Training time is too long

[jasonbau]

I'm training 17000 pictures on GCP compute engine.
Computer engine infomation:
CPU & Memory: n1-highmem-8 (8 vCPU，52 GB memory)
GPU: 8 x NVIDIA Tesla K80
Disk: 100G SSD

I've already executed the python3 watermarks.py --logdir=save/ until now.
but it has not completed yet. Is there any way to get faster?

[arnaudoff]

Same issue here. Compiling Tensorflow with SSE and AVX instructions as recommended seems to have improved the performance on my machine, but not significantly.

[kmrabhay]

I fixed it by removing while True: part in evaluation part(Line after saving the model) which was causing the evaluation to process to go infinite loop resulting training never complete beyond one checkpoint

[kmrabhay]

Fixed it by removing while True: part in evaluation part(Line after saving the model) which was causing the evaluation process to go infinite loop resulting training never complete beyond one checkpoint after saving it.

[arnaudoff]

I haven't changed anything in the code, but training time decreased dramatically after building Tensorflow myself, but now with the -march=native flag. I guess it does better job at optimization than manually setting up what instruction set to use, so you guys might try that and see if it works for you as well.