Results are not stable, even with fixed seed

[yamins81]

I've noticed that output results are not stable, meaning that different runs produce the different results, even with the same initial conditions. Specifically, even with all random seeds held constant, and with the FIFO queue, although initial filter values at t = 0 are identical between runs, they have diverged by t = 1. I haven't been able isolate the source of the divergence, including whether it's the result of numerical error.

[qbilius]

Could it be due to us running train and validation queues at the same time? Maybe they get mixed up somehow? Have you tried disabling validation?