have problem to run on GPU. CUDA8.0+cudnn5+4 * 1080ti GPU

[Juicechen95]

My torch code always running on the CPU, not on the GPU.
In my code, I have put my model and tensor on cuda using ':cuda()' and use 'cutorch.setDevice(GPU_num)'.
when I run in the terminal, I add 'CUDA_VISIBLE_DEVICES=1' before th command.
But my code still fails to run on the GPU well. It uses up all the CPU kernels and GPU-Util is always 0.
During the training time, each epoch sometimes takes 0.5s, while sometimes takes 2s,10s.
The running speed is highly relative to the cpu.
I thought the reason would be that my code is running both on CPU and GPU, how can I make it to run on GPU only?