Fixing the device assignment issues during inference (test_batch) in Sortformer model

Fixing the device assignment issues during inference (test_batch) in Sortformer model #7053