RDMA_CM_EVENT_ADDR_ERROR raised when running distributed training with PyTorch

[anj-s]

Describe the bug
I am unable to get distributed training running with PyTorch backend. I am consistently running into the RDMA_CM_EVENT_ADDR_ERROR. Can someone take a look and let me know if I am missing something?

Run setup: 2 nodes
node 0: worker 0
node 1: worker 1, server, scheduler

scheduler_hostname = IP of the RDMA interface

perf test works using ib_write_bw
single node training works

To Reproduce
General env vars that are set on workers, scheduler and server
os.environ["DMLC_ENABLE_RDMA"] = "ibverbs"
os.environ["DMLC_INTERFACE"] = "front0"
os.environ["ENABLE_RDMA_LOG"] = "1"
os.environ["PS_VERBOSE"] = "1"
os.environ["BYTEPS_LOG_LEVEL"] = "TRACE"
os.environ["NCCL_DEBUG"] = "INFO"
os.environ["NCCL_SHM_DISABLE"] = "1"
os.environ["BYTEPS_ENABLE_GDB"] = "0"
os.environ["BYTEPS_RDMA_RX_DEPTH"]="128"
os.environ["BYTEPS_RDMA_START_DEPTH"]="16"

server env vars
os.environ["DMLC_ROLE"] = "server"
os.environ["DMLC_NUM_WORKER"] = 2
os.environ["DMLC_NUM_SERVER"] = 1
os.environ["DMLC_PS_ROOT_URI"] = scheduler_hostname
os.environ["DMLC_PS_ROOT_PORT"] = SCHEDULER_PORT

scheduler env vars
os.environ["DMLC_ROLE"] = "scheduler"
os.environ["DMLC_NUM_WORKER"] = "2"
os.environ["DMLC_NUM_SERVER"] = "1"
os.environ["DMLC_PS_ROOT_URI"] = scheduler_hostname
os.environ["DMLC_PS_ROOT_PORT"] = SCHEDULER_PORT

worker env vars
os.environ["DMLC_ROLE"] = "worker"
os.environ["DMLC_WORKER_ID"] = str(worker_id)
os.environ["DMLC_NUM_WORKER"] = "2"
os.environ["DMLC_NUM_SERVER"] = "1"
os.environ["DMLC_PS_ROOT_URI"] = scheduler_hostname
os.environ["DMLC_PS_ROOT_PORT"] = SCHEDULER_PORT
os.environ["BYTEPS_LOCAL_RANK"] = "0"
os.environ["BYTEPS_LOCAL_SIZE"] = "1'

Expected behavior
Able to run:

command = "python /private/home/anj/.conda/envs/fairscale/bin/bpslaunch python
                       /private/home/anj/byteps_repro/byteps/example/pytorch/train_mnist_byteps.py"
subprocess.check_call(command,
                      stdout=sys.stdout, stderr=sys.stderr, shell=True)

stack trace: https://gist.github.com/anj-s/6c808731287e9a504cb63c6f8013fad0

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):
OS: Ubuntu
GCC version: gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)
CUDA and NCCL version:
CUDA: 11.0
NCCL: 2.7.8
Framework (TF, PyTorch, MXNet): PyTorch 1.8

Additional context
Add any other context about the problem here.