GridFTP Authentication Hangs

[65sc02]

Hello!

I know today is the last day of January, but I hope you can shed some light and give some help despite the imminent end of support.

We are trying to use GridFTP (over a private network, so not reachable from the normal internet, thus no GlobusOnline possible) to transfer huge amounts of data between two European computing centers: BSC and LRZ. However, GridFTP hangs during the authentication phase without giving an error message. There is also no useful information in the server log files.

To make things more confusing, we also tried with a 3rd European center. They have two machines, and transfers from BSC work with one machine while the other hangs, while transfers from LRZ work with both machines just fine.

We are at our wit's end and desperately need your help! Let me summarize information for the two sites. If needed, I can also supply this for the 3rd site, but I don't want to make it too confusing.

@bsc (from BSC to LRZ):
client (@bsc) globus-version: 6.0.1502136246
client (@bsc) globus-version: 5.2.4 did also not work
GridFTP server: globus-gridftp-server -version
globus_gridftp_server: 12.2 (1502136246-85)

command tried:
globus-url-copy -list gsiftp://LRZ/
=> hangs in authentication

@lrz (from LRZ to BSC):
globus-version: 6.0.1453307864
GridFTP server:
220 XXX.XXX.XXX.XXX GridFTP Server 9.4 (gcc64, 1453307864-85) [Globus Toolkit 6.0.1453307864] ready.

command tried:
globus-url-copy -dbg -list gsiftp://BSC/

output:
gsiftp://BSC/
debug: starting to feat gsiftp://BSC/
debug: connecting to gsiftp://BSC/
debug: response from gsiftp://BSC/:
220-**
220-** Globus/GridFTP Service at PRACE-BSC
220-**
220-** HelpDesk: blablabla
220-**
220 End.

debug: authenticating with gsiftp://BSC/

The frontend log at BSC says for this connection only the following (log-level set to ALL):
[18333] Wed Jan 31 12:26:59 2018 :: Configuration read from /etc/gridftpd/gridftpd_frontend.conf.
[18333] Wed Jan 31 12:26:59 2018 :: Server started in inetd mode.
[18333] Wed Jan 31 12:26:59 2018 :: New connection from: LRZ:59371

We are really at a loss here. All the obvious things have been excluded:

• the DN used is known and valid at both sides. Tested via the 3rd site which works with this DN and cert to/from BSC and LRZ.
• it is not a firewall issue at BSC: communication to BSC port 2811 was also tested with telnet: fine. We even temporarily turned the firewall OFF: no change, still does not work.
• it is not a firewall issue at LRZ: we tested connection to BSC on port 2811 with telnet and it works fine. Also the initial contact is established, which speaks against a firewall problem.
• BSC's installation can't be totally hosed since it works fine from one other site
• LRZ's installation can't be totally hosed since it works fine from many other sites
• we even did TCPdumps from a working connection to BSC and a non-working connection to BSC. If needed, we can provide those. There is no obvious difference, it just seems to stop for no apparent reason. TCPdump says that all packets from BSC have an incorrect checksum, but this is probably due to offloading to the NIC and should be no cause for worries.

Can you give us any hint where to look? What could be going wrong?Are there any known incompatibilities with the various Globus/GridFTP versions?

Thanks so much
Helmut

[j-rodrigu]

The issue was solved by modifying the MTU on the network interface at the endpoint server at BSC.