g-w CI C96C48_hybatmaerosnowDA fails on WCOSS2

[RussTreadon-NOAA]

When g-w CI C96C48_hybatmaerosnowDA is run using g-w PR #2978, the following jobs abort on WCOSS2 (Cactus)

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/praero_pr2978
202112201200        gdas_aeroanlgenb                   158495445                DEAD                 -29         2        1849.0
202112201800            gdas_snowanl                   158472339                DEAD                   1         2          58.0

gdas_aeranlgenb aborts while executing gdas.x fv3jedi convertstate using chem_convertstate.yaml

nid003614.cactus.wcoss2.ncep.noaa.gov 0: Converting state 1 of 1
nid003614.cactus.wcoss2.ncep.noaa.gov 32:
FATAL from PE    32: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./bkg/20211220.180000.anlres.fv_tracer.res.tile3.nc variable:xaxis_1

nid003614.cactus.wcoss2.ncep.noaa.gov 64:
FATAL from PE    64: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./bkg/20211220.180000.anlres.fv_tracer.res.tile5.nc variable:xaxis_1

gdas_snowanl aborts while executing gdas.x fv3jedi localensembleda using letkfoi.yaml

nid001614.cactus.wcoss2.ncep.noaa.gov 0: Local solver completed.
OOPS_STATS LocalEnsembleDA after solver             - Runtime:     10.48 sec,  Memory: total:     8.43 Gb, per task: min =     1.40 Gb, max =     1.41 Gb
nid001614.cactus.wcoss2.ncep.noaa.gov 0:
FATAL from PE     0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./anl/snowinc.20211220.180000.sfc_data.tile1.nc variable:xaxis_1

nid001614.cactus.wcoss2.ncep.noaa.gov 0:
FATAL from PE     0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./anl/snowinc.20211220.180000.sfc_data.tile1.nc variable:xaxis_1

The error message is the same for both failures.

These jobs successfully run to completion on Hera, Hercules, and Orion. GDASApp is built with newer intel compilers and different modules on these machines. It is not clear if the older intel/19 compiler or modulefules used on Cactus are the issue or if there is an actual bug in the JEDI code which needs to be fixed.

This issue is opened to document the WCOSS2 failure and its resolution.