Unable to transform tables > 2GiB

[jordanpadams]

Issue found user:

Hi,

I’m trying to use the transform tool to convert some JUNO/JADE data from PDS3 format to CSV. I was able to run the tool successfully on the low rate data, however, the tool failed on the high rate data (perhaps the data file is too large in size?). The file in question is (https://pds-ppi.igpp.ucla.edu/search/view/?f=null&id=pds://PPI/JNO-J_SW-JAD-3-CALIBRATED-V1.0/DATA/2016/2016240/ELECTRONS/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01&o=1). When running the tool, I’m getting the output:

PDS Transform Tool Log

Version Version 1.8.0

Time Mon, Apr 29 2019 at 07:42:19 AM

Target file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL

Output Directory /Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0

Index 1

Format Type csv

INFO: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL Transforming label file: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL

INFO: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL Finished transforming PDS3 label 'file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL' to a PDS4 label '/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/jad_l30_hrs_elc_all_cnt_2016240_v01.xml'

INFO: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/jad_l30_hrs_elc_all_cnt_2016240_v01.xml Transforming table '1' of file: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.DAT

ERROR: [/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/jad_l30_hrs_elc_all_cnt_2016240_v01.xml] Error occurred while transforming table: Error occurred while reading table '1' of file 'file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.DAT': capacity < 0: (-1040993352 < 0)

ERROR: file:/Users/xzhu/Downloads/JNO-J_SW-JAD-3-CALIBRATED-V1.0/JAD_L30_HRS_ELC_ALL_CNT_2016240_V01.LBL Unsuccessful table transformation. Check transformed PDS4 label for possible errors.

Thanks,

Brian Xiaoyu Zhu

Possible clue towards solution:

Look into the following clue:

the transform program must load the number of bytes in the binary file as an integer. However, because the file size was too large, the integer overflowed and became negative. Changing the capacity variable to long int would likely solve this particular problem.

[jordanpadams]

Duplicate of NASA-PDS/pds4-jparser#21