Corrupted memory errors in dbdreader 0.6

[smwoodman]

I'm running into intermittent memory errors when trying to read a large number of sensor values from a large number of dbd/ebd files with dbdreader 0.6.2. These issues are intermittent, ie I can't pinpoint them to specific files, the exact error messages are different, and I don't usually run into them when working with a small number of files or sensors. I don't run into these issues when using v0.5.8, or when using the pure python backend (setting DBDREADER_C_EXTENSION=0). Maybe these are specific to my environment, although I've run into said errors on several separate Linux VMs running Ubuntu 22.04, of varying sizes/specs.

The errors I've seen include:

double free or corruption (!prev)
Aborted (core dumped)
...
free(): invalid size
Aborted (core dumped)
...
free(): invalid next size (normal)
Aborted (core dumped)
...
Fatal glibc error: malloc.c:4179 (_int_malloc): assertion failed: chunk_main_arena (bck->bk)
Aborted (core dumped)

Reproducibility of these errors has been intermittent at best, but again I only run into them when using v0.6 and the C backend. If I'm extracting data for a small number of sensors or files, I usually don't run into errors, but I usually do when extracting data for ~40 sensor values. I confirmed that in all of my test environments, the lz4 package is v4.4.5. Basic script where I see the errors:

...
    sensors = [
        'm_lat', 'm_lon', 'sci_water_cond', 'sci_water_temp', 'sci_water_pressure', 'sci_flbbcd_chlor_units', 
        'sci_flbbcd_cdom_units', 'sci_flbbcd_bb_units', 'sci_oxy4_oxygen', 'sci_oxy4_saturation', 
        'm_heading', 'm_pitch', 'm_roll', 'm_final_water_vx', 'm_final_water_vy', 'm_depth', 'm_battery', 'm_battpos', 
        'm_coulomb_amphr', 'm_coulomb_amphr_total', 'm_de_oil_vol', 'm_leakdetect_voltage', 'm_leakdetect_voltage_forward', 
        'm_leakdetect_voltage_science', 'm_vacuum', 'm_tot_num_inflections', 'm_altitude', 'm_gps_lat', 'm_gps_lon', 
        'c_de_oil_vol', 'c_dive_target_depth', 'c_wpt_lat', 'c_wpt_lon', 
        'sci_flbbcd_chlor_sig', 'sci_flbbcd_cdom_sig', 'sci_flbbcd_bb_sig'
    ] 

    search = "*.[DEde][BCbc][Dd]"
    dbd = dbdreader.MultiDBD(
        pattern=f"{binarydir}/{search}", 
        cacheDir="standard-glider-files/Cache", 
    )
    data_list = [(t, v) for (t, v) in dbd.get(*sensors, return_nans=True)]
    data_time, data = zip(*data_list)
    print(data)

I haven't been able to narrow the errors to specific files - when I write a loop to read each dbd/ebd pair individually, I rarely get an error, and on different files.

Currently I'm running scripts with DBDREADER_C_EXTENSION=0, to avoid the memory issues.

Please let me know if I can provide any other info. I'm also happy to share the script I'm using/full dataset of binary files where I'm reasonably consistently seeing the errors. There are ~1000 files, so please email me at sam.woodman@noaa.gov if that would be useful.

[smerckel]

Hi @smwoodman,

Thanks for pointing this out and testing. Unfortunately I cannot reproduce the issue loading all the dbd/ebd files I have on my laptop. However, I ran the issue through claude code, and it found potential buffer overflow issue when handling the state bytes. This issue is introduced from 0.5.9 to 0.6.0. This explains it is not there in 0.5.x or in the python only version.
I pushed a fix to the master branch, tentatively versioned as 0.6.3dev1. Since I cannot reproduce the issue, would it be possible for you to clone this version and run your test? If it works out fine, then I commit and tag it as 0.6.3.