Fix-HDF5-buffered-reads-and-writes-for-shape-datasets

[prasad-sawantdesai]

fixes a crash when reading some HDF5 shape datasets through the
buffered read path also fixes the same indexing problem in the buffered write path.

ids.grid_ggd[0].space[0].objects_per_dimension[3].object[0].geometry[0]

    h5ls -r "/home/ITER/pankina/public/d3d/4/163518/75/mhd_1.h5" | grep 'grid_ggd\[\]&space\[\]&objects_per_dimension\[\]&object\[\]&geometry'
    /mhd_1/grid_ggd[]&space[]&objects_per_dimension[]&object[]&geometry Dataset {2/Inf, 1, 4, 3702416/Inf, 3}
    /mhd_1/grid_ggd[]&space[]&objects_per_dimension[]&object[]&geometry_SHAPE Dataset {2/Inf, 1, 4, 3702416/Inf}

geometry stores the actual values and has 5 dimensions

[grid_ggd, space, objects_per_dimension, object, geometry_value]

geometry_SHAPE stores the length of geometry for each object. It has
only 4 dimensions::

[grid_ggd, space, objects_per_dimension, object]

indices.push_back(0); always added one extra index when reading or writing 1D data.
That is correct for geometry, but wrong for geometry_SHAPE.

checks the dataset rank before adding the extra index.

• If the dataset needs one more index, add 0.
• If the dataset already has enough indices, do not add anything.
• If the number of indices is still wrong, throw an exception

$ python test.py
16:14:37 INFO     Parsing data dictionary version 4.1.1 @dd_zip.py:89
╭─────────────────── IDS structure: grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0] (DD version 4.1.1) ────────────────────╮
│ Set of objects for a given dimension                                                                                             │
│ ╭───────────────────────────────────────────────────────── Attributes ─────────────────────────────────────────────────────────╮ │
│ │ metadata = <IDSMetadata for 'object'>                                                                                        │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ ╭──────────────────────────────────────────────────────── Child nodes ─────────────────────────────────────────────────────────╮ │
│ │    boundary = <IDSStructArray (IDS:mhd, grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0]/boundary with 5 items)>      │ │
│ │    geometry = <IDSNumericArray (IDS:mhd, grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0]/geometry, empty FLT_1D)>    │ │
│ │ geometry_2d = <IDSNumericArray (IDS:mhd, grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0]/geometry_2d, empty FLT_2D)> │ │
│ │     measure = <IDSFloat0D (IDS:mhd, grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0]/measure, empty FLT_0D)>          │ │
│ │       nodes = <IDSNumericArray (IDS:mhd, grid_ggd[0]/space[0]/objects_per_dimension[3]/object[0]/nodes, INT_1D)>             │ │
│ │               numpy.ndarray([     1,      2,    643, 231402, 231403, 232044], dtype=int32)                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[prasad-sawantdesai]

Need to test with Fortran code writing this file and check if this issue appears again.

imas:hdf5?path=/home/ITER/pankina/public/d3d/4/163518/75

[fleuryl-ai]

It was not an easy bug... The patch seems fine. Thanks a lot.

[olivhoenen]

It was not an easy bug... The patch seems fine. Thanks a lot.

This is still under investigation, as it is not clear if the fix is hiding the problem. Looking at some of the shapes of the HDF5 datasets there seem to be odd things, possibly due to inconsistent AoS sizes inside two time slices

• we may be facing the same issue than described in https://jira.iter.org/browse/IMAS-5785
• reading the first slice seems to work, so the current suspicion is that when writing inconsistent second slice the HDF5 backend messes up the shapes (some tensors expected to be 5D only have 4D)
• current patch can't be accepted, it could raise a proper exception but we should not silently return with no data while there are clearly some issues in the datasets shapes
• maybe @pankin you can re-create this mhd IDS with a single time slice only, so we can gain confidence in the content of the GGD before complexifying with multiple slices?

[prasad-sawantdesai]

Between I could not reproduce the segfault with the files I have created in Python and Fortran.
create_d3d_like_mhd.py
--> The Python process is being terminated while executing put_slice

create_d3d_like_mhd_f90.txt
---> This could generate the file

import imas

entry = imas.DBEntry("imas:hdf5?path=/home/ITER/sawantp1/github/IMAS-Core/testdata/d3d_mhd_fortran", "r")

ids = entry.get("mhd", occurrence=0, lazy=True, autoconvert=False)
ids.grid_ggd[0].space[0].objects_per_dimension[3].object[0].geometry[0]

$ python test.py
16:21:32 INFO     Parsing data dictionary version 4.1.0 @dd_zip.py:89
Traceback (most recent call last):
  File "/home/ITER/sawantp1/github/IMAS-Core/test.py", line 8, in <module>
    ids.grid_ggd[0].space[0].objects_per_dimension[3].object[0].geometry[0]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/work/imas/opt/EasyBuild/software/IMAS-Python/2.2.0-intel-2023b/lib/python3.11/site-packages/imas/ids_primitive.py", line 224, in __getitem__
    return self.value[index]
           ~~~~~~~~~~^^^^^^^
IndexError: index 0 is out of bounds for axis 0 with size 0
$ module list 2>&1 | grep IMAS
 15) libxml2/2.11.5-GCCcore-13.2.0            43) IMAS-Core/5.5.2-intel-2023b
 16) ncurses/6.4-GCCcore-13.2.0               44) IMAS-MDSplus-models/5.5.2-GCCcore-13.2.0-DD-4.1.0
 17) libreadline/8.2-GCCcore-13.2.0           45) IMAS-Data-Dictionary/4.1.0-GCCcore-13.2.0
 18) Java/21 -> Java/21.0.5                   46) IMAS-Fortran/5.5.0-intel-2023b-DD-4.1.0
 27) fmt/10.2.1-GCCcore-13.2.0                55) IMAS-Python/2.2.0-intel-2023b
$

It seems that files were created in MDSplus and then converted to HDF5 with idscp
From mhd_1.h5
IMAS-Python 2.1.0.post1
Access Layer 5.5.3

@pankin

• Could you please provide path of MDSplus files if there are any..
• Could you please point out codebase writing mhd IDS.

Thank you..

[pankin]

@pankin

• Could you please provide path of MDSplus files if there are any..
• Could you please point out codebase writing mhd IDS.

@prasad-sawantdesai We don't save the data in mdsplus. The tools that we have developed for NIMROD are specifically developed for hdf5 backend. You can find the scripts here:

https://github.com/PrincetonUniversity/nimrod2imas

[prasad-sawantdesai]

@pankin Thank you for sharing code
It seems that IMAS data entry is written with h5py with custom data sets mentioned here and it may have stored wrong ranks of datasets.
https://github.com/PrincetonUniversity/nimrod2imas/blob/main/dump2imas.py#L232
"/home/ITER/pankina/public/d3d/4/163518/75/mhd_1.h5"

I think it is good to fix ranks of the datasets in https://github.com/PrincetonUniversity/nimrod2imas/blob/main/dump2imas.py

[prasad-sawantdesai]

Checked single time slice and found similar behavior. (/home/ITER/pankina/public/d3d/4/163518/99)

$ python
Python 3.11.5 (main, Jul 30 2025, 19:17:56) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import imas
>>> entry = imas.DBEntry("imas:hdf5?path=/home/ITER/pankina/public/d3d/4/163518/99", "r")
>>> ids = entry.get("mhd", occurrence=1, lazy=True, autoconvert=False)
10:03:59 INFO     Parsing data dictionary version 4.1.1 @dd_zip.py:89
>>> ids.grid_ggd[0].space[0].objects_per_dimension[3].object[0].geometry[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ITER/sawantp1/github/IMAS-Core/myenv/lib/python3.11/site-packages/imas/ids_structure.py", line 67, in __getattr__
    self._lazy_context.get_child(child)
  File "/home/ITER/sawantp1/github/IMAS-Core/myenv/lib/python3.11/site-packages/imas/backends/imas_core/al_context.py", line 294, in get_child
    imas.backends.imas_core.db_entry_helpers._get_child(child, self)
  File "/home/ITER/sawantp1/github/IMAS-Core/myenv/lib/python3.11/site-packages/imas/backends/imas_core/db_entry_helpers.py", line 118, in _get_child
    data = ctx.get_context().read_data(new_path, timebase, data_type.al_type, ndim)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ITER/sawantp1/github/IMAS-Core/myenv/lib/python3.11/site-packages/imas/backends/imas_core/al_context.py", line 149, in read_data
    status, data = ll_interface.read_data(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "imas_core/_al_lowlevel.pyx", line 989, in imas_core._al_lowlevel.al_read_data
  File "imas_core/_al_lowlevel.pyx", line 889, in imas_core._al_lowlevel.al_read_data_array
imas_core.exception.ImasCoreBackendException: Error while reading data: al_plugin_read_data: [ALBackendException = HDF5Backend: SHAPE dataset 'grid_ggd[]&space[]&objects_per_dimension[]&object[]&geometry_SHAPE' has rank 4 but rank 5 was expected. ]
Error status=-3