added endpoint for data retrieval

[prasad-sawantdesai]

Temporal endpoint for data retrieval from IMAS data entries as metadata doesn't have arrays

curl -s "http://localhost:5000/v1.2/simulation/bb47e1d63e5111f1a85
548df37d69a88/data?path=summary/global_quantities/ip/value"

[maarten-ic]

Hi Prasad,

I'm also a bit confused about the usecases you're trying to support. I had expected this would be a simple way to extract the summary time traces (1D) or 0D data from the IDSs stored on disk.
However, the logic in some places is more complicated than that (e.g. _to_python seems to support arbitrary dimension arrays, but only for floats).

If this is only intended for 0D or 1D data, then I'd explicitly indicate that (and give an error when trying to retrieve higher dimensional data).

See also some of the specific comments below. Let me know if you have any questions.

[deepakmaroo]

Hello,

I would suggest to use exiting API and enhance to achieve this requirement.

API https://simdb.iter.org/scenarios/api/v1.2/simulation/simid returns metadata, IMAS-URI in inputs and IMAS-URI in outputs.

In API implementation when preparing metadata response, check if element value has min and max call imas-python (IMAS-URI+metadata-element) or any other better method and attach value with min and max JSON i.e.
{"global_quantities/ip/value":{ "min":"0", "max":"5", "value": { "_type": "numpy.ndarray", "bytes": "BrjFoSNuDsA=", "dtype": "float64" }} }

We don't have to call API for each selected quantity.

[prasad-sawantdesai]

Hello,

I would suggest to use exiting API and enhance to achieve this requirement.

API https://simdb.iter.org/scenarios/api/v1.2/simulation/simid returns metadata, IMAS-URI in inputs and IMAS-URI in outputs.

In API implementation when preparing metadata response, check if element value has min and max call imas-python (IMAS-URI+metadata-element) or any other better method and attach value with min and max JSON i.e. {"global_quantities/ip/value":{ "min":"0", "max":"5", "value": { "_type": "numpy.ndarray", "bytes": "BrjFoSNuDsA=", "dtype": "float64" }} }

We don't have to call API for each selected quantity.

Hi @deepakmaroo,
Thanks for the suggestion. Added this temporal /data endpoint so that metadata loaded quickly, and then on-demand data fetched only when the user requests a specific field with 1D array.
Few points:
"Use existing API and enhance" - The existing GET /simulation/<sim_id> is a database query and adding IMAS file reads to it will make every simulation page load disk-bound, even when the user never requests a plot.
"We don't have to call API for each selected quantity" - On the dashboard, the user selects one quantity at a time to plot. So one API call per quantity looks ok to me - there is no batch case to optimize for time being.
"Return {min, max, value} in metadata response" - This will embed large numpy arrays into the simulation metadata JSON on every page load, wasting bandwidth even when the user never ask for a plot.

[Note: This endpoint will be unused or removed when ibex backend is available on the server]

[prasad-sawantdesai]

Hi Prasad,

I'm also a bit confused about the usecases you're trying to support. I had expected this would be a simple way to extract the summary time traces (1D) or 0D data from the IDSs stored on disk. However, the logic in some places is more complicated than that (e.g. _to_python seems to support arbitrary dimension arrays, but only for floats).

If this is only intended for 0D or 1D data, then I'd explicitly indicate that (and give an error when trying to retrieve higher dimensional data).

See also some of the specific comments below. Let me know if you have any questions.

Hello Maarten,

Here, usecase is simply extracting summary IDS fields (0D/1D). I agree it needs clean up and also should give an error when trying to retrieve higher dimensional data.

Thanks

[olivhoenen]

While testing the test server with a client based on the same version (both based on this branch) works so far as expected, if I used a client from the last release 0.15.1 I get an error message that does not clearly identify the problem and will come as a surprise to the user:

[hoeneno@98dci4-srv-1002 simdb]$ simdb remote TEST list
Error: Corrupted data - non-optional metadata is None.

@Yannicked @maarten-ic, while I can't rule out that the problem comes from this PR or from our install in the test server, it's most probably an issue present already in develop (since the merge of JSON column?).

[Yannicked]

While testing the test server with a client based on the same version (both based on this branch) works so far as expected, if I used a client from the last release 0.15.1 I get an error message that does not clearly identify the problem and will come as a surprise to the user:

[hoeneno@98dci4-srv-1002 simdb]$ simdb remote TEST list
Error: Corrupted data - non-optional metadata is None.

@Yannicked @maarten-ic, while I can't rule out that the problem comes from this PR or from our install in the test server, it's most probably an issue present already in develop (since the merge of JSON column?).

Hi Olivier this seems to be an issue in develop. I've created a fix for it: #87