Metadata model

.github/workflows/check_pull_request.yml

@@ -46,6 +46,62 @@ jobs:
           ./run test
 
 
+  test_without_plotly:
+    name: Core installation without Plotly (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12", "3.15"]
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv and set the Python version
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+        with:
+          version: ">=0.13.4"
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install the core package without extras or development dependencies
+        run: uv sync --locked --no-dev --no-editable
+
+      - name: Check core functionality and optional plotting exports
+        run: |
+          uv run --no-sync python -I - <<'PY'
+          import importlib.util
+
+          assert importlib.util.find_spec("plotly") is None, "Plotly must not be installed"
+
+          import pandas as pd
+          import hdhelpers
+          from hdhelpers.metadata import attach_metadata, attrs_to_json, get_series_unit, read_metadata
+          from hdhelpers.metadata.conventions import SERIES_METADATA_EXAMPLE
+
+          series = pd.Series([1.0], index=pd.to_datetime(["2026-01-01T00:00:00Z"]))
+          series.attrs = SERIES_METADATA_EXAMPLE
+          meta = attach_metadata(series)
+          assert meta is not None
+          assert read_metadata(series) is meta
+          assert get_series_unit(series) == "m"
+          assert attrs_to_json(series.attrs)["metrics"][0]["value_dimensions"][0]["unit"] == "m"
+          converted = hdhelpers.modify_timezone(series, to_timezone="Europe/Berlin")
+          assert str(converted.index.tz) == "Europe/Berlin"
+          assert read_metadata(converted) is meta
+
+          # Repeat every export to catch modules cached by an earlier failed import.
+          names = (
+              "plotting", "FUSEKI_COLOR_CYCLE", "resolve_color",
+              "set_agnostic_theme", "TICKFORMATSTOPS", "set_dt_ticks",
+          )
+          for name in names * 2:
+              try:
+                  getattr(hdhelpers, name)
+              except ImportError as error:
+                  assert "hdhelpers[plotting]" in str(error), (name, error)
+              else:
+                  raise AssertionError(f"hdhelpers.{name} is available without Plotly")
+          PY
+
   check_lint_format_mypy:
     runs-on: ubuntu-latest
     steps:

CHANGELOG.md

@@ -1,22 +1,12 @@
 ## [Unreleased]
-- Update python versions to fix CA certificate issue (https://emudhra.we-have-a-problem.com/)
-- metadata accessors can fall back to the properties a metric inherits from objects higher up in an asset hierarchy (under "inherited" in the metric metadata of the current convention), only as last resort after "value_dimensions_shared". This is the default for get_metric_info, get_singlets_metric_info, get_series_info and get_singlets_info. The functions for dedicated information (units, names, display names, short display names and measurements) only do so with keyword argument include_inherited=True, since an inherited property with the same label usually does not describe the metric itself
-- get_metric_info and get_singlets_metric_info fall back to the value dimension "value" and to "value_dimensions_shared" if the metric itself does not provide the information
-- fix: (short) display names of metadata following the current convention treat null as missing and look up each field on the value dimension, the metric and "value_dimensions_shared" before falling back to the next field. So e.g. a display name of the metric wins over the name of its value dimension
-- fix: get_metric_info returns None entries instead of raising if the metadata contains no metric information
-- the series accessors determine their metric like the SingleTSFrame accessors: the one given by "single_metric" in the dataset metadata or, if that is missing, the only metric present. The metric named "series" is only used as default for older metadata structures
-- a "single_metric" in the dataset metadata which is no string counts as missing. Before, e.g. a list let all accessors of Series and SingleTSFrames raise an error
-- fix: get_series_display_name no longer falls back to the short display name, as documented and like get_display_names and get_singlets_display_names
-- fix: the SingleTSFrame accessors no longer use the only metric present if it is another one than the metric given by "single_metric" in the dataset metadata. The information counts as missing then
-- metadata following the current convention is extracted in a single pass in plain Python instead of evaluating glom specs per level and field. The accessors are considerably faster for frames with many metrics, e.g. about 100 times for 1000 metrics when requesting field names or dotted paths. Other glom specs are still evaluated by glom, but only on the levels needed
-- the accessors of Series and SingleTSFrames only resolve the information of their single metric. So they are considerably faster for a Series carrying the metadata of many metrics, e.g. taken from a MultiTSFrame, and a spec is not applied to the other metrics. Also get_series_unit and get_series_name no longer evaluate glom specs for metadata following the current convention. Both are about 90 times faster for the metadata of 1000 metrics
-- the requested information is only looked up on the levels needed, so e.g. a spec transforming the information is no longer applied to null values of levels which are not used anyway
-- entries of metadata following the current convention which do not adhere to the conventions are skipped: metrics which are no objects or whose field named by the metric key is no string, and value dimensions which are no objects or whose column is no string. "value_dimensions" and "value_dimensions_shared" which are no lists count as empty, a metric key which is no string as missing. Before, a single such entry could make all information missing or raise an error
-- new functions get_default_timezones, get_series_default_timezone and get_singlets_default_timezone provide the default time zone of the metrics ("default_timezone" of the current metadata convention), falling back to the one of the dataset metadata. It is meant for computations, e.g. bucketing days when aggregating, not for displaying timestamps, which is what the time zone of the plot target settings is for. New section "Which time zone to compute in?" in the documentation
-- documentation: the information requested from get_metric_info, get_singlets_metric_info, get_series_info and get_singlets_info is interpreted as a glom spec. New section "How to query nested information?" with examples for dotted paths into sub dictionaries, other glom specs and keys containing dots
-- new class Recording and functions get_recordings, get_series_recording and get_singlets_recording provide how the values of the metrics are recorded: "recording_mode", "data_frequency" with "data_frequency_offset" and "data_frequency_timezone", and "recording_aggregation_method" of the current metadata convention. The frequency falls back to "ref_data_frequency", "ref_data_frequency_offset" and "ref_data_frequency_timezone" of the dataset metadata as a whole, i.e. offset and time zone only apply together with the frequency of the same level. The values are provided as given in the metadata, the documentation of Recording explains what they mean
-- new functions get_default_aggregation_methods, get_series_default_aggregation_method and get_singlets_default_aggregation_method provide the default aggregation method of the metrics ("default_aggregation_method"), normalizing "AVG" to "MEAN". Without one, it is "LAST" for metrics whose value dimension "value" has the value data type "boolean" or "string", otherwise "MEAN"
-- new function without_metadata provides a copy of a Series or DataFrame without its metadata (attrs) to compute on. pandas deep-copies the attrs to every object an operation creates, which makes e.g. loops over the metrics of a MultiTSFrame with the metadata of many metrics slow. The copy shares the data until it is changed
+
+- Add validated, immutable metadata models with helpers to read, attach, update, and serialize them. Pandas operations share attached models instead of repeatedly copying metadata.
+- Improve metadata lookups with consistent fallbacks for missing values, display names, and inherited properties (`include_inherited=True` for dedicated unit/name helpers).
+- Add helpers for default time zones, recording information, default aggregation methods, and copying pandas objects without metadata (`copy_without_metadata`).
+- **Breaking:** Only the current metadata conventions are supported. Invalid metadata returns no information. Legacy formats and arbitrary glom specs are removed; queries accept field names and dotted paths.
+- Make Plotly optional: install `hdhelpers[plotting]` for plotting helpers. Core metadata and time zone functionality works without it.
+- Add Python 3.15 support and update Python/tooling requirements to address the CA certificate security issue.
+- Expand validation and regression tests, correct lint configuration, and add CI checks of actual installations without Plotly.
 
 ## [0.0.9] - 2026-09-10
 - singletsframe support

README.md

@@ -18,6 +18,9 @@ The documentation of the package is a [GitHub page](https://hetida.github.io/hdh
 ## Getting Started with hdhelpers
 Since version >0.13.10 the hetida designer runtime comes with an installed version of hdhelpers. To start the hetida designer we recommend following the [hetida designer setup guide](https://github.com/hetida/hetida-designer/blob/release/README.md#getting-started-with-hetida-designer).
 
+The plotting helpers (`hdhelpers.plotting` and their names in `hdhelpers`) need plotly, which is the optional extra
+`plotting`: `pip install "hdhelpers[plotting]"`. Everything else, e.g. the metadata models, works without plotly.
+
 An example is given in [GitHub page](https://hetida.github.io/hdhelpers/first_steps.html#how-to-get-metadata-with-hdhelpers) of how to use functionalities of hdhelpers inside a hetida designer component. Furthermore the base component [Single Time Series Plot](https://github.com/hetida/hetida-designer/blob/release/runtime/transformations/components/visualization) uses hdhelpers to demonstrate the usage (since version >0.13.10).
 
 ## Developing for hdhelpers
@@ -27,7 +30,8 @@ For dependency management and venv setup, building and publishing, [uv](https://
 #### Python environment
 1) Create a virtual environment with `uv venv`. This will create a hidden `.venv` directory.
 2) Activate the virtual environment via `source .venv/bin/activate`
-3) Run `uv sync` to install all dependencies given in pyproject.toml.
+3) Run `uv sync --all-extras` to install all dependencies given in pyproject.toml, including the optional
+   ones of the plotting helpers.
 4) In case you need to add a new dependency, do so via `uv add <new_dependency>`. That way, uv finds versions of all
    dependencies that are compatible with each other.
 5) In case you need a new requirement for development purposes please use `uv add --dev <new_dependency>`
@@ -43,7 +47,7 @@ FROM hetida/designer-backend:<<hd_version>>
 
 USER root
 
-RUN pip install .
+RUN pip install ".[plotting]"
 
 USER hd_app
 
@@ -69,6 +73,34 @@ writing component code.
 ### Code Quality
 Once you are done writing your code, including unit tests, use `./run check` to see if your code quality is sufficient.
 
+#### Metadata schema snapshot
+`tests/metadata/test_conventions_schema.py` compares the generated metadata JSON schemas with
+`tests/data/convention_schemas.json`. Class docstrings become schema descriptions, so documentation
+changes can also require a snapshot update.
+
+After an intentional schema change, regenerate the snapshot in Bash/WSL:
+
+```bash
+UPDATE_SCHEMA_SNAPSHOT=1 uv run pytest tests/metadata/test_conventions_schema.py -q
+```
+
+Or in PowerShell:
+
+```powershell
+$env:UPDATE_SCHEMA_SNAPSHOT = "1"
+uv run pytest tests/metadata/test_conventions_schema.py -q
+Remove-Item Env:UPDATE_SCHEMA_SNAPSHOT
+```
+
+Review the diff for unintended changes and rerun the test without the update flag:
+
+```bash
+git diff -- tests/data/convention_schemas.json
+uv run pytest tests/metadata/test_conventions_schema.py -q
+```
+
+Commit the updated snapshot together with the corresponding model or docstring changes.
+
 ### Documentation
 For documentation we use the tool sphinx. Please apply `./run build_docs` to create the current state of documentation. It will be stored in **docs**. You can open the documentation by opening `docs/index.html`, e.g. with your browser.