Adding catalogs section in the pyaml configuration file

[gupichon]

Discussed in https://github.com/orgs/python-accelerator-middle-layer/discussions/45

^{Originally posted by gupichon February 5, 2026}

Proposal: Introducing a Value Catalog in pyAML

As discussed during the workshop, I would like to propose a new concept in pyAML: a catalog of values.

Motivation

Today, control-system references (Tango attributes, strings, metadata, etc.) are spread across device and model definitions. This makes configurations harder to read, maintain, and reuse.

The proposed value catalog addresses this by centralizing and formalizing how such values are defined and accessed.

Objectives

The main goals of this proposal are:

• Centralize all control-system references in a single, well-identified section of the configuration.
• Expose a catalog of values that can be accessed uniformly across pyAML.
• Make these values usable as metadata, enabling easy recording, logging, and export alongside measurements.

Proposed Configuration Structure

With this approach, device and model definitions remain focused on physics and behavior, while all control-system bindings are moved to dedicated catalogs.

A possible configuration would look like this:

type: pyaml.accelerator
facility: ESRF
machine: sr
energy: 6e9

simulators:
  - type: pyaml.lattice.simulator
    lattice: sr/lattices/ebs.mat
    name: design

data_folder: /data/store

arrays:
  - type: pyaml.arrays.serialized_magnet
    name: series
    elements:
      - mySeriesOfMagnets

devices:
  - type: pyaml.magnet.serialized_magnet
    name: mySeriesOfMagnets
    function: B1
    elements:
      - QF8B-C04
      - QF8D-C04
      - QD5D-C04
      - QF6D-C04
      - QF4D-C04
    model:
      type: pyaml.magnet.linear_serialized_model
      calibration_factors: 1.00504
      calibration_offsets: 0.0
      unit: m-1
      curves: sr/magnet_models/quadcurve.yaml

  - type: pyaml.magnet.quadrupole
    name: QF1E-C04
    model:
      type: pyaml.magnet.linear_model
      calibration_factor: 1.00054
      crosstalk: 1.0
      curve:
        type: pyaml.configuration.csvcurve
        file: sr/magnet_models/QF1_strength.csv
      unit: 1/m

  - type: pyaml.magnet.cfm_magnet
    name: SH1A-C01
    mapping:
      - [B0, SH1A-C01-H]
      - [A0, SH1A-C01-V]
      - [A1, SH1A-C01-SQ]
    model:
      type: pyaml.magnet.identity_cfm_model
      multipoles: [B0, A0, A1]
      units: [rad, rad, m-1]
    use-physics: true

Control-System Catalog

All control-system bindings are moved into a dedicated catalog:

control-system-catalog:
  type: pyaml.common.catalog
  refs:
    - reference: mySeriesOfMagnets
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/current
        unit: A

    - reference: QF1E-C04
      device:
        type: tango.pyaml.attribute
        attribute: srmag/vps-qf1/c04-e/current
        unit: A

    - reference: SH1A-C01-H
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/strength
        unit: rad

    - reference: SH1A-C01-V
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch2/strength
        unit: rad

    - reference: SH1A-C01-SQ
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch3/strength
        unit: m-1

User-Defined Data Catalog

The same mechanism can also be used for global values that are not tied to a specific pyaml model:

user-defined-data:
  type: pyaml.common.catalog
  refs:
    - reference: machine-current
      device:
        type: tango.pyaml.attribute
        attribute: ans/ca/machinestatus/current
        unit: mA

    - reference: machine-mode
      device:
        type: tango.pyaml.string_attribute
        attribute: ans/ca/machinestatus/functionMode
        unit: ""

Key Properties of the Catalog

• All catalog entries are identified by a key (reference).

• Any value can be retrieved anywhere in pyAML using:

  • its exact key, or
  • a regular expression (for bulk access and grouping).

• Catalog entries are first-class objects and can be:

  • recorded automatically as metadata,
  • reused by multiple models or tools,
  • queried independently of the device hierarchy.

Open Design Questions

This proposal raises important points that need further discussion:

• How do models declare which catalog keys they depend on?
  In other words, how do we clearly define which values a model expects and how it resolves them from the catalog?
• Whether a validation step is required to ensure that all mandatory catalog entries are present, and at which stage this validation should occur.

[kparasch]

How would the example catalog look like with a CFM magnet that is not based on an identity model? I think this is one of the "trickier" cases.

[gupichon]

How would the example catalog look like with a CFM magnet that is not based on an identity model? I think this is one of the "trickier" cases.

Do you want to use a specific example? Can you put one here?

[gupichon]

But for a short answer, the matching rule between the catalog element and the magnet model must be handled in the magnet model. The catalog should remain a simple key/value store. At most, the most elaborate element it should return would be an array of values.

[kparasch]

e.g. in tests/config/ESRFOrbit.yaml:

- type: pyaml.magnet.cfm_magnet
  name: SH1A-C04
  mapping:
    - [B0, SH1A-C04-H]
    - [A0, SH1A-C04-V]
    - [A1, SH1A-C04-SQ]
    - [B2, SH1A-C04-S]
  model:
    type: pyaml.magnet.linear_cfm_model
    multipoles: [B0,A0,A1,B2]
    pseudo_factors: [1.0,-1.0,-1.0,-1.0]
    units: [rad,rad,m-1,m-2]
    curves:
      - type: pyaml.configuration.csvcurve
        file: sr/magnet_models/SHI_h_strength.csv
      - type: pyaml.configuration.csvcurve
        file: sr/magnet_models/SHI_v_strength.csv
      - type: pyaml.configuration.csvcurve
        file: sr/magnet_models/SHI_sq_strength.csv
      - type: pyaml.configuration.csvcurve
        file: sr/magnet_models/SHI_sext_strength.csv
    matrix:
      type: pyaml.configuration.csvmatrix
      file: sr/magnet_models/SHI_matrix.csv
    powerconverters:
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c04-a-ch1/current
        unit: A
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c04-a-ch2/current
        unit: A
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c04-a-ch3/current
        unit: A
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c04-a-ch5/current
        unit: A
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c04-a-ch6/current
        unit: A

I guess the "powerconverters" section would be moved to the catalog with some additional names as references?

[gupichon]

Yes, it will look for a set of devices. In this case, it will retrieve five devices.

[JeanLucPons]

I'm also interested on how it will work for BPM and especially how the Catalog pydnatic model look like.
I proposed this structure for various references in catalog.
I expect that the catalog can handle several models and 1 basic model to access one (or more) devices on the fly.
Fell free to improve it and discuss.

PYAMLCLASS = "BPMSimpleModel"

class CatalogModel(BaseModel):

    """
    Configuration model for BPM devices

    Parameters
    ----------
   reference: str
       Name of the BPM element
    x_pos : DeviceAccess
        Horizontal position device
    y_pos : DeviceAccess
        Vertical position device
    """
    model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")

    reference: str
    x_pos: DeviceAccess | None = None
    y_pos: DeviceAccess | None = None

class ConfigModel(BaseModel):
    """
    Configuration model for BPM simple model

    x_pos_index : int, optional
        Index in the array when specified, otherwise scalar
        value is expected
    y_pos_index : int, optional
        Index in the array when specified, otherwise scalar
        value is expected
    """
    model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")

    x_pos_index: int | None = None
    y_pos_index: int | None = None

Same question when BPMOffset and Tilt will be there.
I really would like to avoid confusing situation where you can have 1,2,3,4 or 5 devices for a BPM without knowing which ones are for position(s),offset(s) and tilt.
We can have also stripline based BPM that return only one single position.