[OSSIE][SIGMA] Add bidirectional Sigma Computing data model converter

.github/workflows/converter-sigma-ci.yml

@@ -0,0 +1,65 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Converters Sigma CI
+
+on:
+  push:
+    branches: [ "main" ]
+    paths:
+      - 'converters/sigma/**'
+      - 'python/**'
+      - '.github/workflows/converter-sigma-ci.yml'
+  pull_request:
+    branches: [ "main" ]
+    paths:
+      - 'converters/sigma/**'
+      - 'python/**'
+      - '.github/workflows/converter-sigma-ci.yml'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+
+    steps:
+      - name: Checkout project
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"
+
+      - name: Sync dependencies
+        working-directory: converters/sigma
+        run: |
+          uv sync
+
+      - name: Unit Tests
+        working-directory: converters/sigma
+        run: |
+          uv run pytest

converters/README.md

@@ -76,6 +76,7 @@ The Ossie specification currently defines extensions for the following vendors:
 | `OMNI` | Omni semantic model |
 | `WISDOM` | WisdomAI domain |
 | `NVIDIA_GSF` | NVIDIA Generative Semantic Fabric standalone YAML |
+| `SIGMA` | Sigma Computing data model |
 
 Each vendor may define custom extensions (via the `custom_extensions` field in the Ossie spec) to carry vendor-specific metadata that does not have an equivalent in the core specification.
 

converters/sigma/LIMITATIONS.md

@@ -0,0 +1,130 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Limitations and design tradeoffs
+
+What `converters/sigma` does not map onto a portable Ossie concept, and why. Every
+item here is reported at runtime as a `ConverterIssue`, never dropped silently.
+
+## Presentation and governance state is preserved, not modeled
+
+A Sigma table element carries `filters`, `folders`, `order`, `sort`, `summary`,
+`groupings`, `columnSecurities`, `visibleAsSource`, per-column `hidden`, and per-metric
+`isHighlighted`/`format`/`timeline`. None of these describe the *shape* of a semantic
+model — they describe how Sigma displays it and who may see it — and Ossie has no
+equivalent for any of them.
+
+All of it is preserved verbatim under a `native` key in the owning object's
+`custom_extensions` (`vendor_name: SIGMA`) and restored unchanged on export, matching
+how the Databricks, Omni, and Orion Belt converters handle vendor-only features. The
+residue is captured **by subtraction** — everything the converter does not explicitly
+map — so a future `schemaVersion` that adds fields still round-trips rather than
+silently losing them. Element `filters` additionally raise `FILTER_NOT_MODELED`.
+
+## Only `kind: table` elements are modeled
+
+The data model spec defines table elements; the API docs list input tables, Python
+elements, UI elements, and custom functions as unsupported programmatically. Any
+element with another `kind` is preserved verbatim at the model level under
+`non_table_elements` with an `UNSUPPORTED_ELEMENT_KIND` issue. This is a defensive
+path, not an expected one.
+
+## Non-warehouse-table sources have no `OSIDataset.source`
+
+`source.kind` may be `warehouse-table`, `sql`, `table`, `data-model`, `join`, or
+`union`. Only the first is a `database.schema.table` location, which is what
+`OSIDataset.source` is defined to hold. The other five get a readable marker
+(`sql:<connectionId>`, `join:<elementId>`, ...) plus a `DERIVED_ELEMENT_NOT_MODELED`
+issue; the full native `source` block lives in `custom_extensions`, so export
+reproduces it exactly. An Ossie document that never came from Sigma can only ever
+produce a `warehouse-table` source, since that is all a location string implies.
+
+## Join keys use two addressing schemes
+
+Sigma addresses a relationship key either by the element's own column id or by a raw
+`inode-<file>/<PHYSICAL_COLUMN>` reference straight to the warehouse column, bypassing
+the modeled column list. The converter resolves both to a modeled field name where it
+can, records `RELATIONSHIP_COLUMN_UNRESOLVED` where it cannot, and **always** keeps the
+raw `keys` in `custom_extensions`, so Sigma → Ossie → Sigma is exact either way.
+
+Unsolved: a document authored by another tool has no raw keys to fall back on, so
+export must synthesize key ids from field names. That works when every joined field is
+a modeled column, but cannot recreate a key pointing at a physical column the element
+never redefined.
+
+## Formula coverage is bounded by what Sigma puts in the formula
+
+`ossie_sigma.sigma_formula` parses Sigma's formula language and translates it through a
+sqlglot expression tree, covering nested calls, all operators, literals, and ~30
+functions across aggregation, conditional, string, and date categories.
+
+Table calculations (`RunningSum`, `Rank`, `Lag`, ...) resolve their partition/order
+context from UI configuration rather than from arguments, so nothing in the formula
+string can produce correct SQL. These are reported `EXPRESSION_NOT_TRANSLATABLE` and
+carry a `SIGMA` dialect entry only. Every formula, translatable or not, is preserved
+verbatim in that `SIGMA` entry, so nothing is ever lost on the way in.
+
+Because the intermediate representation is a sqlglot tree rather than SQL text,
+emitting a warehouse dialect instead of ANSI is a `dialect=` argument
+(`sigma_formula.to_sql`). The converter currently emits `ANSI_SQL` only: Sigma formulas
+are warehouse-agnostic, so the spec gives no signal about which vendor dialect would be
+more useful, and the table-calculation gap above is unaffected either way.
+
+## Untranslatable expressions are omitted on export, not approximated
+
+`formula` is required on every Sigma column and metric, and the data model API
+validates the whole document before applying any of it — so one placeholder formula
+fails the entire upload, not one field. When neither a `SIGMA` dialect entry nor a
+translatable `ANSI_SQL` one is available, the column or metric is **omitted** with an
+`EXPRESSION_NOT_TRANSLATABLE` issue naming it.
+
+## Cross-dataset metrics have no Sigma equivalent
+
+A Sigma metric is scoped to exactly one element; an `OSIMetric` is model-level and may
+span datasets via relationships. Sigma → Ossie always promotes cleanly (the owning
+`element_id` is preserved). Ossie → Sigma places a metric by its preserved
+`element_id`, or, failing that, by the single dataset its ANSI SQL unambiguously
+qualifies. A metric that references several datasets or none is dropped with
+`CROSS_DATASET_METRIC_DROPPED`.
+
+## Column formats carry only a coarse datatype
+
+The spec has no column datatype — only a display `format`, with two documented kinds,
+`number` and `date`. So Sigma → Ossie can infer no more than `Decimal`/`DateTime`, any
+other kind becomes `Opaque` (`OPAQUE_DATATYPE`), and a column with no format correctly
+gets no `datatype` at all. Ossie → Sigma emits only those two kinds; `String`,
+`Boolean`, `Time`, and `Opaque` produce no `format` key, because an invented `kind`
+would be rejected for the whole document. The full native format object is always
+preserved, so display detail (`formatString`, `currencySymbol`, ...) survives.
+
+## Synthesized ids
+
+Sigma element/column/relationship ids are load-bearing — controls, other data models,
+and materializations reference them — so the converter never invents an id for an
+object that has one. Native ids ride in `custom_extensions` and are reused verbatim.
+Objects originating outside Sigma get a `uuid5` of a fixed namespace plus their
+dataset/field path: deterministic across runs, processes, and machines (pinned by
+`test_synthesized_ids_are_stable_across_processes`), but not pre-registered with
+Sigma's backend.
+
+## One semantic model per document
+
+Sigma data models are single models; `OSIDocument.semantic_model` is a list. Only
+`semantic_model[0]` is converted, with `EXTRA_MODEL_DROPPED` naming how many were
+dropped.

converters/sigma/README.md

@@ -0,0 +1,181 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# apache-ossie-sigma
+
+Converts between [Sigma Computing](https://www.sigmacomputing.com/) Data Models (the
+"code representation" spec returned by `GET /v2/dataModels/{id}/spec`, and accepted by
+`POST`/`PUT` on the same resource) and the [Apache Ossie](https://github.com/apache/ossie)
+format.
+
+Both conversion directions are supported:
+
+- `sigma-to-osi` — Sigma data model spec JSON → Ossie YAML
+- `osi-to-sigma` — Ossie YAML → Sigma data model spec JSON
+
+## Requirements
+
+- Python 3.11+
+- [uv](https://docs.astral.sh/uv/) (recommended) or pip
+
+## Installation
+
+```bash
+pip install apache-ossie-sigma
+```
+
+Or with uv:
+
+```bash
+uv add apache-ossie-sigma
+```
+
+## CLI usage
+
+### Sigma → Apache Ossie
+
+Export a data model's spec from Sigma (e.g. with [sigcli](https://pypi.org/project/sigcli/)):
+
+```bash
+sigcli data-models spec get --params '{"dataModelId": "<id>"}' > data_model.json
+ossie-sigma sigma-to-osi -i data_model.json -o semantic_model.yaml
+```
+
+### Apache Ossie → Sigma
+
+```bash
+ossie-sigma osi-to-sigma -i semantic_model.yaml -o data_model.json
+```
+
+The output is a Sigma data model spec JSON document suitable for
+`sigcli data-models spec create`/`update`.
+
+### Help
+
+```bash
+ossie-sigma --help
+ossie-sigma sigma-to-osi --help
+ossie-sigma osi-to-sigma --help
+```
+
+## Python API
+
+```python
+import json
+from pathlib import Path
+
+from ossie_sigma import SigmaToOSIConverter, OSIToSigmaConverter
+
+spec = json.loads(Path("data_model.json").read_text())
+result = SigmaToOSIConverter().convert(spec)
+for issue in result.issues:
+    print(f"[warning] {issue.issue_type.value}: {issue.element_name}")
+Path("semantic_model.yaml").write_text(result.output.to_osi_yaml())
+
+# Ossie -> Sigma
+from ossie import OSIDocument
+import yaml
+
+document = OSIDocument.model_validate(yaml.safe_load(Path("semantic_model.yaml").read_text()))
+result = OSIToSigmaConverter().convert(document)
+Path("data_model.json").write_text(json.dumps(result.output, indent=2))
+```
+
+## Mapping overview
+
+| Sigma concept | Ossie concept | Notes |
+|---|---|---|
+| Data model (`name`, `description`) | `OSISemanticModel` | `dataModelId`, `folderId`, `documentVersion`, `schemaVersion` preserved in `custom_extensions` |
+| Page | *(none)* | Ossie has no page/folder-of-elements concept, but `pages` is a required part of the spec, so page membership is preserved per-dataset in `custom_extensions` and rebuilt on export |
+| Element (`kind: table`) | `OSIDataset` | `source` = warehouse path joined with `.` for `warehouse-table`; the other five source kinds get a marker plus the native `source` block in `custom_extensions` |
+| Element (any other `kind`) | *not modeled* | Preserved verbatim in a model-level `custom_extensions` entry so `osi-to-sigma` restores it unchanged |
+| Column (`formula`) | `OSIField.expression` | See [Expression translation](#expression-translation) |
+| Element `uniqueKeys` | `OSIDataset.primary_key` | Column ids resolved to field names in both directions |
+| Element `metrics[]` | `OSIMetric` | Promoted to model level (Ossie metrics are not dataset-scoped); the formula is re-qualified with the owning dataset name |
+| `relationships[]` (join keys) | `OSIRelationship` | See [Relationship resolution](#relationship-resolution) |
+| Column/element/relationship native `id` | *(preserved, not surfaced)* | Stashed in `custom_extensions` (`vendor_name: SIGMA`) so re-export can reuse Sigma's own stable ids rather than minting new ones — see [Stable ids](#stable-ids) |
+| Column `format` | `datatype` (coarse) + `custom_extensions` | Sigma has no column datatype, only a display format with two kinds (`number`, `date`); anything else becomes `Opaque`. The native format object is always preserved |
+| `filters`, `folders`, `order`, `sort`, `summary`, `groupings`, `columnSecurities`, `visibleAsSource`, `hidden`, metric `timeline`/`isHighlighted`/`format`, `relationshipType` | *not modeled* | Presentation/governance state with no Ossie equivalent, preserved verbatim under a `native` key in `custom_extensions`. Captured by subtraction, so fields added by a future `schemaVersion` round-trip too |
+
+### Expression translation
+
+Sigma's formula language is not SQL, so `ossie_sigma.sigma_formula` tokenizes and parses formulas
+like `Sum([Orders/Amount])` or `If([Status] = "closed", 1, 0)` with a recursive-descent parser.
+From there it does what the SQL-native converters in this repo do: it builds a **sqlglot expression
+tree** and lets sqlglot's generator emit the SQL, so identifier quoting, string escaping, and
+operator precedence are the library's job, and targeting a warehouse dialect instead of ANSI is a
+`dialect=` argument rather than a second renderer. The reverse direction walks a sqlglot tree back
+into formula text, so both directions share one intermediate representation.
+
+Translation is deliberately conservative: a formula using a construct with no portable SQL meaning
+(e.g. table calculations like `RunningSum`, whose partition/order context comes from UI
+configuration rather than from a formula argument) is **not** translated.
+
+Every `OSIExpression` produced by `sigma-to-osi` always carries **both**:
+
+1. A `SIGMA`-dialect entry with the original Sigma formula text, verbatim — this is what guarantees
+   lossless round-tripping regardless of how much the ANSI SQL translator understands.
+2. An `ANSI_SQL`-dialect entry, present only when the formula translated successfully.
+
+`osi-to-sigma` prefers the `SIGMA` dialect entry when present (perfect fidelity for anything that
+came from Sigma); for expressions authored by another tool it falls back to translating the
+`ANSI_SQL` entry back into Sigma formula syntax. If neither is possible, the column or metric is
+**omitted** and flagged in `ConverterResult.issues`
+(`ConverterIssueType.EXPRESSION_NOT_TRANSLATABLE`). `formula` is a required property and the data
+model API validates the whole document before applying any of it, so emitting a placeholder would
+fail the entire upload rather than degrade one field.
+
+### Relationship resolution
+
+Sigma relationships (`element.relationships[]`) join two *elements*, not two *Ossie datasets*
+directly, and their `keys[].sourceColumnId`/`targetColumnId` address columns by Sigma's internal
+column id — which is **not** the same id space as the modeled column's own `id` when the key
+references a column that isn't explicitly redefined by the element (Sigma addresses those via an
+`inode-<file>/<PHYSICAL_COLUMN_NAME>` reference straight to the underlying warehouse table/column,
+bypassing the element's own column list entirely). `sigma_to_osi.py` resolves both addressing
+schemes to a modeled column name using the element's own column formulas; when resolution succeeds,
+`OSIRelationship.from_columns`/`to_columns` reference the Ossie field name. When it cannot be
+resolved (the physical column has no corresponding modeled column, e.g. it was never referenced
+anywhere in the element as a column), the physical column name is used verbatim and a converter
+issue is recorded. **The raw, unresolved `sourceColumnId`/`targetColumnId` values are always
+preserved in the relationship's `custom_extensions`,** so `osi-to-sigma` reconstructs the exact
+original join regardless of whether name resolution succeeded — see [Limitations](#limitations).
+
+### Stable ids
+
+Sigma column, element, and relationship ids are load-bearing: other parts of a Sigma workbook
+(controls, other data models' relationships, materializations) reference them, so an export that
+mints new ids for unchanged objects would silently break those references. `sigma_to_osi.py`
+therefore never invents an id for anything that already has one — it always preserves the native
+Sigma id in that object's `custom_extensions` and `osi-to-sigma` reuses it verbatim. Ids are only
+synthesized (as a deterministic `uuid5` of a fixed namespace plus the object's dataset/field path)
+for objects that originate purely in Ossie and have never been round-tripped through Sigma before.
+
+## Limitations
+
+See [`LIMITATIONS.md`](LIMITATIONS.md) for a full accounting of what this converter does not (yet)
+handle faithfully and why.
+
+## Development
+
+```bash
+cd converters/sigma
+uv sync
+uv run pytest
+```