Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,16 +98,16 @@ jobs:
suite:
- name: pipelines
markers: "not requires_spark and not requires_ceph and not docs and not state_audit"
args: --cov tests/cdm_data_loaders/pipelines/
args: tests/cdm_data_loaders/pipelines/
- name: e2e
markers: "not requires_spark and not requires_ceph and not docs and not state_audit"
args: --cov tests/e2e/
args: tests/e2e/
- name: io
markers: "not requires_spark and not requires_ceph and not docs and not state_audit"
args: --cov tests/cdm_data_loaders/io/
args: tests/cdm_data_loaders/io/
- name: everything else
markers: "not requires_spark and not requires_ceph and not docs and not state_audit"
args: --cov --ignore=tests/e2e --ignore=tests/cdm_data_loaders/pipelines --ignore=tests/docs --ignore=tests/cdm_data_loaders/io
args: --ignore=tests/e2e --ignore=tests/cdm_data_loaders/pipelines --ignore=tests/docs --ignore=tests/cdm_data_loaders/io
include:
- python-version: "3.13"
os: "ubuntu-latest"
Expand Down Expand Up @@ -171,7 +171,7 @@ jobs:
- name: Run local tests
shell: bash
continue-on-error: false
run: uv run --all-groups pytest -m "${{ matrix.suite.markers }}" --durations=0 --durations-min=5 --junitxml=junit.xml -o junit_family=legacy ${{ matrix.suite.args }}
run: uv run --all-groups pytest -m "${{ matrix.suite.markers }}" --durations=0 --durations-min=5 --junitxml=junit.xml -o junit_family=legacy --cov src/ ${{ matrix.suite.args }}

- name: Send test reports to Codecov
id: upload_test_reports
Expand Down
2 changes: 0 additions & 2 deletions docs/dlt_pyarrow_definitions.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ This guide summarises what table metadata, column descriptions, types, nullabili
- **Characterisation test suite**: [`tests/dependencies/test_dlt_pyarrow_table_definitions.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/tests/dependencies/test_dlt_pyarrow_table_definitions.py)
- **Buffered Arrow page emitter (`DictBuffer`)**: [`src/cdm_data_loaders/core/buffer.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/buffer.py)
- **Schema registry (`ArrowSchemaRegistry`)**: [`src/cdm_data_loaders/core/registry.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/registry.py)
- **Resource hint registrar (`OutputTableHints`)**: [`src/cdm_data_loaders/core/transforms/hints.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/hints.py)
- **Polaris Iceberg definition sync CLI**: [`src/cdm_data_loaders/cli/polaris_schema_definitions.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/cli/polaris_schema_definitions.py)
- **PyIceberg SQLite connection cleanup**: [`src/cdm_data_loaders/utils/iceberg/cleanup.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/utils/iceberg/cleanup.py)
- **Companion nested output and `max_table_nesting` guide**: [`dlt` Nested Output Behaviour](dlt_nested_output_behaviour.md)
Expand Down Expand Up @@ -100,7 +99,6 @@ Key takeaways for nested schemas:
## Implications for Pipeline and Catalog Authors

- **Populate `metadata={"doc": ...}` for Iceberg field descriptions**: converters and `OUTPUT_SCHEMAS` registries (see [Table and Column Transformations](table_transforms.md)) should write field descriptions under the `"doc"` metadata key so PyIceberg populates `NestedField.doc` during table creation.
- **Register table hints and Iceberg partitions via `OutputTableHints`**: `dlt` drops `pa.Schema.metadata` before creating Iceberg tables, and PyIceberg rewrites `data/*.parquet` without schema-level metadata. Pipeline settings subclasses can return `OutputTableHints` from `get_output_table_hints()` ([`src/cdm_data_loaders/core/transforms/hints.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/hints.py)) to register table descriptions (`x-iceberg-table-properties={"comment": ...}`), top-level column descriptions, and `x-iceberg-partition` specs on `DltResource` variants at ingest time.
- **Sync table comments and field definitions to Polaris Iceberg tables**: for existing tables in a Polaris REST catalog, run [`apply_polaris_schema_definitions`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/cli/polaris_schema_definitions.py) (`uv run apply_polaris_schema_definitions --catalog polaris --table <ns.table> --schema-source <module:ATTR_or_jsonschema>`) to apply table `comment` properties and root or nested `StructType` `NestedField.doc` strings (`doc`, `comment`, and `description` metadata keys), normalizing root PyArrow column names with `dlt`'s `snake_case` convention while matching nested `StructType` field names as written.
- **Expect `pa.int32()` when scanning `pa.int16()` columns from Iceberg**: downstream Arrow consumers reading Iceberg tables via `table.scan().to_arrow()` receive `pa.int32()` for columns ingested as `pa.int16()`.

Expand Down
2 changes: 1 addition & 1 deletion docs/pipeline_e2e_testing_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ How `PipelineTestRunner` ([`tests/e2e/framework.py`](https://github.com/kbase/cd

- **Input directory and `dlt` configuration**: Local `input_dir` contents (along with `dlt_conf/config.toml`, `dlt_conf/secrets.toml`, and `$HOME/.pyiceberg.yaml` when present) are staged on the host and bind-mounted at `/input_dir`, while `output_dir` is bind-mounted at `/output_dir`.
- **Remote URLs (`s3://...`)**: When `--input-dir` is an `s3://` URL, `PipelineTestRunner` passes the remote URL to the CLI command inside the container while mounting `/input_dir` so [`scripts/entrypoint.sh`](https://github.com/kbase/cdm-data-loaders/blob/develop/scripts/entrypoint.sh) copies `/input_dir/dlt_conf` into `$HOME/.dlt`.
- **Auxiliary schema files and Python modules**: Local files passed via `--xsd-file`, `--schema-file`, `-s`, or `--log-config-file` are staged read-only at `/aux_dir`, and generated schema/model modules (`--entity-models`, `--entity-schemas`, `-m`) are materialized read-only at `/pymodules` on `PYTHONPATH`.
- **Auxiliary schema files and Python modules**: Local files passed via `--xsd-file`, `--schema-file`, `-s`, `--log-config-file`, or `.py` paths on module flags are staged read-only at `/aux_dir`, and synthetic or non-container-importable Python modules (`--entity-models`, `--entity-schemas`, `-m`, `--preprocessors-module`, `--transforms-module`, `--output-schemas-module`) along with their dynamic model dependencies are materialized read-only at `/pymodules` on `PYTHONPATH`.

---

Expand Down
39 changes: 22 additions & 17 deletions docs/table_transforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ All six file-format ingestion pipelines support pluggable file pre-processing, r
| [`jsonlines/pydantic.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/jsonlines/pydantic.py) | `jsonlines_pydantic` | `pyarrow.Table` | Keys of `ENTITY_MODELS` |
| [`xsv/pipeline.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/xsv/pipeline.py) | `xsv` | `pyarrow.Table` | `--table-name` (or preset entrypoint `uniprot_idmapping`) |
| [`xmltodict/pipeline.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/xmltodict/pipeline.py) | `xmltodict` | `list[dict]` (`pyarrow.Table` if schema map set) | `--table-name` (`settings.table_name`) |
| [`xmltopyarrow/pipeline.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/xmltopyarrow/pipeline.py) | `xmltopyarrow` | `pyarrow.Table` | `--table-name` (or preset entrypoints `pubmed`, `uniprot`, `uniref50`, `uniref90`, `uniref100`) |
| [`xmltopyarrow/pipeline.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/xmltopyarrow/pipeline.py) | `xmltopyarrow` | `pyarrow.Table` | `--table-name` (or preset entrypoints `pubmed`, `uniprot`, `uniref50`, `uniref90`, `uniref100`, `uniprot_cdm`, `uniref50_cdm`, `uniref90_cdm`, `uniref100_cdm`) |

---

Expand Down Expand Up @@ -65,7 +65,7 @@ Create an importable Python module exporting `FILE_PREPROCESSORS`, `TABLE_TRANSF

**`OUTPUT_SCHEMAS`** (`--output-schemas-module` / `CDL_OUTPUT_SCHEMAS_MODULE`): `dict[str, pa.Schema]`

- Optional mapping of `{destination_table_name: pyarrow.Schema}` overriding or supplying `pyarrow.Schema` definitions for transformed or split destination tables. Preset settings subclasses can also return `OutputTableHints` from `get_output_table_hints()` ([`src/cdm_data_loaders/core/transforms/hints.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/hints.py)) to register table/column descriptions and Iceberg partition specs on `DltResource` variants.
- Optional mapping of `{destination_table_name: pyarrow.Schema}` overriding or supplying `pyarrow.Schema` definitions for transformed or split destination tables. Preset settings subclasses can lock built-in transforms and output schemas off the CLI with `LockedTransformHooksMixin` ([`src/cdm_data_loaders/core/transforms/settings_mixin.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/settings_mixin.py)) and attach table descriptions via `with_schema_description` ([`src/cdm_data_loaders/utils/arrow_schema.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/utils/arrow_schema.py)).

---

Expand Down Expand Up @@ -488,23 +488,28 @@ import pyarrow.compute as pc

from cdm_data_loaders.core.registry import TableTransformFn, TransformContext
from cdm_data_loaders.core.transforms.routing import emit_raw_table

RAW_ID_MAPPING_SCHEMA = pa.schema(
[
pa.field("uniprot_id", pa.string(), nullable=False),
pa.field("db_reference_type", pa.string(), nullable=False),
pa.field("db_reference_id", pa.string(), nullable=False),
],
metadata={"comment": "raw ID mapping data from UniProt"},
from cdm_data_loaders.utils.arrow_schema import with_schema_description

RAW_ID_MAPPING_SCHEMA = with_schema_description(
pa.schema(
[
pa.field("uniprot_id", pa.string(), nullable=False),
pa.field("db_reference_type", pa.string(), nullable=False),
pa.field("db_reference_id", pa.string(), nullable=False),
]
),
"raw ID mapping data from UniProt",
)

ID_MAPPING_SCHEMA = pa.schema(
[
pa.field("uniprot_id", pa.string(), nullable=False),
pa.field("db_reference", pa.string(), nullable=False),
pa.field("source", pa.string(), nullable=False),
],
metadata={"comment": "Standardised mapping data from UniProt"},
ID_MAPPING_SCHEMA = with_schema_description(
pa.schema(
[
pa.field("uniprot_id", pa.string(), nullable=False),
pa.field("db_reference", pa.string(), nullable=False),
pa.field("source", pa.string(), nullable=False),
]
),
"Standardised mapping data from UniProt",
)


Expand Down
5 changes: 2 additions & 3 deletions docs/uniprot.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ uv run uniref90_cdm \

[`presets/uniprot_idmapping.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/presets/uniprot_idmapping.py) provides a preset XSV pipeline (`UniProtIdMappingSettings`, sharing `run_xsv_pipeline`) for ingesting headerless, tab-delimited UniProt `idmapping.dat` and `idmapping.dat.gz` files validated against [`idmapping.schema.json`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/schemas/uniprot/idmapping/idmapping.schema.json).

Each validated `pyarrow.Table` batch passes through two composed `TABLE_TRANSFORMS` steps ([`emit_raw_table`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/routing.py) and [`transform_uniprot_id_mapping`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/datasets/uniprot/idmapping.py)), emitting both a raw table and a standardised cross-reference table under dataset `uniprot_idmapping`. Both tables carry PyArrow schemas ([`schemas/uniprot/idmapping/schema.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/schemas/uniprot/idmapping/schema.py)) and Iceberg hash-bucket partition specs (`bucket(N, "uniprot_id")`, configurable via `--partition-buckets`).
Each validated `pyarrow.Table` batch passes through two composed `TABLE_TRANSFORMS` steps ([`emit_raw_table`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/core/transforms/routing.py) and [`transform_uniprot_id_mapping`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/datasets/uniprot/idmapping.py)), emitting both a raw table and a standardised cross-reference table under dataset `uniprot_idmapping`. Both tables carry PyArrow schemas defined in [`schemas/uniprot/idmapping/schema.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/schemas/uniprot/idmapping/schema.py).

### Output Tables

Expand All @@ -105,8 +105,7 @@ Each validated `pyarrow.Table` batch passes through two composed `TABLE_TRANSFOR
uv run uniprot_idmapping \
--input-dir /path/to/idmapping \
--output-dir /path/to/output \
--loader-file-format iceberg \
--partition-buckets 64
--loader-file-format iceberg
```

---
Expand Down
10 changes: 10 additions & 0 deletions docs_macros/tables/pipeline_tables.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@
from cdm_data_loaders.converters.sources.json_schema.strict_scalars import NON_EXTENDABLE_SCALAR_HINTS
from cdm_data_loaders.core.buffer import REJECTED_PYARROW_SCHEMA
from cdm_data_loaders.pipelines.presets.pubmed import PubMedXmlToPyArrowSettings
from cdm_data_loaders.pipelines.presets.uniprot_cdm import (
UniProtSettings,
UniRef50CdmSettings,
UniRef90CdmSettings,
UniRef100CdmSettings,
)
from cdm_data_loaders.pipelines.presets.uniprot_xml import (
UniProtXmlToPyArrowSettings,
UniRef50XmlToPyArrowSettings,
Expand Down Expand Up @@ -37,9 +43,13 @@
XMLTOPYARROW_SETTINGS_CLASSES: Final[tuple[type[XmlToPyArrowSettings], ...]] = (
XmlToPyArrowSettings,
UniProtXmlToPyArrowSettings,
UniProtSettings,
UniRef50XmlToPyArrowSettings,
UniRef50CdmSettings,
UniRef90XmlToPyArrowSettings,
UniRef90CdmSettings,
UniRef100XmlToPyArrowSettings,
UniRef100CdmSettings,
PubMedXmlToPyArrowSettings,
)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
title: UniProtKB CDM Fan-Out and Missing-Accession Rejection (`_rejected`)
pipeline: uniprot_cdm
settings_file: settings.toml
rules:
- R-XML-DEFAULT-NS
- R-XML-TRANSFORMS
- R-XML-REPORT
- R-UNIPROT-CDM-PRESET
notes: |
A `<uniprot>` XML file with one complete `<entry>` (`P12345`) and one `<entry>` whose `<accession>` is blank.
`UniProtSettings` fans the valid entry across `entity`, `identifier`, `name`, `protein`, and
`entity_x_publication` (`UNIPROT_CDM_SCHEMA_MAP`) and routes the blank-accession entry to `_rejected`
with `table_name="entry"` and `source_file="uniprot_sample.xml"`.
runs:
- id: default
input_dir: input
expected_report:
status: completed
ingested_files:
- uniprot_sample.xml
pipeline_load_info_errs: null
expected_counts:
entity: 1
identifier: 4
name: 3
protein: 1
entity_x_publication: 1
_rejected: 1
expected_tables: null
expected_json: null
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
<?xml version="1.0" encoding="UTF-8"?>
<uniprot xmlns="https://uniprot.org/uniprot">
<entry created="2000-05-30" modified="2020-01-01" version="3" dataset="Swiss-Prot">
<accession>P12345</accession>
<accession>Q67890</accession>
<name>TEST_ENTRY</name>
<protein>
<recommendedName>
<fullName>Test protein</fullName>
</recommendedName>
</protein>
<gene>
<name type="primary">testGene</name>
</gene>
<organism>
<name type="scientific">Test organism</name>
<dbReference type="NCBI Taxonomy" id="9606"/>
</organism>
<reference key="1">
<citation type="journal article" date="2020" name="Test Journal">
<title>A test paper.</title>
<authorList>
<person name="Tester T."/>
</authorList>
<dbReference type="PubMed" id="12345678"/>
</citation>
<scope>SEQUENCE</scope>
</reference>
<comment type="function">
<text>A test function.</text>
</comment>
<dbReference type="GeneID" id="1636"/>
<evidence type="ECO:0000255" key="1"/>
<proteinExistence type="evidence at protein level"/>
<sequence length="5" mass="500" checksum="ABCDEF0123456789" modified="2020-01-01" version="1">ACGTA</sequence>
</entry>
<entry created="2000-05-30" modified="2020-01-01" version="1" dataset="Swiss-Prot">
<accession> </accession>
<name>MISSING_ACC</name>
<protein>
<submittedName>
<fullName>Unindexed entry</fullName>
</submittedName>
</protein>
<organism>
<name type="scientific">Test organism</name>
<dbReference type="NCBI Taxonomy" id="9606"/>
</organism>
<proteinExistence type="uncertain"/>
<sequence length="4" mass="400" checksum="0123456789ABCDEF" modified="2020-01-01" version="1">ACGT</sequence>
</entry>
</uniprot>
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
loader_file_format = "parquet"
use_output_dir_for_pipeline_metadata = true
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
title: UniRef50 CDM Cluster Fan-Out and Missing-ID Rejection (`_rejected`)
pipeline: uniref50_cdm
settings_file: settings.toml
rules:
- R-XML-DEFAULT-NS
- R-XML-TRANSFORMS
- R-XML-REPORT
- R-UNIREF-CDM-PRESET
notes: |
A `<UniRef50>` XML file with one valid `<entry>` (`UniRef50_A0AB34IYJ6`) and one `<entry>` missing `id`.
`UniRef50CdmSettings` fans the valid cluster across `entity`, `cluster`, and `clustermember`
(`UNIREF_CDM_SCHEMA_MAP`) and routes the entry without `id` to `_rejected` with `table_name="entry"`
and `source_file="uniref50_sample.xml"`.
runs:
- id: default
input_dir: input
expected_report:
status: completed
ingested_files:
- uniref50_sample.xml
pipeline_load_info_errs: null
expected_counts:
entity: 1
cluster: 1
clustermember: 2
_rejected: 1
expected_tables: null
expected_json: null
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
<?xml version="1.0" encoding="UTF-8"?>
<UniRef50 xmlns="http://uniprot.org/uniref">
<entry id="UniRef50_A0AB34IYJ6" updated="2025-02-05">
<name>Cluster: PKZILLA-1 giant polyketide synthase</name>
<property type="member count" value="2"/>
<representativeMember>
<dbReference type="UniProtKB ID" id="A0AB34IYJ6_PRYPA">
<property type="UniProtKB accession" value="A0AB34IYJ6"/>
<property type="isSeed" value="true"/>
</dbReference>
<sequence length="4" checksum="385E596EB33D030E">ACGT</sequence>
</representativeMember>
<member>
<dbReference type="UniParc ID" id="UPI0001"/>
</member>
</entry>
<entry updated="2025-02-05">
<name>Cluster: Missing ID</name>
<representativeMember>
<dbReference type="UniParc ID" id="UPI0002"/>
</representativeMember>
</entry>
</UniRef50>
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
loader_file_format = "parquet"
use_output_dir_for_pipeline_metadata = true
Loading
Loading