Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
not_for_github/
cdl_skill/
.DS_Store
mutants/

# git worktree
.history
Expand Down
4 changes: 2 additions & 2 deletions docs/schema_converters.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,8 @@ from cdm_data_loaders.converters.facades.jsonschema_to_pyarrow import JSONSchema
from cdm_data_loaders.converters.sources.json_schema.permissive import JsonSchemaReader

KBASE_EXTENSION_SPEC: Final[ExtensionSpec] = ExtensionSpec(
"x-kbase",
{
namespace="x-kbase",
schema={
"type": "object",
"properties": {
"ontology_term": {"type": "string"},
Expand Down
22 changes: 11 additions & 11 deletions docs/uniprot.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@

`cdm-data-loaders` provides `dlt` pipelines for ingesting UniProt Knowledgebase (UniProtKB) and UniRef XML datasets (`*.xml` and `*.xml.gz`) as well as tabular UniProt ID mapping releases (`*.dat` and `*.dat.gz`), along with parsers for UniProt release metadata.

| Pipeline CLI | Mode | Source Module | Bundled Schema / Dataset Module | Default Dataset / Table |
| :------------------ | :---------------------------- | :---------------------------------------------------- | :-------------------------------------------- | :--------------------------------------------------- |
| `uniprot` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniprot/uniprot.xsd` | `uniprot` / `entry` |
| `uniref50` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref50` / `entry` |
| `uniref90` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref90` / `entry` |
| `uniref100` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref100` / `entry` |
| `uniprot_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.kb` | `uniprot_cdm` (multi-table) |
| `uniref50_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref50_cdm` (multi-table) |
| `uniref90_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref90_cdm` (multi-table) |
| `uniref100_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref100_cdm` (multi-table) |
| Pipeline CLI | Mode | Source Module | Bundled Schema / Dataset Module | Default Dataset / Table |
| :------------------ | :---------------------------- | :----------------------------------------------------- | :--------------------------------------------- | :--------------------------------------------------- |
| `uniprot` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniprot/uniprot.xsd` | `uniprot` / `entry` |
| `uniref50` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref50` / `entry` |
| `uniref90` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref90` / `entry` |
| `uniref100` | XSD-to-PyArrow (single table) | `cdm_data_loaders.pipelines.presets.uniprot_xml` | `cdm_data_loaders/schemas/uniref/uniref.xsd` | `uniref100` / `entry` |
| `uniprot_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniprot_kb` | `uniprot_cdm` (multi-table) |
| `uniref50_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref50_cdm` (multi-table) |
| `uniref90_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref90_cdm` (multi-table) |
| `uniref100_cdm` | CDM-normalized (multi-table) | `cdm_data_loaders.pipelines.presets.uniprot_cdm` | `cdm_data_loaders.datasets.uniprot.uniref` | `uniref100_cdm` (multi-table) |
| `uniprot_idmapping` | XSV preset (dual-table) | `cdm_data_loaders.pipelines.presets.uniprot_idmapping` | `cdm_data_loaders.datasets.uniprot.idmapping` | `uniprot_idmapping` (`raw_id_mapping`, `id_mapping`) |

---
Expand Down Expand Up @@ -45,7 +45,7 @@ When loading directly into the KBase Common Data Model (CDM) relational table st

### `uniprot_cdm`

[`uniprot_cdm`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/presets/uniprot_cdm.py) converts each `{https://uniprot.org/uniprot}entry` element against [`uniprot.xsd`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/schemas/uniprot/uniprot.xsd) and routes rows via [`datasets/uniprot/kb.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/datasets/uniprot/kb.py) into normalized tables under dataset `uniprot_cdm`:
[`uniprot_cdm`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/pipelines/presets/uniprot_cdm.py) converts each `{https://uniprot.org/uniprot}entry` element against [`uniprot.xsd`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/schemas/uniprot/uniprot.xsd) and routes rows via [`datasets/uniprot/uniprot_kb.py`](https://github.com/kbase/cdm-data-loaders/blob/develop/src/cdm_data_loaders/datasets/uniprot/uniprot_kb.py) into normalized tables under dataset `uniprot_cdm`:

| Table Name | Contents |
| :--------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand Down
5 changes: 3 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,9 @@ build-backend = "uv_build"

[tool.mutmut]
source_paths = [ "src/" ]
pytest_add_cli_args_test_selection= [ "-m", "not requires_spark", "-m", "not slow_test", "-m", "not requires_ceph", "-m", "not external_request", "-m", "not docker", "-m", "not docs", "-m", "not state_audit" ]

pytest_add_cli_args_test_selection= [ "-m", "not requires_spark", "-m", "not slow_test", "-m", "not requires_ceph", "-m", "not external_request", "-m", "not docker", "-m", "not state_audit", "tests/cdm_data_loaders/"]
also_copy = ["docs/", "docs_macros/", "scripts/", "specs/", "notebooks/"]
do_not_mutate = [ "src/cdm_data_loaders/schemas/**/schema.py"]


[tool.pytest]
Expand Down
52 changes: 34 additions & 18 deletions specs/_loader/converter_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,41 +9,45 @@
The same manifests drive the specification tests and the generated specification documents.
"""

from dataclasses import dataclass, field
from pathlib import Path
from typing import Any

from pydantic import BaseModel, ConfigDict, Field

from specs._loader.yaml_io import CASE_MANIFEST, load_yaml_dict


@dataclass(frozen=True, slots=True)
class Rule:
class Rule(BaseModel):
"""A single conversion or pipeline specification rule."""

model_config = ConfigDict(extra="forbid", frozen=True)

id: str
statement: str
section_id: str


@dataclass(frozen=True, slots=True)
class Section:
class Section(BaseModel):
"""A group of rules rendered under one heading."""

model_config = ConfigDict(extra="forbid", frozen=True)

id: str
title: str
intro: str
generated: str | None
rules: tuple[Rule, ...]


@dataclass(frozen=True, slots=True)
class RuleCatalogue:
class RuleCatalogue(BaseModel):
"""All rules and front-matter metadata for one converter or pipeline specification."""

model_config = ConfigDict(extra="forbid", frozen=True)

title: str
intro: str
sections: tuple[Section, ...]
metadata: dict[str, Any] = field(default_factory=dict)
metadata: dict[str, Any] = Field(default_factory=dict)

@property
def rules(self) -> dict[str, Rule]:
Expand All @@ -55,38 +59,49 @@ def rules(self) -> dict[str, Rule]:
return {rule.id: rule for section in self.sections for rule in section.rules}


@dataclass(frozen=True, slots=True)
class Run:
class Run(BaseModel):
"""One execution of a converter case: input data, parameters, and expected outputs."""

model_config = ConfigDict(extra="forbid", frozen=True)

id: str
data: Path
expected_json: Path
expected_table: Path
xml_tag: str | None
params: dict[str, Any] = field(default_factory=dict)
params: dict[str, Any] = Field(default_factory=dict)
schema_attr: str = "EXPECTED_SCHEMA"
table_attr: str = "EXPECTED_TABLE"
expected_count: int | None = None
expected_rejected: Path | None = None
extra_expected_json: dict[str, Path] = field(default_factory=dict)
extra_expected_json: dict[str, Path] = Field(default_factory=dict)


@dataclass(frozen=True, slots=True)
class Case:
class Case(BaseModel):
"""A converter fixture directory with its manifest."""

model_config = ConfigDict(extra="forbid", frozen=True, validate_by_name=True, validate_by_alias=True)

suite: str
name: str
directory: Path
title: str
schema: Path
schema_path: Path = Field(alias="schema")
expected_schema: Path
xml_tag: str | None
rules: tuple[str, ...]
notes: str
runs: tuple[Run, ...]

@property
def schema(self) -> Path: # type: ignore[override]
"""Return the input schema file path for this converter case.

:return: Resolved input schema path.
:rtype: Path
"""
return self.schema_path

@property
def anchor(self) -> str:
"""Return the stable HTML anchor for the case.
Expand All @@ -97,16 +112,17 @@ def anchor(self) -> str:
return f"case-{self.suite}-{self.name}".replace("_", "-").lower()


@dataclass(frozen=True, slots=True)
class ErrorCase:
class ErrorCase(BaseModel):
"""An input that must be rejected with a specific error code."""

model_config = ConfigDict(extra="forbid", frozen=True)

file: Path
code: str
match: str
description: str
xml_tag: str | None
params: dict[str, Any] = field(default_factory=dict)
params: dict[str, Any] = Field(default_factory=dict)


def load_rules(path: Path) -> RuleCatalogue:
Expand Down
7 changes: 4 additions & 3 deletions specs/_loader/dlt_contracts.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
"""Static PyArrow probe schemas and dlt normalization contracts shared by docs macros and tests."""

from dataclasses import dataclass
from datetime import UTC, date, datetime
from decimal import Decimal
from enum import StrEnum
from typing import Any, Final

import pyarrow as pa
from frozendict import frozendict
from pydantic import BaseModel, ConfigDict
from pyiceberg.schema import Schema as IcebergSchema
from pyiceberg.types import (
BooleanType,
Expand Down Expand Up @@ -845,10 +845,11 @@ class TableShape(StrEnum):
)


@dataclass(frozen=True, slots=True)
class ShapeExpectation:
class ShapeExpectation(BaseModel):
"""Input schema, rows, and per-layer expectations for one ``TableShape``."""

model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid", frozen=True)

input_arrow_schema: pa.Schema
table_metadata: dict[str, str]
rows: tuple[dict[str, Any], ...]
Expand Down
30 changes: 17 additions & 13 deletions specs/_loader/pipeline_spec.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,52 +2,55 @@

import json
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Final

import pyarrow as pa
from pydantic import BaseModel, ConfigDict, Field

from cdm_data_loaders.core.buffer import REJECTED_PYARROW_SCHEMA
from specs._loader.yaml_io import CASE_MANIFEST, load_settings_file, load_spec_module, load_yaml_dict

RELATIVE_PATH_SETTINGS_KEYS: Final[frozenset[str]] = frozenset({"xsd_file", "schema_file", "log_config_file"})


@dataclass(frozen=True, slots=True)
class ExpectedReport:
class ExpectedReport(BaseModel):
"""Expected contents of ``report.txt`` written by ``save_pipeline_metadata``."""

model_config = ConfigDict(extra="forbid", frozen=True)

status: str = "completed"
ingested_files: tuple[str, ...] = ()
load_errors: tuple[str, ...] = ()
pipeline_load_info_errs: str | None = None


@dataclass(frozen=True, slots=True)
class PipelineRun:
class PipelineRun(BaseModel):
"""One execution of a pipeline case: input path, settings, and expected outputs."""

model_config = ConfigDict(extra="forbid", frozen=True)

id: str
input_dir: Path
input_subdir: str | None = None
settings_file: Path | None = None
settings: dict[str, Any] = field(default_factory=dict)
expected_report: ExpectedReport = field(default_factory=ExpectedReport)
expected_counts: dict[str, int] = field(default_factory=dict)
settings: dict[str, Any] = Field(default_factory=dict)
expected_report: ExpectedReport = Field(default_factory=ExpectedReport)
expected_counts: dict[str, int] = Field(default_factory=dict)
expected_tables: Path | None = None
expected_json: Path | None = None
tables_attr: str = "EXPECTED_TABLES"
primary_table: str | None = None
expected_rejected: Path | None = None
extra_expected_json: dict[str, Path] = field(default_factory=dict)
extra_expected_json: dict[str, Path] = Field(default_factory=dict)
smoke: bool = True


@dataclass(frozen=True, slots=True)
class PipelineCase:
class PipelineCase(BaseModel):
"""A pipeline specification fixture directory with its manifest."""

model_config = ConfigDict(extra="forbid", frozen=True)

pipeline: str
name: str
directory: Path
Expand Down Expand Up @@ -219,10 +222,11 @@ def load_expected_json_tables(run: PipelineRun) -> dict[str, list[dict[str, Any]
return result


@dataclass(frozen=True, slots=True)
class _CaseRunContext:
class _CaseRunContext(BaseModel):
"""Case-level defaults shared across all runs in one ``case.yaml`` manifest."""

model_config = ConfigDict(extra="forbid", frozen=True)

pipeline: str
directory: Path
case_raw: dict[str, Any]
Expand Down
27 changes: 19 additions & 8 deletions src/cdm_data_loaders/converters/core/dlt_normalization.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,29 @@
from collections import deque
from collections.abc import Mapping
from copy import deepcopy
from dataclasses import dataclass, field
from typing import Final, cast
from dataclasses import field
from typing import Annotated, Final, cast

from dlt.common.schema.typing import TTableSchema
from pydantic import ConfigDict, GetPydanticSchema, InstanceOf, StrictStr
from pydantic.dataclasses import dataclass, rebuild_dataclass
from pydantic_core import core_schema

from cdm_data_loaders.converters.core.errors import ConversionError

NESTED_TABLE_SEPARATOR: Final[str] = "__"

type _TableSchemaField = Annotated[
TTableSchema,
GetPydanticSchema(lambda _tp, _handler: core_schema.is_instance_schema(Mapping)),
]


class DltNormalizationError(ConversionError):
"""Raised for malformed tables, invalid parent graphs, or inconsistent trees."""


@dataclass(frozen=True, slots=True)
@dataclass(frozen=True, slots=True, config=ConfigDict(extra="forbid"))
class DltTableNode:
"""A table with its mapping name, declared parent, property key, and raw definition.

Expand All @@ -26,11 +34,14 @@ class DltTableNode:
Identity fields are frozen; table and children mappings are independently owned.
"""

name: str
parent: str | None
key: str
table: TTableSchema
children: dict[str, "DltTableNode"] = field(default_factory=dict)
name: StrictStr
parent: StrictStr | None
key: StrictStr
table: _TableSchemaField
children: dict[StrictStr, InstanceOf["DltTableNode"]] = field(default_factory=dict)


rebuild_dataclass(DltTableNode)


def child_key(parent_name: str, child_name: str) -> str:
Expand Down
Loading
Loading