See .agents/skills/coding-style and .agents/skills/testing for rules that apply to all code edits.
Every code change that alters CLI flags, settings classes, public APIs, schemas, or converter behavior MUST update documentation in the same task before completion:
- CLI and Settings Changes (
core/settings.py,core/fields.py,core/transforms/,pipelines/**/settings.py,pyproject.toml):docs/cli_reference.mdanddocs/bundled_schemas.mdare.gitignored macro stubs materialized byuv run python -m docs_macrosand rendered at build/serve time viadocs_macros(docs/_templates/pages/cli_reference.md.j2anddocs/_templates/pages/bundled_schemas.md.j2).- Search
docs/andREADME.mdfor any renamed/removed field names, CLI flags (--*), or env vars (CDL_*) and update all prose/examples.
- Converter Rules, Scalar Maps, or Error Classes (
converters/,validation/xsd/,specs/schema_converters/*/rules.yaml):- Converter specification pages render at build/serve time via
docs_macros(docs/_templates/pages/converter_spec.md.j2anddocs/_templates/tables/*.md.j2). - Converter suites under
specs/schema_converters/<xxx_to_yyy>/are auto-discovered bydiscover_converter_specs()indocs_macros/specs/discovery.py, anduv run python -m docs_macrosmaterializes their.gitignoreddocs/<xxx_to_yyy>_specification.mdstubs beforezensical build/zensical serve. When adding or removing a converter specification, register/update it inzensical.toml(nav) anddocs/index.md. - See
docs/schema_converter_specs_guide.mdfor creating or editing rules, error codes (E1..En), and fixture manifests.
- Converter specification pages render at build/serve time via
- Pipeline Specifications and Multi-Environment Smoke Tests (
specs/pipelines/,specs/_loader/pipeline_spec.py,tests/helpers/pipeline_env.py,tests/e2e/test_pipeline_smoke_spec.py):- Pipeline specification pages render at build/serve time via
docs_macros(docs/_templates/pages/pipeline_spec.md.j2anddocs/_templates/tables/*.md.j2), with.gitignoreddocs/<name>_specification.mdstubs materialized byuv run python -m docs_macros. - Pipeline suites under
specs/pipelines/<name>/are auto-discovered bydiscover_pipeline_specs()indocs_macros/specs/discovery.pyfromrules.yamlfront-matter (pipeline,cli_command,settings_class,pipeline_runner,presets,use_converter_base_cases). When adding or removing a pipeline specification, register/update it inzensical.toml(nav) anddocs/index.md. - See
docs/pipeline_specs_guide.mdanddocs/pipeline_e2e_testing_guide.mdfor creating or editing pipeline rules, error scenarios (_rejectedandexpected_report), and running smoke tests across environments (--dlt-config-dir,--settings-file,--drop-existing-tables,--drop-tables-after).
- Pipeline specification pages render at build/serve time via
- Public Symbol / Registry Refactors (
core/,pipelines/,readers/,converters/,coercion/,validation/,datasets/,io/,cli/,parsers/,utils/):- Update module and class/function docstrings in the modified files.
- Grep
docs/andREADME.mdfor all references to modified or removed symbols and update code examples.
- Pre-Completion Verification:
- Run
uv run pytest -m docs,uv run python -m docs_macros,uv run --group docs zensical build --clean --strict, anduv run python -m docs_macros --cleanbefore finishing any task that touchessrc/,docs_macros/,scripts/,specs/,pyproject.toml, ordocs/.
- Run
- dlt 1.30 DuckDB destination fails with a Binder Error "Ambiguous reference to catalog or schema" when the db filename stem equals the dataset name; keep the db filename distinct from both pipeline and dataset names.
- dlt 1.30:
AnyDestinationis a quoted TypeAlias (a str at runtime), soAnyDestination | Nonein an evaluated annotation raises TypeError on import. UseTDestinationReferenceArgfromdlt.common.destination.reference. - pyiceberg SqlCatalog engines leak sqlite connections; use the context-manager protocol (
with load_catalog(...) as catalog) soclose()disposes the engine. - pipelines are run on the KBase CTS (CDM Task Service), which only saves the contents of the output directory (or dlt destination) if the container exits with exit code 0. For any other error code, the container STDERR and STDOUT are saved, but nothing else. As a result, even if pipelines experience failure (e.g.
load_info.has_failed_packagesis true), the container must exit 0 to ensure that logs, parsed data, and other artefacts are not deleted. Errors that occur before a pipeline is run (e.g. argument parsing errors) are not caught, so they should be picked up by the container STDERR/STDOUT.