diff --git a/.agents/skills/coding-style/SKILL.md b/.agents/skills/coding-style/SKILL.md new file mode 100644 index 00000000..160228bf --- /dev/null +++ b/.agents/skills/coding-style/SKILL.md @@ -0,0 +1,94 @@ +--- +name: coding-style +description: >- + Coding style, formatting, docstring, comment, typing, and code reuse guidelines for cdm-data-loaders. + Use when writing, modifying, refactoring, formatting, or reviewing Python code in this repository. +--- + +# Coding style + +## Formatting + +- Docstrings are in Sphinx / reST format +- All functions must have a docstring, the list of params and param type, and the return and return type. +- Constants are in all-caps and must never start with an underscore. Where possible, use immutable types or assign the type `Final` to the constant. +- Fully type-annotated code (ruff ANN); datetime values carry an explicit timezone (DTZ); use Decimal for exact arithmetic, never constructed from floats. +- Do not add `__all__` blocks. + + +## Conventions + +- Use Pydantic for models unless the context requires very large numbers of objects + - If Pydantic would have a significant negative impact on performance, use frozen, slotted dataclasses. +- Use pydantic-settings for CLI argument parsing. Settings are frozen once constructed. +- Pipelines use the pipeline structure defined in `cdm_data_loaders.pipelines.core`. When creating a new pipeline, look at the existing pipelines under `cdm_data_loaders.pipelines.jsonlines`, `cdm_data_loaders.pipelines.xmltopyarrow`, `cdm_data_loaders.pipelines.xmltodict`, or `cdm_data_loaders.pipelines.xsv` for examples of how to structure the code. + + +## Code and patterns reuse + +- Before writing new code, search the codebase for existing functionality; if there is not an exact match, is there functionality that could be adapted? +- When using a function from the code base, look at existing usage if in doubt how it works +- Always import from the source module; do not import via another module. +- Do not make aliases solely for importing into another module; use `import as ` instead +- DO NOT disable ruff rules to make the checks pass. Fix the root issue instead. + + +## Prose + +- Use "and" instead of "&" + + +## Code Comments + +- Inline comments start with lowercase (`# resolve the schema`) +- NEVER add comments that separate code blocks (`# -- sqlglot_dialect mapping --`) +- NEVER add comments with a line of hyphens above and below, like the following: +``` +# -------------- +# some comment +# -------------- +``` +- DO NOT state the obvious +- DO NOT document your thought process +- DO NOT talk to yourself or to the coder +- Comments should ONLY refer to the current code and its usage. They should never to historic versions of the code, +sets of edits being made, or anything else that is not directly related to executing the code. + + +# Documentation + +- The aim of standalone documentation is to assist people executing codebase functions, e.g. running scripts from the entrypoints in `pyproject.toml`. +- Do not add lists of symbols, functions, classes, errors, etc., that can be seen in the auto-generated API docs. +- Do not add diagrams of class hierarchies. +- Do not use numbered lists unless you are describing a set of steps and need to refer back to a previous step. +- Do add examples of how to use functions, as in `docs/table_transforms.md` +- Add links to the appropriate codebase files. +- Separate out user documentation from developer documentation. + - User documentation covers executing pipelines, configuration, creating custom transforms or preprocessors, supplying schemas, and anything else about *using* the code. + - Developer documentation includes running tests, how to create new sets of specs for a pipeline or converter, smoke testing, and anything involving codebase internals and editing the codebase. Specific details about implementations should go in comments in the codebase itself. + + +## Types and data structures + +- Assume python 3.13+ +- Always use full generic parametrization -- `dict[str, Any]` not `dict` +- All tests should return `None` +- Prefer `functools.partial` over `lambda` when currying + + +## Testing + +- Use the `testing` skill for testing guidelines. + + +## Python code execution + +- cdm-data-loaders uses uv to manage Python virtual environments and dependencies +- commands should be run using `uv run python ...` or `uv run pytest ...`. +- After writing code, run `uv run ruff format` for formatting. +- Run `uv run ruff check` to check for code linting issues. + +## Generated files and documentation sync + +- Run `uv run python -m scripts.gen_pyarrow_schema` (or `--check`) to regenerate or verify committed PyArrow `schema.py` modules from bundled XSD schemas. +- Run `uv run pytest -m docs`, `uv run python -m docs_macros`, `uv run --group docs zensical build --clean --strict`, and `uv run python -m docs_macros --clean` to verify macro-rendered CLI reference, bundled schemas, and converter/pipeline specification pages. diff --git a/.agents/skills/testing/SKILL.md b/.agents/skills/testing/SKILL.md new file mode 100644 index 00000000..7eb14ef7 --- /dev/null +++ b/.agents/skills/testing/SKILL.md @@ -0,0 +1,88 @@ +--- +name: testing +description: >- + Testing conventions, structure, methodology, parametrization, fixtures, mocking, and execution rules for cdm-data-loaders. + Use when writing, updating, debugging, or reviewing tests of any type. +--- + +# Testing + +## Style and structure + +- Module-based tests (plain `def test_*() -> None:` functions), not class-based +- Use pytest with fixtures in `conftest.py` +- Tests should be named `test__` +- Tests should have a one line description indicating what the test is doing. +- Each test should return `None` and have fully typed parameters. + +## Test methodology + +- Before writing tests, read through the code and understand what it does. If you don't understand it or you need other files for context, ask. Do not proceed without fully understanding the context. + +- Every function and class instantiation should be unit-tested. This includes: + - Happy path + - Every code path must be exercised + - Every exception must be raised + - Boundary conditions (empty strings, empty lists, None, zero, negative values, etc.) + +- Pydantic classes must have all validator paths tested for success and failure conditions. + +- Tests should check precise results: test that the output is exactly the data structure and values that are expected. + +- If fixing a bug, write a test that reproduces the bug but expects the correct answer. The test should initially fail. Fix the bug and verify that the test passes. + +- Ensure all code paths are covered by tests. Use coverage tools to identify untested code if an execution environment is available. + +## Reduce test sprawl + +- Do not create separate tests if all that varies is the assertions at the end of the test. Combine the assertions into a single test. + +- If the same set up is used for numerous tests, check `conftest.py` files, the test module itself, and `test.helpers.*` to see if functions or fixtures already exist. If not, add new fixtures/functions in the appropriate location. + +- When testing a function, do not execute that function more than once within the test _unless_ testing caching. Use `pytest.mark.parametrize` for testing the same function with different parameters. ALWAYS use human readable ids for the test cases when using `pytest.mark.parametrize`. + +## Autouse Fixtures + +- Autouse fixtures may be in `conftest.py` files or in the test module itself. Always check before adding new fixtures. + +## Large files for test cases + +- Do not keep large test cases in Python modules. Use the `data` folder and organise files according to the data source. + +## Parallel safety and isolation + +- Tests should be idempotent and order-independent. + +- Assume that tests may be run in any order, that they may be run multiple times, and that they may be run in parallel. + +- Do not pollute the global state and avoid side effects wherever possible. + +- Do not create files within the working directory; always use a temporary file or directory. + +- Avoid external resources that clash with other tests. + +- Tests must work on Linux, macOS and Windows. Avoid encoding paths and platform-specific code. + +## Mocking + +- Do not mock internal code. Only mock external dependencies (e.g. network calls, database connections, s3 bucket access). + +- use pytest's tmp_path fixture to create temporary files and directories for testing. + +- Calls to external HTTP resources should be recorded and replayed using `vcrpy` and `pytest-recording`. + +- Use the moto library to mock AWS services. + +- Integration and end-to-end tests may use multiple containers to create an appropriate working environment for fully testing code that requires services such as databases, catalogs, and s3. + +## External dependencies + +- Tests that contact an external service (e.g. an FTP site, etc.) should be marked with `@pytest.mark.external_request`. + +- Tests with external dependencies (e.g. Spark, Ceph) should be marked (using `@pytest.mark.requires_spark`, `@pytest.mark.requires_ceph`) and skipped by default. They should only be run in a controlled environment where the external dependencies are available. + +- Spark and Ceph tests are run in containers. Use the `docker-compose.lakehouse.yml` file to start the containers and run the tests inside them. Do not run these tests on the host machine. + +## Running tests + +- Run tests using `uv run pytest`.