Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: CI

on:
push:
branches: [ "main", "master" ]
pull_request:
branches: [ "main", "master" ]

jobs:
lint-and-test:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "latest"

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12.3"

- name: Install dependencies
run: uv sync

- name: Check formatting (ruff)
run: uv run ruff format --check .

- name: Lint (ruff)
run: uv run ruff check .

- name: Type check (pyright)
run: uv run pyright

- name: OCA Lint
run: bash scripts/oca_lint.sh

- name: Run unit tests (pytest)
run: uv run pytest tests/ -x -q
15 changes: 15 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
repos:
- repo: local
hooks:
- id: ruff-format
name: ruff format
entry: uv run ruff format --force-exclude
language: system
types: [python]
require_serial: false
- id: ruff-check
name: ruff check
entry: uv run ruff check --force-exclude
language: system
types: [python]
require_serial: false
26 changes: 26 additions & 0 deletions AGENTS.MD
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# AI Agent Rules for Parqcast Workspace

This document defines the constraints, architecture boundaries, and tooling instructions for AI coding assistants working in the `parqcast` repository. Before generating or editing code, agents must understand and adhere strictly to these rules.

## 1. Toolchain & Workspace Constraints
- **Workspace Tool**: The project uses `uv` as the workspace manager. All dependency management and command running must go through `uv` (e.g., `uv sync`, `uv run pytest`).
- **Strict Type Checking**: We use `pyright` in strict mode. All new code must pass `uv run pyright` without generating `Unknown` or `Any` leakage. Avoid casting unless interfacing with dynamic ORM attributes, and even then, rely on `pyproject.toml` suppressions where configured.
- **Linting & Formatting**: Code must pass `uv run ruff check .` and `uv run ruff format --check .`. The configured line length is 120.

## 2. Architecture Boundaries
Parqcast is strictly layered. Agents must NEVER introduce imports that violate these boundaries:
- **`packages/parqcast-core/`**: Defines data schemas, pyarrow constructs, and protocol interfaces. **Constraint:** Must NEVER import from `odoo`, `psycopg2`, or any other workspace package.
- **`packages/parqcast-collectors/`**: Executes fast raw SQL queries using `psycopg2` to build arrow arrays. **Constraint:** Must NEVER import `odoo` ORM models.
- **`packages/parqcast-ingesters/`**: Pulls data via Odoo ORM to push planning decisions back. This is the ONLY python package (outside the addon itself) permitted to use dynamic Odoo typing.
- **`packages/parqcast-transport-http/`**: Handles HTTP uploads of generated chunks.
- **`packages/parqcast/`**: The Odoo addon. It bridges the environment with the workspace packages. **Constraint:** Must pass Odoo Community Association (OCA) linting via `bash scripts/oca_lint.sh`.

## 3. Odoo Versioning (PEP 695 Generics)
Parqcast certifies per Odoo major.
- Do not use dynamic runtime version checks in core logic.
- Instead, use PEP 695 generics (`type V18`, `type V19`) to tag and enforce version-specific implementations at compile time. Pyright enforces that a `V18` component cannot be accidentally mixed with a `V19` context.
- For local type checking of Odoo symbols, use the `stubs/odoo-stubs/` local directory.

## 4. Testing
- Use `uv run pytest tests/ -x -q` to verify logic.
- Do not run `tests/test_live_db.py` without the `PARQCAST_TEST_DB` environment variable set.
11 changes: 5 additions & 6 deletions packages/parqcast-collectors/src/parqcast/collectors/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# Importing v18 / v19 registers their bundles into
# parqcast.core.registry.REGISTRY. Consumers look up collectors/suites
# via REGISTRY[version_str] rather than importing a flat list from here.
from . import v18 as v18
from . import v19 as v19
from .base import BaseCollector as BaseCollector
from .base import CoreCollector as CoreCollector
from .base import MpsCollector as MpsCollector
Expand All @@ -7,9 +12,3 @@
from .base import QualityCollector as QualityCollector
from .base import SaleCollector as SaleCollector
from .base import StockCollector as StockCollector

# Importing v18 / v19 registers their bundles into
# parqcast.core.registry.REGISTRY. Consumers look up collectors/suites
# via REGISTRY[version_str] rather than importing a flat list from here.
from . import v18 as v18 # noqa: F401
from . import v19 as v19 # noqa: F401
9 changes: 8 additions & 1 deletion packages/parqcast-collectors/src/parqcast/collectors/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,12 @@ def collect(self, key_from: int = 0, key_to: int = 0) -> pa.Table:

# Apply keyset pagination bounds
if key_from > 0 or key_to > 0:
import re

# Strip trailing ORDER BY if present to prevent syntax errors when we append our own.
# Only matches simple trailing ORDER BY clauses to avoid breaking subqueries.
sql = re.sub(r"(?i)\s+ORDER\s+BY\s+[\w\s,.]+$", "", sql)

joiner = " AND" if self._sql_has_where(sql) else "\nWHERE"
if key_from > 0:
sql += f"{joiner} {self.pk_column} >= {key_from}"
Expand Down Expand Up @@ -155,7 +161,8 @@ def _to_table(self, rows: list[OdooRow]) -> pa.Table:
# resulting Field/Table values come back as unknown even though
# their .name / schema usage is well-defined in practice.
cols: dict[str, list[object]] = {
field.name: [] for field in self.schema # pyright: ignore[reportUnknownVariableType]
field.name: []
for field in self.schema # pyright: ignore[reportUnknownVariableType]
}
for row in rows:
for i, field in enumerate(self.schema): # pyright: ignore[reportUnknownVariableType]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@

from __future__ import annotations

from . import bundle as bundle # noqa: F401 — side-effect: registers v18 bundle
from . import bundle as bundle
from .mps import MpsForecastCollectorV18, MpsScheduleCollectorV18
from .mrp_bom import (
BomByproductCollectorV18,
BomCollectorV18,
BomLinesCollectorV18,
BomOperationsCollectorV18,
)
from .mps import MpsForecastCollectorV18, MpsScheduleCollectorV18
from .mrp_production import MrpProductionCollectorV18
from .mrp_workorder import MrpWorkorderCollectorV18
from .orderpoint import OrderpointCollectorV18
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@
from ..currency import CurrencyCollector
from ..partner import PartnerCollector
from ..product_category import ProductCategoryCollector
from .mps import MpsForecastCollectorV18, MpsScheduleCollectorV18
from .mrp_bom import (
BomByproductCollectorV18,
BomCollectorV18,
BomLinesCollectorV18,
BomOperationsCollectorV18,
)
from .mps import MpsForecastCollectorV18, MpsScheduleCollectorV18
from .mrp_production import MrpProductionCollectorV18
from .mrp_workorder import MrpWorkorderCollectorV18
from .orderpoint import OrderpointCollectorV18
Expand Down Expand Up @@ -168,9 +168,7 @@
)


V18_COLLECTORS: tuple[type, ...] = tuple(
cls for suite in V18_SUITES for cls in suite.collector_classes
)
V18_COLLECTORS: tuple[type, ...] = tuple(cls for suite in V18_SUITES for cls in suite.collector_classes)


append_to_bundle(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

from __future__ import annotations

from . import bundle as bundle # noqa: F401 — side-effect: registers v19 bundle
from . import bundle as bundle
from .mps import MpsForecastCollectorV19, MpsScheduleCollectorV19
from .mrp_bom import (
BomByproductCollectorV19,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,7 @@
)


V19_COLLECTORS: tuple[type, ...] = tuple(
cls for suite in V19_SUITES for cls in suite.collector_classes
)
V19_COLLECTORS: tuple[type, ...] = tuple(cls for suite in V19_SUITES for cls in suite.collector_classes)


append_to_bundle(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ def visit(name: str) -> None:
visit(c.name)
return order


# Default time budget: 4.5 minutes, leaving 30s buffer for cleanup
DEFAULT_TIME_BUDGET = 270

Expand Down Expand Up @@ -349,7 +350,9 @@ def _finalize(self, cr: ReadCursor, run: ExportRun, prefix: str, t0: float) -> J
uploaded = self.chunk_cls.find_by_state(cr, run.id, "uploaded")
file_metas = [chunk_rec.get_metadata(cr) for chunk_rec in uploaded]

errors = [f"{c.collector}: {c.state}" for c in self.chunk_cls.find_by_state(cr, run.id, "error")]
errors = [
f"{c.collector}: {c.error_message or c.state}" for c in self.chunk_cls.find_by_state(cr, run.id, "error")
]

# Re-probe for capabilities (needed for manifest)
bundle, probe = _resolve_bundle(cr)
Expand All @@ -359,6 +362,7 @@ def _finalize(self, cr: ReadCursor, run: ExportRun, prefix: str, t0: float) -> J
files=file_metas,
company=self.company,
company_id=self.company_id,
odoo_version=caps.odoo_version,
total_duration=round(time.monotonic() - t0, 3),
errors=errors,
warnings=[],
Expand All @@ -373,9 +377,6 @@ def _finalize(self, cr: ReadCursor, run: ExportRun, prefix: str, t0: float) -> J
run.set_state(cr, "done")
self._commit()

self.run_cls.cleanup_old(cr)
self._commit()

logger.info(
"Run %s complete: %d files, %d errors",
run.run_uuid[:8],
Expand Down
2 changes: 1 addition & 1 deletion packages/parqcast-core/src/parqcast/core/manifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def build_manifest(
files: list[ChunkMetadata],
company: str,
company_id: int,
odoo_version: str = "17.0",
odoo_version: str,
parqcast_version: str = "0.1.0",
errors: list[str] | None = None,
warnings: list[str] | None = None,
Expand Down
15 changes: 7 additions & 8 deletions packages/parqcast-core/src/parqcast/core/protocols.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,21 +16,21 @@
from __future__ import annotations

from collections.abc import Mapping, Sequence
from typing import Any, NotRequired, Protocol, TypeAlias, TypedDict, runtime_checkable
from typing import Any, NotRequired, Protocol, TypedDict, runtime_checkable

# ---------------------------------------------------------------------------
# Named type aliases shared across the workspace.
# ---------------------------------------------------------------------------

SqlParams: TypeAlias = Sequence[object] | Mapping[str, object] | None
type SqlParams = Sequence[object] | Mapping[str, object] | None
"""Positional or named parameters accepted by psycopg2's execute()."""

SqlWithParams: TypeAlias = tuple[str, SqlParams]
type SqlWithParams = tuple[str, SqlParams]
"""Return shape of ``BaseCollector.get_sql()``: a SQL string paired with
bind-parameters (or None). Kept as a named alias so collector subclasses
match the abstract signature verbatim."""

JsonDict: TypeAlias = dict[str, object]
type JsonDict = dict[str, object]
"""A JSON-serializable dict payload.

Used for manifests, capability summaries, chunk metadata, and anywhere
Expand All @@ -39,7 +39,7 @@
nested JSON union because the consumers accept anything ``json.dumps``
can handle (via ``default=str`` if needed)."""

OdooRow: TypeAlias = tuple[object, ...]
type OdooRow = tuple[object, ...]
"""One row returned by a psycopg2/Odoo cursor's ``fetchone``/``fetchall``.

Columns are positional; the tuple's arity depends on the SELECT clause."""
Expand Down Expand Up @@ -159,9 +159,7 @@ def __setattr__(self, name: str, value: Any) -> None: ...
class OdooModel(Protocol):
"""An Odoo model accessor — what ``env["<model>"]`` returns."""

def create(
self, vals: Mapping[str, object] | list[Mapping[str, object]]
) -> OdooRecord: ...
def create(self, vals: Mapping[str, object] | list[Mapping[str, object]]) -> OdooRecord: ...
def search(
self,
domain: Sequence[object],
Expand All @@ -180,5 +178,6 @@ class OdooEnvironment(Protocol):
"""

company: OdooRecord
cr: ReadCursor

def __getitem__(self, model_name: str) -> OdooModel: ...
3 changes: 1 addition & 2 deletions packages/parqcast-core/src/parqcast/core/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,7 @@ def append_to_bundle(
current = REGISTRY.get(version)
if current is None:
raise RuntimeError(
f"append_to_bundle: no bundle registered for Odoo {version!r}; "
f"bootstrap should have installed one."
f"append_to_bundle: no bundle registered for Odoo {version!r}; bootstrap should have installed one."
)
REGISTRY[version] = replace(
current,
Expand Down
9 changes: 5 additions & 4 deletions packages/parqcast-core/src/parqcast/core/tracking.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,7 @@ def ensure_table(cls, cr: ReadCursor):
""")

@classmethod
def create(
cls, cr: ReadCursor, company_id: int | None = None, company_name: str = ""
) -> ExportRun:
def create(cls, cr: ReadCursor, company_id: int | None = None, company_name: str = "") -> ExportRun:
run_uuid = str(_uuid.uuid4())
cr.execute(
"""
Expand Down Expand Up @@ -154,6 +152,7 @@ def __init__(
key_from: int = 0,
key_to: int = 0,
estimated_seconds: float = 0,
error_message: str | None = None,
):
self.id = id
self.run_id = run_id
Expand All @@ -163,6 +162,7 @@ def __init__(
self.key_from = key_from
self.key_to = key_to
self.estimated_seconds = estimated_seconds
self.error_message = error_message

@classmethod
def ensure_table(cls, cr: ReadCursor):
Expand Down Expand Up @@ -220,7 +220,7 @@ def create(
def find_by_state(cls, cr: ReadCursor, run_id: int, state: str) -> list[ExportChunk]:
cr.execute(
"""
SELECT id, run_id, collector, sequence, state, key_from, key_to, estimated_seconds
SELECT id, run_id, collector, sequence, state, key_from, key_to, estimated_seconds, error_message
FROM parqcast_export_chunk
WHERE run_id = %s AND state = %s
ORDER BY sequence
Expand All @@ -237,6 +237,7 @@ def find_by_state(cls, cr: ReadCursor, run_id: int, state: str) -> list[ExportCh
key_from=r[5],
key_to=r[6],
estimated_seconds=r[7],
error_message=r[8],
)
for r in fetch_all(cr)
]
Expand Down
22 changes: 18 additions & 4 deletions packages/parqcast-core/src/parqcast/core/version_gate.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@

from __future__ import annotations

import importlib

from parqcast.core.protocols import ReadCursor
from parqcast.core.registry import REGISTRY
from parqcast.core.version import SupportedVersionStr, UnsupportedOdooVersionError
Expand All @@ -21,10 +23,7 @@ def _read_odoo_major(cr: ReadCursor) -> str:
same source of truth used by :func:`parqcast.core.capabilities.probe`.
Returns an empty string if the row is missing.
"""
cr.execute(
"SELECT latest_version FROM ir_module_module "
"WHERE name = 'base' AND state = 'installed' LIMIT 1"
)
cr.execute("SELECT latest_version FROM ir_module_module WHERE name = 'base' AND state = 'installed' LIMIT 1")
row = cr.fetchone()
if not row or not row[0]:
return ""
Expand All @@ -40,6 +39,21 @@ def assert_supported(cr: ReadCursor) -> SupportedVersionStr:
"""
major = _read_odoo_major(cr)
if major and major in REGISTRY:
bundle = REGISTRY[major]
if bundle.probe_capabilities is None:
# The bundle is registered but hasn't been imported yet.
# Force import now to ensure the version is genuinely supported and not
# just an empty bootstrap placeholder.
try:
importlib.import_module(f"parqcast.collectors.v{major}.bundle")
if REGISTRY[major].probe_capabilities is None:
raise RuntimeError("Import succeeded but probe_capabilities is still None")
except Exception as e:
raise UnsupportedOdooVersionError(
f"Odoo {major} is registered, but the corresponding subpackage "
f"failed to load. This parqcast installation might be corrupt "
f"or incomplete. Error: {e}"
) from e
return major
supported = sorted(REGISTRY.keys()) or ["<none>"]
detected = major or "<unknown>"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
from .base import BaseIngester as BaseIngester

# Importing v18 / v19 registers their ingesters into
# parqcast.core.registry.REGISTRY[version].ingesters.
from . import v18 as v18 # noqa: F401
from . import v19 as v19 # noqa: F401
from . import v18 as v18
from . import v19 as v19
from .base import BaseIngester as BaseIngester
from .v19 import (
DistributionActorV19,
OrderpointActorV19,
Expand Down
Loading
Loading