Skip to content

Commit da3b1a9

Browse files
authored
Add Datasets module for managing geospatial and tabular data uploads (#45)
* feat(gis_datasets): catalog module for uploading & managing geospatial datasets Adds a new ``gis_datasets`` module that lets users upload arbitrary GIS files (GeoJSON, Shapefile, KML, GeoTIFF, CSV, etc.), browse them with extracted metadata (CRS, bbox, feature/band counts), and download them. * Local-filesystem storage abstracted behind ``LocalDatasetStorage`` so an S3 backend can drop in later without touching the service. * Best-effort metadata extraction (stdlib JSON for GeoJSON; optional ``fiona``/``rasterio`` paths for Shapefile/KML/raster) — never blocks an upload when a parser is missing. * REST API (``/api/gis_datasets``) + Inertia views (Browse/Create/Edit/Show), permissions group, sidebar menu item, locale namespace, and a health check that probes the storage dir. * Alembic migration for the new ``gis_datasets_dataset`` table with ``branch_labels=("gis_datasets",)``. Framework changes: * ``i18n_manifest._serialize`` emits single-quoted TS string literals to match the project's biome ``quoteStyle: "single"``, so the generated ``keys.generated.ts`` is biome-clean without a follow-up format pass. * ``test_health.test_ready_no_checks_is_healthy`` now tolerates module- registered checks (the old assertion broke whenever a new module registered one). https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * refactor(datasets): rename gis_datasets module to datasets Renames the module so it can catalog any dataset kind (not only GIS). Mechanical changes only — no behaviour changes: * ``modules/gis_datasets/`` → ``modules/datasets/`` (and inner ``gis_datasets/`` package → ``datasets/``). * Class names: ``GisDatasetsModule`` → ``DatasetsModule``, ``GisDatasetsServices`` → ``DatasetsServices``, ``GisDatasetsSettings`` → ``DatasetsSettings``. * Module meta: ``"GisDatasets"`` → ``"Datasets"``; routes flip from ``/api/gis_datasets`` and ``/gis_datasets`` to ``/api/datasets`` and ``/datasets``; Inertia namespace ``Datasets/Browse`` etc. * Permissions: ``gis_datasets.{view,upload,edit,delete}`` → ``datasets.*`` (group label "Datasets"). * Settings env prefix: ``SM_GIS_DATASETS_*`` → ``SM_DATASETS_*``; default storage dir ``./var/datasets``. * Locale namespace ``gis_datasets`` → ``datasets``; sidebar label "Datasets". * Health check name ``datasets.storage``. Database: * Drops the ``gis_datasets_dataset`` migration and replaces it with ``add_datasets_table`` creating ``datasets_dataset`` (table prefix follows the new module name; SQLite parity intact). Migration carries ``branch_labels=("datasets",)``. All 585 tests still pass; lint and ``make doctor`` clean. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * feat(datasets): expand public contract surface for consumer modules Makes the Datasets module easy to depend on. Downstream modules now have a single stable import path (``datasets.contracts``) and a one-line FastAPI dependency. Service protocol ``IDatasetService`` gains: * ``get_by_slug(slug)`` — slug is the URL-friendly handle other modules are most likely to persist as a foreign reference. * ``list_by_kind(kind, *, limit=None)`` — lets consumers filter to just the kinds they care about (e.g. a tiling module only fetches rasters). * ``get_file(dataset_id)`` → ``DatasetFile`` — opaque handle bundling the on-disk path, filename, mime type, and full ``DatasetOut`` metadata so a consumer can open/parse bytes without reaching into ``datasets.storage`` (which would trip SM009). Public value types: * ``DatasetFile`` — ``path`` + ``metadata`` + ``open()`` helper; designed to survive a future S3 storage backend swap. * URL builders ``download_url`` / ``detail_url`` / ``show_url`` plus ``API_PREFIX`` / ``VIEW_PREFIX`` constants, so dependent UIs don't hard-code ``/api/datasets/...``. Events carry slug: * ``DatasetUploaded`` and ``DatasetDeleted`` now include ``slug`` so subscribers can route/index without a round-trip to the service. Dependency ergonomics: * ``DatasetServiceDep`` ``Annotated`` alias — consuming endpoints write ``datasets: DatasetServiceDep`` instead of ``Depends(get_dataset_service)``. ``datasets.contracts.__init__`` now re-exports the whole surface with a docstring pointing consumers at the right entry points. Contract tests live in ``tests/test_public_surface.py`` — anything that fails there is an API-break for dependent modules. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * build(datasets): add package.json + tsconfig so CI tsc checks the module Without its own ``tsconfig.json`` the datasets module's .tsx pages compiled only indirectly via the host's ``moduleGlobs``, so the ``ci-js-typecheck`` Makefile loop skipped them entirely. Type errors in Browse/Create/Edit/Show would have slipped past CI. Added: * ``package.json`` — declares the workspace name and UI/Inertia peers (mirrors ``modules/products/package.json``). * ``tsconfig.json`` — extends ``@simple-module/tsconfig/base.json`` with the standard ``@/*`` and ``@simple-module/ui/*`` path aliases. * ``pyproject.toml`` hatch ``force-include`` so the wheel ships the module-root ``package.json`` (host discovers JS deps via ``importlib.resources`` after a pip install). ``make ci-js-typecheck`` now runs ``tsc -p modules/datasets/tsconfig.json``. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * chore(i18n): regenerate keys after merging main * refactor(datasets): delegate bytes storage to file_storage.StorageBackend Datasets no longer owns its own filesystem. Bytes live in whatever ``file_storage`` backend is configured (local FS today, S3 tomorrow), so swapping the host's ``SM_FILE_STORAGE_BACKEND`` to ``s3`` gets every dataset upload/download/delete S3-native for free. Module wiring: * ``meta.depends_on = ["FileStorage"]`` — ensures file_storage's ``register_settings`` runs first, so ``app.state.file_storage.backend`` is populated before any datasets endpoint fires. * Dropped ``LocalDatasetStorage`` and the datasets-owned storage health check (file_storage has its own). * Dropped ``SM_DATASETS_STORAGE_DIR`` — file_storage's settings are the single source of truth. Service layer: * ``DatasetService.__init__`` now takes a ``StorageBackend`` directly. * Upload path spools to a tempfile (for fiona/rasterio/stdlib extractors that need a path), then streams bytes to ``backend.put()``. * Keys live under a ``datasets/`` prefix in the backend namespace so they don't collide with generic ``file_storage`` uploads. * Delete cleans up the backend object after the DB row is gone and tolerates already-missing objects via ``StorageNotFoundError``. Public contract: * ``DatasetFile`` is now backend-agnostic — no ``path`` attribute (that only made sense for local FS). Consumers call ``await file.stream()``, ``await file.read()``, or ``await file.materialize_to_tempfile()`` to get a local ``Path`` for GIS libraries that insist on one. * Download endpoint streams via ``StreamingResponse`` and will 302 to a presigned URL when the backend supports it. Tests updated to inject a ``FilesystemBackend(root=tmp_path)`` directly; public-surface tests cover the async ``stream`` / ``materialize_to_*`` pathway that downstream modules will rely on. All 7 CI items green locally: 741 Python tests, 8 JS tests, lint, typecheck (Python + TS), file-size, hardcoded-strings, biome. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * refactor(datasets): drop IDatasetService Protocol (single-impl) Follows the convention main #39 established: ship a Protocol only for real extension points (e.g. ``file_storage.StorageBackend`` with a registry and multiple backends). Datasets has one implementation and consumers already type-hint against the concrete ``DatasetService`` via the ``DatasetServiceDep`` annotated alias — the Protocol was ceremony. * Delete ``datasets/contracts/service.py`` and drop the re-export. * Tighten docstrings so nothing still points at the old Protocol. Also regenerates ``keys.generated.ts`` / ``generated-resources.ts`` to include the new feature_flags keys alongside datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * feat(ui): add layers icon so Datasets sidebar item renders an icon Sidebar menu items render their icon via ``NavIcon``'s static SVG map. My datasets module's ``register_menu_items`` set ``icon="layers"`` but the map had no entry for that name, so the sidebar showed "Datasets" as text only while every other module got a glyph. Added a Heroicons ``layers`` path — the sidebar now renders consistently. Verified visually by running the dev stack and walking through the full flow: upload a GeoJSON, land on Browse with a populated row, follow into Show (dl/dt/dd detail card), then Edit (Card + form). All four datasets pages (Browse, Create, Edit, Show) use the same PageShell / Card / Table / Empty primitives as Products, Files, and Users. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * feat(datasets): defer metadata extraction to a Celery background task Metadata extraction is the only expensive bit of the upload path — GeoJSON is fast but ``fiona`` (shapefile/KML) and ``rasterio`` can block for tens of seconds on large uploads, which is far too long to hold an HTTP request open. Moved extraction off the request path into a Celery task that the ``background_tasks`` module's worker picks up automatically. Flow on upload: 1. Endpoint spools bytes to a temp file for size-validation purposes. 2. Service persists a ``Dataset`` row with ``extraction_status="pending"`` — no bbox/CRS/feature_count yet — and uploads the bytes to ``file_storage.StorageBackend``. 3. Endpoint enqueues ``datasets.extract_metadata`` via Celery (task name exported as ``EXTRACT_METADATA_TASK``). 4. Worker calls ``extract_metadata_task(dataset_id)``: pulls bytes from the backend to a local tempfile, runs ``extract_metadata``, updates the row via the ``background_tasks.sync_db`` session. New files: * ``datasets/tasks.py`` — ``extract_metadata_task`` shared-task. Uses ``asyncio.run`` to drive the async file_storage backend from inside the sync Celery worker. Autodiscovered by ``background_tasks.celery_app`` — no framework hook. * ``tests/test_tasks.py`` — unit tests with ``sync_db`` + file_storage pointed at tmp paths via ``monkeypatch`` env vars. Verifies the task fills in bbox/CRS/feature_count and degrades to ``not_found`` when the row has been deleted between enqueue and run. * ``tests/conftest.py`` — autouse fixture that stubs ``app.state.background_tasks.celery`` with a ``MagicMock`` (mirrors the admin suite's setup) so the upload endpoint can call ``send_task`` without a live Redis broker in CI. Module wiring: * ``meta.depends_on = ["FileStorage", "BackgroundTasks"]``. * ``datasets/pyproject.toml`` picks up ``background-tasks`` + ``celery`` as explicit deps so ``datasets`` can be installed standalone. * New ``get_celery`` dep in ``datasets/deps.py`` reads from ``app.state.background_tasks.celery``. Refactored service: * ``DatasetService.register_upload`` no longer calls ``extract_metadata`` — it persists the row, uploads bytes, and leaves ``extraction_status="pending"``. The endpoint does the enqueue. The background_tasks admin UI at ``/admin/background-tasks`` now shows every dataset upload as a task row, walks it through ``pending → running → success/failed``, and makes it retryable. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * refactor(datasets): extract magic strings to constants; drop GIS framing Introduces ``datasets/constants.py`` as the single source of truth for every module identifier — permissions, menu, routes, pages, Celery task name, env prefix, table/schema names, dependency module names, kinds, extraction-status values, defaults. Every other file imports from it instead of embedding string literals, mirroring the pattern used by ``file_storage``, ``products``, ``background_tasks``, etc. Files touched: * ``module.py`` — ``MODULE_PASCAL``, ``ROUTE_PREFIX_*``, ``MODULE_FILE_STORAGE``/``MODULE_BACKGROUND_TASKS`` (for ``depends_on``), ``PERMISSION_GROUP``, ``ALL_PERMISSIONS``, menu constants. * ``endpoints/api.py`` — ``PERM_DATASETS_*`` via ``RequiresPermission(...)``, ``TASK_EXTRACT_METADATA`` for the Celery enqueue, ``ALL_KINDS`` for the kind allow-list. * ``endpoints/views.py`` — module-local ``_PAGE_*`` string constants (Name-only literals, required so ``simple_module_core.diagnostics`` SM003 orphan-page check can resolve them via its AST walker); ``constants.PERM_DATASETS_*`` for permissions. * ``service.py``, ``tasks.py`` — ``ExtractionStatus.*``, ``DEFAULT_*``, ``STORAGE_KEY_PREFIX``, ``TASK_EXTRACT_METADATA``. * ``models.py``, ``settings.py``, ``extractors.py``, ``schemas.py`` — ``SCHEMA_NAME``, ``TABLE_DATASET``, ``ENV_PREFIX``, ``DatasetKind`` enum, ``DEFAULT_GEOJSON_CRS``, ``DEFAULT_MAX_UPLOAD_MB``. Also drops the remaining "GIS"/"geospatial" framing from user-visible strings and docstrings — the catalog now describes itself as a generic dataset catalog with optional geospatial metadata extraction. The ``fiona``/``rasterio`` library references stay in code comments because those are the actual library names. * ``locales/en.json`` — "geospatial datasets" → "datasets" across browse description, empty state, create description. * ``models.py`` — "A geospatial dataset ..." → "A dataset ...". * ``extractors.py`` — module docstring drops "GIS" framing. * ``contracts/files.py`` — "GIS library" → "parser library" / named libraries. All 7 CI items green locally: 816 python tests, 8 JS tests, ruff format/check, ty, biome, tsc x 11 workspaces, 300-line cap, ``check_hardcoded_strings`` clean, ``make doctor`` shows no SM003 warnings against datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * feat(datasets): wire feature flags, role mapping, settings registration Rounds out the module's integration with the framework's governance primitives. Every identifier lives in ``datasets/constants.py``. Feature flags (registered via ``register_feature_flags``): * ``datasets.auto_extract`` (default on) — guards the Celery ``send_task`` call on the upload endpoint. Admins can freeze auto-extraction during incidents without redeploying; Dataset rows stay ``pending`` and can be re-triggered later. * ``datasets.allow_raster_uploads`` (default on) — 422s GeoTIFF uploads with a clear "raster uploads are disabled" message when turned off, useful on instances without ``rasterio`` installed. Permission role mapping (via ``registry.map_role``): * The ``user`` role now inherits ``datasets.view`` + ``datasets.upload`` so non-admins can browse and self-upload. ``edit`` / ``delete`` stay admin-only through the framework wildcard. Settings registration (registered via ``on_startup`` against the ``settings`` module's registry, skipped gracefully if the settings module isn't installed): * ``datasets.max_upload_mb`` — runtime override for the per-dataset upload size. ``get_max_upload_bytes`` reads this on every upload request, falling back to the env-backed default on the module's ``DatasetsSettings``. * ``datasets.default_kind`` — the kind to assign when filename-based detection falls back to ``other``. Resilience fix: * ``send_task`` wrapped in a logger-suppressed try/except. A dead broker (Redis down, network blip) no longer cascades into a failed upload — the dataset row is already persisted with ``extraction_status="pending"`` and the failure is logged for operator retry. Tests: * ``tests/test_wiring.py`` covers flag registration, permission role mapping, setting registration (via ``getattr(app.state, "settings", None)`` so it skips gracefully), and the flag-toggle behaviours on upload (auto-extract off → no ``send_task``; raster flag off → 422). Verified visually with a live dev run — see ``.playwright-cli/video-2026-04-20T17-02-55-633Z.webm`` for the full walk-through (login → upload → bbox/CRS extraction → browse → show → feature flags admin → module settings admin). All 7 CI items green: ruff/ty/biome/tsc, 823 python tests, 8 JS tests, file-size cap, ``check_hardcoded_strings`` clean, ``make doctor`` 0 errors on datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv * fix(ci): sync package-lock.json with @simple-module/datasets workspace CI's ``make install-js`` runs ``npm ci`` which fails when the lock file disagrees with any workspace ``package.json``. The ``@simple-module/datasets`` workspace was added a while back but its entry never made it into ``package-lock.json``, so every JS job (lint, typecheck, tests, build) was failing during install with:: npm error Missing: @simple-module/datasets@0.1.0 from lock file Local ``npm install`` had been silently fixing this each time I ran it which is why CI was the only place that surfaced the gap. Re-ran ``npm install`` and committed the resulting lock-file delta (17 lines, just the new workspace's metadata).
1 parent e378f91 commit da3b1a9

42 files changed

Lines changed: 3303 additions & 12 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ node_modules/
2626
*.db
2727
*.sqlite3
2828

29+
# Module-managed runtime state (e.g. uploaded dataset files,
30+
# default storage_dir for SM_DATASETS_STORAGE_DIR).
31+
var/
32+
2933
# file_storage filesystem backend default root (override via SM_FILE_STORAGE_FS_ROOT_PATH).
3034
uploads/
3135

‎framework/hosting/simple_module_hosting/i18n_manifest.py‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -75,15 +75,12 @@ def emit_frontend_types(registry: I18nRegistry, project_root: Path) -> None:
7575
logger.exception("Failed to write generated-resources.ts — frontend types will be stale")
7676

7777

78-
_RESOURCES_HEADER = """\
79-
// AUTO-GENERATED by simple_module_hosting.i18n_manifest — do not edit by hand.
80-
// Regenerate by booting the host in development mode.
81-
"""
78+
_RESOURCES_HEADER = (
79+
"// AUTO-GENERATED by simple_module_hosting.i18n_manifest — do not edit by hand.\n"
80+
"// Regenerate by booting the host in development mode."
81+
)
8282

83-
_KEYS_HEADER = """\
84-
// AUTO-GENERATED by simple_module_hosting.i18n_manifest — do not edit by hand.
85-
// Regenerate by booting the host in development mode.
86-
"""
83+
_KEYS_HEADER = _RESOURCES_HEADER
8784

8885
_PLURAL_SUFFIXES: tuple[str, ...] = tuple(f"_{c}" for c in PLURAL_CATEGORIES)
8986

@@ -125,7 +122,7 @@ def _render_resources(keys: list[str]) -> str:
125122
def _render_keys(keys: list[str]) -> str:
126123
"""Render the nested ``keys`` constant tree."""
127124
tree = _build_key_tree(keys)
128-
return f"{_KEYS_HEADER}\nexport const keys = {_serialize(tree, indent=0)} as const;\n"
125+
return f"{_KEYS_HEADER}\n\nexport const keys = {_serialize(tree, indent=0)} as const;\n"
129126

130127

131128
def _build_key_tree(flat_keys: list[str]) -> dict[str, Any]:

‎framework/hosting/tests/test_health.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,12 @@ async def check_broken():
7676
assert "connection refused" in data["checks"]["broken_service"]["detail"]
7777

7878
async def test_ready_no_checks_is_healthy(self, client: httpx.AsyncClient):
79+
# Modules may register their own health checks (e.g. gis_datasets
80+
# probes its storage dir). Assert the aggregate is healthy and
81+
# every reported check is healthy, rather than requiring zero
82+
# checks — which would break whenever a new module joins boot.
7983
resp = await client.get("/health/ready")
8084
data = resp.json()
8185
assert data["status"] == "healthy"
82-
assert data["checks"] == {}
86+
for name, payload in data["checks"].items():
87+
assert payload["status"] == "healthy", f"{name} not healthy: {payload}"
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
"""add datasets table
2+
3+
Revision ID: dad9a134290f
4+
Revises: b7e1af4c9d02
5+
Create Date: 2026-04-19 11:51:19.145277
6+
"""
7+
8+
from collections.abc import Sequence
9+
10+
import sqlalchemy as sa
11+
from alembic import op
12+
13+
# revision identifiers, used by Alembic.
14+
revision: str = "dad9a134290f"
15+
down_revision: str | None = "b7e1af4c9d02"
16+
branch_labels: str | Sequence[str] | None = ("datasets",)
17+
depends_on: str | Sequence[str] | None = None
18+
19+
20+
def upgrade() -> None:
21+
# ### commands auto generated by Alembic - please adjust! ###
22+
op.create_table(
23+
"datasets_dataset",
24+
sa.Column("is_deleted", sa.Boolean(), nullable=False),
25+
sa.Column("deleted_at", sa.DateTime(timezone=True), nullable=True),
26+
sa.Column("deleted_by", sa.String(length=255), nullable=True),
27+
sa.Column(
28+
"created_at",
29+
sa.DateTime(timezone=True),
30+
server_default=sa.text("(CURRENT_TIMESTAMP)"),
31+
nullable=False,
32+
),
33+
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
34+
sa.Column("created_by", sa.String(length=255), nullable=True),
35+
sa.Column("updated_by", sa.String(length=255), nullable=True),
36+
sa.Column("id", sa.Integer(), nullable=False),
37+
sa.Column("name", sa.String(length=200), nullable=False),
38+
sa.Column("slug", sa.String(length=200), nullable=False),
39+
sa.Column("kind", sa.String(length=32), nullable=False),
40+
sa.Column("description", sa.String(length=2000), nullable=True),
41+
sa.Column("original_filename", sa.String(length=255), nullable=False),
42+
sa.Column("mime_type", sa.String(length=127), nullable=True),
43+
sa.Column("size_bytes", sa.Integer(), nullable=False),
44+
sa.Column("storage_key", sa.String(length=512), nullable=False),
45+
sa.Column("crs", sa.String(length=64), nullable=True),
46+
sa.Column("bbox_min_x", sa.Float(), nullable=True),
47+
sa.Column("bbox_min_y", sa.Float(), nullable=True),
48+
sa.Column("bbox_max_x", sa.Float(), nullable=True),
49+
sa.Column("bbox_max_y", sa.Float(), nullable=True),
50+
sa.Column("feature_count", sa.Integer(), nullable=True),
51+
sa.Column("band_count", sa.Integer(), nullable=True),
52+
sa.Column("extraction_status", sa.String(length=16), nullable=False),
53+
sa.PrimaryKeyConstraint("id", name=op.f("pk_datasets_dataset")),
54+
)
55+
op.create_index(
56+
"ix_datasets_dataset_is_deleted", "datasets_dataset", ["is_deleted"], unique=False
57+
)
58+
op.create_index(op.f("ix_datasets_dataset_kind"), "datasets_dataset", ["kind"], unique=False)
59+
op.create_index(op.f("ix_datasets_dataset_slug"), "datasets_dataset", ["slug"], unique=True)
60+
# ### end Alembic commands ###
61+
62+
63+
def downgrade() -> None:
64+
# ### commands auto generated by Alembic - please adjust! ###
65+
op.drop_index(op.f("ix_datasets_dataset_slug"), table_name="datasets_dataset")
66+
op.drop_index(op.f("ix_datasets_dataset_kind"), table_name="datasets_dataset")
67+
op.drop_index("ix_datasets_dataset_is_deleted", table_name="datasets_dataset")
68+
op.drop_table("datasets_dataset")
69+
# ### end Alembic commands ###

‎host/pyproject.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ dependencies = [
88
"auth",
99
"dashboard",
1010
"products",
11+
"datasets",
1112
"permissions",
1213
"background-tasks",
1314
"file-storage",
@@ -21,6 +22,7 @@ simple-module-hosting = { workspace = true }
2122
auth = { workspace = true }
2223
products = { workspace = true }
2324
dashboard = { workspace = true }
25+
datasets = { workspace = true }
2426
permissions = { workspace = true }
2527
background-tasks = { workspace = true }
2628
file-storage = { workspace = true }
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Datasets module."""
Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
"""Stable identifiers for the datasets module.
2+
3+
Every string that identifies a permission, role, page, module dependency,
4+
event, route, or config key lives here. Everything else in the module
5+
imports from this file, so ``rg`` can prove the module has no scattered
6+
magic strings.
7+
"""
8+
9+
from __future__ import annotations
10+
11+
from typing import Final
12+
13+
# ── Module identity ──────────────────────────────────────────────────
14+
MODULE_NAME: Final = "datasets"
15+
MODULE_PASCAL: Final = "Datasets"
16+
MODULE_DISPLAY_NAME: Final = "Datasets"
17+
18+
# ── Configuration ────────────────────────────────────────────────────
19+
ENV_PREFIX: Final = "SM_DATASETS_"
20+
21+
# ── Routing ──────────────────────────────────────────────────────────
22+
ROUTE_PREFIX_API: Final = "/api/datasets"
23+
ROUTE_PREFIX_VIEW: Final = "/datasets"
24+
25+
# REST sub-paths (joined to ROUTE_PREFIX_API by the router)
26+
PATH_DOWNLOAD: Final = "/{dataset_id}/download"
27+
PATH_DATASET: Final = "/{dataset_id}"
28+
29+
# View routes (relative to ROUTE_PREFIX_VIEW, used in redirects)
30+
REDIRECT_BROWSE: Final = "/datasets/"
31+
32+
# ── Module dependencies (for ModuleMeta.depends_on) ──────────────────
33+
MODULE_FILE_STORAGE: Final = "FileStorage"
34+
MODULE_BACKGROUND_TASKS: Final = "BackgroundTasks"
35+
36+
# ── i18n ─────────────────────────────────────────────────────────────
37+
LOCALE_NAMESPACE: Final = MODULE_NAME
38+
39+
# ── Inertia page identifiers ─────────────────────────────────────────
40+
PAGE_BROWSE: Final = "Datasets/Browse"
41+
PAGE_CREATE: Final = "Datasets/Create"
42+
PAGE_EDIT: Final = "Datasets/Edit"
43+
PAGE_SHOW: Final = "Datasets/Show"
44+
45+
# ── Permissions ──────────────────────────────────────────────────────
46+
PERM_DATASETS_VIEW: Final = "datasets.view"
47+
PERM_DATASETS_UPLOAD: Final = "datasets.upload"
48+
PERM_DATASETS_EDIT: Final = "datasets.edit"
49+
PERM_DATASETS_DELETE: Final = "datasets.delete"
50+
51+
PERMISSION_GROUP: Final = "Datasets"
52+
ALL_PERMISSIONS: Final = (
53+
PERM_DATASETS_VIEW,
54+
PERM_DATASETS_UPLOAD,
55+
PERM_DATASETS_EDIT,
56+
PERM_DATASETS_DELETE,
57+
)
58+
59+
# ── Menu ─────────────────────────────────────────────────────────────
60+
MENU_LABEL: Final = "Datasets"
61+
MENU_ICON: Final = "layers"
62+
MENU_ORDER: Final = 40
63+
64+
# ── Celery tasks ─────────────────────────────────────────────────────
65+
TASK_EXTRACT_METADATA: Final = "datasets.extract_metadata"
66+
67+
# ── Feature flags ────────────────────────────────────────────────────
68+
FLAG_AUTO_EXTRACT: Final = "datasets.auto_extract"
69+
FLAG_ALLOW_RASTER_UPLOADS: Final = "datasets.allow_raster_uploads"
70+
71+
# ── Settings keys (registered with the settings module) ──────────────
72+
SETTING_MAX_UPLOAD_MB: Final = "datasets.max_upload_mb"
73+
SETTING_DEFAULT_KIND: Final = "datasets.default_kind"
74+
75+
# ── Roles ────────────────────────────────────────────────────────────
76+
ROLE_USER: Final = "user"
77+
ROLE_ADMIN: Final = "admin"
78+
79+
# Permissions granted to plain ``user`` role via ``map_role``. ``admin``
80+
# inherits everything through the framework's wildcard, so we don't list
81+
# it explicitly here.
82+
USER_ROLE_PERMISSIONS: Final = (
83+
PERM_DATASETS_VIEW,
84+
PERM_DATASETS_UPLOAD,
85+
)
86+
87+
# ── Health check ─────────────────────────────────────────────────────
88+
HEALTH_CHECK_STORAGE: Final = "datasets.storage"
89+
90+
# ── Storage ──────────────────────────────────────────────────────────
91+
STORAGE_KEY_PREFIX: Final = "datasets/"
92+
93+
# ── Database ─────────────────────────────────────────────────────────
94+
SCHEMA_NAME: Final = "datasets"
95+
TABLE_DATASET: Final = "datasets_dataset"
96+
97+
# ── Defaults ─────────────────────────────────────────────────────────
98+
DEFAULT_MAX_UPLOAD_MB: Final = 256
99+
DEFAULT_UPLOAD_CHUNK_SIZE: Final = 1024 * 1024 # 1 MB
100+
DEFAULT_PRESIGN_TTL_SECONDS: Final = 300 # 5 minutes
101+
DEFAULT_FALLBACK_FILENAME: Final = "upload.bin"
102+
DEFAULT_MIME_TYPE: Final = "application/octet-stream"
103+
DEFAULT_GEOJSON_CRS: Final = "EPSG:4326"
104+
105+
106+
class DatasetKind:
107+
"""Built-in dataset kinds.
108+
109+
The ``kind`` column is a free-form string so new providers can register
110+
their own labels, but these are the ones the shipped extractors and
111+
the UI know about.
112+
"""
113+
114+
VECTOR_GEOJSON: Final = "vector_geojson"
115+
VECTOR_SHAPEFILE: Final = "vector_shapefile"
116+
VECTOR_KML: Final = "vector_kml"
117+
RASTER_GEOTIFF: Final = "raster_geotiff"
118+
TABULAR_CSV: Final = "tabular_csv"
119+
OTHER: Final = "other"
120+
121+
122+
ALL_KINDS: Final = (
123+
DatasetKind.VECTOR_GEOJSON,
124+
DatasetKind.VECTOR_SHAPEFILE,
125+
DatasetKind.VECTOR_KML,
126+
DatasetKind.RASTER_GEOTIFF,
127+
DatasetKind.TABULAR_CSV,
128+
DatasetKind.OTHER,
129+
)
130+
131+
132+
class ExtractionStatus:
133+
"""Values stored in ``Dataset.extraction_status``.
134+
135+
``pending`` is set by the upload endpoint; ``ok``/``partial``/``failed``
136+
are set by the Celery worker once extraction completes. ``manual`` is
137+
the fallback for kinds the extractor doesn't understand; ``not_found``
138+
is only used as a task-result marker (never stored).
139+
"""
140+
141+
PENDING: Final = "pending"
142+
OK: Final = "ok"
143+
PARTIAL: Final = "partial"
144+
FAILED: Final = "failed"
145+
MANUAL: Final = "manual"
146+
NOT_FOUND: Final = "not_found"
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
"""Datasets contracts — public interface for other modules.
2+
3+
Downstream modules should import from here, not from
4+
``datasets.models`` or ``datasets.service`` internals. The SM009
5+
diagnostic enforces framework→plugin purity; this package is the
6+
supported surface for plugin→plugin coupling::
7+
8+
from datasets.contracts import (
9+
DatasetOut, # DTO returned by service lookups
10+
DatasetFile, # handle for stored file access
11+
DatasetUploaded, # event subscribers listen for
12+
download_url, # URL helper for UIs
13+
)
14+
15+
For the FastAPI dependency, prefer
16+
``from datasets.deps import DatasetServiceDep``. Type-hint against the
17+
concrete ``DatasetService`` — the module is single-impl, so there's no
18+
Protocol abstraction (matching the framework's "ship a Protocol only
19+
for real extension points" rule).
20+
"""
21+
22+
from datasets.contracts.events import DatasetDeleted, DatasetUploaded
23+
from datasets.contracts.files import DatasetFile
24+
from datasets.contracts.schemas import (
25+
KIND_VALUES,
26+
DatasetKind,
27+
DatasetOut,
28+
DatasetUpdate,
29+
)
30+
from datasets.contracts.urls import (
31+
API_PREFIX,
32+
VIEW_PREFIX,
33+
detail_url,
34+
download_url,
35+
show_url,
36+
)
37+
38+
__all__ = [
39+
"API_PREFIX",
40+
"KIND_VALUES",
41+
"VIEW_PREFIX",
42+
"DatasetDeleted",
43+
"DatasetFile",
44+
"DatasetKind",
45+
"DatasetOut",
46+
"DatasetUpdate",
47+
"DatasetUploaded",
48+
"detail_url",
49+
"download_url",
50+
"show_url",
51+
]
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
"""Domain events emitted by the Datasets module.
2+
3+
Events carry the handful of fields subscribers most often route on
4+
(``slug``, ``kind``) so downstream handlers don't have to make a round
5+
trip to ``IDatasetService`` just to decide whether the event is for
6+
them. Anything beyond this — CRS, bbox, feature counts — is still a
7+
``get_by_id`` away.
8+
"""
9+
10+
from __future__ import annotations
11+
12+
from dataclasses import dataclass
13+
14+
from simple_module_core.events import Event
15+
16+
17+
@dataclass
18+
class DatasetUploaded(Event):
19+
dataset_id: int
20+
name: str
21+
slug: str
22+
kind: str
23+
24+
25+
@dataclass
26+
class DatasetDeleted(Event):
27+
dataset_id: int
28+
slug: str

0 commit comments

Comments
 (0)