From bdf0b0937bb5fcfbfafcf1d90d1d1d8cbb04ad7c Mon Sep 17 00:00:00 2001 From: fstwn Date: Wed, 21 Oct 2026 17:36:57 +0200 Subject: [PATCH 1/4] docs: roadmap 0.6.3.1 to 0.6.7, search and location map in 0.6.4, xeokit for later --- docs/adr/FUTURE.md | 8 +++- docs/adr/IMPLEMENTATION_PLAN_0.6.md | 59 +++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 2 deletions(-) diff --git a/docs/adr/FUTURE.md b/docs/adr/FUTURE.md index 78a51e42..586e91a7 100644 --- a/docs/adr/FUTURE.md +++ b/docs/adr/FUTURE.md @@ -28,8 +28,12 @@ gap. - **IFC export via IfcOpenShell** --- proxies map to `IfcExtrudedAreaSolid` / `IfcTriangulatedFaceSet`, properties to a PSet (spec section 10.4). Unlocks professional AEC workflows. *(On the 0.6.0.x backlog since 2026-10-08, plan P12 "Backlog after P12"; to grill.)* -- **xeokit** --- open-source WebGL BIM viewer; could replace or augment the component viewer, - with IFC support. High effort, high visual impact. +- **xeokit** (for later) --- open-source WebGL BIM viewer for whole IFC building models. In CSC it could show a + piece in its donor building (floor, axis, facade highlighted) and check our own IFC export; replacing the + three.js component viewer brings nothing for a single piece. Only once donor-building IFC models exist (no + dataset has one so far), and only after clearing the licence: the xeokit SDK is AGPL-3.0, CSC is MIT, and + AGPL code in the served web app would bind it to the AGPL (a commercial licence costs money). The IFC + export does not need it. High effort. ## Geometry search and processing diff --git a/docs/adr/IMPLEMENTATION_PLAN_0.6.md b/docs/adr/IMPLEMENTATION_PLAN_0.6.md index 8ffeab00..a43f967e 100644 --- a/docs/adr/IMPLEMENTATION_PLAN_0.6.md +++ b/docs/adr/IMPLEMENTATION_PLAN_0.6.md @@ -944,6 +944,65 @@ The SH descriptor stays blocked on its licence. --- +### Roadmap after 0.6.3.0 --- proposed 2026-10-21, adjusted by the user the same day; each release is grilled before it is built + +**0.6.3.1 --- hardening after the release (about 3 days):** a per-address limit on failed access-token lookups; +token clean-up (tokens of deleted accounts removed, revoked rows deleted after about 90 days); viewer: a version +switch with the mesh Off no longer loads the hidden level; viewer: mesh edges and faces clip each other at some +angles (user 2026-10-21) --- the edges should win, most likely through `polygonOffset` on the mesh material so the +faces sit a hair behind their edges (edges drawn with depth test off would show the hidden back edges too, so not +that), and check the camera near / far planes against the piece's size in case the faces themselves are cut +when zooming in; stale file markers after a geometry PATCH on a draft (`mesh_ply_resolutions`, +`point_cloud_files`, `FUTURE.md`); the geometry cache drops the descriptor keys on a frame-only cut; the planar +outline's vertex limit counts holes; the paired rebound linked on the core record's evidence card; RL1 checked +(two layouts on one piece show both cards and bar sets); the snapshot-scoped fold confirmed by a test; +housekeeping (em dashes as empty values in `app/admin/users/page.tsx` and `InvitationsTable.tsx`, the long +README line on `SMTP_REPLY_TO`, `tests/api/test_mail_templates.py` and its `delenv`, a fresh `npm audit`). + +**0.6.4 --- search and the location map (about 7--9 days; user 2026-10-21):** a catalog of about 2300 pieces +needs new kinds of questions more than indexes. (a) **Dimensions** ("what fits?"): range filters on the stored +size (`bbx`) per axis with tolerances, an orientation-independent envelope match (the three sizes sorted), and +sorting by closeness to the asked size; in MCP as parameters of `search_components` or a `find_fitting` tool +(about 1--2 days). (b) **Location**: the versions' `location` and the origin building's `Place.location` also +stored as GeoJSON points with a `2dsphere` index (Atlas M0 has it); `$geoNear` gives the nearest k or everything +within a radius, with the distance in metres, combined with the other filters; in MCP as `near` (coordinates, +radius, count) or a `find_near` tool (about 1--1.5 days on top of (c)). (c) The **location map** (user +2026-10-21): a second map page next to the Similarity Map with every visible component as a marker (clustered), +a click opening the shared preview panel, visibility as on Browse, on the field and index of (b); to grill: +which location (current, origin, or both as layers), the map library and tile source (the CSP of 8.164 allows +only the Google embed; OpenStreetMap tiles send each visitor's address to OSM and allow light use only, so the +CSP and the privacy page change) (about 2--3 days). (d) **Sorting and overview**: sort by catalog number, date, +size, distance, dimension match, rising or falling; a `catalog_overview` MCP tool with counts by material, +function, dataset and shape class (about 1 day). (e) **True similarity**: kNN on the descriptor vectors +(radial signature, scalars) instead of the 2D map layout, a NumPy pass at this size (no Atlas Vector Search +needed), combinable with (a) and (b) (about 1--2 days). Later: search by measured properties from the evidence +fold (once it is clear which quantities are searched); Atlas Search for typo-tolerant text (weakest gain). + +**0.6.5 --- reinforcement from Grasshopper (about 6--8 days; moved from 0.6.4 by the user 2026-10-21):** (a) units +in GH: every component scales by the document unit (a prerequisite for placing scans); (b) RL8, a GH builder +for the `rebar_spec` claim and its display, grilled together with (c); (c) Ferroscan to reinforcement layout, +parts (a) report reader, detection, placement and a draft through `AddEvidence`, (b) the report image and PDF +as attachments, (d) several scans of one face; (c) "the image in 3D" later. + +**0.6.6 --- interoperability (about 5--7 days):** the IFC export (`compose?format=ifc`; first the `ifcopenshell` +wheel check for the server, then IFC4 vs IFC4x3 and single piece vs selection); the timeline as PROV-O / +JSON-LD / GraphML (8.140 h); optionally a vocab page the JSON-LD `@vocab` resolves to. + +**0.6.7 --- shape descriptors (research, about 3--7 days):** HKS v2 behind a retrieval check (nothing ships if +the neighbours do not look right); if it holds, the HKS basis on the Similarity Map, the level 1 curve and the +level 2 heat map, and the descriptor kNN of 0.6.4 (e) gains from it; the SH descriptor only if its licence has +arrived. + +**Outside a release (the maintainer):** the legal check of the privacy page by the university's data protection +office; the Atlas traffic measurement of 0.6.1.0 (c); the backend CI job back to `ubuntu-latest` once MongoDB +8.0 starts on the runner's kernel (SERVER-121912). + +**Later or ideas:** the GH sign-in on macOS (Keychain); Ferroscan "the image in 3D"; MCP opened to members +(after 0.6.3.1); the CSC logo and brand, then the logo in mails, Open Graph and the favicon; the partners on +the credits page (their consent); usage statistics without cookies; timber evidence methods; the analytics +page; transactions for record plus change log; xeokit (for later, see `FUTURE.md`); a similarity-search +endpoint and the other `FUTURE.md` ideas. + ## 4. Dependencies at a glance ``` From d4945b0b9df6acd6efb16b4bdd4add3731b9f43a Mon Sep 17 00:00:00 2001 From: fstwn Date: Wed, 21 Oct 2026 19:51:13 +0200 Subject: [PATCH 2/4] fix: hardening 0.6.3.1 (token limits and clean-up, viewer depth and edges, file markers, paired rebound, release-prep, local launch and scripts without user paths) --- .claude/launch.json | 65 ++-- .gitattributes | 1 + CHANGELOG.md | 30 ++ CONTRIBUTING.md | 2 +- README.md | 38 +- docs/adr/DESIGN_DECISIONS.md | 4 +- docs/adr/FUTURE.md | 5 - docs/adr/IMPLEMENTATION_PLAN_0.6.md | 2 +- scripts/ci/release_prep.py | 131 +++++++ scripts/conversion/convert_ply_to_gltf.py | 9 +- scripts/conversion/convert_sas_meshes_RPY3.py | 18 +- scripts/db_maintenance/import_sas_database.py | 21 +- scripts/dev/run_csc_python.cmd | 60 ++++ scripts/dev/serve_migrated.py | 15 +- scripts/import/epfl_stacking/README.md | 5 +- scripts/import/recreate/README.md | 5 +- scripts/prototyping/hks_features.py | 16 +- src/backend/apps/catalog/access_tokens.py | 24 ++ src/backend/apps/catalog/api/access_tokens.py | 32 +- src/backend/apps/catalog/api/auth.py | 3 +- .../apps/catalog/api/snapshot_lifecycle.py | 25 ++ src/backend/apps/catalog/documents.py | 3 + src/backend/apps/catalog/file_markers.py | 109 ++++++ src/backend/apps/catalog/geometry_cache.py | 14 +- src/backend/apps/catalog/mcp_server.py | 21 +- src/backend/apps/catalog/oauth_server.py | 13 +- src/backend/apps/catalog/proxies/specs.py | 13 +- src/backend/apps/catalog/token_check.py | 55 ++- src/backend/apps/catalog/token_cleanup.py | 60 ++++ src/backend/main_fastapi.py | 7 + src/backend/usermaintenance.py | 11 + src/frontend/app/admin/users/page.tsx | 6 +- .../components/admin/InvitationsTable.tsx | 4 +- .../components/components/ComponentViewer.tsx | 156 ++++++--- .../detail/ComponentRecordsCard.tsx | 7 +- .../components/evidence/EvidenceCard.tsx | 28 +- src/frontend/components/viewer/ProxySolid.tsx | 6 +- src/frontend/lib/evidence/paired.test.ts | 68 ++++ src/frontend/lib/evidence/paired.ts | 51 +++ src/frontend/lib/reinforcementColors.test.ts | 56 +++ src/frontend/lib/reinforcementColors.ts | 36 ++ src/frontend/lib/viewerDepth.test.ts | 150 ++++++++ src/frontend/lib/viewerDepth.ts | 79 +++++ src/frontend/lib/viewerSwitches.test.ts | 20 ++ src/frontend/lib/viewerSwitches.ts | 15 + .../public/backgroundmeshes/manifest.json | 3 +- tasks.py | 97 +++++- tests/api/support.py | 34 +- tests/api/test_dump_helpers.py | 73 ++++ tests/api/test_evidence_fold_routes.py | 28 ++ tests/api/test_geometry_patch_markers.py | 222 ++++++++++++ tests/api/test_geometry_remote.py | 27 ++ tests/api/test_mail_templates.py | 19 +- tests/api/test_token_cleanup.py | 326 ++++++++++++++++++ tests/api/test_token_failure_limit.py | 160 +++++++++ tests/catalog/test_proxies.py | 47 +++ tests/ci/test_release_prep.py | 173 ++++++++++ 57 files changed, 2533 insertions(+), 175 deletions(-) create mode 100644 scripts/ci/release_prep.py create mode 100644 scripts/dev/run_csc_python.cmd create mode 100644 src/backend/apps/catalog/file_markers.py create mode 100644 src/backend/apps/catalog/token_cleanup.py create mode 100644 src/frontend/lib/evidence/paired.test.ts create mode 100644 src/frontend/lib/evidence/paired.ts create mode 100644 src/frontend/lib/viewerDepth.test.ts create mode 100644 src/frontend/lib/viewerDepth.ts create mode 100644 tests/api/test_dump_helpers.py create mode 100644 tests/api/test_geometry_patch_markers.py create mode 100644 tests/api/test_token_cleanup.py create mode 100644 tests/api/test_token_failure_limit.py create mode 100644 tests/ci/test_release_prep.py diff --git a/.claude/launch.json b/.claude/launch.json index 55a220fb..a8539b66 100644 --- a/.claude/launch.json +++ b/.claude/launch.json @@ -2,14 +2,22 @@ "version": "0.0.1", "configurations": [ { - "name": "backend-migrated", - "runtimeExecutable": "C:\\Users\\EFESTWIN\\miniconda3\\envs\\csc\\python.exe", + "name": "backend-local", + "runtimeExecutable": "cmd", "runtimeArgs": [ - "-m", - "invoke", - "dev-migrated" + "/c", + "set CSC_PUBLIC_API_URL=http://127.0.0.1:8791&& scripts\\dev\\run_csc_python.cmd -u scripts/dev/serve_migrated.py --dump latest --no-migrate --assets .dev/assets-261012 --port 8791 --mongo-port 27791" ], - "port": 8000 + "port": 8791 + }, + { + "name": "frontend-local", + "runtimeExecutable": "cmd", + "runtimeArgs": [ + "/c", + "set FASTAPI_URL=http://127.0.0.1:8791&& set NEXTAUTH_URL=http://localhost:3000&& npm --prefix src/frontend run dev" + ], + "port": 3000 }, { "name": "frontend", @@ -23,58 +31,29 @@ "port": 3000 }, { - "name": "frontend-migrated", - "runtimeExecutable": "cmd", - "runtimeArgs": [ - "/c", - "set FASTAPI_URL=http://127.0.0.1:8000&& npm --prefix src/frontend run dev" - ], - "port": 3000 - }, - { - "name": "frontend-migrated-3100", + "name": "backend-migrated", "runtimeExecutable": "cmd", "runtimeArgs": [ "/c", - "set FASTAPI_URL=http://127.0.0.1:8000&& set NEXTAUTH_URL=http://127.0.0.1:3100&& npm --prefix src/frontend run dev -- -p 3100" + "scripts\\dev\\run_csc_python.cmd -m invoke dev-migrated" ], - "port": 3100, - "url": "http://127.0.0.1:3100" + "port": 8000 }, { - "name": "frontend-migrated-8010", + "name": "frontend-migrated", "runtimeExecutable": "cmd", "runtimeArgs": [ "/c", - "set FASTAPI_URL=http://127.0.0.1:8010&& set NEXTAUTH_URL=http://127.0.0.1:3100&& set NEXT_DIST_DIR=.next-3100&& npm --prefix src/frontend run dev -- -p 3100" - ], - "port": 3100, - "url": "http://127.0.0.1:3100" - }, - { - "name": "backend-261012", - "runtimeExecutable": "C:/Users/EFESTWIN/miniconda3/envs/csc/python.exe", - "runtimeArgs": [ - "-u", - "scripts/dev/serve_migrated.py", - "--dump", - "261012", - "--no-migrate", - "--assets", - ".dev/assets-261012", - "--port", - "8791", - "--mongo-port", - "27791" + "set FASTAPI_URL=http://127.0.0.1:8000&& npm --prefix src/frontend run dev" ], - "port": 8791 + "port": 3000 }, { - "name": "frontend-261012", + "name": "frontend-migrated-3100", "runtimeExecutable": "cmd", "runtimeArgs": [ "/c", - "set FASTAPI_URL=http://127.0.0.1:8791&& set NEXTAUTH_URL=http://127.0.0.1:3100&& set NEXT_DIST_DIR=.next-3100&& npm --prefix src/frontend run dev -- -p 3100" + "set FASTAPI_URL=http://127.0.0.1:8000&& set NEXTAUTH_URL=http://127.0.0.1:3100&& npm --prefix src/frontend run dev -- -p 3100" ], "port": 3100, "url": "http://127.0.0.1:3100" diff --git a/.gitattributes b/.gitattributes index 17a66083..0cc2affe 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,3 +2,4 @@ *.sh text eol=lf uberspaceconfig/crontab/crontab.example text eol=lf uberspaceconfig/html/** text eol=lf +*.cmd text eol=crlf diff --git a/CHANGELOG.md b/CHANGELOG.md index ee2f04fc..9a5770cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,36 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.6.3.1] - unreleased + +A hardening release after 0.6.3.0. + +### Changed + +- **Viewer:** mesh edges stay crisp over the faces instead of breaking up at some angles; close + zooms no longer cut a piece open, and small pieces open filling the view. A version switch + while the mesh or the point cloud is Off loads nothing hidden. +- **Evidence:** a core record links to its paired rebound record; a piece with several + reinforcement layouts shows all of them. +- **Geometry:** replacing the meshes or clouds of a draft also removes the files and markers + of the dropped ones; a frame-only cache entry no longer carries descriptor data; the outline + vertex limit counts holes. +- **Release tooling:** `invoke release-prep` sets the CHANGELOG date, writes `.zenodo.json` and + runs the version checks in one step; `bump-version --gh` leaves the Session component alone + when its client version already matches. The local launch entries work without a user path. + +### Security + +- **Failed access-token lookups are limited per address** (429 with `Retry-After`); valid tokens + are never counted. +- **Token clean-up:** the tokens and connections of a deleted account are removed with it; + revoked and expired tokens are deleted 90 days later. + +### Server notes + +- Existing token rows get their clean-up date from the daily user maintenance run (not while the + read-only flag is set); no step needed. + ## [0.6.3.0] - 2026-10-21 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4d9a6973..457c25cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,7 @@ Thank you for helping with the Catalog of Second Chances (CSC). Everyone taking Releases are made by the maintainers only; tags `v*` are protected. - `VERSION` and the changelog section are set with `invoke bump-version `; `.zenodo.json` follows with - `invoke zenodo`. + `invoke release-prep`, which also dates the section and runs the release checks. - Pushing the tag `v` on `main` runs CI, builds the three bundles (backend, frontend, Grasshopper interface), publishes the GitHub release and asks for approval before the deploy. - The Grasshopper bundle takes its files from `grasshopper_release/`: diff --git a/README.md b/README.md index 242bd877..5816314e 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,19 @@ port (a throwaway `invoke dev-migrated` with `NEXT_DIST_DIR` and `NEXTAUTH_URL=http://127.0.0.1:`) would otherwise share its NextAuth session with the first. +**A local copy of the newest dump** (for checking the web app against real +data) starts from `.claude/launch.json`: `backend-local` (the newest +`mongodb_collections_local/` folder, a throwaway `mongod` on 27791, the +backend on http://127.0.0.1:8791, assets copied to `.dev/assets-261012`) and +`frontend-local` (http://localhost:3000). Both run through +`scripts/dev/run_csc_python.cmd`, which finds the conda environment `csc` +without a path of your account: `CSC_PYTHON` (a `python.exe` you name), the +active environment when it is `csc`, the conda base from `CONDA_EXE`, the usual +install folders under `%USERPROFILE%` and `%ProgramData%`, then `conda run -n +csc`. The sign-in `dev-admin` has a new password at every start, written to +`.dev/dev-migrated-credentials-8791.txt`. A collection with no export file in +the dump is not loaded, and an export file of 0 bytes holds no documents. + ### Tests ``` @@ -198,10 +211,10 @@ two places you need to set them: **1. `~/.bash_profile`** - for the cron jobs and the deploy script: copy the template `uberspaceconfig/.bash_profile.example` and fill it in (MongoDB, JWT, SMTP with the optional `SMTP_REPLY_TO`, the support address that replies to the -verification and password-reset mails go to, asset folders, CORS origins; optionally `GITHUB_CSC_GH_TOKEN` for higher -GitHub rate limits in `CSC_Update`). Optional: `CSC_MAX_DATASETS_PER_USER` -(default 3), how many datasets a non-admin account may create; admins are -not limited. +verification and password-reset mails go to, asset folders, CORS origins; +optionally `GITHUB_CSC_GH_TOKEN` for higher GitHub rate limits in +`CSC_Update`). Optional: `CSC_MAX_DATASETS_PER_USER` (default 3), how many +datasets a non-admin account may create; admins are not limited. Apply immediately by running: `source ~/.bash_profile` @@ -273,10 +286,19 @@ push is automated. **Cutting a release** -1. On your working branch: `invoke bump-version --version 0.5.1.1` (add `--gh` - if the Grasshopper UserObjects changed --- then re-export the changed - `.ghuser` / XML in Rhino). Fill in the new `CHANGELOG.md` section: it becomes - the release notes. +1. On your working branch, in this order: + 1. `invoke bump-version --version 0.5.1.1` writes the version everywhere and + opens a `## [0.5.1.1] - unreleased` section in `CHANGELOG.md`. Add `--gh` + if the Grasshopper UserObjects changed --- then re-export the changed + `.ghuser` / XML in Rhino. With `--gh`, a client version that already is + the new version leaves `DDU_CSC_Session.py` as it is (the `Version:` + header keeps matching the exported `.ghuser`). + 2. Fill in that `CHANGELOG.md` section: it becomes the release notes. + 3. `invoke release-prep` (`--date YYYY-MM-DD`, default today) dates the + section, writes `.zenodo.json`, runs the version check for the tag and + the Zenodo check, and runs `tests/ci`. It stops with an error if the + section is missing or empty or a check fails, and prints the next steps; + a second run only checks. 2. Run the whole suite once locally, including the slow tests: `invoke test-release` (set `CSC_TEST_RHINO=1` for the headless Rhino tests and `CSC_DUMP_DIR` for the migration on a dump). CI no longer runs `tests/catalog`, diff --git a/docs/adr/DESIGN_DECISIONS.md b/docs/adr/DESIGN_DECISIONS.md index fcee1b51..2a57b90f 100644 --- a/docs/adr/DESIGN_DECISIONS.md +++ b/docs/adr/DESIGN_DECISIONS.md @@ -219,11 +219,11 @@ into one **metadata object**. | 8.82 | **Evidence files are capped at 25 MB each** (user 2026-10-03). The backend refuses a larger file with 413 before storing anything (the cap is checked on the bytes received, not on a client header); the web form says so before upload. The frontend proxy's body limit for evidence uploads sits just above the cap (multipart overhead) so the backend, not the proxy, answers; images are still re-encoded to the image budget (7.3). Settles the size in 8.74. | Rejected: no fixed cap (a proxy limit as the only bound). | | 8.83 | **P6 part 2 build decisions 8.72--8.81 confirmed, with amendments from the review** (user 2026-10-03). (a) `GET /evidence` gains `recorded_by=me`, `order=newest|oldest` and `limit`; "repeat from my last record" asks for the caller's newest record of the same method in the same dataset in one request (accepts the proposal of 8.73). (b) Upload size follows 8.82: 25 MB per evidence file, checked by the backend (413); the frontend proxy allows just above it for evidence uploads only, every other path keeps its own limit (settles 8.74). (c) The proxy passes `X-Content-Type-Options` (and `Content-Length`) through. (d) Point picking caches its per-mesh analysis and picks on the reduced mesh where one exists. | --- | | 8.84 | **P6 part 2 review fixes, as built** (build notes, 2026-10-03, confirmed by the user). (a) `GET /evidence`: `recorded_by=me` (signed in, else 401; any other value 422), `order=newest|oldest` sorts **by creation** (when the record was made, which is what "my last record" means; without `order` the list stays oldest observation first), `limit` as before; the form's "repeat from my last record" is one request `?dataset=&method=&status=all&recorded_by=me&order=newest&limit=1` (replaces the paged walk of 8.73). (b) The 25 MB cap is a ceiling in the backend: `EVIDENCE_UPLOAD_LIMIT_MB` may lower it and the backend refuses to start above 25; the check is on the bytes read (64 kB chunks), 413 `File too large (limit: 25 MB)` before anything is stored; exactly 25 MB is accepted. The form says "Up to 25 MB per file" under the file buttons, refuses a larger file when it is chosen (also the "Add a file" on a record) and the proxy lets 26 MB through for `POST /evidence/attachments` only (`proxyClientMaxBodySize: '26mb'`, every other path keeps its 5 MB), so the backend answers; measured through the proxy: 25 MB = 201, 25 MB + 1 byte = 413 from the backend, above 26 MB = 413 from the proxy. (c) The `/api/backend` proxy passes `X-Content-Type-Options` and, for a body it passes on as it came (no `Content-Encoding`), `Content-Length`. (d) Picking starts on the reduced mesh where the snapshot has one (the user can still choose another detail), and the welded vertices and triangle graph of a mesh are built once per geometry (cached while the mesh keeps its place in the pick root): the first pick on the reduced test mesh took about 90 ms, the next ones 2--7 ms. (e) Register links of the verification panel: a `iso_17025` id of the DAkkS shape (`D-XX-12345-01-00`) links to that body's certificate `https://www.dakks.de/files/data/as/pdf/.pdf`; any other id, and every notified body, links to the register's **entry page** (the DAkkS search page, the NANDO notified-body list) and says "search it": neither takes an id in the address that could be verified. (f) A method description that only repeats its name is not shown; dates of day precision show without a time in the cards, the pairing list and the outranked list; the generator's import filter now runs after the root interface is written (an import only the root interface uses was dropped) and the generated files equal the ones in the tree. | Spec 7.2; 8.73, 8.74, 8.82, 8.83. | -| 8.85 | **One drawing of the proxy** (build notes of the viewer fix, 2026-10-03, confirmed by the user). The grey solid, its edges and the deviation overlay draw the same faces of the primary proxy, built once (`lib/proxyShape.ts`, hook `useProxyShape`) from the overlay's primitive builder and placed by one matrix (Rhino z-up to y-up, mm to m, the proxy's `placement`); the earlier path that drew a box or prism without its placement (`boxAsPrism`, `ExtrusionVisualization`) is removed. No request is added: hull and cylinder come from the stored `params`, not from `/snapshots/{sid}/proxies/{i}/mesh`. (a) "Proxy" is offered as a detail whenever the primary proxy has parameters to draw, so hulls and cylinders show too; a proxy without usable parameters (non-positive size, empty hull) is not offered. (b) "Show mesh" in the Proxy detail is the solid's surface (own state, default on), "Show edges" its edge lines; the per-mesh list is for the mesh details only. Where a snapshot has only point clouds and a proxy, the Proxy block carries the same two boxes. (c) The explanations of the viewer panel (canonical vs stored, fit p95 / max, colour legends per overlay) sit behind "?" (the form's Help popover, moved to `components/ui/help.tsx` as it is now shared; for a hull proxy the distance legend says the value is the depth of the scan below the hull surface, never negative, so the map shows red only); the panel keeps controls and numbers (bounding box in mm, `box (obb): p95 .. mm, max .. mm`, `Scale ..`). The popover had a tap bug: the emulated mouse-enter of a tap opened it and the click that follows toggled it shut; a click within 600 ms of a hover-open now keeps it open (also fixes the evidence form). Keyboard: Enter or Space on the "?" toggles it; opening on focus alone was left out, because the popover takes focus when it opens and would close itself. (d) First frontend unit test: `npm test` (`tsx --test lib/proxyShape.test.ts`, no new package) checks, for a rotated box, a prism along x, a tilted cylinder and a hull, that the solid and the overlay faces have equal scene vertices and that these equal the stored placement applied by hand. | Spec 4.3, App. B; 8.23, 8.43, 8.72. | +| 8.85 | **One drawing of the proxy** (build notes of the viewer fix, 2026-10-03, confirmed by the user). The grey solid, its edges and the deviation overlay draw the same faces of the primary proxy, built once (`lib/proxyShape.ts`, hook `useProxyShape`) from the overlay's primitive builder and placed by one matrix (Rhino z-up to y-up, mm to m, the proxy's `placement`); the earlier path that drew a box or prism without its placement (`boxAsPrism`, `ExtrusionVisualization`) is removed. No request is added: hull and cylinder come from the stored `params`, not from `/snapshots/{sid}/proxies/{i}/mesh`. (a) "Proxy" is offered as a detail whenever the primary proxy has parameters to draw, so hulls and cylinders show too; a proxy without usable parameters (non-positive size, empty hull) is not offered. (b) "Show mesh" in the Proxy detail is the solid's surface (own state, default on), "Show edges" its edge lines; the per-mesh list is for the mesh details only. Where a snapshot has only point clouds and a proxy, the Proxy block carries the same two boxes. (c) The explanations of the viewer panel (canonical vs stored, fit p95 / max, colour legends per overlay) sit behind "?" (the form's Help popover, moved to `components/ui/help.tsx` as it is now shared; for a hull proxy the distance legend says the value is the depth of the scan below the hull surface, never negative, so the map shows red only); the panel keeps controls and numbers (bounding box in mm, `box (obb): p95 .. mm, max .. mm`, `Scale ..`). The popover had a tap bug: the emulated mouse-enter of a tap opened it and the click that follows toggled it shut; a click within 600 ms of a hover-open now keeps it open (also fixes the evidence form). Keyboard: Enter or Space on the "?" toggles it; opening on focus alone was left out, because the popover takes focus when it opens and would close itself. (d) First frontend unit test: `npm test` (`tsx --test lib/proxyShape.test.ts`, no new package) checks, for a rotated box, a prism along x, a tilted cylinder and a hull, that the solid and the overlay faces have equal scene vertices and that these equal the stored placement applied by hand. **Amended by 8.165 C (0.6.3.0):** the "Show mesh" checkbox of (b) is gone; the Mesh switch has an "Off" choice instead. | Spec 4.3, App. B; 8.23, 8.43, 8.72. | | 8.86 | **"Recent" belongs to the signed-in user** (build notes, 2026-10-03, confirmed by the user). The sidebar list is shown only with a valid session (not while loading, not for a session that carries an error) and nothing is recorded for anonymous visitors; it is stored per user (`csc-recent-components:`, the id is the session's `user.id`); the account menu's sign-out and both expiry paths of `SessionMonitor` remove the user's list before `signOut`; the key before this fix (one list per browser) is removed on first load. Left as it was: a session that ends while the browser is closed leaves that user's list in storage until the same user signs in again (other accounts never read it); `SessionMonitor` ignores component pages (public demo path), so on those a session expiry hides the list but clears it only on the next page that is not a component page. | Spec 8.29. | | 8.87 | **P7 shape: one snapshot form, a scan path for unused tags, corrections without a bulk tool** (user 2026-10-03). (a) One new `SnapshotForm` with four modes --- new component, cut, record new state, correct --- and the steps details, size (authored box), photos, optional inspection (new component only), submit; the 0.5 wizard (`ComponentAddWizard`, `lib/catalogCreate.ts`) is deleted, its photo capture, location and dimension parts reused; the P4 "Cut a piece from it" dialog becomes the form's cut mode. (b) `/id/{uuid}` for an unknown id offers a signed-in contributor *New component* and *Cut from pieces*; everyone else keeps 404 (amends 7.5 / spec section 7.5). (c) A correction that changes the geometry lists the evidence positioned on the previous version, each linked to its own correction in the evidence form; no bulk re-placing tool in 0.6 (amends spec section 3.3.2). (d) P7 removes the web consumers of 0.5 fields and the retired `/utility/compute-snapshot-orientation` route with `orientation.py`; the `gh-interface` page texts change with the bridge in P8. | Rejected: porting the 0.5 wizard; creation only through `/add-component?id=`; a bulk re-placing tool in P7; all text cleanup in P7. | | 8.88 | **The snapshot form, as built** (build notes of P7, 2026-10-03, confirmed, 8.91). (a) One `SnapshotForm` (`components/snapshot/`), four modes, steps by mode: new component (tag when the link carries none, details, size, photos, inspection, review), cut (parents with the new piece's own tag, details, size, photos, review), record new state and correct (details, size, photos, review). Routes: `/add-component[?id=&mode=new|cut&parent=]`, `/components/{id}/snapshot/new?mode=state` and `?mode=correct&snapshot={sid}`; buttons: *Cut a piece from it* and *Record new state* on the component page, *Correct* on each live version, *Add component* in the Capture group of the sidebar. The 0.5 wizard, `catalogCreate.ts`, its material / dataset picker and the P4 cut dialog are deleted; photo capture, the location fields (moved to `MetadataFields`) and the dimension inputs are reused. (b) Submitting is draft --> photos (photo i goes to slot i, so a retry overwrites instead of doubling) --> `submit?publish=1&promote=1`; the draft exists after the first call, so a failure in a later step resumes there and says where it stopped. A moderator's submit publishes (a cut then ends its parents' circulation, 8.8), anyone else's waits. After a new component the queued id is consumed as in 0.5. (c) *New component* asks dataset, original function, material (the materials registry), trade name and, folded away, the origin block of P4 (`OriginFields`); everything else about provenance stays on the component page. *Cut* asks only what it may override (function, material: needed when merged parents disagree, 8.33), a cut date and the fields every snapshot has; the new piece's tag is optional (generated when empty). Every snapshot form has name, quantity, fragment flag, colour, location, notes. (d) **Visual inspection** is a hand-over, not an embedded step: the step is a tick box, and after the piece exists the done page leads to the evidence form with `?method=visual_inspection` (the form takes an `initialMethod`); the evidence form needs the component to exist, and doing it there keeps one inspection form (7.9). (e) **A correction is offered for authored-box snapshots only.** `supersede` takes the corrected snapshot as a whole and copies no files, so a correction of a scan from the web would keep the preview mesh and lose the reduced and original files; the form says so for such a version (name, notes, location, colour are edited in place; a wrong scan is corrected with Grasshopper). For an authored box the geometry is sent as stored unless the size is entered again; a new size lists the published evidence positioned on the previous version, each linked to its correction in the evidence form (8.87 c; in a new tab before the submit, on the done page after it). **Open question for the coordinator:** copy the stored mesh and point-cloud files in `supersede` (backend), so that scanned snapshots can be corrected from the web? (f) *Record new state* needs a size (only a changed shape is a state) and warns when it equals the current one; the date defaults to today. (g) The edit page patches only what changed (name, notes, location, colour) of the version on screen and says who may edit it by its status (draft: author or moderator(D); pending, published: moderator(D); rejected, withdrawn: nobody); the backend decides per field (3.2.2). | Spec 3.2.2, 3.3.2, 7.6; 7.5, 7.9, 8.16, 8.87. | -| 8.89 | **The unknown tag, the scanners and 0.5 leftovers** (build notes of P7, 2026-10-03, confirmed, 8.91). (a) `/id/{uuid}` looks the id up (`GET /identities/{id}?expand=none`); anything but a 404 redirects to the component page as before. For a 404 a signed-in caller with `contributor` in some dataset (admin counts: every role) sees *This tag is not in the catalog yet* with *New component* and *Cut from pieces* (to `/add-component?id=..&mode=new|cut`); everyone else gets the plain 404, so the page tells a non-contributor nothing a 404 does not. (b) The scanners lead there by a line under their result, shown only for a contributor and only for an id the lookup says is unknown: *Scan and identify* (not found), *Locate by ID* (reference id) and *Transmit ID* (active id); *Scan and identify* now reads a link that ends in an id as well as a bare id (7.5). (c) Removed: `POST /utility/compute-snapshot-orientation`, `orientation.py`, their models and test; no web code reads or writes `type`, `extrusions`, `condition`, `consumed*`, `validated`, `iframe` or `pca_frame` (the `gh-interface` page texts stay for P8); `package.json` is 0.6.0.0, so the client header is `web/0.6.0.0`. (d) Dev tooling: `invoke dev-migrated --test-accounts [--test-dataset]` creates `dev-contrib`, `dev-mod`, `dev-mod2` (moderator and contributor) and `dev-rev` in one dataset (default `dbu_zirkus`), and `dev-outsider` with no role anywhere (what a non-member sees), with fresh passwords in the gitignored `.dev/dev-test-accounts[-].json`, never printed; `NEXT_DIST_DIR` (unset: `.next`) gives a second dev server its own build folder, because Next allows one dev server per folder: launch entry `frontend-migrated-8010` (port 3100, backend 8010, mongod 27019; `.next-*` is ignored, and `next dev` adds its type folders to `tsconfig.json`: do not commit that). | Spec 7.4, 7.5; 8.81, 8.87. | +| 8.89 | **The unknown tag, the scanners and 0.5 leftovers** (build notes of P7, 2026-10-03, confirmed, 8.91). (a) `/id/{uuid}` looks the id up (`GET /identities/{id}?expand=none`); anything but a 404 redirects to the component page as before. For a 404 a signed-in caller with `contributor` in some dataset (admin counts: every role) sees *This tag is not in the catalog yet* with *New component* and *Cut from pieces* (to `/add-component?id=..&mode=new|cut`); everyone else gets the plain 404, so the page tells a non-contributor nothing a 404 does not. (b) The scanners lead there by a line under their result, shown only for a contributor and only for an id the lookup says is unknown: *Scan and identify* (not found), *Locate by ID* (reference id) and *Transmit ID* (active id); *Scan and identify* now reads a link that ends in an id as well as a bare id (7.5). (c) Removed: `POST /utility/compute-snapshot-orientation`, `orientation.py`, their models and test; no web code reads or writes `type`, `extrusions`, `condition`, `consumed*`, `validated`, `iframe` or `pca_frame` (the `gh-interface` page texts stay for P8); `package.json` is 0.6.0.0, so the client header is `web/0.6.0.0`. (d) Dev tooling: `invoke dev-migrated --test-accounts [--test-dataset]` creates `dev-contrib`, `dev-mod`, `dev-mod2` (moderator and contributor) and `dev-rev` in one dataset (default `dbu_zirkus`), and `dev-outsider` with no role anywhere (what a non-member sees), with fresh passwords in the gitignored `.dev/dev-test-accounts[-].json`, never printed; `NEXT_DIST_DIR` (unset: `.next`) gives a second dev server its own build folder, because Next allows one dev server per folder: launch entry `frontend-migrated-8010` (port 3100, backend 8010, mongod 27019; `.next-*` is ignored, and `next dev` adds its type folders to `tsconfig.json`: do not commit that). **Amended (0.6.3.1, 2026-10-21):** the local launch entries are now one pair, `backend-local` / `frontend-local` (newest dump, `serve_migrated.py --dump latest`), started through `scripts/dev/run_csc_python.cmd`; the entry `frontend-migrated-8010` of (d) is gone. | Spec 7.4, 7.5; 8.81, 8.87. | | 8.90 | **Found while walking Appendix A, and the Next.js patch** (build notes of P7, 2026-10-03, confirmed, 8.91). (a) A piece nobody has published has no current state, so its component page showed only "no current state" and its author could neither see, recall, delete nor edit the draft: the page now lists the versions (`ComponentSnapshotVersionList`, with the lifecycle buttons) and links *Edit details*, and the edit page loads the latest version when the passport has none (the backend already shows these to the author and moderator(D), 3.1.5). (b) The evidence form crashed on such a piece (`primarySnapshot` throws on a passport without snapshots): it now says the piece has no published state yet, because evidence is recorded about a state (positions need its id); the done page of *New component* offers the inspection only once the piece is published. (c) A flat face of a mesh of at most 500 triangles counts as a face for the edge-distance check (an authored box has two triangles a face, a fitted hull one a facet, so the 6-triangle rule of 8.76 said "not a flat face" for them); scans keep the rule. (d) Next.js 16.3.6 --> 16.3.8 with `eslint-config-next` (and its `@next/eslint-plugin-next`, locked by it to 16.3.8): 16.3.7 is a backport of one Turbopack fix (a strongly consistent read hanging on a cancelled task); 16.3.8 is a security release (high: SSRF in image optimization; medium: information disclosure in metadata image routes via `dynamicParams`, cache poisoning of SSG / ISR pages (two advisories), Draft Mode content leaking into regular responses via a pending `use cache` fill, a cache leak across root params in nested `use cache`; low: information disclosure in the dev server's MCP endpoint); no breaking changes are listed; `npm run build` passes. (e) **Not changed, for the coordinator:** the form's observation date is day precision (UTC midnight), and the as-of fold puts an observation of the same day *before* a state that began later that day, so a visual inspection recorded the day a piece was catalogued does not move the condition badge (a later, timed one does), and the timeline lists such a record under "before the piece was catalogued". | Spec 3.1.5, 4.4; 8.9, 8.76, 8.88. | | 8.91 | **P7 build decisions 8.88--8.90 confirmed; scan corrections stay with Grasshopper; a today-dated observation is timed** (user 2026-10-03, after the P7 review). (a) A web correction is offered for authored-box versions only (8.88 e as built); a scanned version's name, notes, location and colour are edited in place (3.2.2), and a correction of a scan comes with a new scan from Grasshopper; whether `supersede` should keep a scan's files for a web correction is decided in P8. (b) The evidence form sends an observation dated today as *now* with minute precision, so a same-day inspection lands on the state recorded earlier that day; a past date stays day-precise, and the context resolution of spec section 4.1 is unchanged (a day-precise instant is its midnight). (c) `/id/{uuid}` answers 410 for a purged piece, as spec section 7.5 says. | Rejected: copying a scan's files in `supersede` in P7; resolving a coarse date to the end of its period. | | 8.92 | **The Grasshopper client stays CPython script UserObjects in 0.6** (user 2026-10-03, after a rundown with measurements). After P8 the heavy maths runs in the geometry runner (P5); what stays on the client is HTTP + JSON (network- and server-bound in any language) and mesh in / out, where only building a Rhino mesh from Python is bound by the CPython-to-.NET bridge (a median reduced mesh loads in under 1 s). P8 adds keep-alive, bulk mesh export with numpy for uploads, numpy parsing for downloads, a measurement of the mesh build, and two environments (the bridge needs `requests` only); the header stays `gh-userobjects/0.6.0.0`. After the cutover, gated by that measurement: a pip-installable client package with tests outside Rhino; a small .NET PLY / mesh helper via Yak only if the build is too slow; a compiled `.gha` only if tests inside Rhino, Rhino 7 or one-click installs outweigh editable scripts. | Rejected for 0.6: C# script components (port of 44 components, students lose readable scripts); a compiled `.gha` in P8 (P8 would become XL, header, `CSC_Update` and `/ghinterface` change). | diff --git a/docs/adr/FUTURE.md b/docs/adr/FUTURE.md index 586e91a7..d3eaf2e0 100644 --- a/docs/adr/FUTURE.md +++ b/docs/adr/FUTURE.md @@ -45,11 +45,6 @@ gap. and a point cloud of the same L-panel landed a quarter turn apart). Changing the descriptor invalidates every stored signature. - **Open3D** --- mesh quality analysis, better preprocessing for the geometry runner. -- **Stale file markers after a geometry PATCH** (review of 0.6.2.0 T1, 2026-10-16): a - `PATCH /snapshots/{sid}` with new `geometry` on a draft removes the fitted maps but keeps the mesh - and point cloud files and both markers (`mesh_ply_resolutions`, `point_cloud_files`), so an index - can point past the new primitives. Reads fall back to the inline copy, so nothing fails; clear the - markers (and the files) of indices the new geometry no longer has. ## Component map diff --git a/docs/adr/IMPLEMENTATION_PLAN_0.6.md b/docs/adr/IMPLEMENTATION_PLAN_0.6.md index a43f967e..b580e966 100644 --- a/docs/adr/IMPLEMENTATION_PLAN_0.6.md +++ b/docs/adr/IMPLEMENTATION_PLAN_0.6.md @@ -957,7 +957,7 @@ when zooming in; stale file markers after a geometry PATCH on a draft (`mesh_ply outline's vertex limit counts holes; the paired rebound linked on the core record's evidence card; RL1 checked (two layouts on one piece show both cards and bar sets); the snapshot-scoped fold confirmed by a test; housekeeping (em dashes as empty values in `app/admin/users/page.tsx` and `InvitationsTable.tsx`, the long -README line on `SMTP_REPLY_TO`, `tests/api/test_mail_templates.py` and its `delenv`, a fresh `npm audit`). +README line on `SMTP_REPLY_TO`, `tests/api/test_mail_templates.py` and its `delenv`, a fresh `npm audit`). Added (user 2026-10-21): `invoke release-prep` (the CHANGELOG date, `.zenodo.json`, `check_version --tag`, `tests/ci` in one step) and `bump-version --gh` leaving `DDU_CSC_Session.py` alone when its client version already matches. And `.claude/launch.json` committable: a launcher that finds the `csc` environment without an account path, `serve_migrated.py --dump latest` (empty exports load as no documents), one `backend-local` / `frontend-local` pair. **0.6.4 --- search and the location map (about 7--9 days; user 2026-10-21):** a catalog of about 2300 pieces needs new kinds of questions more than indexes. (a) **Dimensions** ("what fits?"): range filters on the stored diff --git a/scripts/ci/release_prep.py b/scripts/ci/release_prep.py new file mode 100644 index 00000000..e0b5fd2e --- /dev/null +++ b/scripts/ci/release_prep.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +""" +The pure parts of the release preparation (``invoke release-prep``, +``invoke bump-version --gh``); standard library only. + +* ``changelog_section`` / ``section_is_empty``: the CHANGELOG section of a + version and whether it holds anything to release. +* ``set_changelog_date``: ``## [] - unreleased`` becomes the date, + the file's line endings kept; a date that is already set is left alone. +* ``gh_client_update``: the Grasshopper client version in DDU_CSC_Session.py + moves only when it is not the new version already; then ``CSC_CLIENT`` and + the ``Version: `` header change together (the header date has to + keep matching the exported ``.ghuser``). +""" + +from __future__ import annotations + +import datetime +import re +from typing import Optional, Tuple + +DATE_RE = r'\d{4}-\d{2}-\d{2}' + +# the files a release preparation changes (bump-version and release-prep), by +# name, for the commit the next step asks for (no `git add -A`) +RELEASE_FILES = ( + 'CHANGELOG.md', + '.zenodo.json', + 'VERSION', + 'src/backend/csc_version.py', + 'src/frontend/package.json', + 'src/frontend/package-lock.json', + 'README.md', +) +# and with --gh, when the Grasshopper client version moved +GH_RELEASE_FILES = ( + 'grasshopper_userobjects_src/DDU_CSC_Session.py', +) + + +def next_steps(version: str) -> str: + """The text release-prep prints when it is done: the commit by file name, + then the PR and the tag.""" + files = ' '.join(RELEASE_FILES) + return ( + f'release-prep: v{version} is ready. Next, by you:\n' + f' 1. git add {files}\n' + f' (with bump-version --gh also {" ".join(GH_RELEASE_FILES)} and the\n' + f' re-exported .ghuser / XML), then git commit\n' + f' 2. invoke test-release, then open the PR into main and wait for CI\n' + f' 3. after the merge: git tag v{version} && git push origin v{version}') + + +def _heading(version: str) -> str: + return rf'^## \[{re.escape(version)}\][^\r\n]*' + + +def changelog_section(text: str, version: str) -> Optional[str]: + """The body of the section of ``version`` (without its heading), or None + when the CHANGELOG has no such section.""" + start = re.search(_heading(version), text, flags=re.MULTILINE) + if start is None: + return None + rest = text[start.end():] + end = re.search(r'^## \[', rest, flags=re.MULTILINE) + return rest[:end.start() if end else len(rest)] + + +def section_is_empty(body: Optional[str]) -> bool: + """A section says nothing when it has no line besides blanks and the + ``###`` sub-headings.""" + if body is None: + return True + for line in body.splitlines(): + stripped = line.strip() + if stripped and not stripped.startswith('###'): + return False + return True + + +def set_changelog_date(text: str, version: str, date: str + ) -> Tuple[str, str]: + """``(new_text, state)``. ``state``: ``dated`` (unreleased replaced by + ``date``), ``already`` (the heading has a date: nothing changes, whatever + the date) or ``missing`` (no section). The line endings are untouched: + only the text after the version in the heading line is replaced.""" + datetime.date.fromisoformat(date) # ValueError if not a date + match = re.search(_heading(version), text, flags=re.MULTILINE) + if match is None: + return text, 'missing' + heading = match.group(0) + if re.search(rf'\] - {DATE_RE}\s*$', heading): + return text, 'already' + if not re.search(r'\] - unreleased\s*$', heading): + return text, 'missing' + done = re.sub(r'unreleased(\s*)$', rf'{date}\1', heading) + return text[:match.start()] + done + text[match.end():], 'dated' + + +def existing_date(text: str, version: str) -> Optional[str]: + """The date in the heading of ``version``, or None.""" + match = re.search(_heading(version), text, flags=re.MULTILINE) + if match is None: + return None + found = re.search(rf'\] - ({DATE_RE})\s*$', match.group(0)) + return found.group(1) if found else None + + +def gh_client_update(text: str, version: str, today: datetime.date + ) -> Tuple[str, bool]: + """``(new_text, changed)``: the Session source with the client version of + ``version``. When ``CSC_CLIENT`` already is ``gh-userobjects/`` + nothing changes (neither it nor the ``Version:`` header). Else both are + rewritten: the client, and the header to ``today`` as ``yymmdd``. A source + without either line is an error.""" + client = re.search(r"^CSC_CLIENT = 'gh-userobjects/([^']*)'", text, + flags=re.MULTILINE) + header = re.search(r'^Version: \d+(\.\d+)?[a-z]?(?=\r?$)', text, + flags=re.MULTILINE) + if client is None: + raise ValueError('CSC_CLIENT line not found') + if client.group(1) == version: + return text, False + if header is None: + raise ValueError('Version: header not found') + out = (text[:client.start()] + f"CSC_CLIENT = 'gh-userobjects/{version}'" + + text[client.end():]) + header = re.search(r'^Version: \d+(\.\d+)?[a-z]?(?=\r?$)', out, + flags=re.MULTILINE) + out = out[:header.start()] + f'Version: {today:%y%m%d}' + out[header.end():] + return out, True diff --git a/scripts/conversion/convert_ply_to_gltf.py b/scripts/conversion/convert_ply_to_gltf.py index fe15f291..7d7dfce1 100644 --- a/scripts/conversion/convert_ply_to_gltf.py +++ b/scripts/conversion/convert_ply_to_gltf.py @@ -21,9 +21,11 @@ "6dc08bb0-4ae3-42e6-8cd9-23b49f624706" ] -# Source and destination paths +# Source and destination paths; the destination is inside the repository +# (from this script's own place), the source is a data folder of the machine SOURCE_DIR = r"D:\02_DATASETS\geometry-features-v1\meshes\original\components" -DEST_DIR = r"C:\Users\EFESTWIN\source\repos\csc\src\frontend\public\meshes" +DEST_DIR = str(Path(__file__).resolve().parents[2] + / 'src' / 'frontend' / 'public' / 'meshes') def convert_ply_to_gltf(ply_path, output_path): """Convert a PLY file to GLTF format""" @@ -99,8 +101,7 @@ def main(): } for component_id in COMPONENT_IDS ], - "total_count": converted_count, - "converted_at": str(Path().cwd()) + "total_count": converted_count } manifest_path = os.path.join(DEST_DIR, "manifest.json") diff --git a/scripts/conversion/convert_sas_meshes_RPY3.py b/scripts/conversion/convert_sas_meshes_RPY3.py index e91e9cde..3bf53375 100644 --- a/scripts/conversion/convert_sas_meshes_RPY3.py +++ b/scripts/conversion/convert_sas_meshes_RPY3.py @@ -474,10 +474,20 @@ def sas_processing( if __name__ == '__main__': - source_geometry_dir = r'C:\Users\EFESTWIN\Documents\07_tu_darmstadt_ddu\05_Promotion\07_Source\240404_SAS_Debris_Files' # NOQA - target_geometry_dir = r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\250115' # NOQA - json_source_dir = r'C:\Users\EFESTWIN\source\repos\csc\mongodb_backup\250115_0825' # NOQA - json_target_dir = r'C:\Users\EFESTWIN\source\repos\csc\mongodb_backup\SAS_NEW' # NOQA + # the SAS debris files are outside the repository: name the folder in the + # environment variable CSC_SAS_SOURCE_DIR (there is no default) + source_geometry_dir = os.environ.get('CSC_SAS_SOURCE_DIR') + if not source_geometry_dir: + raise SystemExit( + 'CSC_SAS_SOURCE_DIR is not set: set it to the folder of the SAS ' + 'debris files (one sub-folder per object) before running.') + # the rest is inside the repository, found from this script's own place + repo_dir = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))) + target_geometry_dir = os.path.join( + repo_dir, 'component_geometry', '250115') + json_source_dir = os.path.join(repo_dir, 'mongodb_backup', '250115_0825') + json_target_dir = os.path.join(repo_dir, 'mongodb_backup', 'SAS_NEW') sas_processing( source_geometry_dir, target_geometry_dir, diff --git a/scripts/db_maintenance/import_sas_database.py b/scripts/db_maintenance/import_sas_database.py index 2b91e598..086f5182 100644 --- a/scripts/db_maintenance/import_sas_database.py +++ b/scripts/db_maintenance/import_sas_database.py @@ -9,9 +9,24 @@ 'high': 3 } -SOURCE_DIR = r'C:\Users\EFESTWIN\Documents\07_tu_darmstadt_ddu\05_Promotion\07_Source\240404_SAS_Debris_Files' # NOQA -GEOMETRY_TARGET_DIR = r'C:\Users\EFESTWIN\Documents\07_tu_darmstadt_ddu\05_Promotion\07_Source\241212_MES_Geometry_Repo' # NOQA -JSON_TARGET_DIR = r'C:\Users\EFESTWIN\Documents\07_tu_darmstadt_ddu\05_Promotion\07_Source\241212_MES_JSON_Repo' # NOQA + + +def required_dir(name): + """A folder outside the repository, named in an environment variable; + there is no default, the script stops and says which one is missing.""" + value = os.environ.get(name) + if not value: + raise SystemExit( + f'{name} is not set: set it to the folder before running ' + f'(CSC_SAS_SOURCE_DIR: the SAS debris files, one sub-folder per ' + f'object; CSC_MES_GEOMETRY_DIR and CSC_MES_JSON_DIR: where the ' + f'geometry and the JSON go).') + return value + + +SOURCE_DIR = required_dir('CSC_SAS_SOURCE_DIR') +GEOMETRY_TARGET_DIR = required_dir('CSC_MES_GEOMETRY_DIR') +JSON_TARGET_DIR = required_dir('CSC_MES_JSON_DIR') if not os.path.exists(JSON_TARGET_DIR): os.makedirs(JSON_TARGET_DIR) diff --git a/scripts/dev/run_csc_python.cmd b/scripts/dev/run_csc_python.cmd new file mode 100644 index 00000000..dd975576 --- /dev/null +++ b/scripts/dev/run_csc_python.cmd @@ -0,0 +1,60 @@ +@echo off +rem Runs the Python of the "csc" conda environment with the given arguments: +rem +rem scripts\dev\run_csc_python.cmd -u scripts/dev/serve_migrated.py --dump latest ... +rem +rem The launch configurations of .claude/launch.json use it, so that no path of +rem a user account stands in a committed file. The environment is looked for in +rem this order, the first that exists wins: +rem 1. CSC_PYTHON a python.exe you name yourself (any environment) +rem 2. CONDA_PREFIX the active environment, when it is called csc +rem 3. CONDA_EXE \envs\csc, the base found from conda +rem 4. %USERPROFILE%\miniconda3 (anaconda3, miniforge3, mambaforge) and +rem %ProgramData%\miniconda3 (anaconda3): the usual install places +rem 5. `conda run -n csc --no-capture-output python` when conda is on the PATH +rem Nothing is installed or changed; with no environment found it says so and +rem stops with exit code 1. +setlocal EnableDelayedExpansion +set "PYEXE=" + +if defined CSC_PYTHON if exist "%CSC_PYTHON%" set "PYEXE=%CSC_PYTHON%" + +if not defined PYEXE if defined CONDA_PREFIX ( + for %%I in ("%CONDA_PREFIX%") do ( + if /I "%%~nxI"=="csc" if exist "%CONDA_PREFIX%\python.exe" set "PYEXE=%CONDA_PREFIX%\python.exe" + ) +) + +if not defined PYEXE if defined CONDA_EXE ( + for %%I in ("%CONDA_EXE%") do set "CONDA_DIR=%%~dpI" + for %%I in ("!CONDA_DIR!..") do set "CONDA_BASE=%%~fI" + if exist "!CONDA_BASE!\envs\csc\python.exe" set "PYEXE=!CONDA_BASE!\envs\csc\python.exe" +) + +if not defined PYEXE ( + for %%D in (miniconda3 anaconda3 miniforge3 mambaforge) do ( + if not defined PYEXE if exist "%USERPROFILE%\%%D\envs\csc\python.exe" set "PYEXE=%USERPROFILE%\%%D\envs\csc\python.exe" + ) +) + +if not defined PYEXE if defined ProgramData ( + for %%D in (miniconda3 anaconda3) do ( + if not defined PYEXE if exist "%ProgramData%\%%D\envs\csc\python.exe" set "PYEXE=%ProgramData%\%%D\envs\csc\python.exe" + ) +) + +if defined PYEXE goto run_python + +where conda >nul 2>nul +if errorlevel 1 goto not_found +conda run -n csc --no-capture-output python %* +exit /b %ERRORLEVEL% + +:run_python +"%PYEXE%" %* +exit /b %ERRORLEVEL% + +:not_found +echo run_csc_python.cmd: no conda environment "csc" found. 1>&2 +echo Create it, or set CSC_PYTHON to the python.exe to use. 1>&2 +exit /b 1 diff --git a/scripts/dev/serve_migrated.py b/scripts/dev/serve_migrated.py index bd803190..09c89d99 100644 --- a/scripts/dev/serve_migrated.py +++ b/scripts/dev/serve_migrated.py @@ -26,6 +26,10 @@ passwords, written to ``.dev/dev-test-accounts.json`` (gitignored); nothing but the user names is printed. +``--dump latest`` takes the newest folder of ``mongodb_collections_local`` +whose name is six digits. A collection with no export file is not loaded, and +an export file of 0 bytes loads as no documents. + ``--no-migrate`` loads a dump that already is 0.6 data (a production dump from after the cutover) as it is: no migration, no gripper copy, no derive run. ``--assets DIR`` points every asset folder (meshes, previews, photos, @@ -147,7 +151,9 @@ def restore_capture_fixtures(snapshots, assets: Path, capture: Path): def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument('--dump', default='261001') + parser.add_argument('--dump', default='261001', + help='a folder of mongodb_collections_local, or ' + '"latest" for the newest one (six digits)') parser.add_argument('--port', type=int, default=8000, help='backend port (default 8000)') parser.add_argument('--mongo-port', type=int, default=27018, @@ -178,6 +184,13 @@ def main() -> int: parser.add_argument('--test-dataset', default='dbu_zirkus', help='dataset the test accounts are members of') args = parser.parse_args() + if args.dump == 'latest': + from support import latest_dump + args.dump = latest_dump(str(_REPO / 'mongodb_collections_local')) + if args.dump is None: + sys.exit('--dump latest: no yymmdd folder in ' + 'mongodb_collections_local') + print(f'latest dump: {args.dump}') from mongod import ThrowawayMongod, find_mongod, sweep_stale_dirs from pymongo import MongoClient diff --git a/scripts/import/epfl_stacking/README.md b/scripts/import/epfl_stacking/README.md index d29fbae1..76420890 100644 --- a/scripts/import/epfl_stacking/README.md +++ b/scripts/import/epfl_stacking/README.md @@ -118,8 +118,9 @@ output goes outside the repository or into `.dev/`; a path anywhere else in the ## Local rehearsal -1. Start the instance: launch configs `backend-261012` (backend on 8791, throwaway mongod on 27791, a copy of - the production assets in `.dev/assets-261012`) and `frontend-261012` (3100) in `.claude/launch.json`. +1. Start the instance: launch configs `backend-local` (backend on 8791 with the newest dump of + `mongodb_collections_local`, throwaway mongod on 27791, a copy of the production assets in + `.dev/assets-261012`) and `frontend-local` (3000) in `.claude/launch.json`. Never point `--assets` at the original backup of the assets, and never run a runner with `--sweep` on it. 2. Fetch, prepare, plan; `run.py --dry-run`; `run.py --server http://127.0.0.1:8791 --username dev-admin` (the password is in `.dev/dev-migrated-credentials-8791.txt`); run it again (it creates nothing). diff --git a/scripts/import/recreate/README.md b/scripts/import/recreate/README.md index d9e84410..3edb5848 100644 --- a/scripts/import/recreate/README.md +++ b/scripts/import/recreate/README.md @@ -122,8 +122,9 @@ Paths below are the defaults (`--data` = `.dev/open_datasets/recreate/zenodo-168 ## Local rehearsal -1. Start the instance: launch configs `backend-261012` (backend on 8791, throwaway mongod on 27791, a copy - of the production assets in `.dev/assets-261012`) and `frontend-261012` (3100) in `.claude/launch.json`. +1. Start the instance: launch configs `backend-local` (backend on 8791 with the newest dump of + `mongodb_collections_local`, throwaway mongod on 27791, a copy of the production assets in + `.dev/assets-261012`) and `frontend-local` (3000) in `.claude/launch.json`. Never point `--assets` at the original backup of the assets, and never run a runner with `--sweep` on it. 2. Fetch, parse, compute geometry, plan, check (the order above). 3. `run.py --dry-run`, then `run.py --server http://127.0.0.1:8791 --username dev-admin` (the password of diff --git a/scripts/prototyping/hks_features.py b/scripts/prototyping/hks_features.py index cf040200..6493c4e5 100644 --- a/scripts/prototyping/hks_features.py +++ b/scripts/prototyping/hks_features.py @@ -18,6 +18,7 @@ Mass-weighted pooling reduces sensitivity to non-uniform sampling density. """ +import os from typing import Tuple import numpy as np import scipy.sparse as sp @@ -565,12 +566,17 @@ def visualize_descriptor_values(descriptors: np.ndarray, shape_indices=[0, 1]): # Process each shape with its own adaptive time grid try: descriptors = [] + # the test meshes are in the repository (component_geometry/hks_test) + hks_test_dir = os.path.join( + os.path.dirname(os.path.dirname(os.path.dirname( + os.path.abspath(__file__)))), + 'component_geometry', 'hks_test') mesh_files = [ - r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\hks_test\0aad9436-ead8-4651-81a1-8b435012d799.ply', - r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\hks_test\0aad9436-ead8-4651-81a1-8b435012d799_b.ply', - r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\hks_test\0b8c24c4-54d5-41b2-9eb0-43acdc34dfc1.ply', - r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\hks_test\0dd38d21-87ea-4c1d-a0b8-7245b45cd633.ply', - r'C:\Users\EFESTWIN\source\repos\csc\component_geometry\hks_test\1c3a2fdc-6522-43ab-900c-8f196804cd8d.ply', + os.path.join(hks_test_dir, '0aad9436-ead8-4651-81a1-8b435012d799.ply'), + os.path.join(hks_test_dir, '0aad9436-ead8-4651-81a1-8b435012d799_b.ply'), + os.path.join(hks_test_dir, '0b8c24c4-54d5-41b2-9eb0-43acdc34dfc1.ply'), + os.path.join(hks_test_dir, '0dd38d21-87ea-4c1d-a0b8-7245b45cd633.ply'), + os.path.join(hks_test_dir, '1c3a2fdc-6522-43ab-900c-8f196804cd8d.ply'), ] # NOTE: n_times * 2 = output descriptor dimension diff --git a/src/backend/apps/catalog/access_tokens.py b/src/backend/apps/catalog/access_tokens.py index 1cb52207..b8e3c8f6 100644 --- a/src/backend/apps/catalog/access_tokens.py +++ b/src/backend/apps/catalog/access_tokens.py @@ -54,6 +54,13 @@ OAUTH_ACCESS_SECONDS = 3600 OAUTH_REFRESH_DAYS = 30 OAUTH_GRANT_DAYS = 90 +# a row that is revoked, or expired past its refresh limit, is deleted this many +# days later: a TTL index on ``purge_at`` (0.6.3.1), no job +PURGE_AFTER_DAYS = 90 +# failed lookups (unknown, revoked, expired, wrong scope) per client address and +# minute, counted in the shared store; /token is called from addresses many +# people share (Anthropic's), so this is not tight +FAILED_LOOKUPS_PER_MINUTE = 60 def now() -> datetime: @@ -107,6 +114,23 @@ def status_of(doc: dict, moment: Optional[datetime] = None) -> str: return 'active' +def purge_at(doc: dict) -> datetime: + """When a row may be deleted: ``PURGE_AFTER_DAYS`` after it died. A row + dies when it is revoked, reaches its hard limit (``expires``) or, for a + connection, its refresh limit (``refresh_expires``): the earliest of them.""" + ends = [doc['expires']] + if doc.get('refresh_expires'): + ends.append(doc['refresh_expires']) + if doc.get('revoked'): + ends.append(doc['revoked']) + return parse_stamp(min(ends)) + timedelta(days=PURGE_AFTER_DAYS) + + +def purge_by_revoke(moment: Optional[datetime] = None) -> datetime: + """The purge date of a row revoked now (``$min`` with the one it has).""" + return (moment or now()) + timedelta(days=PURGE_AFTER_DAYS) + + def lifetime_days(doc: dict) -> int: """The lifetime a token was made with, as one of the offered choices; the default when it is none of them.""" diff --git a/src/backend/apps/catalog/api/access_tokens.py b/src/backend/apps/catalog/api/access_tokens.py index 7c2467d1..b025a853 100644 --- a/src/backend/apps/catalog/api/access_tokens.py +++ b/src/backend/apps/catalog/api/access_tokens.py @@ -24,7 +24,9 @@ same text, so the answer does not tell which check failed (the log does). A JWT is refused here, and ``auth.lookup_user_from_token`` refuses ``csc_`` tokens. 120 requests a minute per token are allowed, counted per worker in -memory (8.152) after the token has proved itself. +memory (8.152) after the token has proved itself. A token that fails counts +for the client address in the shared store (60 a minute): over that the answer +is 429 with Retry-After instead of 401; a valid token is never affected. """ import uuid @@ -41,8 +43,10 @@ TokenRefused, lookup_access_token, may_hold_pat, + note_failure, note_use, ) +from limiter import client_ip from logging_setup import get_logger from .auth import get_current_active_user, require_admin @@ -152,6 +156,7 @@ async def _create(request: Request, user: UserInDB, name: str, days: int, token_sha256=tk.token_hash(token), short_id=tk.short_id_of(token), created=tk.stamp(moment), expires=tk.expires_at(days, moment), ).model_dump(by_alias=True) + doc['purge_at'] = tk.purge_at(doc) await _tokens(request).insert_one(doc) if await _count_active(request, user.id, except_id) \ > tk.MAX_ACTIVE_TOKENS: @@ -196,7 +201,8 @@ async def _revoke(request: Request, doc: dict) -> dict: revoked = tk.stamp(tk.now()) await _tokens(request).update_one( {'_id': doc['_id'], 'revoked': None}, - {'$set': {'revoked': revoked}}) + {'$set': {'revoked': revoked}, + '$min': {'purge_at': tk.purge_by_revoke()}}) doc = {**doc, 'revoked': revoked} return doc @@ -347,17 +353,23 @@ def access_token_user(scope: str, www_authenticate: str = 'Bearer'): ``Bearer resource_metadata="..."``).""" async def dependency(request: Request) -> UserInDB: + token = _bearer(request) try: - doc, user = await lookup_access_token( - request.app, _bearer(request), scope) - route = getattr(request.scope.get('route'), 'path', '-') - await note_use(request.app, doc, route) - except TokenRefused as refusal: - raise _refuse(refusal.reason, refusal.short_id, www_authenticate) + try: + doc, user = await lookup_access_token( + request.app, token, scope) + route = getattr(request.scope.get('route'), 'path', '-') + await note_use(request.app, doc, route) + except TokenRefused as refusal: + # a token was sent and failed: it counts for the address + # (429 once over the limit), a missing header does not + if token: + await note_failure(client_ip(request)) + raise _refuse(refusal.reason, refusal.short_id, + www_authenticate) except RateLimited as limited: raise HTTPException( - status.HTTP_429_TOO_MANY_REQUESTS, - 'Too many requests for this access token.', + status.HTTP_429_TOO_MANY_REQUESTS, limited.message, headers={'Retry-After': str(limited.wait)}) request.state.access_token = doc return user diff --git a/src/backend/apps/catalog/api/auth.py b/src/backend/apps/catalog/api/auth.py index bb8f4ff0..85961187 100644 --- a/src/backend/apps/catalog/api/auth.py +++ b/src/backend/apps/catalog/api/auth.py @@ -124,7 +124,8 @@ async def revoke_all_access_tokens(app, user_id: str) -> int: row holds. Returns how many were revoked.""" done = await app.mongodb_access_tokens.update_many( {'owner_user_id': user_id, 'revoked': None}, - {'$set': {'revoked': access_tokens.stamp(access_tokens.now())}}) + {'$set': {'revoked': access_tokens.stamp(access_tokens.now())}, + '$min': {'purge_at': access_tokens.purge_by_revoke()}}) if done.modified_count: log.info('access tokens revoked after a password change: %d', done.modified_count) diff --git a/src/backend/apps/catalog/api/snapshot_lifecycle.py b/src/backend/apps/catalog/api/snapshot_lifecycle.py index 1ddbf99d..1337925b 100644 --- a/src/backend/apps/catalog/api/snapshot_lifecycle.py +++ b/src/backend/apps/catalog/api/snapshot_lifecycle.py @@ -30,6 +30,7 @@ from pymongo import ReturnDocument # LOCAL IMPORTS --------------------------------------------------------------- +from apps.catalog.file_markers import reconcile_markers, remove_files from apps.catalog.documents import ( Capture, ComponentSnapshot, @@ -206,6 +207,15 @@ def _validated(doc: dict) -> dict: return doc +def _plain_geometry(geometry: Any) -> Any: + """A geometry as the model stores it, so that one the client sent and one + stored at creation compare equal (the PATCH stores the body as it came).""" + try: + return Geometry.model_validate(geometry).model_dump(mode='json') + except ValidationError: + return geometry + + def _check_client_geometry(body: SnapshotDraftBody) -> None: """Fitted proxies are derived (3.2.1): a client sends authored ones.""" for proxy in body.geometry.proxies: @@ -742,9 +752,19 @@ async def patch_snapshot( derivation = dict(updated.get('derivation') or {}) derivation.pop(key, None) updated['derivation'] = derivation + stale_files = None if 'geometry' in body: _check_client_geometry(SnapshotDraftBody.model_validate( {'geometry': body['geometry']})) + # the file markers follow the new geometry: a mesh or cloud that was + # replaced or removed no longer has its file (0.6.3.1) + stale_files = reconcile_markers( + _plain_geometry(snapshot.get('geometry')), + _plain_geometry(body['geometry']), + snapshot.get('mesh_ply_resolutions'), + snapshot.get('point_cloud_files')) + updated['mesh_ply_resolutions'] = stale_files.resolutions + updated['point_cloud_files'] = stale_files.cloud_files updated['lastmodified'] = now_iso() if set(body) & STAGE_INPUT_FIELDS: # the previews read the colour, the stages the geometry and the @@ -765,6 +785,11 @@ async def patch_snapshot( await asyncio.to_thread( remove_map_files, request.app.snapshot_proxies_dir, fitted_map_files(snapshot)) + if stale_files is not None and stale_files.changed: + await asyncio.to_thread( + remove_files, request.app.snapshot_meshes_dir, + request.app.snapshot_point_clouds_dir, snapshot['_id'], + stale_files) if 'geometry' in body or 'shape_class' in body: # frame and class follow at once (spec 4.3, stage 1: every draft # geometry write; 8.55: a cleared override is recomputed) diff --git a/src/backend/apps/catalog/documents.py b/src/backend/apps/catalog/documents.py index ddc3874f..cd046275 100644 --- a/src/backend/apps/catalog/documents.py +++ b/src/backend/apps/catalog/documents.py @@ -1088,6 +1088,9 @@ class AccessToken(_Document): refresh_expires: Optional[Timestamp] = None client_id: Optional[str] = None redirect_host: Optional[str] = None + # TTL field (0.6.3.1): a date 90 days after the row died, see + # ``access_tokens.purge_at`` + purge_at: Optional[datetime] = None class ChangeLogEntry(_Document): diff --git a/src/backend/apps/catalog/file_markers.py b/src/backend/apps/catalog/file_markers.py new file mode 100644 index 00000000..9374c5ec --- /dev/null +++ b/src/backend/apps/catalog/file_markers.py @@ -0,0 +1,109 @@ +""" +File markers that follow the geometry (0.6.3.1). + +A snapshot lists the files it has stored: ``mesh_ply_resolutions`` (per mesh +index, the levels that have a PLY) and ``point_cloud_files`` (the indices of +the clouds that have a PLY). A file belongs to the primitive at its index. A +``PATCH`` that replaces ``geometry`` on a draft can change or remove the +primitive at an index: the file of that index then describes something that is +no longer there, and the marker would point past the new primitives or at the +wrong shape. ``reconcile_markers`` keeps the markers of the primitives that are +still the same and drops the others; ``remove_files`` deletes their files. +""" + +import os +import shutil +from typing import Any, Dict, List, Optional, Tuple + +from logging_setup import get_logger + +log = get_logger(__name__) + + +class Reconciled: + """The markers after a geometry change and what went.""" + + def __init__(self, resolutions: Dict[str, List[str]], + cloud_files: List[int], + dropped_meshes: List[Tuple[int, str]], + dropped_clouds: List[int]): + self.resolutions = resolutions + self.cloud_files = cloud_files + self.dropped_meshes = dropped_meshes + self.dropped_clouds = dropped_clouds + + @property + def changed(self) -> bool: + return bool(self.dropped_meshes or self.dropped_clouds) + + +def _same(old: List[Any], new: List[Any], index: int) -> bool: + """The primitive at ``index`` is still there and unchanged.""" + return (0 <= index < len(old) and index < len(new) + and old[index] == new[index]) + + +def _list(geometry: Optional[dict], key: str) -> List[Any]: + value = (geometry or {}).get(key) + return value if isinstance(value, list) else [] + + +def reconcile_markers(old_geometry: Optional[dict], + new_geometry: Optional[dict], + resolutions: Optional[Dict[str, List[str]]], + cloud_files: Optional[List[int]]) -> Reconciled: + """Keep a marker only where its primitive is unchanged: same index, equal + content. Everything else (replaced, moved, gone, or already beyond the old + geometry) is dropped.""" + old_meshes = _list(old_geometry, 'meshes') + new_meshes = _list(new_geometry, 'meshes') + old_clouds = _list(old_geometry, 'point_clouds') + new_clouds = _list(new_geometry, 'point_clouds') + kept_resolutions: Dict[str, List[str]] = {} + dropped_meshes: List[Tuple[int, str]] = [] + for key, levels in (resolutions or {}).items(): + try: + index = int(key) + except (TypeError, ValueError): + index = -1 + if _same(old_meshes, new_meshes, index): + kept_resolutions[key] = list(levels) + else: + dropped_meshes.extend((index, level) for level in levels) + kept_clouds = [index for index in (cloud_files or []) + if _same(old_clouds, new_clouds, index)] + dropped_clouds = [index for index in (cloud_files or []) + if index not in kept_clouds] + return Reconciled(kept_resolutions, kept_clouds, dropped_meshes, + dropped_clouds) + + +def remove_files(meshes_dir: str, clouds_dir: str, snapshot_id: str, + reconciled: Reconciled) -> int: + """Delete the files of the dropped markers; a file that is not there is + fine. A mesh folder left empty goes too. Returns how many files went.""" + removed = 0 + for index, level in reconciled.dropped_meshes: + path = os.path.join(meshes_dir, snapshot_id, str(index), f'{level}.ply') + removed += _remove(path) + folder = os.path.dirname(path) + try: + if os.path.isdir(folder) and not os.listdir(folder): + shutil.rmtree(folder) + except OSError: + log.warning('mesh folder not removed', exc_info=True) + for index in reconciled.dropped_clouds: + removed += _remove(os.path.join( + clouds_dir, snapshot_id, f'{index}.ply')) + return removed + + +def _remove(path: str) -> int: + try: + os.remove(path) + return 1 + except FileNotFoundError: + return 0 + except OSError: + log.warning('stale file not removed', exc_info=True) + return 0 diff --git a/src/backend/apps/catalog/geometry_cache.py b/src/backend/apps/catalog/geometry_cache.py index a911286e..51872da5 100644 --- a/src/backend/apps/catalog/geometry_cache.py +++ b/src/backend/apps/catalog/geometry_cache.py @@ -28,7 +28,7 @@ # LOCAL IMPORTS --------------------------------------------------------------- from apps.catalog.geometry_source import inline_fingerprint -from apps.catalog.geometry_stages import STAGES, VERSIONS, Outcome +from apps.catalog.geometry_stages import HULL_SCORES, STAGES, VERSIONS, Outcome # the derived fields each stage writes (the worker's upload names them) FIELDS_OF_STAGE = { @@ -183,12 +183,16 @@ def _outcome(self, entry: Mapping[str, Any], stages: List[str] folder = self._folder(entry['key']['snapshot_id']) out = Outcome() out.ran = list(stages) - # a cut to ``frame`` alone would carry the whole ``descriptors`` dict - # of the entry (the field is shared with the descriptors stage); the - # day's run asks for the heavy stages, whose cut is exact, so the - # shared field is only a hazard for a frame-only cut keys = {k for s in stages for k in FIELDS_OF_STAGE.get(s, ())} out.set = {k: v for k, v in entry['set'].items() if k in keys} + # ``descriptors`` is one field of two stages: the frame stage writes + # the hull scores into it, the descriptors stage the rest. A cut that + # has the frame stage and not the descriptors stage carries the hull + # scores only, not the keys the descriptors stage wrote (0.6.3.1) + if 'descriptors' in out.set and 'descriptors' not in stages: + out.set['descriptors'] = { + k: v for k, v in (out.set['descriptors'] or {}).items() + if k in HULL_SCORES} if 'proxies' in stages: for n, name in enumerate(entry['files']): with open(os.path.join(folder, 'files', str(n)), 'rb') as h: diff --git a/src/backend/apps/catalog/mcp_server.py b/src/backend/apps/catalog/mcp_server.py index 289eb5ab..ba8ae36e 100644 --- a/src/backend/apps/catalog/mcp_server.py +++ b/src/backend/apps/catalog/mcp_server.py @@ -16,6 +16,7 @@ ASGI app that ``/mcp`` serves (token check, 401 with the resource metadata, 429). """ +import json import math import re from typing import Annotated, Any, Optional @@ -34,12 +35,14 @@ from mcp.server.transport_security import TransportSecuritySettings from pydantic import AnyHttpUrl, Field from starlette.middleware.authentication import AuthenticationMiddleware +from starlette.requests import Request +import limiter as lim from apps.catalog import access_tokens as tk from apps.catalog.internal_call import InternalError, rest_get from apps.catalog.models import UserInDB from apps.catalog.oauth_server import AccessTokenVerifier -from apps.catalog.token_check import RateLimited +from apps.catalog.token_check import REQUEST_ADDRESS, RateLimited from logging_setup import get_logger log = get_logger(__name__) @@ -474,26 +477,34 @@ async def similar_components( # THE GATE OF /mcp ------------------------------------------------------------------------ class RateGate: - """Outside the SDK's authentication: a valid token over its limit gets a - 429 with Retry-After (the SDK middleware only knows 401 and 403).""" + """Outside the SDK's authentication: a valid token over its limit, or an + address over its limit of failed lookups, gets a 429 with Retry-After (the + SDK middleware only knows 401 and 403). It also tells the token verifier + which address the request comes from.""" def __init__(self, inner): self.inner = inner async def __call__(self, scope, receive, send): + marker = None + if scope['type'] == 'http': + marker = REQUEST_ADDRESS.set(lim.client_ip(Request(scope))) try: await self.inner(scope, receive, send) except RateLimited as limited_: if scope['type'] != 'http': raise - body = b'{"error":"rate_limited","error_description":' \ - b'"Too many requests for this access token."}' + body = json.dumps({'error': 'rate_limited', + 'error_description': limited_.message}).encode() await send({'type': 'http.response.start', 'status': 429, 'headers': [ (b'content-type', b'application/json'), (b'content-length', str(len(body)).encode()), (b'retry-after', str(limited_.wait).encode())]}) await send({'type': 'http.response.body', 'body': body}) + finally: + if marker is not None: + REQUEST_ADDRESS.reset(marker) def build_gate(app, mcp: FastMCP, metadata_url: str): diff --git a/src/backend/apps/catalog/oauth_server.py b/src/backend/apps/catalog/oauth_server.py index a545d20b..4e521555 100644 --- a/src/backend/apps/catalog/oauth_server.py +++ b/src/backend/apps/catalog/oauth_server.py @@ -58,9 +58,11 @@ import read_only from apps.catalog import access_tokens as tk from apps.catalog.token_check import ( + REQUEST_ADDRESS, TokenRefused, lookup_access_token, may_hold_pat, + note_failure, note_use, ) from logging_setup import get_logger @@ -365,6 +367,7 @@ async def exchange_authorization_code(self, client, authorization_code 'refresh_expires': tk.stamp( min(now + timedelta(days=tk.OAUTH_REFRESH_DAYS), hard)), 'client_id': client.client_id, 'redirect_host': host} + row['purge_at'] = tk.purge_at(row) if gh: # the Grasshopper client: the access token is the normal JWT of an # hour (every route keeps working), the row keeps only the refresh @@ -457,6 +460,10 @@ async def exchange_refresh_token(self, client, refresh_token, scopes 'refresh_expires': tk.stamp( min(now + timedelta(days=tk.OAUTH_REFRESH_DAYS), hard)), 'last_used': stamp_now} + # the refresh limit moved: so does the date the row may go + changes['purge_at'] = tk.purge_at( + {'expires': tk.stamp(hard), + 'refresh_expires': changes['refresh_expires']}) if gh: access, seconds = self._gh_access_token(owner, now, hard) else: @@ -529,7 +536,8 @@ async def _revoke_grant(self, grant_id: str, why: str) -> None: done = await self.tokens.update_one( {'_id': grant_id, 'kind': {'$in': ['oauth', 'device']}, 'revoked': None}, - {'$set': {'revoked': tk.stamp(tk.now())}}) + {'$set': {'revoked': tk.stamp(tk.now())}, + '$min': {'purge_at': tk.purge_by_revoke()}}) if done.modified_count: log.warning('oauth connection ended: %s (%s)', grant_id[:8], why) @@ -550,6 +558,9 @@ async def verify_token(self, token: str) -> Optional[SdkAccessToken]: except TokenRefused as refusal: log.info('access token refused: %s (%s)', refusal.reason, refusal.short_id) + # counted for the client address; over the limit this raises + # RateLimited, which the gate around /mcp answers with a 429 + await note_failure(REQUEST_ADDRESS.get()) return None await note_use(self.app, doc, '/mcp') ends = doc['expires'] diff --git a/src/backend/apps/catalog/proxies/specs.py b/src/backend/apps/catalog/proxies/specs.py index 0e0fad4b..77f7b09d 100644 --- a/src/backend/apps/catalog/proxies/specs.py +++ b/src/backend/apps/catalog/proxies/specs.py @@ -63,8 +63,17 @@ def _length_placement(ctx: FitContext) -> Dict[str, list]: return frame_dict(ctx.frame['o'], axes[list(order)]) +def outline_vertices(polygon: Polygon) -> int: + """The vertices of the outline, the holes counted too: the limit is on + what is stored (``profile`` and ``holes``), not on the exterior ring + alone (0.6.3.1).""" + return len(polygon.exterior.coords) + sum( + len(ring.coords) for ring in polygon.interiors) + + def _simplify(polygon: Polygon, extent: float) -> Optional[Polygon]: - """The polygon with at most ``OUTLINE_MAX_VERTICES`` vertices, or None. + """The polygon with at most ``OUTLINE_MAX_VERTICES`` vertices (exterior + and holes together), or None. The tolerance grows by half each step from ``SIMPLIFY_START`` up to ``SIMPLIFY_CAP`` of the extent. A topology-preserving simplification can @@ -78,7 +87,7 @@ def _simplify(polygon: Polygon, extent: float) -> Optional[Polygon]: simple = polygon.simplify(tolerance, preserve_topology=True) if not simple.is_valid or simple.is_empty: simple = polygon - if len(simple.exterior.coords) <= OUTLINE_MAX_VERTICES: + if outline_vertices(simple) <= OUTLINE_MAX_VERTICES: return simple if tolerance >= cap: return None diff --git a/src/backend/apps/catalog/token_check.py b/src/backend/apps/catalog/token_check.py index 16976f4d..1a155dc3 100644 --- a/src/backend/apps/catalog/token_check.py +++ b/src/backend/apps/catalog/token_check.py @@ -15,16 +15,26 @@ ``note_use`` is the rest of a successful check: the rate limit (120 a minute per token, counted per worker in memory), ``last_used`` (at most once a minute, never while the read-only flag is set) and the log line of the use. + +``note_failure`` counts a failed lookup (a token was sent and did not pass) +for the client address, in the shared store the sign-in limit uses, so that +every worker counts the same failures: over ``FAILED_LOOKUPS_PER_MINUTE`` the +answer is 429 with Retry-After instead of 401. A token that passes never +touches that counter, so a valid token is not affected by what else comes from +its address. """ +import contextvars import time from datetime import timedelta from typing import Optional, Tuple +import anyio from limits import parse from limits.storage import MemoryStorage from limits.strategies import FixedWindowRateLimiter +import limiter as lim import read_only from apps.catalog import access_tokens as tk from apps.catalog.models import UserInDB @@ -43,13 +53,29 @@ def __init__(self, reason: str, short_id: str = '-'): self.short_id = short_id +TOKEN_LIMIT_TEXT = 'Too many requests for this access token.' +ADDRESS_LIMIT_TEXT = 'Too many failed attempts from this address; wait a minute.' + +# the client address of the request being checked, for the verifier of the MCP +# layer (the SDK hands it the token only); set by the gate around /mcp +REQUEST_ADDRESS: contextvars.ContextVar = contextvars.ContextVar( + 'csc_request_address', default=None) + + class RateLimited(Exception): - """A valid token over its limit; ``wait`` seconds to the next window.""" + """Over a limit; ``wait`` seconds to the next window. By default a valid + token over its own limit; ``by_address`` when the address made too many + failed lookups.""" - def __init__(self, wait: int, short_id: str): + def __init__(self, wait: int, short_id: str, by_address: bool = False): super().__init__('rate limited') self.wait = wait self.short_id = short_id + self.by_address = by_address + + @property + def message(self) -> str: + return ADDRESS_LIMIT_TEXT if self.by_address else TOKEN_LIMIT_TEXT def may_hold_pat(role: str) -> bool: @@ -103,6 +129,31 @@ def _within_rate(short_id: str) -> Optional[int]: return max(1, int(reset - time.time()) + 1) +_FAILED = parse(f'{tk.FAILED_LOOKUPS_PER_MINUTE}/minute') + + +def _failure_wait(address: str) -> Optional[int]: + """One failed lookup for ``address``: None while it is within the limit, + else the seconds to wait. The shared store is synchronous (a thread).""" + counter = lim.shared_limiter._limiter + if counter.hit(_FAILED, 'token-failures', address): + return None + reset = counter.get_window_stats( + _FAILED, 'token-failures', address).reset_time + return max(1, int(reset - time.time()) + 1) + + +async def note_failure(address: Optional[str]) -> None: + """Count a failed lookup for ``address``; ``RateLimited`` when over. No + address (a call outside a request) counts nothing.""" + if not address: + return + wait = await anyio.to_thread.run_sync(_failure_wait, address) + if wait is not None: + log.info('failed access-token lookups over the limit for an address') + raise RateLimited(wait, '-', by_address=True) + + async def touch(app, doc: dict) -> None: """``last_used``, at most once a minute per token and never while the read-only flag is set. The condition is in the update, so two workers do diff --git a/src/backend/apps/catalog/token_cleanup.py b/src/backend/apps/catalog/token_cleanup.py new file mode 100644 index 00000000..279387d9 --- /dev/null +++ b/src/backend/apps/catalog/token_cleanup.py @@ -0,0 +1,60 @@ +""" +Clean-up of access tokens and OAuth rows (0.6.3.1). + +* ``remove_tokens_of``: the account is gone, so its access tokens and its + OAuth rows go with it (a Claude connection and a Grasshopper sign-in are rows + of ``access_tokens``; the rest are the pending authorization codes and + requests of the account and the used refresh tokens of its connections). + Called wherever an account is deleted (``usermaintenance``). +* ``backfill_purge_dates``: the TTL index on ``purge_at`` deletes a row 90 + days after it was revoked or expired; rows made by an earlier build have no + such date yet and get it once, when the API starts. +""" + +from typing import Dict, Iterable + +import read_only +from apps.catalog import access_tokens as tk +from logging_setup import get_logger + +log = get_logger(__name__) + + +async def remove_tokens_of(db, user_ids: Iterable[str]) -> Dict[str, int]: + """Delete the access tokens and OAuth rows of the accounts ``user_ids``; + ``db`` is the database handle. Returns how many rows went per collection.""" + ids = list(user_ids) + if not ids: + return {} + grants = [row['_id'] async for row in db['access_tokens'].find( + {'owner_user_id': {'$in': ids}, 'kind': {'$in': ['oauth', 'device']}}, + {'_id': 1})] + counts = { + 'oauth_used_refresh': (await db['oauth_used_refresh'].delete_many( + {'grant_id': {'$in': grants}})).deleted_count if grants else 0, + 'oauth_codes': (await db['oauth_codes'].delete_many( + {'user_id': {'$in': ids}})).deleted_count, + 'oauth_requests': (await db['oauth_requests'].delete_many( + {'bound_user': {'$in': ids}})).deleted_count, + 'access_tokens': (await db['access_tokens'].delete_many( + {'owner_user_id': {'$in': ids}})).deleted_count, + } + log.info('tokens of deleted accounts removed: %s', counts) + return counts + + +async def backfill_purge_dates(tokens) -> int: + """Give every row without a ``purge_at`` its date; returns how many. The + condition is in the update, so two workers starting together do no harm. + Nothing is written while the read-only flag is set.""" + if read_only.is_on(): + return 0 + done = 0 + async for doc in tokens.find({'purge_at': {'$exists': False}}): + result = await tokens.update_one( + {'_id': doc['_id'], 'purge_at': {'$exists': False}}, + {'$set': {'purge_at': tk.purge_at(doc)}}) + done += result.modified_count + if done: + log.info('purge dates added to access token rows: %d', done) + return done diff --git a/src/backend/main_fastapi.py b/src/backend/main_fastapi.py index 62fd901e..453b2173 100644 --- a/src/backend/main_fastapi.py +++ b/src/backend/main_fastapi.py @@ -36,6 +36,7 @@ ) from apps.catalog.api.access import TombstoneHit from apps.catalog import oauth_wiring +from apps.catalog.token_cleanup import backfill_purge_dates from csc_version import CSC_VERSION from logging_setup import configure as configure_logging, get_logger @@ -139,6 +140,12 @@ async def lifespan(app: FastAPI): await app.mongodb_access_tokens.create_index('owner_user_id') await app.mongodb_access_tokens.create_index('expires') await app.mongodb_access_tokens.create_index('refresh_sha256', sparse=True) + # a revoked row, or one expired past its refresh limit, goes 90 days later + # (0.6.3.1): a TTL on the date written when the row dies; rows of an earlier + # build get theirs here, once + await app.mongodb_access_tokens.create_index( + 'purge_at', expireAfterSeconds=0) + await backfill_purge_dates(app.mongodb_access_tokens) # a registered client that never completes a consent goes after 1 day, one # that did, 30 days after its last use (one TTL index, expires_at moves); # the pending requests, the codes and the used refresh tokens go by their diff --git a/src/backend/usermaintenance.py b/src/backend/usermaintenance.py index c70dcf41..4a6674f7 100644 --- a/src/backend/usermaintenance.py +++ b/src/backend/usermaintenance.py @@ -23,6 +23,7 @@ # LOCAL MODULE IMPORTS -------------------------------------------------------- from logging_setup import configure as configure_logging, get_logger import read_only +from apps.catalog.token_cleanup import backfill_purge_dates, remove_tokens_of from settings import get_settings from utility import ( get_database_name, @@ -83,6 +84,13 @@ async def cleanup_unverified_users(dry_run=False): db = client[get_database_name()] users = db['users'] + # rows of access_tokens made before the purge dates existed get theirs + # here: the API start does it too, but deploys start it under the + # read-only flag, which skips it; this job skips under the flag as well + # (and the next run after the flag finishes it) + if not dry_run: + await backfill_purge_dates(db['access_tokens']) + # Calculate cutoff date (7 days ago) cutoff_date = datetime.now(timezone.utc) - timedelta(days=7) log.info('Cutoff date: %s', cutoff_date.isoformat()) @@ -103,6 +111,9 @@ async def cleanup_unverified_users(dry_run=False): if dry_run: log.info('DRY RUN: no users were actually deleted') else: + # their access tokens and OAuth rows first: if this stops here the + # accounts are still found by the next run, which finishes it + await remove_tokens_of(db, ids) # Delete the users result = await users.delete_many(query) log.info('Deleted %d user(s)', result.deleted_count) diff --git a/src/frontend/app/admin/users/page.tsx b/src/frontend/app/admin/users/page.tsx index 4993e864..83f2750b 100644 --- a/src/frontend/app/admin/users/page.tsx +++ b/src/frontend/app/admin/users/page.tsx @@ -280,12 +280,12 @@ export default function AdminUsersPage() { {user.invited && invited} -
{user.full_name ?? '—'}
-
{user.email ?? '—'}
+
{user.full_name ?? '-'}
+
{user.email ?? '-'}
{(user.memberships ?? []).length === 0 ? ( - — + - ) : (
{(user.memberships ?? []).map((m) => ( diff --git a/src/frontend/components/admin/InvitationsTable.tsx b/src/frontend/components/admin/InvitationsTable.tsx index 77461ead..f2f52110 100644 --- a/src/frontend/components/admin/InvitationsTable.tsx +++ b/src/frontend/components/admin/InvitationsTable.tsx @@ -82,8 +82,8 @@ export default function InvitationsTable({ {invitations.map((invitation) => ( {invitation.email} - {showDataset && {invitation.dataset ?? '—'}} - {invitation.roles.join(', ') || '—'} + {showDataset && {invitation.dataset ?? '-'}} + {invitation.roles.join(', ') || '-'} {invitation.state} {invitation.mail_failed && invitation.state !== 'used' && invitation.state !== 'revoked' && ( diff --git a/src/frontend/components/components/ComponentViewer.tsx b/src/frontend/components/components/ComponentViewer.tsx index 086a5592..cf706284 100644 --- a/src/frontend/components/components/ComponentViewer.tsx +++ b/src/frontend/components/components/ComponentViewer.tsx @@ -38,7 +38,7 @@ import { barLegend, defaultColourMode, diameterRange, - layoutLabels, + layoutsOnSnapshot, type BarColourMode, type BarLayout, } from '@/lib/reinforcementColors' @@ -62,11 +62,19 @@ import { applyOpacity, visibleBodyLayers, } from '@/lib/viewerOpacity' +import { + bodyFaceMaterial, + depthChanged, + depthRange, + edgeLineMaterial, + sizeChanged, +} from '@/lib/viewerDepth' import { chooseSwitch, edgesControl, hasMeshSwitch, pointCloudStartsOff, + shouldLoadLevel, shownFlags, switchValue, withOff, @@ -95,11 +103,31 @@ function ToolbarSwitch({ label, children }: { label: string; children: React.Rea ) } -/** Fit camera to scene bounds; registers ref for manual zoom-extents. */ +/** Frames between two measurements of the piece (it changes with a load, not with a frame). */ +const MEASURE_EVERY_FRAMES = 15 + +/** + * Fit camera to scene bounds; registers ref for manual zoom-extents. Also + * owns two things that depend on the size of the piece (0.6.3.1), measured + * together with one reused box, at the first frame that has content and then + * every `MEASURE_EVERY_FRAMES` frames: + * - the first fit can come before the piece is in the scene (it is attached a + * moment later, and drei measures a made-up box of the camera's distance + * for an empty scene), so a small piece was left as a dot. Until the user + * moves the camera, the view is fitted again whenever the size of the piece + * changes (`sizeChanged`); + * - the camera planes follow the camera: drei's `clip` sets them once, for the + * distance of the fit, so zooming close to a face cut it away. Each frame + * they are a function of the distance to the orbit target and the radius of + * the piece (`depthRange`; no measuring in the frame, only that distance). + */ function FitCameraController({ fitRef, + contentRef, }: { fitRef: React.MutableRefObject<(() => void) | null> + /** The group of the piece: its box is the size of the piece. */ + contentRef: React.RefObject }) { const bounds = useBounds() const controls = useThree((state) => state.controls) @@ -122,6 +150,48 @@ function FitCameraController({ hasFit.current = true }) + const box = useMemo(() => new THREE.Box3(), []) + const extent = useMemo(() => new THREE.Vector3(), []) + const radius = useRef(0) + const fittedSize = useRef(0) + const touched = useRef(false) + const frames = useRef(0) + useEffect(() => { + const events = controls as unknown as { + addEventListener: (type: string, listener: () => void) => void + removeEventListener: (type: string, listener: () => void) => void + } | null + if (!events) return + const touch = () => { touched.current = true } + events.addEventListener('start', touch) + return () => events.removeEventListener('start', touch) + }, [controls]) + useFrame((state) => { + const content = contentRef.current + if (hasFit.current && content + && (radius.current === 0 || frames.current % MEASURE_EVERY_FRAMES === 0)) { + box.setFromObject(content) + if (!box.isEmpty()) { + const diagonal = box.getSize(extent).length() + if (Number.isFinite(diagonal) && diagonal > 0) radius.current = diagonal / 2 + if (!touched.current && sizeChanged(fittedSize.current, diagonal)) { + fittedSize.current = diagonal + fitToScene() + } + } + } + frames.current += 1 + const target = (state.controls as { target?: THREE.Vector3 } | null)?.target + if (!target || radius.current === 0) return + const { camera } = state + const wanted = depthRange(camera.position.distanceTo(target), radius.current) + if (depthChanged({ near: camera.near, far: camera.far }, wanted)) { + camera.near = wanted.near + camera.far = wanted.far + camera.updateProjectionMatrix() + } + }) + return null } @@ -387,19 +457,8 @@ function buildThreeGroupFromPLYGeometry( const hasColors = !!geometry.getAttribute('color') - const material = hasColors - ? new THREE.MeshBasicMaterial({ - vertexColors: true, - side: THREE.DoubleSide, - transparent: false, - opacity: 1.0, - }) - : new THREE.MeshBasicMaterial({ - color: 0x888888, - side: THREE.DoubleSide, - transparent: false, - opacity: 1.0, - }) + // the faces sit a hair behind their edges (0.6.3.1) + const material = bodyFaceMaterial(hasColors) const mesh = new THREE.Mesh(geometry, material) mesh.name = meshLabel @@ -408,7 +467,7 @@ function buildThreeGroupFromPLYGeometry( object.add(mesh) { const edgeGeometry = new THREE.EdgesGeometry(geometry) - const edgeMaterial = new THREE.LineBasicMaterial({ color: 0x000000 }) + const edgeMaterial = edgeLineMaterial() const edges = new THREE.LineSegments(edgeGeometry, edgeMaterial) edges.name = `${meshLabel}_edges` object.add(edges) @@ -895,12 +954,10 @@ const VisualizeMultipleMeshes = React.memo(({ geom.computeVertexNormals() geom.normalizeNormals() - const material = rawColors && rawColors.length > 0 - ? new THREE.MeshBasicMaterial({ vertexColors: true, side: THREE.DoubleSide }) - : new THREE.MeshBasicMaterial({ color: 0x888888, side: THREE.DoubleSide }) + const material = bodyFaceMaterial(!!rawColors && rawColors.length > 0) const edgeGeometry = new THREE.EdgesGeometry(geom) - const edgeMaterial = new THREE.LineBasicMaterial({ color: 0x000000 }) + const edgeMaterial = edgeLineMaterial() // both are built here for this viewer alone: fade them in place applyOpacity(material, opacity) applyOpacity(edgeMaterial, opacity) @@ -1156,6 +1213,9 @@ export default function ComponentViewer({ const pointCloudVisibilitySeedRef = useRef( `${identityId?.toString() ?? ''}:${primitivePointCloudCount}`, ) + const pointCloudOffSeedRef = useRef( + `${identityId?.toString() ?? ''}:${primitivePointCloudCount}`, + ) const snapshotMeshCacheKey = useMemo(() => { if (!snapshotRouting?.snapshot_id) return '' @@ -1235,20 +1295,11 @@ export default function ComponentViewer({ }[]) => { if (cancelled) return // one layout per record (RL3); a record without drawable bars is no layout - const drawable = (bar: ReinforcementBar) => Array.isArray(bar.points) && bar.points.length >= 2 && bar.diameter_mm > 0 - const here = records - .filter((record) => record.position?.snapshot_id === snapshotIdForOverlays) - .filter((record) => (record.payload?.bars ?? []).some(drawable)) - const labels = layoutLabels(here) - const bars: ViewerBar[] = here.flatMap((record, layout) => - (record.payload?.bars ?? []).filter(drawable).map((bar) => ({ ...bar, layout }))) + // one layout per record (RL1), every bar of every layout drawn + const { bars, layouts } = layoutsOnSnapshot(records, snapshotIdForOverlays) setReinforcementBars(bars) - setBarLayouts(here.map((record, layout) => ({ - id: String(record._id ?? layout), - label: labels[layout], - count: bars.filter((bar) => bar.layout === layout).length, - }))) - setBarColourMode(defaultColourMode(here.length)) + setBarLayouts(layouts) + setBarColourMode(defaultColourMode(layouts.length)) }) .catch(() => { if (!cancelled) { setReinforcementBars([]); setBarLayouts([]) } }) return () => { cancelled = true } @@ -1264,9 +1315,20 @@ export default function ComponentViewer({ const visibilitySeed = `${identityId?.toString() ?? ''}:${primitiveMeshCount}` const resetVisibilityToDefault = meshVisibilitySeedRef.current !== visibilitySeed - meshVisibilitySeedRef.current = visibilitySeed - - if (isMeshExternalMode && !isProxyOnly && identityId) { + // an Off layer loads nothing and leaves its visibility alone: it must not + // use up the reset either, so that after a version switch while Off the + // new version's default visibility applies when the mesh is switched back + // on (0.6.3.1) + const meshHeldByOff = isMeshExternalMode && meshOff && !isProxyOnly && !!identityId + if (!meshHeldByOff) meshVisibilitySeedRef.current = visibilitySeed + + if (meshHeldByOff) { + // Off hides and loads nothing, also not after a version switch; the + // level is fetched when the mesh is switched back on (0.6.3.1) + setIsLoadingExternalMeshes(false) + } else if (isMeshExternalMode && shouldLoadLevel({ + external: isMeshExternalMode, off: meshOff, proxyOnly: isProxyOnly, hasIdentity: !!identityId, + })) { setIsLoadingExternalMeshes(true) setMeshGeometryError(null) setShowEdges(false) @@ -1323,6 +1385,7 @@ export default function ComponentViewer({ } }, [ meshGeometryMode, + meshOff, isProxyOnly, identityId, isMeshExternalMode, @@ -1337,10 +1400,22 @@ export default function ComponentViewer({ const visibilitySeed = `${identityId?.toString() ?? ''}:${primitivePointCloudCount}` const resetVisibilityToDefault = pointCloudVisibilitySeedRef.current !== visibilitySeed - pointCloudVisibilitySeedRef.current = visibilitySeed - if (resetVisibilityToDefault) setPointCloudOff(pointCloudStartsOff(primitiveMeshCount)) + // as for the mesh, an Off cloud holding a stored level does not use up the + // reset of its visibility (0.6.3.1); the Off switch itself starts again + // for another piece at once (a seed of its own) + const cloudHeldByOff = isPointCloudExternalMode && pointCloudOff && !isProxyOnly && !!identityId + if (!cloudHeldByOff) pointCloudVisibilitySeedRef.current = visibilitySeed + if (pointCloudOffSeedRef.current !== visibilitySeed) { + pointCloudOffSeedRef.current = visibilitySeed + setPointCloudOff(pointCloudStartsOff(primitiveMeshCount)) + } - if (isPointCloudExternalMode && !isProxyOnly && identityId) { + if (cloudHeldByOff) { + // as for the mesh: Off loads nothing (0.6.3.1) + setIsLoadingExternalPointClouds(false) + } else if (shouldLoadLevel({ + external: isPointCloudExternalMode, off: pointCloudOff, proxyOnly: isProxyOnly, hasIdentity: !!identityId, + })) { setIsLoadingExternalPointClouds(true) loadExternalPointClouds( identityId.toString(), @@ -1390,6 +1465,7 @@ export default function ComponentViewer({ } }, [ pointCloudGeometryMode, + pointCloudOff, isProxyOnly, identityId, isPointCloudExternalMode, @@ -1981,7 +2057,7 @@ export default function ComponentViewer({ margin={1.2} maxDuration={1} > - + r.status === 'published') const newestFirst = (a: (typeof live)[number], b: (typeof live)[number]) => b.observed_at.localeCompare(a.observed_at) const help = registry?.help(...UPLOAD_HELP) + // the cards of this page: a paired record is linked only when it is one of them + const shownIds = [...live, ...(showCorrected ? corrected : [])].map((record) => String(record._id)) const card = (record: (typeof live)[number]) => ( + uploadHelp={help} menu shownIds={shownIds} /> ) return ( @@ -68,7 +70,8 @@ export default function ComponentRecordsCard({ catalog }: { catalog: CatalogComp {showCorrected ? 'Hide' : 'Show'} corrected records ({corrected.length}) {showCorrected && [...corrected].sort(newestFirst).map((record) => ( - + ))}
)} diff --git a/src/frontend/components/evidence/EvidenceCard.tsx b/src/frontend/components/evidence/EvidenceCard.tsx index 69270f7a..3f17ca4f 100644 --- a/src/frontend/components/evidence/EvidenceCard.tsx +++ b/src/frontend/components/evidence/EvidenceCard.tsx @@ -7,7 +7,7 @@ * verification. Used in the component page's list, in the form's review * step and in the queues. */ -import { useState } from 'react' +import { useEffect, useState } from 'react' import Link from 'next/link' import { ChevronDown, ChevronRight, MapPin, TriangleAlert } from 'lucide-react' @@ -25,6 +25,7 @@ import { roundVec, type Vec3 } from '@/lib/evidence/grid' import { useRegistry } from '@/lib/evidence/useRegistry' import { formatDateAtPrecision } from '@/components/components/componentDetailShared' import { documentOf } from '@/lib/evidence/document' +import { pairedReboundLink, recordAnchor } from '@/lib/evidence/paired' function who(actor: Actor): string { return [actor.name, actor.organization].filter(Boolean).join(', ') || 'someone' @@ -37,6 +38,7 @@ export default function EvidenceCard({ defaultOpen = false, uploadHelp, menu = false, + shownIds, }: { record: EvidenceView dataset: string | null | undefined @@ -45,8 +47,18 @@ export default function EvidenceCard({ uploadHelp?: string /** The lifecycle actions in a menu of the card's own (the component page, 8.118). */ menu?: boolean + /** The ids of the cards the page lists (the paired record is linked only when it is one of them). */ + shownIds?: readonly string[] }) { const [open, setOpen] = useState(defaultOpen) + const anchor = recordAnchor(String(record._id)) + // a link to this card (the paired record's) opens it + useEffect(() => { + const opens = () => { if (window.location.hash === anchor) setOpen(true) } + opens() + window.addEventListener('hashchange', opens) + return () => window.removeEventListener('hashchange', opens) + }, [anchor]) const { registry } = useRegistry() const method = registry?.methods.find((m) => m.name === record.method) const facts = serverFacts(method, record.payload) @@ -57,6 +69,7 @@ export default function EvidenceCard({ const point = record.position?.point as Vec3 | null | undefined const attachments = (record.attachments ?? []).filter((a) => !a.removed).length const cited = documentOf(record.payload) + const paired = pairedReboundLink(record.method, record.payload, shownIds) // no result: the record documents the piece and never enters the fold (8.106) const isDocument = !record.summary @@ -112,6 +125,19 @@ export default function EvidenceCard({

Record id

+ {paired && ( +

+ Paired rebound record: + {paired.onPage ? ( + + {paired.id.slice(0, 8)} + + ) : ( + {paired.id.slice(0, 8)} + )} +

+ )} {(record.derived ?? []).map((d, i) => (

{quantityLabel(d.quantity)} ({d.model.kind}): diff --git a/src/frontend/components/viewer/ProxySolid.tsx b/src/frontend/components/viewer/ProxySolid.tsx index ce1ea6b8..af5009e5 100644 --- a/src/frontend/components/viewer/ProxySolid.tsx +++ b/src/frontend/components/viewer/ProxySolid.tsx @@ -4,6 +4,7 @@ import { useEffect, useLayoutEffect, useMemo } from 'react' import * as THREE from 'three' import { solidGeometry, type ProxyShape } from '@/lib/proxyShape' +import { FACE_POLYGON_OFFSET, edgeLineMaterial } from '@/lib/viewerDepth' import { OPACITY_DEFAULT, applyOpacity, materialSettings } from '@/lib/viewerOpacity' /** @@ -27,7 +28,7 @@ export default function ProxySolid({ opacity?: number }) { const solid = useMemo(() => solidGeometry(shape), [shape]) - const edgeMaterial = useMemo(() => new THREE.LineBasicMaterial({ color: 0x000000 }), []) + const edgeMaterial = useMemo(() => edgeLineMaterial(), []) useEffect(() => () => solid.dispose(), [solid]) useEffect(() => () => edgeMaterial.dispose(), [edgeMaterial]) useLayoutEffect(() => applyOpacity(edgeMaterial, opacity), [edgeMaterial, opacity]) @@ -46,6 +47,9 @@ export default function ProxySolid({ transparent={fade.transparent} opacity={fade.opacity} depthWrite={fade.depthWrite} + polygonOffset={FACE_POLYGON_OFFSET.polygonOffset} + polygonOffsetFactor={FACE_POLYGON_OFFSET.polygonOffsetFactor} + polygonOffsetUnits={FACE_POLYGON_OFFSET.polygonOffsetUnits} /> )} diff --git a/src/frontend/lib/evidence/paired.test.ts b/src/frontend/lib/evidence/paired.test.ts new file mode 100644 index 00000000..65cadae2 --- /dev/null +++ b/src/frontend/lib/evidence/paired.test.ts @@ -0,0 +1,68 @@ +/** + * The paired rebound record of a core record: the link on the core's card + * (0.6.3.1). + * + * Run: npm test + */ +import assert from 'node:assert/strict' +import { test } from 'node:test' + +import { + PAIRED_OFF_PAGE_TITLE, + isOnPage, + pairedReboundId, + pairedReboundLink, + recordAnchor, +} from '@/lib/evidence/paired' + +test('a core record names its paired rebound record', () => { + const payload = { sampling: { paired_rebound_id: 'abc-123' } } + assert.equal(pairedReboundId('core_compression', payload), 'abc-123') +}) + +test('only a core record has one', () => { + const payload = { sampling: { paired_rebound_id: 'abc-123' } } + assert.equal(pairedReboundId('rebound_hammer', payload), null) + assert.equal(pairedReboundId('reinforcement_layout', payload), null) +}) + +test('a core without a pairing, or a value that is no id, has none', () => { + assert.equal(pairedReboundId('core_compression', { sampling: {} }), null) + assert.equal(pairedReboundId('core_compression', { sampling: { paired_rebound_id: null } }), null) + assert.equal(pairedReboundId('core_compression', { sampling: { paired_rebound_id: 5 } }), null) + assert.equal(pairedReboundId('core_compression', { sampling: { paired_rebound_id: ' ' } }), null) + assert.equal(pairedReboundId('core_compression', {}), null) + assert.equal(pairedReboundId('core_compression', null), null) + assert.equal(pairedReboundId('core_compression', undefined), null) +}) + +test('the link points at the card of the record', () => { + assert.equal(recordAnchor('abc-123'), '#record-abc-123') +}) + +test('the link is shown for a record on the page', () => { + assert.equal(isOnPage('a', ['a', 'b']), true) + assert.equal(isOnPage('c', ['a', 'b']), false) +}) + +test('the core links to its rebound record when that record is on the page', () => { + const payload = { sampling: { paired_rebound_id: 'reb-1' } } + assert.deepEqual(pairedReboundLink('core_compression', payload, ['core-1', 'reb-1']), { + id: 'reb-1', onPage: true, href: '#record-reb-1', + }) +}) + +test('the id is plain text with a title when the record is not on the page', () => { + const payload = { sampling: { paired_rebound_id: 'reb-1' } } + const off = { id: 'reb-1', onPage: false, title: PAIRED_OFF_PAGE_TITLE } + assert.deepEqual(pairedReboundLink('core_compression', payload, ['core-1']), off) + assert.deepEqual(pairedReboundLink('core_compression', payload, []), off) + // no list at all (a queue, the review step of the form): nothing is on the page + assert.deepEqual(pairedReboundLink('core_compression', payload), off) + assert.match(PAIRED_OFF_PAGE_TITLE, /another version|filtered out/) +}) + +test('a record without a pairing has no link and no text', () => { + assert.equal(pairedReboundLink('core_compression', { sampling: {} }, ['a']), null) + assert.equal(pairedReboundLink('rebound_hammer', { sampling: { paired_rebound_id: 'x' } }, ['x']), null) +}) diff --git a/src/frontend/lib/evidence/paired.ts b/src/frontend/lib/evidence/paired.ts new file mode 100644 index 00000000..c3e2e81d --- /dev/null +++ b/src/frontend/lib/evidence/paired.ts @@ -0,0 +1,51 @@ +/** + * The rebound record a core record is paired with (EN 13791 pairs, 0.6.3.1): + * the card of the core links to it. The id is read from the payload of a + * `core_compression` record; any other record, or a value that is no text, + * has none. + * + * Run: npm test + */ + +/** The `paired_rebound_id` of a core record's payload, or null. */ +export function pairedReboundId(method: string, payload: unknown): string | null { + if (method !== 'core_compression') return null + const sampling = (payload as { sampling?: { paired_rebound_id?: unknown } } | null | undefined)?.sampling + const id = sampling?.paired_rebound_id + return typeof id === 'string' && id.trim() ? id.trim() : null +} + +/** The anchor of a record's card on the component page (`EvidenceCard` ids). */ +export function recordAnchor(recordId: string): string { + return `#record-${recordId}` +} + +/** The record is on this page: the link can point at its card. */ +export function isOnPage(recordId: string, shownIds: readonly string[]): boolean { + return shownIds.includes(recordId) +} + +export const PAIRED_OFF_PAGE_TITLE = + 'This record is on another version or filtered out of this list' + +export type PairedReboundLink = + | { id: string; onPage: true; href: string } + | { id: string; onPage: false; title: string } + +/** + * What the core's card shows for its paired rebound record: a link to the card + * when that record is on the page (`shownIds`: the cards the page lists), else + * the id as plain text with a short title saying why there is no link. Null for + * a record that has no pairing. Without `shownIds` (a list that is no + * component page: a queue, the form's review) nothing is on the page. + */ +export function pairedReboundLink( + method: string, + payload: unknown, + shownIds?: readonly string[], +): PairedReboundLink | null { + const id = pairedReboundId(method, payload) + if (!id) return null + if (shownIds && isOnPage(id, shownIds)) return { id, onPage: true, href: recordAnchor(id) } + return { id, onPage: false, title: PAIRED_OFF_PAGE_TITLE } +} diff --git a/src/frontend/lib/reinforcementColors.test.ts b/src/frontend/lib/reinforcementColors.test.ts index b7c3f0f6..740763aa 100644 --- a/src/frontend/lib/reinforcementColors.test.ts +++ b/src/frontend/lib/reinforcementColors.test.ts @@ -9,6 +9,7 @@ import { diameterRange, layoutColour, layoutLabels, + layoutsOnSnapshot, LAYOUT_COLOURS, rampColour, } from '@/lib/reinforcementColors' @@ -78,3 +79,58 @@ test('the legend lists the layouts or the diameter range', () => { assert.equal(byDiameter?.kind === 'diameter' && byDiameter.max, 16) assert.equal(barLegend('steel', layouts, bars), null) }) + + +// TWO LAYOUTS ON ONE PIECE (RL1, 0.6.3.1) ----------------------------------------- +const bar = (diameter: number, x: number) => ({ + spec: `B${diameter}`, diameter_mm: diameter, points: [[x, 0, 0], [x, 0, 100]], +}) + +test('two layouts on one snapshot are two layouts with their own bars', () => { + const records = [ + { _id: 'a', observed_at: '2026-01-05', position: { snapshot_id: 's1' }, + payload: { bars: [bar(12, 0), bar(12, 50)], document: { title: 'Drawing A' } } }, + { _id: 'b', observed_at: '2026-02-09', position: { snapshot_id: 's1' }, + payload: { bars: [bar(8, 10), bar(8, 20), bar(8, 30)], document: { title: 'Drawing B' } } }, + ] + const { bars, layouts } = layoutsOnSnapshot(records, 's1') + assert.equal(bars.length, 5) + assert.deepEqual(layouts, [ + { id: 'a', label: 'Drawing A', count: 2 }, + { id: 'b', label: 'Drawing B', count: 3 }, + ]) + assert.deepEqual(bars.map((b) => b.layout), [0, 0, 1, 1, 1]) + assert.equal(defaultColourMode(layouts.length), 'layout') + // each layout has its own colour in the legend + assert.notEqual(layoutColour(0), layoutColour(1)) + const legend = barLegend('layout', layouts, bars) + assert.equal(legend?.kind, 'layout') +}) + +test('a layout of another snapshot, or without a drawable bar, is not a layout here', () => { + const records = [ + { _id: 'a', position: { snapshot_id: 's1' }, payload: { bars: [bar(12, 0)] } }, + { _id: 'other', position: { snapshot_id: 's2' }, payload: { bars: [bar(12, 0)] } }, + { _id: 'empty', position: { snapshot_id: 's1' }, payload: { bars: [] } }, + { _id: 'flat', position: { snapshot_id: 's1' }, + payload: { bars: [{ spec: 'x', diameter_mm: 0, points: [[0, 0, 0], [1, 0, 0]] }] } }, + { _id: 'none', position: null, payload: { bars: [bar(12, 0)] } }, + ] + const { bars, layouts } = layoutsOnSnapshot(records, 's1') + assert.deepEqual(layouts, [{ id: 'a', label: 'Layout 1', count: 1 }]) + assert.equal(bars.length, 1) +}) + +test('two layouts with the same title are told apart in the legend', () => { + const records = ['a', 'b'].map((id) => ({ + _id: id, position: { snapshot_id: 's1' }, + payload: { bars: [bar(10, 0)], document: { title: 'Ground floor' } } })) + const { layouts } = layoutsOnSnapshot(records, 's1') + assert.deepEqual(layouts.map((l) => l.label), ['Ground floor (1)', 'Ground floor (2)']) +}) + +test('a piece without a layout has no bars and starts steel', () => { + const { bars, layouts } = layoutsOnSnapshot([], 's1') + assert.deepEqual([bars, layouts], [[], []]) + assert.equal(defaultColourMode(layouts.length), 'steel') +}) diff --git a/src/frontend/lib/reinforcementColors.ts b/src/frontend/lib/reinforcementColors.ts index ad6e5609..a99e2c62 100644 --- a/src/frontend/lib/reinforcementColors.ts +++ b/src/frontend/lib/reinforcementColors.ts @@ -94,6 +94,42 @@ export function layoutLabels(records: RecordLike[]): string[] { export type ColouredBar = { diameter_mm: number; layout: number } +/** A bar the viewer can draw: a centreline of two points or more and a diameter. */ +export type DrawableBar = { diameter_mm: number; points: number[][] } + +export function isDrawableBar(bar: DrawableBar): boolean { + return Array.isArray(bar.points) && bar.points.length >= 2 && bar.diameter_mm > 0 +} + +export type LayoutRecord = RecordLike & { + position?: { snapshot_id?: string | null } | null + payload?: { bars?: B[]; document?: { title?: string | null } | null } | null +} + +/** + * The layouts of one snapshot and their bars (RL1, 0.6.3.1): every record that + * is positioned on the snapshot and has a drawable bar is one layout, in the + * order given, and every bar of every layout is drawn, each with the index of + * its layout. Two layouts on one piece are two cards and two sets of bars. + */ +export function layoutsOnSnapshot( + records: LayoutRecord[], + snapshotId: string, +): { bars: (B & { layout: number })[]; layouts: BarLayout[] } { + const here = records + .filter((record) => record.position?.snapshot_id === snapshotId) + .filter((record) => (record.payload?.bars ?? []).some(isDrawableBar)) + const labels = layoutLabels(here) + const bars = here.flatMap((record, layout) => + (record.payload?.bars ?? []).filter(isDrawableBar).map((bar) => ({ ...bar, layout }))) + const layouts = here.map((record, layout) => ({ + id: String(record._id ?? layout), + label: labels[layout], + count: bars.filter((bar) => bar.layout === layout).length, + })) + return { bars, layouts } +} + /** The colour of a bar, or null for the steel material. */ export function barColour( mode: BarColourMode, diff --git a/src/frontend/lib/viewerDepth.test.ts b/src/frontend/lib/viewerDepth.test.ts new file mode 100644 index 00000000..d20626d4 --- /dev/null +++ b/src/frontend/lib/viewerDepth.test.ts @@ -0,0 +1,150 @@ +/** + * Faces behind their edges (0.6.3.1): the face materials carry a polygon + * offset, also in the faded copies of the opacity slider; the edge lines keep + * their depth test and have no offset. + * + * Run: npm test + */ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import * as THREE from 'three' + +import { + FACE_POLYGON_OFFSET, + NEAR_FLOOR, + bodyFaceMaterial, + depthChanged, + depthRange, + edgeLineMaterial, + pushFacesBack, + sizeChanged, +} from '@/lib/viewerDepth' +import { applyOpacity, applyOpacityToObject, copyForViewer } from '@/lib/viewerOpacity' + +function offsetOf(material: THREE.Material) { + return [material.polygonOffset, material.polygonOffsetFactor, material.polygonOffsetUnits] +} + +test('the offset is small and positive: faces go back, not forward', () => { + assert.equal(FACE_POLYGON_OFFSET.polygonOffset, true) + assert.ok(FACE_POLYGON_OFFSET.polygonOffsetFactor > 0 && FACE_POLYGON_OFFSET.polygonOffsetFactor <= 2) + assert.ok(FACE_POLYGON_OFFSET.polygonOffsetUnits > 0 && FACE_POLYGON_OFFSET.polygonOffsetUnits <= 2) +}) + +test('the face material of a mesh has the offset, with vertex colours or grey', () => { + for (const hasColors of [true, false]) { + const material = bodyFaceMaterial(hasColors) + assert.deepEqual(offsetOf(material), [true, 1, 1]) + assert.equal(material.side, THREE.DoubleSide) + assert.equal(material.vertexColors, hasColors) + // opaque and writing depth, as before the offset + assert.equal(material.transparent, false) + assert.equal(material.depthWrite, true) + assert.equal(material.depthTest, true) + } + assert.equal(bodyFaceMaterial(false).color.getHex(), 0x888888) +}) + +test('the edge lines have no offset and keep their depth test', () => { + const lines = edgeLineMaterial() + assert.deepEqual(offsetOf(lines), [false, 0, 0]) + assert.equal(lines.depthTest, true) + assert.equal(lines.color.getHex(), 0) +}) + +test('an existing material can be pushed back', () => { + const standard = pushFacesBack(new THREE.MeshStandardMaterial({ color: 0x336699 })) + assert.deepEqual(offsetOf(standard), [true, 1, 1]) + assert.equal(standard.color.getHex(), 0x336699) +}) + +test('the faded copy of the opacity slider keeps the offset, the original is not changed', () => { + const faces = bodyFaceMaterial(true) + const edges = edgeLineMaterial() + const group = new THREE.Group() + group.add(new THREE.Mesh(new THREE.BoxGeometry(1, 1, 1), faces)) + group.add(new THREE.LineSegments(new THREE.EdgesGeometry(new THREE.BoxGeometry(1, 1, 1)), edges)) + const copy = copyForViewer(group) + applyOpacityToObject(copy, 40) + const [copiedFaces, copiedEdges] = copy.children.map( + (child) => (child as THREE.Mesh).material as THREE.Material) + assert.deepEqual(offsetOf(copiedFaces), [true, 1, 1]) + assert.deepEqual(offsetOf(copiedEdges), [false, 0, 0]) + assert.equal(copiedFaces.opacity, 0.4) + assert.deepEqual(offsetOf(faces), [true, 1, 1]) + assert.equal(faces.opacity, 1) +}) + +test('fading and restoring a face material leaves its offset', () => { + const faces = bodyFaceMaterial(false) + applyOpacity(faces, 30) + applyOpacity(faces, 100) + assert.deepEqual(offsetOf(faces), [true, 1, 1]) +}) + + +// THE CAMERA PLANES (0.6.3.1) ----------------------------------------------------- +test('near is a hundredth of the distance and far lies past the piece', () => { + const { near, far } = depthRange(6, 2.3) // a 4.6 m beam, seen from the fit distance + assert.equal(near, 0.06) + assert.ok(far > 6 + 2.3, 'the far side of the piece and the grid are inside') + assert.ok(far / near < 5000) +}) + +test('close to the piece the near plane follows, so a face is not cut away', () => { + const radius = 2.3 + for (const distance of [1, 0.2, 0.05, 0.01]) { + const { near } = depthRange(distance, radius) + assert.ok(near <= distance / 100 + 1e-12 || near === radius * NEAR_FLOOR, String(distance)) + assert.ok(near < distance, 'a face at the target is in front of the near plane: ' + distance) + } + // a camera right at the target: still a positive, tiny near plane + assert.ok(depthRange(0, radius).near > 0) + assert.equal(depthRange(1e-9, radius).near, radius * NEAR_FLOOR) +}) + +test('the planes keep the ratio the depth buffer can resolve', () => { + for (const radius of [0.09, 0.5, 2.3, 40]) { + for (const distance of [radius * 1e-3, radius * 0.1, radius, radius * 3, radius * 10]) { + const { near, far } = depthRange(distance, radius) + assert.ok(near > 0 && far > near) + assert.ok(far / near <= 1e6, `radius ${radius}, distance ${distance}: ${far / near}`) + } + } +}) + +test('a small piece and a large piece get planes of their own size', () => { + const small = depthRange(0.3, 0.09) // 185 mm + const large = depthRange(6, 2.3) // 4.6 m + assert.ok(small.near < large.near && small.far < large.far) + assert.ok(small.far > 0.3 + 0.09) +}) + +test('a distance or radius that is no number falls back to something sane', () => { + for (const [distance, radius] of [[Number.NaN, 1], [1, Number.NaN], [-3, 1], [1, 0], [Infinity, 1]]) { + const { near, far } = depthRange(distance, radius) + assert.ok(Number.isFinite(near) && Number.isFinite(far) && near > 0 && far > near) + } +}) + +test('the camera is only touched when a plane moved by more than a hundredth', () => { + const now = { near: 1, far: 100 } + assert.equal(depthChanged(now, { near: 1.005, far: 100.4 }), false) + assert.equal(depthChanged(now, { near: 1.2, far: 100 }), true) + assert.equal(depthChanged(now, { near: 1, far: 150 }), true) +}) + +test('the view is fitted again when the piece changes size, not for a small wobble', () => { + assert.equal(sizeChanged(0, 0.3), true) // the first size + assert.equal(sizeChanged(0.3, 0.3), false) + assert.equal(sizeChanged(0.3, 0.31), false) + assert.equal(sizeChanged(0.3, 4.6), true) // a version of another size + assert.equal(sizeChanged(4.6, 0.3), true) +}) + +test('a size that is no number never asks for a fit', () => { + for (const bad of [0, -1, Number.NaN, Infinity]) { + assert.equal(sizeChanged(0.3, bad), false) + assert.equal(sizeChanged(0, bad), false) + } +}) diff --git a/src/frontend/lib/viewerDepth.ts b/src/frontend/lib/viewerDepth.ts new file mode 100644 index 00000000..ac8deb35 --- /dev/null +++ b/src/frontend/lib/viewerDepth.ts @@ -0,0 +1,79 @@ +/** + * Faces sit a hair behind their edges (0.6.3.1): a mesh and the lines of its + * edges lie in the same surface, so at some angles the depth test let the + * faces cut the lines. The face materials carry a polygon offset, which moves + * the faces a little back in depth while the edges stay where they are. The + * depth test of the edges stays on: back edges are hidden by the faces as + * before, only the edges of the visible side win. + * + * Run: npm test + */ +import * as THREE from 'three' + +/** Offset of the faces: the slope factor handles steep faces, the units the flat ones. */ +export const FACE_POLYGON_OFFSET = { + polygonOffset: true, + polygonOffsetFactor: 1, + polygonOffsetUnits: 1, +} as const + +/** Move the faces of a material back (an existing one, e.g. a built-in mesh). */ +export function pushFacesBack(material: T): T { + material.polygonOffset = FACE_POLYGON_OFFSET.polygonOffset + material.polygonOffsetFactor = FACE_POLYGON_OFFSET.polygonOffsetFactor + material.polygonOffsetUnits = FACE_POLYGON_OFFSET.polygonOffsetUnits + return material +} + +/** The material of the faces of a mesh: vertex colours or grey, both sides, faces set back. */ +export function bodyFaceMaterial(hasColors: boolean): THREE.MeshBasicMaterial { + return pushFacesBack(hasColors + ? new THREE.MeshBasicMaterial({ vertexColors: true, side: THREE.DoubleSide }) + : new THREE.MeshBasicMaterial({ color: 0x888888, side: THREE.DoubleSide })) +} + +/** The material of the edge lines: black, depth test on, no offset. */ +export function edgeLineMaterial(): THREE.LineBasicMaterial { + return new THREE.LineBasicMaterial({ color: 0x000000 }) +} + +/** The camera planes (near, far) in the units of the scene. */ +export type DepthRange = { near: number; far: number } + +/** Nearest the near plane may come, as a share of the piece's radius. */ +export const NEAR_FLOOR = 1e-4 + +/** + * The camera's near and far planes (0.6.3.1). They follow the camera, not the + * first fit: ``distance`` is the distance of the camera to its orbit target + * (the middle of the piece) and ``radius`` the radius of the piece. The near + * plane is a hundredth of the distance, never below a ten-thousandth of the + * radius, so that zooming close to a face does not cut it away; the far plane + * lies ten times as far as the far side of the piece, so that the grid and + * the axes stay in view. The ratio of the two stays small enough for the + * depth buffer next to the polygon offset. + */ +export function depthRange(distance: number, radius: number): DepthRange { + const r = Number.isFinite(radius) && radius > 0 ? radius : 1 + const d = Number.isFinite(distance) && distance > 0 ? distance : r + const near = Math.max(d / 100, r * NEAR_FLOOR) + const far = Math.max((d + r) * 10, near * 100) + return { near, far } +} + +/** Whether a camera needs the new planes: a change of more than 1 %. */ +export function depthChanged(current: DepthRange, wanted: DepthRange): boolean { + const moved = (a: number, b: number) => Math.abs(a - b) > 0.01 * Math.max(a, b) + return moved(current.near, wanted.near) || moved(current.far, wanted.far) +} + +/** + * Whether the piece is another size than the one the view was last fitted to + * (its diagonal differs by more than 5 %); the first size, from 0, always is. + * A size that is no positive number never is. + */ +export function sizeChanged(fitted: number, current: number): boolean { + if (!Number.isFinite(current) || current <= 0) return false + if (!(fitted > 0)) return true + return Math.abs(current - fitted) > 0.05 * fitted +} diff --git a/src/frontend/lib/viewerSwitches.test.ts b/src/frontend/lib/viewerSwitches.test.ts index d640fa52..b0dd09ca 100644 --- a/src/frontend/lib/viewerSwitches.test.ts +++ b/src/frontend/lib/viewerSwitches.test.ts @@ -13,6 +13,7 @@ import { edgesControl, hasMeshSwitch, pointCloudStartsOff, + shouldLoadLevel, shownFlags, switchValue, withOff, @@ -101,3 +102,22 @@ test('the mesh switch is for a mesh or a proxy', () => { assert.equal(hasMeshSwitch(0, true), true) assert.equal(hasMeshSwitch(0, false), false) }) + +test('a stored level is fetched only while its layer is shown', () => { + const shown = { external: true, off: false, proxyOnly: false, hasIdentity: true } + assert.equal(shouldLoadLevel(shown), true) + assert.equal(shouldLoadLevel({ ...shown, off: true }), false) + assert.equal(shouldLoadLevel({ ...shown, external: false }), false) + assert.equal(shouldLoadLevel({ ...shown, proxyOnly: true }), false) + assert.equal(shouldLoadLevel({ ...shown, hasIdentity: false }), false) +}) + +test('switching back on from Off loads the level that was chosen before', () => { + const state: SwitchState = { level: 'original', off: false } + const off = chooseSwitch(state, 'off') + const base = { external: true, proxyOnly: false, hasIdentity: true } + assert.equal(shouldLoadLevel({ ...base, off: off.off }), false) + const back = chooseSwitch(off, off.level) + assert.equal(back.level, 'original') + assert.equal(shouldLoadLevel({ ...base, off: back.off }), true) +}) diff --git a/src/frontend/lib/viewerSwitches.ts b/src/frontend/lib/viewerSwitches.ts index 09081449..c2cd9482 100644 --- a/src/frontend/lib/viewerSwitches.ts +++ b/src/frontend/lib/viewerSwitches.ts @@ -43,6 +43,21 @@ export function edgesControl(showEdges: boolean, meshOff: boolean): { checked: b return { checked: showEdges, disabled: meshOff } } +/** + * Whether a stored level (the reduced or original mesh, the original point + * cloud) is fetched now: only while its layer is shown. A layer that is Off + * loads nothing, also not when the version changes; choosing a level again + * loads it (0.6.3.1). + */ +export function shouldLoadLevel(state: { + external: boolean + off: boolean + proxyOnly: boolean + hasIdentity: boolean +}): boolean { + return state.external && !state.off && !state.proxyOnly && state.hasIdentity +} + /** A piece with a mesh starts with its point cloud hidden (as before the switch had Off). */ export function pointCloudStartsOff(meshCount: number): boolean { return meshCount > 0 diff --git a/src/frontend/public/backgroundmeshes/manifest.json b/src/frontend/public/backgroundmeshes/manifest.json index 703885fd..9a511ddd 100644 --- a/src/frontend/public/backgroundmeshes/manifest.json +++ b/src/frontend/public/backgroundmeshes/manifest.json @@ -36,6 +36,5 @@ "type": "reduced" } ], - "total_count": 7, - "converted_at": "C:\\Users\\EFESTWIN\\source\\repos\\csc" + "total_count": 7 } \ No newline at end of file diff --git a/tasks.py b/tasks.py index 80fa4f70..9acdb3cd 100644 --- a/tasks.py +++ b/tasks.py @@ -431,21 +431,104 @@ def bump_version(c, version, gh=False): _rewrite(changelog, r'^## \[', f'## [{version}] - unreleased{eol}{eol}## [') if gh: + release_prep = _release_prep_module() session = os.path.join(REPO_DIR, 'grasshopper_userobjects_src', 'DDU_CSC_Session.py') - _rewrite(session, rf"^CSC_CLIENT = 'gh-userobjects/{v}'", - f"CSC_CLIENT = 'gh-userobjects/{version}'") - _rewrite(session, r'^Version: \d+(\.\d+)?[a-z]?(?=\r?$)', - f"Version: {datetime.date.today():%y%m%d}") - print('GH: CSC_Session changed -> re-export DDU_CSC_Session.ghuser ' - '(and its XML) in Rhino before releasing') + # latin-1 reads and writes every byte back unchanged: the file is + # saved by Rhino's editor and may not be UTF-8 + with open(session, encoding='latin-1', newline='') as handle: + source = handle.read() + try: + new_source, changed = release_prep.gh_client_update( + source, version, datetime.date.today()) + except ValueError as exc: + sys.exit(f'bump-version: DDU_CSC_Session.py: {exc}') + if changed: + with open(session, 'w', encoding='latin-1', newline='') as handle: + handle.write(new_source) + print('GH: CSC_Session changed -> re-export ' + 'DDU_CSC_Session.ghuser (and its XML) in Rhino before ' + 'releasing') + else: + print(f'GH: CSC_CLIENT is gh-userobjects/{version} already: ' + f'DDU_CSC_Session.py is left as it is (the Version: ' + f'header keeps matching the exported .ghuser)') print(f'VERSION -> {version}; now fill in the CHANGELOG section, then ' - f'run invoke zenodo') + f'run invoke release-prep') with chdir(REPO_DIR): c.run(f'{sys.executable} scripts/ci/check_version.py', pty=False, warn=True) +def _release_prep_module(): + """scripts/ci/release_prep.py: the pure parts, shared with the tests.""" + import importlib.util + path = os.path.join(REPO_DIR, 'scripts', 'ci', 'release_prep.py') + spec = importlib.util.spec_from_file_location('release_prep', path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@task(help={'date': 'release date YYYY-MM-DD (default: today)'}) +def release_prep(c, date=''): + """ + Prepare the release of the version in VERSION: date the CHANGELOG section, + write .zenodo.json, check version, tag and Zenodo metadata, run tests/ci. + Run after bump-version and after filling in the CHANGELOG section; stops + with an error on any failure. Running it again only checks. + """ + import datetime + release_prep = _release_prep_module() + with open(os.path.join(REPO_DIR, 'VERSION'), encoding='utf-8') as handle: + version = handle.read().strip() + try: + day = datetime.date.fromisoformat(date) if date \ + else datetime.date.today() + except ValueError: + sys.exit(f'release-prep: --date must be YYYY-MM-DD, not "{date}"') + changelog = os.path.join(REPO_DIR, 'CHANGELOG.md') + # newline='': the file's own line endings (CRLF) are kept + with open(changelog, encoding='utf-8', newline='') as handle: + text = handle.read() + body = release_prep.changelog_section(text, version) + if body is None: + sys.exit(f'release-prep: CHANGELOG.md has no "## [{version}]" ' + f'section; run invoke bump-version or add it') + if release_prep.section_is_empty(body): + sys.exit(f'release-prep: the "## [{version}]" section of ' + f'CHANGELOG.md is empty; fill it in first (it becomes the ' + f'release notes)') + new_text, state = release_prep.set_changelog_date( + text, version, day.isoformat()) + if state == 'dated': + with open(changelog, 'w', encoding='utf-8', newline='') as handle: + handle.write(new_text) + print(f'CHANGELOG: [{version}] dated {day.isoformat()}') + elif state == 'already': + print(f'CHANGELOG: [{version}] is dated already ' + f'({release_prep.existing_date(text, version)}): only checking') + else: + sys.exit(f'release-prep: the heading of [{version}] in CHANGELOG.md ' + f'is neither "unreleased" nor dated') + steps = [ + ('Zenodo metadata', f'{sys.executable} scripts/ci/zenodo_metadata.py'), + ('version check', f'{sys.executable} scripts/ci/check_version.py ' + f'--tag v{version}'), + ('Zenodo check', f'{sys.executable} scripts/ci/zenodo_metadata.py ' + f'--check'), + ('tests/ci', f'{sys.executable} -m pytest tests/ci -q'), + ] + with chdir(REPO_DIR): + for name, command in steps: + print(f'--- {name}') + result = c.run(command, pty=False, warn=True) + if not result.ok: + sys.exit(f'release-prep: {name} failed (see above); fix it ' + f'and run invoke release-prep again') + print('\n' + release_prep.next_steps(version)) + + @task(help={'check': 'only report whether .zenodo.json is stale'}) def zenodo(c, check=False): """ diff --git a/tests/api/support.py b/tests/api/support.py index 49e3094a..0c49d9c3 100644 --- a/tests/api/support.py +++ b/tests/api/support.py @@ -44,13 +44,39 @@ def dump_files(dump_dir): return files -def load_dump(database, dump_dir, replace=False): - """Insert every exported collection; returns {collection: count}.""" +def latest_dump(dumps_dir): + """The newest dump folder of ``mongodb_collections_local``: a folder whose + name is six digits (yymmdd); the greatest name is the newest. None when + there is none.""" + import os + import re + if not os.path.isdir(dumps_dir): + return None + names = [n for n in os.listdir(dumps_dir) + if re.fullmatch(r'\d{6}', n) + and os.path.isdir(os.path.join(dumps_dir, n))] + return max(names) if names else None + + +def read_export(path): + """The documents of one Compass export. An export file that is empty (0 + bytes, or only white space), as Compass writes for a collection with no + documents, holds none.""" from bson import json_util + with open(path, encoding='utf-8') as handle: + text = handle.read() + if not text.strip(): + return [] + return json_util.loads(text) + + +def load_dump(database, dump_dir, replace=False): + """Insert every exported collection; returns {collection: count}. A + collection with no export file is simply not loaded (the dump predates + it); an empty export file loads as no documents.""" counts = {} for collection, path in dump_files(dump_dir).items(): - with open(path, encoding='utf-8') as handle: - docs = json_util.loads(handle.read()) + docs = read_export(path) if replace: database[collection].delete_many({}) if docs: diff --git a/tests/api/test_dump_helpers.py b/tests/api/test_dump_helpers.py new file mode 100644 index 00000000..02dc0722 --- /dev/null +++ b/tests/api/test_dump_helpers.py @@ -0,0 +1,73 @@ +"""The helpers of the local dump folders (tests/api/support.py): which folder +is the newest, and an export file with no documents (0 bytes) loads as none, +a collection with no export file is not loaded.""" + +from __future__ import annotations + +import json + +from support import dump_files, latest_dump, load_dump, read_export + + +def test_the_latest_dump_is_the_greatest_six_digit_folder(tmp_path): + for name in ('260520', '261009', '261020', '261016'): + (tmp_path / name).mkdir() + assert latest_dump(str(tmp_path)) == '261020' + + +def test_folders_that_are_not_six_digits_and_files_are_not_dumps(tmp_path): + for name in ('261009', 'archive', '2610200', '99999', 'backup-261231'): + (tmp_path / name).mkdir() + (tmp_path / '261231').write_text('a file, not a dump', encoding='utf-8') + assert latest_dump(str(tmp_path)) == '261009' + + +def test_no_dump_and_no_folder_give_none(tmp_path): + assert latest_dump(str(tmp_path)) is None + assert latest_dump(str(tmp_path / 'missing')) is None + + +def test_an_export_file_of_zero_bytes_has_no_documents(tmp_path): + empty = tmp_path / 'csc.access_tokens.json' + empty.write_bytes(b'') + assert read_export(str(empty)) == [] + blank = tmp_path / 'csc.blank.json' + blank.write_text(' \r\n\n', encoding='utf-8') + assert read_export(str(blank)) == [] + + +def test_an_export_file_with_documents_is_read(tmp_path): + path = tmp_path / 'csc.datasets.json' + path.write_text(json.dumps([{'_id': 'a'}, {'_id': 'b'}]), encoding='utf-8') + assert [d['_id'] for d in read_export(str(path))] == ['a', 'b'] + path.write_text('[]', encoding='utf-8') + assert read_export(str(path)) == [] + + +class FakeCollection: + def __init__(self): + self.docs = [] + + def delete_many(self, _filter): + self.docs = [] + + def insert_many(self, docs, ordered=True): + self.docs.extend(docs) + + +class FakeDatabase(dict): + def __missing__(self, name): + self[name] = FakeCollection() + return self[name] + + +def test_a_dump_with_an_empty_file_and_a_missing_collection_loads(tmp_path): + (tmp_path / 'csc.datasets.json').write_text(json.dumps([{'_id': 'a'}]), encoding='utf-8') + (tmp_path / 'csc.access_tokens.json').write_bytes(b'') # exported, no documents + # csc.invitations.json is not there at all: the dump predates it + database = FakeDatabase() + counts = load_dump(database, str(tmp_path)) + assert counts == {'access_tokens': 0, 'datasets': 1} + assert [d['_id'] for d in database['datasets'].docs] == ['a'] + assert database['access_tokens'].docs == [] + assert 'invitations' not in counts and sorted(dump_files(str(tmp_path))) == ['access_tokens', 'datasets'] diff --git a/tests/api/test_evidence_fold_routes.py b/tests/api/test_evidence_fold_routes.py index 814dbca1..e8edb967 100644 --- a/tests/api/test_evidence_fold_routes.py +++ b/tests/api/test_evidence_fold_routes.py @@ -401,3 +401,31 @@ def test_the_timeline_merges_everything_and_hides_what_it_must(world): row = [e for e in outside if e.get('record_id') == mid['_id']][0] assert row['status'] == 'withdrawn' and 'secret' not in str(row) assert row['quantity'] is None + + +def test_two_versions_each_see_only_their_own_records(world): + """The snapshot-scoped fold (0.6.3.1 check): a record belongs to the + version that was current when it was observed; a claim about the piece + as a whole belongs to neither. Each version's properties hold its own + records and nothing of the other's.""" + w = world + v0, v1 = sid('beam', 0), sid('beam', 1) + a = put(w, visual_body('spalling', 1, observed_at=IN_V0)) + b = put(w, visual_body('crack_width', 0.2, observed_at=IN_V0)) + c = put(w, visual_body('spalling', 3, observed_at=IN_V1)) + d = put(w, visual_body('crack_width', 0.9, observed_at=IN_V1)) + put(w, claim_body('compressive_strength', (18, 28))) + first, second = snap_props(w, v0), snap_props(w, v1) + assert first['spalling']['evidence_ids'] == [a['_id']] + assert first['crack_width']['evidence_ids'] == [b['_id']] + assert second['spalling']['evidence_ids'] == [c['_id']] + assert second['crack_width']['evidence_ids'] == [d['_id']] + assert first['spalling']['range'] == [1, 1] and second['spalling']['range'] == [3, 3] + assert first['crack_width']['range'] == [0.2, 0.2] and second['crack_width']['range'] == [0.9, 0.9] + for props in (first, second): + assert 'compressive_strength' not in props + assert props['spalling']['n'] == 1 + # a record added to v1 later leaves v0 alone + e = put(w, visual_body('spalling', 2, observed_at=IN_V1)) + assert snap_props(w, v0)['spalling']['evidence_ids'] == [a['_id']] + assert sorted(snap_props(w, v1)['spalling']['evidence_ids']) == sorted([c['_id'], e['_id']]) diff --git a/tests/api/test_geometry_patch_markers.py b/tests/api/test_geometry_patch_markers.py new file mode 100644 index 00000000..3a24b04a --- /dev/null +++ b/tests/api/test_geometry_patch_markers.py @@ -0,0 +1,222 @@ +"""File markers follow a geometry PATCH on a draft (0.6.3.1): a mesh or cloud +that was replaced or removed loses its marker and its file; one that is +unchanged keeps both. ``reconcile_markers`` is the rule.""" + +from __future__ import annotations + +import os + +import numpy as np +import pytest +import trimesh + +from apps.catalog.file_markers import reconcile_markers, remove_files +from apps.catalog.migration06.steps import CUTOVER_STEPS, Context, run +from support import iid, seed_05_catalog + +BEAM = iid('beam') + + +# THE RULE ---------------------------------------------------------------------- +def geometry(meshes=(), clouds=()): + return {'meshes': [{'v': m} for m in meshes], + 'point_clouds': [{'p': c} for c in clouds]} + + +def test_markers_of_unchanged_primitives_are_kept(): + old = geometry(['a', 'b'], ['x', 'y']) + done = reconcile_markers(old, geometry(['a', 'b'], ['x', 'y']), + {'0': ['reduced', 'detailed'], '1': ['reduced']}, [0, 1]) + assert done.resolutions == {'0': ['reduced', 'detailed'], '1': ['reduced']} + assert done.cloud_files == [0, 1] and not done.changed + + +def test_a_replaced_primitive_loses_its_marker_the_others_keep_theirs(): + done = reconcile_markers(geometry(['a', 'b'], ['x', 'y']), + geometry(['a', 'CHANGED'], ['x', 'NEW']), + {'0': ['reduced'], '1': ['reduced', 'detailed']}, [0, 1]) + assert done.resolutions == {'0': ['reduced']} + assert done.dropped_meshes == [(1, 'reduced'), (1, 'detailed')] + assert done.cloud_files == [0] and done.dropped_clouds == [1] + + +def test_a_primitive_that_is_gone_loses_its_marker(): + done = reconcile_markers(geometry(['a', 'b'], ['x', 'y']), + geometry(['a'], ['x']), + {'0': ['reduced'], '1': ['reduced']}, [0, 1]) + assert done.resolutions == {'0': ['reduced']} and done.cloud_files == [0] + assert done.dropped_meshes == [(1, 'reduced')] and done.dropped_clouds == [1] + + +def test_a_shifted_primitive_is_another_primitive(): + """The file of index 0 described 'a'; index 0 is 'b' now.""" + done = reconcile_markers(geometry(['a', 'b']), geometry(['b']), {'0': ['reduced']}, []) + assert done.resolutions == {} and done.dropped_meshes == [(0, 'reduced')] + + +def test_a_marker_beyond_the_old_geometry_is_dropped(): + done = reconcile_markers(geometry(['a']), geometry(['a', 'b']), {'5': ['reduced']}, [7]) + assert done.resolutions == {} and done.cloud_files == [] and done.changed + + +def test_no_markers_and_no_geometry_are_fine(): + done = reconcile_markers(None, None, None, None) + assert done.resolutions == {} and done.cloud_files == [] and not done.changed + done = reconcile_markers({}, {'meshes': 'not a list'}, {'0': ['reduced']}, [0]) + assert done.changed + + +def test_a_key_that_is_no_index_is_dropped(): + done = reconcile_markers(geometry(['a']), geometry(['a']), {'x': ['reduced']}, []) + assert done.resolutions == {} and done.dropped_meshes == [(-1, 'reduced')] + + +def test_the_files_of_dropped_markers_are_removed_and_a_missing_file_is_fine(tmp_path): + meshes, clouds = tmp_path / 'meshes', tmp_path / 'clouds' + (meshes / 'sid' / '1').mkdir(parents=True) + (meshes / 'sid' / '1' / 'reduced.ply').write_bytes(b'ply') + (meshes / 'sid' / '0').mkdir(parents=True) + (meshes / 'sid' / '0' / 'reduced.ply').write_bytes(b'ply') + (clouds / 'sid').mkdir(parents=True) + (clouds / 'sid' / '1.ply').write_bytes(b'ply') + (clouds / 'sid' / '0.ply').write_bytes(b'ply') + done = reconcile_markers(geometry(['a', 'b'], ['x', 'y']), geometry(['a'], ['x']), + {'0': ['reduced'], '1': ['reduced', 'detailed']}, [0, 1]) + assert remove_files(str(meshes), str(clouds), 'sid', done) == 2 # detailed.ply was never there + assert not (meshes / 'sid' / '1').exists() # the empty folder went + assert (meshes / 'sid' / '0' / 'reduced.ply').exists() + assert (clouds / 'sid' / '0.ply').exists() and not (clouds / 'sid' / '1.ply').exists() + + +# THE ROUTE --------------------------------------------------------------------- +@pytest.fixture +def story(api, db, member_headers): + seed_05_catalog(db) + run(Context(db=db, files=False, log=lambda _m: None), CUTOVER_STEPS) + contributor, _ = member_headers({'dbu_zirkus': ['contributor']}) + return {'api': api, 'db': db, 'c': contributor} + + +def box(size): + mesh = trimesh.creation.box(extents=(size, 200, 150)) + return {'vertices': mesh.vertices.tolist(), 'faces': mesh.faces.tolist()} + + +def cloud(seed): + return {'points': (np.random.default_rng(seed).random((10, 3)) * 100).tolist()} + + +def draft(story, meshes, clouds): + geometry_doc = {'meshes': meshes, 'point_clouds': clouds, 'proxies': []} + response = story['api'].post(f'/identities/{BEAM}/snapshots', + json={'geometry': geometry_doc}, headers=story['c']) + assert response.status_code == 201, response.text + return response.json()['_id'] + + +def ply(seed=1): + points = np.random.default_rng(seed).random((300, 3)) * 100 + return ('x.ply', trimesh.PointCloud(points).export(file_type='ply'), + 'application/octet-stream') + + +def put_cloud(story, sid, index): + done = story['api'].put(f'/snapshots/{sid}/point_clouds/{index}.ply', + files={'point_cloud_file': ply(index)}, headers=story['c']) + assert done.status_code == 200, done.text + + +def put_mesh(story, sid, index, level='reduced'): + done = story['api'].put(f'/snapshots/{sid}/meshes/{index}/{level}', + files={'mesh_file': ply(index)}, headers=story['c']) + assert done.status_code == 200, done.text + + +def stored(story, sid): + return story['db']['component_snapshots'].find_one({'_id': sid}) + + +def cloud_path(sid, index): + return os.path.join(os.environ['SNAPSHOT_POINT_CLOUDS_DIR'], sid, f'{index}.ply') + + +def mesh_path(sid, index, level='reduced'): + return os.path.join(os.environ['SNAPSHOT_MESHES_DIR'], sid, str(index), f'{level}.ply') + + +def patch(story, sid, meshes, clouds): + done = story['api'].patch( + f'/snapshots/{sid}', headers=story['c'], + json={'geometry': {'meshes': meshes, 'point_clouds': clouds, 'proxies': []}}) + assert done.status_code == 200, done.text + return done.json() + + +def test_replacing_the_meshes_and_clouds_clears_the_markers_and_the_files(story): + sid = draft(story, [box(300), box(400)], [cloud(1), cloud(2)]) + for index in (0, 1): + put_cloud(story, sid, index) + put_mesh(story, sid, index) + put_mesh(story, sid, 1, 'detailed') + assert os.path.isfile(cloud_path(sid, 1)) and os.path.isfile(mesh_path(sid, 1, 'detailed')) + body = patch(story, sid, [box(500), box(600)], [cloud(8), cloud(9)]) + assert body['mesh_ply_resolutions'] == {} and body['point_cloud_files'] == [] + row = stored(story, sid) + assert row['mesh_ply_resolutions'] == {} and row['point_cloud_files'] == [] + for index in (0, 1): + assert not os.path.exists(cloud_path(sid, index)) + assert not os.path.exists(mesh_path(sid, index)) + assert not os.path.exists(mesh_path(sid, 1, 'detailed')) + + +def test_a_patch_that_keeps_the_primitives_keeps_markers_and_files(story): + sid = draft(story, [box(300)], [cloud(1)]) + put_cloud(story, sid, 0) + put_mesh(story, sid, 0) + body = patch(story, sid, [box(300)], [cloud(1)]) + assert body['mesh_ply_resolutions'] == {'0': ['reduced']} + assert body['point_cloud_files'] == [0] + assert os.path.isfile(cloud_path(sid, 0)) and os.path.isfile(mesh_path(sid, 0)) + + +def test_only_the_replaced_primitive_loses_its_marker_and_file(story): + sid = draft(story, [box(300), box(400)], [cloud(1), cloud(2)]) + for index in (0, 1): + put_cloud(story, sid, index) + put_mesh(story, sid, index) + body = patch(story, sid, [box(300), box(450)], [cloud(1), cloud(7)]) + assert body['mesh_ply_resolutions'] == {'0': ['reduced']} + assert body['point_cloud_files'] == [0] + assert os.path.isfile(mesh_path(sid, 0)) and not os.path.exists(mesh_path(sid, 1)) + assert os.path.isfile(cloud_path(sid, 0)) and not os.path.exists(cloud_path(sid, 1)) + + +def test_fewer_primitives_leave_no_marker_past_the_new_list(story): + sid = draft(story, [box(300), box(400)], [cloud(1), cloud(2)]) + for index in (0, 1): + put_cloud(story, sid, index) + put_mesh(story, sid, index) + body = patch(story, sid, [box(300)], [cloud(1)]) + assert body['mesh_ply_resolutions'] == {'0': ['reduced']} + assert body['point_cloud_files'] == [0] + assert not os.path.exists(cloud_path(sid, 1)) and not os.path.exists(mesh_path(sid, 1)) + + +def test_the_cloud_is_served_from_the_new_inline_points_after_the_patch(story): + sid = draft(story, [box(300)], [cloud(1)]) + put_cloud(story, sid, 0) + patch(story, sid, [box(300)], [cloud(5)]) + served = story['api'].get(f'/snapshots/{sid}/point_clouds/0.ply', headers=story['c']) + assert served.status_code == 200 + assert served.content[:3] == b'ply' + + +def test_a_patch_without_geometry_leaves_markers_alone(story): + sid = draft(story, [box(300)], [cloud(1)]) + put_cloud(story, sid, 0) + put_mesh(story, sid, 0) + done = story['api'].patch(f'/snapshots/{sid}', json={'color': [10, 20, 30]}, + headers=story['c']) + assert done.status_code == 200, done.text + assert done.json()['point_cloud_files'] == [0] + assert done.json()['mesh_ply_resolutions'] == {'0': ['reduced']} diff --git a/tests/api/test_geometry_remote.py b/tests/api/test_geometry_remote.py index c63d13dc..2258b41f 100644 --- a/tests/api/test_geometry_remote.py +++ b/tests/api/test_geometry_remote.py @@ -638,3 +638,30 @@ def test_a_value_that_is_not_json_fails_loudly_and_leaves_no_entry(tmp_path): cache.store(key, bad) assert cache.entry('sid') is None assert [p for p in os.listdir(str(tmp_path))] == [] # no half-written folder + + +def test_a_frame_only_cut_drops_the_descriptor_stage_keys(tmp_path): + """``descriptors`` is shared by two stages: a cut to ``frame`` carries + the hull scores the frame stage wrote, not the radial signature or the HKS + of the descriptors stage; a cut with the descriptors stage carries all.""" + from apps.catalog.geometry_cache import cache_key + from apps.catalog.geometry_stages import HULL_SCORES, Outcome + key = cache_key({'_id': 'sid', 'geometry': {}}, {}, {'meshes': {}, 'point_clouds': {}}) + outcome = Outcome() + outcome.ran = ['frame', 'descriptors'] + scores = {name: 0.5 for name in HULL_SCORES} + outcome.set = {'frame': {'o': [0, 0, 0]}, 'bbx': [1, 2, 3], + 'descriptors': {**scores, 'hks': [1.0], 'radial_signature': [2.0], + 'sas_vectors': [[0.0]]}} + cache = ResultCache(str(tmp_path)) + cache.store(key, outcome) + frame_only = cache.lookup(key, ['frame']).outcome + assert frame_only.set['descriptors'] == scores + assert frame_only.set['frame'] == {'o': [0, 0, 0]} and frame_only.set['bbx'] == [1, 2, 3] + both = cache.lookup(key, ['frame', 'descriptors']).outcome + assert set(both.set['descriptors']) == set(scores) | {'hks', 'radial_signature', 'sas_vectors'} + only_descriptors = cache.lookup(key, ['descriptors']).outcome + assert 'hks' in only_descriptors.set['descriptors'] + assert 'frame' not in only_descriptors.set + # what the cache holds is not changed by a cut + assert 'hks' in cache.entry('sid')['set']['descriptors'] diff --git a/tests/api/test_mail_templates.py b/tests/api/test_mail_templates.py index ccfe5a04..e79e490a 100644 --- a/tests/api/test_mail_templates.py +++ b/tests/api/test_mail_templates.py @@ -289,14 +289,21 @@ def test_a_test_mail_needs_an_admin_and_an_address(api, db, auth_headers, sent): assert sent['mails'] == [] -def test_mail_that_is_not_set_up_says_so(api, auth_headers, sent, monkeypatch): - monkeypatch.delenv('SMTP_HOST', raising=False) - got = api.post('/admin/mail-templates/reset/test', - headers=auth_headers('admin')) +def test_mail_that_is_not_set_up_says_so(api, auth_headers, sent, env): + admin = auth_headers('admin') + # set up (the test environment has SMTP_HOST): the test mail goes out + assert api.post('/admin/mail-templates/reset/test', + headers=admin).status_code == 200 + assert len(sent['mails']) == 1 + sent['mails'].clear() + # the settings are cached, so the variable is removed through ``env``, + # which reads them again (a bare delenv would change nothing) + env(SMTP_HOST=None) + got = api.post('/admin/mail-templates/reset/test', headers=admin) assert got.status_code == 502 and 'SMTP_HOST' in got.json()['detail'] + assert sent['mails'] == [] # the preview does not need the SMTP settings - assert api.get('/admin/mail-templates', - headers=auth_headers('admin')).status_code == 200 + assert api.get('/admin/mail-templates', headers=admin).status_code == 200 def test_the_test_send_is_rate_limited(api, auth_headers, sent): diff --git a/tests/api/test_token_cleanup.py b/tests/api/test_token_cleanup.py new file mode 100644 index 00000000..2332acac --- /dev/null +++ b/tests/api/test_token_cleanup.py @@ -0,0 +1,326 @@ +""" +Token clean-up (0.6.3.1): the access tokens and OAuth rows of a deleted account +go with it (the one deletion path, the unverified-account cleanup); a revoked +row, or one expired past its refresh limit, gets a ``purge_at`` date 90 days +after it died and a TTL index deletes it, with no job; rows of an earlier build +get the date once at start. +""" + +from __future__ import annotations + +import asyncio +from datetime import datetime, timedelta, timezone + +import pytest + +import usermaintenance +from apps.catalog import access_tokens as tk +from apps.catalog.token_cleanup import backfill_purge_dates, remove_tokens_of +from oauth_helpers import bearer, connect, gh_sign_in, refresh, reset_address_counters +from support import DEFAULT_PASSWORD + +NOW = datetime.now(timezone.utc) +OLD = NOW - timedelta(days=9) + + +@pytest.fixture(autouse=True) +def fresh_counters(): + reset_address_counters() + yield + reset_address_counters() + + +@pytest.fixture +def owner(make_user, login): + user = make_user('cleanowner', role='admin') + return {**user, 'headers': bearer(login('cleanowner').json()['access_token'])} + + +def aware(value): + """A date read back from Mongo is naive UTC.""" + return value.replace(tzinfo=timezone.utc) + + +def same(a, b): + """Mongo keeps milliseconds.""" + return abs((a - b).total_seconds()) < 1 + + +# THE DATE A ROW MAY GO --------------------------------------------------------- +def stamp_days(days): + return tk.stamp(NOW + timedelta(days=days)) + + +def test_a_row_may_go_90_days_after_it_expires(): + row = {'expires': stamp_days(30)} + assert tk.purge_at(row) == tk.parse_stamp(row['expires']) + timedelta(days=90) + + +def test_a_connection_dies_with_its_refresh_limit_when_that_comes_first(): + row = {'expires': stamp_days(90), 'refresh_expires': stamp_days(30)} + assert tk.purge_at(row) == tk.parse_stamp(row['refresh_expires']) + timedelta(days=90) + + +def test_a_revoked_row_goes_90_days_after_the_revoke_if_that_is_earlier(): + row = {'expires': stamp_days(300), 'revoked': stamp_days(-1)} + assert tk.purge_at(row) == tk.parse_stamp(row['revoked']) + timedelta(days=90) + + +def test_the_purge_date_is_never_before_the_row_died(): + row = {'expires': stamp_days(-100), 'refresh_expires': stamp_days(-120)} + assert tk.purge_at(row) == tk.parse_stamp(row['refresh_expires']) + timedelta(days=90) + + +# WRITTEN WHERE A ROW IS MADE AND REVOKED --------------------------------------- +def test_a_personal_token_gets_its_purge_date_when_made(api, db, owner): + row = api.post('/auth/access-tokens', json={'name': 'a', 'days': 30}, + headers=owner['headers']).json() + stored = db['access_tokens'].find_one({'_id': row['id']}) + assert same(aware(stored['purge_at']), tk.parse_stamp(row['expires']) + timedelta(days=90)) + + +def test_revoking_a_personal_token_pulls_the_purge_date_in(api, db, owner): + row = api.post('/auth/access-tokens', json={'name': 'a', 'days': 365}, + headers=owner['headers']).json() + before = aware(db['access_tokens'].find_one({'_id': row['id']})['purge_at']) + api.delete(f'/auth/access-tokens/{row["id"]}', headers=owner['headers']) + after = aware(db['access_tokens'].find_one({'_id': row['id']})['purge_at']) + assert after < before + assert abs((after - (datetime.now(timezone.utc) + timedelta(days=90))).total_seconds()) < 120 + + +def test_a_token_revoked_late_keeps_its_earlier_date(api, db, owner): + """$min: a token that dies in 30 days is revoked in 29: still 120 days.""" + row = api.post('/auth/access-tokens', json={'name': 'a', 'days': 30}, + headers=owner['headers']).json() + api.delete(f'/auth/access-tokens/{row["id"]}', headers=owner['headers']) + stored = db['access_tokens'].find_one({'_id': row['id']}) + # revoked now: now + 90 days is later than expires + 90 - 30 days + assert aware(stored['purge_at']) <= tk.parse_stamp(row['expires']) + timedelta(days=90) + + +def test_a_connection_has_its_date_and_a_refresh_moves_it(api, db, owner): + claude = connect(api, owner['headers']) + row = db['access_tokens'].find_one({'kind': 'oauth'}) + assert same(aware(row['purge_at']), tk.parse_stamp(row['refresh_expires']) + timedelta(days=90)) + db['access_tokens'].update_one({'_id': row['_id']}, {'$set': { + 'refresh_expires': stamp_days(2)}}) + assert refresh(api, claude['client_id'], claude['refresh']).status_code == 200 + after = db['access_tokens'].find_one({'_id': row['_id']}) + assert same(aware(after['purge_at']), tk.parse_stamp(after['refresh_expires']) + timedelta(days=90)) + assert aware(after['purge_at']) > aware(row['purge_at']) - timedelta(days=30) + + +def test_a_grasshopper_sign_in_has_its_date_too(api, db, owner): + gh_sign_in(api, owner['headers']) + row = db['access_tokens'].find_one({'kind': 'device'}) + assert row['purge_at'] is not None + + +def test_ending_a_connection_and_a_password_change_set_the_date(api, db, owner): + connect(api, owner['headers']) + gh_sign_in(api, owner['headers']) + before = {r['_id']: aware(r['purge_at']) for r in db['access_tokens'].find({})} + done = api.post('/auth/change-password', json={ + 'current_password': DEFAULT_PASSWORD, 'new_password': 'another long password'}, + headers=owner['headers']) + assert done.status_code == 200 + for row in db['access_tokens'].find({}): + assert row['revoked'] + assert aware(row['purge_at']) < before[row['_id']] + + +def test_an_admin_revoke_of_a_connection_sets_the_date(api, db, owner): + connect(api, owner['headers']) + row = db['access_tokens'].find_one({'kind': 'oauth'}) + before = aware(row['purge_at']) + done = api.delete(f'/admin/access-tokens/{row["_id"]}', headers=owner['headers']) + assert done.status_code == 200 + assert aware(db['access_tokens'].find_one({'_id': row['_id']})['purge_at']) < before + + +# THE TTL INDEX ----------------------------------------------------------------- +def test_a_ttl_index_on_purge_at_deletes_at_the_date(db): + found = [spec for spec in db['access_tokens'].index_information().values() + if spec['key'] == [('purge_at', 1)]] + assert len(found) == 1 and found[0]['expireAfterSeconds'] == 0 + + +# ROWS OF AN EARLIER BUILD ------------------------------------------------------- +def old_row(row_id, **extra): + return {'_id': row_id, 'kind': 'pat', 'owner_user_id': 'u', 'scopes': [tk.SCOPE_CATALOG], + 'name': 'old', 'token_sha256': row_id, 'short_id': row_id[:8], + 'created': tk.stamp(NOW - timedelta(days=200)), 'expires': stamp_days(-100), + 'last_used': None, 'revoked': None, **extra} + + +def test_rows_without_a_date_get_one_at_start_and_a_second_run_changes_nothing(db): + db['access_tokens'].insert_many([ + old_row('expired-row'), + old_row('revoked-row', expires=stamp_days(200), revoked=stamp_days(-10)), + old_row('has-date', purge_at=NOW + timedelta(days=5)), + ]) + done = asyncio.run(_backfill(db)) + assert done == 2 + expired = db['access_tokens'].find_one({'_id': 'expired-row'}) + assert same(aware(expired['purge_at']), tk.parse_stamp(expired['expires']) + timedelta(days=90)) + revoked = db['access_tokens'].find_one({'_id': 'revoked-row'}) + assert same(aware(revoked['purge_at']), tk.parse_stamp(revoked['revoked']) + timedelta(days=90)) + untouched = db['access_tokens'].find_one({'_id': 'has-date'}) + assert abs((aware(untouched['purge_at']) - (NOW + timedelta(days=5))).total_seconds()) < 1 + assert asyncio.run(_backfill(db)) == 0 + + +def test_nothing_is_written_at_start_while_the_read_only_flag_is_set(db, read_only_flag): + db['access_tokens'].insert_one(old_row('expired-row')) + read_only_flag.on() + assert asyncio.run(_backfill(db)) == 0 + read_only_flag.off() + assert 'purge_at' not in db['access_tokens'].find_one({'_id': 'expired-row'}) + + +async def _backfill(db): + from pymongo import AsyncMongoClient + client = AsyncMongoClient(db.client.address[0] + ':' + str(db.client.address[1])) + try: + return await backfill_purge_dates(client[db.name]['access_tokens']) + finally: + await client.close() + + +# A DELETED ACCOUNT -------------------------------------------------------------- +def _unverified(uid): + return {'_id': uid, 'username': f'name-{uid}', 'email': f'{uid}@mail.example.org', + 'full_name': uid, 'hashed_password': 'x', 'role': 'user', 'disabled': False, + 'email_verified': False, 'verification_token_expires': OLD} + + +def _rows_for(db, uid, tag): + db['access_tokens'].insert_many([ + old_row(f'{tag}-pat', owner_user_id=uid), + old_row(f'{tag}-oauth', kind='oauth', owner_user_id=uid, expires=stamp_days(50)), + old_row(f'{tag}-device', kind='device', owner_user_id=uid, expires=stamp_days(50)), + ]) + db['oauth_codes'].insert_one({'_id': f'{tag}-code', 'user_id': uid}) + db['oauth_requests'].insert_one({'_id': f'{tag}-request', 'bound_user': uid}) + db['oauth_used_refresh'].insert_many([ + {'_id': f'{tag}-used-1', 'grant_id': f'{tag}-oauth', 'at': NOW}, + {'_id': f'{tag}-used-2', 'grant_id': f'{tag}-device', 'at': NOW}]) + + +def names(db, collection): + return sorted(doc['_id'] for doc in db[collection].find({})) + + +@pytest.fixture +def maintenance(mongod, db, db_name, monkeypatch): + monkeypatch.setattr(usermaintenance, 'get_db_connectionstring', + lambda: f'{mongod.uri}/{db_name}') + monkeypatch.setattr(usermaintenance, 'get_database_name', lambda: db_name) + return lambda dry_run=False: asyncio.run( + usermaintenance.cleanup_unverified_users(dry_run=dry_run)) + + +def test_deleting_an_unverified_account_removes_its_tokens_and_oauth_rows(db, maintenance): + db['users'].insert_one(_unverified('gone')) + db['users'].insert_one({**_unverified('stays'), 'email_verified': True}) + _rows_for(db, 'gone', 'gone') + _rows_for(db, 'stays', 'stays') + assert maintenance() == 1 + assert names(db, 'users') == ['stays'] + assert names(db, 'access_tokens') == ['stays-device', 'stays-oauth', 'stays-pat'] + assert names(db, 'oauth_codes') == ['stays-code'] + assert names(db, 'oauth_requests') == ['stays-request'] + assert names(db, 'oauth_used_refresh') == ['stays-used-1', 'stays-used-2'] + + +def test_a_dry_run_removes_nothing(db, maintenance): + db['users'].insert_one(_unverified('gone')) + _rows_for(db, 'gone', 'gone') + assert maintenance(dry_run=True) == 1 + assert len(names(db, 'access_tokens')) == 3 and len(names(db, 'oauth_codes')) == 1 + + +def test_a_run_with_no_account_to_delete_touches_no_token(db, maintenance): + db['users'].insert_one({**_unverified('stays'), 'email_verified': True}) + _rows_for(db, 'stays', 'stays') + assert maintenance() == 0 + assert len(names(db, 'access_tokens')) == 3 + + +def test_remove_tokens_of_reports_what_it_removed_and_ignores_no_ids(db): + _rows_for(db, 'gone', 'gone') + + async def run(ids): + from pymongo import AsyncMongoClient + client = AsyncMongoClient(db.client.address[0] + ':' + str(db.client.address[1])) + try: + return await remove_tokens_of(client[db.name], ids) + finally: + await client.close() + + assert asyncio.run(run([])) == {} + assert asyncio.run(run(['gone'])) == { + 'oauth_used_refresh': 2, 'oauth_codes': 1, 'oauth_requests': 1, 'access_tokens': 3} + assert asyncio.run(run(['gone'])) == { + 'oauth_used_refresh': 0, 'oauth_codes': 0, 'oauth_requests': 0, 'access_tokens': 0} + + +def test_the_account_deletion_in_the_dev_server_script_is_not_a_path_of_the_app(): + """The only code of the app that deletes accounts is usermaintenance.""" + import pathlib + root = pathlib.Path(__file__).resolve().parents[2] / 'src' / 'backend' + hits = [] + for path in root.rglob('*.py'): + text = path.read_text(encoding='utf-8', errors='ignore') + if ("mongodb_users.delete" in text or "['users'].delete" in text + or "users.delete_many" in text or "users.delete_one" in text): + hits.append(path.name) + assert hits == ['usermaintenance.py'] + + +# THE BACKFILL FROM THE DAILY MAINTENANCE RUN (0.6.3.1) ---------------------------- +def test_the_maintenance_run_gives_old_rows_their_purge_date(db, maintenance): + db['access_tokens'].insert_many([ + old_row('expired-row'), + old_row('revoked-row', expires=stamp_days(200), revoked=stamp_days(-10)), + old_row('has-date', purge_at=NOW + timedelta(days=5)), + ]) + # no account to delete: the backfill still runs + assert maintenance() == 0 + expired = db['access_tokens'].find_one({'_id': 'expired-row'}) + assert same(aware(expired['purge_at']), tk.parse_stamp(expired['expires']) + timedelta(days=90)) + revoked = db['access_tokens'].find_one({'_id': 'revoked-row'}) + assert same(aware(revoked['purge_at']), tk.parse_stamp(revoked['revoked']) + timedelta(days=90)) + assert same(aware(db['access_tokens'].find_one({'_id': 'has-date'})['purge_at']), + NOW + timedelta(days=5)) + + +def test_a_dry_run_writes_no_purge_date(db, maintenance): + db['access_tokens'].insert_one(old_row('expired-row')) + maintenance(dry_run=True) + assert 'purge_at' not in db['access_tokens'].find_one({'_id': 'expired-row'}) + + +def test_under_the_read_only_flag_the_job_waits_and_the_next_run_finishes_it( + db, maintenance, read_only_flag, mongod, db_name, monkeypatch): + db['access_tokens'].insert_one(old_row('expired-row')) + # the cleanup itself: the backfill alone does nothing under the flag + read_only_flag.on() + assert asyncio.run(usermaintenance.main()) == 0 # the job is skipped + assert 'purge_at' not in db['access_tokens'].find_one({'_id': 'expired-row'}) + assert maintenance() == 0 # called directly: the backfill skips + assert 'purge_at' not in db['access_tokens'].find_one({'_id': 'expired-row'}) + # the flag ends: the next run does it + read_only_flag.off() + assert asyncio.run(usermaintenance.main()) == 0 + assert db['access_tokens'].find_one({'_id': 'expired-row'})['purge_at'] is not None + + +def test_deleting_accounts_and_the_backfill_work_in_one_run(db, maintenance): + db['users'].insert_one(_unverified('gone')) + _rows_for(db, 'gone', 'gone') + db['access_tokens'].insert_one(old_row('left-over', owner_user_id='someone-else')) + assert maintenance() == 1 + assert names(db, 'access_tokens') == ['left-over'] + assert db['access_tokens'].find_one({'_id': 'left-over'})['purge_at'] is not None diff --git a/tests/api/test_token_failure_limit.py b/tests/api/test_token_failure_limit.py new file mode 100644 index 00000000..38060089 --- /dev/null +++ b/tests/api/test_token_failure_limit.py @@ -0,0 +1,160 @@ +""" +Failed access-token lookups are limited per client address (0.6.3.1): a token +that is unknown, revoked, expired or of the wrong scope counts for the address +in the shared store, and over the limit the answer is 429 with Retry-After +instead of 401. A valid token never counts and is never refused because of +what else comes from its address; no header at all counts nothing. The same +holds for /mcp. +""" + +from __future__ import annotations + +import pytest +from limits import parse + +from apps.catalog import access_tokens as tk +from apps.catalog import token_check +from oauth_helpers import bearer, connect, mcp_post, reset_address_counters + +LONG_AGO = '2000-01-01T00:00:00.000000Z' +UNKNOWN = 'csc_pat_' + 'A' * 43 +LIMIT = 4 + + +@pytest.fixture(autouse=True) +def tight_limit(monkeypatch): + """Four failures a minute, and clean counters before and after.""" + monkeypatch.setattr(token_check, '_FAILED', parse(f'{LIMIT}/minute')) + reset_address_counters() + yield + reset_address_counters() + + +@pytest.fixture +def owner(make_user, login): + user = make_user('tokenowner', role='admin') + return {**user, 'headers': bearer(login('tokenowner').json()['access_token'])} + + +def make_pat(api, owner): + return api.post('/auth/access-tokens', json={'name': 'script'}, + headers=owner['headers']).json() + + +def check(api, token, **headers): + return api.get('/auth/token-check', headers={**bearer(token), **headers}) + + +def fail_until_limited(api, token=UNKNOWN, limit=LIMIT): + """The failed lookups the limit allows, each a 401.""" + for _ in range(limit): + assert check(api, token).status_code == 401 + + +def test_over_the_limit_a_failed_lookup_is_a_429_with_retry_after(api): + fail_until_limited(api) + answer = check(api, UNKNOWN) + assert answer.status_code == 429 + assert 1 <= int(answer.headers['Retry-After']) <= 61 + assert 'failed attempts' in answer.json()['detail'] + assert 'WWW-Authenticate' not in answer.headers + + +def test_a_valid_token_is_not_affected_by_the_failures_of_its_address(api, owner): + pat = make_pat(api, owner)['token'] + fail_until_limited(api) + assert check(api, UNKNOWN).status_code == 429 + assert check(api, pat).status_code == 200 + assert check(api, pat).status_code == 200 + + +def test_valid_requests_never_count(api, owner): + pat = make_pat(api, owner)['token'] + for _ in range(LIMIT * 3): + assert check(api, pat).status_code == 200 + assert check(api, UNKNOWN).status_code == 401 # the first failure only + + +def test_a_request_without_a_token_counts_nothing(api): + for _ in range(LIMIT * 3): + assert api.get('/auth/token-check').status_code == 401 + assert check(api, UNKNOWN).status_code == 401 + + +def test_a_revoked_token_counts(api, owner): + row = make_pat(api, owner) + api.delete(f'/auth/access-tokens/{row["id"]}', headers=owner['headers']) + fail_until_limited(api, row['token']) + assert check(api, row['token']).status_code == 429 + + +def test_an_expired_token_counts(api, db, owner): + row = make_pat(api, owner) + db['access_tokens'].update_one({'_id': row['id']}, {'$set': {'expires': LONG_AGO}}) + fail_until_limited(api, row['token']) + assert check(api, row['token']).status_code == 429 + + +def test_a_token_of_the_wrong_scope_counts(api, db, owner): + row = make_pat(api, owner) + db['access_tokens'].update_one( + {'_id': row['id']}, {'$set': {'scopes': [tk.SCOPE_GH]}}) + fail_until_limited(api, row['token']) + assert check(api, row['token']).status_code == 429 + + +def test_a_jwt_sent_as_an_access_token_counts(api, owner): + jwt = owner['headers']['Authorization'].split(' ', 1)[1] + fail_until_limited(api, jwt) + assert check(api, jwt).status_code == 429 + + +def test_the_count_is_per_address(api, monkeypatch): + from apps.catalog.api import access_tokens as routes + monkeypatch.setattr(routes, 'client_ip', lambda request: request.headers['x-test-address']) + for _ in range(LIMIT): + assert check(api, UNKNOWN, **{'x-test-address': '203.0.113.1'}).status_code == 401 + assert check(api, UNKNOWN, **{'x-test-address': '203.0.113.1'}).status_code == 429 + assert check(api, UNKNOWN, **{'x-test-address': '203.0.113.2'}).status_code == 401 + + +def test_the_count_is_in_the_shared_store_not_in_the_worker(api): + """Another worker counts in the same store: its hits show here.""" + import limiter as lim + other_worker = lim.shared_limiter._limiter # what every worker uses + item = token_check._FAILED + for _ in range(LIMIT): + assert other_worker.hit(item, 'token-failures', 'testclient') + assert check(api, UNKNOWN).status_code == 429 + + +def test_the_limit_is_not_applied_to_a_wrong_sign_in(api): + """The sign-in has its own limit; failed tokens do not use it up.""" + fail_until_limited(api) + assert api.post('/auth/token', data={'username': 'nobody', 'password': 'x'}).status_code == 401 + + +# /mcp --------------------------------------------------------------------------- +def test_mcp_failures_are_limited_and_a_valid_connection_still_works(api, owner): + good = connect(api, owner['headers']) + for _ in range(LIMIT): + assert mcp_post(api, UNKNOWN, 'tools/list').status_code == 401 + limited = mcp_post(api, UNKNOWN, 'tools/list') + assert limited.status_code == 429 + assert int(limited.headers['Retry-After']) >= 1 + assert 'failed attempts' in limited.json()['error_description'] + assert mcp_post(api, good['access'], 'tools/list').status_code == 200 + + +def test_failures_on_mcp_and_on_the_rest_share_one_count(api): + for _ in range(LIMIT - 1): + assert mcp_post(api, UNKNOWN, 'tools/list').status_code == 401 + assert check(api, UNKNOWN).status_code == 401 # the 4th + assert check(api, UNKNOWN).status_code == 429 + assert mcp_post(api, UNKNOWN, 'tools/list').status_code == 429 + + +def test_mcp_without_a_token_counts_nothing(api): + for _ in range(LIMIT * 2): + assert mcp_post(api, None, 'tools/list').status_code == 401 + assert mcp_post(api, UNKNOWN, 'tools/list').status_code == 401 diff --git a/tests/catalog/test_proxies.py b/tests/catalog/test_proxies.py index 255e4569..8bd35a0b 100644 --- a/tests/catalog/test_proxies.py +++ b/tests/catalog/test_proxies.py @@ -458,3 +458,50 @@ def stats(p95, rms=None): lambda: stats(0.0, rms=50.0)) assert _accept('box', 'planar', stats(1e6), bbx) # the last fit always stands assert _accept('prism', 'linear', stats(10.0), bbx) # the linear rule is its own + + +# THE VERTEX LIMIT COUNTS THE HOLES (0.6.3.1) ----------------------------------- +def _round_hole_plate(hole_points): + from shapely.geometry import Point, box + plate = box(0, 0, 1000, 800) + hole = Point(500, 400).buffer(200, quad_segs=hole_points // 4) + return plate.difference(hole) + + +def test_the_vertex_count_of_an_outline_includes_its_holes(): + from apps.catalog.proxies.specs import outline_vertices + plate = _round_hole_plate(64) + assert len(plate.exterior.coords) == 5 + assert outline_vertices(plate) == 5 + len(plate.interiors[0].coords) + assert outline_vertices(plate) > 60 + + +def test_a_plate_with_a_fine_hole_is_simplified_until_exterior_and_hole_fit(): + from apps.catalog.proxies.specs import ( + OUTLINE_MAX_VERTICES, _simplify, outline_vertices) + plate = _round_hole_plate(1200) # the exterior alone is 5 + assert len(plate.exterior.coords) < OUTLINE_MAX_VERTICES + assert outline_vertices(plate) > OUTLINE_MAX_VERTICES + simple = _simplify(plate, 1000.0) + assert simple is not None + assert outline_vertices(simple) <= OUTLINE_MAX_VERTICES + assert len(simple.interiors) == 1 # the hole is still there + assert abs(simple.area - plate.area) / plate.area < 0.01 + + +def test_a_plate_within_the_limit_keeps_its_hole_and_its_shape(): + from apps.catalog.proxies.specs import OUTLINE_MAX_VERTICES, _simplify, outline_vertices + plate = _round_hole_plate(64) + simple = _simplify(plate, 1000.0) + assert simple is not None and outline_vertices(simple) <= OUTLINE_MAX_VERTICES + assert len(simple.interiors) == 1 + assert abs(simple.area - plate.area) / plate.area < 0.01 + + +def test_a_hole_that_cannot_be_simplified_gives_up_like_a_jagged_exterior(): + from shapely.geometry import LineString, box + from apps.catalog.proxies.specs import _simplify + zigzag = LineString([(1.6 * i + 100, 400 + 150.0 * (i % 2)) for i in range(600)]) + hole = zigzag.buffer(0.3, cap_style='flat', join_style='mitre') + plate = box(0, 0, 1200, 800).difference(hole) + assert _simplify(plate, 1000.0) is None diff --git a/tests/ci/test_release_prep.py b/tests/ci/test_release_prep.py new file mode 100644 index 00000000..1125e7c1 --- /dev/null +++ b/tests/ci/test_release_prep.py @@ -0,0 +1,173 @@ +"""The pure parts of ``invoke release-prep`` and ``bump-version --gh`` +(scripts/ci/release_prep.py): the CHANGELOG date, the empty-section rule, and +the rule that the Grasshopper client version moves only when it changes. +Standard library only (tests/ci runs with pytest alone).""" + +import datetime +import importlib.util +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +_spec = importlib.util.spec_from_file_location( + 'release_prep', ROOT / 'scripts' / 'ci' / 'release_prep.py') +rp = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(rp) + +LF = ('# Changelog\n\nintro\n\n## [0.6.3.1] - unreleased\n\n### Added\n\n- one thing\n\n' + '## [0.6.3.0] - 2026-10-21\n\n### Added\n\n- older\n') +CRLF = LF.replace('\n', '\r\n') +TODAY = datetime.date(2026, 10, 22) + + +# THE SECTION ------------------------------------------------------------------- +def test_the_section_of_a_version_is_found(): + body = rp.changelog_section(LF, '0.6.3.1') + assert '- one thing' in body and '- older' not in body + assert '- older' in rp.changelog_section(LF, '0.6.3.0') + + +def test_a_missing_section_is_none_and_counts_as_empty(): + assert rp.changelog_section(LF, '9.9.9.9') is None + assert rp.section_is_empty(None) + + +def test_a_version_is_not_a_prefix_match(): + text = '## [0.6.3.10] - unreleased\n\n- x\n' + assert rp.changelog_section(text, '0.6.3.1') is None + + +def test_a_section_with_only_headings_and_blanks_is_empty(): + assert rp.section_is_empty('\n\n### Added\n\n### Changed\n\n') + assert rp.section_is_empty('') + assert rp.section_is_empty(' \r\n\r\n') + + +def test_a_section_with_a_line_of_text_is_not_empty(): + assert not rp.section_is_empty('\n### Added\n\n- something\n') + assert not rp.section_is_empty('A plain paragraph.\n') + + +def test_the_last_section_of_the_file_is_read_to_the_end(): + text = '## [1.0.0.0] - unreleased\n\n- last line without a newline' + assert rp.changelog_section(text, '1.0.0.0').strip() == '- last line without a newline' + + +# THE DATE ---------------------------------------------------------------------- +def test_unreleased_becomes_the_date(): + out, state = rp.set_changelog_date(LF, '0.6.3.1', '2026-10-22') + assert state == 'dated' + assert '## [0.6.3.1] - 2026-10-22\n' in out and 'unreleased' not in out + assert out.replace('2026-10-22', 'unreleased', 1) == LF # nothing else moved + + +def test_crlf_is_kept_exactly(): + out, state = rp.set_changelog_date(CRLF, '0.6.3.1', '2026-10-22') + assert state == 'dated' + assert out == CRLF.replace('unreleased', '2026-10-22') + assert out.count('\r\n') == CRLF.count('\r\n') and '\n' not in out.replace('\r\n', '') + + +def test_only_the_heading_of_that_version_changes(): + text = LF + '\n## [0.6.2.0] - unreleased\n\n- an old unreleased heading\n' + out, _ = rp.set_changelog_date(text, '0.6.3.1', '2026-10-22') + assert '## [0.6.2.0] - unreleased' in out + + +def test_a_second_run_changes_nothing_and_says_so(): + once, _ = rp.set_changelog_date(LF, '0.6.3.1', '2026-10-22') + twice, state = rp.set_changelog_date(once, '0.6.3.1', '2026-10-22') + assert state == 'already' and twice == once + + +def test_a_date_that_is_set_is_not_overwritten_by_another_one(): + once, _ = rp.set_changelog_date(LF, '0.6.3.1', '2026-10-22') + again, state = rp.set_changelog_date(once, '0.6.3.1', '2026-11-30') + assert state == 'already' and again == once + assert rp.existing_date(again, '0.6.3.1') == '2026-10-22' + + +def test_a_missing_section_or_a_heading_without_unreleased_is_reported(): + assert rp.set_changelog_date(LF, '9.9.9.9', '2026-10-22')[1] == 'missing' + odd = '## [0.6.3.1] - soon\n\n- x\n' + assert rp.set_changelog_date(odd, '0.6.3.1', '2026-10-22') == (odd, 'missing') + + +def test_a_date_that_is_not_a_date_is_refused(): + for bad in ('2026-13-01', '22.10.2026', 'today', '2026-10-32'): + with pytest.raises(ValueError): + rp.set_changelog_date(LF, '0.6.3.1', bad) + + +def test_a_date_has_to_be_there_for_existing_date(): + assert rp.existing_date(LF, '0.6.3.1') is None + assert rp.existing_date(LF, '0.6.3.0') == '2026-10-21' + assert rp.existing_date(LF, '1.2.3.4') is None + + +# THE GRASSHOPPER CLIENT --------------------------------------------------------- +SESSION = ('"""\r\nDDU_CSC_Session\r\nVersion: 261020\r\n"""\r\n' + "CSC_CLIENT = 'gh-userobjects/0.6.3.0'\r\nCSC_GH_MINIMUM = '261020'\r\n") + + +def test_an_equal_client_version_changes_nothing_at_all(): + out, changed = rp.gh_client_update(SESSION, '0.6.3.0', TODAY) + assert changed is False and out == SESSION # the header date stays + + +def test_a_new_client_version_rewrites_the_client_and_the_header_date(): + out, changed = rp.gh_client_update(SESSION, '0.6.3.1', TODAY) + assert changed is True + assert "CSC_CLIENT = 'gh-userobjects/0.6.3.1'" in out + assert 'Version: 261022' in out and 'Version: 261020' not in out + assert "CSC_GH_MINIMUM = '261020'" in out # the minimum is its own + assert out.count('\r\n') == SESSION.count('\r\n') + assert out.replace('0.6.3.1', '0.6.3.0').replace('261022', '261020') == SESSION + + +def test_a_prerelease_version_is_a_different_version(): + out, changed = rp.gh_client_update(SESSION, '0.6.3.0-beta.1', TODAY) + assert changed and "gh-userobjects/0.6.3.0-beta.1" in out + + +def test_the_rule_applies_again_after_the_change(): + first, changed = rp.gh_client_update(SESSION, '0.6.3.1', TODAY) + again, changed_again = rp.gh_client_update(first, '0.6.3.1', datetime.date(2027, 1, 1)) + assert changed and not changed_again and again == first + + +def test_a_source_without_the_lines_is_an_error(): + with pytest.raises(ValueError): + rp.gh_client_update('no client here\r\n', '0.6.3.1', TODAY) + with pytest.raises(ValueError): + rp.gh_client_update("CSC_CLIENT = 'gh-userobjects/0.6.3.0'\r\n", '0.6.3.1', TODAY) + # an equal version needs no header + out, changed = rp.gh_client_update("CSC_CLIENT = 'gh-userobjects/0.6.3.1'\r\n", '0.6.3.1', TODAY) + assert changed is False + + +# THE NEXT STEPS NAME THEIR FILES (0.6.3.1) ------------------------------------- +def test_the_next_steps_name_the_files_and_never_add_everything(): + text = rp.next_steps('0.6.3.1') + assert 'git add -A' not in text and 'git add .' not in text + for name in ('CHANGELOG.md', '.zenodo.json', 'VERSION', 'src/backend/csc_version.py', + 'src/frontend/package.json', 'src/frontend/package-lock.json', 'README.md'): + assert name in text, name + assert 'git tag v0.6.3.1 && git push origin v0.6.3.1' in text + assert 'DDU_CSC_Session.py' in text # the --gh case is mentioned + + +def test_every_named_file_exists_in_the_repository(): + for name in rp.RELEASE_FILES + rp.GH_RELEASE_FILES: + assert (ROOT / name).is_file(), name + + +def test_the_named_files_are_the_ones_bump_version_writes(): + """bump-version rewrites exactly these (tasks.py): a file added there has + to be named here.""" + source = (ROOT / 'tasks.py').read_text(encoding='utf-8') + for needle in ("'VERSION'", "'csc_version.py'", "'package.json'", "'package-lock.json'", + "'README.md'", "'CHANGELOG.md'", "'DDU_CSC_Session.py'"): + assert needle in source, needle + assert '.zenodo.json' in ' '.join(rp.RELEASE_FILES) From e2a06c05416dd91ba8d2b9586cc2b046418d0f5f Mon Sep 17 00:00:00 2001 From: fstwn Date: Wed, 21 Oct 2026 20:09:39 +0200 Subject: [PATCH 3/4] fix: hardening 0.6.3.1 (token limits and clean-up, viewer depth and edges, file markers, paired rebound, release-prep, serve_dump, no user paths) --- .claude/launch.json | 4 +- CHANGELOG.md | 5 +- README.md | 6 +- scripts/dev/run_csc_python.cmd | 2 +- .../dev/{serve_migrated.py => serve_dump.py} | 83 ++++++++++++------- scripts/import/epfl_stacking/README.md | 2 +- scripts/import/recreate/README.md | 2 +- scripts/import/recreate/derive_local.py | 2 +- src/frontend/next.config.ts | 2 +- tasks.py | 58 ++++++++++--- ...ixtures.py => test_serve_dump_fixtures.py} | 4 +- 11 files changed, 117 insertions(+), 53 deletions(-) rename scripts/dev/{serve_migrated.py => serve_dump.py} (83%) rename tests/catalog/{test_serve_migrated_fixtures.py => test_serve_dump_fixtures.py} (92%) diff --git a/.claude/launch.json b/.claude/launch.json index a8539b66..ef306c52 100644 --- a/.claude/launch.json +++ b/.claude/launch.json @@ -6,7 +6,7 @@ "runtimeExecutable": "cmd", "runtimeArgs": [ "/c", - "set CSC_PUBLIC_API_URL=http://127.0.0.1:8791&& scripts\\dev\\run_csc_python.cmd -u scripts/dev/serve_migrated.py --dump latest --no-migrate --assets .dev/assets-261012 --port 8791 --mongo-port 27791" + "set CSC_PUBLIC_API_URL=http://127.0.0.1:8791&& scripts\\dev\\run_csc_python.cmd -u scripts/dev/serve_dump.py --dump latest --assets .dev/assets-261012 --port 8791 --mongo-port 27791" ], "port": 8791 }, @@ -35,7 +35,7 @@ "runtimeExecutable": "cmd", "runtimeArgs": [ "/c", - "scripts\\dev\\run_csc_python.cmd -m invoke dev-migrated" + "scripts\\dev\\run_csc_python.cmd -m invoke dev-local --migrate --dump 261001" ], "port": 8000 }, diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a5770cc..8ff912fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,10 @@ A hardening release after 0.6.3.0. vertex limit counts holes. - **Release tooling:** `invoke release-prep` sets the CHANGELOG date, writes `.zenodo.json` and runs the version checks in one step; `bump-version --gh` leaves the Session component alone - when its client version already matches. The local launch entries work without a user path. + when its client version already matches. The local launch entries and the old scripts work + without a user path; the local server script is now `scripts/dev/serve_dump.py` (`invoke + dev-local`; `invoke dev-migrated` still works and says so), migrating an old 0.5 dump only + with `--migrate`; it writes `.dev/dev-local-credentials-.txt` and `.dev/local-.json`. ### Security diff --git a/README.md b/README.md index 5816314e..ba62bfa6 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ UserObjects talk to production by default; to use a local backend, set the `Serv For browser checks open the web app as `http://127.0.0.1:`, not `localhost`: cookies do not separate by port, so a second dev server on another -port (a throwaway `invoke dev-migrated` with `NEXT_DIST_DIR` and +port (a throwaway `invoke dev-local` with `NEXT_DIST_DIR` and `NEXTAUTH_URL=http://127.0.0.1:`) would otherwise share its NextAuth session with the first. @@ -119,7 +119,9 @@ without a path of your account: `CSC_PYTHON` (a `python.exe` you name), the active environment when it is `csc`, the conda base from `CONDA_EXE`, the usual install folders under `%USERPROFILE%` and `%ProgramData%`, then `conda run -n csc`. The sign-in `dev-admin` has a new password at every start, written to -`.dev/dev-migrated-credentials-8791.txt`. A collection with no export file in +`.dev/dev-local-credentials-8791.txt`. The script is `scripts/dev/serve_dump.py` +(`invoke dev-local` runs it too); it loads a dump as it is, and `--migrate` +runs the 0.6 cutover steps on an old 0.5 dump. A collection with no export file in the dump is not loaded, and an export file of 0 bytes holds no documents. ### Tests diff --git a/scripts/dev/run_csc_python.cmd b/scripts/dev/run_csc_python.cmd index dd975576..b264a25a 100644 --- a/scripts/dev/run_csc_python.cmd +++ b/scripts/dev/run_csc_python.cmd @@ -1,7 +1,7 @@ @echo off rem Runs the Python of the "csc" conda environment with the given arguments: rem -rem scripts\dev\run_csc_python.cmd -u scripts/dev/serve_migrated.py --dump latest ... +rem scripts\dev\run_csc_python.cmd -u scripts/dev/serve_dump.py --dump latest ... rem rem The launch configurations of .claude/launch.json use it, so that no path of rem a user account stands in a committed file. The environment is looked for in diff --git a/scripts/dev/serve_migrated.py b/scripts/dev/serve_dump.py similarity index 83% rename from scripts/dev/serve_migrated.py rename to scripts/dev/serve_dump.py index 09c89d99..74727d8a 100644 --- a/scripts/dev/serve_migrated.py +++ b/scripts/dev/serve_dump.py @@ -1,13 +1,10 @@ -"""Serve a migrated copy of a local dump for frontend work (`invoke dev-migrated`). +"""Serve a local dump for frontend work (`invoke dev-local`). -Starts a throwaway ``mongod``, loads ``mongodb_collections_local/``, -runs the 0.6 migration, derives frame, shape class, proxies (with deviation -maps) and complexity with the geometry runner (steps 5 and 7 of the spec; -``--no-derive`` skips it, the web app then has no frames), builds the -component-map cache, adds a local admin account and serves the backend on it with ``src/backend/dev.env``. The asset -folders are only read: step 6c moves the gripper meshes out of a temporary -copy into a temporary ``SNAPSHOT_CAPTURE_DIR``. Your usual database is not -touched; everything is gone when the script stops (Ctrl+C). +Starts a throwaway ``mongod``, loads ``mongodb_collections_local/`` as it +is (a dump of the 0.6 data; the newest one by default), builds the +component-map cache, adds a local admin account and serves the backend on it +with ``src/backend/dev.env``. Your usual database is not touched; everything +is gone when the script stops (Ctrl+C). The database listens on a fixed port (``--mongo-port``, default 27018), so the frontend only needs its NextAuth database pointed there:: @@ -16,7 +13,11 @@ npm run dev # in src/frontend; FASTAPI_URL from .env.development.local The admin account (``dev-admin``) and its generated password are written to -``.dev/dev-migrated-credentials.txt`` (gitignored). +``.dev/dev-local-credentials.txt`` (``-`` after ``credentials`` for any +other backend port; gitignored), and the address of the database and the +backend to ``.dev/local.json`` (``local-.json``). Before the rename of +this script (it was ``serve_migrated.py``) the files were called +``dev-migrated-credentials*.txt`` and ``migrated*.json``. ``--test-accounts`` also creates one account per dataset role in one dataset (``dev-contrib``, ``dev-mod`` = moderator and contributor, ``dev-rev``, and @@ -30,12 +31,19 @@ whose name is six digits. A collection with no export file is not loaded, and an export file of 0 bytes loads as no documents. -``--no-migrate`` loads a dump that already is 0.6 data (a production dump -from after the cutover) as it is: no migration, no gripper copy, no derive -run. ``--assets DIR`` points every asset folder (meshes, previews, photos, -point clouds, proxies, capture, evidence) at the sub-folders of DIR, a copy of -the server's asset folder; they are read and written in place, so never name -the backup itself and run no runner with ``--sweep`` on it. +``--migrate`` is for an old 0.5 dump: it runs the 0.6 cutover steps on it +(gripper meshes out of a temporary copy of the asset folder into a temporary +``SNAPSHOT_CAPTURE_DIR``, the owner mapping of ``.dev/reattribute_06.json`` +when present), then derives frame, shape class, proxies (with deviation maps) +and complexity with the geometry runner (``--no-derive`` skips that, the web app +then has no frames). Without it the dump is loaded as it is: no migration, no +gripper copy, no derive run. ``--no-migrate`` is accepted and does nothing (it +is the default now), so that old command lines keep working. + +``--assets DIR`` points every asset folder (meshes, previews, photos, point +clouds, proxies, capture, evidence) at the sub-folders of DIR, a copy of the +server's asset folder; they are read and written in place, so never name the +backup itself and run no runner with ``--sweep`` on it. ``--existing`` serves a database that is already running (``--mongo-port``) again: no mongod, no dump, no migration, no accounts. The capture fixture @@ -64,6 +72,11 @@ _PATH_KEYS = ('_DIR', 'CLIENT_LOG_PATH') +# the temporary folder of a run; the sweep of stale ones finds the old prefix +# too for a while (the script was ``serve_migrated.py``) +TEMP_PREFIX = 'csc-dev-local-' +OLD_TEMP_PREFIX = 'csc-dev-migrated-' + def _dev_env() -> dict: env = {} @@ -151,7 +164,7 @@ def restore_capture_fixtures(snapshots, assets: Path, capture: Path): def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument('--dump', default='261001', + parser.add_argument('--dump', default='latest', help='a folder of mongodb_collections_local, or ' '"latest" for the newest one (six digits)') parser.add_argument('--port', type=int, default=8000, @@ -168,8 +181,13 @@ def main() -> int: help='create dev-contrib, dev-mod, dev-mod2 and ' 'dev-rev with their roles in --test-dataset, ' 'and dev-outsider with none') - parser.add_argument('--no-migrate', action='store_true', - help='the dump is already 0.6 data: load it as it is') + migration = parser.add_mutually_exclusive_group() + migration.add_argument('--migrate', action='store_true', + help='the dump is a 0.5 one: run the 0.6 cutover ' + 'steps on it (and derive, see --no-derive)') + migration.add_argument('--no-migrate', action='store_true', + help='does nothing: loading the dump as it is is ' + 'the default now') parser.add_argument('--assets', default=None, metavar='DIR', help='use the sub-folders of this asset folder copy ' 'for every asset directory') @@ -184,6 +202,9 @@ def main() -> int: parser.add_argument('--test-dataset', default='dbu_zirkus', help='dataset the test accounts are members of') args = parser.parse_args() + if args.no_migrate: + print('--no-migrate: the dump is loaded as it is by default now ' + '(use --migrate for a 0.5 dump)') if args.dump == 'latest': from support import latest_dump args.dump = latest_dump(str(_REPO / 'mongodb_collections_local')) @@ -208,7 +229,7 @@ def main() -> int: if keep_work: work = Path(args.work_dir) else: - work = Path(tempfile.mkdtemp(prefix='csc-dev-migrated-')) + work = Path(tempfile.mkdtemp(prefix=TEMP_PREFIX)) (work / 'owner.pid').write_text(str(os.getpid()), encoding='ascii') else: binary = find_mongod() @@ -216,8 +237,9 @@ def main() -> int: sys.exit('no mongod binary (install MongoDB Community Server)') server = ThrowawayMongod(binary, port=args.mongo_port).start() uri = server.uri - sweep_stale_dirs('csc-dev-migrated-') - work = Path(tempfile.mkdtemp(prefix='csc-dev-migrated-')) + for prefix in (TEMP_PREFIX, OLD_TEMP_PREFIX): + sweep_stale_dirs(prefix) + work = Path(tempfile.mkdtemp(prefix=TEMP_PREFIX)) (work / 'owner.pid').write_text(str(os.getpid()), encoding='ascii') env = _dev_env() if args.assets: @@ -245,7 +267,7 @@ def main() -> int: # those folders, never the asset folder itself meshes = work / 'meshes' copied = 0 - if not args.no_migrate: + if args.migrate: for snap in db['component_snapshots'].find( {'geometry.meshes.1': {'$exists': True}}, {'_id': 1}): source = assets / snap['_id'] / '1' @@ -257,16 +279,17 @@ def main() -> int: CUTOVER_STEPS) print(f'migrated to 0.6 ({copied} gripper meshes -> {capture})') else: - print('no migration: the dump is loaded as it is') + print('no migration: the dump is loaded as it is ' + '(--migrate for a 0.5 dump)') # the untracked owner mapping (decisions 8.24-8.27), when present mapping = dev / 'reattribute_06.json' - if mapping.is_file() and not args.no_migrate: + if mapping.is_file() and args.migrate: run(Context(db=db, files=False, log=lambda m: None, shared_mapping=json.loads( mapping.read_text(encoding='utf-8'))), ['11b']) print('step 11b applied (.dev/reattribute_06.json)') - if not args.no_derive and not args.no_migrate: + if not args.no_derive and args.migrate: from apps.catalog.geometry_runner import derive_and_store_sync from apps.catalog.geometry_stages import Env stages = [s for s in args.derive_stages.split(',') if s] @@ -301,13 +324,13 @@ def main() -> int: dev / f'dev-test-accounts{tag}.json') print(f'test accounts in {args.test_dataset}: {", ".join(names)} ' f'(passwords in .dev/dev-test-accounts{tag}.json)') - (dev / f'dev-migrated-credentials{tag}.txt').write_text( + (dev / f'dev-local-credentials{tag}.txt').write_text( f'dev-admin\n{password}\n', encoding='utf-8') - (dev / f'migrated{tag}.json').write_text(json.dumps( + (dev / f'local{tag}.json').write_text(json.dumps( {'mongo_uri': mongo_uri, 'backend': f'http://127.0.0.1:{args.port}' }), encoding='utf-8') print(f'mongo {mongo_uri}; login dev-admin ' - f'(password in .dev/dev-migrated-credentials{tag}.txt)') + f'(password in .dev/dev-local-credentials{tag}.txt)') # on every start, a restart too: the temporary capture folder is # gone with the last stop (step 6c's files were moved into it) restored, present, missing = restore_capture_fixtures( @@ -326,7 +349,7 @@ def main() -> int: # the backend refuses a secret under 32 bytes (0.6.1.1): a short one in # dev.env is replaced for this throwaway instance if len(env.get('JWT_SECRET', '').encode('utf-8')) < 32: - env['JWT_SECRET'] = 'dev-migrated-only-secret-at-least-32-bytes' + env['JWT_SECRET'] = 'dev-local-only-secret-at-least-32-bytes' env.setdefault('SMTP_PASSWORD', 'dev') os.environ.update(env) for key, value in env.items(): diff --git a/scripts/import/epfl_stacking/README.md b/scripts/import/epfl_stacking/README.md index 76420890..604bdbf0 100644 --- a/scripts/import/epfl_stacking/README.md +++ b/scripts/import/epfl_stacking/README.md @@ -123,7 +123,7 @@ output goes outside the repository or into `.dev/`; a path anywhere else in the `.dev/assets-261012`) and `frontend-local` (3000) in `.claude/launch.json`. Never point `--assets` at the original backup of the assets, and never run a runner with `--sweep` on it. 2. Fetch, prepare, plan; `run.py --dry-run`; `run.py --server http://127.0.0.1:8791 --username dev-admin` - (the password is in `.dev/dev-migrated-credentials-8791.txt`); run it again (it creates nothing). + (the password is in `.dev/dev-local-credentials-8791.txt`); run it again (it creates nothing). 3. `derive_local.py --attribute epfl_stacking`, then `spotcheck.py`. ## Production diff --git a/scripts/import/recreate/README.md b/scripts/import/recreate/README.md index 3edb5848..a4ac0381 100644 --- a/scripts/import/recreate/README.md +++ b/scripts/import/recreate/README.md @@ -128,7 +128,7 @@ Paths below are the defaults (`--data` = `.dev/open_datasets/recreate/zenodo-168 Never point `--assets` at the original backup of the assets, and never run a runner with `--sweep` on it. 2. Fetch, parse, compute geometry, plan, check (the order above). 3. `run.py --dry-run`, then `run.py --server http://127.0.0.1:8791 --username dev-admin` (the password of - the local admin account is in `.dev/dev-migrated-credentials-8791.txt`). + the local admin account is in `.dev/dev-local-credentials-8791.txt`). 4. `derive_local.py`, then `spotcheck.py --out OUT --username dev-admin`. ## Production diff --git a/scripts/import/recreate/derive_local.py b/scripts/import/recreate/derive_local.py index 409f5dd1..58c8b940 100644 --- a/scripts/import/recreate/derive_local.py +++ b/scripts/import/recreate/derive_local.py @@ -6,7 +6,7 @@ ``--attribute`` names the import: its components carry `attributes.` (default `recreate`; the EPFL stones use `epfl_stacking`). -Rehearsal only: the database is the throwaway mongod of scripts/dev/serve_migrated.py (any host that is not +Rehearsal only: the database is the throwaway mongod of scripts/dev/serve_dump.py (any host that is not this machine is refused), and the files go to the asset copy `--assets` of that instance, which must be inside `.dev/` (the tool writes into it: a copy, never the original asset folders). In production the cron of decision 8.136 does this (`main_geometry.py --due`). No sweep: nothing is deleted. Run in the `csc` diff --git a/src/frontend/next.config.ts b/src/frontend/next.config.ts index 04d3e458..659e3141 100644 --- a/src/frontend/next.config.ts +++ b/src/frontend/next.config.ts @@ -2,7 +2,7 @@ import type { NextConfig } from "next"; import { headerRules } from "./lib/securityHeaders"; const nextConfig: NextConfig = { - // A second dev server for browser checks (`invoke dev-migrated` plus a + // A second dev server for browser checks (`invoke dev-local` plus a // frontend on another port) needs a build folder of its own: Next allows // one dev server per folder. Unset, this is `.next` as always. distDir: process.env.NEXT_DIST_DIR || '.next', diff --git a/tasks.py b/tasks.py index 9acdb3cd..3026f8e4 100644 --- a/tasks.py +++ b/tasks.py @@ -23,7 +23,7 @@ def help(c): @task() -def gource(c, mode='overview', output_dir='viz'): +def gource(c, mode='overview', out='viz'): """ Create gource video in specified output directory. @@ -37,8 +37,8 @@ def gource(c, mode='overview', output_dir='viz'): repodir = os.path.normpath(os.path.dirname(__file__)) with chdir(repodir): # Create output directory if it doesn't exist - if not os.path.exists(output_dir): - os.makedirs(output_dir) + if not os.path.exists(out): + os.makedirs(out) # Gource visualization try: @@ -49,14 +49,14 @@ def gource(c, mode='overview', output_dir='viz'): "gource {0} -1920x1080 -f --multi-sampling -a 1 " "-s 1 --hide bloom,mouse,progress --camera-mode " "overview -r 60 -o {1}/{2}.ppm" - ).format(repodir, output_dir, mode) + ).format(repodir, out, mode) c.run(gource_cmd) else: # track mode gource_cmd = ( "gource {0} -1920x1080 -f --multi-sampling -a 1 " "-s 1 --hide bloom,mouse,progress --camera-mode " "track -r 60 -o {1}/{2}.ppm" - ).format(repodir, output_dir, mode) + ).format(repodir, out, mode) c.run(gource_cmd) except exceptions.UnexpectedExit: print("Gource is not installed or not in the current PATH! " @@ -71,10 +71,10 @@ def gource(c, mode='overview', output_dir='viz'): "{0}/{1}.ppm -vcodec libx264 -preset medium " "-pix_fmt yuv420p -crf 1 -threads 0 -bf 0 " "{0}/{1}.mp4" - ).format(output_dir, mode) + ).format(out, mode) c.run(ffmpeg_cmd) - os.remove(f"{output_dir}/{mode}.ppm") - print(f"Video saved to {output_dir}/{mode}.mp4") + os.remove(f"{out}/{mode}.ppm") + print(f"Video saved to {out}/{mode}.mp4") except exceptions.UnexpectedExit: print("FFmpeg is not installed or not in the current PATH! " "See https://ffmpeg.org/ for info on installation.") @@ -309,6 +309,30 @@ def rehearse(c, dump='261001', assets='', mapping=''): f'--dump {dump}{args}', pty=False) +@task(help={ + 'dump': 'dump folder name in mongodb_collections_local, or "latest" ' + '(default latest)', + 'port': 'backend port (default 8000)', + 'mongo_port': 'throwaway mongod port (default 27018)', + 'derive': 'with --migrate: geometry-runner stages to run (default ' + 'frame,shape_class,proxies,complexity); "none" skips them', + 'test_accounts': 'also create dev-contrib, dev-mod, dev-mod2, dev-rev ' + 'and dev-outsider (no role); passwords in ' + '.dev/dev-test-accounts.json, not printed', + 'test_dataset': 'dataset of the test accounts (default dbu_zirkus)', + 'migrate': 'the dump is a 0.5 one: run the 0.6 cutover steps on it ' + '(otherwise it is loaded as it is)', +}) +def dev_local(c, dump='latest', port=8000, mongo_port=27018, + derive='frame,shape_class,proxies,complexity', + test_accounts=False, test_dataset='dbu_zirkus', migrate=False): + """ + Serve a local dump (throwaway mongod) for frontend work. + """ + _serve_dump(c, dump, port, mongo_port, derive, test_accounts, + test_dataset, migrate) + + @task(help={ 'dump': 'dump folder name in mongodb_collections_local (default 261001)', 'port': 'backend port (default 8000)', @@ -324,13 +348,25 @@ def dev_migrated(c, dump='261001', port=8000, mongo_port=27018, derive='frame,shape_class,proxies,complexity', test_accounts=False, test_dataset='dbu_zirkus'): """ - Serve a migrated copy of a dump (throwaway mongod) for frontend work. + Deprecated: use "invoke dev-local --migrate" (same as before). """ - flags = ' --no-derive' if derive == 'none' else f' --derive-stages {derive}' + print('invoke dev-migrated is deprecated: use "invoke dev-local --migrate" ' + '(a dump that is 0.6 data already: "invoke dev-local")') + _serve_dump(c, dump, port, mongo_port, derive, test_accounts, + test_dataset, True) + + +def _serve_dump(c, dump, port, mongo_port, derive, test_accounts, + test_dataset, migrate): + flags = '' + if migrate: + flags += ' --migrate' + flags += (' --no-derive' if derive == 'none' + else f' --derive-stages {derive}') if test_accounts: flags += f' --test-accounts --test-dataset {test_dataset}' with chdir(REPO_DIR): - c.run(f'{sys.executable} scripts/dev/serve_migrated.py --dump {dump} ' + c.run(f'{sys.executable} scripts/dev/serve_dump.py --dump {dump} ' f'--port {port} --mongo-port {mongo_port}{flags}', pty=False) diff --git a/tests/catalog/test_serve_migrated_fixtures.py b/tests/catalog/test_serve_dump_fixtures.py similarity index 92% rename from tests/catalog/test_serve_migrated_fixtures.py rename to tests/catalog/test_serve_dump_fixtures.py index 3f96f1b2..874322f9 100644 --- a/tests/catalog/test_serve_migrated_fixtures.py +++ b/tests/catalog/test_serve_dump_fixtures.py @@ -1,4 +1,4 @@ -"""``scripts/dev/serve_migrated.py`` restores the capture fixture files on +"""``scripts/dev/serve_dump.py`` restores the capture fixture files on every start (P12 preparation): copied from the asset folder, never moved, and only where a snapshot records a fixture and the file is missing.""" @@ -9,7 +9,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[2] / 'scripts' / 'dev')) -from serve_migrated import restore_capture_fixtures # noqa: E402 +from serve_dump import restore_capture_fixtures # noqa: E402 WITH = {'_id': 'a1', 'capture': {'fixtures': [{'label': 'end_effector', 'file': 'capture/a1/fixtures/0.ply'}]}} From ee03a7b8a1be83ae1fdbde3fa22e71a245dd9da6 Mon Sep 17 00:00:00 2001 From: fstwn Date: Wed, 21 Oct 2026 20:10:17 +0200 Subject: [PATCH 4/4] chore: release prep --- .zenodo.json | 6 +++--- CHANGELOG.md | 2 +- README.md | 4 ++-- VERSION | 2 +- src/backend/csc_version.py | 2 +- src/frontend/package-lock.json | 4 ++-- src/frontend/package.json | 2 +- 7 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.zenodo.json b/.zenodo.json index 2b6c65fa..d9896f3a 100644 --- a/.zenodo.json +++ b/.zenodo.json @@ -1,8 +1,8 @@ { - "title": "Catalog of Second Chances (CSC) - Digital Database and Corresponding Interfaces for ReUse of Architectural Components (v0.6.3.0)", - "version": "0.6.3.0", + "title": "Catalog of Second Chances (CSC) - Digital Database and Corresponding Interfaces for ReUse of Architectural Components (v0.6.3.1)", + "version": "0.6.3.1", "upload_type": "software", - "description": "

The Catalog of Second Chances (CSC) is a prototypical digital component database. It catalogs digital representations of uniquely identifiable architectural components for reuse. Furthermore, it provides corresponding interfaces and tools to leverage them via a backend REST-API, a web-frontend as well as a interface for direct interaction within Rhino Grasshopper.

Changes in v0.6.3.0 (2026-10-21)

Added

  • Access tokens (decision 8.167): admins create personal access tokens in Settings (30 days, 90 days or one year; shown once, stored only as a hash, at most 10 active), renew and revoke them; admins see and revoke every token on a new admin page. A token reads the catalog only where its scope allows; GET /auth/token-check tests one. The ground for the MCP layer and the Grasshopper device sign-in of this release.
  • Using CSC from Claude (decision 8.168): an MCP endpoint at https://<api host>/mcp with nine read-only tools (search, components, versions, evidence, datasets, timeline, similar components). Add it as a custom connector in Claude; a browser window opens the consent page on the website, where an admin allows read access to the catalog as they see it in the web app. The connection renews itself for up to 90 days and can be revoked in the token list. Answers carry no e-mail addresses and mark everything that is not public.
  • Grasshopper signs in without a password (decision 8.168): the Session component has a SignIn button (Windows). It opens the browser, you allow \"Grasshopper on this computer\" on the website, and the sign-in is stored for your Windows user (encrypted with DPAPI, one per server), outside every Grasshopper file. Later runs sign in by themselves and renew their token; SignOut ends the sign-in on the computer and the server. Each sign-in appears in the token list in Settings, where it can be revoked.

Changed

  • Session component: new inputs SignIn and SignOut; Refresh is gone, because the sign-in renews itself. Connected Username and Password still win and stay the way to sign in on macOS; with them the component signs in again by itself when its 6-hour token runs out. Update the components with CSC_Update.
  • Viewer: a \"Piece opacity\" slider in the Overlay panel fades the mesh, proxy or point cloud so reinforcement and other overlays show through. The Mesh and Point cloud switches are labelled and have an \"Off\" choice instead of the \"Show mesh\" and \"Show point cloud\" checkboxes; \"Show edges\" moved to the overlays.

Security

  • A password change or reset ends every access token of the account: personal access tokens, Claude connections and Grasshopper sign-ins. Afterwards create a new token, allow Claude again and press SignIn again.

Server notes

  • CSC_PAT_ROLES (optional, default admin) names the roles that may create access tokens.
  • The new collections access_tokens, oauth_clients, oauth_requests, oauth_codes and oauth_used_refresh need the same database rights as the others; their indexes are created at the start.
  • Set CSC_PUBLIC_API_URL=https://<api host> (no path) in the backend environment: it turns on /mcp and the OAuth server and is the address the connector is given. It also becomes the base of the links in the JSON-LD export (if it was unset so far, those links took the request's own address). After the restart, check that https://<api host>/.well-known/oauth-protected-resource/mcp answers, then add the connector once in claude.ai.
  • The deploy installs the MCP Python SDK (mcp) and its dependencies (cryptography among them).
  • In read-only mode /mcp and the token renewal keep working; a code issued just before the switch can still be exchanged once. New consents are refused until the flag is gone.

Full history: CHANGELOG.md

", + "description": "

The Catalog of Second Chances (CSC) is a prototypical digital component database. It catalogs digital representations of uniquely identifiable architectural components for reuse. Furthermore, it provides corresponding interfaces and tools to leverage them via a backend REST-API, a web-frontend as well as a interface for direct interaction within Rhino Grasshopper.

Changes in v0.6.3.1 (2026-10-21)

A hardening release after 0.6.3.0.

Changed

  • Viewer: mesh edges stay crisp over the faces instead of breaking up at some angles; close zooms no longer cut a piece open, and small pieces open filling the view. A version switch while the mesh or the point cloud is Off loads nothing hidden.
  • Evidence: a core record links to its paired rebound record; a piece with several reinforcement layouts shows all of them.
  • Geometry: replacing the meshes or clouds of a draft also removes the files and markers of the dropped ones; a frame-only cache entry no longer carries descriptor data; the outline vertex limit counts holes.
  • Release tooling: invoke release-prep sets the CHANGELOG date, writes .zenodo.json and runs the version checks in one step; bump-version --gh leaves the Session component alone when its client version already matches. The local launch entries and the old scripts work without a user path; the local server script is now scripts/dev/serve_dump.py (invoke dev-local; invoke dev-migrated still works and says so), migrating an old 0.5 dump only with --migrate; it writes .dev/dev-local-credentials-<port>.txt and .dev/local-<port>.json.

Security

  • Failed access-token lookups are limited per address (429 with Retry-After); valid tokens are never counted.
  • Token clean-up: the tokens and connections of a deleted account are removed with it; revoked and expired tokens are deleted 90 days later.

Server notes

  • Existing token rows get their clean-up date from the daily user maintenance run (not while the read-only flag is set); no step needed.

Full history: CHANGELOG.md

", "creators": [ { "name": "Eschenbach, Max Benjamin", diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ff912fb..c682bc0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.6.3.1] - unreleased +## [0.6.3.1] - 2026-10-21 A hardening release after 0.6.3.0. diff --git a/README.md b/README.md index ba62bfa6..1b856b1f 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ framework ## Current Versions -- **CSC**: 0.6.3.0 --- backend, web frontend and Grasshopper interface are released together - under one version (tag `v0.6.3.0`); the single source is the `VERSION` file. +- **CSC**: 0.6.3.1 --- backend, web frontend and Grasshopper interface are released together + under one version (tag `v0.6.3.1`); the single source is the `VERSION` file. See `CHANGELOG.md` for release notes. diff --git a/VERSION b/VERSION index 1e40a508..a50a1dcf 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.6.3.0 +0.6.3.1 diff --git a/src/backend/csc_version.py b/src/backend/csc_version.py index b157596b..0dbc3e4f 100644 --- a/src/backend/csc_version.py +++ b/src/backend/csc_version.py @@ -6,7 +6,7 @@ against VERSION, package.json and the release tag by scripts/ci/check_version.py. """ -CSC_VERSION = '0.6.3.0' +CSC_VERSION = '0.6.3.1' def release_tag(version: str = CSC_VERSION) -> str: diff --git a/src/frontend/package-lock.json b/src/frontend/package-lock.json index fd9b26df..75f41bc5 100644 --- a/src/frontend/package-lock.json +++ b/src/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "csc-frontend", - "version": "0.6.3.0", + "version": "0.6.3.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "csc-frontend", - "version": "0.6.3.0", + "version": "0.6.3.1", "dependencies": { "@radix-ui/react-accordion": "^1.2.12", "@radix-ui/react-checkbox": "^1.3.3", diff --git a/src/frontend/package.json b/src/frontend/package.json index 2ef9222e..2e34b399 100644 --- a/src/frontend/package.json +++ b/src/frontend/package.json @@ -1,6 +1,6 @@ { "name": "csc-frontend", - "version": "0.6.3.0", + "version": "0.6.3.1", "private": true, "scripts": { "dev": "next dev",