User-dataset collection + admission gatekeeper

[aperson30]

What

Collects users' CT scans + our segmentation into an in-house dataset — but only what's worth keeping, via an admission gatekeeper. Completes the data-saving feature and the "thing that decides what goes in and what doesn't."

Design principle: never touches the user

It runs on a daemon thread AFTER the result is already delivered, every failure is swallowed, and it's a no-op unless USER_DATASET_PATH is set. A user who uploads garbage still gets their result — the garbage just doesn't enter the dataset. Merging this changes nothing until it's switched on in the environment.

The gatekeeper (services/user_dataset.py) — four gates

1. Real CT? valid 3D volume, plausible dimensions, CT-like HU range (air near −1000 + tissue/bone present), not constant.
2. Duplicate? exact (sha256 of the file) and near-duplicate (downsampled, quantized perceptual hash) vs a registry — re-uploads that differ only in compression/metadata are caught.
3. Usable? our own segmentation is the judge — reject if it didn't find plausible organs (voxel + distinct-organ thresholds), which is what garbage/non-abdominal input produces.
4. Abuse? rolling 24h per-user / per-IP / global quotas + a file-size cap. A competitor dumping thousands of files hits the quota; identical floods hit dedup — either way they can't poison the dataset.

What accepted scans become (PanTS-mirroring layout)

UserData/
  image_only/USER_00000001/ct.nii.gz
  mask_only/USER_00000001/combined_labels.nii.gz
  mask_only/USER_00000001/segmentations/<organ>.nii.gz   # per-organ sublabels
  mask_only/USER_00000001/metadata.json                   # user, IP, model, date, hashes, stats
  registry.json        # case-id counter, fingerprints, rolling quota events
  rejections.jsonl     # audit trail — nothing is silently dropped

So it can sit beside PanTS/CancerVerse once a writable location is granted. Uses a file-based registry with atomic writes — no DB migration.

Wiring

• api_blueprint.py fires collect_user_scan_async(...) right after a job is marked completed; client IP is captured in the request context (the worker thread has none) for the per-IP quota.
• constants.py: documented USER_DATASET_PATH.

Tests / CI

• tests/unit/test_user_dataset.py: quotas (user/IP/global + pruning), dedup, registry round-trip, CT validity (accepts a real CT, rejects constant / non-CT / 2D), segmentation-quality gate, and end-to-end promotion (accepted → full layout; duplicate → logged, not stored).
• Added the test to the CI backend job (+nibabel to its deps). compileall covers the module too.

To switch on (deploy)

Set USER_DATASET_PATH to a writable staging dir (e.g. /home/visitor/UserData) and restart. Tune quotas via USER_DATASET_DAILY_PER_USER / _PER_IP / _GLOBAL, USER_DATASET_MAX_CT_BYTES, USER_DATASET_MIN_ORGAN_VOXELS.

Note: authored without server access (no VPN), so validated by CI rather than a live run — worth one manual smoke test (set the env, run one scan, confirm a USER_00000001 appears) before relying on it.