Camera-fed room memory pipeline (watcher + labeler + SNN-native memory)

[w4ffl35]

Extension of the memory-system work in #22: an always-on, camera-fed system that watches a room, identifies what's in it, and remembers specific instances of objects it has seen before — using the same SNN-native one-shot memory module being scoped there.

Depends on #22. This is queued up for documentation now; #22 (digit-holdout memory POC) is the active work and should land first, since the memory module built there is reused here unchanged.

The pipeline (three stages, not one monolithic system)

1. Watcher (SNN, cheap, always running). Spikes on motion/change in the incoming feed rather than running full perception continuously — this is application [PT-W2] Encode-at-inference contract #3 from Explore: modular multi-region SNN architectures — candidate applications #22 ("always-on device that mostly sleeps"), applied as the front door to this pipeline. Natural fit for an event camera (DVS) feeding spikeforge's existing event pipeline (N-MNIST/DVS128/CIFAR10-DVS support) rather than converting a normal RGB feed to spikes frame-by-frame.
2. Labeler (conventional, runs only when the watcher flags something). An off-the-shelf object-detection/scene-understanding model identifies and tags what changed ("that's a mug," "that's a lamp"). Not built from scratch — a fixed, pretrained front end, same pattern as the face-embedding backbone discussed for Explore: modular multi-region SNN architectures — candidate applications #22's application [Architecture] Phased repo split: core library, deploy targets, dashboard #1.
3. Memory (SNN-native — the actual reused piece from Explore: modular multi-region SNN architectures — candidate applications #22). Once something is labeled, bind it: "have I seen this specific mug before?" Same one-shot associative memory mechanism scoped in Explore: modular multi-region SNN architectures — candidate applications #22, applied to real tagged objects instead of held-out digit classes.

Explicitly out of scope

Internet scraping for images of similar items — considered and dropped. It doesn't help the core claim (recognizing the same specific instance again doesn't need outside images of similar-but-different objects) and only marginally helps two narrower things (hardening a single weak capture via hard negatives, or human-facing metadata enrichment) that aren't part of this system's goal. Also carries ToS/legal risk around scraping vs. using a proper image-search API. Not worth the dependency for what it buys — revisit only if the goal shifts toward describing/contextualizing objects for a person, rather than the system remembering them itself.

Open questions for when this is picked up

• Real event camera (DVS) vs. simulated/converted RGB feed for the watcher stage — depends on available hardware.
• Which off-the-shelf labeler to use for stage 2, and how its output (label + bounding box/crop) maps to the input the SNN-native memory module expects (this shape needs to match whatever Explore: modular multi-region SNN architectures — candidate applications #22 lands on for the digit-holdout memory module's input format).
• How "novelty" is judged to decide watcher → labeler handoff (motion alone vs. some notion of "this doesn't match anything already tracked").

[w4ffl35]

Design doc landed in #27 (plans/memory_system_research.md §3): three stages (event-camera watcher reusing UC-3's architecture, a fixed off-the-shelf labeler, and #22's memory module unchanged as the readout), the open integration question on bridging a labeled crop into the memory module's input contract, and the open questions/out-of-scope items carried over from the original scoping. Design only, as scoped — no implementation yet.