Skip to content

feat(write-gate): reject machine output and session-notification noise (Gate 1b)#94

Open
RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:upstream-pr/quality-scorer-machine-noise-denylist
Open

feat(write-gate): reject machine output and session-notification noise (Gate 1b)#94
RobertSigmundsson wants to merge 1 commit into
acidkill:mainfrom
RobertSigmundsson:upstream-pr/quality-scorer-machine-noise-denylist

Conversation

@RobertSigmundsson

Copy link
Copy Markdown
Contributor

Problem

Passive auto-capture stores a lot of harness output as if it were knowledge. Measured over one day of
gate_decision false-accepts on a live brain: 180× "Session activity:", 30× <summary>,
20× "Background command", 19× <status>, 12× task-notification, plus tool-use-id,
*-caveat> and raw shell lines (ps -o …, echo …, curl -…). None of it is a memory. All of it
passes the length and quality-score gates because it is long, structured, and full of distinct tokens.

Change

A new Gate 1b in score_memory, running before the scoring heuristics:

def _is_machine_noise(content: str) -> bool:
    """Two signals: (a) a known auto-capture/shell noise prefix, or
    (b) a structural notification/tool tag that never appears in a real memory."""

A match returns QualityResult(score=0, quality="low", rejected=True, rejection_reason="Machine-noise rejected: …") — same shape as the existing rejection gates.

Why upstream benefits

The denylist is derived from measured false-accepts, not guesswork, and it targets exactly the content
this project's own hooks generate. Every entry is a structural marker (a tag or a fixed prefix), so the
false-positive surface on genuine prose is small.

Evidence

Check Result
cherry-pick onto origin/main clean
import surreal_memory OK
ruff check All checks passed!
ruff format --check 701 files already formatted
mypy src/ --ignore-missing-imports Success: no issues found in 369 source files
pytest -m "not stress" -n auto 6535 passed, 84 skipped, 1 xfailed — identical to baseline

Risks / notes for the reviewer

  • No new tests (suite unchanged at 6535). This is the weakest of the eight on that axis: it is a
    rejection gate with no test asserting either that the noise is rejected or that ordinary prose is not.
    Recommend requiring a test before merge — happy to add one.
  • Hardcoded list. The markers are Claude-Code-harness-specific (<system-reminder>,
    task-notification, tool-use-id). Reasonable for this project, but a reviewer may prefer it
    config-driven (write_gate.noise_patterns) so other harnesses can extend it without a code change.
  • A memory that legitimately quotes one of these tags (e.g. a note about debugging the hooks) would be
    rejected. Rare, and the rejection is explicit with the reason in the message, not silent.
  • The commit was reformatted with ruff format while preparing this branch (quote-style normalisation in
    _NOISE_TAG_RE); author preserved.

Empiryczny denylist z gate_decision false-accepts: hard-reject dla struktur
powiadomień/tagów które nigdy nie są wiedzą — <summary>/<status>/task-notification,
tool-use-id, *-caveat>, Monitor event, Background command, oraz prefiksów
Session activity:/ps -o/echo/curl. Zamyka lukę: enforce przepuszczał 89% szumu
przy score=5 (monitor-events, task-notif, echa shell). Zwalidowane fixture:
JUNK 8/8 reject, GOOD (z tagami/typem) 3/3 pass. + config auto_capture_min_score 5->6.

(cherry picked from commit 2bb52f47da177ca077b2239cac2d0966fda7b3ba)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant