Skip to content

Remove two committed page dumps, and close the hole that let them through - #2

Merged
jehrr merged 1 commit into
mainfrom
fix-committed-dumps
Sep 14, 2026
Merged

jehrr merged 1 commit into
mainfrom
fix-committed-dumps

Conversation

@jehrr

@jehrr jehrr commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

The v0.1.0 squash merge committed live_results.page1 and live_results.page2 — 1.5 MB each, raw --dump-html output from a live run.

Content, read before deciding: each is one motors listing page as the site served it — 26 per-seller UUIDs, 12 copies of dubizzle's public Algolia search key, 2 Sentry public keys, ~820 of its own ad identifiers. All of it data the site ships to every visitor. Nothing of ours: no 2Captcha key, no proxy credential, no session cookie, no Bearer token, no email. A motors page carries no agent_profile, so no individual is named.

Why the guards missed it

  • .gitignore had live_results/ — the DIRECTORY. --dump-html live_results writes live_results.page1, a FILE.
  • ci_checks.py scanned six suffixes (.py .md .txt .yml .yaml .example). .page1 was not one of them, so the check ran, passed, and never opened them. Neither had it ever opened fixtures_generated.json, sample_output.json or sample_output.csv.json and .csv were not on the list either.

Fixes

  • Both files removed; .gitignore now covers live_results.* and *.pageN.
  • The working-tree scan reads everything git tracks, plus anything new that is not ignored, at any suffix, skipping only binaries. git ls-files --cached --others --exclude-standard — so a developer's own .env and captures stay invisible while a new file is caught before it is added.
  • A raw capture is refused by shape (*.pageN, *_debug.html, live_results.*, captures/) whether or not its content looks dangerous — the two that got through would have passed any content rule.
  • A new KEY_SHAPED_FIELD rule applies everywhere, generated data included. The bare-32-hex rule is deliberately not applied to the three generated data files, with the reason written down: this site emits 32-hex ids in five public contexts and the rule fired 221 times on correct data. Those files are covered by that rule plus make_fixtures.py's scrub and the suite's corpus scan.
  • --history-check now reports any capture that has ever been committed, with its size, and fails unless a decision is recorded — because a later commit cannot reach the blob.

Verified

Both new rules were checked by planting a violation and watching them fail, and the history rule by removing the recorded decision (rc=1) and putting it back (rc=0).

520 offline checks, pytest green, ci_checks --all and --history-check clean.

@jehrr
jehrr merged commit e01da35 into main Sep 14, 2026
7 checks passed
@jehrr
jehrr deleted the fix-committed-dumps branch September 14, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant