ci: check the Markdown the demo reads, on every change by sstriker · Pull Request #42 · apache/tooling-memory

sstriker · 2026-10-31T12:05:24Z

Summary

This change depends on #37. Merge #37 first. Until then, this diff also shows the commits of #37.

scripts/docs-only.sh lets a change to Markdown outside crates/ and taps/ skip the build and the tests. But the demo reads some of that Markdown:

scripts/seed-corpus.py seeds the demo record from docs/ and examples/decisions/.
golden/docs.toml, the recall suite of act 7, and evals/demo.toml name entries of that record as answers.
Acts 4, 7 and 10 ask about DR-19 and DR-22.
Act 11 prints the two lists at the top of README.md.

A rename of a heading, or an edit to a list, can break the demo, and no check finds it. #37 corrects seven golden answers and two eval tasks that broke in this way.

This change adds make corpus-check (scripts/corpus-check.sh), and a corpus job in CI that runs it. The job runs on every change, and also on a change to Markdown only. The check seeds a scratch record from $(DEMO_FROM), as make demo-ready does. It fails when:

a question in golden/docs.toml, golden/platform.toml or golden/acts.toml names no entry of the record;
the keywords of a question in the first two suites do not find its answer in the top eight;
an eval task names no entry of the record in answers, or has no answers;
the text of the entries that an eval task names does not match its expect. For asf-pressures, the check counts the data rows of the table (counts), and matches expect against that number;
act 11 does not print every item of the two lists in README.md;
the seed fails, or one write of the seed fails (seed-corpus.py --strict).

The check does not run a harness, so it does not show what a model answers.

Other changes:

Act 7 reads its recall suite from golden/platform.toml, not from text in the script. Its third answer was "Trust, review", a heading that no longer exists. Now it is "Trust and review".
golden/acts.toml holds the questions that acts 4, 7 and 10 ask about DR-19 and DR-22.
Each task in evals/demo.toml has answers. scripts/eval.py reads only id, prompt and expect, so it does not change.
scripts/demo/lib.sh takes the binary from AMEM when it is set, so the check can run act 11 with the binary it built.
AGENTS.md, scripts/docs-only.sh and a comment in ci.yml say that the corpus job runs on a change to Markdown only.

Tests

The check passes on this branch. I made each change below in a copy of the branch, and ran the check of the first version of this branch and the check of this version against it.

Change to the Markdown	First version	This version
`examples/decisions/DR-22-interchange-format.md` deleted	exit 0	exit 1: `golden/acts.toml` names no entry for the DR-22 question
`S2` changed to `S4` in DR-19	exit 0	exit 1: task `dr19-spike`, `expect` matches no text
the status of DR-19 changed to "Accepted"	exit 0	exit 1: task `dr19-decision`, `expect` matches no text
a seventh row added to part 10 of the seeding notes	exit 0	exit 1: task `asf-pressures`, `expect` matches no text
a file in `docs/` with a byte that is not UTF-8	exit 0 (the seed failed in a pipe)	exit 1: `UnicodeDecodeError`
bold removed from 8 of the 9 items of the first README list	exit 0	exit 1: "act 11: 1 and 14 items, of 9 and 14 in README.md"
the heading "1. One rule: the domain is the partition key" renamed	exit 1	exit 1: `golden/platform.toml` names no entry for that question
act 7's answer "Trust, review", as on `main`	not checked	exit 1: `golden/platform.toml` names no entry
part 10 renumbered to 11	exit 1	exit 0. The prompts name the part by its title, so the demo still works
a seventh table row without bold	not checked	the count is 7, so `asf-pressures` fails. The pattern before this version counted 6
one write of the seed fails, in a section that no suite names	exit 0	exit 1: "1 writes failed"

Earlier, the first version of this branch failed on main's golden suite, on a stale eval answer, on tasks with no answers, on a renamed README heading, and on an emptied list.

Checks

make corpus-check passes: 104 entries from 9 files. golden/docs.toml: keywords 15 of 15, question 15 of 15, stripped 14 of 15. golden/platform.toml: 3 of 3 for each strategy. Evals: 4 of 4 tasks hold their answer. Act 11: 9 and 14 items, of 9 and 14.
make check passes: 737 tests run, 737 passed.

Adversarial pass

A fresh agent attacked the first version of this branch. The second commit corrects what it found:

The check did not read act 7's recall suite, which was already broken. It did not check that the record holds DR-22.
An eval task could lose its answer from the text, and the check still passed.
A seed that crashed exited 0, because the seed ran in a pipe.
Act 11 could print one item of nine, and the check still passed.
A renumbered part failed the check, but the demo still worked.
A character outside the BMP in an answer gave a TOML error.
The temporary directory stayed when the run was stopped with a signal.
The check seeded from fixed directories, not from DEMO_FROM.
A comment in ci.yml said that every job below it skips on a change to Markdown only.

Two findings are not changed:

A section of less than 40 words joins the section before it, so its title goes away. The check then fails, but the demo can still find the text. A failure of this type tells the author to change answers in the same change.
With CARGO_TARGET_DIR set, the target runs a binary from target/. The other targets of the Makefile do the same. A change to all of them is a separate change.

Copilot then found that counts counted only rows with bold text, and that the seed went on after a failed write and exited 0. The last commit corrects both. No pass ran after the second or the last commit.

Posted by Claude on behalf of @sstriker

A change to docs/ or examples/decisions/ skips make check and CI, but those files are the demo's record. A renamed heading broke 6 of the 14 golden questions and 2 eval tasks, and no check saw it. scripts/corpus-check.sh seeds a scratch record as make demo-ready does. It fails when: - a golden question or an eval task names no entry in the record; - the keywords of a golden question do not find its answer in the top 8; - act 11 does not find its two lists in README.md, or a list is empty. The new CI job, corpus, runs make corpus-check on every change. Each eval task now names its answer in `answers`. Act 11 takes the binary from AMEM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2

The check now scores act 7's recall suite, which moves to golden/platform.toml, with the answer "Trust and review" in place of the old heading. golden/acts.toml names the DR-19 and DR-22 entries that acts 4, 7 and 10 ask about. The text of each eval task's entries must match its expect; for asf-pressures, the number of table rows must. Act 11 must print every item of the two README lists. The seed writes to a file, so a crash fails the check. The eval answers name parts by title, not by number. The check seeds from DEMO_FROM, keeps non-ASCII text as it is, and removes its directory on a signal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2

Copilot

🟡 Changes recommended

The table check miscounts rows based on bold formatting, and the corpus check can pass despite failed writes that leave incomplete records.

2 open issues

Count table rows without requiring bold text · New
Fail the seed check when any write fails · New

What changed in this PR

This PR adds a CI check for Markdown that supplies the demo record and README lists. It also includes corrections from prerequisite PR #37.

Changes:

Add make corpus-check and run it in CI, including on Markdown-only changes.
Check golden answers, keyword retrieval, eval answer text, and act 11 list output.
Correct stale demo expectations, documentation, command names, and diagnostic text.

File	Description
scripts/seed-corpus.py	Default to the `amem` command.
scripts/fetch-wslcsdk.test.sh	Correct the executable name in a comment.
scripts/fetch-wslcsdk.sh	Correct the executable name in usage text and comments.
scripts/docs-only.sh	Explain corpus checks in comments.
scripts/demo/lib.sh	Allow the `AMEM` binary override.
scripts/demo/7-platform.sh	Read the recall suite from TOML.
scripts/demo/10-moves.sh	Clarify that the export step writes a new note.
scripts/corpus-check.sh	Add corpus, eval, retrieval, and README checks.
README.md	Clarify gateway credentials and authentication modes.
PLAN.md	Distinguish historical measurements from current questions.
Makefile	Add the `corpus-check` target.
local/README.md	Correct launcher, gateway, handoff, and evaluation guidance.
images/harness/README.md	Correct image contents and binary location.
images/harness/Dockerfile	Correct image-description comments.
golden/platform.toml	Extract and correct act 7’s recall suite.
golden/docs.toml	Update stale questions, answers, and keywords.
golden/acts.toml	Add entry expectations for demo act questions.
evals/demo.toml	Add answer references and correct grading patterns.
docs/reference-stack.md	Separate planned architecture from current implementation.
docs/harness-containers.md	Correct Dependabot status.
docs/demo-narrative.md	Clarify historical results and export counts.
crates/wslc-api/src/dll.rs	Correct the executable name in DLL guidance.
crates/web/tests/reader_browser.rs	Correct the command name in test documentation.
crates/web/src/lib.rs	Correct launcher commands in documentation.
crates/sandbox/src/seatbelt.rs	Correct binary names in comments.
crates/sandbox/src/linux.rs	Correct relay command names in documentation.
crates/sandbox/src/lib.rs	Update harness documentation and generated comments.
crates/policy/src/lib.rs	Correct the init command in documentation.
crates/otel/Cargo.toml	Correct the binary name in package metadata.
crates/memory/tests/launch.rs	Correct command names in test comments.
crates/memory/src/wslc.rs	Clarify gateway diagnostics and command documentation.
crates/memory/src/wslc_api.rs	Correct command documentation and refresh guidance.
crates/memory/src/taps.rs	Correct tap command documentation and error guidance.
crates/memory/src/savings.rs	Correct the recall command in output.
crates/memory/src/main.rs	Correct command names in help, errors, and comments.
crates/memory/src/credentials.rs	Clarify gateway credential usage text.
crates/egress/src/lib.rs	Correct the usage command in documentation.
crates/client/src/lib.rs	Correct the serve command in documentation.
CONTRIBUTING.md	Correct the binary name and init example.
AGENTS.md	Document corpus checks and Markdown validation steps.
.github/workflows/image.yml	Correct the build-step name and explain binary staging.
.github/workflows/ci.yml	Add the corpus job without the docs-only skip condition.

_{Review effort: Balanced}

sstriker · 2026-11-01T07:21:28Z

+answers = ["What the Foundation taught the design"]
+# the number is not in the text: scripts/corpus-check.sh counts the lines of the answer
+# entry that match `counts`, and checks `expect` against that count
+counts = "^\\| \\*\\*"


Fixed in 1436c8e. counts now matches each table row that follows another row and is not the separator, so bold does not matter. On the table of part 10, the old pattern counted 6, and the new pattern counts 6. With a seventh row without bold, the old pattern counted 6 and the new pattern counts 7. With one row without bold, the old pattern counted 5 and the new pattern counts 6.

Posted by Claude on behalf of @sstriker

sstriker · 2026-11-01T07:21:31Z

+cd "$here"
+
+"$amem" --root "$root" init --owner human:demo:check --domain dom-demo >/dev/null
+python3 scripts/seed-corpus.py --root "$root" --memory "$amem" "$@" >"$work/seed.txt"


Fixed in 1436c8e. seed-corpus.py has a --strict option. With it, the script fails when a write fails. corpus-check.sh uses it. I ran the check with an amem that fails the write of one section that no suite names. Before the change, the check exited 0. Now it exits 1 with "1 writes failed".

Posted by Claude on behalf of @sstriker

Copilot found two gaps. The asf-pressures pattern counted only rows with bold text, so a seventh row without bold was not counted. Now it counts each row that follows another row and is not the separator. The seed went on after a failed write and exited 0, so the check could pass on a partial record. Now seed-corpus.py has --strict, and the check uses it. With one failed write, the check exited 0 before this change and exits 1 now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bhj4Ck4y5iwZUsjpgQx8A2

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

# Conflicts: # .github/workflows/ci.yml # Makefile

Brings in what landed since the last merge (#37, #38, #41, #42, #48 to #52, #54 to #56). No conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D5DyJNQBUKom7QzrdkqRHn

Brings in what landed since the last merge (#37, #38, #41, #42, #39, #48 to #52, #54 to #56). Two conflicts: - Makefile .PHONY: keep corpus-check from main, and drop reader, which this branch removes. - crates/web/static/reader/dm_reader_bg.wasm: #49 built it again on main. This branch deletes it, because crates/web/build.rs builds it from the sources, the change of #49 included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D5DyJNQBUKom7QzrdkqRHn

Brings in what landed since the last merge (#37, #38, #41, #42, #39, #48 to #52, #54 to #56). One conflict, in crates/server/src/lib.rs: #52 adds `last_stamp`, so that a write of this service is not dated before the one before it. This branch adds `Memory::for_request()`, a view that writes for the same session. The view now shares `last_stamp` with its session, as it shares the other state that is kept per session. New test: a_request_view_dates_its_writes_after_its_session_s_last_one. With a view that starts its own `last_stamp`, the write is dated at the clock, 30 s before the last stamp of the session, and the test fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D5DyJNQBUKom7QzrdkqRHn

sstriker and others added 2 commits October 31, 2026 13:03

Copilot AI balanced review requested due to automatic review settings October 31, 2026 12:05

sstriker mentioned this pull request Oct 31, 2026

fix: correct the facts that the ASD-STE100 rewrite left open #37

Merged

Copilot started reviewing on behalf of sstriker October 31, 2026 12:05

View session

Copilot AI reviewed Oct 31, 2026

View reviewed changes

sstriker requested a balanced review from Copilot October 31, 2026 16:45

Copilot AI reviewed Oct 31, 2026

sstriker mentioned this pull request Nov 1, 2026

docs: correct the incident account, and name MCP 2026-07-28 and A2A #48

Merged

sstriker added 2 commits November 2, 2026 23:32

Merge remote-tracking branch 'origin/main' into ci/corpus-check

93baa82

# Conflicts: # .github/workflows/ci.yml # Makefile

Merge remote-tracking branch 'origin/main' into ci/corpus-check

e205dc3

sstriker merged commit abef714 into main Nov 2, 2026
10 checks passed

sstriker deleted the ci/corpus-check branch November 2, 2026 23:06

Navigation Menu

Repository navigation

ci: check the Markdown the demo reads, on every change - #42

ci: check the Markdown the demo reads, on every change#42
sstriker merged 5 commits into
mainfrom
ci/corpus-check

sstriker commented Oct 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

sstriker Nov 1, 2026

Uh oh!

sstriker Nov 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sstriker commented Oct 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Checks

Adversarial pass

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

sstriker Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

sstriker Nov 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sstriker commented Oct 31, 2026 •

edited

Loading