test(server): check OpenAPI descriptions with the pulldown-cmark rustdoc link guard (#2330)

[cyberlife-coder]

Fixes #2330

What

velesdb-server checked its published OpenAPI descriptions for rustdoc link syntax with a hand-written raw-text scan (#2270). velesdb-memory had already replaced the same kind of scan with pulldown-cmark after #2261's 40 review rounds. The server now uses that guard. The guard is shared, not copied, and it stays out of velesdb-core.

• New crate velesdb-rustdoc-guard (publish = false). It starts from the guard in velesdb-memory/tests/support/rustdoc_link_guard.rs, moved with git mv; the review rounds below then corrected how it reads a bare label. Its JSON walk takes the keys to read (description, plus summary for OpenAPI) and escapes pointers per RFC 6901, as the server's did.
• Consumers. velesdb-memory and velesdb-server take the crate as a path-only dev-dependency with no version. Cargo strips that when they publish, so cargo publish --dry-run, the release list and version sync don't see it. Both manifests carry a comment on this. One consequence, stated in velesdb-memory's manifest: its packaged tests no longer compile outside the workspace. They already read docs/reference/mcp-tools.json, which the package does not ship, so they only ran from the repository before this change too.
• Shared pin. pulldown-cmark is pinned once in [workspace.dependencies]. velesdb-memory's runtime rewrite and the guard use the same version. Rounds 4 and 5 below change what that rewrite reads as a path, and the CHANGELOG states the change.
• Removed from velesdb-server: holds_rustdoc_link, brackets_a_path, target_start, is_url, collect_rustdoc_links and their four tests. The tables became the guard's own tests.

Red first

I compiled the server's old scanner on its own. It returned false for each of these, and the guard flags each one:

• see <crate::Point>.: an autolink to a path;
• see [SegmentInfo] and [optional].: bare item paths, which rustdoc 1.90 resolves or warns about;
• see [u64] and [Self].: [u64] resolves to the primitive.

The guard, measured against the old tables and against rustdoc

I ran the guard on all 55 strings in the server's tables. The old scan flagged three forms the guard did not, and all three are real rustdoc links: a label with backticks inside ([stream_traverse`()`], [vec`!`]), which rustdoc drops before reading the path; the bare [&] reference primitive; and a mailto::X link target, which is a path and not an address.

Review round 1 found more outside that table: [foo ()], [vec !], [vec !()], [!] and [!][], each of which rustdoc 1.90 links. It also found [struct @Foo] and [fn @ f], which velesdb-memory's rewrite rewrites and the guard passed, so "the guard flags at least what the rewrite rewrites" was false. names_an_item no longer handles these one by one. It now reads a label in the order rustdoc's preprocess_link does: backticks dropped, the item before a #, a one-word disambiguator and a call or macro suffix removed along with the spaces around them, then the path tested, with the bare !, () and & primitives accepted (round 3 found that rustdoc keeps () as prose). Against the previous guard, flags_each_link_form misses exactly those 8 new strings, and all of them pass now. For the three fixes from the table, a mutation removing each one makes the test fail on exactly its string. A spaced disambiguator is covered by explicit cases in velesdb-memory (a_spaced_disambiguator_is_rewritten_and_flagged). one_pass_is_final's pseudo-random mix can't draw a known kind before a spaced @, so no token was added there.

Review round 2 found two more gaps.

• A link hidden by a reference. In [a, b][](crate::Foo), a client renders the inline link [](crate::Foo). The guard, which accepts every reference so as to see what rustdoc might link, read [a, b][] as a collapsed reference and never saw that link. The old server scan did catch it, through its ][. The guard now reads each text twice: once as rustdoc does, and once as a client renders it, with no reference invented. What either reading flags is a link.
• Path characters. The guard allowed (){}@ anywhere in a path. rustdoc's should_ignore_link allows only :_<>, !*&;. So [f(x)] and [x()y] now pass, as rustdoc ignores them, and (){} only count as a suffix that gets stripped. Round 3 below found the labels it still read differently.

Each new test fails against the guard it fixes. flags_each_link_form misses [a, b][](crate::Foo) and passes_web_links_code_and_prose_brackets flags [f(x)] under the round-2 guard, and the spaced-disambiguator case fails under the round-1 guard. The reviewer's randomized check (300,000 labels of 1 to 7 tokens, asserting that the guard flags every text the rewrite rewrites) finds see [(){}][](crate::Foo)] end in the round-2 guard, and nothing in 36,450 rewritten texts in this one.

Review round 3 measured the guard against rustdoc 1.90 on 3,202 labels, and I repeated the measurement at a larger scale.

• Two misses. [!<u8>] and [::<u8>&] passed, and rustdoc links them to the never and reference primitives. The guard tested for a bare primitive before it stripped generics, and rustdoc strips generics first. rustdoc also drops the empty :: segments that stripping leaves, and counts generic depth with a sign, so [><f] reads as f.
• Labels rustdoc neither links nor warns about. [()], [!{}], [Result<(), u8>], [Vec<f32.5>], [Option<&'static str>] and [`()`] were flagged. rustdoc checks its path characters against the whole path, generics included, and keeps [()] as prose. It also drops a code span's backticks before it reads the label, where the guard took any single code span for a path.
• A false premise. rustdoc does not trim a kind before its @. It warns about [struct @Foo] and [fn @ f] and shows the brackets as written, and it trims only after the @ ([fn@ f]). The guard still flags the spaced forms: failing closed costs nothing where rustdoc warns anyway. Its docs and velesdb-memory's test comment now say so.

names_an_item now takes preprocess_link's steps in rustdoc's order:

1. a / anywhere means no item (added in round 4);
2. backticks;
3. the # fragment;
4. the disambiguator;
5. a suffix, only when removing it leaves something ([!] stays the never primitive);
6. the character set, over the whole path;
7. generics, then the empty segments they leave;
8. no space.

I measured it against cargo +1.90 doc (the rendered HTML and the JSON warnings) on two corpora: 22,621 labels (every sequence of 1 to 3 of 28 tokens, plus the named cases) and 20,000 random labels of 4 to 7 tokens. On both, 0 labels rustdoc links pass the guard, and the guard flags 0 labels that rustdoc neither links nor warns about. Every other flag is a label rustdoc warns about: an unresolved path, an unknown kind, or malformed generics. Two things are left out of the count, both by design: the mailto: link rustdoc renders from an e-mail autolink inside a label, which is publishable, and the images the guard flags.

Red first:

• Against the round-3 guard, the new cases fail. flags_each_link_form reports the guard misses "see [!<u8>].", and passes_web_links_code_and_prose_brackets gets ["[()]", "[ () ]", "[!{}]", "[()]"].
• Each of three mutations fails on its own case:
  • depth clamped at zero: [><f];
  • empty segments kept: [::<u8>&];
  • the character set checked after generics: [Result<(), u8>], [Vec<f32.5>] and [Option<&'static str>].

Review round 4 found two things.

• Guard ⊉ rewrite, again. Round 3 made the guard check path characters over the whole label, generics included, as rustdoc does. velesdb-memory's is_rustdoc_target still stripped generics first, so the rewrite kept rewriting labels that rustdoc and the guard leave alone: [Result<(), u8>], [Vec<f32.5>], [Option<&'static str>], [Vec<a/b>]. one_pass_is_final had no <, > or . token, so it could not see this. The rewrite now checks rustdoc's characters before it strips generics. This is a behaviour change in velesdb-memory's published-schema rewrite, stated in the CHANGELOG: such a label stays as written, as rustdoc shows it. The committed mcp-tools.json snapshot is unchanged.
• The / step. preprocess_link returns early on any /. The guard skipped that step and flagged [a#/] and [S0#a/b], which rustdoc neither links nor warns about.

Red first:

• With the three tokens added, one_pass_is_final fails against the round-4 rewrite with the guard misses "[<a\">crate::x!]> # b]b]".
• The new a_path_rustdoc_ignores_is_left_as_written fails with left: Some("see Result<(), u8>.").
• Removing the / step makes passes_web_links_code_and_prose_brackets flag ["[a#/]", "[S0#a/b]", "[Vec<u8>#x/y]"].

A third differential against rustdoc 1.90 found 0 misses and 0 silent over-flags. It ran on 20,000 random labels of 2 to 6 tokens, and adds /, a/b, #, ", |, =, + and é to the tokens.

Review round 5 found that round 4's rewrite fix was itself out of rustdoc's order.

• A regression. The rewrite checked its path characters before it dropped backticks, so it stopped rewriting [Vec<`u8`>], which rustdoc 1.90 links. It stripped only a single enclosing code span.
• A missing / step. The rewrite still rewrote [S0#a/b] to S0, which rustdoc shows as written and the guard now passes.

is_rustdoc_target now takes the guard's steps in rustdoc's order: it stops on a /, drops every backtick, then checks characters over the whole path. one_pass_is_final also draws / and #.

Red first: against the round-5 rewrite,

• the new code_inside_a_path_is_rewritten_and_flagged (renamed in round 6) fails with left: None;
• a_path_rustdoc_ignores_is_left_as_written, with the / cases, fails with left: Some("see a.");
• one_pass_is_final fails with the guard misses "`b`)\n\n[a# `/>\t][" .

All 853 velesdb-memory --features http --lib tests pass after the fix.

Review round 6 found one more undeclared rewrite change and a misplaced doc comment.

• The backtick change. Dropping every backtick also makes the rewrite remove links that develop left as written: [f`()`], [Foo] and [x](crate::`Foo`). rustdoc links all three, and the guard flags them. It also refuses [x](a#b/c), which develop rewrote to x. The CHANGELOG now states both. code_inside_a_path_is_rewritten_and_flagged pins the backtick cases, and round 7's a_target_rustdoc_reads_as_no_item_is_refused pins the / refusal. Put back develop's single code-span strip and that test fails with left: None, right: Some("see Vec<u8>.").
• The doc comment. The spaced-disambiguator paragraph is back on its own test.

Review round 7 found that the CHANGELOG, and the round-4 and round-6 sections above, scoped the rewrite change too narrowly: they covered labels and one inline link. is_rustdoc_target reads every target the same way, whether a label, an inline destination or a reference definition. So develop also rewrote [x](Foo<'a>), [x](Vec<f32.5>#a), [x](Vec<a/b>) and [x][y] with [y]: a#b/c, which HEAD refuses (left as written, flagged by the guard). HEAD also removes [x](crate::Foo), whose lone backtick rustdoc drops. The CHANGELOG now states the change once, for every target. The new a_target_rustdoc_reads_as_no_item_is_refusedpins the destination and definition cases; with develop'sschema_walks.rsit fails withleft: Some("see x."), right: None`.

Strings the old scan failed as a documented cost now pass, because rustdoc does not link them: code spans (`[x](crate::y)`, `&[Vec<f32>]`), web links whose text holds code or a path ([`Point`](https://…), [issue #2261](https://…)), and prose brackets ([#2261], [`asc`, `desc`]). The other way round, a bare [Point] or [sic], and a VelesQL pattern written outside a code span (-[:KNOWS]->), now fail. That's the cost of failing closed, and flags_prose_that_reads_as_an_item_path states it. The committed docs/openapi.json passes: its bracketed text all sits in code spans or code blocks.

Also

• deny.toml: licenses.clarify entry for the new crate, like every other workspace crate.
• CONTRIBUTING.md: crate map row. ARCHITECTURE.md describes the runtime layers, so a test-only crate is not listed there.
• CHANGELOG.md: Changed entry, stating what the guard now flags and passes compared with develop.

Gates (local, aarch64, replayed from ci.yml; from round 5 on, reduced to the changed crates to spare the machine; CI runs the whole workspace)

• fmt; clippy -D warnings -D clippy::pedantic with both of CI's workspace feature sets
• tests: velesdb-rustdoc-guard (4), velesdb-server --lib (default features, and --all-features for the OpenAPI tests), velesdb-memory --features http, and velesdb-memory --lib --no-default-features --features extract,persistence
• cargo publish -p velesdb-memory --dry-run --locked passes, which shows the path-only dev-dependency is stripped. velesdb-server's dry-run fails against the velesdb-core on crates.io, which does not have parse_search_mode yet. That has nothing to do with this change, and CI does not run it.
• cargo deny check licenses bans sources, rustdoc -D warnings on the new crate, and check-inline-tests, check-file-budgets, check-doc-freshness, check-feature-claims, check-ai-attribution and check-version-sync
• end to end: with /// … see [SegmentInfo]. added to TraversalResultItem's doc comment, test_openapi_descriptions_carry_no_rustdoc_link fails at /components/schemas/TraversalResultItem/description

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 25 complexity

Metric	Results
Complexity	25

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}