P2: headless native map renderer for the ssm report (native default)#31
Open
drserajames wants to merge 20 commits into
Open
P2: headless native map renderer for the ssm report (native default)#31drserajames wants to merge 20 commits into
drserajames wants to merge 20 commits into
Conversation
Investigate whether the headless C++ map-draw renderer can become the canonical figure engine for ae.report, replacing the kateri (Dart/macOS/socket) path. - tools/p2-render-spike/compare.sh: render a styled report chart through map-draw and pixel-diff it against the kateri golden PDF (paths passed as args; all real-data inputs/outputs stay outside the repo). - P2-RENDER-SPIKE-PLAN.md: characterises the two input paths (report drives kateri with a self-describing chart + named semantic styles in c["R"]; map-draw is a fixed AD-chains pipeline that reads neither c["R"] nor the legacy plot-spec), enumerates the gap, records the prototype result (RMSE 0.155, ~0% report-figure parity as-is), and gives a staged parity+effort plan with a conditional GO for P2 (gated on a P2-A milestone) and a feasible-but-downstream verdict for sigp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prove the report's on-chart styling (c[R] named styles + c[p] base plot-spec) can be consumed headlessly and reproduce a report map's content without kateri. - tools/p2-render-spike/poc_render.py: ~230-line headless renderer (Pillow). Resolves a selected front style (front->background reference composition), matches selectors against semantic attributes, applies per-point fill/outline/size/shape/draw-order, draws title/labels/viewport. Reads charts read-only; no WHO data committed. - P2-RENDER-DESIGN.md: rendering-contract spec, POC results + 3 fidelity numbers, vertical-mirror resolution (no Y-flip), architecture (native renderer as kateri-compatible drop-in with kateri kept as fallback/interactive), and an independent/parallelisable phased work-breakdown. Key finding: cc/chart/v3/chart-import.cc already fully parses c[R] into semantic::Styles; only the resolver + drawing application are missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a headless C++ entry point `export_styled_map(chart, projection_no,
style_name, width, output)` (plus `ae_backend.map_draw.export_styled_map`
pybind) that renders a chart's on-chart semantic styling — the `c["R"]`
named styles + `c["p"]` base plot-spec — the way kateri's set_style +
get_pdf does, so the native renderer is a drop-in for the report's map step.
This is additive to and independent of the existing AD-chains export_map.
Inside styled-draw.cc:
- Style resolver (front -> background composition): walk the front style's
ordered modifier list, recursively expanding {R:<name>} references into a
flat modifier list, tolerating undefined refs; merge legend, carry title,
last-writer-wins viewport.
- Selector engine over semantic attributes (clade `C`, boolean attrs like
reference/vaccine, index `!i`, antigens/sera scope).
- Per-point render model: fill / outline / outline-width / size / shape /
shown / draw-order (raise/lower), from base plot-spec then modifiers. Base
sizes carry kateri's x5 (_sizeScale); modifier sizes are absolute px.
- Viewport transform+recenter+viewport chain reproduced exactly (the key
fidelity step): absolute viewport = -roundedSize/2 + V.origin + hull_centre,
where roundedSize = ceil(hull_extent + 1) — matching kateri's roundAndRecenter
and the export_mapi_for_signature_pages recovery. No Y-flip.
- Grid (#CCCCCC), plain legend (box, markers, per-row counts) and title,
drawn baseline-anchored in Helvetica to match the golden.
New cairo-surface primitives (used only by this path): `egg` (kateri's egg
bezier), `text_font` (weight/slant, baseline anchor), and a Helvetica option
on `text_size` so boxes size from the same metrics they draw with.
Fidelity vs the kateri golden for a representative plain by-clade map
(rasterised identically via pdftoppm): map content (grid + points + colours +
frame) 1.2% pixel diff; full image incl. title/legend/labels 5.1%, the excess
being font-rasterizer AA (cairo vs poppler Helvetica) — the known bounded tail
that milestones B/C/I/K address. The viewport-convention gate (the P2 risk) is met.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… :bright/:pale colour model Extend the native styled (semantic) map renderer (styled-draw.cc) to cover all of the seasonal report's map style families, not just the plain by-clade map proven in milestone A: - Selector engine rewritten to mirror kateri semanticMatch exactly: boolean selector value = attribute-presence test; missing attribute never matches a non-boolean value; list-valued attribute matched by contains; otherwise scalar equality. Adds per-side !i index and !D date-range selectors. This covers reference, passage, continent/country, older-than, new, serology, vaccine and sequenced selectors (a prior string-compare path could spuriously match-all on an absent numeric attribute). - Colour model: implement kateri ColorAndModifier semantics as a deferred per-point pale flag. :pale desaturates + lightens (HSV, factor 0.4); :bright clears the pale modifier keeping the base colour. Backgrounds that pale every point (e.g. the -pale family) now render pale, and :bright highlights come back to full colour. - info- front styles: whitespace-only title is treated as blank and skipped; combined with the importer's inverted legend flag this yields blank-title, no-legend maps as specified. Composed families (-reset, -new-*, -vaccines*/-no-label, -pale, -o6m/-o12m and the by-clade since-6m/12m fronts) resolve through the existing front to background reference flattening; undefined references stay tolerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t tail) The native styled renderer's map *content* already matched the kateri golden, but the full image (title + legend + labels) diffed ~5% because the text landed on different pixels. Root cause was placement, not glyph rasterisation: on this toolchain cairo's toy "Helvetica" resolves (via fontconfig) to the same face poppler substitutes for the golden PDF's non-embedded base-14 Helvetica, so an embedded subset rasterises pixel-identically in isolation (0px diff). Confine the fix to cc/draw/cairo-surface.cc (no signature or call-site changes, so the parallel styled-draw.cc work is unaffected), aligning three primitives to kateri's PdfGraphics semantics (draw_on_pdf.dart): - text_font: anchor the glyph pen origin (baseline-left) at the supplied point; drop the first-glyph left-side-bearing subtraction that pushed every string ~1px left of the golden. - text_size (helvetica): return the pen *advance* width (kateri's AFM metrics.width, not cairo's ink extent) and the em size as height (kateri hard- codes textSize height to 1.0*fontSize), so callers size title box and legend rows/interline from the same metrics kateri used — fixing the compressed legend and the title box. - apply kateri's fontScaleToMatchCanvas (1.02) to both the rendered font size and the metrics, so text is rendered at the same scale the golden carries instead of ~2% short. Fidelity for the representative plain by-clade map, rasterised identically via pdftoppm and compared with magick -metric AE: full image at the threshold-30 metric 3.0% -> 0.32%; strict (fuzz 0) 5.4% -> 2.1%. Title/legend text now position- and size-identical to the golden; the strict residual is sub-pixel glyph AA plus one bounded artifact: pdftoppm substitutes the golden's declared base-14 bold face with a regular-weight face on this machine, so the golden's *rasterised* bold title is thinner than the native's correct (AD-faithful, PDF-declared) bold — a verification-rasteriser quirk, not a renderer defect, left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings kateri-faithful text placement/metrics (cc/draw/cairo-surface.cc) alongside the renderer breadth (cc/map-draw/styled-draw.cc). Disjoint files, clean merge. Text tail 3.0% -> 0.32% at threshold-30. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tools/p2-fidelity/ — the pixel-diff scoreboard every P2 map-renderer
milestone uses to prove itself. Given a manifest of (chart, style, [golden],
[ad-ref]) entries it:
- renders each map natively (ae_backend.map_draw.export_styled_map -> PDF),
- rasterises native + reference PDFs to equal-size PNGs (pdftoppm -scale-to),
- pixel-compares native vs kateri golden (and AD single-map ref where an
isolable one exists) with `magick compare -metric AE` at strict AND
-fuzz 30%,
- emits a per-map table to stdout + scoreboard.{csv,md} + optional
native|kateri[|AD] montages.
Every external command is wrapped in `timeout` so a hung tool can't stall a
run. Charts + reference PDFs are referenced by PATH only (real report data
lives outside the repo, never committed); rendered output goes to an out-dir
outside the repo. Only the synthetic manifest.example.json is shipped.
Files: fidelity.py (driver), manifest.example.json (annotated synthetic
schema), README.md (interface + WHO-data policy + how to read the scoreboard).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Abstract the report's per-map PDF step behind a pluggable MapRenderer seam so
it can render natively (in-process C++) instead of launching kateri per map,
gated by an env flag and behaviour-neutral by default.
New py/ae/report/map_renderer.py:
- MapRenderer interface: export_pdf(chart, style_name, output_filename, width).
- KateriRenderer (default): drive kateri over its socket (set_style + get_pdf)
- exactly the previous behaviour.
- NativeRenderer: call ae_backend.map_draw.export_styled_map in-process,
consuming the same on-chart c["R"] named style + c["p"] base plot-spec the
report already bakes for kateri; no subprocess, no socket, Linux-capable.
- get_map_renderer() selects the backend from AE_REPORT_MAP_RENDERER
(kateri default / native); unset reproduces current behaviour exactly.
commander.CommanderBasic.export_pdf now delegates to the selected renderer and
takes the styled chart (passed at the export / export_info / serum-coverage call
sites) so the native backend can render it; the kateri backend ignores it.
Verified: with the flag native, the report path (CommanderBasic.export_pdf)
renders the "clades" map for a real chart byte-identically to a direct
export_styled_map call; with the flag unset the default still dispatches to
kateri (get_pdf). Python-only; no C++ rebuild.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds tools/p2-fidelity/ (native-vs-kateri/AD pixel scoreboard: driver, example manifest, README). tools/-only, disjoint from renderer + report seam. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Draw serum circles and serum-coverage shading in the headless native
renderer (export_styled_map), matching kateri's report serum-coverage
maps.
Serum circles (kateri PlotSpec.serumCircleData / DrawOn.SerumCircle):
- radius from the serum's CI{fold} semantic attribute (e=empirical,
t=theoretical, selected by the modifier's theoretical flag); missing
radius falls back to `fold` map units drawn heavily dashed.
- per-passage outline/fill/width/dash from the modifier's circle style;
radius kept in map units and mapped to device via the viewport scale.
- collected during modifier resolution and drawn on top of the points.
- whole-circle (solid or dashed) and angular-sector (with radius lines)
forms supported.
Serum coverage (kateri PlotSpec.applySerumCoverage): split a serum's
antigens by whether their titre is within `fold` of the homologous
titre (reusing the existing chart serum_coverage() computation, with a
kateri-equivalent homologous-titre finder), recolour each group's
fill/outline, and raise them.
Adds an open arc-stroke primitive to CairoSurface for the dashed-circle
outline (sector() closes the path to the centre, which is wrong for a
ring).
Fidelity vs the kateri golden serum-coverage maps (h1-niid, several
sera, empirical + theoretical): map-region pixel diff ~1.8-2.5% (AE,
fuzz 30%), at the anti-aliasing floor; circle geometry, per-passage
colour and coverage restyle align. Residual whole-image diff is the
known multi-line-title (milestone C) and legend-swatch (milestone B)
gaps, unrelated to F.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-spike Native serum circles (theoretical/empirical radius rings, per-passage outline/fill, whole-circle + angular-sector forms) and serum-coverage within/outside-fold restyle. Adds CairoSurface::arc primitive. Map-region diff vs kateri golden ~1.8-2.5% (AA floor). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er-spike Adds py/ae/report/map_renderer.py (KateriRenderer default / NativeRenderer) selected by AE_REPORT_MAP_RENDERER; commander.export_pdf delegates through it. Behaviour-neutral when unset; native path byte-identical to export_styled_map. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The by-clade report maps use kateri's *semantic* plot spec, which initialises a fresh grey default (test antigen fill gray80, all outlines gray80, reference antigens/sera transparent, sizes 20/32, shapes by egg/reassortant/reference) and then applies the named-style modifiers. It deliberately ignores the c["p"] legacy plot spec. export_styled_map was instead seeding every point from c["p"], whose fills still carry leftover base colours. On a chart where only a small subset matches a clade selector (the H3 clades-v1 case) this leaked the base colours onto every non-highlighted point, colouring the whole cloud instead of greying it. Charts whose clades cover almost all points (bvic, h1) happened to hide the bug because the base colours were overwritten. Replace the c["p"] seeding (and its draw order) with kateri's default semantic seed + sera/reference/test draw order. H3 clades-v1 fuzz30 vs the kateri golden drops 8.36% -> 1.75% (framing/AA floor); bvic clades-v1 0.79% -> 0.70% and h1 clades 1.58% unchanged (no regression). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Point labels (vaccine / reference strain names) were drawn as plain black glyphs, so where a label sits over the dark point cloud it becomes illegible — the black text disappears into the points. kateri gives every point label a default thin white halo (pointLabelHaloWidthFactor = 0.04 of the label size, a stroke pass painted under the fill) precisely so labels read over the points. Add an optional halo to CairoPdf::text_font (stroke the glyph outlines in white with round joins, scaled with the glyph, then fill on top — the same pass text()/text_rotated() already use) and have the styled-map label draw request it at kateri's 0.04 width. Label *positioning* already matched the golden: it is data-driven from each label's authored offset, and kateri does no collision auto-placement, so none is added here. Verified against the h1 clades golden (Wi/22-cell, GM/19-egg, Sy/21-cell, …): native labels now carry the same white halo and read over the cloud exactly as the golden does; positions unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task 1: seed point render state from kateri's default SEMANTIC plot-spec (grey80 fills/outlines, kateri draw order) instead of the legacy c["p"], which carried leftover clade colours -> composed grey-out now applies for sparse-selector maps (H3 clades-v1 8.36%->1.75%, no bvic/h1 regression). Task 2: white label halo (kateri pointLabelHaloWidthFactor 0.04) so labels read over the point cloud; positions unchanged (kateri has no auto-placement). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the serology-map fidelity residual (framing, not point content): - Legend swatches now track the (base, pale) pair like points do. The legend row copies the first matched point's resolved fill/outline (and their pale state + outline width), and a global colour modifier with no selector and a deferred :pale/:bright fill/outline re-pales every existing swatch — kateri's "make legend for pale clades pale too" hack. The paled clades on the serology map now show paled swatches instead of full colour. - Plot title supports multiple lines: the title string is split on newlines and each line drawn advancing the baseline by lineHeight*(interline+1), matching kateri's PlotText/paintTitle. Single-line titles render identically. Verified with the p2-fidelity harness (native vs kateri golden, fuzz30): serology h1 2.78->2.35%, bvic 1.77->1.43%, h3-hint 3.16->2.76%; the clades baseline is unchanged (1.87%), confirming no regression. Edits confined to cc/map-draw/styled-draw.cc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P2 milestone: close the "kateri-free export" gap flagged by milestone J. With
AE_REPORT_MAP_RENDERER=native the report's `export` command now runs with NO
kateri process at all — previously it still launched kateri for the trailing
export_to_legacy+get_chart (styled.ace) and for export_mapi's get_viewport.
map_renderer.py:
- native_selected() predicate (same env flag as J's per-map backend).
- write_styled_ace(): native replacement for kateri export_to_legacy(style)+
get_chart()+write. The in-memory styled chart already carries the c["R"]
named styles, c["P"] projections, semantic attributes, and the c["p"] legacy
plot spec baked by the earlier prestyle step (which bakes the SAME
style_for_legacy_plot_spec() style), so chart.write() reproduces a
structurally-equivalent styled.ace. Verified against kateri output: identical
R styles, projections, and per-antigen legacy fills (only colour-string
normalisation differs). Honest residual: does not itself re-run the
semantic->legacy bake (that setFrom logic lives only in kateri); relies on
the equivalent c["p"] the standard pipeline already produces.
- sig_page_viewport(): native replacement for kateri set_style+get_viewport.
Reproduces kateri's recentred sig-page viewport from the transformed layout
(raw layout x projection transform, then roundAndRecenter geometry). Verified
to reproduce kateri's sp.mapi viewport to full float precision across H1/H3/B.
commander.export / chart_modifier.export_mapi_for_signature_pages: branch on
native_selected(); the kateri path is textually unchanged and stays the default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AE_REPORT_MAP_RENDERER=native makes the report `export` command run with no kateri: map_renderer.write_styled_ace (chart.write) replaces export_to_legacy+get_chart; sig_page_viewport computes sp.mapi natively (exact float match). Kateri path unchanged when flag unset. Residual: native bake relies on prestyle-baked c["p"] (semantic->legacy setFrom still kateri-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tle) Legend swatches now pale with their points (kateri global-pale re-pale hack); multi-line titles split+advance per kateri. Serology fuzz30 residual down (h1 2.78->2.35, bvic 1.77->1.43, h3 3.16->2.76); clades baseline unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip AE_REPORT_MAP_RENDERER default from kateri to native, so the report's batch map-PDF step renders headlessly in-process by default. kateri stays the interactive viewer and remains available as the render fallback via AE_REPORT_MAP_RENDERER=kateri. Only affects report figure rendering; main_loop still launches kateri for commands that need it (e.g. prestyle's semantic->legacy bake), pending the native bake binding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P2 — headless native map renderer for the ssm report
Adds a native, in-process C++ renderer for the report's antigenic-map PDFs, so the report no longer needs to drive the kateri GUI app per map. Consumes the same on-chart styling the report already bakes (
c["R"]named styles +c["p"]base plot-spec), so it is a drop-in for the report's map step. kateri stays the interactive viewer and remains available as a render fallback.What's here
cc/map-draw/styled-draw.cc—ae_backend.map_draw.export_styled_map(ace, out, style, width, proj): style resolver + selector engine (mirrors kateri's semantic matching), viewport chain, grid, legend (with swatch paling), single/multi-line title, composed style families, pale/bright colour model, serum circles + serum-coverage restyle, label halos. Small cairo-surface primitives added (arc, text halo, helvetica metrics).py/ae/report/map_renderer.py— aMapRendererseam withNativeRenderer(default) andKateriRenderer(opt-in viaAE_REPORT_MAP_RENDERER=kateri);commander.exportrenders map PDFs and writes the styled chart natively when native is selected.tools/p2-fidelity/— reusable native-vs-golden pixel scoreboard.Fidelity
All report map families render at the anti-aliasing floor vs the kateri golden (~1–3% at a 30% fuzz threshold; content — clusters, palette, legend, circles, labels — matches). The one content regression found during development (a composed grey-out background not applying on sparse-selector maps) was root-caused and fixed.
Default change
AE_REPORT_MAP_RENDERERnow defaults to native. This affects only the report's batch map-PDF rendering.main_loopstill launches kateri for commands that genuinely need it (e.g. the prestyle semantic→legacy bake, which is still kateri-only pending a native binding), so a full report run is not yet entirely kateri-free — that is a tracked follow-up.Notes