a11ymeter

A Bayesian accessibility measurement tool. Instead of a pass/fail score, a11ymeter models belief in conformance for every (WCAG success criterion × component) pair as a BetaDistribution, and updates that belief as evidence comes in — automated scanner output, manual tester results, or both. Manual tests are treated as near-ground-truth and dominate the posterior when present; automated results are weighted by the scanning tool's known sensitivity/specificity rather than taken at face value.

This is an active, evolving research project, not a finished tool. The model, the reliability estimates, and the CLI surface are all still being shaped by real scans and real manual review — see CLAUDE.md for the full design rationale, known bug patterns, and what's still a placeholder.

Why Bayesian, why not just pass/fail

Automated scanners are precise but incomplete (they can't see everything a manual tester can), and manual testing is thorough but expensive and can't cover every page. a11ymeter combines both: each success criterion's conformance belief starts from a base-rate prior, gets nudged by automated evidence in proportion to how reliable that evidence actually is, and gets overridden by manual test results when they're available. The output is a distribution, not a single number — so a report can show how confident the tool actually is, not just what it currently believes.

Three-way evidence split

Not every signal means the same thing, so a11ymeter keeps them separate instead of blending them into one score:

• Conformance (primary, weakest-link) — the headline number. WCAG criteria are AND-composed: one failing component fails the whole criterion, so this rolls up via a weakest-link combination, not an average.
• Graded score (secondary, diagnostic) — a weighted average used to rank "what to fix next." Never shown as the headline number.
• Practice-hygiene (separate, non-normative) — signals like landmark and heading structure that matter for screen-reader navigability but don't map to a specific WCAG criterion. These never feed the conformance number; they roll up into their own named sub-scores instead.

See CLAUDE.md for the full modeling details, including how manual tests decay toward uninformative as tester confidence drops, and how the tiered reliability estimates were derived.

Requirements

• Wolfram Engine + wolframscript on PATH
• Node.js 18+ (for the axe-core/Puppeteer scanner)

Setup

npm install
chmod +x cli.wls

Usage

Run an automated scan against a URL. This creates a timestamped session under data/sessions/ with the scan evidence, a manual-review queue for anything the scanner couldn't determine on its own, and a running site tree of posteriors:

./cli.wls scan https://example.com

Deploy manual-review forms for the items the scan couldn't resolve (requires a Wolfram Cloud login):

./cli.wls review deploy <sessionDir> <page>

Pull down submitted review results, fold them into the session's site tree, and log them for calibration. Safe to re-run — already-synced submissions are skipped:

./cli.wls review sync <sessionDir>

Or resolve the same review queue entirely at the terminal, one item at a time, instead of via a deployed web form — prints full context (rule, description, page, selector, HTML) before each prompt, persists as it goes, and can be safely quit (q) and resumed later:

./cli.wls review answer <sessionDir>

Sample a batch of axe's own "violations"/"passes" verdicts for spot-check re-review, and resolve them at the terminal the same way — see Calibration for why this is a separate path from review answer:

./cli.wls review spotcheck sample <sessionDir> <sampleSize>
./cli.wls review spotcheck answer <sessionDir>

See how often spot-check testers agreed with axe's own automated verdict, broken down by reliability tier:

./cli.wls review calibration-report

Run the test suite:

wolframscript -file tests/TestUpdates.wl
wolframscript -file tests/TestRollup.wl
wolframscript -file tests/TestReport.wl
wolframscript -file tests/TestEvidence.wl

Project layout

a11ymeter/
├── CLAUDE.md               # full design rationale and modeling detail
├── cli.wls                 # command-line entry point
├── package.json            # Node deps for the axe-core scanner
├── src/
│   ├── Updates.wl          # Beta-Bernoulli soft updates, manual test intake
│   ├── Rollup.wl           # component -> page -> site aggregation
│   ├── Evidence.wl         # axe-core JSON -> evidence tuples
│   └── Report.wl           # manual-review form deployment/sync, reporting
├── data/
│   ├── Data.wl             # WCAG priors, rule metadata, reliability tiers
│   └── calibration-log.example.json  # schema example (real log is gitignored)
├── scripts/
│   └── axe-runner.js       # headless Puppeteer + axe-core scan
└── tests/

Calibration

Every resolved (automated observation, manual result) pair gets logged to data/calibration-log.json — this is real tester data and is gitignored, not part of this repo's history. See data/calibration-log.example.json for the schema. Reliability tiers in data/Data.wl are currently engineering estimates, not empirical calibration; this log is how they'll eventually be replaced with real numbers.

review answer/review deploy resolve items from manualReviewQueue — axe's "incomplete" results, the ones it declined to judge automatically. Those pairs can validate the conformance posterior for that specific component, but can never calibrate the reliability tiers themselves: there's no automated verdict in an "incomplete" result to check the manual result against. review spotcheck sample/review spotcheck answer fill that gap — they sample from axe's actual "violations"/"passes" verdicts instead, so each resulting calibration-log entry pairs a real automated verdict with an independent manual judgment (the tester is never shown which verdict axe gave — reviewItemAppearance never surfaces it). review calibration-report reads those pairs back and reports raw agreement rates per tier. Spot-check submissions are deliberately never folded into a site tree via applyManualTest — that node's automated evidence is already reflected there once, from the scan that produced it; doing so again would double-count the same observation under a different name. Agreement rates are diagnostic, not automatically applied — adjusting ruleReliabilityTiers in data/Data.wl from them is a deliberate, human-reviewed decision.

A note on accessibility of this project itself

Developed entirely from the terminal by a project owner who is blind and uses VoiceOver. No Mathematica notebook (.nb) files anywhere in this repo — all Wolfram Language code is plain-text .wl/.wls, run via wolframscript, with text/JSON/file output only.

License

MIT — see LICENSE.