Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,10 +36,20 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm test
working-directory: extensions/kit
- run: npm ci
working-directory: extensions/har-openapi
# The unit tests, and the quote example end to end: a run, this extension, `expectation docs`.
- run: npm test
working-directory: extensions/har-openapi
env:
EXPECTATION_E2E: "1"
# Coverage an app hands over as attachments, the extension, and `expectation docs`, end to end.
- run: cargo build -p expectation_cli -p expectation_remote
- run: npm ci
working-directory: extensions/code-coverage
- run: npm test
working-directory: extensions/code-coverage
env:
EXPECTATION_E2E: "1"
55 changes: 55 additions & 0 deletions book/src/evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,63 @@ branch of a JSON response becomes an outcome. It also reads a HAR made by any ot
it leaves unknown. `crates/harness/examples/quote` runs the whole pipeline, with a stand-in
quote service.

## Code coverage

Contract coverage asks which documented outcomes were seen. Code coverage asks which of the
program's own source ran, and links that to the test that ran it. It is a separate thing with a
separate section on the test's page, because it means something different.

The application collects it (Istanbul, or anything that writes LCOV) and hands it over as step
attachments, in JavaScript with `onAttachments`. Then
[`extensions/code-coverage`](https://github.com/ptescher/expectation/tree/main/extensions/code-coverage)
reads LCOV or Istanbul's `coverage-final.json` and writes `results/<runner>[@platform].code.tsv`
(`app.code.tsv` for the default runner):

```bash
expectation-run expectations --docs docs --attachments captures
npx expectation-code-coverage --results docs/results/app.tsv --attachments captures \
--revision "$(git rev-parse HEAD)" --scope "src/**, without tests" \
--report-url "https://coverage.example.com/{run}/{slug}/"
expectation docs expectations docs
```

Each test's page then has a "Code coverage" section: the source revision and scope that were
instrumented, the lines, branches and functions hit at the start and after each step, what each
step added, and a link to a full HTML report that is hosted somewhere else (`{run}` and `{slug}`
are filled in for each test). Raw coverage files stay in the attachments directory, outside the
manual.

What the section claims is limited on purpose:

- **It is observed execution.** It never changes a test's result or an expectation's verdict,
and it doesn't say anything was asserted. A line that ran in a failed test ran.
- **Missing is not uncovered.** A step with no capture, or with one that can't be read, is shown
as that. Zero lines hit and "no data" are different.
- **A step's part is only claimed when it can be.** Counters count from when the app started, so
a step's part is the difference from the capture before. If a counter goes down, the app
restarted, and nothing is attributed to that step. If a step's capture is missing, the next
difference is labelled as since the earlier step it is compared with. Anything else that ran in the same
window, such as background work, is in the numbers, and the page says so. With
`--counters per-step` the app empties its counters each time it hands them over.
- **It counts for its own run and test.** Like contract evidence, a capture is shown only with
the run it came from and a test that hasn't changed since.

Code coverage files are read the same way as the others: tab-separated, a few hundred characters
a field, 16 MiB at most, and the report link must be an `http` or `https` address. The format is
described at the top of
[`crates/story/src/codecov.rs`](https://github.com/ptescher/expectation/blob/main/crates/story/src/codecov.rs).
Expectation doesn't instrument the app: collecting coverage is the app's and the driver's job.

## Writing an extension

Extensions in JavaScript can start from
[`extensions/kit`](https://github.com/ptescher/expectation/tree/main/extensions/kit): it reads the
results file, finds the attached files (refusing any path that leaves the attachments directory,
links included) and writes records that can't break the format. Expectation reads every
extension's file the same way, whatever kind it is: the file for another platform is left out, a
file over 16 MiB isn't read, fields are cut, and evidence counts only for its own run and test as
it was.

An extension reads:

- the results file (`results/<runner>[@platform].tsv`), for the run id, each test's slug and
Expand Down
17 changes: 16 additions & 1 deletion crates/cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,7 @@ fn docs(
.map_or_else(Default::default, expectation_docs::Screenshots::parse),
reader,
shared: None,
code: &[],
};
if let Some(expectation_docs::Screenshots::Dir(d)) =
shots.as_deref().map(expectation_docs::Screenshots::parse)
Expand All @@ -264,6 +265,7 @@ fn docs(
};
// Every platform's tests and results, then the pages: each one's dropdown knows the others' pages.
let mut manuals = vec![];
let mut codes = vec![];
for name in &names {
let platform = name
.as_deref()
Expand Down Expand Up @@ -316,6 +318,18 @@ fn docs(
expectation_story::Coverage::judge(contract, &c.plans, &report)
})
.collect();
// How much of the program's source ran, from files an extension wrote next to the results.
let (code, errors) =
expectation_story::load_code_coverage_for(&results_dir, name.as_deref());
for e in errors {
eprintln!("warning: code coverage file {e}");
}
for c in &code {
for p in &c.problems {
eprintln!("warning: {}: {p}", c.file);
}
}
codes.push(code);
// Just some sections, for a preview: resolved in this platform's suite.
let partial = (!sections.is_empty()).then(|| {
let ids: Vec<_> = sections
Expand Down Expand Up @@ -370,6 +384,7 @@ fn docs(
partial: partial.as_ref(),
look: expectation_docs::Look {
shared: (i > 0 && names[i].is_some()).then_some((Path::new(out), 2)),
code: &codes[i],
..look
},
};
Expand Down Expand Up @@ -535,7 +550,7 @@ fn prune_screenshots(store: &str, dirs: &[&str], dry_run: bool) {
for entry in std::fs::read_dir(d).into_iter().flatten().flatten() {
let path = entry.path();
if path.extension().is_none_or(|e| e != "tsv")
|| expectation_story::is_coverage_file(&path)
|| expectation_story::sidecar::is_sidecar(&path, None)
{
continue;
}
Expand Down
111 changes: 109 additions & 2 deletions crates/docs/src/html.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,42 @@ use expectation_graph::Node;
use expectation_model::text::paragraphs;
use expectation_model::{ExpId, FeatureId, Keyword, PhraseBody, SectionId, StepText, Suite};
use expectation_story::{
Attribution, Because, Check, CheckResult, Completeness, Credit, Origin, Source, State, Status,
StepResult, Verdict,
Attribution, Basis, Because, CaptureState, Check, CheckResult, Completeness, Credit, Origin,
Source, State, Status, StepResult, Verdict,
};
use std::collections::HashMap;
use std::fmt::Write;

/// "2 lines, 1 branch, 0 functions".
pub(crate) fn counted(l: u64, b: u64, f: u64) -> String {
let pl = |n: u64, one: &str, many: &str| format!("{n} {}", if n == 1 { one } else { many });
format!(
"{}, {}, {}",
pl(l, "line", "lines"),
pl(b, "branch", "branches"),
pl(f, "function", "functions")
)
}

/// What a capture's "new" numbers say, in words.
fn new_since(c: &expectation_story::Capture) -> String {
let Some([l, b, f]) = c.new else {
return match c.basis {
Basis::Baseline => "the first capture: what ran before the test".into(),
Basis::Restart => "the program restarted, so nothing is attributed".into(),
_ => "can't be told".into(),
};
};
let what = counted(l, b, f);
match c.basis {
Basis::Step => what,
Basis::Since(from) => {
format!("{what}, since step {from} (the captures between are missing)")
}
_ => what,
}
}

pub fn esc(s: &str) -> String {
s.replace('&', "&amp;")
.replace('<', "&lt;")
Expand Down Expand Up @@ -1859,9 +1889,86 @@ impl Site<'_> {
o.push_str(&self.step_evidence(i, n + 1));
o.push_str("</section>");
}
o.push_str(&self.code_coverage(i));
self.page(Here::Test(i), &plan.title, &o)
}

/// Code coverage of the instrumented program while this test ran, apart from its result.
fn code_coverage(&self, plan: usize) -> String {
let mut o = String::new();
for (n, (cov, caps)) in self.d.code_coverage_of(plan).into_iter().enumerate() {
let id = if n == 0 {
"code-coverage".to_string()
} else {
format!("code-coverage-{}", n + 1)
};
let _ = write!(
o,
r#"<section class="step code-coverage" id="{id}"><h2>Code coverage</h2><p class="muted">Which of the program's own source ran while this test did. It is observed execution: it doesn't change the result above, and it doesn't show that anything was checked. Counts are running totals over the test; "new" is what a step added since the capture before it, when that can be told, and can include activity the test didn't cause.</p>"#
);
let mut about = vec![];
for (label, v) in [
("Source", &cov.name),
("Revision", &cov.revision),
("Scope", &cov.scope),
] {
if !v.is_empty() {
about.push(format!("{label}: <code>{}</code>", esc(v)));
}
}
if !about.is_empty() {
let _ = write!(o, "<p>{}</p>", about.join(" · "));
}
o.push_str("<table><thead><tr><th>Capture</th><th>Lines</th><th>Branches</th><th>Functions</th><th>New since</th></tr></thead><tbody>");
for c in &caps {
let at = if c.step == 0 {
r##"<a href="#step-0">Start</a>"##.to_string()
} else {
format!(r##"<a href="#step-{0}">Step {0}</a>"##, c.step)
};
match c.state {
CaptureState::Captured => {
let n = |k: expectation_story::Count| format!("{} of {}", k.hit, k.found);
let _ = write!(
o,
"<tr><td>{at}</td><td>{}</td><td>{}</td><td>{}</td><td>{}</td></tr>",
n(c.lines),
n(c.branches),
n(c.functions),
esc(&new_since(c))
);
}
other => {
let what = if other == CaptureState::Missing {
"No capture at this step"
} else {
"A capture that can't be used"
};
let note = if c.note.is_empty() {
String::new()
} else {
format!(": {}", esc(&c.note))
};
let _ = write!(
o,
r#"<tr><td>{at}</td><td colspan="4"><span class="badge warn">{what}</span>{note}. That isn't the same as nothing being covered.</td></tr>"#
);
}
}
}
o.push_str("</tbody></table>");
if let Some(url) = caps.first().and_then(|c| cov.report_for(&c.plan, &c.run)) {
let _ = write!(
o,
r#"<p><a href="{}" rel="noopener noreferrer">The full coverage report</a>, hosted separately.</p>"#,
esc(&url)
);
}
o.push_str("</section>");
}
o
}

/// The contract evidence that counts for a step: what it did, and the contract it was held against.
fn step_evidence(&self, plan: usize, step: usize) -> String {
let found = self.d.evidence_at(plan, step);
Expand Down
Loading
Loading