From 3edd8d1e5d3350af14d58fff5eed943d875bc945 Mon Sep 17 00:00:00 2001 From: Patrick Tescher Date: Fri, 13 Nov 2026 11:38:51 -0800 Subject: [PATCH] An API's manual shows each step's request as a curl command A driver can hand over the HTTP requests it made, with their responses (Driver::take_exchanges, Bindings::on_exchanges). The runner marks each with what it was for (setting up, the action, or a check), redacts it and keeps it in the results. The manual shows a step's action request as a curl command with the response it got, where a screenshot would be; the test page also has the requests that set up and checked it. Redaction keeps a short list of headers, turns API keys into $API_KEY (or a name per key), replaces secret-looking query, form and JSON fields, and swaps a loopback origin for $BASE_URL. A JSON body that doesn't parse isn't shown. The invoices example runs a suite against a stand-in API over HTTP, with admin and member keys and an exception only a member meets. Co-Authored-By: Claude Opus 5.5 --- README.md | 2 + book/src/SUMMARY.md | 1 + book/src/api.md | 150 +++ book/src/drivers.md | 8 + crates/docs/src/html.rs | 94 +- crates/docs/src/lib.rs | 41 +- crates/docs/src/manual.rs | 10 +- crates/docs/src/markdown.rs | 116 +- crates/harness/examples/invoices/api.rs | 420 ++++++ .../harness/examples/invoices/invoices.expect | 109 ++ crates/harness/examples/invoices/main.rs | 42 + crates/harness/src/bindings.rs | 19 +- crates/harness/src/driver.rs | 13 +- crates/harness/src/lib.rs | 2 +- crates/harness/src/run.rs | 31 +- crates/harness/tests/exchanges.rs | 177 +++ crates/story/src/exchange.rs | 1127 +++++++++++++++++ crates/story/src/lib.rs | 37 + crates/story/src/report.rs | 49 + site/build.sh | 6 +- site/llms.txt | 2 + 21 files changed, 2400 insertions(+), 56 deletions(-) create mode 100644 book/src/api.md create mode 100644 crates/harness/examples/invoices/api.rs create mode 100644 crates/harness/examples/invoices/invoices.expect create mode 100644 crates/harness/examples/invoices/main.rs create mode 100644 crates/harness/tests/exchanges.rs create mode 100644 crates/story/src/exchange.rs diff --git a/README.md b/README.md index 410e5f3..43b61f1 100644 --- a/README.md +++ b/README.md @@ -131,12 +131,14 @@ fn behaves_as_expected() { | --- | --- | | [turnstile](crates/harness/examples/turnstile.rs) | The smallest whole run: an inline suite, closures for the driver, one story and a generated test. | | [kettle](crates/harness/examples/kettle) | A suite across several files: two exceptions, four stories, three generated tests, a toy world with time in it, SVG screenshots and the generated manual. | +| [invoices](crates/harness/examples/invoices) | An API's manual: the driver calls a stand-in invoices API over HTTP and hands over each request, and the manual shows them as `curl` commands with their responses, where a screenshot would be. Admin and member API keys, and an exception only a member's request meets. | | [quote](crates/harness/examples/quote) | Contract coverage end to end: the driver hands over each step's HTTP requests as HAR, [the HAR and OpenAPI extension](extensions/har-openapi) holds them against the contract, and the manual shows what the tests saw. | ```bash cargo run -p expectation_harness --example turnstile cargo run -p expectation_harness --example kettle # docs in target/expectation-docs/kettle cargo run -p expectation_harness --example kettle -- out --bug # a seeded defect, to see a failure +cargo run -p expectation_harness --example invoices # an API's manual, in target/expectation-docs/invoices cargo run -p expectation_cli -- check crates/harness/examples/kettle # check it without running anything cargo test --workspace # the examples, as tests too git config core.hooksPath .githooks # check fmt and clippy before each commit diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index 5720cbe..b305067 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -27,6 +27,7 @@ - [Remote drivers](remote.md) - [External services](services.md) - [The generated manual](manual.md) +- [APIs: requests as evidence](api.md) - [Evidence and contract coverage](evidence.md) # Reference diff --git a/book/src/api.md b/book/src/api.md new file mode 100644 index 0000000..f71c7be --- /dev/null +++ b/book/src/api.md @@ -0,0 +1,150 @@ +# APIs: requests as evidence + +A suite can describe a system that has no screen: an HTTP API. The vocabulary, the +expectations, the stories and the checks work as they do for an app. Only the evidence is +different. Where an app's manual shows a screenshot of each step, an API's manual shows the +request the step sent, as a `curl` command, and the response it got. + +```sh +curl -X POST "$BASE_URL/v1/invoices/in_0002/void" \ + -H "Authorization: Bearer $MEMBER_API_KEY" +``` + +```http +HTTP/1.1 403 Forbidden +Content-Type: application/json + +{ + "error": { + "type": "forbidden", + "message": "Only an admin of the account can void an invoice." + } +} +``` + +Every example in the manual is a request a test made against the API, so it can't drift away +from what the API really does. [`crates/harness/examples/invoices`](https://github.com/ptescher/expectation/tree/main/crates/harness/examples/invoices) +runs the whole thing against a stand-in invoices API: + +```bash +cargo run -p expectation_harness --example invoices # docs in target/expectation-docs/invoices +cargo run -p expectation_harness --example invoices -- out --bug # members can void: a failure +``` + +## Writing the suite + +Write it about what callers do and what the API promises, as you would for an app. The events +are what a caller asks for ("the caller voids the invoice"), and the states are what a caller +can read back ("the invoice is open"). Who is calling is a state too, so the rules that depend +on it can be written as exceptions: + +```gherkin +Vocabulary: + State: the invoice is none | draft | open | void + Shows: void: the invoice's `status` is `void` + State: the caller is admin | member + Set: use the {value}'s API key + Event: the caller voids the invoice + Say: void the invoice + +Feature: Voiding + Expectation: Voiding an open invoice voids it + Given the invoice is open + When the caller voids the invoice + Then the invoice is void + + Expectation: Members can't void + Overrides: Voiding an open invoice voids it + Given the invoice is open + And the caller is member + When the caller voids the invoice + Then the invoice is open +``` + +`expectation check` finds the permission rule that conflicts with the rule it is an exception +to, as it would for any two expectations that promise different things in the same situation. + +## The driver + +The driver calls the API with whatever HTTP client you use, and keeps each request it sends +with the response it got as an `Exchange`. The runner takes them with `on_exchanges` (or +`Driver::take_exchanges`): + +```rust,ignore +use expectation_harness::{Bindings, Exchange}; + +let x = Exchange::new("POST", format!("{base}/v1/invoices/{id}/void")) + .header("Authorization", format!("Bearer {key}")) + .response(403) + .response_header("Content-Type", "application/json") + .response_body(body); + +Bindings::new(World::new()) + .observe("the invoice", World::status) // GET /v1/invoices/{id} + .on("the caller voids the invoice", World::void) + .on_exchanges(|w| std::mem::take(&mut w.log)) +``` + +Hand them over as they were sent, credentials and all: the runner redacts them (below). It asks +after arranging a test's start, after each action and after each step's checks, so it knows +what each request was for: + +| Role | When | Where the manual shows it | +| --- | --- | --- | +| `setup` | while the test's start was arranged | on the test's page, folded away | +| `action` | while the step's action ran | on the tutorial, the feature page and the test's page | +| `check` | while the step's Thens were read back | on the test's page, folded away | + +The same request sent again in a row, such as a check that polled until it held, is shown +once, with how many times it was sent. + +## What is kept + +Exchanges are kept in the results file (`results/api.tsv`), which is published with the +manual. So the runner keeps only what `RunOptions::redaction` lets through: + +- Only a few headers: `Accept`, `Content-Type`, `Idempotency-Key`, `If-Match`, `If-None-Match` + and `Prefer` on requests, and `Content-Type`, `ETag`, `Location` and `Retry-After` on + responses. Cookies and anything a gateway added are dropped. +- `Authorization` keeps its scheme, and the key becomes `$API_KEY`, or the name you give it in + `keys`. So does the value of an `X-Api-Key` or `Api-Key` header. +- Query parameters, form fields and JSON fields named like a secret (`password`, `token`, + `client_secret`, `api_key`, `cvc` and the like), and those you list in `fields`, read + `REDACTED`. +- A loopback address (`http://127.0.0.1:53211`) becomes `$BASE_URL`. Set `base_url` to show + another origin instead, such as `https://api.example.com`. +- JSON bodies are pretty-printed, with numbers kept as they were written. A JSON body that + doesn't parse can't be checked for secrets, so it is replaced with a note. A body over + `max_body` (16 KiB) is cut short. + +```rust,ignore +RunOptions { + redaction: Redaction { + keys: vec![ + (ADMIN_KEY.into(), "$ADMIN_API_KEY".into()), + (MEMBER_KEY.into(), "$MEMBER_API_KEY".into()), + ], + fields: vec!["email".into()], + ..Default::default() + }, + ..Default::default() +} +``` + +Name the keys when who is calling matters, so the reader can tell an admin's request from a +member's. A name is a `$` followed by capitals, digits and `_`. The `curl` commands leave those +for the shell to fill in, and escape any other `$`. + +Redaction is by name and by header, so it can't know that a field called `note` holds a +customer's address. Read what the manual shows before you publish it, and add such fields to +`fields`. + +## Limits + +- Bodies are text. Hand over a binary body as a note of what it was. +- The [remote drivers](remote.md) (Swift, JavaScript, the embedded Rust client) don't send + exchanges yet. +- A step with both a screenshot and requests shows both: an app that calls an API can show its + screen and its requests. +- Exchanges are for the manual. To hold the traffic against an OpenAPI document, capture it as + HAR too and use [the HAR and OpenAPI extension](evidence.md#the-har-and-openapi-extension). diff --git a/book/src/drivers.md b/book/src/drivers.md index a70f693..40f6683 100644 --- a/book/src/drivers.md +++ b/book/src/drivers.md @@ -36,6 +36,7 @@ Terms are matched the way steps are: ignoring case, extra spaces and a leading " | `on_settle(f)` | Letting the system settle after each action, before anything is observed. | | `on_advance(f)` | Letting a little time pass before the runner looks again. | | `on_screenshot(f)` | A picture of the system after each step: `Screenshot::png`, `jpeg` or `svg`. | +| `on_exchanges(f)` | The HTTP requests the system made since the last call, with their responses: what an API's manual shows instead of a screenshot. See [APIs](api.md). | | `needs(&[…])`, `fresh(&[…])` | The [external services](services.md) the tests need. | The values passed to `set` and `arrange` are `Value`s: `v.as_str()` for a named value, @@ -115,6 +116,12 @@ A driver can hand over files it captured during each step, such as a network log Extensions use them to report which outcomes of a contract the tests saw. See [Evidence and contract coverage](evidence.md). +## HTTP exchanges + +A driver for an API hands over the requests it sent and the responses it got, with +`Bindings::on_exchanges` or `Driver::take_exchanges`. The runner redacts them and keeps them in +the results, and the manual shows each step's request as a `curl` command. See [APIs](api.md). + ## The Driver trait For full control, implement `Driver` yourself. Only `set`, `perform` and `observe` are @@ -133,6 +140,7 @@ pub trait Driver { fn can_arrange(&self, var: &Variable, value: &Value) -> bool { true } fn settle(&mut self) {} fn advance(&mut self) -> Progress { Progress::Static } + fn take_exchanges(&mut self) -> Vec { vec![] } fn screenshot(&mut self) -> Result, String> { Ok(None) } fn supports(&self, need: Need<'_>) -> bool { true } } diff --git a/crates/docs/src/html.rs b/crates/docs/src/html.rs index cd3ae6f..7afee35 100644 --- a/crates/docs/src/html.rs +++ b/crates/docs/src/html.rs @@ -14,8 +14,8 @@ use expectation_graph::Node; use expectation_model::text::paragraphs; use expectation_model::{ExpId, FeatureId, Keyword, PhraseBody, SectionId, StepText, Suite}; use expectation_story::{ - Attribution, Basis, Because, CaptureState, Check, CheckResult, Completeness, Credit, Origin, - Source, State, Status, StepResult, Verdict, + Attribution, Basis, Because, CaptureState, Check, CheckResult, Completeness, Credit, Exchange, + Origin, Role, Source, State, Status, StepResult, Verdict, exchange, }; use std::collections::HashMap; use std::fmt::Write; @@ -1054,6 +1054,44 @@ impl Site<'_> { } } + /// Requests, each as a `curl` command with the response it got, where a screenshot would + /// be. The same request sent again in a row (a check that polled) is shown once. + fn exchanges(&self, xs: &[&Exchange], stale: bool, failed: bool) -> String { + let mut o = String::new(); + for (x, times) in crate::collapse(xs) { + let response = exchange::response(x); + let (status_line, rest) = response.split_once('\n').unwrap_or((&response, "")); + let class = match x.status { + 0 | 500.. => "s5", + 400..=499 => "s4", + _ => "s2", + }; + let mut notes = vec![]; + if times > 1 { + notes.push(format!("Sent {times} times.")); + } + if stale { + notes.push("From before this page changed.".into()); + } else if failed { + notes.push("What came back when this failed.".into()); + } + let caption = if notes.is_empty() { + String::new() + } else { + format!("
{}
", esc(¬es.join(" "))) + }; + let _ = write!( + o, + r#"
{}
{}{}{}
{caption}
"#, + esc(&exchange::curl(x)), + esc(status_line), + if rest.is_empty() { "" } else { "\n" }, + esc(rest), + ); + } + o + } + /// Verdict counts for a set of expectations: "12 verified · 3 not supported yet". fn tally(&self, exps: &[ExpId]) -> String { let mut counts: Vec<(&str, &str, usize)> = vec![]; @@ -1404,8 +1442,13 @@ impl Site<'_> { let _ = write!(o, r#"

{}

"#, self.prose(&how)); o.push_str(&self.exception_notes(e)); match self.f.shot_of(self.d, e) { - Some((_, _, shot)) => { - o.push_str(&self.figure(Some(&shot), Status::Passed, false, false, &how)) + Some((p, step, shot)) => { + if let Some(shot) = &shot { + o.push_str(&self.figure(Some(shot), Status::Passed, false, false, &how)); + } + let r = crate::step_result(self.d, p, step); + let xs = r.map(|r| crate::actions(&r.exchanges)).unwrap_or_default(); + o.push_str(&self.exchanges(&xs, false, false)); } None => { let status = self @@ -1481,7 +1524,11 @@ impl Site<'_> { } else { &step.seen }; - o.push_str(&self.figure(step.shot.as_deref(), step.status, step.stale, false, alt)); + if step.shot.is_some() || step.exchanges.is_empty() { + o.push_str(&self.figure(step.shot.as_deref(), step.status, step.stale, false, alt)); + } + let xs: Vec<&Exchange> = step.exchanges.iter().collect(); + o.push_str(&self.exchanges(&xs, step.stale, step.status == Status::Failed)); o.push_str(""); } o.push_str(""); @@ -2223,13 +2270,29 @@ impl Site<'_> { plan_status: Status, ) -> String { let status = result.map_or(plan_status, |r| r.status); - let mut o = self.figure( - result.and_then(|r| r.screenshot.as_deref()), - status, - result.is_some_and(|r| r.stale), - true, - "", - ); + let stale = result.is_some_and(|r| r.stale); + let shot = result.and_then(|r| r.screenshot.as_deref()); + let all: &[Exchange] = result.map_or(&[], |r| &r.exchanges); + let mut o = if shot.is_some() || all.is_empty() { + self.figure(shot, status, stale, true, "") + } else { + String::new() + }; + o.push_str(&self.exchanges(&crate::actions(all), stale, status == Status::Failed)); + for (role, summary) in [ + (Role::Setup, "Requests that set it up"), + (Role::Check, "Requests that checked it"), + ] { + let xs: Vec<&Exchange> = all.iter().filter(|x| x.role == role).collect(); + if !xs.is_empty() { + let _ = write!( + o, + r#"
{summary} ({}){}
"#, + xs.len(), + self.exchanges(&xs, stale, false) + ); + } + } if let Some(e) = result .and_then(|r| r.error.as_ref()) .filter(|_| status == Status::Failed) @@ -2933,6 +2996,13 @@ section.exp:first-of-type { border-top: 0; } figure { margin: 14px 0; max-width: 870px; padding: var(--bezel-pad); background: var(--bezel); border: 1px solid var(--bezel-edge); border-radius: var(--radius-panel); } figure img { display: block; width: 100%; aspect-ratio: 16 / 9; object-fit: cover; border-radius: var(--radius-screen); background: var(--screen); } figcaption { color: var(--muted); font-size: 12px; padding: 7px 4px 2px; } +/* A request and its response: two screens in one bezel. */ +figure.exchange pre.http { margin: 0; padding: 12px 16px; overflow-x: auto; background: var(--screen); color: var(--text); border-radius: var(--radius-screen); font: 13px/1.55 var(--font-mono); white-space: pre; } +figure.exchange pre.http + pre.http { margin-top: var(--bezel-pad); } +figure.exchange pre.response .status { font-weight: 700; color: var(--ok); } +figure.exchange pre.response.s4 .status, figure.exchange pre.response.s5 .status { color: var(--warning); } +details.requests { margin: 10px 0; } +details.requests > summary { cursor: pointer; color: var(--muted); font-size: 13px; } /* A missing screenshot is a gap to note, not something to look at: keep it small. */ figure.placeholder { max-width: 360px; } dl.meta { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; font-size: 15px; margin: 12px 0 0; } diff --git a/crates/docs/src/lib.rs b/crates/docs/src/lib.rs index f72c715..9d5c1b6 100644 --- a/crates/docs/src/lib.rs +++ b/crates/docs/src/lib.rs @@ -33,7 +33,9 @@ pub(crate) const CONTINUED: &str = "This run started where the test that ran bef use expectation_graph::Analysis; use expectation_model::{ExpId, Feature, StepText, Suite, VarKind, Variable}; -use expectation_story::{Because, Plan, Report, Status, Verdict, coverage}; +use expectation_story::{ + Because, Exchange, Plan, Report, Role, Status, StepResult, Verdict, coverage, +}; pub use links::broken_links; use std::path::{Path, PathBuf}; @@ -658,7 +660,10 @@ impl Facts { /// The best picture of `e` in action: the screenshot after a step whose checks for it /// passed, else after any step that exercises it. Returns (plan, step) with the path. - fn shot_of(&self, d: &Docs, e: ExpId) -> Option<(usize, Option, String)> { + /// + /// For an API, a step's requests show it as a screenshot would: a step with action requests + /// and no screenshot counts, and the screenshot is then `None`. + fn shot_of(&self, d: &Docs, e: ExpId) -> Option<(usize, Option, Option)> { let report = d.report?; let mut fallback = None; for &(p, step) in &self.coverage[e.0 as usize] { @@ -667,7 +672,10 @@ impl Facts { None => (&result.start, &d.plans[p].start_checks), Some(s) => (&result.steps[s], &d.plans[p].steps[s].checks), }; - let Some(shot) = &r.screenshot else { continue }; + if r.screenshot.is_none() && actions(&r.exchanges).is_empty() { + continue; + } + let shot = &r.screenshot; let verified = checks .iter() .zip(&r.checks) @@ -681,6 +689,33 @@ impl Facts { } } +/// The result of step `step` (`None` for the start) of plan `p`, if it ran. +fn step_result<'d>(d: &Docs<'d>, p: usize, step: Option) -> Option<&'d StepResult> { + let r = d.report?.results.get(p)?; + match step { + None => Some(&r.start), + Some(s) => r.steps.get(s), + } +} + +/// Requests with the same request sent again in a row counted instead of repeated: a check +/// that polled until it held asked the same thing many times. +fn collapse<'a>(xs: &[&'a Exchange]) -> Vec<(&'a Exchange, usize)> { + let mut out: Vec<(&Exchange, usize)> = vec![]; + for &x in xs { + match out.last_mut() { + Some((last, n)) if **last == *x => *n += 1, + _ => out.push((x, 1)), + } + } + out +} + +/// The requests that were a step's action: what a reading page shows of it. +fn actions(xs: &[Exchange]) -> Vec<&Exchange> { + xs.iter().filter(|x| x.role == Role::Action).collect() +} + /// `to` relative to `from`, both directories: "../expectations". fn relative(from: &Path, to: &Path) -> Option { let (from, to) = (from.canonicalize().ok()?, to.canonicalize().ok()?); diff --git a/crates/docs/src/manual.rs b/crates/docs/src/manual.rs index ad4e0b3..e9cd8d0 100644 --- a/crates/docs/src/manual.rs +++ b/crates/docs/src/manual.rs @@ -6,7 +6,7 @@ use crate::{Docs, Facts, voice}; use expectation_graph::ConflictKind; use expectation_model::text::{join_and, paragraphs}; use expectation_model::{ExpId, FeatureId, Said, SectionId, StepText, Suite, VarId}; -use expectation_story::{Origin, Plan, Status}; +use expectation_story::{Exchange, Origin, Plan, Status}; /// A section's "Before you begin". pub struct Before { @@ -176,6 +176,8 @@ pub struct Step { /// What you'll see: "The tool is PEN." pub seen: String, pub shot: Option, + /// The requests the step's actions made, for an API: shown as a `curl` command each. + pub exchanges: Vec, pub status: Status, /// The picture, if any, is from before the page changed. pub stale: bool, @@ -278,6 +280,12 @@ pub fn tutorial(d: &Docs, f: &Facts, i: usize) -> Option { doing: voice::instruction(s, &whens), seen: outcome(s, plan, &group), shot: last.and_then(|r| r.screenshot.clone()), + exchanges: group + .iter() + .filter_map(|&n| result.and_then(|r| r.steps.get(n))) + .flat_map(|r| crate::actions(&r.exchanges)) + .cloned() + .collect(), status: last.map_or(f.status[i], |r| r.status), stale: last.is_some_and(|r| r.stale), }); diff --git a/crates/docs/src/markdown.rs b/crates/docs/src/markdown.rs index 6cdf9cb..29e8d9e 100644 --- a/crates/docs/src/markdown.rs +++ b/crates/docs/src/markdown.rs @@ -11,7 +11,8 @@ use expectation_model::text::paragraphs; use expectation_model::{ExpId, PhraseBody, SectionId, StepText, Suite}; use expectation_story::{Attribution, Credit, State}; use expectation_story::{ - Because, Check, CheckResult, ControlRow, Origin, Plan, Status, StepResult, Verdict, + Because, Check, CheckResult, ControlRow, Exchange, Origin, Plan, Role, Status, StepResult, + Verdict, exchange, }; use std::fmt::Write; @@ -256,6 +257,34 @@ fn image( } } +/// Requests, each as a `curl` command and the response it got, where a screenshot would be. +fn exchanges(o: &mut String, xs: &[&Exchange], stale: bool, failed: bool) { + for (x, times) in crate::collapse(xs) { + fenced(o, "sh", &exchange::curl(x)); + fenced(o, "http", &exchange::response(x)); + if times > 1 { + let _ = writeln!(o, "*Sent {times} times.*\n"); + } + if stale { + let _ = writeln!(o, "*From before this page changed.*\n"); + } else if failed { + let _ = writeln!(o, "*What came back when this failed.*\n"); + } + } +} + +/// A fenced code block, its fence longer than any run of backticks in `code`. +fn fenced(o: &mut String, lang: &str, code: &str) { + let mut longest = 0; + let mut run = 0; + for c in code.chars() { + run = if c == '`' { run + 1 } else { 0 }; + longest = longest.max(run); + } + let fence = "`".repeat((longest + 1).max(3)); + let _ = writeln!(o, "{fence}{lang}\n{code}\n{fence}\n"); +} + fn index(d: &Docs) -> String { let s = d.suite; let mut o = String::new(); @@ -497,19 +526,23 @@ fn tutorial_page(d: &Docs, f: &Facts, i: usize, t: &manual::Tutorial) -> String } o.push('\n'); let mut img = String::new(); - image( - &mut img, - &d.look, - step.shot.as_deref(), - step.status, - step.stale, - d.reader(), - if step.seen.is_empty() { - &step.doing - } else { - &step.seen - }, - ); + if step.shot.is_some() || step.exchanges.is_empty() { + image( + &mut img, + &d.look, + step.shot.as_deref(), + step.status, + step.stale, + d.reader(), + if step.seen.is_empty() { + &step.doing + } else { + &step.seen + }, + ); + } + let xs: Vec<&Exchange> = step.exchanges.iter().collect(); + exchanges(&mut img, &xs, step.stale, step.status == Status::Failed); for line in img.lines() { let _ = writeln!(o, " {line}"); } @@ -583,15 +616,22 @@ fn feature_page(d: &Docs, f: &Facts, fi: usize) -> String { let _ = writeln!(o, "> {line}\n"); } match f.shot_of(d, e) { - Some((_, _, shot)) => image( - &mut o, - &d.look, - Some(&shot), - Status::Passed, - false, - d.reader(), - &voice::how(s, &x.steps), - ), + Some((p, step, shot)) => { + if let Some(shot) = &shot { + image( + &mut o, + &d.look, + Some(shot), + Status::Passed, + false, + d.reader(), + &voice::how(s, &x.steps), + ); + } + let r = crate::step_result(d, p, step); + let xs = r.map(|r| crate::actions(&r.exchanges)).unwrap_or_default(); + exchanges(&mut o, &xs, false, false); + } None => image( &mut o, &d.look, @@ -1458,17 +1498,25 @@ fn step_body( ) { let status = result.map_or(plan_status, |r| r.status); let stale = result.is_some_and(|r| r.stale); - image( - o, - &d.look, - result.and_then(|r| r.screenshot.as_deref()), - status, - stale, - false, - "", - ); - if result.and_then(|r| r.screenshot.as_ref()).is_none() { - let _ = writeln!(o, "*{}*\n", missing_shot(status, stale)); + let shot = result.and_then(|r| r.screenshot.as_deref()); + let all: &[Exchange] = result.map_or(&[], |r| &r.exchanges); + if shot.is_some() || all.is_empty() { + image(o, &d.look, shot, status, stale, false, ""); + if shot.is_none() { + let _ = writeln!(o, "*{}*\n", missing_shot(status, stale)); + } + } + exchanges(o, &crate::actions(all), stale, status == Status::Failed); + for (role, summary) in [ + (Role::Setup, "Requests that set it up"), + (Role::Check, "Requests that checked it"), + ] { + let xs: Vec<&Exchange> = all.iter().filter(|x| x.role == role).collect(); + if !xs.is_empty() { + let _ = writeln!(o, "
{summary} ({})\n", xs.len()); + exchanges(o, &xs, stale, false); + o.push_str("
\n\n"); + } } if let Some(e) = result .and_then(|r| r.error.as_ref()) diff --git a/crates/harness/examples/invoices/api.rs b/crates/harness/examples/invoices/api.rs new file mode 100644 index 0000000..e84f9ce --- /dev/null +++ b/crates/harness/examples/invoices/api.rs @@ -0,0 +1,420 @@ +//! A stand-in invoices API, served over real HTTP on a loopback port, and a client for it that +//! keeps every request it sends as an [`Exchange`]. Only the standard library: a test or an +//! example can start it without anything installed. +//! +//! The service keeps its invoices in memory. Two API keys are known: the admin's and a member's. +//! Members can create, finalize and read invoices; only admins can void one. + +use expectation_harness::Exchange; +use std::io::{BufRead, BufReader, Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::sync::{Arc, Mutex}; + +pub const ADMIN_KEY: &str = "sk_test_admin_9f8e7d6c5b4a"; +pub const MEMBER_KEY: &str = "sk_test_member_1a2b3c4d5e6f"; + +#[derive(Clone, Debug)] +struct Invoice { + id: String, + customer: String, + amount_due: u64, + status: &'static str, + created_by: &'static str, +} + +impl Invoice { + fn json(&self) -> String { + format!( + r#"{{"id":"{}","object":"invoice","customer":"{}","amount_due":{},"currency":"usd","status":"{}","created_by":{{"email":"{}"}}}}"#, + self.id, self.customer, self.amount_due, self.status, self.created_by + ) + } +} + +#[derive(Default)] +struct State { + invoices: Vec, + next: u32, + /// A defect, for the example's `--bug`: members can void invoices too. + members_can_void: bool, +} + +/// The service, running until the process ends. +pub struct Service { + pub url: String, + state: Arc>, +} + +impl Service { + pub fn start() -> Service { + let listener = TcpListener::bind("127.0.0.1:0").expect("a loopback port"); + let url = format!("http://{}", listener.local_addr().unwrap()); + let state: Arc> = Default::default(); + let shared = state.clone(); + std::thread::spawn(move || { + for stream in listener.incoming().flatten() { + let state = shared.clone(); + std::thread::spawn(move || serve(stream, &state)); + } + }); + Service { url, state } + } + + /// Forget every invoice: a fresh account for the next test. + pub fn reset(&self) { + let mut s = self.state.lock().unwrap(); + s.invoices.clear(); + } + + /// A defect to see a failure: members can void invoices too. + pub fn let_members_void(&self) { + self.state.lock().unwrap().members_can_void = true; + } +} + +struct Request { + method: String, + path: String, + headers: Vec<(String, String)>, + body: String, +} + +fn serve(stream: TcpStream, state: &Mutex) { + let mut reader = BufReader::new(&stream); + let Some(req) = read_request(&mut reader) else { + return; + }; + let (status, body) = route(&req, &mut state.lock().unwrap()); + let mut out = format!( + "HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nX-Request-Id: req_{:x}\r\nConnection: close\r\n", + expectation_story::exchange::reason(status), + body.len(), + expectation_story::now_ms() + ); + if status == 201 + && let Some(id) = field(&body, "id") + { + out.push_str(&format!("Location: {}/{id}\r\n", req.path)); + } + out.push_str("\r\n"); + out.push_str(&body); + let _ = (&stream).write_all(out.as_bytes()); +} + +fn read_request(reader: &mut impl BufRead) -> Option { + let mut line = String::new(); + reader.read_line(&mut line).ok()?; + let mut parts = line.split_whitespace(); + let (method, path) = (parts.next()?.to_string(), parts.next()?.to_string()); + let mut headers = vec![]; + loop { + let mut h = String::new(); + reader.read_line(&mut h).ok()?; + let h = h.trim_end(); + if h.is_empty() { + break; + } + let (k, v) = h.split_once(':')?; + headers.push((k.trim().to_string(), v.trim().to_string())); + } + let len = header(&headers, "content-length") + .and_then(|v| v.parse().ok()) + .unwrap_or(0); + let mut body = vec![0; len]; + reader.read_exact(&mut body).ok()?; + Some(Request { + method, + path, + headers, + body: String::from_utf8(body).ok()?, + }) +} + +fn error(status: u16, kind: &str, message: &str) -> (u16, String) { + ( + status, + format!(r#"{{"error":{{"type":"{kind}","message":"{message}"}}}}"#), + ) +} + +fn route(req: &Request, s: &mut State) -> (u16, String) { + let admin = match header(&req.headers, "authorization") { + Some(v) if v == format!("Bearer {ADMIN_KEY}") => true, + Some(v) if v == format!("Bearer {MEMBER_KEY}") => false, + _ => return error(401, "unauthorized", "Pass your API key as a bearer token."), + }; + let parts: Vec<&str> = req.path.trim_matches('/').split('/').collect(); + match (req.method.as_str(), parts.as_slice()) { + ("POST", ["v1", "invoices"]) => { + let Some(customer) = field(&req.body, "customer") else { + return error(422, "invalid_request", "An invoice needs a customer."); + }; + s.next += 1; + let invoice = Invoice { + id: format!("in_{:04}", s.next), + customer, + amount_due: number(&req.body, "amount_due").unwrap_or(0), + status: "draft", + created_by: if admin { + "ada@example.com" + } else { + "grace@example.com" + }, + }; + let body = invoice.json(); + s.invoices.push(invoice); + (201, body) + } + (method, ["v1", "invoices", id, rest @ ..]) => { + let Some(invoice) = s.invoices.iter_mut().find(|i| i.id == *id) else { + return error(404, "not_found", "No such invoice."); + }; + match (method, rest) { + ("GET", []) => (200, invoice.json()), + ("POST", ["finalize"]) if invoice.status == "draft" => { + invoice.status = "open"; + (200, invoice.json()) + } + ("POST", ["void"]) if !admin && !s.members_can_void => error( + 403, + "forbidden", + "Only an admin of the account can void an invoice.", + ), + ("POST", ["void"]) if invoice.status == "open" => { + invoice.status = "void"; + (200, invoice.json()) + } + ("POST", ["finalize"]) => error( + 409, + "invalid_state", + &format!( + "Only a draft invoice can be finalized; this one is {}.", + invoice.status + ), + ), + ("POST", ["void"]) => error( + 409, + "invalid_state", + &format!( + "Only an open invoice can be voided; this one is {}.", + invoice.status + ), + ), + _ => error(404, "not_found", "No such route."), + } + } + _ => error(404, "not_found", "No such route."), + } +} + +fn header<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> { + headers + .iter() + .find(|(k, _)| k.eq_ignore_ascii_case(name)) + .map(|(_, v)| v.as_str()) +} + +/// A string field of a flat JSON object: `"id":"in_1"` → `in_1`. Enough for the bodies this +/// service writes and the example sends. +pub fn field(json: &str, name: &str) -> Option { + let key = format!("\"{name}\""); + let at = json.find(&key)? + key.len(); + let rest = json[at..].trim_start().strip_prefix(':')?.trim_start(); + let rest = rest.strip_prefix('"')?; + Some(rest[..rest.find('"')?].to_string()) +} + +fn number(json: &str, name: &str) -> Option { + let key = format!("\"{name}\""); + let at = json.find(&key)? + key.len(); + let rest = json[at..].trim_start().strip_prefix(':')?.trim_start(); + let end = rest + .find(|c: char| !c.is_ascii_digit()) + .unwrap_or(rest.len()); + rest[..end].parse().ok() +} + +/// A client for the service, as a caller would write one: it sends the API key it is given, +/// and keeps every request it sends, with what came back, until they are taken. +pub struct Client { + pub base: String, + pub log: Vec, +} + +impl Client { + pub fn new(base: &str) -> Client { + Client { + base: base.to_string(), + log: vec![], + } + } + + /// Send a request and say what came back: the status and the body. + pub fn send( + &mut self, + key: &str, + method: &str, + path: &str, + body: Option<&str>, + ) -> (u16, String) { + let host = self.base.trim_start_matches("http://"); + let mut x = Exchange::new(method, format!("{}{path}", self.base)) + .header("Authorization", format!("Bearer {key}")) + .header("User-Agent", "invoices-example/1.0"); + let mut req = format!( + "{method} {path} HTTP/1.1\r\nHost: {host}\r\nAuthorization: Bearer {key}\r\nUser-Agent: invoices-example/1.0\r\nConnection: close\r\n" + ); + if let Some(b) = body { + x = x.header("Content-Type", "application/json").body(b); + req.push_str(&format!( + "Content-Type: application/json\r\nContent-Length: {}\r\n", + b.len() + )); + } + req.push_str("\r\n"); + req.push_str(body.unwrap_or("")); + let answer = TcpStream::connect(host).and_then(|mut s| { + s.write_all(req.as_bytes())?; + let mut text = String::new(); + s.read_to_string(&mut text)?; + Ok(text) + }); + let Ok(answer) = answer else { + self.log.push(x); + return (0, String::new()); + }; + let (head, body) = answer.split_once("\r\n\r\n").unwrap_or((&answer, "")); + let mut lines = head.lines(); + let status = lines + .next() + .and_then(|l| l.split_whitespace().nth(1)) + .and_then(|s| s.parse().ok()) + .unwrap_or(0); + x = x.response(status); + for l in lines { + if let Some((k, v)) = l.split_once(':') { + x = x.response_header(k.trim(), v.trim()); + } + } + self.log.push(x.response_body(body)); + (status, body.to_string()) + } +} + +/// What the tests drive: the service, a client, the invoice the test is about and whose key +/// the client is sending. +pub struct World { + pub service: Service, + /// Behind a `RefCell` because observing reads the API, and reading it is a request too. + client: std::cell::RefCell, + invoice: Option, + member: bool, +} + +impl World { + pub fn new() -> World { + let service = Service::start(); + let client = Client::new(&service.url).into(); + World { + service, + client, + invoice: None, + member: false, + } + } + + fn key(&self) -> &'static str { + if self.member { MEMBER_KEY } else { ADMIN_KEY } + } + + fn create(&mut self, key: &str) { + let (status, body) = self.client.get_mut().send( + key, + "POST", + "/v1/invoices", + Some(r#"{"customer":"cus_0042","amount_due":12000,"currency":"usd"}"#), + ); + if status == 201 { + self.invoice = field(&body, "id"); + } + } + + fn act(&mut self, key: &str, action: &str) { + if let Some(id) = self.invoice.clone() { + let path = format!("/v1/invoices/{id}/{action}"); + self.client.get_mut().send(key, "POST", &path, None); + } + } + + /// The invoice's `status`, read from the API. + fn status(&self) -> String { + let Some(id) = &self.invoice else { + return "none".into(); + }; + let (_, body) = + self.client + .borrow_mut() + .send(self.key(), "GET", &format!("/v1/invoices/{id}"), None); + field(&body, "status").unwrap_or_else(|| "none".into()) + } + + /// Put an invoice in place as the admin, whoever the test then calls as. + fn arrange(&mut self, status: &str) { + self.invoice = None; + let steps: &[&str] = match status { + "draft" => &[], + "open" => &["finalize"], + "void" => &["finalize", "void"], + _ => return, + }; + self.create(ADMIN_KEY); + for step in steps { + self.act(ADMIN_KEY, step); + } + } +} + +/// The suite's vocabulary bound to the API. With `bug`, members can void invoices. +pub fn driver(bug: bool) -> expectation_harness::Bindings { + let world = World::new(); + if bug { + world.service.let_members_void(); + } + expectation_harness::Bindings::new(world) + .observe("the invoice", World::status) + .observe( + "the caller", + |w: &World| { + if w.member { "member" } else { "admin" } + }, + ) + .arrange("the invoice", |w, v| w.arrange(v.as_str().unwrap_or(""))) + .set("the caller", |w, v| w.member = v.as_str() == Some("member")) + .on("the caller creates an invoice", |w| w.create(w.key())) + .on("the caller finalizes the invoice", |w| { + w.act(w.key(), "finalize") + }) + .on("the caller voids the invoice", |w| w.act(w.key(), "void")) + .on_reset(|w| { + w.service.reset(); + w.invoice = None; + w.member = false; + }) + .on_exchanges(|w| std::mem::take(&mut w.client.get_mut().log)) +} + +/// The run's options: the docs call each key by whose it is, and who created an invoice is a +/// person's email, which they don't show. +pub fn options() -> expectation_harness::RunOptions { + expectation_harness::RunOptions { + redaction: expectation_harness::Redaction { + keys: vec![ + (ADMIN_KEY.into(), "$ADMIN_API_KEY".into()), + (MEMBER_KEY.into(), "$MEMBER_API_KEY".into()), + ], + fields: vec!["email".into()], + ..Default::default() + }, + ..Default::default() + } +} diff --git a/crates/harness/examples/invoices/invoices.expect b/crates/harness/examples/invoices/invoices.expect new file mode 100644 index 0000000..c6fe850 --- /dev/null +++ b/crates/harness/examples/invoices/invoices.expect @@ -0,0 +1,109 @@ +Suite: Invoices API + +Vocabulary: + State: the invoice is none | draft | open | void + Shows: draft: the invoice's `status` is `draft` + Shows: open: the invoice's `status` is `open` + Shows: void: the invoice's `status` is `void` + State: the caller is admin | member + # The API key a request is sent with decides who it is from. + Set: use the {value}'s API key + Event: the caller creates an invoice + Say: create an invoice for the customer + Event: the caller finalizes the invoice + Say: finalize the invoice + Event: the caller voids the invoice + Say: void the invoice + + Phrase: a new account, as its admin + Say: start on an account with no invoices, with the admin's API key + Given the invoice is none + And the caller is admin + +Term: API key + A secret that says which account a request is for and who in it is asking: an admin or a + member. Send it as a bearer token in the `Authorization` header. The examples here read an + admin's key from `$ADMIN_API_KEY`, a member's from `$MEMBER_API_KEY`, and the address of the + API from `$BASE_URL`. + +Background: + Given a new account, as its admin + +Feature: Invoices + Approval: proposed + + An invoice is what your account bills a customer. You create it as a draft, which you can + still change, then finalize it, which opens it for payment and gives it its number. Each + request returns the invoice as it is now, and its `status` says where it is: `draft`, `open` + or `void`. + + Every request is made with an [[API key]]. Admins and members of the account can both create + and finalize invoices. + + Expectation: Creating an invoice makes a draft + Given the invoice is none + When the caller creates an invoice + Then the invoice is draft + + Expectation: Finalizing a draft opens it + Given the invoice is draft + When the caller finalizes the invoice + Then the invoice is open + +Feature: Voiding + Approval: proposed + + Void an open invoice that was issued by mistake: it stays on the account for the record, but + can no longer be paid. Voiding can't be undone, so only an admin of the account can do it. A + member who tries gets `403 Forbidden` and the invoice stays open. + + Expectation: Voiding an open invoice voids it + Given the invoice is open + When the caller voids the invoice + Then the invoice is void + + Expectation: Members can't void + Overrides: Voiding an open invoice voids it + Given the invoice is open + And the caller is member + When the caller voids the invoice + Then the invoice is open + +Story: Bill a customer + Approval: proposed + As an admin of the account + I want to issue an invoice, and take it back if it was a mistake + So that my customer pays what they owe and nothing else + + You bill a customer in two requests: create the invoice as a draft, then finalize it. Until + it is finalized the customer can't see it, so a draft is the place to get it right. The + response to each request is the whole invoice, so you can read its `status` without asking + again. + + If an open invoice turns out to be wrong, void it and issue a new one. A void invoice stays on + the account, so the record shows what was sent. + + When the caller creates an invoice + Then the invoice is draft + When the caller finalizes the invoice + Then the invoice is open + When the caller voids the invoice + Then the invoice is void + +Story: Ask an admin to void + Approval: proposed + As a member of the account + I want to bill customers without being able to void what was billed + So that money can't quietly be taken back + + Members bill customers the same way admins do. Voiding is the one thing a member's + [[API key]] can't do: the API answers `403 Forbidden` and leaves the invoice open, so ask an + admin of the account to void it. + + Given the caller is member + When the caller creates an invoice + Then the invoice is draft + When the caller finalizes the invoice + Then the invoice is open + When the caller voids the invoice + Then the invoice is open diff --git a/crates/harness/examples/invoices/main.rs b/crates/harness/examples/invoices/main.rs new file mode 100644 index 0000000..ec59c2f --- /dev/null +++ b/crates/harness/examples/invoices/main.rs @@ -0,0 +1,42 @@ +//! An API's manual, with a `curl` command where a screenshot would be. The system under test is +//! a stand-in invoices API on a loopback port; the driver calls it over HTTP and hands the +//! runner each request it sent with the response it got. The runner keeps them redacted (the +//! API keys become `$ADMIN_API_KEY` and `$MEMBER_API_KEY`, the loopback address `$BASE_URL`, +//! people's emails `REDACTED`), and the manual shows each step's request and response. +//! +//! ```text +//! cargo run -p expectation_harness --example invoices # docs in target/expectation-docs/invoices +//! cargo run -p expectation_harness --example invoices -- out --bug # members can void: a failure +//! ``` + +mod api; + +use expectation_harness::Acceptance; +use std::path::{Path, PathBuf}; +use std::process::exit; + +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + let bug = args.iter().any(|a| a == "--bug"); + let docs = args.iter().find(|a| !a.starts_with("--")).map_or_else( + || PathBuf::from("target/expectation-docs/invoices"), + PathBuf::from, + ); + let suite = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/invoices"); + let acceptance = Acceptance::load(&suite) + .unwrap_or_else(|e| { + eprint!("{e}"); + exit(1) + }) + .with_coverage(); + let report = acceptance.run(&mut api::driver(bug), &api::options()); + if let Err(e) = acceptance.publish(&docs, "api", &report) { + eprintln!("couldn't write the docs to {}: {e}", docs.display()); + exit(1); + } + println!("docs: {}", docs.join("site/index.html").display()); + if !report.passed() { + eprint!("{}", report.failures(&acceptance.suite, &acceptance.plans)); + exit(1); + } +} diff --git a/crates/harness/src/bindings.rs b/crates/harness/src/bindings.rs index 3824bf4..25a42d9 100644 --- a/crates/harness/src/bindings.rs +++ b/crates/harness/src/bindings.rs @@ -12,7 +12,7 @@ use crate::driver::{Capture, Driver, Need, Progress, Screenshot}; use expectation_model::text; use expectation_model::{Event, Value, Variable}; -use expectation_story::Control; +use expectation_story::{Control, Exchange}; use std::collections::HashMap; type Observer = Box Value>; @@ -23,6 +23,7 @@ type Lister = Box Vec>; type Taker = Box Vec>; type Shooter = Box Screenshot>; type Capturer = Box Vec>; +type Exchanger = Box Vec>; pub struct Bindings { pub world: W, @@ -37,6 +38,7 @@ pub struct Bindings { controls: Option>, used: Option>, attachments: Option>, + exchanges: Option>, needs: Vec, fresh: Vec, always_reset: bool, @@ -57,6 +59,7 @@ impl Bindings { controls: None, used: None, attachments: None, + exchanges: None, needs: vec![], fresh: vec![], always_reset: false, @@ -146,6 +149,13 @@ impl Bindings { self } + /// The HTTP requests the world made since this was last called, emptied as it is read (see + /// [`Driver::take_exchanges`]). + pub fn on_exchanges(mut self, f: impl FnMut(&mut W) -> Vec + 'static) -> Self { + self.exchanges = Some(Box::new(f)); + self + } + pub fn on_screenshot(mut self, f: impl FnMut(&W) -> Screenshot + 'static) -> Self { self.screenshot = Some(Box::new(f)); self @@ -237,6 +247,13 @@ impl Driver for Bindings { .unwrap_or_default() } + fn take_exchanges(&mut self) -> Vec { + self.exchanges + .as_mut() + .map(|f| f(&mut self.world)) + .unwrap_or_default() + } + fn screenshot(&mut self) -> Result, String> { Ok(self.screenshot.as_mut().map(|f| f(&self.world))) } diff --git a/crates/harness/src/driver.rs b/crates/harness/src/driver.rs index 1b720e4..3b6512e 100644 --- a/crates/harness/src/driver.rs +++ b/crates/harness/src/driver.rs @@ -1,7 +1,7 @@ //! The system under test, as the runner sees it. use expectation_model::{Event, Value, Variable}; -use expectation_story::Control; +use expectation_story::{Control, Exchange}; /// Whether waiting could change anything. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -169,6 +169,17 @@ pub trait Driver { vec![] } + /// The HTTP requests the driver made since this was last called, and what came back (see + /// [`Exchange`]): for a system that is an API, what the docs show where they'd show a + /// screenshot. The runner asks after arranging a test's start, after each action and after + /// each step's checks, so it knows which request was the action and which only read the + /// system back, and keeps each one redacted ([`RunOptions::redaction`](crate::RunOptions::redaction)): + /// hand over the requests as sent, credentials and all. It asks even when nothing is kept, + /// so a driver's buffer never grows past one step. + fn take_exchanges(&mut self) -> Vec { + vec![] + } + fn screenshot(&mut self) -> Result, String> { Ok(None) } diff --git a/crates/harness/src/lib.rs b/crates/harness/src/lib.rs index 989115c..b2fdf6d 100644 --- a/crates/harness/src/lib.rs +++ b/crates/harness/src/lib.rs @@ -25,7 +25,7 @@ use expectation_graph::Analysis; use expectation_language::Loaded; use expectation_model::diagnostic::{has_errors, render_all}; use expectation_model::{Diagnostic, Suite}; -pub use expectation_story::{Began, Control, Policy, Provenance}; +pub use expectation_story::{Began, Control, Exchange, Policy, Provenance, Redaction, Role}; use expectation_story::{Plan, Report}; use std::path::Path; diff --git a/crates/harness/src/run.rs b/crates/harness/src/run.rs index 2147c87..fd37018 100644 --- a/crates/harness/src/run.rs +++ b/crates/harness/src/run.rs @@ -4,8 +4,8 @@ use crate::driver::{Driver, Need, Progress}; use crate::service::Services; use expectation_model::{Suite, Trigger}; use expectation_story::{ - Attachment, Began, Check, CheckResult, Control, Plan, PlanResult, Report, Run, Status, - StepResult, Window, now_ms, + Attachment, Began, Check, CheckResult, Control, Exchange, Plan, PlanResult, Redaction, Report, + Role, Run, Status, StepResult, Window, now_ms, }; use std::path::PathBuf; use std::sync::Mutex; @@ -50,6 +50,10 @@ pub struct RunOptions { /// refuse it for itself ([`Driver::may_continue`]), and one that wants services afresh /// for each test ([`Driver::fresh`]) never continues. pub share_starts: bool, + /// What is kept of the HTTP requests the driver hands over ([`Driver::take_exchanges`]): + /// which headers, which fields are secret, what origin the docs show. Exchanges are kept + /// in the results, which are published with the manual, so only what this lets through is. + pub redaction: Redaction, } impl Default for RunOptions { @@ -63,6 +67,7 @@ impl Default for RunOptions { attachments_dir: None, provenance: Default::default(), share_starts: true, + redaction: Redaction::default(), } } } @@ -432,6 +437,17 @@ fn run_steps( attachments.extend(save_captures(dir, run, &plan.slug, n, taken)); } }; + // The requests the driver made since it was last asked, as `role`, redacted. + let exchanges = |driver: &mut dyn Driver, role: Role| -> Vec { + driver + .take_exchanges() + .into_iter() + .map(|mut x| { + x.role = role; + opts.redaction.apply(&x) + }) + .collect() + }; let skip_all = |start: StepResult, attachments: Vec| PlanResult { status: Status::Failed, start, @@ -458,6 +474,7 @@ fn run_steps( let setup_failed = |error: String, driver: &mut dyn Driver, keep: &mut dyn FnMut(&mut dyn Driver, usize)| { let mut start = failed(error); + start.exchanges = exchanges(driver, Role::Setup); start.window = Some(Window { start_ms: began, end_ms: now_ms(), @@ -481,7 +498,11 @@ fn run_steps( && driver.may_continue() && driver.fresh().is_empty() && observe(suite, driver, &plan.start_checks, 0).status == Status::Passed; + // What looking cost: this test's first check if it continues; if not, about a system it + // is about to reset, so nobody's. + let mut taken = exchanges(driver, Role::Check); if !continued { + taken.clear(); if let Err(e) = driver.reset() { let start = setup_failed(format!("couldn't reset: {e}"), driver, &mut keep); return skip_all(start, attachments); @@ -498,6 +519,7 @@ fn run_steps( } } } + taken.extend(exchanges(driver, Role::Setup)); driver.settle(); // Setting the test up is not using the controls. let mut seen = Seen::new(opts.controls); @@ -505,6 +527,8 @@ fn run_steps( seen.used.clear(); seen.look(driver); let mut start = observe(suite, driver, &plan.start_checks, opts.patience); + taken.extend(exchanges(driver, Role::Check)); + start.exchanges = taken; start.screenshot = shoot(driver, 0); start.window = Some(Window { start_ms: began, @@ -521,6 +545,7 @@ fn run_steps( } let began = now_ms(); let acted = act(suite, driver, &step.action); + let mut taken = exchanges(driver, Role::Action); seen.take(driver); let mut r = match acted { Ok(()) => { @@ -530,6 +555,8 @@ fn run_steps( } Err(e) => failed(e), }; + taken.extend(exchanges(driver, Role::Check)); + r.exchanges = taken; r.screenshot = shoot(driver, n + 1); r.window = Some(Window { start_ms: began, diff --git a/crates/harness/tests/exchanges.rs b/crates/harness/tests/exchanges.rs new file mode 100644 index 0000000..d5b64d2 --- /dev/null +++ b/crates/harness/tests/exchanges.rs @@ -0,0 +1,177 @@ +//! An API's manual: the driver hands over the HTTP requests it made, the runner keeps them +//! redacted in the results, and the docs show each step's request as a `curl` command with the +//! response it got, where a screenshot would be. +#[path = "../examples/invoices/api.rs"] +mod api; + +use expectation_harness::{Acceptance, Role}; +use expectation_story::Status; +use std::path::{Path, PathBuf}; + +/// The invoices example, with a story that starts from an open invoice, so a test arranges one. +fn suite(name: &str) -> (Acceptance, PathBuf) { + let root = Path::new(env!("CARGO_TARGET_TMPDIR")).join(name); + let _ = std::fs::remove_dir_all(&root); + let src = root.join("suite"); + std::fs::create_dir_all(&src).unwrap(); + let example = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/invoices/invoices.expect"); + std::fs::copy(example, src.join("invoices.expect")).unwrap(); + std::fs::write( + src.join("correct.expect"), + " +Story: Void an invoice sent by mistake + As an admin of the account + I want to take back an invoice + So that the customer isn't asked to pay it + + Given the invoice is open + When the caller voids the invoice + Then the invoice is void +", + ) + .unwrap(); + let a = Acceptance::load(&src) + .unwrap_or_else(|e| panic!("{e}")) + .with_coverage(); + (a, root.join("docs")) +} + +/// Every file under `dir`, as text. +fn every_file(dir: &Path) -> Vec<(PathBuf, String)> { + let mut out = vec![]; + let mut todo = vec![dir.to_path_buf()]; + while let Some(d) = todo.pop() { + for e in std::fs::read_dir(&d).unwrap().flatten() { + let p = e.path(); + if p.is_dir() { + todo.push(p); + } else if let Ok(text) = std::fs::read_to_string(&p) { + out.push((p, text)); + } + } + } + out +} + +#[test] +fn the_manual_shows_each_steps_request_and_keeps_nothing_secret() { + let (a, docs) = suite("exchanges"); + let report = a.run(&mut api::driver(false), &api::options()); + assert!(report.passed(), "{}", report.failures(&a.suite, &a.plans)); + let i = |slug: &str| { + a.plans + .iter() + .position(|p| p.slug.starts_with(slug)) + .unwrap() + }; + + // Each request is marked with what it did in its step. + let billed = &report.results[i("bill-a-customer")]; + assert!( + billed.start.exchanges.is_empty(), + "nothing to set up or read: {:?}", + billed.start.exchanges + ); + let roles = |n: usize| -> Vec<(Role, String)> { + billed.steps[n] + .exchanges + .iter() + .map(|x| (x.role, x.label())) + .collect() + }; + let step = roles(0); + assert_eq!(step.len(), 2, "{step:?}"); + assert_eq!(step[0], (Role::Action, "POST /v1/invoices → 201".into())); + assert_eq!(step[1].0, Role::Check); + assert!(step[1].1.starts_with("GET /v1/invoices/in_"), "{step:?}"); + let corrected = &report.results[i("void-an-invoice-sent-by-mistake")]; + let setup: Vec = corrected + .start + .exchanges + .iter() + .filter(|x| x.role == Role::Setup) + .map(|x| x.label()) + .collect(); + assert_eq!(setup.len(), 2, "created and finalized: {setup:?}"); + + a.publish(&docs, "api", &report).unwrap(); + // Nothing secret anywhere in what is published, results included. + let files = every_file(&docs); + assert!(files.iter().any(|(p, _)| p.ends_with("results/api.tsv"))); + for (path, text) in &files { + for secret in [ + api::ADMIN_KEY, + api::MEMBER_KEY, + "ada@example.com", + "grace@example.com", + "127.0.0.1", + "X-Request-Id", + "User-Agent", + ] { + assert!(!text.contains(secret), "{secret} in {}", path.display()); + } + } + let read = |page: &str| std::fs::read_to_string(docs.join(page)).unwrap(); + + // The feature page shows the exception as the member saw it. + let md = read("features/voiding.md"); + assert!( + md.contains("/void\" \\\n -H \"Authorization: Bearer $MEMBER_API_KEY\"\n```"), + "{md}" + ); + assert!(md.contains("```http\nHTTP/1.1 403 Forbidden\n"), "{md}"); + assert!(!md.contains("no-screenshot"), "{md}"); + let html = read("site/feature-voiding.html"); + assert!(html.contains("Bearer $MEMBER_API_KEY"), "{html}"); + assert!(html.contains(r#"HTTP/1.1 403 Forbidden"#)); + assert!(!html.contains("no-screenshot"), "{html}"); + + // A tutorial step shows the request it tells the reader to send, not the reads that checked it. + let tutorial = read("tutorials/bill-a-customer.md"); + assert!(tutorial.contains("curl -X POST \"$BASE_URL/v1/invoices\"")); + assert!( + !tutorial.contains("curl \"$BASE_URL/v1/invoices/in_"), + "{tutorial}" + ); + // The test page has those too, folded away. + let test = read("site/test-bill-a-customer.html"); + assert!(test.contains("Requests that checked it (1)"), "{test}"); + let test = read(&format!( + "site/test-{}.html", + a.plans[i("void-an-invoice-sent-by-mistake")].slug + )); + assert!(test.contains("Requests that set it up (2)"), "{test}"); + + // From the saved results alone, the same pages. + let saved = expectation_story::load_results_for(&docs.join("results"), &a.plans, None); + let again = docs.with_file_name("again"); + a.docs(Some(&saved)).write(&again).unwrap(); + for page in ["features/voiding.md", "tutorials/bill-a-customer.md"] { + assert_eq!( + std::fs::read_to_string(again.join(page)).unwrap(), + read(page), + "{page}" + ); + } +} + +#[test] +fn a_failure_shows_what_came_back() { + let (a, docs) = suite("exchanges-bug"); + let report = a.run(&mut api::driver(true), &api::options()); + let p = a + .plans + .iter() + .position(|p| p.slug.starts_with("ask-an-admin-to-void")) + .unwrap(); + assert_eq!(report.results[p].status, Status::Failed); + let void = &report.results[p].steps[2]; + let label = void.exchanges[0].label(); + assert!( + label.starts_with("POST /v1/invoices/in_") && label.ends_with("/void → 200"), + "{label}" + ); + a.publish(&docs, "api", &report).unwrap(); + let md = std::fs::read_to_string(docs.join("tutorials/ask-an-admin-to-void.md")).unwrap(); + assert!(md.contains("*What came back when this failed.*"), "{md}"); +} diff --git a/crates/story/src/exchange.rs b/crates/story/src/exchange.rs new file mode 100644 index 0000000..5d9ea72 --- /dev/null +++ b/crates/story/src/exchange.rs @@ -0,0 +1,1127 @@ +//! HTTP exchanges: the requests a step made and the responses it got, kept as evidence a manual +//! can show. Where a screenshot shows a person what to expect on screen, an exchange shows a +//! developer the request to send and what comes back: the docs draw it as a `curl` command and a +//! response. +//! +//! Unlike an [`crate::Attachment`], an exchange is made to be published, so it is redacted before +//! it is kept (see [`Redaction`]): only a short list of headers survives, credentials become +//! `$API_KEY`, and the values of secret-looking query parameters and JSON fields are replaced. + +use std::fmt::Write; + +/// What an exchange did in its step. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Role { + /// Putting the test's start in place ("Given …"). + Setup, + /// The step's action: what the reader is told to do. + Action, + /// Reading the system back to check the step's Thens. + Check, +} + +impl Role { + pub fn name(self) -> &'static str { + match self { + Role::Setup => "setup", + Role::Action => "action", + Role::Check => "check", + } + } + + fn parse(s: &str) -> Option { + [Role::Setup, Role::Action, Role::Check] + .into_iter() + .find(|r| r.name() == s) + } +} + +/// One request and its response. +/// +/// ``` +/// use expectation_story::Exchange; +/// let x = Exchange::new("POST", "http://127.0.0.1:8080/v1/invoices") +/// .header("Content-Type", "application/json") +/// .body(r#"{"customer":"cus_1"}"#) +/// .response(201) +/// .response_header("Content-Type", "application/json") +/// .response_body(r#"{"id":"in_1","status":"draft"}"#); +/// assert_eq!(x.status, 201); +/// ``` +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Exchange { + /// Set by the runner from when the driver handed it over. + pub role: Role, + pub method: String, + /// The whole URL, with its query. + pub url: String, + pub request_headers: Vec<(String, String)>, + pub request_body: Option, + /// 0 if no response came (the connection failed). + pub status: u16, + pub response_headers: Vec<(String, String)>, + pub response_body: Option, +} + +impl Exchange { + pub fn new(method: impl Into, url: impl Into) -> Self { + Exchange { + role: Role::Action, + method: method.into().to_ascii_uppercase(), + url: url.into(), + request_headers: vec![], + request_body: None, + status: 0, + response_headers: vec![], + response_body: None, + } + } + + pub fn header(mut self, name: impl Into, value: impl Into) -> Self { + self.request_headers.push((name.into(), value.into())); + self + } + + pub fn body(mut self, body: impl Into) -> Self { + self.request_body = Some(body.into()); + self + } + + pub fn response(mut self, status: u16) -> Self { + self.status = status; + self + } + + pub fn response_header(mut self, name: impl Into, value: impl Into) -> Self { + self.response_headers.push((name.into(), value.into())); + self + } + + pub fn response_body(mut self, body: impl Into) -> Self { + self.response_body = Some(body.into()); + self + } + + /// The path and query, without the origin: `/v1/invoices?limit=2`. + pub fn path(&self) -> &str { + let (_, rest) = split_origin(&self.url); + if rest.is_empty() { "/" } else { rest } + } + + /// `POST /v1/invoices → 201`, for a caption or a list. + pub fn label(&self) -> String { + let path = self.path(); + let path = path.split_once('?').map_or(path, |(p, _)| p); + match self.status { + 0 => format!("{} {path} (no response)", self.method), + s => format!("{} {path} → {s}", self.method), + } + } + + /// A request header's value, by name ignoring case. + pub fn request_header(&self, name: &str) -> Option<&str> { + find(&self.request_headers, name) + } + + /// A response header's value, by name ignoring case. + pub fn response_header_value(&self, name: &str) -> Option<&str> { + find(&self.response_headers, name) + } +} + +fn find<'a>(headers: &'a [(String, String)], name: &str) -> Option<&'a str> { + headers + .iter() + .find(|(k, _)| k.eq_ignore_ascii_case(name)) + .map(|(_, v)| v.as_str()) +} + +/// `("http://127.0.0.1:8080", "/v1/x?y=1")`, and `("$BASE_URL", "/v1/x")` once redacted. A URL +/// without a scheme is all path. +fn split_origin(url: &str) -> (&str, &str) { + if let Some(n) = variable(url) { + return url.split_at(n); + } + let Some(i) = url.find("://") else { + return ("", url); + }; + match url[i + 3..].find(['/', '?']) { + Some(j) => url.split_at(i + 3 + j), + None => (url, ""), + } +} + +/// What may be published of an exchange. [`Redaction::apply`] is the only way the runner keeps +/// one, so a driver that hands over everything it sent still publishes nothing secret: +/// +/// - Request headers other than [`Redaction::request_headers`] and response headers other than +/// [`Redaction::response_headers`] are dropped, and with them cookies and anything a gateway +/// added. +/// - `Authorization` keeps its scheme and its credential becomes `$API_KEY`; so does the value of +/// each of [`Redaction::key_headers`] (`X-Api-Key`). A key listed in [`Redaction::keys`] +/// becomes its own name instead (`$MEMBER_API_KEY`), so the docs can say whose it was. +/// - Query parameters, form fields and JSON fields named like a secret (a `password`, a `token`, +/// a `client_secret`, an `api_key`), and those in [`Redaction::fields`], read `REDACTED`. +/// - A loopback origin (`http://127.0.0.1:53211`) becomes `$BASE_URL`, or +/// [`Redaction::base_url`] replaces every origin. +/// - JSON bodies are pretty-printed. A JSON body that doesn't parse can't be checked for secrets, +/// so it is replaced with a note; so is anything past [`Redaction::max_body`]. +#[derive(Clone, Debug)] +pub struct Redaction { + /// The origin the docs show instead of the one the tests used: `https://api.example.com`. + pub base_url: Option, + /// Request headers to keep, besides `Authorization`. Matched ignoring case. + pub request_headers: Vec, + /// Response headers to keep. Matched ignoring case. + pub response_headers: Vec, + /// Headers that carry a key, kept with the value `$API_KEY`. + pub key_headers: Vec, + /// Keys the tests use, and what the docs call each: `("sk_test_4f…", "$MEMBER_API_KEY")`. A + /// name is `$` and capitals, digits and `_`; any other reads `$API_KEY`. The key itself is + /// never kept. + pub keys: Vec<(String, String)>, + /// More fields (JSON keys, query and form parameters) whose values aren't shown, besides those + /// named like a secret: `email`, `phone`. Matched ignoring case. + pub fields: Vec, + /// The longest body kept, in bytes, after pretty-printing. + pub max_body: usize, +} + +impl Default for Redaction { + fn default() -> Self { + let list = |l: &[&str]| l.iter().map(|s| s.to_string()).collect(); + Redaction { + base_url: None, + request_headers: list(&[ + "Accept", + "Content-Type", + "Idempotency-Key", + "If-Match", + "If-None-Match", + "Prefer", + ]), + response_headers: list(&["Content-Type", "ETag", "Location", "Retry-After"]), + key_headers: list(&["X-Api-Key", "Api-Key"]), + keys: vec![], + fields: vec![], + max_body: 16 * 1024, + } + } +} + +/// What a secret value reads as. +pub const REDACTED: &str = "REDACTED"; + +impl Redaction { + pub fn apply(&self, x: &Exchange) -> Exchange { + let has = |l: &[String], name: &str| l.iter().any(|k| k.eq_ignore_ascii_case(name)); + let mut request_headers = vec![]; + for (k, v) in &x.request_headers { + if k.eq_ignore_ascii_case("authorization") { + let v = match v.trim().split_once(' ') { + Some((scheme, key)) => format!("{scheme} {}", self.key_name(key)), + None => self.key_name(v).to_string(), + }; + request_headers.push((k.clone(), v)); + } else if has(&self.key_headers, k) { + request_headers.push((k.clone(), self.key_name(v).to_string())); + } else if has(&self.request_headers, k) { + request_headers.push((k.clone(), clean(v, 500))); + } + } + let response_headers = x + .response_headers + .iter() + .filter(|(k, _)| has(&self.response_headers, k)) + .map(|(k, v)| { + let v = if k.eq_ignore_ascii_case("location") { + self.url(v) + } else { + clean(v, 500) + }; + (k.clone(), v) + }) + .collect(); + let request_type = find(&x.request_headers, "content-type"); + let response_type = find(&x.response_headers, "content-type"); + Exchange { + role: x.role, + method: clean(&x.method, 20), + url: self.url(&x.url), + request_headers, + request_body: x + .request_body + .as_deref() + .map(|b| self.body(b, request_type)), + status: x.status, + response_headers, + response_body: x + .response_body + .as_deref() + .map(|b| self.body(b, response_type)), + } + } + + /// What the docs call a key: its name in [`Redaction::keys`], or `$API_KEY`. + fn key_name(&self, key: &str) -> &str { + self.keys + .iter() + .find(|(k, name)| k == key.trim() && variable(name) == Some(name.len())) + .map_or("$API_KEY", |(_, name)| name.as_str()) + } + + fn secret(&self, name: &str) -> bool { + secret_name(name) || self.fields.iter().any(|f| f.eq_ignore_ascii_case(name)) + } + + /// The URL as the docs show it: the origin swapped, secret query values replaced. + fn url(&self, url: &str) -> String { + let url = clean(url, 2000); + let (origin, rest) = split_origin(&url); + let origin = match &self.base_url { + Some(base) if !origin.is_empty() => base.trim_end_matches('/').to_string(), + _ if loopback(origin) => "$BASE_URL".to_string(), + _ => origin.to_string(), + }; + let (path, query) = match rest.split_once('?') { + Some((p, q)) => (p, Some(q)), + None => (rest, None), + }; + let mut out = format!("{origin}{path}"); + if let Some(q) = query { + out.push('?'); + out.push_str(&self.form(q)); + } + out + } + + /// `a=1&password=x` → `a=1&password=REDACTED`. + fn form(&self, q: &str) -> String { + q.split('&') + .map(|pair| match pair.split_once('=') { + Some((k, _)) if self.secret(&percent_decode(k)) => format!("{k}={REDACTED}"), + _ => pair.to_string(), + }) + .collect::>() + .join("&") + } + + fn body(&self, body: &str, media_type: Option<&str>) -> String { + let media = media_type + .and_then(|m| m.split(';').next()) + .unwrap_or("") + .trim() + .to_ascii_lowercase(); + let looks_json = media.ends_with("json") + || media.ends_with("+json") + || (media.is_empty() && matches!(body.trim_start().chars().next(), Some('{' | '['))); + let out = if looks_json { + match Json::parse(body) { + Ok(mut j) => { + j.redact(&|k| self.secret(k)); + j.pretty() + } + Err(_) => { + return format!( + "(a {}-byte body that isn't valid JSON, not shown)", + body.len() + ); + } + } + } else if media == "application/x-www-form-urlencoded" { + self.form(body.trim()) + } else { + body.replace("\r\n", "\n") + }; + if out.len() > self.max_body { + let mut cut = self.max_body; + while !out.is_char_boundary(cut) { + cut -= 1; + } + format!("{}\n… ({} more bytes)", &out[..cut], out.len() - cut) + } else { + out + } + } +} + +fn loopback(origin: &str) -> bool { + let host = origin.split_once("://").map_or("", |(_, h)| h); + let host = host.rsplit_once('@').map_or(host, |(_, h)| h); + let host = match host.strip_prefix('[') { + Some(v6) => v6.split(']').next().unwrap_or(""), + None => host.split(':').next().unwrap_or(""), + }; + host == "localhost" + || host.ends_with(".localhost") + || host.starts_with("127.") + || host == "::1" + || host == "0.0.0.0" +} + +/// A field named like a secret: `password`, `client_secret`, `accessToken`, `api-key`. +pub fn secret_name(name: &str) -> bool { + // Words, split at `_`, `-`, `.`, spaces and camelCase: `clientSecret` is `client secret`. + let mut words: Vec = vec![]; + let mut word = String::new(); + let mut prev_lower = false; + for c in name.chars() { + if !c.is_alphanumeric() { + words.extend((!word.is_empty()).then(|| std::mem::take(&mut word))); + prev_lower = false; + continue; + } + if c.is_uppercase() && prev_lower { + words.push(std::mem::take(&mut word)); + } + prev_lower = c.is_lowercase() || c.is_ascii_digit(); + word.extend(c.to_lowercase()); + } + words.extend((!word.is_empty()).then_some(word)); + let one = |w: &str| { + matches!( + w, + "password" + | "passwd" + | "passphrase" + | "secret" + | "token" + | "apikey" + | "credential" + | "credentials" + | "authorization" + | "cookie" + | "otp" + | "pin" + | "cvc" + | "cvv" + | "ssn" + ) + }; + let two = |a: &str, b: &str| { + b == "key" && matches!(a, "api" | "private" | "access" | "secret" | "signing") + }; + words.iter().any(|w| one(w)) || words.windows(2).any(|w| two(&w[0], &w[1])) +} + +/// Control characters out, and at most `max` bytes. +fn clean(s: &str, max: usize) -> String { + let mut out: String = s.chars().filter(|c| !c.is_control()).collect(); + if out.len() > max { + let mut cut = max; + while !out.is_char_boundary(cut) { + cut -= 1; + } + out.truncate(cut); + out.push('…'); + } + out +} + +fn percent_decode(s: &str) -> String { + let hex = |c: u8| (c as char).to_digit(16).map(|d| d as u8); + let b = s.as_bytes(); + let mut out = vec![]; + let mut i = 0; + while i < b.len() { + match ( + b[i], + b.get(i + 1).copied().and_then(hex), + b.get(i + 2).copied().and_then(hex), + ) { + (b'%', Some(h), Some(l)) => { + out.push(h * 16 + l); + i += 3; + continue; + } + (b'+', ..) => out.push(b' '), + (c, ..) => out.push(c), + } + i += 1; + } + String::from_utf8_lossy(&out).into_owned() +} + +/// JSON as it was written: numbers keep their text, so an id past 2^53 reads as sent. +#[derive(Clone, Debug, PartialEq)] +enum Json { + Lit(String), + Str(String), + Arr(Vec), + Obj(Vec<(String, Json)>), +} + +impl Json { + fn parse(s: &str) -> Result { + let mut p = Parser { + s: s.as_bytes(), + i: 0, + depth: 0, + }; + let v = p.value()?; + p.ws(); + if p.i != p.s.len() { + return Err(format!("unexpected text at {}", p.i)); + } + Ok(v) + } + + /// Replace the values of the fields `secret` names, wherever they are. + fn redact(&mut self, secret: &dyn Fn(&str) -> bool) { + match self { + Json::Arr(a) => a.iter_mut().for_each(|v| v.redact(secret)), + Json::Obj(o) => { + for (k, v) in o { + if secret(k) && *v != Json::Lit("null".into()) { + *v = Json::Str(REDACTED.into()); + } else { + v.redact(secret); + } + } + } + _ => {} + } + } + + fn pretty(&self) -> String { + let mut out = String::new(); + self.write(&mut out, 0); + out + } + + fn write(&self, out: &mut String, indent: usize) { + let pad = |out: &mut String, n: usize| out.extend(std::iter::repeat_n(" ", n)); + match self { + Json::Lit(s) => out.push_str(s), + Json::Str(s) => quote(out, s), + Json::Arr(a) if a.is_empty() => out.push_str("[]"), + Json::Obj(o) if o.is_empty() => out.push_str("{}"), + Json::Arr(a) => { + out.push_str("[\n"); + for (i, v) in a.iter().enumerate() { + pad(out, indent + 1); + v.write(out, indent + 1); + out.push_str(if i + 1 < a.len() { ",\n" } else { "\n" }); + } + pad(out, indent); + out.push(']'); + } + Json::Obj(o) => { + out.push_str("{\n"); + for (i, (k, v)) in o.iter().enumerate() { + pad(out, indent + 1); + quote(out, k); + out.push_str(": "); + v.write(out, indent + 1); + out.push_str(if i + 1 < o.len() { ",\n" } else { "\n" }); + } + pad(out, indent); + out.push('}'); + } + } + } +} + +fn quote(out: &mut String, s: &str) { + out.push('"'); + for c in s.chars() { + match c { + '"' => out.push_str("\\\""), + '\\' => out.push_str("\\\\"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + '\t' => out.push_str("\\t"), + c if (c as u32) < 0x20 => { + let _ = write!(out, "\\u{:04x}", c as u32); + } + c => out.push(c), + } + } + out.push('"'); +} + +struct Parser<'a> { + s: &'a [u8], + i: usize, + depth: usize, +} + +impl Parser<'_> { + fn ws(&mut self) { + while self.i < self.s.len() && matches!(self.s[self.i], b' ' | b'\t' | b'\n' | b'\r') { + self.i += 1; + } + } + + fn peek(&self) -> Option { + self.s.get(self.i).copied() + } + + fn eat(&mut self, c: u8) -> Result<(), String> { + self.ws(); + if self.peek() == Some(c) { + self.i += 1; + Ok(()) + } else { + Err(format!("expected `{}` at {}", c as char, self.i)) + } + } + + fn value(&mut self) -> Result { + self.ws(); + self.depth += 1; + if self.depth > 128 { + return Err("nested too deeply".into()); + } + let v = match self.peek() { + Some(b'{') => { + self.i += 1; + let mut fields = vec![]; + self.ws(); + if self.peek() == Some(b'}') { + self.i += 1; + } else { + loop { + self.ws(); + let k = self.string()?; + self.eat(b':')?; + fields.push((k, self.value()?)); + self.ws(); + match self.peek() { + Some(b',') => self.i += 1, + Some(b'}') => { + self.i += 1; + break; + } + _ => return Err(format!("expected `,` or `}}` at {}", self.i)), + } + } + } + Json::Obj(fields) + } + Some(b'[') => { + self.i += 1; + let mut items = vec![]; + self.ws(); + if self.peek() == Some(b']') { + self.i += 1; + } else { + loop { + items.push(self.value()?); + self.ws(); + match self.peek() { + Some(b',') => self.i += 1, + Some(b']') => { + self.i += 1; + break; + } + _ => return Err(format!("expected `,` or `]` at {}", self.i)), + } + } + } + Json::Arr(items) + } + Some(b'"') => Json::Str(self.string()?), + Some(_) => { + let start = self.i; + while self.i < self.s.len() + && matches!(self.s[self.i], b'-' | b'+' | b'.' | b'0'..=b'9' | b'a'..=b'z' | b'E') + { + self.i += 1; + } + let lit = std::str::from_utf8(&self.s[start..self.i]).unwrap_or(""); + let number = lit.parse::().is_ok() + && lit + .bytes() + .all(|c| matches!(c, b'-' | b'+' | b'.' | b'e' | b'E' | b'0'..=b'9')) + && lit + .bytes() + .next() + .is_some_and(|c| c == b'-' || c.is_ascii_digit()); + if !(number || matches!(lit, "true" | "false" | "null")) { + return Err(format!("unexpected `{lit}` at {start}")); + } + Json::Lit(lit.to_string()) + } + None => return Err("unexpected end".into()), + }; + self.depth -= 1; + Ok(v) + } + + fn string(&mut self) -> Result { + if self.peek() != Some(b'"') { + return Err(format!("expected a string at {}", self.i)); + } + self.i += 1; + let mut out = String::new(); + loop { + let start = self.i; + while self.i < self.s.len() && !matches!(self.s[self.i], b'"' | b'\\') { + if self.s[self.i] < 0x20 { + return Err(format!("a control character in a string at {}", self.i)); + } + self.i += 1; + } + out.push_str( + std::str::from_utf8(&self.s[start..self.i]).map_err(|_| "not UTF-8".to_string())?, + ); + match self.peek() { + Some(b'"') => { + self.i += 1; + return Ok(out); + } + Some(b'\\') => { + let c = *self.s.get(self.i + 1).ok_or("unexpected end")?; + self.i += 2; + match c { + b'"' => out.push('"'), + b'\\' => out.push('\\'), + b'/' => out.push('/'), + b'b' => out.push('\u{8}'), + b'f' => out.push('\u{c}'), + b'n' => out.push('\n'), + b'r' => out.push('\r'), + b't' => out.push('\t'), + b'u' => { + let mut code = self.hex4()?; + if (0xD800..0xDC00).contains(&code) + && self.s.get(self.i) == Some(&b'\\') + && self.s.get(self.i + 1) == Some(&b'u') + { + self.i += 2; + let low = self.hex4()?; + code = 0x10000 + + ((code - 0xD800) << 10) + + (low.wrapping_sub(0xDC00) & 0x3FF); + } + out.push(char::from_u32(code).unwrap_or('\u{FFFD}')); + } + _ => return Err(format!("a bad escape at {}", self.i)), + } + } + _ => return Err("an unterminated string".into()), + } + } + } + + fn hex4(&mut self) -> Result { + let h = self.s.get(self.i..self.i + 4).ok_or("unexpected end")?; + let h = std::str::from_utf8(h).map_err(|_| "a bad escape".to_string())?; + self.i += 4; + u32::from_str_radix(h, 16).map_err(|_| "a bad escape".into()) + } +} + +/// A field of the results file: `\`, tabs and line breaks escaped, so a body fits on one line. +pub(crate) fn escape(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + for c in s.chars() { + match c { + '\\' => out.push_str("\\\\"), + '\t' => out.push_str("\\t"), + '\n' => out.push_str("\\n"), + '\r' => out.push_str("\\r"), + c => out.push(c), + } + } + out +} + +pub(crate) fn unescape(s: &str) -> String { + let mut out = String::with_capacity(s.len()); + let mut chars = s.chars(); + while let Some(c) = chars.next() { + if c != '\\' { + out.push(c); + continue; + } + match chars.next() { + Some('t') => out.push('\t'), + Some('n') => out.push('\n'), + Some('r') => out.push('\r'), + Some(c) => out.push(c), + None => out.push('\\'), + } + } + out +} + +/// The lines [`crate::Report::save`] writes for an exchange of step `n`. +pub(crate) fn save(out: &mut String, n: usize, x: &Exchange) { + let _ = writeln!( + out, + "exchange\t{n}\t{}\t{}\t{}\t{}", + x.role.name(), + escape(&x.method), + escape(&x.url), + x.status + ); + for (k, v) in &x.request_headers { + let _ = writeln!(out, "request-header\t{}\t{}", escape(k), escape(v)); + } + if let Some(b) = &x.request_body { + let _ = writeln!(out, "request-body\t{}", escape(b)); + } + for (k, v) in &x.response_headers { + let _ = writeln!(out, "response-header\t{}\t{}", escape(k), escape(v)); + } + if let Some(b) = &x.response_body { + let _ = writeln!(out, "response-body\t{}", escape(b)); + } +} + +/// An `exchange` line read back: its step and the exchange, still without headers or bodies. +pub(crate) fn load(f: &[&str]) -> Option<(usize, Exchange)> { + let [_, n, role, method, url, status] = f else { + return None; + }; + let mut x = Exchange::new(unescape(method), unescape(url)); + x.role = Role::parse(role)?; + x.status = status.parse().ok()?; + Some((n.parse().ok()?, x)) +} + +/// What a status code says: `201` → `Created`. +pub fn reason(status: u16) -> &'static str { + match status { + 200 => "OK", + 201 => "Created", + 202 => "Accepted", + 204 => "No Content", + 301 => "Moved Permanently", + 302 => "Found", + 303 => "See Other", + 304 => "Not Modified", + 307 => "Temporary Redirect", + 308 => "Permanent Redirect", + 400 => "Bad Request", + 401 => "Unauthorized", + 402 => "Payment Required", + 403 => "Forbidden", + 404 => "Not Found", + 405 => "Method Not Allowed", + 406 => "Not Acceptable", + 409 => "Conflict", + 410 => "Gone", + 412 => "Precondition Failed", + 413 => "Content Too Large", + 415 => "Unsupported Media Type", + 422 => "Unprocessable Content", + 428 => "Precondition Required", + 429 => "Too Many Requests", + 500 => "Internal Server Error", + 501 => "Not Implemented", + 502 => "Bad Gateway", + 503 => "Service Unavailable", + 504 => "Gateway Timeout", + _ => "", + } +} + +/// The request as a `curl` command, one option per line. Variables (`$BASE_URL`, `$API_KEY`: a +/// `$` and capitals, digits and `_`) are left for the shell to fill in; any other `$` is +/// escaped. A JSON body is indented under the command. +pub fn curl(x: &Exchange) -> String { + let mut lines = vec![]; + let url = dq(&x.url); + lines.push(match (x.method.as_str(), &x.request_body) { + ("GET", None) => format!("curl {url}"), + (m, _) => format!("curl -X {m} {url}"), + }); + for (k, v) in &x.request_headers { + lines.push(format!(" -H {}", dq(&format!("{k}: {v}")))); + } + if let Some(b) = &x.request_body { + let json = x + .request_header("content-type") + .is_some_and(|t| t.split(';').next().unwrap_or("").trim().ends_with("json")); + // Whitespace between JSON tokens means nothing, so it can line up with the command. + let b = if json { + b.replace('\n', "\n ") + } else { + b.clone() + }; + lines.push(format!(" -d {}", sq(&b))); + } + lines.join(" \\\n") +} + +/// The response as HTTP shows it: the status line, the headers kept, a blank line and the body. +pub fn response(x: &Exchange) -> String { + if x.status == 0 { + return "(no response)".into(); + } + let mut out = format!("HTTP/1.1 {} {}", x.status, reason(x.status)) + .trim_end() + .to_string(); + for (k, v) in &x.response_headers { + let _ = write!(out, "\n{k}: {v}"); + } + if let Some(b) = x.response_body.as_deref().filter(|b| !b.is_empty()) { + let _ = write!(out, "\n\n{b}"); + } + out +} + +/// How long the variable `s` starts with is (`$BASE_URL`: a `$`, a capital, then capitals, +/// digits and `_`), if it starts with one. +fn variable(s: &str) -> Option { + let name = s.strip_prefix('$')?; + if !name.starts_with(|c: char| c.is_ascii_uppercase()) { + return None; + } + let len = name + .find(|c: char| !(c.is_ascii_uppercase() || c.is_ascii_digit() || c == '_')) + .unwrap_or(name.len()); + Some(1 + len) +} + +/// In double quotes, for the shell, leaving variables (`$BASE_URL`) to expand. +fn dq(s: &str) -> String { + let mut out = String::from("\""); + let mut rest = s; + while let Some(c) = rest.chars().next() { + if let Some(n) = variable(rest) { + out.push_str(&rest[..n]); + rest = &rest[n..]; + continue; + } + if matches!(c, '"' | '\\' | '`' | '$') { + out.push('\\'); + } + out.push(c); + rest = &rest[c.len_utf8()..]; + } + out.push('"'); + out +} + +/// In single quotes, for the shell: nothing in it expands. +fn sq(s: &str) -> String { + format!("'{}'", s.replace('\'', r"'\''")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn invoice() -> Exchange { + Exchange::new("post", "http://127.0.0.1:53211/v1/invoices?expand=customer&access_token=abc") + .header("Authorization", "Bearer sk_live_123") + .header("Cookie", "session=xyz") + .header("Content-Type", "application/json") + .header("X-Forwarded-For", "10.0.0.1") + .body(r#"{"customer":"cus_1","amount":12345678901234567890,"card":{"number":"4242","cvc":"123"},"client_secret":null}"#) + .response(201) + .response_header("Content-Type", "application/json; charset=utf-8") + .response_header("Set-Cookie", "session=abc") + .response_header("Location", "http://127.0.0.1:53211/v1/invoices/in_1") + .response_body(r#"{"id":"in_1","status":"draft","lines":[],"meta":{},"owner":{"email":"a@b.c","apiKey":"k"}}"#) + } + + #[test] + fn redaction_keeps_what_can_be_published() { + let r = Redaction { + fields: vec!["email".into()], + ..Default::default() + }; + let x = r.apply(&invoice()); + assert_eq!(x.method, "POST"); + assert_eq!( + x.url, + "$BASE_URL/v1/invoices?expand=customer&access_token=REDACTED" + ); + assert_eq!(x.label(), "POST /v1/invoices → 201"); + assert_eq!( + x.request_headers, + vec![ + ("Authorization".to_string(), "Bearer $API_KEY".to_string()), + ("Content-Type".to_string(), "application/json".to_string()), + ] + ); + assert_eq!( + x.request_body.as_deref(), + Some( + r#"{ + "customer": "cus_1", + "amount": 12345678901234567890, + "card": { + "number": "4242", + "cvc": "REDACTED" + }, + "client_secret": null +}"# + ) + ); + assert_eq!( + x.response_header_value("location"), + Some("$BASE_URL/v1/invoices/in_1") + ); + assert!(x.response_header_value("set-cookie").is_none()); + let body = x.response_body.unwrap(); + assert!(body.contains(r#""lines": []"#), "{body}"); + assert!(body.contains(r#""meta": {}"#), "{body}"); + assert!(body.contains(r#""email": "REDACTED""#), "{body}"); + assert!(body.contains(r#""apiKey": "REDACTED""#), "{body}"); + } + + #[test] + fn named_keys_say_whose_they_were() { + let r = Redaction { + keys: vec![ + ("sk_member".into(), "$MEMBER_API_KEY".into()), + ("sk_odd".into(), "$not a name".into()), + ], + ..Default::default() + }; + let auth = |key: &str| { + let x = Exchange::new("GET", "/").header("Authorization", format!("Bearer {key}")); + r.apply(&x).request_headers[0].1.clone() + }; + assert_eq!(auth("sk_member"), "Bearer $MEMBER_API_KEY"); + assert_eq!(auth("sk_admin"), "Bearer $API_KEY"); + assert_eq!(auth("sk_odd"), "Bearer $API_KEY"); + let x = Exchange::new("GET", "/").header("X-Api-Key", "sk_member"); + assert_eq!(r.apply(&x).request_headers[0].1, "$MEMBER_API_KEY"); + let c = curl(&r.apply( + &Exchange::new("GET", "$BASE_URL/").header("Authorization", "Bearer sk_member"), + )); + assert_eq!( + c, + "curl \"$BASE_URL/\" \\\n -H \"Authorization: Bearer $MEMBER_API_KEY\"" + ); + } + + #[test] + fn a_base_url_replaces_every_origin() { + let r = Redaction { + base_url: Some("https://api.example.com/".into()), + ..Default::default() + }; + let x = r.apply(&Exchange::new("GET", "http://10.1.2.3:9000/v1/x")); + assert_eq!(x.url, "https://api.example.com/v1/x"); + // Only loopback addresses are swapped without one. + let x = Redaction::default().apply(&Exchange::new("GET", "https://api.example.com/v1/x")); + assert_eq!(x.url, "https://api.example.com/v1/x"); + let x = Redaction::default().apply(&Exchange::new("GET", "http://[::1]:80")); + assert_eq!(x.url, "$BASE_URL"); + } + + #[test] + fn json_that_does_not_parse_is_not_shown() { + let x = Exchange::new("GET", "/x") + .response(200) + .response_header("Content-Type", "application/json") + .response_body(r#"{"password": "hunter2""#); + let x = Redaction::default().apply(&x); + let body = x.response_body.unwrap(); + assert!(!body.contains("hunter2"), "{body}"); + assert!(body.contains("isn't valid JSON"), "{body}"); + } + + #[test] + fn form_bodies_and_long_bodies() { + let x = Exchange::new("POST", "/oauth/token") + .header("Content-Type", "application/x-www-form-urlencoded") + .body("grant_type=client_credentials&client_secret=s3cr3t") + .response(200) + .response_header("Content-Type", "text/plain") + .response_body("é".repeat(100)); + let x = Redaction { + max_body: 60, + ..Default::default() + } + .apply(&x); + assert_eq!( + x.request_body.as_deref(), + Some("grant_type=client_credentials&client_secret=REDACTED") + ); + let body = x.response_body.unwrap(); + assert!(body.ends_with("… (140 more bytes)"), "{body}"); + } + + #[test] + fn secret_names() { + for yes in [ + "password", + "client_secret", + "accessToken", + "api-key", + "API_KEY", + "privateKey", + "refresh_token", + "card.cvc", + ] { + assert!(secret_name(yes), "{yes}"); + } + for no in [ + "customer", + "keyboard", + "monkey", + "status", + "tokenizer", + "key", + ] { + assert!(!secret_name(no), "{no}"); + } + } + + #[test] + fn curl_and_response() { + let x = Redaction::default().apply(&invoice()); + let c = curl(&x); + assert!( + c.starts_with( + "curl -X POST \"$BASE_URL/v1/invoices?expand=customer&access_token=REDACTED\" \\\n -H \"Authorization: Bearer $API_KEY\" \\\n" + ), + "{c}" + ); + assert!(c.contains(" -d '{\n \"customer\""), "{c}"); + assert!(c.ends_with("\n }'"), "{c}"); + let get = Exchange::new("GET", "$BASE_URL/a?q=$5 `x`"); + assert_eq!(curl(&get), r#"curl "$BASE_URL/a?q=\$5 \`x\`""#); + let quoted = Exchange::new("POST", "/a").body("it's"); + assert_eq!(curl("ed), "curl -X POST \"/a\" \\\n -d 'it'\\''s'"); + let r = response(&x); + assert!( + r.starts_with("HTTP/1.1 201 Created\nContent-Type: application/json; charset=utf-8\nLocation: $BASE_URL/v1/invoices/in_1\n\n{\n \"id\": \"in_1\""), + "{r}" + ); + assert_eq!(response(&Exchange::new("GET", "/")), "(no response)"); + assert_eq!( + response(&Exchange::new("GET", "/").response(299)), + "HTTP/1.1 299" + ); + } + + #[test] + fn json_keeps_what_was_written() { + let j = + Json::parse(r#" {"a":[1,-2.5e3,true,null],"b":"\u00e9\ud83d\ude00\"\n"} "#).unwrap(); + assert_eq!( + j.pretty(), + "{\n \"a\": [\n 1,\n -2.5e3,\n true,\n null\n ],\n \"b\": \"é😀\\\"\\n\"\n}" + ); + for bad in [ + "{", + "[1,]", + "{\"a\" 1}", + "nope", + "01x", + "\"\u{1}\"", + "[1] 2", + ] { + assert!(Json::parse(bad).is_err(), "{bad}"); + } + let deep = "[".repeat(200) + &"]".repeat(200); + assert!(Json::parse(&deep).is_err()); + } + + #[test] + fn escaping_round_trips() { + let s = "a\tb\nc\\d\re"; + assert_eq!(unescape(&escape(s)), s); + assert!(!escape(s).contains(['\t', '\n'])); + } +} diff --git a/crates/story/src/lib.rs b/crates/story/src/lib.rs index a29bfdb..830e861 100644 --- a/crates/story/src/lib.rs +++ b/crates/story/src/lib.rs @@ -5,6 +5,7 @@ pub mod codecov; pub mod controls; pub mod coverage; +pub mod exchange; pub mod plan; pub mod report; pub mod sha256; @@ -19,6 +20,7 @@ pub use coverage::{ Attribution, Completeness, Contract, Coverage, Credit, Evidence, Item, Judged, Note, Source, State, coverage_file, is_coverage_file, load_contracts_for, }; +pub use exchange::{Exchange, Redaction, Role}; pub use plan::{ Because, Check, Origin, Plan, PlanStep, Planned, coverage, coverage_plans, plan_stories, reaching_plans, untested, visit_plans, @@ -315,4 +317,39 @@ Background: generated.iter().map(|p| &p.title).collect::>() ); } + + #[test] + fn exchanges_are_saved_with_their_steps() { + let (_, _, p) = planned( + " +Story: Boil dry + When the dial is 80 % + When the water runs out + Then the kettle is dry +", + ); + let plans = &p.plans[..1]; + let mut r = PlanResult::with(Status::Passed, &plans[0]); + let mut setup = Exchange::new("PUT", "$BASE_URL/dial").body("{\n \"to\": 0\n}"); + setup.role = Role::Setup; + r.start.exchanges.push(setup); + let action = Exchange::new("POST", "$BASE_URL/water/out?a=1\tb") + .header("Content-Type", "application/json") + .response(204); + let mut check = Exchange::new("GET", "$BASE_URL/kettle") + .response(200) + .response_header("Content-Type", "application/json") + .response_body("{\n \"phase\": \"dry\\\\\"\n}"); + check.role = Role::Check; + r.steps[1].exchanges = vec![action, check]; + let report = Report { results: vec![r] }; + let text = report.save(plans); + let back = Report::load(&text, plans); + let (a, b) = (&report.results[0], &back.results[0]); + assert_eq!(a.start.exchanges, b.start.exchanges, "{text}"); + assert!(b.steps[0].exchanges.is_empty()); + assert_eq!(a.steps[1].exchanges, b.steps[1].exchanges, "{text}"); + // Every exchange is on lines of its own, so older readers skip them. + assert!(text.lines().all(|l| l.split('\t').count() <= 6), "{text}"); + } } diff --git a/crates/story/src/report.rs b/crates/story/src/report.rs index 661636a..2afee82 100644 --- a/crates/story/src/report.rs +++ b/crates/story/src/report.rs @@ -1,6 +1,7 @@ //! Results of running plans, and what they say about each expectation. use crate::controls::Control; +use crate::exchange::{self, Exchange}; use crate::plan::{Because, Check, Plan}; use expectation_model::{ExpId, Suite, Value}; use std::fmt::Write; @@ -228,6 +229,9 @@ pub struct StepResult { pub checks: Vec, /// Relative to the docs directory. pub screenshot: Option, + /// The HTTP requests it made and what came back, redacted to be published (see + /// [`crate::exchange`]): what the docs show of an API where they'd show a screenshot. + pub exchanges: Vec, pub error: Option, /// How many times the checks were observed before they passed or it gave up. pub polls: u32, @@ -248,6 +252,7 @@ impl StepResult { status, checks: vec![], screenshot: None, + exchanges: vec![], error: None, polls: 0, window: None, @@ -304,10 +309,12 @@ impl PlanResult { if r.stale { r.start.stale = true; r.start.screenshot = old.start.screenshot; + r.start.exchanges = old.start.exchanges; if old.steps.len() == plan.steps.len() { for (new, old) in r.steps.iter_mut().zip(old.steps) { new.stale = true; new.screenshot = old.screenshot; + new.exchanges = old.exchanges; } } else { for step in &mut r.steps { @@ -642,6 +649,9 @@ impl Report { if let Some(w) = step.window { let _ = writeln!(out, "window\t{n}\t{}\t{}", w.start_ms, w.end_ms); } + for x in &step.exchanges { + exchange::save(&mut out, n, x); + } for a in r.attachments.iter().filter(|a| a.step == n) { let _ = writeln!( out, @@ -714,6 +724,44 @@ impl Report { step.window = Some(Window { start_ms, end_ms }); } } + ["exchange", ..] => { + let Some((_, r)) = current.as_mut() else { + continue; + }; + let Some((n, x)) = exchange::load(&f) else { + continue; + }; + let step = if n == 0 { + Some(&mut r.start) + } else { + r.steps.get_mut(n - 1) + }; + if let Some(step) = step { + step.exchanges.push(x); + } + } + [ + kind + @ ("request-header" | "request-body" | "response-header" | "response-body"), + rest @ .., + ] => { + // Part of the exchange before it: the last one of the last step read. + let Some((_, r)) = current.as_mut() else { + continue; + }; + let step = r.steps.last_mut().unwrap_or(&mut r.start); + let Some(x) = step.exchanges.last_mut() else { + continue; + }; + let un = exchange::unescape; + match (*kind, rest) { + ("request-header", [k, v]) => x.request_headers.push((un(k), un(v))), + ("response-header", [k, v]) => x.response_headers.push((un(k), un(v))), + ("request-body", [b]) => x.request_body = Some(un(b)), + ("response-body", [b]) => x.response_body = Some(un(b)), + _ => {} + } + } ["attachment", n, media_type, path] => { if let (Some((_, r)), Ok(step)) = (current.as_mut(), n.parse()) { r.attachments.push(Attachment { @@ -769,6 +817,7 @@ impl Report { status: Status::parse(status).unwrap_or(Status::NotRun), checks: vec![], screenshot: opt(shot), + exchanges: vec![], error: opt(error), polls: polls.parse().unwrap_or(0), window: None, diff --git a/site/build.sh b/site/build.sh index dc479da..9496efb 100755 --- a/site/build.sh +++ b/site/build.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Builds the GitHub Pages site into target/pages (or $1): the landing page, the guide # (mdBook), the API reference (rustdoc), the kettle example's generated manual, and the -# multi-platform demo's manual, and the smart-kettle example's two manuals. +# multi-platform demo's manual, the smart-kettle example's two manuals, and the invoices API's +# manual. set -euo pipefail cd "$(dirname "$0")/.." out="${1:-target/pages}" @@ -12,6 +13,9 @@ out="$(cd "$out" && pwd)" # The example's manual, from a real run: tutorials, topics, the test report and screenshots. cargo run -q -p expectation_harness --example kettle -- "$out/example" +# An API's manual, from a real run against a stand-in API: curl commands where screenshots would be. +cargo run -q -p expectation_harness --example invoices -- "$out/invoices" + # The multi-platform demo's manual, from the results and screenshots its runs saved. mkdir -p "$out/demo" cp -R demos/kettle/docs/. "$out/demo/" diff --git a/site/llms.txt b/site/llms.txt index ad06535..c6ed5e0 100644 --- a/site/llms.txt +++ b/site/llms.txt @@ -24,6 +24,7 @@ For AI agents: install the skill at https://expectation.sim001.net/skills/expect - [Drivers](https://expectation.sim001.net/guide/drivers.html) - [Remote drivers](https://expectation.sim001.net/guide/remote.html): Swift, JavaScript and Rust clients, embedded in the app or in a UI-automation test process, driven by `expectation-run` - [The generated manual](https://expectation.sim001.net/guide/manual.html): includes custom themes and approved-only manuals +- [APIs: requests as evidence](https://expectation.sim001.net/guide/api.html): suites for an HTTP API, where the manual shows each step's request as a curl command and the response it got; what the runner redacts before it keeps them - [Evidence and contract coverage](https://expectation.sim001.net/guide/evidence.html): run ids, step windows and attachments for extensions; the coverage file format and the rules it is held to - [The expectation command](https://expectation.sim001.net/guide/cli.html) - [Diagnostics](https://expectation.sim001.net/guide/diagnostics.html) @@ -33,6 +34,7 @@ For AI agents: install the skill at https://expectation.sim001.net/skills/expect - [Smart kettle, a worked feature-docs suite](https://expectation.sim001.net/smart-kettle/site/index.html) ([approved only](https://expectation.sim001.net/smart-kettle-approved/site/index.html)) - [The kettle manual](https://expectation.sim001.net/example/site/index.html) - [One suite, three apps](https://expectation.sim001.net/demo/site/index.html) +- [An invoices API's manual](https://expectation.sim001.net/invoices/site/index.html): curl commands and responses where screenshots would be ## Optional