Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,12 +131,14 @@ fn behaves_as_expected() {
| --- | --- |
| [turnstile](crates/harness/examples/turnstile.rs) | The smallest whole run: an inline suite, closures for the driver, one story and a generated test. |
| [kettle](crates/harness/examples/kettle) | A suite across several files: two exceptions, four stories, three generated tests, a toy world with time in it, SVG screenshots and the generated manual. |
| [invoices](crates/harness/examples/invoices) | An API's manual: the driver calls a stand-in invoices API over HTTP and hands over each request, and the manual shows them as `curl` commands with their responses, where a screenshot would be. Admin and member API keys, and an exception only a member's request meets. |
| [quote](crates/harness/examples/quote) | Contract coverage end to end: the driver hands over each step's HTTP requests as HAR, [the HAR and OpenAPI extension](extensions/har-openapi) holds them against the contract, and the manual shows what the tests saw. |

```bash
cargo run -p expectation_harness --example turnstile
cargo run -p expectation_harness --example kettle # docs in target/expectation-docs/kettle
cargo run -p expectation_harness --example kettle -- out --bug # a seeded defect, to see a failure
cargo run -p expectation_harness --example invoices # an API's manual, in target/expectation-docs/invoices
cargo run -p expectation_cli -- check crates/harness/examples/kettle # check it without running anything
cargo test --workspace # the examples, as tests too
git config core.hooksPath .githooks # check fmt and clippy before each commit
Expand Down
1 change: 1 addition & 0 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
- [Remote drivers](remote.md)
- [External services](services.md)
- [The generated manual](manual.md)
- [APIs: requests as evidence](api.md)
- [Evidence and contract coverage](evidence.md)

# Reference
Expand Down
150 changes: 150 additions & 0 deletions book/src/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# APIs: requests as evidence

A suite can describe a system that has no screen: an HTTP API. The vocabulary, the
expectations, the stories and the checks work as they do for an app. Only the evidence is
different. Where an app's manual shows a screenshot of each step, an API's manual shows the
request the step sent, as a `curl` command, and the response it got.

```sh
curl -X POST "$BASE_URL/v1/invoices/in_0002/void" \
-H "Authorization: Bearer $MEMBER_API_KEY"
```

```http
HTTP/1.1 403 Forbidden
Content-Type: application/json

{
"error": {
"type": "forbidden",
"message": "Only an admin of the account can void an invoice."
}
}
```

Every example in the manual is a request a test made against the API, so it can't drift away
from what the API really does. [`crates/harness/examples/invoices`](https://github.com/ptescher/expectation/tree/main/crates/harness/examples/invoices)
runs the whole thing against a stand-in invoices API:

```bash
cargo run -p expectation_harness --example invoices # docs in target/expectation-docs/invoices
cargo run -p expectation_harness --example invoices -- out --bug # members can void: a failure
```

## Writing the suite

Write it about what callers do and what the API promises, as you would for an app. The events
are what a caller asks for ("the caller voids the invoice"), and the states are what a caller
can read back ("the invoice is open"). Who is calling is a state too, so the rules that depend
on it can be written as exceptions:

```gherkin
Vocabulary:
State: the invoice is none | draft | open | void
Shows: void: the invoice's `status` is `void`
State: the caller is admin | member
Set: use the {value}'s API key
Event: the caller voids the invoice
Say: void the invoice

Feature: Voiding
Expectation: Voiding an open invoice voids it
Given the invoice is open
When the caller voids the invoice
Then the invoice is void

Expectation: Members can't void
Overrides: Voiding an open invoice voids it
Given the invoice is open
And the caller is member
When the caller voids the invoice
Then the invoice is open
```

`expectation check` finds the permission rule that conflicts with the rule it is an exception
to, as it would for any two expectations that promise different things in the same situation.

## The driver

The driver calls the API with whatever HTTP client you use, and keeps each request it sends
with the response it got as an `Exchange`. The runner takes them with `on_exchanges` (or
`Driver::take_exchanges`):

```rust,ignore
use expectation_harness::{Bindings, Exchange};

let x = Exchange::new("POST", format!("{base}/v1/invoices/{id}/void"))
.header("Authorization", format!("Bearer {key}"))
.response(403)
.response_header("Content-Type", "application/json")
.response_body(body);

Bindings::new(World::new())
.observe("the invoice", World::status) // GET /v1/invoices/{id}
.on("the caller voids the invoice", World::void)
.on_exchanges(|w| std::mem::take(&mut w.log))
```

Hand them over as they were sent, credentials and all: the runner redacts them (below). It asks
after arranging a test's start, after each action and after each step's checks, so it knows
what each request was for:

| Role | When | Where the manual shows it |
| --- | --- | --- |
| `setup` | while the test's start was arranged | on the test's page, folded away |
| `action` | while the step's action ran | on the tutorial, the feature page and the test's page |
| `check` | while the step's Thens were read back | on the test's page, folded away |

The same request sent again in a row, such as a check that polled until it held, is shown
once, with how many times it was sent.

## What is kept

Exchanges are kept in the results file (`results/api.tsv`), which is published with the
manual. So the runner keeps only what `RunOptions::redaction` lets through:

- Only a few headers: `Accept`, `Content-Type`, `Idempotency-Key`, `If-Match`, `If-None-Match`
and `Prefer` on requests, and `Content-Type`, `ETag`, `Location` and `Retry-After` on
responses. Cookies and anything a gateway added are dropped.
- `Authorization` keeps its scheme, and the key becomes `$API_KEY`, or the name you give it in
`keys`. So does the value of an `X-Api-Key` or `Api-Key` header.
- Query parameters, form fields and JSON fields named like a secret (`password`, `token`,
`client_secret`, `api_key`, `cvc` and the like), and those you list in `fields`, read
`REDACTED`.
- A loopback address (`http://127.0.0.1:53211`) becomes `$BASE_URL`. Set `base_url` to show
another origin instead, such as `https://api.example.com`.
- JSON bodies are pretty-printed, with numbers kept as they were written. A JSON body that
doesn't parse can't be checked for secrets, so it is replaced with a note. A body over
`max_body` (16 KiB) is cut short.

```rust,ignore
RunOptions {
redaction: Redaction {
keys: vec![
(ADMIN_KEY.into(), "$ADMIN_API_KEY".into()),
(MEMBER_KEY.into(), "$MEMBER_API_KEY".into()),
],
fields: vec!["email".into()],
..Default::default()
},
..Default::default()
}
```

Name the keys when who is calling matters, so the reader can tell an admin's request from a
member's. A name is a `$` followed by capitals, digits and `_`. The `curl` commands leave those
for the shell to fill in, and escape any other `$`.

Redaction is by name and by header, so it can't know that a field called `note` holds a
customer's address. Read what the manual shows before you publish it, and add such fields to
`fields`.

## Limits

- Bodies are text. Hand over a binary body as a note of what it was.
- The [remote drivers](remote.md) (Swift, JavaScript, the embedded Rust client) don't send
exchanges yet.
- A step with both a screenshot and requests shows both: an app that calls an API can show its
screen and its requests.
- Exchanges are for the manual. To hold the traffic against an OpenAPI document, capture it as
HAR too and use [the HAR and OpenAPI extension](evidence.md#the-har-and-openapi-extension).
8 changes: 8 additions & 0 deletions book/src/drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Terms are matched the way steps are: ignoring case, extra spaces and a leading "
| `on_settle(f)` | Letting the system settle after each action, before anything is observed. |
| `on_advance(f)` | Letting a little time pass before the runner looks again. |
| `on_screenshot(f)` | A picture of the system after each step: `Screenshot::png`, `jpeg` or `svg`. |
| `on_exchanges(f)` | The HTTP requests the system made since the last call, with their responses: what an API's manual shows instead of a screenshot. See [APIs](api.md). |
| `needs(&[…])`, `fresh(&[…])` | The [external services](services.md) the tests need. |

The values passed to `set` and `arrange` are `Value`s: `v.as_str()` for a named value,
Expand Down Expand Up @@ -115,6 +116,12 @@ A driver can hand over files it captured during each step, such as a network log
Extensions use them to report which outcomes of a contract the tests saw. See
[Evidence and contract coverage](evidence.md).

## HTTP exchanges

A driver for an API hands over the requests it sent and the responses it got, with
`Bindings::on_exchanges` or `Driver::take_exchanges`. The runner redacts them and keeps them in
the results, and the manual shows each step's request as a `curl` command. See [APIs](api.md).

## The Driver trait

For full control, implement `Driver` yourself. Only `set`, `perform` and `observe` are
Expand All @@ -133,6 +140,7 @@ pub trait Driver {
fn can_arrange(&self, var: &Variable, value: &Value) -> bool { true }
fn settle(&mut self) {}
fn advance(&mut self) -> Progress { Progress::Static }
fn take_exchanges(&mut self) -> Vec<Exchange> { vec![] }
fn screenshot(&mut self) -> Result<Option<Screenshot>, String> { Ok(None) }
fn supports(&self, need: Need<'_>) -> bool { true }
}
Expand Down
94 changes: 82 additions & 12 deletions crates/docs/src/html.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ use expectation_graph::Node;
use expectation_model::text::paragraphs;
use expectation_model::{ExpId, FeatureId, Keyword, PhraseBody, SectionId, StepText, Suite};
use expectation_story::{
Attribution, Basis, Because, CaptureState, Check, CheckResult, Completeness, Credit, Origin,
Source, State, Status, StepResult, Verdict,
Attribution, Basis, Because, CaptureState, Check, CheckResult, Completeness, Credit, Exchange,
Origin, Role, Source, State, Status, StepResult, Verdict, exchange,
};
use std::collections::HashMap;
use std::fmt::Write;
Expand Down Expand Up @@ -1054,6 +1054,44 @@ impl Site<'_> {
}
}

/// Requests, each as a `curl` command with the response it got, where a screenshot would
/// be. The same request sent again in a row (a check that polled) is shown once.
fn exchanges(&self, xs: &[&Exchange], stale: bool, failed: bool) -> String {
let mut o = String::new();
for (x, times) in crate::collapse(xs) {
let response = exchange::response(x);
let (status_line, rest) = response.split_once('\n').unwrap_or((&response, ""));
let class = match x.status {
0 | 500.. => "s5",
400..=499 => "s4",
_ => "s2",
};
let mut notes = vec![];
if times > 1 {
notes.push(format!("Sent {times} times."));
}
if stale {
notes.push("From before this page changed.".into());
} else if failed {
notes.push("What came back when this failed.".into());
}
let caption = if notes.is_empty() {
String::new()
} else {
format!("<figcaption>{}</figcaption>", esc(&notes.join(" ")))
};
let _ = write!(
o,
r#"<figure class="exchange"><pre class="http request" aria-label="Request"><code>{}</code></pre><pre class="http response {class}" aria-label="Response"><code><span class="status">{}</span>{}{}</code></pre>{caption}</figure>"#,
esc(&exchange::curl(x)),
esc(status_line),
if rest.is_empty() { "" } else { "\n" },
esc(rest),
);
}
o
}

/// Verdict counts for a set of expectations: "12 verified · 3 not supported yet".
fn tally(&self, exps: &[ExpId]) -> String {
let mut counts: Vec<(&str, &str, usize)> = vec![];
Expand Down Expand Up @@ -1404,8 +1442,13 @@ impl Site<'_> {
let _ = write!(o, r#"<p class="how">{}</p>"#, self.prose(&how));
o.push_str(&self.exception_notes(e));
match self.f.shot_of(self.d, e) {
Some((_, _, shot)) => {
o.push_str(&self.figure(Some(&shot), Status::Passed, false, false, &how))
Some((p, step, shot)) => {
if let Some(shot) = &shot {
o.push_str(&self.figure(Some(shot), Status::Passed, false, false, &how));
}
let r = crate::step_result(self.d, p, step);
let xs = r.map(|r| crate::actions(&r.exchanges)).unwrap_or_default();
o.push_str(&self.exchanges(&xs, false, false));
}
None => {
let status = self
Expand Down Expand Up @@ -1481,7 +1524,11 @@ impl Site<'_> {
} else {
&step.seen
};
o.push_str(&self.figure(step.shot.as_deref(), step.status, step.stale, false, alt));
if step.shot.is_some() || step.exchanges.is_empty() {
o.push_str(&self.figure(step.shot.as_deref(), step.status, step.stale, false, alt));
}
let xs: Vec<&Exchange> = step.exchanges.iter().collect();
o.push_str(&self.exchanges(&xs, step.stale, step.status == Status::Failed));
o.push_str("</li>");
}
o.push_str("</ol>");
Expand Down Expand Up @@ -2223,13 +2270,29 @@ impl Site<'_> {
plan_status: Status,
) -> String {
let status = result.map_or(plan_status, |r| r.status);
let mut o = self.figure(
result.and_then(|r| r.screenshot.as_deref()),
status,
result.is_some_and(|r| r.stale),
true,
"",
);
let stale = result.is_some_and(|r| r.stale);
let shot = result.and_then(|r| r.screenshot.as_deref());
let all: &[Exchange] = result.map_or(&[], |r| &r.exchanges);
let mut o = if shot.is_some() || all.is_empty() {
self.figure(shot, status, stale, true, "")
} else {
String::new()
};
o.push_str(&self.exchanges(&crate::actions(all), stale, status == Status::Failed));
for (role, summary) in [
(Role::Setup, "Requests that set it up"),
(Role::Check, "Requests that checked it"),
] {
let xs: Vec<&Exchange> = all.iter().filter(|x| x.role == role).collect();
if !xs.is_empty() {
let _ = write!(
o,
r#"<details class="requests"><summary>{summary} ({})</summary>{}</details>"#,
xs.len(),
self.exchanges(&xs, stale, false)
);
}
}
if let Some(e) = result
.and_then(|r| r.error.as_ref())
.filter(|_| status == Status::Failed)
Expand Down Expand Up @@ -2933,6 +2996,13 @@ section.exp:first-of-type { border-top: 0; }
figure { margin: 14px 0; max-width: 870px; padding: var(--bezel-pad); background: var(--bezel); border: 1px solid var(--bezel-edge); border-radius: var(--radius-panel); }
figure img { display: block; width: 100%; aspect-ratio: 16 / 9; object-fit: cover; border-radius: var(--radius-screen); background: var(--screen); }
figcaption { color: var(--muted); font-size: 12px; padding: 7px 4px 2px; }
/* A request and its response: two screens in one bezel. */
figure.exchange pre.http { margin: 0; padding: 12px 16px; overflow-x: auto; background: var(--screen); color: var(--text); border-radius: var(--radius-screen); font: 13px/1.55 var(--font-mono); white-space: pre; }
figure.exchange pre.http + pre.http { margin-top: var(--bezel-pad); }
figure.exchange pre.response .status { font-weight: 700; color: var(--ok); }
figure.exchange pre.response.s4 .status, figure.exchange pre.response.s5 .status { color: var(--warning); }
details.requests { margin: 10px 0; }
details.requests > summary { cursor: pointer; color: var(--muted); font-size: 13px; }
/* A missing screenshot is a gap to note, not something to look at: keep it small. */
figure.placeholder { max-width: 360px; }
dl.meta { display: grid; grid-template-columns: max-content 1fr; gap: 4px 16px; font-size: 15px; margin: 12px 0 0; }
Expand Down
Loading
Loading