Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions book/src/drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,36 @@ A test whose start can't be set up (the reset or an arrangement fails) fails the
other steps are skipped. The runner still asks for a screenshot, so the report shows what the
system looked like when it went wrong. If there is none, the failure stands as it is.

### A run as a gate

A run passes when nothing failed. Tests the driver can't run yet, pending ones and ones that
weren't run don't count against it, so a suite can be adopted a binding at a time. A release
gate needs more: that the tests ran. Ask for it with a policy.

| Policy | Passes when | `expectation-run` |
|---|---|---|
| `Policy::Lenient` | nothing failed | the default |
| `Policy::Required` | every test that isn't pending ran and passed | `--strict` |
| `Policy::All` | every test ran and passed, with nothing pending | `--strict=all` |

```rust
acceptance.assert_passes_with(&mut driver, &opts, Policy::Required);

// Or with the report:
let report = acceptance.run(&mut driver, &opts);
if !report.meets(Policy::Required) {
eprint!("{}", report.failures(&acceptance.suite, &acceptance.plans));
eprint!("{}", report.incomplete(&acceptance.plans, Policy::Required));
}
```

- `report.incomplete` lists each test that didn't run with its status and why, and under
`Required` the pending ones it let by, apart.
- A run of no tests doesn't meet `Required` or `All`: it verified nothing.
- A policy judges the run that was just made. The report `publish` returns is merged with what
earlier runs saved, and a pass saved earlier doesn't stand in for a test that didn't run now.
- An expectation's approval is about the spec, not the run. It plays no part.

### Several runners

`Acceptance::publish(dir, runner, &report)` saves a run's results as
Expand Down
2 changes: 1 addition & 1 deletion book/src/remote.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ screenshot "$1"'` takes screenshots from outside, for apps that don't take their
(see [Screenshots in large projects](manual.md#screenshots-in-large-projects)).

It exits 1 if a test failed and 0 otherwise, **including when every test was "not supported
yet"**. A failed screenshot is only a warning on stderr. Read the per-plan lines it prints
yet"**, unless the run is `--strict` (see [A run as a gate](drivers.md#a-run-as-a-gate)). A failed screenshot is only a warning on stderr. Read the per-plan lines it prints
(`✓` passed, `✗` failed, `–` anything else) or the `plan` rows in
`docs/results/<runner>@<platform>.tsv`, and check the ones you expect to pass, before calling
anything verified.
Expand Down
29 changes: 25 additions & 4 deletions crates/harness/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ use expectation_graph::Analysis;
use expectation_language::Loaded;
use expectation_model::diagnostic::{has_errors, render_all};
use expectation_model::{Diagnostic, Suite};
pub use expectation_story::Control;
pub use expectation_story::{Control, Policy};
use expectation_story::{Plan, Report};
use std::path::Path;

Expand Down Expand Up @@ -286,8 +286,22 @@ impl Acceptance {
}

/// Run every plan, write the docs to `opts.docs_dir` (if set) whatever happened, and panic
/// with what failed. For `#[test]`s.
/// with what failed. For `#[test]`s. Tests that aren't supported yet, are pending or weren't
/// run don't fail it: see [`Acceptance::assert_passes_with`] for a gate that wants them run.
pub fn assert_passes(&self, driver: &mut dyn Driver, opts: &RunOptions) -> Report {
self.assert_passes_with(driver, opts, Policy::Lenient)
}

/// As [`Acceptance::assert_passes`], held to `policy`: with [`Policy::Required`] it also
/// panics when a test that isn't pending didn't run (the driver can't yet, a service is
/// missing), and with [`Policy::All`] when anything is pending too. It judges this run:
/// results saved by earlier ones don't stand in for a test that didn't run now.
pub fn assert_passes_with(
&self,
driver: &mut dyn Driver,
opts: &RunOptions,
policy: Policy,
) -> Report {
let report = self.run(driver, opts);
if let Some(dir) = &opts.docs_dir {
self.docs(Some(&report))
Expand All @@ -296,8 +310,15 @@ impl Acceptance {
}
if !report.passed() {
panic!(
"acceptance tests failed:\n{}",
report.failures(&self.suite, &self.plans)
"acceptance tests failed:\n{}{}",
report.failures(&self.suite, &self.plans),
report.incomplete(&self.plans, policy)
);
}
if !report.meets(policy) {
panic!(
"acceptance tests didn't all run:\n{}",
report.incomplete(&self.plans, policy)
);
}
report
Expand Down
200 changes: 200 additions & 0 deletions crates/harness/tests/strict.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
//! A gate that wants the tests to have run: what isn't supported yet, or wasn't run, stops it,
//! where an ordinary run lets it by.

use expectation_harness::{Acceptance, Bindings, Policy, RunOptions};
use expectation_story::{Report, Status};

const LAMP: &str = "
Vocabulary:
State: the lamp is off | on
State: the dimmer is low | high
Event: the user flips the switch
Event: the user turns the dimmer
Background:
Given the lamp is off
And the dimmer is low
Feature: Lamp
Expectation: Flipping turns it on
Given the lamp is off
When the user flips the switch
Then the lamp is on
Expectation: Turning the dimmer brightens it
Given the dimmer is low
When the user turns the dimmer
Then the dimmer is high
Story: Light
When the user flips the switch
Then the lamp is on
Story: Bright
When the user turns the dimmer
Then the dimmer is high
";

const PENDING: &str = "
Feature: Dimming
Expectation: Turning the dimmer again dims it
Status: pending
Given the dimmer is high
When the user turns the dimmer
Then the dimmer is low
Story: Dim again
Given the dimmer is high
When the user turns the dimmer
Then the dimmer is low
";

fn lamp(src: &str) -> Acceptance {
let loaded = expectation_language::load_sources("lamp", &[("lamp.expect", src)]);
Acceptance::from_loaded(loaded).unwrap_or_else(|e| panic!("{e}"))
}

/// A lamp whose dimmer the driver can turn, or not yet.
fn driver(dimmer: bool, broken: bool) -> Bindings<(bool, bool)> {
let b = Bindings::new((false, false))
.observe(
"the lamp",
|w: &(bool, bool)| if w.0 { "on" } else { "off" },
)
.observe(
"the dimmer",
|w: &(bool, bool)| if w.1 { "high" } else { "low" },
)
.arrange("the lamp", |w, v| w.0 = v.to_string() == "on")
.arrange("the dimmer", |w, v| w.1 = v.to_string() == "high")
.on("the user flips the switch", move |w| w.0 = !broken)
.on_reset(|w| *w = (false, false));
if dimmer {
b.on("the user turns the dimmer", |w| w.1 = true)
} else {
b
}
}

fn status(a: &Acceptance, report: &Report, title: &str) -> Status {
let i = a.plans.iter().position(|p| p.title == title).unwrap();
report.results[i].status
}

#[test]
fn a_test_the_driver_cant_run_passes_an_ordinary_run_and_stops_a_strict_one() {
let a = lamp(LAMP);
let report = a.run(&mut driver(false, false), &RunOptions::default());
assert_eq!(status(&a, &report, "Light"), Status::Passed);
assert_eq!(status(&a, &report, "Bright"), Status::Unsupported);

// As it always was.
assert!(report.passed());
assert!(report.meets(Policy::Lenient));
assert_eq!(report.incomplete(&a.plans, Policy::Lenient), "");
// Required execution is incomplete, and it says which test and why.
assert!(!report.meets(Policy::Required));
assert!(!report.meets(Policy::All));
let why = report.incomplete(&a.plans, Policy::Required);
assert!(
why.contains("– unsupported: Bright (The driver can't"),
"{why}"
);
assert!(!why.contains("Light"), "{why}");

// With the dimmer bound, everything runs and every policy is met.
let report = a.run(&mut driver(true, false), &RunOptions::default());
for policy in [Policy::Lenient, Policy::Required, Policy::All] {
assert!(report.meets(policy));
assert_eq!(report.incomplete(&a.plans, policy), "");
}
}

#[test]
fn pending_is_let_by_unless_everything_is_asked_for_and_is_listed_apart() {
let a = lamp(&format!("{LAMP}{PENDING}"));
let report = a.run(&mut driver(true, false), &RunOptions::default());
assert_eq!(status(&a, &report, "Dim again"), Status::Pending);
assert!(report.meets(Policy::Lenient));
assert!(report.meets(Policy::Required));
assert_eq!(report.incomplete(&a.plans, Policy::Required), "");
assert!(!report.meets(Policy::All));
let why = report.incomplete(&a.plans, Policy::All);
assert!(
why.contains("– pending: Dim again (Pending: it depends on"),
"{why}"
);

// With something else missing, the pending test is named, but not as what is missing.
let report = a.run(&mut driver(false, false), &RunOptions::default());
let why = report.incomplete(&a.plans, Policy::Required);
let (missing, pending) = why.split_once("Pending, and not required:\n").expect(&why);
assert!(missing.contains("– unsupported: Bright"), "{why}");
assert!(!missing.contains("Dim again"), "{why}");
assert!(pending.contains("– pending: Dim again"), "{why}");
}

#[test]
fn a_failure_fails_every_policy_and_is_told_as_a_failure() {
let a = lamp(LAMP);
let report = a.run(&mut driver(true, true), &RunOptions::default());
assert_eq!(status(&a, &report, "Light"), Status::Failed);
for policy in [Policy::Lenient, Policy::Required, Policy::All] {
assert!(!report.meets(policy));
// What failed is in `failures`, once.
assert_eq!(report.incomplete(&a.plans, policy), "");
}
assert!(report.failures(&a.suite, &a.plans).contains("✗ Light"));
}

#[test]
fn what_other_runs_saved_doesnt_stand_in_for_a_test_that_didnt_run() {
let a = lamp(LAMP);
let before = a.run(&mut driver(true, false), &RunOptions::default());
assert!(before.meets(Policy::Required));
let dir = std::path::Path::new(env!("CARGO_TARGET_TMPDIR")).join("strict-merged");
let _ = std::fs::remove_dir_all(&dir);

// An earlier run saved a pass for every test; this one can't run the dimmer's.
a.publish(&dir, "earlier", &before).unwrap();
let now = a.run(&mut driver(false, false), &RunOptions::default());
let merged = a.publish(&dir, "now", &now).unwrap();
// The manual has the earlier pass, and the gate still sees that this run didn't make it.
assert_eq!(status(&a, &merged, "Bright"), Status::Passed);
assert!(!now.meets(Policy::Required));
}

#[test]
fn a_run_of_no_tests_verified_nothing() {
let none = Report::default();
assert!(none.passed());
assert!(none.meets(Policy::Lenient));
for policy in [Policy::Required, Policy::All] {
assert!(!none.meets(policy));
assert!(
none.incomplete(&[], policy)
.contains("No tests were selected")
);
}
// And tests that were never run aren't complete either.
let a = lamp(LAMP);
let not_run = Report::not_run(&a.plans);
assert!(not_run.passed());
assert!(!not_run.meets(Policy::Required));
assert!(
not_run
.incomplete(&a.plans, Policy::Required)
.contains("– not-run: Light")
);
}

#[test]
#[should_panic(expected = "acceptance tests didn't all run")]
fn a_strict_assertion_panics_when_a_test_didnt_run() {
lamp(LAMP).assert_passes_with(
&mut driver(false, false),
&RunOptions::default(),
Policy::Required,
);
}

#[test]
fn an_ordinary_assertion_still_lets_it_by() {
let a = lamp(LAMP);
let report = a.assert_passes(&mut driver(false, false), &RunOptions::default());
assert_eq!(report.count(Status::Unsupported), 1);
}
25 changes: 20 additions & 5 deletions crates/remote/src/bin/expectation-run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
//! ```text
//! expectation-run <suite-dir> [--platform NAME] [--addr 127.0.0.1:4477] [--wait SECONDS]
//! [--docs DIR] [--runner NAME] [--settle MS] [--screenshot-cmd "COMMAND"]
//! [--attachments DIR] [--screenshot-store DIR]
//! [--attachments DIR] [--screenshot-store DIR] [--strict[=all]]
//! ```
//!
//! It listens, waits for the app to say hello, runs every story and a generated test for each
Expand All @@ -15,9 +15,12 @@
//! `DIR/<run>/<plan>/<step>/<name>`; keep it outside `--docs`, which is for publishing.
//! `--screenshot-store` keeps screenshots by content in `DIR` (`<sha256>.png`), so identical
//! frames are stored once and unchanged tests write nothing new; the docs take them from there.
//! Exits 1 if anything failed.
//! Exits 1 if anything failed. Tests that aren't supported yet, are pending or didn't run don't
//! fail it, unless it is `--strict`: then every test that isn't pending must have run and passed
//! in this run (`--strict=all`: every test, pending or not), and there must be at least one.
//! Results saved by earlier runs don't count towards it.

use expectation_harness::{Acceptance, RunOptions, Screenshot};
use expectation_harness::{Acceptance, Policy, RunOptions, Screenshot};
use expectation_remote::{DEFAULT_ADDR, RemoteDriver};
use expectation_story::Status;
use std::path::PathBuf;
Expand All @@ -26,7 +29,7 @@ use std::time::Duration;

const USAGE: &str = "usage: expectation-run <suite-dir> [--platform NAME] [--addr HOST:PORT] [--wait SECONDS] \
[--command-timeout SECONDS] \
[--stylesheet CSS] [--reader] [--state-shot-ratio W:H] [--docs DIR] [--runner NAME] [--settle MS] [--screenshot-cmd COMMAND] [--attachments DIR] [--screenshot-store DIR]";
[--stylesheet CSS] [--reader] [--state-shot-ratio W:H] [--docs DIR] [--runner NAME] [--settle MS] [--screenshot-cmd COMMAND] [--attachments DIR] [--screenshot-store DIR] [--strict[=all]]";

fn main() {
let mut args = std::env::args().skip(1);
Expand All @@ -53,6 +56,7 @@ fn main() {
);
let (mut stylesheet, mut ratio, mut attachments, mut store) = (None, None, None, None);
let mut reader = false;
let mut policy = Policy::Lenient;
while let Some(a) = args.next() {
let mut value = || {
args.next()
Expand Down Expand Up @@ -87,6 +91,12 @@ fn main() {
Some(expectation_harness::parse_ratio(&value()).unwrap_or_else(|e| fail(&e)))
}
"--reader" => reader = true,
"--strict" => policy = Policy::Required,
_ if a.starts_with("--strict=") => {
policy = Policy::parse(&a["--strict=".len()..]).unwrap_or_else(|| {
fail("--strict is --strict, --strict=required or --strict=all")
})
}
"--docs" => docs = Some(PathBuf::from(value())),
"--runner" => runner = Some(value()),
"--settle" => {
Expand Down Expand Up @@ -196,11 +206,16 @@ fn main() {
)),
}
}
if !report.passed() {
// This run's own report: what earlier runs saved doesn't stand in for a test that didn't run.
if !report.meets(policy) {
eprint!(
"\n{}",
report.failures(&acceptance.suite, &acceptance.plans)
);
let incomplete = report.incomplete(&acceptance.plans, policy);
if !incomplete.is_empty() {
eprint!("\nNot every test ran and passed (--strict):\n{incomplete}");
}
exit(1);
}
}
Expand Down
Loading
Loading