Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion book/src/drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ pub trait Driver {
fn needs(&self) -> Vec<String> { vec![] }
fn fresh(&self) -> Vec<String> { vec![] }
fn reset(&mut self) -> Result<(), String> { Ok(()) }
fn may_continue(&self) -> bool { true }
fn arrange(&mut self, var: &Variable, value: &Value) -> Result<(), String> { self.set(var, value) }
fn set(&mut self, var: &Variable, value: &Value) -> Result<(), String>;
fn perform(&mut self, event: &Event) -> Result<(), String>;
Expand Down Expand Up @@ -162,6 +163,34 @@ A test whose start can't be set up (the reset or an arrangement fails) fails the
other steps are skipped. The runner still asks for a screenshot, so the report shows what the
system looked like when it went wrong. If there is none, the failure stands as it is.

### Starting where the last test ended

Resetting and arranging a start can be the slow part of a test. So when a test has passed, and
everything the next one is given already holds, the runner starts the next one there: no
reset, nothing arranged. It decides by looking. It observes each thing the test is given,
once, and only continues if all of them hold as they are. It never continues from a test that
failed, into a test that is given nothing, or across a `Launch:` that starts the program
again.

To make the most of this, the runner picks the next test: one that starts where the last one
ended if there is one, and otherwise the next in order. The report stays in the order of the
tests. Each result says how the test began (`PlanResult::began`, a `began` line in the saved
results), and the manual's page for a test that continued says so. If a share looks wrong,
that is where to look first.

The check only sees what the suite describes. Say no when a test leaves something behind that
no state of the suite covers and that the next test could notice: a log, a clock, a cache, a
counter.

```rust,ignore
fn may_continue(&self) -> bool { false } // a Driver
Bindings::new(world).always_reset() // Bindings
RunOptions { share_starts: false, ..Default::default() } // one run
```

A driver that asks for services afresh before each test (`fresh`) never continues: those
services must not keep what the last test left.

### A run as a gate

A run passes when nothing failed. Tests the driver can't run yet, pending ones and ones that
Expand Down Expand Up @@ -218,5 +247,6 @@ let report = acceptance.run_parallel(4, &RunOptions::default(), |n| {
});
```

Workers take the next test as they finish one, and the report is in the order of the tests.
Workers take a test as they finish one (one that starts where their last one ended, if there
is one), and the report is in the order of the tests.
The closure runs on the worker's thread, so a driver need not be `Send`.
3 changes: 3 additions & 0 deletions book/src/remote.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,9 @@ expectation-run expectations --platform SwiftUI --docs docs --runner app
`--command-timeout SECONDS` sets that per-command deadline (`--wait` only covers connecting).
`--no-state-pictures` leaves out the tests that only visit a state for the state map's picture
of it: a quicker run with the same verdicts (see [Stories](stories.md)).
A test starts where the last one ended when what it is given already holds there, and is
printed with `(continued)`; `--no-shared-starts` resets before every test (see
[Drivers](drivers.md#starting-where-the-last-test-ended)).
It listens, waits for the app, runs the suite for that platform, prints the results, and
with `--docs` saves them and writes the manual. `--screenshot-cmd 'xcrun simctl io booted
screenshot "$1"'` takes screenshots from outside, for apps that don't take their own.
Expand Down
3 changes: 3 additions & 0 deletions crates/docs/src/html.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1856,6 +1856,9 @@ impl Site<'_> {
}
let _ = write!(o, r#"<p class="{class} against">{}</p>"#, line.trim_end());
}
if result.and_then(|r| r.began) == Some(expectation_story::Began::Continued) {
let _ = write!(o, r#"<p class="note began">{}</p>"#, esc(crate::CONTINUED));
}
if let Some(e) = result.and_then(|r| r.start.error.as_ref()) {
let class = if status == Status::Unsupported {
"note warn"
Expand Down
3 changes: 3 additions & 0 deletions crates/docs/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ mod states;
mod thumbs;
mod voice;

/// What a test's page says when it began where another ended (`Began::Continued`).
pub(crate) const CONTINUED: &str = "This run started where the test that ran before it had ended: everything it is given was seen to hold there, so nothing was reset or arranged.";

use expectation_graph::Analysis;
use expectation_model::{ExpId, Feature, StepText, Suite, VarKind, Variable};
use expectation_story::{Because, Plan, Report, Status, Verdict, coverage};
Expand Down
3 changes: 3 additions & 0 deletions crates/docs/src/markdown.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1067,6 +1067,9 @@ fn test_page(d: &Docs, f: &Facts, i: usize) -> String {
}
let _ = writeln!(o, "{}\n", line.trim_end());
}
if result.and_then(|r| r.began) == Some(expectation_story::Began::Continued) {
let _ = writeln!(o, "{}\n", crate::CONTINUED);
}
if let Some(e) = result.and_then(|r| r.start.error.as_ref()) {
let _ = writeln!(o, "> {e}\n");
}
Expand Down
13 changes: 13 additions & 0 deletions crates/harness/src/bindings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ pub struct Bindings<W> {
attachments: Option<Capturer<W>>,
needs: Vec<String>,
fresh: Vec<String>,
always_reset: bool,
}

impl<W> Bindings<W> {
Expand All @@ -58,9 +59,17 @@ impl<W> Bindings<W> {
attachments: None,
needs: vec![],
fresh: vec![],
always_reset: false,
}
}

/// Reset before every test, even one that could start where the last one ended (see
/// [`Driver::may_continue`]): for a world that keeps what no state of the suite describes.
pub fn always_reset(mut self) -> Self {
self.always_reset = true;
self
}

/// The services to start afresh before each test (a subset of those it needs).
pub fn fresh(mut self, services: &[&str]) -> Self {
self.fresh = services.iter().map(|s| s.to_string()).collect();
Expand Down Expand Up @@ -156,6 +165,10 @@ impl<W> Driver for Bindings<W> {
self.fresh.clone()
}

fn may_continue(&self) -> bool {
!self.always_reset
}

fn reset(&mut self) -> Result<(), String> {
if let Some(f) = &mut self.reset {
f(&mut self.world);
Expand Down
8 changes: 8 additions & 0 deletions crates/harness/src/driver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,14 @@ pub trait Driver {
Ok(())
}

/// May a test start where the last one ended, without a reset? The runner only does so when
/// the last test passed and everything this one is given is observed to hold already, so
/// say no when that isn't enough: when what a test leaves behind that no state of the
/// suite describes (a log, a clock, a cache) could change what the next one sees.
fn may_continue(&self) -> bool {
true
}

/// Make `var` hold `value` before the first step ("Given …").
fn arrange(&mut self, var: &Variable, value: &Value) -> Result<(), String> {
self.set(var, value)
Expand Down
2 changes: 1 addition & 1 deletion crates/harness/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ use expectation_graph::Analysis;
use expectation_language::Loaded;
use expectation_model::diagnostic::{has_errors, render_all};
use expectation_model::{Diagnostic, Suite};
pub use expectation_story::{Control, Policy, Provenance};
pub use expectation_story::{Began, Control, Policy, Provenance};
use expectation_story::{Plan, Report};
use std::path::Path;

Expand Down
Loading
Loading