Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions book/src/drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,10 @@ acceptance.docs(Some(&report)).write("docs")?;

With a `docs_dir`, screenshots go to `<docs_dir>/screenshots/<test>/<step>.<ext>`.

A test whose start can't be set up (the reset or an arrangement fails) fails there, and its
other steps are skipped. The runner still asks for a screenshot, so the report shows what the
system looked like when it went wrong. If there is none, the failure stands as it is.

### Several runners

`Acceptance::publish(dir, runner, &report)` saves a run's results as
Expand Down
17 changes: 17 additions & 0 deletions book/src/platforms.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,23 @@ and merges them with what other runners saved.
under `platforms/<name>/`. Every HTML page has a Platform menu that opens the same page in the
other manual.

The platforms share one docs directory, and so one `screenshots/`. The default platform's are in
`screenshots/<test>/`, and each other platform's in a folder of its own,
`screenshots/platforms/<name>/<test>/`, so the same test on two platforms never shows the other's
picture. `expectation-run --docs` does this. A driver in Rust gets it from
`acceptance.run_options("docs")`, or by setting `RunOptions::screenshots_under` to
`acceptance.screenshots_under()`:

```rust
let acceptance = Acceptance::load_for("expectations", "iOS")?;
let report = acceptance.run(&mut driver, &acceptance.run_options("docs"));
acceptance.publish("docs".as_ref(), "app", &report)?;
```

Results saved before this name a later platform's screenshots as `screenshots/<test>/…`. Its
manual shows a placeholder for those until the tests run again. Screenshots kept by content in
a store are the same on every platform.

## Saying what each platform is

The docs have a menu that picks the version of the manual. Say what each platform is with
Expand Down
3 changes: 2 additions & 1 deletion clients/js/test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ test('a suite runs through the JS driver', async () => {

/** Runs the turnstile suite through a JS driver that `extra` adds handlers to, keeping attachments in `keep`. */
async function runTurnstile(extra, keep) {
const port = 5000 + Math.floor(Math.random() * 1000);
// Not 5060 or 5061: fetch refuses to connect to them (they are SIP's), and no app would arrive.
const port = 5100 + Math.floor(Math.random() * 900);
const docs = fs.mkdtempSync(path.join(os.tmpdir(), 'expectation-js-docs-'));
const args = [path.join(here, 'fixture'), '--addr', `127.0.0.1:${port}`, '--settle', '0', '--wait', '20', '--docs', docs];
if (keep) args.push('--attachments', keep);
Expand Down
8 changes: 7 additions & 1 deletion crates/cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,13 @@ fn docs(
|| Path::new(out).join("results"),
|r| Path::new(r).to_path_buf(),
);
let report = expectation_story::load_results_for(&results_dir, &c.plans, name.as_deref());
let mut report =
expectation_story::load_results_for(&results_dir, &c.plans, name.as_deref());
// A later platform's pictures are in its own folder of the shared screenshots; a
// per-test name outside it is another platform's picture, or nobody's.
if let Some(p) = platform {
report.only_shots_under(&expectation_story::platform_shots(p));
}
// Coverage files that extensions wrote next to the results, held to these results.
let (found, errors) = expectation_story::load_contracts_for(&results_dir, name.as_deref());
for e in errors {
Expand Down
3 changes: 3 additions & 0 deletions crates/docs/src/html.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1024,6 +1024,9 @@ impl Site<'_> {
esc(&alt),
if stale {
"<figcaption>From before this page changed.</figcaption>"
} else if status == Status::Failed {
// Evidence of what went wrong, not a picture of what should be there.
"<figcaption>What it looked like when this failed.</figcaption>"
} else {
""
}
Expand Down
2 changes: 2 additions & 0 deletions crates/docs/src/markdown.rs
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,8 @@ fn image(
);
if stale {
let _ = writeln!(o, "*From before this page changed.*\n");
} else if status == Status::Failed {
let _ = writeln!(o, "*What it looked like when this failed.*\n");
}
}
None if reader => {}
Expand Down
30 changes: 29 additions & 1 deletion crates/harness/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,29 @@ impl Acceptance {
}
}

/// The folder under the docs' `screenshots/` for this platform's per-test screenshots
/// (`platforms/<name>`), so each platform's results name its own pictures. `None` for the
/// default platform and a suite without platforms: theirs are in `screenshots/` itself.
pub fn screenshots_under(&self) -> Option<String> {
match self.suite.platform.as_deref() {
Some(p) if self.suite.platforms.first().map(String::as_str) != Some(p) => {
Some(expectation_story::platform_shots(p))
}
_ => None,
}
}

/// What to run with so that [`Acceptance::publish`] to the docs directory `dir` finds the
/// screenshots: they are taken into `dir/screenshots/`, a later platform's in a folder of
/// its own there.
pub fn run_options(&self, dir: impl Into<std::path::PathBuf>) -> RunOptions {
RunOptions {
docs_dir: Some(dir.into()),
screenshots_under: self.screenshots_under(),
..Default::default()
}
}

pub fn from_loaded(loaded: Loaded) -> Result<Self, String> {
let Loaded {
suite,
Expand Down Expand Up @@ -233,11 +256,16 @@ impl Acceptance {
std::fs::create_dir_all(&results)?;
let file = expectation_story::results_file(runner, self.platform.as_deref());
std::fs::write(results.join(file), report.save(&self.plans))?;
let merged = expectation_story::load_results_for(
let mut merged = expectation_story::load_results_for(
&results,
&self.plans,
self.suite.platform.as_deref(),
);
// A later platform's pictures are in its own folder of the shared screenshots; a
// per-test name outside it is another platform's picture, or nobody's.
if let Some(under) = self.screenshots_under() {
merged.only_shots_under(&under);
}
// What extensions made of this platform's evidence, held to these results.
let (found, _) =
expectation_story::load_contracts_for(&results, self.suite.platform.as_deref());
Expand Down
27 changes: 25 additions & 2 deletions crates/harness/src/run.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@ pub struct RunOptions {
/// --screenshots <store>`), or make it the docs' own `screenshots/` directory. Screenshots
/// are taken when this or `docs_dir` is set.
pub screenshot_store: Option<PathBuf>,
/// A folder under `screenshots/` for this run's per-test screenshots, so that runs which
/// share a docs directory don't write over each other's: with `platforms/android`, they go
/// to `<docs_dir>/screenshots/platforms/android/<plan>/<step>.<ext>` and the results name
/// them so. [`Acceptance::run_options`](crate::Acceptance::run_options) sets it for a
/// platform other than the suite's first. A store has no use for it: its names are the
/// content's.
pub screenshots_under: Option<String>,
/// How many times to [`Driver::advance`] while waiting for a step's checks to hold.
pub patience: u32,
/// Ask the driver for its controls and which ones each test used (see
Expand All @@ -39,6 +46,7 @@ impl Default for RunOptions {
RunOptions {
docs_dir: None,
screenshot_store: None,
screenshots_under: None,
patience: 100,
controls: true,
attachments_dir: None,
Expand Down Expand Up @@ -302,10 +310,18 @@ fn run_steps(
}

// Per test and step under the docs, replaced each run; or by content in a store.
let named = match opts
.screenshots_under
.as_deref()
.map(|u| u.trim_matches('/'))
{
Some(under) if !under.is_empty() => format!("screenshots/{under}/{}", plan.slug),
_ => format!("screenshots/{}", plan.slug),
};
let shots = match (&opts.screenshot_store, &opts.docs_dir) {
(Some(_), _) => None,
(None, Some(d)) => {
let dir = d.join("screenshots").join(&plan.slug);
let dir = d.join(&named);
let _ = std::fs::remove_dir_all(&dir);
Some(dir)
}
Expand All @@ -321,7 +337,7 @@ fn run_steps(
let name = format!("{n:02}.{}", shot.extension);
std::fs::create_dir_all(dir).ok()?;
std::fs::write(dir.join(&name), &shot.bytes).ok()?;
Some(format!("screenshots/{}/{name}", plan.slug))
Some(format!("{named}/{name}"))
};
let mut attachments = vec![];
let mut keep = |driver: &mut dyn Driver, n: usize| {
Expand Down Expand Up @@ -357,6 +373,13 @@ fn run_steps(
start_ms: began,
end_ms: now_ms(),
});
// What it looked like when it went wrong, if the driver can still say. The failure
// is the setup's whether or not there is a picture of it.
start.screenshot = shoot(driver, 0);
start.window = Some(Window {
start_ms: began,
end_ms: now_ms(),
});
keep(driver, 0);
start
};
Expand Down
6 changes: 4 additions & 2 deletions crates/harness/tests/demo_results.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ fn the_demos_saved_results_still_match_its_tests() {
}

/// A platform other than the first is written under `platforms/<name>/`, but its screenshots
/// are the run's, in the docs' own `screenshots/`: its pages show them from there.
/// are in the docs' own `screenshots/`, in a folder of their own: its pages show them from there.
#[test]
fn a_later_platforms_manual_shows_the_shared_screenshots() {
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../demos/kettle");
Expand Down Expand Up @@ -75,7 +75,9 @@ fn a_later_platforms_manual_shows_the_shared_screenshots() {
let mut shown = 0;
for entry in std::fs::read_dir(tree.join("site")).unwrap().flatten() {
let page = std::fs::read_to_string(entry.path()).unwrap_or_default();
shown += page.matches("../../../screenshots/").count();
shown += page
.matches("../../../screenshots/platforms/swiftui/")
.count();
assert!(
!page.contains(r#"src="../screenshots/"#),
"{}",
Expand Down
Loading
Loading