From a1e7d42a5d0790f4022065e6ca2d06f88fa2aab5 Mon Sep 17 00:00:00 2001 From: Arnau Sanchez Date: Wed, 16 Sep 2026 11:38:17 +0200 Subject: [PATCH] Add intersectional-test skill Claude Code skill that runs the MSF DHIS2 intersectional testing workbook for an Operational Directorate: browser-driven steps, per-step artifacts, and an HTML/xlsx report. Copied from the intersectional-testing project so it lives alongside the other OCBA tooling. Co-Authored-By: Claude Opus 5 --- skills/intersectional-test/SKILL.md | 99 +++++++++ .../intersectional-test/config.template.yaml | 6 + .../dhis2-browser-interaction.md | 196 ++++++++++++++++++ 3 files changed, 301 insertions(+) create mode 100644 skills/intersectional-test/SKILL.md create mode 100644 skills/intersectional-test/config.template.yaml create mode 100644 skills/intersectional-test/dhis2-browser-interaction.md diff --git a/skills/intersectional-test/SKILL.md b/skills/intersectional-test/SKILL.md new file mode 100644 index 0000000..a642ccf --- /dev/null +++ b/skills/intersectional-test/SKILL.md @@ -0,0 +1,99 @@ +--- +name: intersectional-test +description: Run a guided MSF DHIS2 intersectional testing for some MSF Operational Directorate (OD) +version: 0.0.1 +--- + +# Description + +This skill guides a user (= focal point) of a particular OD (Operational Directorate) of MSF in the intersectional testing required whenever a new DHIS2 version needs to be validated. The skill expects a workbook file (ie an Excel file) as source of truth to verify the testing points, using either API calls or a browser when UI interaction is required. Finally, write as an output the updated copy of the workbook and some custom user-friendly report. + +# Requirements + +Assumes a coding agent with shell and HTTP access to the testing instances. Uses curl, jq, openpyxl, playwright. + +# Folder structure of the execution + +## `config.yaml` + +Configuration file describing the full testing environment, copied into the execution folder from `config.template.yaml` (which ships with this skill, beside `SKILL.md`) and filled in with the user. + +## Source workbook + +The workbook to use as source of truth (ie `DHIS2_VERSION - DHIS2 Intersectional testing.xlsx`). It contains the testing conditions, the ODs/focal points, and the tests to perform for each subdomain `Main domain - Inner domain` (`General - Menus`, `Users`, `Maintenance - Constants`). Known sheets to ignore: `Aux`, `Main New Features`, `All Bugs And Features`. + +Domain sheets carry roughly these columns: `Scenario`, `Action`, `Steps to Follow`, `Expected Result`, `Comments/conditions`, `Passed?`, `Test done by`, `Fail details`. Treat that as indicative, not a schema — the sheets were written by hand over years and none of it is guaranteed. Notes: + +- Resolve columns by header name, never by position: the order varies between sheets (some swap `Passed?`/`Test done by`) and so does the spelling (`Test Done by`, `Section` instead of `Scenario`). Read only those columns; some sheets report thousands of spurious empty ones. +- Sanity-check what you read rather than trusting the layout: if headers and values disagree (a status column full of OD names), believe the values; skip rows that merely repeat the header; expect sheet names to differ from their Summary label by wording or stray whitespace. Match on meaning, not on an exact structure. +- A fully blank row is a separator, not a step. Count real steps before reporting progress, and if a row range the user gave does not match the number of steps in it, say so (rows 2-6 of Capture (Tracker programs) is 4 steps, not 5). +- A blank `Scenario`/`Action` means "same as the row above": to build a step, walk up to the nearest non-empty cell. Steps within such a block are ordered and often depend on the previous ones (ie. "repeat the above for..."). +- `Passed?` is written by humans, so expect variants (`Passed`, `Failed`, `Fail`, `NA`, `N/A`, blank). Read case-insensitively; write only `Passed`/`Failed`/`Not Executed`/`N/A` (with the appropriate background color). +- `N/A` and `Not Executed` are not the same: `N/A` means the step can never apply to this version/OD (feature removed, or the OD does not use that module) and is final, so later runs skip it; `Not Executed` means it applies but has not run yet, so it stays pending. + +# Interactions + +- Perform only browser UI interactions/assertions: `dhis2-browser-interaction.md` has the technical details on how to interact with the browser. Do not open/close the browser for each action step. Keep it open and reuse the tab. Do not perform manual requests to the API (unless specifically stated by the action step), always use the UI to perform assertions. + +## `output/artifacts/` + +This output folder will contain all the technical data and scripts generated by the agent in the session. This data will be used to: + +1. generate the final report. +2. serve as a technical report in itself for reproducibility. +3. store the necessary data for resumability, in case we continue the session at a later time. + +- `output/artifacts/Subdomain/` +- `output/artifacts/Subdomain/NNN`: Folder for a specific subdomain step `NNN` (ie. `002` means row 2 in the workbook, for easy matching). Store here any script or data that was used/generated while testing the step: `test.py`, whatever API payloads it asserted against, `trace.zip`, final screenshots `success.png` if ok, `failure.png` if it failed. +- `output/artifacts/Subdomain/NNN/result.json`: the step's verdict, written as soon as the step ends - status, timestamp, the note that feeds `Comments/conditions`, and the evidence paths. This is what makes a run resumable and what Step 3 assembles the reports from, so a step without one has not been run. +- `output/artifacts/Subdomain/NNN/transcript.txt`: the slice of the agent's own session transcript (prompts, reasoning, tool calls, output) spent on this step, cut in Step 3 from `started_at`/`timestamp` in `result.json`. It is the "why" behind the verdict - the script shows what was asserted, the transcript shows how the agent got there and what it tried first. +- `output/artifacts/Subdomain/results.json`: the roll-up for the subdomain, plus the run's context (OD, focal point, server, DHIS2 version, start/finish). + +# Execution workflow + +The focal point user will invoke the skill (`/intersectional-test`) and the process will begin: + +## Step 1 - Environment setup + +- Read the workbook to get the ODs list, their focal point and the DHIS2 target version (`Testing conditions` -> `DHIS2 Core Version`). Show a very concise report to the user of that analysis. If some important information is missing or unclear, ask the user before continuing the process. +- With that information in hand, guide the user to fill `config.yaml` (if not existing, copy from the template). That includes all the keys (except the field server, which is optional) +- Check that the testing DHIS2 central server is up and its version matches the target DHIS2. +- Read this OD's assignment from the Summary sheet and show a report of what their domain are, which are completed and which are still pending, so have an idea of the status of what's pending to do. +- The Summary's progress colours are hand-maintained and can be wrong, so treat them as a hint and derive real status by counting `Passed?` in the domain sheet itself. Row 36 was coloured "Test completed" while every `Passed?` cell on that sheet was blank. Report any disagreement to the user rather than silently picking one. + +## Step 2 - Execution of a subdomain and its steps + +- The user has at this point a clear view of the task status, so ask them how to proceed. Typically, they will instruct the agent to run tests for pending subdomains/steps or just the some tests for some specific subdomain. By default, actions in the domain tab already marked as Passed/NA will be skipped (inform the user of that), unless that step is necessary for the next to succeed, or the user instructs otherwise. +- Before starting the execution of a subdomain, the agent will check if the version of the app (or apps) to be used in that subdomain match the requirements in the Testing Conditions. If not, guide the user on how to install them. +- Now, for each step intended to be executed, follow the steps specified in the corresponding sheet (the summary label and the sheets may not match 100%, use your judgement) and its comments/description/expected-results and so on, and interact with the DHIS2 instance (either using CLI commands or a real browser). +- If a user with some specific permissions (other than admin) should be used, create it on-the-fly (inform the user about that). +- An Expected Result that the app has outgrown is not a failing step. The workbook was written over years and describes UI that has since changed (row 4 expects a "List button" Capture 105.7 no longer has). Assert the closest real behaviour, pass the step if that behaviour is correct, and record what changed in `Comments/conditions` - a `Failed` here would report a defect that does not exist. Flag these to the user; they are the rows the next round should rewrite. +- Prefer a contrast to a presence check. "Program is selectable in the dropdown" is nearly unfalsifiable on its own; comparing an org unit that has the program (14 offered) against one that does not (1 offered), both matching the API, is a real test. +- Derive expected values from the API at runtime, never hardcode them. Names, uids and counts differ per OD, and a hardcoded expectation is a step that only ever worked on one instance. +- The browser runs headless, so a run looks like nothing is happening. The first time a step needs the browser, tell the user how to watch it live and how to replay a step afterwards; both commands are printed by the run itself (see `dhis2-browser-interaction.md`). +- The process should be mostly automatic, so show to user clearly which action you are executing at each moment (name of the subdomain, name of the action, the overall progress, and so on) +- The moment you begin a step (before reading its row in detail or writing any script), record the wall-clock time (`date -Is`) and store it as `started_at` in that step's `result.json`; `timestamp` is when the verdict was written. Step 3 uses that window to cut the step's transcript slice, and `elapsed_seconds` only covers the script run - the agent's own time on the step (reading, writing, debugging) is usually far longer and is what the slice has to cover. +- If something fundamental is going wrong (ie. you cannot open the browser or the apps), stop and fix that with the help of the user. When things go normal, a step can pass, fail, or flagged as `Not Executed` if something was missing. Write the artifacts (data/scripts) needed for debugging/reporting/resuming and continue to the next step. +- After the requested execution has finished, show in the screen a summary of what you have done and what output you have generated (see next step) + +## Step 3 — Report + +The previous step will have created some artifacts data. From this data, create these reports files (assuming `DHIS2_VERSION - DHIS2 Intersectional testing.xlsx` was the input file): + +- `output/skill/`: Clone of the skill that guided the session. + +- `output/DHIS2_VERSION - DHIS2 Intersectional testing (original).xlsx` + +- `output/DHIS2_VERSION - DHIS2 Intersectional testing (updated).xlsx`: a cloned version of the input workbook with fields filled from every accumulated `result.json`, not only this session's. Typical columns to fill: `Passed?`, `Test done by`, `Fail details`. + - openpyxl keeps formulas but drops their cached results, so cells like the Summary totals read back as `None` and show empty until Excel or LibreOffice recalculates them on open. Expected, not corruption - do not try to "fix" it by writing literals over the formulas. + - Verify the output by diffing every cell against the input and confirming the only changes are the cells this run meant to write. + +- `output/report.html` A report that converts the current session. Contents: + - A brief summary of what was tested (OD, focal point, subdomains and the success/failed steps), with links to the input/output xlsx files. + - A detailed table with filters showing the result (columns: `Row`, `Subdomain`, `Scenario`, `Action`, `Passed?`, `Elapsed (secs)`). To keep it readable, strive for single-line rows. For that, 1) cells containing long text should be ellipsed (...), and a hover would show the rest of the text, and 2) The row has an icon (position: at the left of the row) to open more details, showing extra info (non-table form) like: `Steps to follow`, `Expected result`, `Passed?`, `What was asserted`, `Resources` (links to the most relevant artifact files of that step), `Playback`(the command a user should run to retrace the step, ie `.venv/bin/playwright show-trace "output/artifacts/Capture (Tracker programs)/003/trace.zip"` - adjust for the current OS) + - Coding agent (with version) used, model, input/output tokens, and so on. Read the agent/token figures from the coding agent's own session transcript rather than estimating them. + - Agent time of the testing session. If the transcript carries no duration field, derive both from entry timestamps. + - This skill's `version` from its frontmatter. The ODs run independently over weeks, so a returned result has to say which revision produced it - and `output/skill/` is the copy it refers to. + - Full agent conversation, internal reasoning and output, with a button that downloads it as a `.txt`. Build that file client-side from the text already in the page (a Blob and ``), not by linking a sidecar file - the report gets shared on its own, and a link to a neighbouring file breaks the moment it does. + - Per-step transcript: cut one slice per step into `output/artifacts/Subdomain/NNN/transcript.txt` - the transcript entries between the step's `started_at` and `timestamp` (widen to the next step's `started_at` when the verdict was written late), with the same header/format as the full file. Link it from the step's `Resources` as `Agent transcript`, next to `trace.zip` and the screenshots: it is an artifact like them (so the sidecar rule above does not apply), and a reviewer questioning one verdict should not have to scroll a 100-entry session to find its 8 entries. + How to display: Keep the summary always visible and put the other sections in accordions, so the contents of the report are visible at a glance. Add a button for dark/light theme (dark by default). diff --git a/skills/intersectional-test/config.template.yaml b/skills/intersectional-test/config.template.yaml new file mode 100644 index 0000000..ead4420 --- /dev/null +++ b/skills/intersectional-test/config.template.yaml @@ -0,0 +1,6 @@ +od: "Operational Directorate" # OCG, OCB, OCA, OCBA, OCP, WACA +focal_point: "Name Surname" # Focal point responsible for the tests in the OD +workbook: "SOMEFILE.xlsx" # ie "VERSION - DHIS2 Intersectional testing.xlsx" +servers: # DHIS2 servers (central: required, field: required only when running the Sync domain) + central: { url: "http://localhost:8080", username: admin, password: district } + field: { url: "http://localhost:8081", username: admin, password: district } diff --git a/skills/intersectional-test/dhis2-browser-interaction.md b/skills/intersectional-test/dhis2-browser-interaction.md new file mode 100644 index 0000000..2c291fb --- /dev/null +++ b/skills/intersectional-test/dhis2-browser-interaction.md @@ -0,0 +1,196 @@ +# Browser interaction + +UI steps are driven with **Playwright for Python** (sync API). Install once, into a venv — +some distros refuse system-wide installs: +`python3 -m venv .venv && .venv/bin/pip install playwright && .venv/bin/playwright install chromium`. +Run every step script with `.venv/bin/python`. + +## Two phases per step + +1. **Explore live.** Drive the running app and dump what is actually in the DOM to find the *real* + selectors (watch it through the DevTools URL below; `HEADED=1` if you truly need to click by + hand). DHIS2 org-unit trees, dimension pickers and + modals have no stable accessible names — never guess them. Note findings in + `artifacts//NNN/explore.md`. +2. **Commit a script.** Write `artifacts//NNN/test.py`: rerunnable, asserts the + workbook's Expected Result, `exit 0` = pass / non-zero = fail (same contract as the curl+jq + steps). Expect 2–4 fix-and-rerun cycles; that is normal, not a blocker. + +## One browser for the whole run + +The browser is opened once per run and every step reuses that tab, so a step script must not +launch or close one. Put the lifecycle in a shared harness that yields a session, and have each +step expose `run(s)` against it; a runner then walks the requested steps in that single session. +Keep a `__main__` block in each step so it still runs standalone by borrowing its own session. + +Never attach to a debugging port just because something is listening on it. The browser is +started with a non-default port and its own throwaway profile, and the pid and CDP +id it reported are recorded. Delete that profile when the browser stops: it is rebuilt on every +start and is tens of MB of Chromium state in a folder meant to be handed over. + +**Suppress Chrome's own dialogs before the first launch, not after.** "Save password?", +notification prompts and, on Linux, the keyring unlock will each hang a run waiting for an answer +nobody is there to give. That means launch flags (`--password-store=basic`, `--use-mock-keychain`, +`--disable-save-password-bubble`, notifications denied) plus seeded profile preferences, since some +of these are read only at first start. Headless removes the focus-stealing but not the keyring. +Page-level `alert()`/`confirm()` are separate and block every later Playwright call until +answered — `page.on("dialog", lambda d: d.dismiss())` on the shared tab. + +Navigation helpers belong on the session (`open_capture()` and friends): navigate only if the tab +is not already there, then put the app back into the state the step expects. + +That is more than closing popups — **app scope has to be reset too**. Capture's landing screen only +exists while nothing is selected, so a step that asserts against it fails if an earlier step left a +program selected. A `clear_scope()`-style helper belongs next to the popup dismissal; row 002 +failed on a second run for exactly this reason and nothing was wrong with it. + +### Closing a DHIS2 popup + +Every obvious approach fails, so it is worth stating what works: + +| Attempt | Result | +| ------- | ------ | +| `keyboard.press("Escape")` | no effect — the layer stays open | +| `backdrop.click()` | closes the org-unit tree; times out when a menu is open | +| `backdrop.dispatch_event("click")` | works in both cases | + +The org-unit tree stays open after a unit is picked, and its backdrop then swallows every later +click as *subtree intercepts pointer events*. Clicking the backdrop is what closes it — but an open +menu popper sits **above** the backdrop and intercepts a real click aimed at it, which is why +dispatching the event straight at the backdrop is the only thing that works for both: it skips hit +testing entirely. + +Dismiss every overlay, not just the ones with a `data-test`: the plain backdrop that intercepts +clicks may carry nothing but a class. Do it on entry to a step as well as on exit — the step that +pays for a stray overlay is the next one. + +## Headless, and how to see what it is doing + +Run headless. A visible window that loses focus or gets covered makes Chromium throttle timers +and stop `requestAnimationFrame`, so a polling app stalls, auto-wait expires, and a step records +a `Failed` that only means someone switched windows. Headless has no window, so the whole class +is gone — on Linux, macOS and Windows alike. (`--disable-backgrounding-occluded-windows` and +friends are set anyway for anyone who does run headed; nothing can stop a real window firing +`blur`, which closes DHIS2 dropdowns and pickers on its own.) + +Nothing is lost by it: + +- **Live** — Chromium serves the DevTools front-end off its own debugging port, so a URL is all + it takes to watch the running page: open it in a Chromium-based browser (Chrome, Edge, Brave — + not Firefox) and turn on Screencast. No VNC, no X server, no internet — but build that URL + from Chromium's own front-end (`/devtools/inspector.html?ws=/devtools/page/`), because + the one `/json/list` advertises points at a Google-hosted copy. A run prints the URL + before its first step and `browser.py watch` reprints it; take it from that output rather than + saving it, because the page id changes whenever the tab is recreated. Watching cannot disturb + the run — that DevTools tab is a separate browser process from the headless one, so defocusing + or closing it changes nothing. +- **Afterwards** — every step writes `artifacts//NNN/trace.zip`, replayed with + `playwright show-trace ` or by dropping it on trace.playwright.dev (which processes + locally). It holds a screencast filmstrip, a DOM snapshot per action with the selector that + action used, the network log and the source line — far more than a video would give. + +Per-step evidence has to be a trace, not a video: video is per-context and only written when the +context closes, and is unavailable on a CDP-attached context at all (`page.video` is `None`), so +one video per step would mean one browser per step — losing the shared tab that dependent steps +need. Tracing runs for the life of the context and **the runner** brackets each step with a chunk — +a step script does not manage its own trace, which is why the skeleton below has no tracing in it. + +**A trace records only the resources it observes being fetched**, and the browser deliberately +outlives each script. So a run that attaches to a tab already sitting on the app never re-fetches +its CSS, and the chunk ends up holding the DOM with no stylesheets: the Trace Viewer then renders a +blank page next to a perfectly correct filmstrip, which reads as a corrupt trace and is not one. +One `page.reload()` immediately after `tracing.start()` fixes every chunk in that session — +Playwright stores the bodies even though Chromium serves them from cache — and costs a single app +load per run, before the first step. Check it rather than assuming: a chunk with no `.css` entries +has this bug. Frames are not the problem; an iframe's own HTML and CSS trace fine. + +**Say this to the focal point, do not assume they know it.** A headless run looks like nothing is +happening, and the natural reaction is to conclude it is broken. When the first browser step of a +run starts, show them the watch URL and the `show-trace` command for the artifacts folder. + +## Script skeleton + +```python +from playwright.sync_api import expect +import harness # owns config, api(), the session + +def run(s): # s.page = the shared tab + app = s.open_capture() # frame_locator, already waited on + try: + app.get_by_role("button", name="New").click() + expect(app.get_by_text("...")).to_be_visible() # the Expected Result + except Exception: + s.shot("screenshot.png") # screenshots: page, never app + raise + +if __name__ == "__main__": # standalone rerun + with harness.session() as s: + run(s) +``` + +Exceptions propagate → non-zero exit. Nothing else is needed to report a failure. + +## Login + +The browser process owns its own profile, so **the session cookie lives in that profile** and +outlives every script that attaches. There is no `storage_state` file to write, load or gitignore. + +Logging in is therefore a fallback, not a step: on attaching, if the tab is sitting on +`dhis-web-login`, submit the `config.yaml` credentials on `page` (the login page is not iframed) +and carry on. Every later script finds itself already authenticated. + +## The app runs in an iframe + +DHIS2 2.42's global shell iframes the app: `page` holds only the shell (header bar, sidebar), and +every app element is one frame down. Locating on `page` finds nothing and fails with a timeout that +looks like a broken selector. + +- `app = page.frame_locator("iframe")` once, right after `goto`, then locate on `app` throughout. +- **Wait inside the frame** for the app's own first element — the shell resolves long before the app + does, so `wait_until="domcontentloaded"` on `page` proves nothing. Allow ~60s on the first wait. +- **`page.screenshot()`, not `app`** — a frame locator cannot screenshot, and the page capture shows + the whole window anyway. +- The login page is **not** iframed: `dhis-web-login` is driven on `page` directly. +- `frame_locator` cannot run DOM introspection. While exploring, grab the real frame instead — + `fr = [f for f in page.frames if "dhis-web-capture" in f.url][0]` — and use + `fr.eval_on_selector_all(...)` to dump `[data-test]` attributes and roles. + +## Rules + +- **Never pin the layout of Playwright's browser cache.** `chrome-linux/chrome` became + `chrome-linux64/chrome`, and every run then died with "chromium not found" before a single step. + Glob loosely, choose the newest by parsed version number rather than string order + (`chromium-1234` sorts *below* `chromium-999`), and fall back to asking Playwright for + `chromium.executable_path`. +- **Always headless.** `HEADED=1` forces a real window for debugging a selector by hand; never + for a run whose results anyone will rely on. +- **Never `time.sleep` in sync Playwright.** It blocks the loop that receives CDP events, so + cached state like `page.frames` never refreshes and a poll on it spins until it times out, + looking exactly like a broken app. Use `page.wait_for_timeout`, or better, a locator that + auto-waits. +- **Normalise whitespace when matching UI text against API names.** Metadata written by hand over + years contains stray spaces (`" Panderu MCHP"`) that the rendered label trims away, so an exact + match on the raw name finds nothing. +- **Assert with `expect()`**, which auto-waits. Never `sleep`. Never assert on global counts or + "the list is empty" — scope to the objects this step created. +- **Screenshot on failure**, into the step's artifacts folder. That is the evidence for the report. +- **Locators, in order of preference**: `get_by_role`, `get_by_label`, `get_by_test_id` + (DHIS2 ships `data-test` on many components), then CSS. Avoid text that is translated. + Check the roles are actually there before preferring them — several DHIS2 widgets ship none at + all (the org-unit tree has no `treeitem`, only `data-test`), and a role locator that matches + nothing looks exactly like a broken app. +- **`exact=True` is unreliable here.** `get_by_role("menuitem", name="Child Programme", exact=True)` + matches nothing: the item is an `` wrapping the program icon, whose + `title="child_program_positive"` is folded into the accessible name. Match the rendered label + without `exact` in menus. +- **A row existing is not data having loaded.** DHIS2 grids render skeleton rows first, so + `expect(rows.first).to_be_visible()` passes on a row whose cells are empty. Wait for the loader + count to reach 0 *and* for non-empty text in a data cell. +- **`has_text` matches substrings.** Filtering a nested widget by a short name silently selects an + ancestor or a sibling ("Bo" also matches "Bombali"). Match the label exactly, then walk up to the + node you meant. +- **Delete stale evidence when a step later passes.** Fix-and-rerun cycles leave `failure.png` and + a `trace.zip` from the run that failed; leave them and the report links evidence contradicting + its own verdict. Clear the step's evidence at the start of each attempt. +- A selector that broke because the app changed is a **skill bug**, not a DHIS2 regression. Fix the + script and rerun before recording a failure.