@@ -6,7 +6,14 @@ description: Run a moderated usability study of Surla. One subagent plays a firs
66# Surla UX review
77
88This skill runs a moderated, think-aloud usability test of Surla and turns it
9- into a UX research report. Two subagents take part:
9+ into a UX research report. The product goal is a journey driven by the
10+ participant's existing coding-agent session: after installation and one-time
11+ ` surla up ` , the participant describes the goal through ` /surla ` (` $surla ` ),
12+ uses the browser for human decisions, and reaches an approved merge without
13+ typing terminal commands to move between stages. Commands the coding agent
14+ runs internally are distinct from commands the user must enter. Observe whether the product
15+ actually enables that journey; never supply missing steps as hidden coaching.
16+ Two subagents take part:
1017
1118- ** The participant** plays a developer who has never used Surla. It may read
1219 only Surla's ` README.md ` and whatever Surla itself prints or generates, and
@@ -69,45 +76,59 @@ source out of its view.
6976` attempt-1/transcript.md ` :
7077
7178- the Surla commit, the date, and the OS;
72- - whether the folder in the README's ` surla init -dir ` example (such as
73- ` /tmp/shop ` ) already exists. ` surla init ` refuses an existing folder, so a
74- leftover would look like a usability problem. Do not delete it. Tell the
75- tester it exists, labeled "ownership unconfirmed";
76- - the ` -surla-version ` that the README tells ` surla init ` to use. A scaffolded
77- project downloads that version, not the commit under test;
79+ - whether the participant's ` work/ ` folder is empty apart from permitted
80+ coding-agent settings, and any known pre-existing example destinations.
81+ Leave existing folders alone and label their ownership unconfirmed;
82+ - the global setup writes this attempt may make. ` surla up ` and
83+ ` surla setup -global ` update the real machine's detected coding-agent MCP
84+ configuration and skill files, and may install the Herdr mod and add its
85+ free macOS shortcut in ` keys.json ` . Record the paths setup reports and
86+ pre-existing registrations or bindings without recording credentials.
87+ These files are outside ` work/ ` and may be used by other sessions; do not
88+ delete or restore them as attempt cleanup;
89+ - the version of Surla installed by the README and the version its starter
90+ module will download. Record whether they match the commit under test;
7891- the versions of ` go ` , ` goa ` , and ` temporal ` , or that they are missing;
7992- which credentials are set, such as ` ANTHROPIC_API_KEY ` . Record that each is
8093 set, never its value;
8194- any personal instruction files your harness loads into every subagent, such
8295 as a global ` CLAUDE.md ` or ` AGENTS.md ` . These can tell the participant
8396 things the README does not, so the report lists them as a limitation;
8497- whether commands run inside your coding agent's own sandbox, and whether
85- they have network access. ` surla init ` downloads modules, and Surla builds
98+ they have network access. Starter initialization downloads modules, and Surla builds
8699 and runs the code under test as ordinary processes with the user's file
87100 and network access. Working copies and process groups do not confine that
88101 code. Keep any coding-agent sandbox restrictions separate from Surla's
89102 documented execution policy. Give the participant the host access you are
90- authorized to give. Trusted Surla commands run as the docs show, including
91- ` surla init ` , which generates code from Surla's own unmodified starter
92- templates. The product's own code is never compiled, generated from, or run
93- except through Surla's commands . That
94- includes ` goa gen ` on its design, which runs that design code. If the
95- documented workflow needs a step Surla offers no command for, the study
103+ authorized to give. Trusted Surla operations run as the docs show,
104+ including initialization from Surla's own unmodified starter templates.
105+ The product's own code is never compiled, generated from, or run except
106+ through Surla's documented operations . That includes ` goa gen ` on its
107+ design, which runs that design code. If the
108+ documented workflow needs a step Surla offers no operation for, the study
96109 reports it as a missing capability. If the access Surla needs is not
97110 available, that is an environment limitation: the tester marks the
98111 milestones it blocks as not supported;
99- - ** how the participant reaches Surla's MCP server.** The README tells a user
100- to point a coding agent at ` http://<addr>/mcp ` . A subagent cannot add an MCP
101- server to its own session. The participant therefore runs its own
102- coding-agent CLI non-interactively in ` work/ ` , with an MCP configuration it
103- writes itself, for example ` claude -p ` , ` codex exec ` , or ` gemini -p ` .
104- Record which CLIs it may use, or that none can run here. Do not hand it an
105- undocumented HTTP route, because working that out from outside the README
106- is coaching.
112+ - ** how the participant reaches Surla's MCP server.** The current README
113+ uses one-time ` surla up ` (or ` surla setup -global ` ) to register ` surla stdio `
114+ and the ` /surla ` (` $surla ` ) skill. A subagent cannot add an MCP server to
115+ its own session, so the participant may run an installed coding-agent CLI
116+ in ` work/ ` and invoke the skill there. Record which clients can run, which
117+ load the global registration, whether a session reload is needed, and
118+ whether the environment can show the browser for human decisions. Record
119+ harness-only CLI launches separately from user workflow steps; a harness
120+ that cannot preserve the coding session is a study limitation. Do not
121+ hand it an HTTP route, credential, prewritten MCP configuration or lifecycle
122+ command. Failure to discover or use the documented integration is evidence;
123+ - whether Herdr and Chorus are installed, whether the agent starts inside a
124+ live Herdr pane, whether its client has been observed forwarding
125+ ` HC_TERMINAL_ID ` and provider settings to stdio, the installed agent
126+ families and background-worker support, and any project choices or
127+ dismissed recommendations already present. Do not install a second family just to hide the one-family path;
107128- ** what each mid-build change needs** (see the tester brief's "Mid-build
108129 changes"), recorded as two separate facts: what this machine provides,
109130 such as model credentials or the access a live demo needs, and what the
110- commit under test provides, such as a command that regenerates code after a
131+ commit under test provides, such as an operation that regenerates code after a
111132 design change. The tester uses them to tell "not supported on this
112133 machine" apart from "blocked by a missing Surla capability".
113134
@@ -157,8 +178,8 @@ participant.
157178Each round trip works like this:
158179
1591801 . The participant works until a natural pause and replies with a step
160- report. Its brief lists when it pauses, including before the first run of
161- each new ` surla ` command .
181+ report. Its brief lists when it pauses, including before one-time setup,
182+ its first skill invocation and each new human decision on the page .
1621832 . Append the report to the transcript and send it unchanged to the tester,
163184 together with the elapsed time since the previous report.
1641853 . The tester replies with up to three questions, an optional hint, an
@@ -190,7 +211,7 @@ harness keeps a subagent transcript. Also read the logs its coding agent saved
190211in ` work/agent-logs/ ` . A leak is reading anything outside the participant's
191212allowed list, loading this skill, building the product without Surla, or
192213compiling, generating from, or running the product's code except through
193- Surla's commands .
214+ Surla's documented operations .
194215
195216A participant cannot unlearn what it read, so a leak ends the attempt. Tell
196217the tester what was read and when. The tester records the leak as a finding,
@@ -207,15 +228,16 @@ limitations that leaks were only self-reported.
207228 ` ps -p <pid> -o command= ` shows the reported binary and folder. PIDs get
208229 reused. Leave running any process you cannot match.
209230- Delete nothing else. Every project folder the participant created is
210- evidence for the report, including folders outside ` work/ ` such as
211- ` /tmp/shop ` , and may hold a Git repository with unpushed work. Each attempt
231+ evidence for the report, including a project initialized outside ` work/ ` ,
232+ and may hold a Git repository with unpushed work. Keep machine-wide setup
233+ changes recorded above; they are not disposable attempt files. Each attempt
212234 gets a new folder, so nothing needs to be cleared for the next one.
213235
214236In the next transcript, record the exact path of every folder outside ` work/ `
215237that the participant reported or that appears in ` work/agent-logs/ ` , and
216238every process left running. Label them "ownership unconfirmed". Tell the
217239tester when the next attempt starts, so it does not log a leftover, such as
218- ` surla init ` refusing an existing ` -dir ` or a port already in use, as a
240+ initialization refusing a nonempty folder or a port already in use, as a
219241Surla bug. If someone later asks for cleanup, inspect each exact target
220242first: nested repositories, ` git status ` , unpushed commits, and whether
221243deleting it is authorized.
@@ -224,17 +246,18 @@ deleting it is authorized.
224246
225247When the tester decides ` stop_for_bug ` , it names the bug and gives the steps
226248to reproduce it. A blocking bug is one where Surla crashes, behaves
227- differently from its documentation, or offers no command to run
228- something its documented workflow needs, and no documented workaround
229- exists. The participant cannot get past it however well it uses Surla.
249+ differently from its documentation, or lacks an MCP/browser operation its
250+ documented workflow needs, and no documented workaround exists. A terminal
251+ workaround can unblock the product task while still failing the zero-command
252+ journey goal; record both outcomes rather than silently counting success.
230253
2312541 . Reproduce the bug yourself with the trusted Surla CLI, using the host and
232255 network access you are authorized to use, so a restriction in your coding
233256 agent is not mistaken for a Surla bug. The product's code is still
234- compiled, generated from, and run only through Surla's commands .
257+ compiled, generated from, and run only through Surla's documented operations .
235258 - If the failure comes from the harness, tell the tester "environment
236259 limitation, not a Surla bug" and which milestones it affects.
237- - A missing capability, meaning no Surla command for a step the
260+ - A missing capability, meaning no documented Surla operation for a step the
238261 documented workflow needs, counts as a real bug even when Surla lacks
239262 it by design. Go to step 2.
240263 - If Surla behaves as designed, tell the tester "reproduced; behavior is as
@@ -244,17 +267,18 @@ exists. The participant cannot get past it however well it uses Surla.
244267 separate clone and branch, the design gate, tests, an independent review,
245268 and a pull request. Fix the cause, not the symptom the participant
246269 happened to hit.
247- 3 . If the fix is in code that a scaffolded project downloads, the participant
248- would still get the old version through the README's ` -surla-version ` .
249- Push the fix. Then add a follow-up commit that sets ` -surla-version ` to the
250- fix commit's Go pseudo-version (` v0.0.0-<yyyymmddhhmmss>-<12-char-sha> ` ,
251- as ` go list -m github.com/goadesign/surla@<sha> ` prints it). This is the
252- same way main updates ` examples/orders ` after a release.
270+ 3 . If the fix is in code a starter downloads, publish a version containing
271+ it before restarting. Confirm that the documented installation and starter
272+ initialization resolve that version, rather than an earlier dependency.
273+ An exact Go pseudo-version is printed by
274+ ` go list -m github.com/goadesign/surla@<sha> ` ; update a documented version
275+ pin when that is what selects the dependency. Do not patch the participant's
276+ project or supply an undocumented override to make the attempt pass.
2532774 . Restart from scratch:
254278 1 . Stop processes and record leftovers as in section 3.
255279 2 . Create a new participant folder, ` ~/src/surla-ux-<id>-a<N+1>/ ` , holding
256280 a clone at the newest commit of the fix, including any
257- ` -surla- version` follow-up, and an empty ` work/ ` . Leave the previous
281+ version-pin follow-up, and an empty ` work/ ` . Leave the previous
258282 attempt's folder in place as evidence.
259283 3 . Copy the user docs from that commit into the tester's ` docs/ ` again, so
260284 the tester judges the participant against the docs it is reading.
@@ -271,6 +295,18 @@ set a limit on attempts, stop and report when it is reached.
271295
272296## 5. Debrief and report
273297
298+ Keep these outcomes separate: product built, verified, approved, and merged;
299+ one-time installation/setup; later user-entered terminal commands; commands
300+ the coding agent runs internally; harness-only invocations; and steps
301+ completed with help. For each applicable path, record observed success or why it was not
302+ exercised: live guide adoption, page recovery through the guide or companion,
303+ inline agent choices, builder-only builds with configured tests, one-family
304+ operation, missing Herdr, persisted choices/notices after reload, and human
305+ Merge with refusal recovery. Do not add fixtures or hints to force coverage.
306+ A source claim, successful command or assignment is not proof of a visible
307+ outcome. The report's severity ratings and transcript evidence remain the
308+ moderator's responsibility.
309+
274310When the tester decides ` end_session ` , it sends the participant its debrief
275311questions. Relay the questions and the answers. Then send the tester the text
276312of ` references/report-template.md ` and ask it to write ` report.md ` .
0 commit comments