Skip to content

Commit 1237cd8

Browse files
authored
Merge pull request #260 from goadesign/s3-views-docs
Add inline build team picker, one-click merge, degradation notices, and agent-first docs
2 parents 44f46f5 + b3e04da commit 1237cd8

44 files changed

Lines changed: 2458 additions & 447 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/surla-ux-review/SKILL.md‎

Lines changed: 77 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,14 @@ description: Run a moderated usability study of Surla. One subagent plays a firs
66
# Surla UX review
77

88
This skill runs a moderated, think-aloud usability test of Surla and turns it
9-
into a UX research report. Two subagents take part:
9+
into a UX research report. The product goal is a journey driven by the
10+
participant's existing coding-agent session: after installation and one-time
11+
`surla up`, the participant describes the goal through `/surla` (`$surla`),
12+
uses the browser for human decisions, and reaches an approved merge without
13+
typing terminal commands to move between stages. Commands the coding agent
14+
runs internally are distinct from commands the user must enter. Observe whether the product
15+
actually enables that journey; never supply missing steps as hidden coaching.
16+
Two subagents take part:
1017

1118
- **The participant** plays a developer who has never used Surla. It may read
1219
only Surla's `README.md` and whatever Surla itself prints or generates, and
@@ -69,45 +76,59 @@ source out of its view.
6976
`attempt-1/transcript.md`:
7077

7178
- the Surla commit, the date, and the OS;
72-
- whether the folder in the README's `surla init -dir` example (such as
73-
`/tmp/shop`) already exists. `surla init` refuses an existing folder, so a
74-
leftover would look like a usability problem. Do not delete it. Tell the
75-
tester it exists, labeled "ownership unconfirmed";
76-
- the `-surla-version` that the README tells `surla init` to use. A scaffolded
77-
project downloads that version, not the commit under test;
79+
- whether the participant's `work/` folder is empty apart from permitted
80+
coding-agent settings, and any known pre-existing example destinations.
81+
Leave existing folders alone and label their ownership unconfirmed;
82+
- the global setup writes this attempt may make. `surla up` and
83+
`surla setup -global` update the real machine's detected coding-agent MCP
84+
configuration and skill files, and may install the Herdr mod and add its
85+
free macOS shortcut in `keys.json`. Record the paths setup reports and
86+
pre-existing registrations or bindings without recording credentials.
87+
These files are outside `work/` and may be used by other sessions; do not
88+
delete or restore them as attempt cleanup;
89+
- the version of Surla installed by the README and the version its starter
90+
module will download. Record whether they match the commit under test;
7891
- the versions of `go`, `goa`, and `temporal`, or that they are missing;
7992
- which credentials are set, such as `ANTHROPIC_API_KEY`. Record that each is
8093
set, never its value;
8194
- any personal instruction files your harness loads into every subagent, such
8295
as a global `CLAUDE.md` or `AGENTS.md`. These can tell the participant
8396
things the README does not, so the report lists them as a limitation;
8497
- whether commands run inside your coding agent's own sandbox, and whether
85-
they have network access. `surla init` downloads modules, and Surla builds
98+
they have network access. Starter initialization downloads modules, and Surla builds
8699
and runs the code under test as ordinary processes with the user's file
87100
and network access. Working copies and process groups do not confine that
88101
code. Keep any coding-agent sandbox restrictions separate from Surla's
89102
documented execution policy. Give the participant the host access you are
90-
authorized to give. Trusted Surla commands run as the docs show, including
91-
`surla init`, which generates code from Surla's own unmodified starter
92-
templates. The product's own code is never compiled, generated from, or run
93-
except through Surla's commands. That
94-
includes `goa gen` on its design, which runs that design code. If the
95-
documented workflow needs a step Surla offers no command for, the study
103+
authorized to give. Trusted Surla operations run as the docs show,
104+
including initialization from Surla's own unmodified starter templates.
105+
The product's own code is never compiled, generated from, or run except
106+
through Surla's documented operations. That includes `goa gen` on its
107+
design, which runs that design code. If the
108+
documented workflow needs a step Surla offers no operation for, the study
96109
reports it as a missing capability. If the access Surla needs is not
97110
available, that is an environment limitation: the tester marks the
98111
milestones it blocks as not supported;
99-
- **how the participant reaches Surla's MCP server.** The README tells a user
100-
to point a coding agent at `http://<addr>/mcp`. A subagent cannot add an MCP
101-
server to its own session. The participant therefore runs its own
102-
coding-agent CLI non-interactively in `work/`, with an MCP configuration it
103-
writes itself, for example `claude -p`, `codex exec`, or `gemini -p`.
104-
Record which CLIs it may use, or that none can run here. Do not hand it an
105-
undocumented HTTP route, because working that out from outside the README
106-
is coaching.
112+
- **how the participant reaches Surla's MCP server.** The current README
113+
uses one-time `surla up` (or `surla setup -global`) to register `surla stdio`
114+
and the `/surla` (`$surla`) skill. A subagent cannot add an MCP server to
115+
its own session, so the participant may run an installed coding-agent CLI
116+
in `work/` and invoke the skill there. Record which clients can run, which
117+
load the global registration, whether a session reload is needed, and
118+
whether the environment can show the browser for human decisions. Record
119+
harness-only CLI launches separately from user workflow steps; a harness
120+
that cannot preserve the coding session is a study limitation. Do not
121+
hand it an HTTP route, credential, prewritten MCP configuration or lifecycle
122+
command. Failure to discover or use the documented integration is evidence;
123+
- whether Herdr and Chorus are installed, whether the agent starts inside a
124+
live Herdr pane, whether its client has been observed forwarding
125+
`HC_TERMINAL_ID` and provider settings to stdio, the installed agent
126+
families and background-worker support, and any project choices or
127+
dismissed recommendations already present. Do not install a second family just to hide the one-family path;
107128
- **what each mid-build change needs** (see the tester brief's "Mid-build
108129
changes"), recorded as two separate facts: what this machine provides,
109130
such as model credentials or the access a live demo needs, and what the
110-
commit under test provides, such as a command that regenerates code after a
131+
commit under test provides, such as an operation that regenerates code after a
111132
design change. The tester uses them to tell "not supported on this
112133
machine" apart from "blocked by a missing Surla capability".
113134

@@ -157,8 +178,8 @@ participant.
157178
Each round trip works like this:
158179

159180
1. The participant works until a natural pause and replies with a step
160-
report. Its brief lists when it pauses, including before the first run of
161-
each new `surla` command.
181+
report. Its brief lists when it pauses, including before one-time setup,
182+
its first skill invocation and each new human decision on the page.
162183
2. Append the report to the transcript and send it unchanged to the tester,
163184
together with the elapsed time since the previous report.
164185
3. The tester replies with up to three questions, an optional hint, an
@@ -190,7 +211,7 @@ harness keeps a subagent transcript. Also read the logs its coding agent saved
190211
in `work/agent-logs/`. A leak is reading anything outside the participant's
191212
allowed list, loading this skill, building the product without Surla, or
192213
compiling, generating from, or running the product's code except through
193-
Surla's commands.
214+
Surla's documented operations.
194215

195216
A participant cannot unlearn what it read, so a leak ends the attempt. Tell
196217
the tester what was read and when. The tester records the leak as a finding,
@@ -207,15 +228,16 @@ limitations that leaks were only self-reported.
207228
`ps -p <pid> -o command=` shows the reported binary and folder. PIDs get
208229
reused. Leave running any process you cannot match.
209230
- Delete nothing else. Every project folder the participant created is
210-
evidence for the report, including folders outside `work/` such as
211-
`/tmp/shop`, and may hold a Git repository with unpushed work. Each attempt
231+
evidence for the report, including a project initialized outside `work/`,
232+
and may hold a Git repository with unpushed work. Keep machine-wide setup
233+
changes recorded above; they are not disposable attempt files. Each attempt
212234
gets a new folder, so nothing needs to be cleared for the next one.
213235

214236
In the next transcript, record the exact path of every folder outside `work/`
215237
that the participant reported or that appears in `work/agent-logs/`, and
216238
every process left running. Label them "ownership unconfirmed". Tell the
217239
tester when the next attempt starts, so it does not log a leftover, such as
218-
`surla init` refusing an existing `-dir` or a port already in use, as a
240+
initialization refusing a nonempty folder or a port already in use, as a
219241
Surla bug. If someone later asks for cleanup, inspect each exact target
220242
first: nested repositories, `git status`, unpushed commits, and whether
221243
deleting it is authorized.
@@ -224,17 +246,18 @@ deleting it is authorized.
224246

225247
When the tester decides `stop_for_bug`, it names the bug and gives the steps
226248
to reproduce it. A blocking bug is one where Surla crashes, behaves
227-
differently from its documentation, or offers no command to run
228-
something its documented workflow needs, and no documented workaround
229-
exists. The participant cannot get past it however well it uses Surla.
249+
differently from its documentation, or lacks an MCP/browser operation its
250+
documented workflow needs, and no documented workaround exists. A terminal
251+
workaround can unblock the product task while still failing the zero-command
252+
journey goal; record both outcomes rather than silently counting success.
230253

231254
1. Reproduce the bug yourself with the trusted Surla CLI, using the host and
232255
network access you are authorized to use, so a restriction in your coding
233256
agent is not mistaken for a Surla bug. The product's code is still
234-
compiled, generated from, and run only through Surla's commands.
257+
compiled, generated from, and run only through Surla's documented operations.
235258
- If the failure comes from the harness, tell the tester "environment
236259
limitation, not a Surla bug" and which milestones it affects.
237-
- A missing capability, meaning no Surla command for a step the
260+
- A missing capability, meaning no documented Surla operation for a step the
238261
documented workflow needs, counts as a real bug even when Surla lacks
239262
it by design. Go to step 2.
240263
- If Surla behaves as designed, tell the tester "reproduced; behavior is as
@@ -244,17 +267,18 @@ exists. The participant cannot get past it however well it uses Surla.
244267
separate clone and branch, the design gate, tests, an independent review,
245268
and a pull request. Fix the cause, not the symptom the participant
246269
happened to hit.
247-
3. If the fix is in code that a scaffolded project downloads, the participant
248-
would still get the old version through the README's `-surla-version`.
249-
Push the fix. Then add a follow-up commit that sets `-surla-version` to the
250-
fix commit's Go pseudo-version (`v0.0.0-<yyyymmddhhmmss>-<12-char-sha>`,
251-
as `go list -m github.com/goadesign/surla@<sha>` prints it). This is the
252-
same way main updates `examples/orders` after a release.
270+
3. If the fix is in code a starter downloads, publish a version containing
271+
it before restarting. Confirm that the documented installation and starter
272+
initialization resolve that version, rather than an earlier dependency.
273+
An exact Go pseudo-version is printed by
274+
`go list -m github.com/goadesign/surla@<sha>`; update a documented version
275+
pin when that is what selects the dependency. Do not patch the participant's
276+
project or supply an undocumented override to make the attempt pass.
253277
4. Restart from scratch:
254278
1. Stop processes and record leftovers as in section 3.
255279
2. Create a new participant folder, `~/src/surla-ux-<id>-a<N+1>/`, holding
256280
a clone at the newest commit of the fix, including any
257-
`-surla-version` follow-up, and an empty `work/`. Leave the previous
281+
version-pin follow-up, and an empty `work/`. Leave the previous
258282
attempt's folder in place as evidence.
259283
3. Copy the user docs from that commit into the tester's `docs/` again, so
260284
the tester judges the participant against the docs it is reading.
@@ -271,6 +295,18 @@ set a limit on attempts, stop and report when it is reached.
271295

272296
## 5. Debrief and report
273297

298+
Keep these outcomes separate: product built, verified, approved, and merged;
299+
one-time installation/setup; later user-entered terminal commands; commands
300+
the coding agent runs internally; harness-only invocations; and steps
301+
completed with help. For each applicable path, record observed success or why it was not
302+
exercised: live guide adoption, page recovery through the guide or companion,
303+
inline agent choices, builder-only builds with configured tests, one-family
304+
operation, missing Herdr, persisted choices/notices after reload, and human
305+
Merge with refusal recovery. Do not add fixtures or hints to force coverage.
306+
A source claim, successful command or assignment is not proof of a visible
307+
outcome. The report's severity ratings and transcript evidence remain the
308+
moderator's responsibility.
309+
274310
When the tester decides `end_session`, it sends the participant its debrief
275311
questions. Relay the questions and the answers. Then send the tester the text
276312
of `references/report-template.md` and ask it to write `report.md`.

‎.agents/skills/surla-ux-review/references/participant-brief.md‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -31,16 +31,26 @@ You are taking part in a usability study of a developer tool called
3131
real first-time user works from the README and the tool, and that is what
3232
this study measures. Documents the README links to are not available. If
3333
you want one, say so in your report, because that is useful evidence.
34-
- Run Surla itself the way the README shows, including `surla init`, which
35-
sets up your starter project.
34+
- Install and set up Surla the way the README shows. After one-time setup,
35+
use your coding-agent conversation and Surla's browser page to reach the
36+
goal. Setup changes global coding-agent configuration and skill files,
37+
and may install the Herdr companion and its free macOS shortcut outside
38+
`work/`. Report the paths it changes; do not remove shared settings during
39+
cleanup. Report every terminal command you still need to enter as the user;
40+
distinguish commands your coding agent runs internally and harness-only
41+
agent launches. Do not hide a manual workaround as ordinary progress.
42+
- You may read and invoke the runtime `/surla` (`$surla`) skill that Surla
43+
installs for your coding agent. This is separate from the forbidden
44+
`surla-ux-review` study skill. Learn the entrypoint from the README; the
45+
facilitator will not configure a connection or supply a page credential.
3646
- Surla runs your product's code as ordinary processes with your file and
3747
network access. Its working copies and process groups do not confine that
3848
code. Any sandbox imposed by your coding-agent environment is a separate
3949
environment restriction.
4050
- Do not compile, generate code from, or run your product's code yourself,
4151
for example with `goa gen`, `go build`, `go run`, or `go test` in your
42-
project. Your product's code runs only through Surla's commands, such as
43-
`surla verify`. If Surla's README or generated files tell you to run one
52+
project. Your product's code runs only through Surla's documented
53+
operations. If Surla's README or generated files tell you to run one
4454
of those commands yourself, or you need a step Surla gives you no way to
4555
run, do not run it. Say so in your report instead.
4656
- This machine has: {{ENVIRONMENT_FACTS}}. Never print the value of a
@@ -49,9 +59,9 @@ You are taking part in a usability study of a developer tool called
4959
{{CODING_AGENTS}}. If you start one, save its full output under
5060
`{{PARTICIPANT}}/work/agent-logs/`. Tell it to use Surla, and tell it not to
5161
read Surla's source, use the `surla-ux-review` skill, or compile, generate
52-
code from, or run the product's code except through Surla's commands. If
53-
it needs such a step, it must stop and say so in its output, not skip the
54-
step or edit generated files by hand. Surla's source is
62+
code from, or run the product's code except through Surla's documented
63+
operations. If it needs such a step, it must stop and say so in its output,
64+
not skip the step or edit generated files by hand. Surla's source is
5565
the clone apart from its README, and `github.com/goadesign/surla` in the Go
5666
module cache. It may read other libraries it needs to write code.
5767
- Run long-running processes, such as servers, in the background. Report
@@ -70,8 +80,12 @@ Send a step report when any of these happens:
7080
- you finish a meaningful part of the goal;
7181
- you are stuck;
7282
- you are choosing between different ways of using Surla;
73-
- you are about to run a `surla` command for the first time. Report before
74-
running it, and say what you expect it to do;
83+
- you are about to install or set up Surla, invoke its skill for the first
84+
time, or make a new human decision in its browser page. Report before the
85+
action and say what you expect;
86+
- you need to enter a terminal command as the user after one-time setup to
87+
move the product forward; report the reason and what the interface made
88+
available;
7589
- about ten actions have passed since your last report.
7690

7791
Use this format:

0 commit comments

Comments
 (0)