Found while comparing the harness against
KiroCrew, where "corrections and task
failures become durable lessons".
This issue is a decision before it is a task. The mechanism is easy; whether
it is compatible with this repository's measurement discipline is the actual
question, and it should be ruled on before anything is built.
The mechanism
Each item is executed independently. There is no store of project-scoped
constraints, and nothing carries a correction from one item to the next:
$ grep -rin "lesson" src/
$
So a project whose checks reject a particular formatting rule, or whose build
needs a step no prompt mentions, rejects it once per item. Twenty items fail the
same gate twenty times, and each failure is a full planner and implementer call
paid again. Related: #155, where a gate no model reliably passes is already
recognised as a repeated cost.
The case for
The definition of done includes:
Delivery rate is no worse than the workload's own pre-harness baseline, at
lower cost.
A human working the same backlog learns the constraint on item one. The harness
cannot, and pays the difference on every item after it. This is one of the few
identified costs that is repeated, mechanical, and attributable.
The case against, which is the stronger one today
A lesson store mutates the implementer's prompt between items. That breaks the
property this repository is built to protect:
- Two runs are no longer comparable, because they did not run the same prompt.
- "Observed" becomes unfalsifiable — a run that improved cannot be
distinguished from a run that improved because of a lesson nobody read.
- The README's own three-column discipline (tested / observed / proven) has no
column for behaviour that changes itself, and audit is explicitly a record
rather than a behaviour store.
An unversioned lesson store would quietly convert the harness from something
measurable into something that merely feels better over time.
Conditions that would make it acceptable
If it is built, all of these, not a subset:
- Versioned. A lesson set has an identity, and every attempt records which
version it ran under. Without this, no two runs are comparable and the
baseline machinery is worthless.
- Project-scoped. Never global. A constraint learned on one repository is
not evidence about another.
- Derived from evidence, not introspection. A lesson comes from a check-gate
failure with the failing output attached — not from asking a model what it
learned. Model-authored self-assessment is exactly the unfalsifiable input the
repository refuses elsewhere.
- Inspectable and reversible. Listed over the API, readable as text,
individually removable. A lesson that cannot be deleted is a prompt change
nobody approved.
- Attributable in the audit. It must be possible to ask which items ran with
which lessons, and answer it after the fact.
What would settle whether to build it at all
Data this repository does not yet have: across a real multi-item run, how many
check failures share a cause with an earlier item's check failure? If the answer
is near zero, this is cost with no benefit and should be closed. #155 and the
soak runs (#33, #44, #51) would produce that number as a by-product.
Recommendation: do not build this until that number exists. It is filed so
the reasoning is recorded, not because it is ready.
Blind spots
Found while comparing the harness against
KiroCrew, where "corrections and task
failures become durable lessons".
This issue is a decision before it is a task. The mechanism is easy; whether
it is compatible with this repository's measurement discipline is the actual
question, and it should be ruled on before anything is built.
The mechanism
Each item is executed independently. There is no store of project-scoped
constraints, and nothing carries a correction from one item to the next:
So a project whose checks reject a particular formatting rule, or whose build
needs a step no prompt mentions, rejects it once per item. Twenty items fail the
same gate twenty times, and each failure is a full planner and implementer call
paid again. Related: #155, where a gate no model reliably passes is already
recognised as a repeated cost.
The case for
The definition of done includes:
A human working the same backlog learns the constraint on item one. The harness
cannot, and pays the difference on every item after it. This is one of the few
identified costs that is repeated, mechanical, and attributable.
The case against, which is the stronger one today
A lesson store mutates the implementer's prompt between items. That breaks the
property this repository is built to protect:
distinguished from a run that improved because of a lesson nobody read.
column for behaviour that changes itself, and
auditis explicitly a recordrather than a behaviour store.
An unversioned lesson store would quietly convert the harness from something
measurable into something that merely feels better over time.
Conditions that would make it acceptable
If it is built, all of these, not a subset:
version it ran under. Without this, no two runs are comparable and the
baseline machinery is worthless.
not evidence about another.
failure with the failing output attached — not from asking a model what it
learned. Model-authored self-assessment is exactly the unfalsifiable input the
repository refuses elsewhere.
individually removable. A lesson that cannot be deleted is a prompt change
nobody approved.
which lessons, and answer it after the fact.
What would settle whether to build it at all
Data this repository does not yet have: across a real multi-item run, how many
check failures share a cause with an earlier item's check failure? If the answer
is near zero, this is cost with no benefit and should be closed. #155 and the
soak runs (#33, #44, #51) would produce that number as a by-product.
Recommendation: do not build this until that number exists. It is filed so
the reasoning is recorded, not because it is ready.
Blind spots
repeated, which has not been tested.
behaviour at all is unknown, and would need its own A/B — the same shape as
T43: A/B whether the reviewer seeing the plan changes its verdict #84.