Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 19 additions & 15 deletions .github/workflows/daily-experiment-report.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 18 additions & 7 deletions .github/workflows/daily-experiment-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ safe-outputs:
max: 10
add-labels:
max: 10
mentions: false
mentions: true
allowed-github-references: []
max-bot-mentions: 1

Expand Down Expand Up @@ -117,6 +117,7 @@ by the experiments CLI:
- Primary metric (`metric:` field), if set
- Secondary metrics (`secondary_metrics:` list), if set
- Tracking issue number, if an `issue:` field is set
- Notify target (`notify.issue` and/or `notify.discussion`), if set, to determine where readiness/significance/guardrail notifications should be posted

If no workflows declare `experiments:`, append the following to `$GITHUB_STEP_SUMMARY` and exit:

Expand Down Expand Up @@ -530,16 +531,20 @@ After the discussion is created, also write a one-line summary to `$GITHUB_STEP_
Daily experiment report: N experiments analysed, M reached significance (p < 0.05). Discussion: <url>
```

## Step 8 — Notify Tracking Issues
## Step 8 — Notify Experiment Owners

For each experiment that has a `issue:` field set, post a comment to that tracking issue when any
of the following conditions are met **for the first time today**:
For each experiment that has a `notify.issue` or `notify.discussion` target set, post a comment to
that target when any of the following conditions are met **for the first time today**. In this
workflow, the experiment owner is defined by convention as the repository owner, so every
notification comment MUST include `@${{ github.repository_owner }}`.

**Condition A — All variants reached `min_samples`:**
Post a comment:
```
🧪 **Experiment `<name>` is ready for analysis!**

Owner: @${{ github.repository_owner }}

All variants have reached the minimum sample size of `<min_samples>` runs:
<variant>: <n>/<min_samples>
...
Expand All @@ -553,6 +558,8 @@ Post a comment:
```
📊 **Experiment `<name>` has reached statistical significance (p = <p_value>)**

Owner: @${{ github.repository_owner }}

Recommendation: **<PROMOTE | EXTEND | ABANDON>**
<one sentence rationale>

Expand All @@ -564,15 +571,19 @@ Post a comment:
```
⚠️ **Guardrail violation in experiment `<name>`**

Owner: @${{ github.repository_owner }}

The following guardrail metric failed:
- `<metric_name>` expected `<threshold>`, got `<actual_value>` for variant `<variant>`

Recommendation: **ABANDON** — investigate immediately.
```

Use the `add-comment` safe-output tool to post comments. Skip experiments with no
`issue:` field. Do not post duplicate comments if the same condition was already reported in a
previous run today.
Use the `add-comment` safe-output tool to post comments. Select the target in this order:
1) `notify.issue` (when present), 2) `notify.discussion` (when present), 3) skip. If both are
present, post to `notify.issue` only so each notification is emitted once and tracking stays tied to
the canonical experiment thread.
Do not post duplicate comments if the same condition was already reported in a previous run today.

## Step 9 — Update Experiment Lifecycle Labels

Expand Down