Skip to content

feat(experiments): manage draft and running lifecycle - #180

Open
joshdholtz wants to merge 17 commits into
feat/experiments-readfrom
feat/experiments-lifecycle
Open

joshdholtz wants to merge 17 commits into
feat/experiments-readfrom
feat/experiments-lifecycle

Conversation

@joshdholtz

@joshdholtz joshdholtz commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Why

Let humans and agents create, duplicate, and manage experiments with clear review before enrollment changes.

Second PR in the stack: #179 → #180 → #181.

Changes

  • Add create, duplicate, update, delete, start, pause, resume, and stop. Duplication composes get and create and always creates a draft.
  • Accept simple create flags or a JSON config; rc schema experiments create|update describes config fields, placements, duration, conditions, and enums.
  • Preview configuration and require confirmation or --yes before start, pause, resume, running updates, stop, and delete.
  • Validate enrollment changes locally and add API, CLI, snapshot, and mock-server coverage.
  • Reject deletion of non-draft experiments before confirmation and sanitize conflict names in human notices.

Review notes

  • The development OpenAPI contract and backend validator allow enrollment from 1–100%. The dashboard guide says 10% minimum; the CLI follows the API contract.
  • Create, draft update, and draft delete were exercised on a disposable project. Start reached the API but required a published control paywall; publishing was blocked by the separate agent permission layer, so start/pause/resume/stop and duplicate still need a live run.

Verification

go test ./..., go vet ./..., make gen, and make preview passed locally after rebasing onto current main.


Note

Medium Risk
Changes affect live customer enrollment and offering assignment when experiments are started, paused, resumed, or updated; guards and --yes gates reduce accidental mutation but the operational impact is still significant.

Overview
Adds full experiment lifecycle management to rc experiments: create, duplicate, update, delete, start, pause, resume, and stop, wired to new development v2 API routes and documented in the command surface, CLI coverage map, and beta OpenAPI overlay.

Create/update accept simple flags or a JSON --config (stdin/file); rc schema experiments create|update exposes config_fields for agents (metrics, placements, targeting, duration, etc.). Duplicate copies configuration into a new draft via get+create (no results/status). Local validation covers enrollment 1–100%, mutual exclusion of audience_id vs targeting_conditions, and placements rules (fallback offering IDs require at least one placement_offerings entry).

Lifecycle mutations show the current experiment configuration first and require confirmation or --yes (start, pause, resume, running enrollment updates, stop, delete). Updates are status-gated: drafts accept partial JSON; running experiments only enrollment_percentage; paused/stopped cannot be edited; delete is draft-only. The API client gains create/update/delete and action endpoints; human output tweaks include a consolidated conflicts line and sanitized notices/confirmations. Broad API, CLI, schema, and snapshot test coverage accompanies the change.

Reviewed by Cursor Bugbot for commit b772a60. Bugbot is set up for automated code reviews on this repo. Configure here.

@joshdholtz
joshdholtz added this pull request to stack #182 September 25, 2026 14:26
@joshdholtz
joshdholtz force-pushed the feat/experiments-lifecycle branch 4 times, most recently from e36d06b to fc06edb Compare September 25, 2026 19:58
@joshdholtz
joshdholtz force-pushed the feat/experiments-lifecycle branch from fc06edb to 17890ca Compare September 28, 2026 17:52
@joshdholtz
joshdholtz marked this pull request as ready for review September 28, 2026 17:56
Copilot AI lite review requested due to automatic review settings September 28, 2026 17:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/config_schema.go Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4550e5a. Configure here.

Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/experiments_lifecycle.go
Comment thread internal/cli/experiments_lifecycle.go Outdated
@joshdholtz
joshdholtz requested a review from JZDesign October 6, 2026 22:23
Comment thread internal/cli/config_schema.go
if create {
return map[string]any{"type": "object", "properties": fields, "required": []string{"display_name", "enrollment_percentage", "offering_a_id", "offering_b_id"}}
}
audience["description"] = "Omit to keep current targeting unless targeting_conditions is supplied. Set null to clear the audience and conditions, or supply targeting_conditions to replace them. A string selects an audience and cannot be combined with targeting_conditions."

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, My null parameter comment caused you to mark this one omitable/nullable too, but this one returns a 422 when omitted

@joshdholtz
joshdholtz requested a review from JZDesign October 7, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants