Skip to content

experiment action: write the ci mode and runsPerItem to the results file, and skip ci: false results #161

Closed

Description

@aikins01

The experiment action fails the job for each failed experiment in the results file (#154). That includes an experiment run with ci: false / ci=False, whose job was green with action 1.0.0. The action cannot skip those results, because the line that the SDKs write has no CI mode. The line also has no runsPerItem, so the summary can only say "118 of 120 runs completed", not "N items, M runs each". This must land before the next release moves experiment-action-v1. A major version is not planned.

It waits for telemetry-dev/telemetry.dev#617, which changes packages/sdk/src/experiments.ts and sdks/python/src/telemetry_dev/_experiments.py.

expected:

  • Both SDKs add the CI mode and runsPerItem to each line of TELEMETRY_DEV_EXPERIMENT_RESULTS, next to the body of GET /v1/experiments/{id}
  • report.mjs does not fail the job for a result written with the CI mode off, and its summary says "N items, M runs each" from itemsTotal / runsPerItem when runsPerItem is above 1
  • The score log of runExperiment / run_experiment shows a number score, its pass mark, and its change with one decimal, as the comment and the experiment page do. Today formatScore in experiments.ts uses two decimals, so 4.46 against a pass mark of 4.5 logs "4.46, under the pass mark 4.5" while the comment shows 4.5 and 4.5
  • docs/sdk-conformance.md describes the new fields, and the action README and both SDK READMEs no longer say that ci: false fails the job

records of the QA run that #154 and #157 left, and why:

  • VGAP1-CI-COMMENT-002: after a rename of the workflow, the job, the dataset, or the script, the old comment stays on the pull request. It now names its older commit. The marker comes from those names, and the action cannot tell a renamed step from another step of the same job
  • G2 (GAP1-CI-COMMENT-002), check-status part: the job now fails for any failed experiment, but check-status is still the status of the last experiment. v1 documents that, and a change of the output would need a major version
  • GAP1-CI-COMMENT-009: a comment value other than true or false gets a warning and still posts, as in 1.0.0. Rejecting it would break v1 workflows that pass such a value
  • GAP1-CI-COMMENT-007: with the token of a GitHub App, the action still changes a comment with its marker by any app. An app token cannot read its own login from GET /user. The GITHUB_TOKEN case is fixed (only github-actions[bot])

relevant files:

  • actions/experiment/report.mjs
  • actions/experiment/README.md
  • packages/sdk/src/ci.ts (writeResult)
  • packages/sdk/src/experiments.ts
  • sdks/python/src/telemetry_dev/_ci.py
  • sdks/python/src/telemetry_dev/_experiments.py
  • packages/sdk/tests/experiment-action.test.ts

Refs telemetry-dev/telemetry.dev#618, telemetry-dev/telemetry.dev#619, telemetry-dev/telemetry.dev#620

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions