Repository navigation
Description
The experiment action fails the job for each failed experiment in the results file (#154). That includes an experiment run with ci: false / ci=False, whose job was green with action 1.0.0. The action cannot skip those results, because the line that the SDKs write has no CI mode. The line also has no runsPerItem, so the summary can only say "118 of 120 runs completed", not "N items, M runs each". This must land before the next release moves experiment-action-v1. A major version is not planned.
It waits for telemetry-dev/telemetry.dev#617, which changes packages/sdk/src/experiments.ts and sdks/python/src/telemetry_dev/_experiments.py.
expected:
- Both SDKs add the CI mode and
runsPerItemto each line ofTELEMETRY_DEV_EXPERIMENT_RESULTS, next to the body ofGET /v1/experiments/{id} -
report.mjsdoes not fail the job for a result written with the CI mode off, and its summary says "N items, M runs each" fromitemsTotal / runsPerItemwhenrunsPerItemis above 1 - The score log of
runExperiment/run_experimentshows a number score, its pass mark, and its change with one decimal, as the comment and the experiment page do. TodayformatScoreinexperiments.tsuses two decimals, so 4.46 against a pass mark of 4.5 logs "4.46, under the pass mark 4.5" while the comment shows 4.5 and 4.5 -
docs/sdk-conformance.mddescribes the new fields, and the action README and both SDK READMEs no longer say thatci: falsefails the job
records of the QA run that #154 and #157 left, and why:
- VGAP1-CI-COMMENT-002: after a rename of the workflow, the job, the dataset, or the script, the old comment stays on the pull request. It now names its older commit. The marker comes from those names, and the action cannot tell a renamed step from another step of the same job
- G2 (GAP1-CI-COMMENT-002), check-status part: the job now fails for any failed experiment, but
check-statusis still the status of the last experiment. v1 documents that, and a change of the output would need a major version - GAP1-CI-COMMENT-009: a
commentvalue other than true or false gets a warning and still posts, as in 1.0.0. Rejecting it would break v1 workflows that pass such a value - GAP1-CI-COMMENT-007: with the token of a GitHub App, the action still changes a comment with its marker by any app. An app token cannot read its own login from
GET /user. The GITHUB_TOKEN case is fixed (onlygithub-actions[bot])
relevant files:
actions/experiment/report.mjsactions/experiment/README.mdpackages/sdk/src/ci.ts(writeResult)packages/sdk/src/experiments.tssdks/python/src/telemetry_dev/_ci.pysdks/python/src/telemetry_dev/_experiments.pypackages/sdk/tests/experiment-action.test.ts
Refs telemetry-dev/telemetry.dev#618, telemetry-dev/telemetry.dev#619, telemetry-dev/telemetry.dev#620
Activity