Skip to content

ci(coverage): build coverage on push to master so Codecov compares the right base#854

Merged
IvanaGyro merged 2 commits into
masterfrom
claude/codecov-master-baseline
May 30, 2026
Merged

ci(coverage): build coverage on push to master so Codecov compares the right base#854
IvanaGyro merged 2 commits into
masterfrom
claude/codecov-master-baseline

Conversation

@IvanaGyro
Copy link
Copy Markdown
Collaborator

@IvanaGyro IvanaGyro commented May 29, 2026

Problem

The Tests and Codecov workflow (.github/workflows/ci-cmake_tests.yml) ran only on pull_request (to master) and workflow_dispatch. Coverage was therefore uploaded to Codecov only for PR-head commitsmaster HEAD (the post-merge state of the default branch) was never built directly, so Codecov never received a coverage report for it.

Because PRs land on master via merge commits, those PR-head commits sit on the second-parent side of history, off master's first-parent line. When a new PR opens, its base is the current master tip — a first-parent merge commit with no coverage report. Codecov then walks back the ancestry to the nearest commit that does have a report and uses that as the comparison base, which resolves to an arbitrary old PR-head commit far behind master.

Observed on #849: Codecov picked base 98e5842, which is 194 commits behind master and not on the first-parent line. Its own comment said "Report is 211 commits behind head." The "project" diff then showed Files +29, Lines +4844, Coverage 37.16% → 28.15% (−9.01%) — hundreds of commits of drift, not the 3-file / +172−17 change in the PR. The comparison is effectively useless.

Changes

  1. Add a push: branches: [master] trigger so the full build runs on master after every merge and uploads a coverage report for the actual master commit. The PR-only "Merge with latest target branch" step is already guarded by github.event_name == 'pull_request', so push builds test master directly. With a report attached to each master first-parent commit, Codecov can use the true PR base and produce a meaningful diff.

  2. Harden the workflow shell: defaults.run.shell was bash -l {0}. Overriding the shell discards GitHub Actions' default -eo pipefail, so multi-line run: blocks no longer aborted on the first failing command and pipe failures were masked. Switch to bash -leo pipefail {0} — keep the login shell (-l) for conda activation, add -e (fail on first non-zero command) and -o pipefail (propagate failures through pipes). This matches the fail-fast behavior conda-incubator/setup-miniconda documents (bash -el {0}) while also restoring pipefail.

Note on rollout

This fix takes effect going forward: Codecov gets a usable base for a PR once the master commit it is based on has its own coverage report — i.e. after this lands and the next merge fires the new push build. The first PR or two after merge may still show drift until master accumulates push-built reports.

Test plan

  • .github/workflows/ci-cmake_tests.yml parses as valid YAML; triggers are push, pull_request, workflow_dispatch; job shell is bash -leo pipefail {0}.

This is a CI-config-only change; no library code is touched, so the C++/Python test suites are unaffected.


Generated by Claude Code

IvanaGyro and others added 2 commits May 29, 2026 14:18
The "Tests and Codecov" workflow ran only on `pull_request` to master
and `workflow_dispatch`, so coverage was uploaded to Codecov solely for
PR-head commits. Master HEAD (the post-merge state of the default
branch) was never built directly, so Codecov never received a coverage
report for it.

Because PRs land on master via merge commits, those PR-head commits sit
on the second-parent side of history, off master's first-parent line.
When a new PR opens, its base is the current master tip -- a first-parent
merge commit that has no coverage report. Codecov then walks back the
ancestry to the nearest commit that does have a report and uses that as
the comparison base. In practice this resolves to an arbitrary old
PR-head commit far behind master (e.g. ~200 commits), so the project
coverage diff reflects hundreds of commits of drift rather than the
PR under review, making the comparison useless.

Add a `push: branches: [master]` trigger so the full build runs on
master after every merge and uploads coverage for the actual master
commit. The PR-only "Merge with latest target branch" step is already
guarded by `github.event_name == 'pull_request'`, so push builds test
master directly. With a coverage report attached to each master
first-parent commit, Codecov can use the true PR base and report a
meaningful diff.

Co-authored-by: Claude <noreply@anthropic.com>
The job set `defaults.run.shell: bash -l {0}` to get a login shell for
conda activation. Overriding the shell discards GitHub Actions' default
`bash --noprofile --norc -eo pipefail {0}`, so multi-line `run:` blocks
no longer aborted on the first failing command and pipe failures were
masked by the exit status of the last pipe stage. A dependency install
or build command could fail mid-step while the step still reported
success.

Switch to `bash -leo pipefail {0}`: keep the login shell (-l) for conda,
add -e so a step fails on the first non-zero command, and -o pipefail so
failures propagate through pipes. This matches the fail-fast behavior
conda-incubator/setup-miniconda documents (`bash -el {0}`) while also
restoring pipefail.

Co-authored-by: Claude <noreply@anthropic.com>
@gemini-code-assist
Copy link
Copy Markdown

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

@IvanaGyro IvanaGyro marked this pull request as ready for review May 29, 2026 14:33
@IvanaGyro IvanaGyro requested review from manuschneider and pcchen May 29, 2026 14:34
@codecov
Copy link
Copy Markdown

codecov Bot commented May 29, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 27.96%. Comparing base (98e5842) to head (a063b58).
⚠️ Report is 211 commits behind head on master.
✅ All tests successful. No failed tests found.

❗ There is a different number of reports uploaded between BASE (98e5842) and HEAD (a063b58). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (98e5842) HEAD (a063b58)
1 0
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #854      +/-   ##
==========================================
- Coverage   37.16%   27.96%   -9.20%     
==========================================
  Files         213      241      +28     
  Lines       30671    35495    +4824     
  Branches    12486    14781    +2295     
==========================================
- Hits        11399     9927    -1472     
- Misses      17438    18586    +1148     
- Partials     1834     6982    +5148     
Flag Coverage Δ
cpp 27.56% <ø> (?)
python 51.07% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
C++ backend 29.02% <17.28%> (∅)
Python bindings 16.89% <23.65%> (∅)
Python package 51.07% <86.00%> (∅)
see 165 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2a0d31...a063b58. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

# flags: -e aborts the step on the first failing command and
# -o pipefail propagates failures through pipes. Overriding the
# shell drops GitHub's default `-eo pipefail`, so set it back here.
shell: bash -leo pipefail {0}
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shell-flag inconsistency across workflows:
conda_build.yml / conda_build_release.yml → bash -el {0} (no pipefail)
coverity-scan.yml → step-level bash -l {0} in 4 places (neither errexit nor pipefail)
The other workflows have the same latent issue this PR fixes — fail-fast not enabled.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-el is fine if there is no pipe in the script. We should check if -l is needed for each workflow. Just use the default value for the workflow that doesn't need -l.

@IvanaGyro IvanaGyro merged commit f465599 into master May 30, 2026
9 of 10 checks passed
@IvanaGyro IvanaGyro deleted the claude/codecov-master-baseline branch May 30, 2026 09:47
@manuschneider
Copy link
Copy Markdown
Collaborator

The build on master failed, however:
https://github.com/Cytnx-dev/Cytnx/actions/runs/26680863969

@manuschneider manuschneider mentioned this pull request May 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants