e2e tests on Cirrus Labs runners #5485

alwx · 2025-12-18T10:38:02Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Enabled Cirrus Labs runners for e2e tests. Fixes #5418

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
I updated the docs if needed.
I updated the wizard if needed.
All tests passing
No breaking changes

🔮 Next steps

github-actions · 2025-12-18T10:38:37Z

	Fails
🚫	Please consider adding a changelog entry for the next release.

Instructions and example for changelog

Please add an entry to CHANGELOG.md to the "Unreleased" section. Make sure the entry includes this PR's number.

Example:

## Unreleased

### Features

- e2e tests on Cirrus Labs runners ([#5485](https://github.com/getsentry/sentry-react-native/pull/5485))

If none of the above apply, you can opt out of this check by adding #skip-changelog to the PR description or adding a skip-changelog label.

Generated by 🚫 dangerJS against 6a25aaa

.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


github-actions · 2026-01-26T14:18:26Z

Semver Impact of This PR

⚪ None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

e2e tests on Cirrus Labs runners by alwx in #5485

_{🤖 This preview updates automatically when you update the PR.}

github-actions · 2026-01-28T11:02:23Z

Android (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	426.59 ms	454.29 ms	27.69 ms
Size	43.75 MiB	48.40 MiB	4.65 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	501.18 ms	536.70 ms	35.52 ms
`bfe454a`+dirty	573.44 ms	579.46 ms	6.02 ms
`294387d`+dirty	424.30 ms	465.40 ms	41.10 ms
`955f2eb`+dirty	422.74 ms	410.19 ms	-12.55 ms
`3bd3f0d`+dirty	447.21 ms	472.31 ms	25.10 ms
`d081295`+dirty	408.08 ms	453.62 ms	45.54 ms
`ad27f6e`+dirty	471.44 ms	516.23 ms	44.79 ms
`07808fb`+dirty	419.10 ms	419.08 ms	-0.02 ms
`170d5ea`+dirty	407.92 ms	422.49 ms	14.57 ms
`8e653ac`+dirty	360.28 ms	372.04 ms	11.76 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`bfe454a`+dirty	17.75 MiB	19.69 MiB	1.94 MiB
`294387d`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`955f2eb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`3bd3f0d`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`d081295`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`ad27f6e`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`07808fb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`170d5ea`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`8e653ac`+dirty	17.75 MiB	19.75 MiB	2.00 MiB

github-actions · 2026-01-28T11:02:34Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	441.85 ms	465.69 ms	23.83 ms
Size	43.94 MiB	49.26 MiB	5.32 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	422.90 ms	468.30 ms	45.40 ms
`294387d`+dirty	359.44 ms	393.40 ms	33.97 ms
`1d62dde`+dirty	366.59 ms	408.80 ms	42.21 ms
`d081295`+dirty	416.95 ms	461.24 ms	44.29 ms
`ad27f6e`+dirty	484.67 ms	532.79 ms	48.12 ms
`1226664`+dirty	377.65 ms	453.94 ms	76.29 ms
`083f560`+dirty	383.96 ms	417.76 ms	33.80 ms
`ec14be7`+dirty	401.58 ms	475.26 ms	73.68 ms
`eb07ba3`+dirty	419.49 ms	482.12 ms	62.63 ms
`d1fd647`+dirty	374.46 ms	409.51 ms	35.05 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`294387d`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`1d62dde`+dirty	7.15 MiB	8.46 MiB	1.31 MiB
`d081295`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`ad27f6e`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`1226664`+dirty	7.15 MiB	8.46 MiB	1.30 MiB
`083f560`+dirty	7.15 MiB	8.43 MiB	1.28 MiB
`ec14be7`+dirty	7.15 MiB	8.42 MiB	1.26 MiB
`eb07ba3`+dirty	7.15 MiB	8.42 MiB	1.27 MiB
`d1fd647`+dirty	7.15 MiB	8.43 MiB	1.28 MiB

.github/workflows/e2e-v2.yml

github-actions · 2026-01-29T09:24:54Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1211.33 ms	1218.35 ms	7.02 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1207.02 ms	1199.27 ms	-7.76 ms
`d73150f`+dirty	1224.52 ms	1227.17 ms	2.65 ms
`d081295`+dirty	1205.24 ms	1207.52 ms	2.28 ms
`d1fd647`+dirty	1218.16 ms	1225.82 ms	7.65 ms
`ea3e26e`+dirty	1216.61 ms	1214.15 ms	-2.47 ms
`80e4616`+dirty	1206.90 ms	1205.94 ms	-0.96 ms
`955f2eb`+dirty	1225.78 ms	1239.27 ms	13.49 ms
`5ee3314`+dirty	1234.25 ms	1235.44 ms	1.19 ms
`70250df`+dirty	1211.96 ms	1222.31 ms	10.35 ms
`664c66f`+dirty	1195.94 ms	1194.80 ms	-1.14 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	3.19 MiB	4.56 MiB	1.37 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	3.19 MiB	4.55 MiB	1.36 MiB
`5ee3314`+dirty	3.19 MiB	4.55 MiB	1.37 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

github-actions · 2026-01-29T09:24:54Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1215.32 ms	1224.27 ms	8.95 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1212.20 ms	1217.89 ms	5.70 ms
`d73150f`+dirty	1198.44 ms	1210.06 ms	11.62 ms
`d081295`+dirty	1214.40 ms	1211.27 ms	-3.13 ms
`d1fd647`+dirty	1219.35 ms	1233.18 ms	13.83 ms
`ea3e26e`+dirty	1229.13 ms	1228.46 ms	-0.67 ms
`80e4616`+dirty	1221.32 ms	1225.64 ms	4.32 ms
`955f2eb`+dirty	1235.06 ms	1253.88 ms	18.81 ms
`5ee3314`+dirty	1215.18 ms	1207.64 ms	-7.54 ms
`70250df`+dirty	1214.51 ms	1215.04 ms	0.53 ms
`664c66f`+dirty	1215.37 ms	1221.30 ms	5.92 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	2.63 MiB	3.99 MiB	1.36 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	2.63 MiB	3.98 MiB	1.35 MiB
`5ee3314`+dirty	2.63 MiB	3.99 MiB	1.35 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

.github/workflows/e2e-v2.yml

alwx · 2026-01-29T11:12:17Z

.github/workflows/e2e-v2.yml

+          echo "=== Check for conflicting idb installations ==="
+          find /Users/admin -name "idb_companion" -o -name "idb-companion" 2>/dev/null | head -10 || true
+          ls -la /opt/homebrew/bin/idb* 2>/dev/null || true
+


I think it's useful to have it here just to help us debugging potential problems that might arise (it helped me a lot)

lucas-zimerman · 2026-01-29T14:54:58Z

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

is it ready for review?

e2e tests on Cirrus Labs runners

e2397d0

alwx self-assigned this Dec 18, 2025

alwx marked this pull request as ready for review December 18, 2025 10:44

alwx requested review from antonis and lucas-zimerman as code owners December 18, 2025 10:44

alwx added the ready-to-merge Triggers the full CI test suite label Dec 18, 2025

alwx changed the title ~~e2e tests on Cirrus Labs runners~~ DO NOT MERGE: e2e tests on Cirrus Labs runners Dec 18, 2025

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 18, 2025 14:08

Checking another group

a04f351

rolling back last change

83a3d6a

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 23, 2025 09:36

Merge branch 'main' into alwx/experiment/cirrus-labs

1cb4e72

Merge branch 'main' into alwx/experiment/cirrus-labs

bd991eb

alwx added 3 commits January 28, 2026 09:08

Corepack enable

a933ded

Corepack enable

9899e3d

Ninja

61211d5

Ruby install

b5c48c1

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

Ruby install

9142e2f

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

alwx added 2 commits January 29, 2026 10:06

Updated env params

da9811a

Trying to fix metrics job

3034d58

alwx changed the title ~~DO NOT MERGE: e2e tests on Cirrus Labs runners~~ e2e tests on Cirrus Labs runners Jan 29, 2026

Merge branch 'main' into alwx/experiment/cirrus-labs

8c3a337

Trying it out with Macos 26.0.1

550e671

sentry bot reviewed Jan 29, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

alwx added 3 commits January 29, 2026 11:20

iOS version env param change

064ae98

Debug outputs

78bf149

iDB companion update to potentially fix things on iOS side

92b254f

alwx commented Jan 29, 2026

View reviewed changes

alwx added 8 commits January 29, 2026 12:34

Another attempt

bd23cc4

Typo

7da4011

GH runners for tests

4c7db57

e2e fix attempt (with a temp workaround of using GH runners for tests)

d9d34c0

Rolling back some of experiments

0c1d899

Solving concurrency issues

3ca98e6

More GH runner issues

a685372

Merge branch 'main' into alwx/experiment/cirrus-labs

6a25aaa

Uh oh!

e2e tests on Cirrus Labs runners #5485

Are you sure you want to change the base?

e2e tests on Cirrus Labs runners #5485

Conversation

alwx commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

github-actions bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Instructions and example for changelog

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Semver Impact of This PR

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

Uh oh!

alwx Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucas-zimerman commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alwx commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Jan 26, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

alwx Jan 29, 2026 •

edited

Loading