Measure the CI cost after the #834 changes #846

Activity

1. laughingman7743

   added

   enhancement

   on Sep 26, 2026
2. laughingman7743

   added a parent issue on Sep 26, 2026

   • Reduce the AWS cost of integration tests without reducing real-service coverage #834
3. laughingman7743 commented on Sep 28, 2026

   

   laughingman7743

   on Sep 28, 2026

   MemberAuthor

   More actions

   Measurement window, now that the last cost-related changes of #834 have merged:

   • Before: 2026-09-15 to 2026-09-28. The window ends on the day Give the executemany and partition tests their own tables #873 and Define the shared test tables in Python and generate their data #874 merged (Redesign the PyAthena test fixtures: define once, generate data, set up once per job #848's merged steps); the earlier changes (Run AWS test suites only on ready pull requests and related changes #837, Batch SQLAlchemy executemany inserts into multi-row INSERT statements #838, Run pull-request AWS tests on the newest Python version only #863, Run the full test matrix in the Release workflow before publishing #862, Set up the PyAthena test fixtures only in processes that run tests #866, Insert SQLAlchemy compliance fixture rows once for read-only classes #867) also fall in it.
   • After: 2026-09-29 to 2026-10-12, with no further cost changes planned. Measure on or after 2026-10-13.

   Because some of the earlier changes landed inside the "before" window, it is a mixed baseline. Also compare against 2026-09-01 to 2026-09-14, which predates all of #834.

   What to compare, from Cost Explorer with daily granularity, for the test account:

   • S3: PutObject requests and cost
   • Athena: SQL and Spark DPU
   • Glue requests
   • Total

   Normalize by activity with the Test workflow runs in each window (gh run list --workflow test.yaml): the number of runs and of AWS jobs, split by event (pull_request, schedule, dispatch, release). Report the cost per run as well as the totals.

   #834 closes after this measurement.
4. laughingman7743 commented on Oct 10, 2026

   

   laughingman7743

   on Oct 10, 2026

   MemberAuthor

   More actions

   Measurement results (Cost Explorer, daily granularity, test account).

   The test account started incurring test costs on 2026-09-18 (2026-09-02 to 2026-09-17 are close to $0 per day), so the 2026-09-01 to 2026-09-14 comparison does not apply. The "before" window is 2026-09-18 to 2026-09-28, and the "after" window is the same length, 2026-09-29 to 2026-10-09 (measured on 2026-10-10 instead of waiting for 2026-10-12).

   Window	Total	Per day	Excluded
   Before: 2026-09-18 to 2026-09-28	$193.5	$17.6	EC2 $25.0 (one-off #644 benchmark)
   After: 2026-09-29 to 2026-10-09	$25.8	$2.3	Tax $2.25 (September tax booked on 2026-10-01)

   Per day by service (before → after):

   Service	Before	After
   S3	$9.4	$1.2
   Athena (SQL + Spark)	$6.4	$1.1
   Glue	$1.3	$0.04

   • About an 87% reduction per day, from roughly $530 to roughly $70 per month.
   • The highest day in the "after" window was $9.0 (2026-10-03), compared with $30 to $37 on the busiest "before" days.
   • Test workflow runs went from 243 to 758, mostly Draft PRs that now run lint only without AWS. AWS jobs were not counted per run, so no cost-per-run figure is given.

   The reduction is clear, so this closes the measurement.
5. laughingman7743

   closed this as completedon Oct 10, 2026