Skip to content

test: ttl_expire, the #1339 findings in both twins: a timestamp column, a zone that always differs, accurate premises - #1353

Merged
jdatcmd merged 3 commits into
commandprompt:mainfrom
OffgridwithJD:test/1339-ttl
Oct 14, 2026
Merged

jdatcmd merged 3 commits into
commandprompt:mainfrom
OffgridwithJD:test/1339-ttl

Conversation

@OffgridwithJD

Copy link
Copy Markdown
Collaborator

Part of #1339, batch three of the files you assigned me: test_ttl_expire and its shell twin. Red first, on PG17, main against this branch.

What changes

finding change red first
:86: no timestamp retention column anywhere New arm in both twins. A timestamp column expired in Kiritimati (UTC+14): 6,000 rows, six groups of ten hours, starting 40 h before the local cutoff, so four groups go and 2000 rows from id 4001 survive. Cutoff from the UTC wall clock: main has no arm to redden; here got [4000|2001] want [2000|4001] in both twins.
:608 (weak): Pacific/Midway shares UTC's date from 11:00 UTC on The second zone is chosen at run time to be on another date: Midway before 11:00 UTC, Kiritimati from 10:00. A premise says so. The arms name it "a zone on another date", so their ledger keys don't move with the clock. Run at 22:00 UTC with the date cutoff from the UTC wall clock: main 67/67, pytest green (Midway had UTC's date). Here: got [3000] want [2000] and got [1000] want [0], red in both twins.
:121 (inaccurate): "rows exist on both sides of the cutoff" counted one two-day window Each side is counted on its own. An accuracy fix; no mutation separates the two.
:209 (inaccurate): "the SAME row group holds expired rows and NULL ones" counted groups with NULLs only It also requires an expired row in that group, with rows mapped to their group through row_tid() over the group's row numbers. Fixture edit putting the NULLs in a statement of their own: main's premise passes; here got [1|false] want [1|true].
:304, :415, :718, :731 (mislabelled) Four premise: checks that test expire, VACUUM or the zone map become arms (#1236), renamed in both twins. With expire returning 0, the two expire ones are red (n=0, got [0] want [2]). The VACUUM and zone-map ones have no mutation here and stay never.

Evidence

  • Clean: ttl_expire 70/70 and pytest 10 passed on each of 15–19. compare_to_bash: 0 missing. shellcheck -S error is clean (the SC2034 warning is on main too). The guard half passes.
  • Ledger:
    • Three new rows: the timestamp arm (dated) and two premises (never).
    • Eight rows renamed with their history: the four de-premised checks, and the four Midway checks now named "a zone on another date".
    • Expire returning 0 also reddens three existing arms never before seen red: "a group whose only NULLs have been deleted is retired", "and the table lost exactly the groups it retired", "expire retires the all-visible expired group".
    • Totals: rows 2316 → 2319, never 1796 → 1793. No dated premise: added (27, as on main). Gate: census agrees, ceiling 246.

Left

🤖 Generated with Claude Code

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

A correction to my own PR, pushed as ad1cf94. The new pytest test had no TESTS.md row, and cluster_tests didn't count it. The guard half caught the first (test_docs_cover_the_corpus: 1 failed), and I opened this before reading that result. Both are fixed: a TESTS.md row, and cluster_tests 532 → 533 by collection (533 collected). The guard half now gives 425 passed. Nothing else changed.

@jdatcmd

jdatcmd commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

@OffgridwithJD, I reviewed ad1cf94 and reproduced the Midway claim on PG18 at 22:33 UTC, the case you describe. I changed expire's DATE cutoff to take the UTC wall clock (TimestampGetDatum(GetCurrentTimestamp()) instead of timestamptz_timestamp(...)):

  • main: 67/67. Midway shares UTC's date at that hour, so nothing sees it.
  • this head: 68 + 2 failed: "expire in a zone on another date keeps exactly the rows at or after that session's cutoff: got [3000] want [2000]", and "…survives: got [1000] want [0]". Your numbers.

The zone choice holds up. Midway (UTC−11) is on another date while the UTC hour is below 11, and Kiritimati (UTC+14) from 10:00, so the < 11 split always lands on one that differs, and the premise re-checks it.

A minor race, not blocking: the zone is picked in one statement and the arms run later. A run crossing 11:00 UTC exactly could pick Midway a moment before its date rejoins UTC's. The premise would then fail, so the failure is loud, not a false pass.

The timestamp arm's arithmetic also checks out: ten-hour groups, 40 h before the local cutoff, so a local cutoff gives 4 retired and a UTC cutoff gives 2. The row_tid() mapping for the NULL group and the per-side straddle counts read right.

Needs a rebase before I approve. Base d0ae0b1; main is d926b62 at 2321 / 1798. With your three new rows (one dated, two never) and three newly dated arms, that should come to 2324 / 1795. #1350 and #1351 are queued too; whoever lands later restates.

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the content at ad1cf94. At 22:33 UTC on PG18 I reproduced the Midway claim: with expire's DATE cutoff taken from the UTC wall clock, main passes 67/67 and this head fails with got [3000] want [2000] and got [1000] want [0]. The zone choice, the timestamp arm's arithmetic, the row_tid grouping and the per-side straddle counts all check out. The race at exactly 11:00 UTC fails loudly through your premise, so it isn't blocking.

Not mergeable yet: check_ledger_budget.txt conflicts with main (74cde39) at 2321 / 1798. With your rows I expect 2324 / 1795. Rebase, restate from a recount, and I'll check the ledger and merge.

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd, rebased onto main 74cde39 (#1346 and #1352 merged). The new head is efa0235, replacing ad1cf94.

Thanks for the 11:00 UTC race note. Agreed that the premise makes it loud rather than a false pass, so I've left it.

@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd, rebased again, onto 95b4112 (#1347 merged). The new head is 4881a1f, replacing efa0235.

Whichever of this and #1354 lands second restates.

jdatcmd and others added 3 commits October 14, 2026 00:47
- :86 (missing): no arm had a `timestamp` retention column. A new arm expires
  one in Kiritimati, UTC+14, and requires the groups wholly before the
  session's LOCAL cutoff to go: 2000 rows from 4001 survive. With the cutoff
  taken from the UTC wall clock: main has nothing to redden; here
  "got [4000|2001] want [2000|4001]" in both twins.
- :608 (weak): the second zone was Pacific/Midway, which shares UTC's date
  from 11:00 UTC on, so an expire ignoring the session zone passed for half
  of every day. It is now chosen at run time to be on another date (Midway
  before 11:00 UTC, Kiritimati from 10:00), with a premise saying so, and the
  arms name it "a zone on another date" so their ledger keys hold still.
  Run at 22:00 UTC with the date cutoff taken from the UTC wall clock: main
  67/67 and pytest green; here red in both twins.
- :121 (inaccurate): "rows exist on both sides of the cutoff" counted one
  two-day window. Each side is counted now. An accuracy fix.
- :209 (inaccurate): "the SAME row group holds expired rows and NULL ones"
  counted groups with NULLs only. It now also requires an expired row in
  that group, mapped through row_tid(). With the NULLs inserted in a
  statement of their own: main's premise passes; here "1|false".
- :304, :415, :718, :731 (mislabelled): four checks named premise: that test
  expire, VACUUM or the zone map are arms (commandprompt#1236). With expire returning 0,
  the two expire ones are red.
Clean: 70/70 and pytest 10 passed on each of 15-19.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five-major clean runs and three product mutations, each merged with
--target naming every arm it reddens; eight rows renamed with their
history. Rows 2316 -> 2319, never 1796 -> 1793: three existing arms are
seen red for the first time (expire returning 0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…commandprompt#1339)

test_a_timestamp_retention_column_uses_the_sessions_local_time is a new
test, so it needs its TESTS.md row (test_docs_cover_the_corpus failed
without it) and cluster_tests 532 -> 533, by collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@OffgridwithJD

Copy link
Copy Markdown
Collaborator Author

@jdatcmd, rebased onto bd99606 (#1351 merged). The new head is 6786f2c, replacing 4881a1f.

Ledger: 2343 / 1798, restated on top of #1351's block. Only check_ledger_budget.txt conflicted. The tsv equals main plus my delta, with no duplicate keys.

Patch unchanged. Outside the budget, the changed lines match 4881a1f exactly. #1350 and #1351 didn't touch the ttl files.

Evidence still applies. Since this PR's five-major runs (base d0ae0b1), main changed no src/ or SQL file, only the Makefile's #1346 refusal. So ttl_expire 70/70 and test_ttl_expire.py 10 passed on each of 15–19 still describe these test files on this product. CI adds 17 and 18.

Checks.

  • Gate: --against origin/main gives rc=0, the census agrees at 1798, and the ceiling stays 246 → 246.
  • Database-free set: 429 passed.
  • Cluster collection: one above main, matching cluster_tests 533.

I'll ping on green CI.

@jdatcmd jdatcmd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving 6786f2c.

  • Ledger: a recount gives 2343 / 1798, which agrees with the budget. No duplicate keys, no dated premise: row.
  • Patch: the four changed test and doc files carry the same patch as ad1cf94, the head I reviewed and reproduced.
  • Counts, on this head: database-free set 429 passed; cluster collection 533, matching cluster_tests.

Merging.

@jdatcmd
jdatcmd merged commit 25118c0 into commandprompt:main Oct 14, 2026
15 checks passed
OffgridwithJD pushed a commit to OffgridwithJD/pgcolumnar that referenced this pull request Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants