Skip to content

fix(warehouse-sources): upgrade pyarrow to 26.0.1, drop brotli guard - #136823

Merged
trunk-io[bot] merged 3 commits into
masterfrom
tom/warehouse-schema-heartbeat-silent-phase
Nov 17, 2026
Merged

trunk-io[bot] merged 3 commits into
masterfrom
tom/warehouse-schema-heartbeat-silent-phase

Conversation

@Gilbert09

@Gilbert09 Gilbert09 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Problem

Changes

  • pyarrow 26.0.1 bundles the fixed Brotli 1.2.1 (apache/arrow#55167), so the guard goes.
  • Object storage, Delta, Cost and Usage Report and autoresearch now read Brotli Parquet files. Delta scans start at once.
  • No other locked package changes. Snowflake's pyarrow cap covers only its uninstalled pandas extra.
  • The autoresearch sandbox image pins 26.0.1 too, so the sandbox writes scores with the same pyarrow the worker reads them with.
  • Mechanical: removed refusal tests, and one assertion now expects pyarrow 26's error text.

How did you test this code?

Test rationale: no new test, because no PostHog code handles Brotli now.

  • BrotliDecoderVersion in the Linux x86_64 wheels returns 1.1.0 for 23.0.1 and 1.2.1 for 26.0.1.
  • Ran only the changed test files locally. CI covers the suites and mypy.

Release status

  • No feature flag controls this change

Docs update

  • None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Opus 5.5 (claude-opus-5-5)

Skills: /writing-tests, /writing-pr-descriptions

Security review: needed

More context for AI agents
  • Session: Talyn session, PostHog MCP (logs, warehouse tables), GitHub REST proxy.
  • Decisions: parallel footer reads were the alternative; the person chose the upgrade and full removal.
  • Decisions: uv.lock keeps only the pyarrow block; the box's registry filter reorders a torch entry otherwise.
  • Decisions: the ClickHouse .br refusal in posthog/hogql/database/s3_table.py (fix(warehouse-sources): read warehouse files in chdb with an explicit codec #123070) stays; it covers ClickHouse's Brotli, not pyarrow's.
  • Read closely: uv.lock pyarrow block, delta.py iter_tables.

View this session in Talyn

@trunk-io

trunk-io Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Nov 16, 2026 — with Talyn App
@Gilbert09
Gilbert09 requested a review from a team November 16, 2026 23:42
@stamphog
stamphog Bot requested review from a team November 16, 2026 23:42
@veria-ai

veria-ai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Reviews

@bill-ph bill-ph left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed fd7402a against master, including the PR description and existing discussion, dependency and lockfile changes, removal of the Brotli guards in Delta/object storage, AWS Cost and Usage Report and autoresearch, and the associated test changes. Checked correctness, security, data safety, performance, test coverage, overengineering and scope creep. The Arrow 26.0.1 dependency manifest pins Brotli 1.2.1. Tests, builds and linters were not run, as requested.

No issues found.

— Robo Bill v2 (gpt-6.1-sol, high reasoning)

@github-actions

Copy link
Copy Markdown
Contributor

Hey @Gilbert09! 👋

It looks like your git author email on this PR isn't your @posthog.com address (owerstom@gmail.com). Since you're on the PostHog team, it's worth pointing your local git author email at your @posthog.com address. Why it matters:

  • Consistent work identity in git history — internal tooling that attributes commits to team members keys off your @posthog.com address.
  • Keeps team contributions easy to tell apart from external community ones when scanning history.

You can fix it for this repo with:

git config user.email "you@posthog.com"

Or set it globally with git config --global user.email "you@posthog.com". No need to redo this PR — just a nudge for next time. 🙂

@posthog-security-review-bot posthog-security-review-bot Bot added the security-review Request a security review label Nov 16, 2026
@parameterai

parameterai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Risk: No findings

This increment adds one test-only change (an async access-denied assertion in the object-store tests) and bumps modal 1.5.5 → 1.6.4 in pyproject.toml with a matching uv.lock pin. No security-relevant regression: modal 1.6.x's only security-adjacent change hardens block_network, and both repo call sites set it without ports.

Sentinel reviewed 323ce89 · Review settings

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

stamphog declined to auto-review this pull request because two gates failed. The deny-list gate matched the deps_toolchain category, most likely because pyproject.toml and uv.lock change the pyarrow dependency. The tier gate classified the change as T2-never, so it requires a human reviewer regardless of its size (195 lines across 13 files).

The author should ask a human reviewer to look at it. Since the dependency bump touches a security-relevant library, that reviewer should probably be someone who owns dependencies or security.

Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✗ matches: deps_toolchain
every_file_attested ✓ not a bot PR
size ✓ 67L, 8F substantive, 1 binary; 195L/13F incl. docs/generated/snapshots — within ceiling
tier ✗ classified as T2-never: T2-never (195L, 13F, single-area, fix)
stamphog 2.9.0 .stamphog/policy.yml @ unknown · reviewed head fd7402a

@stamphog stamphog Bot removed the stamphog Request AI approval (no full review) label Nov 16, 2026
@posthog-security-review-bot

posthog-security-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ Security review complete — fd7402a03b9b

1 finding posted as a review, in 6 min.

Add the security-review label to run again on the latest commit.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

ℹ️ MCP UI apps size — 35 app(s), 37024.1 KB JS

Built size of each MCP UI app (main.js + styles.css).

App JS CSS
debug 1067.4 KB 225.2 KB
explain-chart 1352.7 KB 225.2 KB
action 994.8 KB 225.2 KB
action-list 1058.5 KB 225.2 KB
cohort 993.8 KB 225.2 KB
cohort-list 1057.5 KB 225.2 KB
email-template 993.6 KB 225.2 KB
error-details 998.7 KB 225.2 KB
error-issue 995.7 KB 225.2 KB
error-issue-list 1059.7 KB 225.2 KB
experiment 1055.6 KB 225.2 KB
experiment-list 1059.3 KB 225.2 KB
experiment-results 1057.7 KB 225.2 KB
feature-flag 1057.9 KB 225.2 KB
feature-flag-list 1061.6 KB 225.2 KB
feature-flag-testing 997.0 KB 225.2 KB
inline-scan 994.3 KB 225.2 KB
insight-actors 1057.0 KB 225.2 KB
llm-costs 1054.3 KB 225.2 KB
session-recording 996.0 KB 225.2 KB
survey 995.4 KB 225.2 KB
survey-global-stats 1054.3 KB 225.2 KB
survey-list 1059.2 KB 225.2 KB
survey-stats 1054.3 KB 225.2 KB
trace-span 994.2 KB 225.2 KB
trace-span-list 1058.8 KB 225.2 KB
vision-observation-list 1057.6 KB 225.2 KB
workflow 994.1 KB 225.2 KB
workflow-list 1057.9 KB 225.2 KB
workflow-setup-status 993.8 KB 225.2 KB
query-results 1322.0 KB 225.2 KB
render-ui 1429.0 KB 225.2 KB
visual-review-snapshots 998.7 KB 225.2 KB
workflow-data-suggestions 996.5 KB 225.2 KB
workflow-sending-rules 995.1 KB 225.2 KB
⚠️ Playwright — 1 flaky

🎭 Playwright report · View test results →

⚠️ 1 flaky test:

  • a secret header entered in the workflow builder is stored encrypted and read back masked (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

✅ Hobby preview — passed

Hobby deployment smoke test passed successfully.


Run 44082460376

Comment thread pyproject.toml Resolved

@posthog-security-review-bot posthog-security-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent-driven security review - findings inline.

Comment thread pyproject.toml Resolved
@posthog-security-review-bot posthog-security-review-bot Bot removed the security-review Request a security review label Nov 16, 2026

Copy link
Copy Markdown
Member Author

/trunk merge

@trunk-io
trunk-io Bot merged commit 144bf9b into master Nov 17, 2026
298 checks passed
@trunk-io
trunk-io Bot deleted the tom/warehouse-schema-heartbeat-silent-phase branch November 17, 2026 13:14
@deployment-status-posthog

deployment-status-posthog Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-11-17 13:47 UTC Run
prod-us ✅ Deployed 2026-11-17 14:25 UTC Run
prod-eu ✅ Deployed 2026-11-17 14:31 UTC Run

This table shows the posthog release only. Hogstation shows every service that this PR reaches, Kargo services included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants