Skip to content

fix(gateway): prevent replay of delivered queries (FB-4383) - #111

Open
simonzelenskifbio wants to merge 4 commits into
mainfrom
FB-4383-request-safety
Open

simonzelenskifbio wants to merge 4 commits into
mainfrom
FB-4383-request-safety

Conversation

@simonzelenskifbio

@simonzelenskifbio simonzelenskifbio commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Background

FB-4383: retrying a generic upstream reset can replay a SQL mutation that the Engine already received and executed.

Summary

Replace reset with reset-before-request in the Gateway retry policy. Retain connection-failure, refused-stream, and the Engine's explicit pre-work X-Firebolt-Drained retries. Generic failures after delivery are returned to the client rather than automatically replayed.

Add render regression coverage and a Docker fixture using the chart's Envoy image and rendered retry policy. The fixture checks that a disconnect after accepting a mutation produces one upstream call, while an explicit pre-work drained response can retry.

This change reduces automatic replay risk; it does not establish exactly-once execution or tell a client whether an interrupted mutation committed. Signing-key rotation guidance belongs to the final documentation PR, #120 (FB-4481).

Test Plan

  • Helm 3.16.3 strict lint passed after the split; retry implementation and fixtures are unchanged by the split.
  • Previously executed real Envoy fault injection passed: accepted-then-disconnected request delivered once; pre-work drained response retried.
  • Revised final stack: full chart validation and 20 render regressions passed.
  • Local kind upgrade, manual workload restarts, and all Helm hooks passed, including SQL writes on both PVC and ephemeral Engines and Engine TLS verification.

Stack navigation: #111 → #112 → #113 → #114 → #117 → #120. Parent: FB-4382. Merge in this order. Signing-key rotation documentation is in #120 (FB-4481).


Note

High Risk
Changes gateway retry semantics for SQL traffic; incorrect policy could duplicate mutations or leave clients without safe failover during drain.

Overview
Tightens Envoy gateway retries so a mutation is not automatically replayed after the engine has already accepted the request.

The chart’s retry_on policy drops generic reset in favor of reset-before-request, while keeping connect/refused-stream failures and X-Firebolt-Drained pre-work 503 retries. Inline comments in gateway-configmap.yaml spell out that post-delivery resets must not be retried.

Adds regression coverage: test_render.py asserts the rendered retry contract; test_gateway_runtime.py runs the chart’s Envoy image against a fault-injecting backend (one upstream call on accept-then-disconnect, retry on drained fence). CI’s validate-chart.sh runs the render tests; make test-render and make test-gateway-runtime expose them locally. helm/AGENTS.md documents the new hooks.

Reviewed by Cursor Bugbot for commit 33c5dfd. Bugbot is set up for automated code reviews on this repo. Configure here.

Restrict retries to pre-delivery failures and explicit drained responses. Document staged signing-key distribution and promotion to keep token verification compatible across rolling pods.
@fireboltdb-docs-integration

fireboltdb-docs-integration Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Mintlify docs preview

Docs were aggregated into the Firebolt documentation site.

Status Preview Updated (UTC)
🟢 Ready View Preview Oct 01, 2026, 3:13 PM

Mintlify may take a few minutes to finish building. This comment updates on each sync while the source PR is open.

@simonzelenskifbio
simonzelenskifbio marked this pull request as ready for review October 1, 2026 15:33
cursor[bot]
cursor Bot previously approved these changes Oct 1, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Risk: medium. Approved: Cursor Bugbot and Cursor Security Agent both completed successfully with no findings that need human review. The gateway retry tightening is a bounded behavior change with render and runtime test coverage.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

cursor[bot]
cursor Bot previously approved these changes Oct 2, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium. Approved: Cursor Bugbot completed successfully with no findings that need human review, and Cursor Security Agent was not running on this update. The gateway retry tightening remains a bounded behavior change with render and runtime test coverage.

Open in Web View Automation 

Sent by Cursor Approval Agent: Infra Reviewer

@simonzelenskifbio
simonzelenskifbio dismissed cursor[bot]’s stale review October 13, 2026 07:54

The merge-base changed after approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants