Support cross-account deployment via Iceberg + Lake Formation migration

[drernie]

Customer Request

From: Jon
Date: Saturday, November 1st

Problem

Cross-account deployment requirement: Customer wants to deploy API Gateway in an external-facing AWS account, separate from their data lake account where the webhook processing would run.

Current blocker: The webhook uses Athena for queries, but Athena is strictly single-account and cannot query Glue databases that exist in a different AWS account.

Customer's Proposed Solution

Use AWS Lake Formation with proxy resource links:

1. Register Quilt Glue tables via Lake Formation in the data lake account
2. Create Lake Formation proxy resource links in the webhook account
3. This allows Athena in the webhook account to query the shared Glue catalog

Architecture:

Account A (external-facing):
  Benchling → API Gateway → Fargate → Athena (with proxy links)
  
Account B (data lake):
  Lake Formation → Glue Catalog

Requirement: Quilt would need to register Glue tables via Lake Formation first.

Customer Quote

"looking at the webhook CDK and noticed that the new version also interacts directly with Athena - we were hoping to deploy the api gateway in a different account where we could keep externally facing services isolated. If my memory serves me correctly Athena is strictly single account meaning it cannot be used to query Glue databases in other accounts. The way around this is to use Lake Formation - you register its glue tables via Lake Formation first...It would require Quilt to register the glue tables via Lake Formation first."

References

• Related to secrets manager #156 (secrets configuration)

[drernie]

Our Proposed Solution: Iceberg + Lake Formation Migration

Rather than implementing Lake Formation proxy links for Athena views (customer's suggestion), we should migrate to Iceberg with Lake Formation (already planned).

Why this is better:

• Native cross-account data sharing via Lake Formation
• Better performance than Athena views
• ACID transactions and schema evolution
• Time travel and incremental reads
• Solves the cross-account problem as part of architectural improvement, not a workaround

Implementation:

1. Complete Iceberg migration (separate effort)
2. Configure Lake Formation for cross-account grants
3. Add CDK support for multi-account deployment
4. Document cross-account setup

Timeline: Depends on Iceberg migration schedule

Impact: Customer can use current workaround (same-account deployment) until Iceberg migration is complete.