fix: EventBridge runtime filtering + remove pending canvas

AGENTS.md

@@ -6,156 +6,27 @@
 - Always fix IDE diagnostics after edits
 - Docker images ALWAYS pull from centralized ECR: `712023778557.dkr.ecr.us-east-1.amazonaws.com/quiltdata/benchling:latest`
 
-## Testing (Essential Guide)
+## Critical: Use Project npm Scripts
 
-**Full details:** [spec/a08-test-scenarios.md](spec/a08-test-scenarios.md)
+**Version bumps:** Use `npm run version -- minor|patch|major`, NOT `npm version`. The project script syncs `package.json`, `pyproject.toml`, `app-manifest.yaml`, and `uv.lock` in one commit.
 
-### Quick Reference: When to Run What
+**General rule:** Always check `npm run` before assuming a built-in npm command. The project script is always preferred.
 
-```bash
-# PRE-COMMIT (always run before committing)
-npm test                     # ~20s: TS + Python unit tests + lint + typecheck
-
-# BEFORE MERGING PR
-npm run test:integration     # ~2m: Full TypeScript integration tests
-npm run test:local           # ~45s: Local Docker dev build + webhook tests
-
-# AFTER MAKING DOCKER CHANGES (requires local build only)
-npm run test:local:prod      # ~60s: Test production Docker build locally
-
-# AFTER CI BUILDS IMAGE (requires git tag + CI completion)
-npm run test:dev             # ~5m: Deploy to dev + test (auto-deploys if needed)
-npm run test:prod            # ~10s: Health check prod deployment (non-invasive)
-cd docker && make test-ecr   # ~90s: Validate published ECR image
-```
-
-### Critical Dependencies
-
-**Docker Images:** All deployments pull from centralized ECR `712023778557.dkr.ecr.us-east-1.amazonaws.com/quiltdata/benchling:latest`
-
-**CI Build Required For:**
-
-- `npm run test:dev` - Requires Docker image in ECR
-- `npm run test:prod` - Requires Docker image in ECR
-- `make test-ecr` - Requires Docker image in ECR
-
-**CI Build Trigger:**
-
-```bash
-npm run version:tag:dev      # Creates git tag → triggers CI → builds/pushes Docker image
-```
-
-### Test Categories Summary
-
-| Test | Duration | CI Dep | When |
-|------|----------|--------|------|
-| `npm test` | ~20s | ❌ | Pre-commit (always) |
-| `npm run test:integration` | ~2m | ❌ | Before merging PR |
-| `npm run test:local` | ~45s | ❌ | Docker dev changes |
-| `npm run test:local:prod` | ~60s | ❌ | Docker prod changes |
-| `npm run test:dev` | ~5m | ✅ | After CI builds image |
-| `npm run test:prod` | ~10s | ✅ | Validate prod deployment |
-
-**Legend:** CI Dep = Requires CI to have built Docker image
-
-### Common Test Issues
-
-| Error | Solution |
-|-------|----------|
-| `No profile found` | `npm run setup` |
-| `Image not found in ECR` | `npm run version:tag:dev` (triggers CI build) |
-| `ECR authentication failed` | `aws sso login` |
-| `Timeout waiting for health` | Check logs: `docker logs {container}` |
-
-## Key Repository Commands
-
-### Development Workflow
+## Testing: When to Run What
 
 ```bash
-npm run setup                # Interactive config wizard (one-time)
-npm test                     # Fast pre-commit tests
-npm run test:local           # Test local Docker build
+npm test                     # PRE-COMMIT (always)
+npm run test:integration     # BEFORE MERGING PR
+npm run test:local           # AFTER DOCKER CHANGES
+npm run test:local:prod      # AFTER DOCKER PROD CHANGES
+npm run test:dev             # AFTER CI BUILDS IMAGE (needs git tag first)
 ```
 
-### Release Workflow (Maintainers)
-
-```bash
-npm run version:tag          # Create git tag → triggers CI → builds Docker image
-npm run deploy:prod          # Deploy to production
-```
-
-### Key Files
-
-- [lib/types/stack-config.ts](lib/types/stack-config.ts) - Minimal CDK stack configuration interface
-- [lib/utils/config-transform.ts](lib/utils/config-transform.ts) - ProfileConfig → StackConfig transformation
-- [lib/benchling-webhook-stack.ts](lib/benchling-webhook-stack.ts) - Main CDK stack
-- [bin/commands/deploy.ts](bin/commands/deploy.ts) - Deployment orchestration
-- [docker/](docker/) - FastAPI webhook processor (Python)
-
-## High-Level Architecture
-
-**Flow:** API Gateway (REST v1) → VPC Link → Network Load Balancer → ECS Fargate (FastAPI) → S3 + SQS
-
-**Configuration:** Profile-based XDG config in `~/.config/benchling-webhook/{profile}/config.json`
-
-**Key Concepts:**
-
-- **Profile** - Named config set (e.g., `default`, `sales`, `dev`)
-- **Stage** - API Gateway deployment target (e.g., `dev`, `prod`)
-- **StackConfig** - Minimal interface for CDK stack (v0.10.0+, decoupled from ProfileConfig)
-
-**Security:**
-
-- Primary: HMAC signature verification in FastAPI
-- Optional: Resource Policy IP filtering (free, when `webhookAllowList` configured)
-
-## Configuration v0.10.0+
-
-**Breaking Change:** Removed unused Iceberg fields (`quilt.athenaUserPolicy`, `quilt.athenaResultsBucketPolicy`, `quilt.athenaResultsBucket`)
-
-**New Architecture:**
-
-- CDK stack uses minimal `StackConfig` interface (only required fields)
-- `config-transform.ts` converts ProfileConfig → StackConfig
-- Eliminated subprocess env var round-trip
-- Deployment flow: deploy.ts passes config via stdin to CDK
-
-## Common Patterns
-
-### Creating a PR
-
-```bash
-npm run test                 # Ensure tests pass
-git commit -m "type(scope): description"
-gh pr create                 # Creates PR with conventional format
-```
-
-### Checking Logs
-
-```bash
-npm run logs -- --profile default
-# Checks both API Gateway and ECS container logs
-```
-
-### Multi-Stack Deployments
-
-```bash
-npm run deploy:prod -- --profile sales --yes
-# Creates: BenchlingWebhookStack-sales
-```
-
-## Troubleshooting
-
-| Issue                  | Solution                                                   |
-| ---------------------- | ---------------------------------------------------------- |
-| Missing profile        | Run `npm run setup`                                        |
-| 403 Forbidden (HMAC)   | Check ECS logs for signature errors                        |
-| 403 Forbidden (IP)     | Add IP to `webhookAllowList` or disable filtering          |
-| Stack name conflict    | Use unique `stackName` per profile in config.json          |
+CI build trigger: `npm run version:tag:dev`
 
-## Documentation
+## Gotchas
 
-- [CLAUDE.md](CLAUDE.md) - Comprehensive project documentation
-- [CHANGELOG.md](CHANGELOG.md) - Release notes and migration guides
-- [spec/](spec/) - Architecture specs and implementation details
-- [docker/README.md](docker/README.md) - FastAPI application documentation
+- `npm run test:dev`, `test:prod`, `make test-ecr` all require CI to have built the Docker image first
+- Integrated mode (`integratedStack: true`) blocks `deploy` — the Quilt stack handles deployment
+- The prefix (`pkg_prefix`) is a runtime secret — never bake it into CloudFormation/IaC
+- Pass args to npm scripts with `--`: `npm run deploy:prod -- --profile sales --yes`

CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+### Changed
+
+- EventBridge rule now matches only on `source` and `detail-type` — bucket and prefix filtering moved to the Docker runtime, where values are available from Secrets Manager. This unblocks integrated-mode stacks where the IaC layer cannot reference secret-derived values.
+- Canvas footer now shows "Pending update" (honest) instead of "Up to date" (a lie) when no confirmed update timestamp exists
+
+### Removed
+
+- `PackagePrefix` CloudFormation parameter (filtering now happens at runtime in the Docker handler)
+- Pending canvas state — the intermediate "Updating..." canvas with disabled buttons and stripped links was removed entirely; the canvas now stays unchanged until the EventBridge package event confirms the update, then renders the final state with a real timestamp
+
 ## [0.15.0] - 2026-04-10
 
 ### Changed

bin/commands/deploy.ts

@@ -761,7 +761,6 @@ export async function deploy(
             `BenchlingSecretARN=${benchlingSecret}`,
             `ImageTag=${options.imageTag}`,
             `PackageBucket=${config.packages.bucket}`,
-            `PackagePrefix=${config.packages.prefix || "benchling"}`,
             `QuiltDatabase=${config.quilt.database || ""}`,  // IAM permissions only (same value as AthenaUserDatabase)
             `LogLevel=${config.logging?.level || "INFO"}`,
         ];

docker/app-manifest.yaml

@@ -2,7 +2,7 @@ manifestVersion: 1
 info:
   name: nightly-quilttest-com
   description: Packaging Benchling Notebooks as Quilt packages
-  version: 0.15.0
+  version: 0.16.0
 features:
   - name: Quilt Connector
     id: quilt_entry

docker/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "benchling-quilt-integration"
-version = "0.15.0"
+version = "0.16.0"
 description = "Benchling-Quilt Integration Webhook Service"
 license = {text = "Apache-2.0"}
 authors = [

docker/src/app.py

@@ -697,21 +697,9 @@ async def _handle_event_impl(request: Request, _verified: None = None):
                     event_type=payload.event_type,
                     canvas_id=payload.canvas_id,
                 )
-                canvas_manager = CanvasManager(benchling, config, payload)
-
                 logger.debug("Starting background export workflow from /event", entry_id=payload.entry_id)
                 entry_packager.execute_workflow_async(payload)
 
-                def async_pending_canvas_from_event():
-                    try:
-                        canvas_manager.update_canvas_pending()
-                    except Exception as e:
-                        logger.error(
-                            "Failed to set pending canvas from /event", canvas_id=payload.canvas_id, error=str(e)
-                        )
-
-                threading.Thread(target=async_pending_canvas_from_event, daemon=True).start()
-
                 logger.info("Canvas update initiated from /event endpoint", canvas_id=payload.canvas_id)
                 return JSONResponse(
                     {"status": "ACCEPTED", "message": "Canvas update initiated"},
@@ -850,6 +838,15 @@ async def _handle_package_event_impl(request: Request):
                 )
                 return JSONResponse({"error": "Forbidden"}, status_code=403)
 
+            expected_prefix = f"{config.pkg_prefix}/"
+            if not package_name.startswith(expected_prefix):
+                logger.info(
+                    "Ignored package event outside prefix",
+                    package_name=package_name,
+                    expected_prefix=expected_prefix,
+                )
+                return JSONResponse({"status": "IGNORED"}, status_code=200)
+
             top_hash = detail.get("topHash")
             if top_hash is not None and not isinstance(top_hash, str):
                 raise ValueError("package event detail.topHash must be a string when present")
@@ -1004,21 +1001,8 @@ async def _handle_canvas_impl(request: Request, _verified: None = None):
             logger.debug("Starting background export workflow", entry_id=payload.entry_id)
             execution_arn = entry_packager.execute_workflow_async(payload)
 
-            canvas_manager = CanvasManager(benchling, config, payload)
-
-            # Push pending canvas via SDK in background thread.
-            # Benchling ignores blocks in the webhook response body —
-            # canvas content must be set via the update_canvas API.
-            def async_pending_canvas():
-                try:
-                    canvas_manager.update_canvas_pending()
-                except Exception as e:
-                    logger.error("Failed to set pending canvas", canvas_id=payload.canvas_id, error=str(e))
-
-            threading.Thread(target=async_pending_canvas, daemon=True).start()
-
             logger.info(
-                "Canvas update initiated (pending state)",
+                "Canvas update initiated",
                 canvas_id=payload.canvas_id,
                 entry_id=payload.entry_id,
                 event_type=payload.event_type,

docker/src/canvas.py

@@ -183,17 +183,13 @@ def upload_url(self) -> str:
         """
         return self.package.upload_url
 
-    def _make_markdown_content(self, pending: bool = False) -> str:
+    def _make_markdown_content(self) -> str:
         """Generate markdown content for the Canvas.
 
         Composes the full canvas content in the following order:
         1. Primary package information
-        2. Notice and status
-        3. Linked packages (if any)
-        4. Error notifications (if any)
-
-        Args:
-            pending: If True, render links as plain text (package not yet created)
+        2. Linked packages (if any)
+        3. Error notifications (if any)
 
         Returns:
             Formatted markdown string with package links
@@ -204,7 +200,6 @@ def _make_markdown_content(self, pending: bool = False) -> str:
             display_id=self.entry.display_id,
             catalog_url=self.catalog_url,
             sync_url=self.sync_uri(),
-            pending=pending,
         )
 
         # Linked packages
@@ -220,7 +215,7 @@ def _make_markdown_content(self, pending: bool = False) -> str:
             # Store linked packages as instance variable
             self._linked_packages = linked_packages
 
-            content += fmt.format_linked_packages(linked_packages, pending=pending)
+            content += fmt.format_linked_packages(linked_packages)
 
         except Exception as e:
             error_msg = f"Failed to search for linked packages: {str(e)}"
@@ -240,46 +235,35 @@ def _make_markdown_content(self, pending: bool = False) -> str:
 
         return content
 
-    def _make_blocks(self, pending: bool = False, updated_at: str | None = None) -> list:
+    def _make_blocks(self, updated_at: str | None = None) -> list:
         """Create UI blocks for the Canvas.
 
         Args:
-            pending: If True, disable buttons and links, show "Updating..." footer
-            updated_at: ISO timestamp of last package update (shown when not pending)
+            updated_at: ISO timestamp of last package update
         """
-        markdown_content = self._make_markdown_content(pending=pending)
+        markdown_content = self._make_markdown_content()
         markdown_block = blocks.create_markdown_block(markdown_content, "md1")
 
         result = [
-            *blocks.create_main_navigation_buttons(self.entry_id, enabled=not pending),
+            *blocks.create_main_navigation_buttons(self.entry_id),
             markdown_block,
         ]
 
-        # Add linked package browse buttons if any exist (disabled when pending)
-        if self._linked_packages and not pending:
+        # Add linked package browse buttons if any exist
+        if self._linked_packages:
             result.extend(blocks.create_linked_package_browse_buttons(self.entry_id, self._linked_packages))
 
         # Add footer as markdown block
         footer_markdown = fmt.format_canvas_footer(
             version=__version__,
             quilt_host=self.config.quilt_catalog,
             bucket=self.config.s3_bucket_name,
-            pending=pending,
             updated_at=updated_at,
         )
         result.append(blocks.create_markdown_block(footer_markdown, "md-footer"))
 
         return result
 
-    def update_canvas_pending(self) -> dict[str, Any]:
-        """Push pending canvas with full content but disabled links/buttons.
-
-        Shows the same information as the final canvas so the user can see
-        where the package will be. Only the status line and button state differ.
-        """
-        blocks = self._make_blocks(pending=True)
-        return self.update_canvas_with_blocks(blocks)
-
     def update_canvas(self, updated_at: str | None = None) -> dict[str, Any]:
         """Update existing Canvas using Benchling SDK.
 

docker/src/canvas_blocks.py

@@ -90,12 +90,11 @@ def create_section(section_id: str, buttons: List[ButtonUiBlock]) -> SectionUiBl
     )
 
 
-def create_main_navigation_buttons(entry_id: str, enabled: bool = True) -> List:
+def create_main_navigation_buttons(entry_id: str) -> List:
     """Create main view navigation buttons (Browse Package, Update Package).
 
     Args:
         entry_id: Entry identifier for button IDs
-        enabled: Whether buttons are interactive (False during pending state)
 
     Returns:
         List containing section with navigation buttons
@@ -104,12 +103,10 @@ def create_main_navigation_buttons(entry_id: str, enabled: bool = True) -> List:
         create_button(
             button_id=f"browse-files-{entry_id}-p0-s15",
             text="Browse Package",
-            enabled=enabled,
         ),
         create_button(
             button_id=f"update-package-{entry_id}",
             text="Update Package",
-            enabled=enabled,
         ),
     ]