SEO: Crawler-only reverse proxy for state legislative tracker

[PavelMakarchuk]

Summary

Closes #702
Companion to PolicyEngine/state-legislative-tracker#107 (already merged and deployed to Modal).

Search engines and social media crawlers hitting /us/state-legislative-tracker/* now receive pre-rendered HTML directly from Modal with:

• Canonical URLs on policyengine.org (not modal.run)
• Per-bill OG tags (unique title, description per state/bill)
• JSON-LD structured data
• Noscript content with bill provisions and revenue impacts

Regular users see no change — they still get the iframe version with the full policyengine.org nav and footer.

Changes

vercel.json — Add /_tracker/:path* rewrite to proxy tracker assets from Modal. These are referenced by the pre-rendered HTML that crawlers receive. Placed before the /(.*) → website.html catch-all so it takes priority.

app/middleware.ts — Add early interception for /us/state-legislative-tracker/* routes:

• Social crawlers (Facebook, Twitter, LinkedIn, etc.) and search engines (Googlebot, Bingbot, etc.) → fetch pre-rendered HTML from Modal and return it
• Everyone else → fall through to catch-all rewrite → website.html → iframe (unchanged)

Search engine bots are detected separately from social crawlers to avoid affecting OG tag behavior on other routes.

Architecture

Crawlers:  policyengine.org/us/state-legislative-tracker/GA
           → middleware detects bot → fetches modal.run/GA → returns HTML

Users:     policyengine.org/us/state-legislative-tracker/GA
           → middleware skips → rewrite → website.html → iframe (unchanged)

Assets:    policyengine.org/_tracker/index-*.js
           → Vercel rewrite → modal.run/_tracker/index-*.js

Test plan

• curl -A "Googlebot" https://policyengine.org/us/state-legislative-tracker/GA → returns tracker HTML with <link rel="canonical" href="https://www.policyengine.org/us/state-legislative-tracker/GA">
• curl -A "facebookexternalhit" https://policyengine.org/us/state-legislative-tracker/GA → returns tracker HTML with OG tags
• curl -A "Mozilla/5.0" https://policyengine.org/us/state-legislative-tracker → returns website.html (iframe version)
• Browser visit to /us/state-legislative-tracker → iframe with PE nav/footer (unchanged)
• /_tracker/ assets load via Vercel rewrite (check network tab when crawled page renders)

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
policyengine-app-v2	Ready	Preview, Comment	Feb 18, 2026 8:22pm
policyengine-calculator	Ready	Preview, Comment	Feb 18, 2026 8:22pm

[anth-volk]

Will review tomorrow

[anth-volk]

@PavelMakarchuk currently reviewing. Having some challenges running locally, but will keep you updated.

[anth-volk]

These changes don't currently work on the preview branch. Did some cursory research and it's unclear to me if SPAs like Vite without server-side rendering can even use Vercel edge functions. At the very least, I believe you need a middleware.ts file at project root. Happy to chat about how to proceed on this if you hit a wall; the learning may be that we want some SSR platform like Next.js and/or partial pre-hydration of some sort from React (but I've never worked with this before).

[PavelMakarchuk]

Good catch — you were right that it wasn't working. The root cause is that middleware.ts was inside app/, but our Vercel project has Root Directory set to . (repo root). Vercel Edge Middleware needs to be at the project root (same level as package.json), so app/middleware.ts was never being picked up.

This also means the existing OG tag middleware on main has never been running in production — confirmed by testing with curl -A "facebookexternalhit" against a blog post URL and getting generic OG tags instead of post-specific ones.

I've pushed a fix (1eb72c8) that moves middleware.ts to the repo root and updates all import paths. This should fix both the new tracker proxy and the existing OG tags. Tests all pass.

Note: the preview still can't be tested with curl due to Vercel's deployment protection (SSO auth wall intercepts before middleware runs), but once merged to production we can verify with:

curl -s -A "facebookexternalhit" https://policyengine.org/us/research/some-post | head -5
curl -s -A "Googlebot" https://policyengine.org/us/state-legislative-tracker/GA | head -5

Re: your Vite question — Edge Middleware is a Vercel platform feature, not framework-specific. It works with any project type (Vite, Next.js, static, etc.). The file just needs to be at the right location.

[anth-volk]

Turns out that you can verify this works on a test deployment by removing deployment protection temporarily. Did so and confirmed that it works on the preview deploy link. Thanks for this, @PavelMakarchuk!