|
| 1 | +--- |
| 2 | +paths: |
| 3 | + - "apps/sim/lib/**/*.ts" |
| 4 | + - "apps/sim/providers/**/*.ts" |
| 5 | + - "apps/sim/executor/**/*.ts" |
| 6 | + - "apps/sim/tools/**/*.ts" |
| 7 | +--- |
| 8 | + |
| 9 | +# In-Process Caching |
| 10 | + |
| 11 | +**Never hand-roll TTL arithmetic.** `lru-cache` is a direct dependency of `apps/sim` and owns |
| 12 | +expiry, the ceiling, and — through `fetchMethod` — request coalescing. A |
| 13 | +`Map` plus `Date.now() - entry.fetchedAt < TTL` re-implements all three, badly. |
| 14 | + |
| 15 | +## First decide whether it is a cache at all |
| 16 | + |
| 17 | +Most module-level `Map`s in this codebase are **not** caches, and forcing them into one is worse |
| 18 | +than leaving them alone. |
| 19 | + |
| 20 | +| Shape | Key dies when | Right tool | |
| 21 | +| --- | --- | --- | |
| 22 | +| **Lifecycle map** — `activeStreams`, `pendingChildRuns`, `memoryStreams`, `handlerRegistry` | the tracked thing ends, and the code deletes it there | plain `Map`. No TTL, no ceiling. | |
| 23 | +| **TTL cache** — a remote read keyed by tenant (org id, user id, workspace id) | time passes | `LRUCache` | |
| 24 | + |
| 25 | +A lifecycle map's key space is unbounded and that is fine, because every key has a defined death. |
| 26 | +Adding a TTL to one introduces an expiry that races the lifecycle. Adding a ceiling silently drops |
| 27 | +live state. |
| 28 | + |
| 29 | +## TTL caches: always set `max` |
| 30 | + |
| 31 | +```ts |
| 32 | +const policyCache = new LRUCache<string, ResolvedSessionPolicy>({ |
| 33 | + max: 20_000, |
| 34 | + ttl: SESSION_POLICY_CACHE_TTL_MS, |
| 35 | +}) |
| 36 | +``` |
| 37 | + |
| 38 | +`ttl` alone does **not** bound memory. Without `ttlAutopurge` (itself expensive — one timer per |
| 39 | +entry) an expired entry lingers until something touches its key or the ceiling evicts it. `max` is |
| 40 | +what actually caps the process, which is why a tenant-keyed `Map` grew for the life of the process |
| 41 | +before this rule existed. |
| 42 | + |
| 43 | +**The ceiling is a memory backstop, not an operating limit.** Exceeding it makes the LRU evict |
| 44 | +*inside* the TTL, so each miss becomes one more read — never a wrong answer, it degrades to exactly |
| 45 | +the pre-cache behavior, but it is a hit-rate cliff on whatever path the cache sits on. Entries are |
| 46 | +tens of bytes, so set the cap far above any plausible per-instance working set within the TTL |
| 47 | +window and let it stay a backstop. |
| 48 | + |
| 49 | +**Reads test `!== undefined`, not truthiness**, whenever the value can be `false`, `0`, or `null`. |
| 50 | +`if (cached)` on a cached `false` re-queries on every single call, for exactly the tenants the |
| 51 | +cache exists to protect. |
| 52 | + |
| 53 | +## Async read-through: prefer `fetchMethod` |
| 54 | + |
| 55 | +`fetchMethod` + `cache.fetch(key)` gives TTL, coalescing (concurrent callers share one promise), |
| 56 | +and eviction-on-rejection (`noDeleteOnFetchRejection` defaults to `false`) in one primitive. Reach |
| 57 | +for it before composing anything yourself. |
| 58 | + |
| 59 | +**The one reason to compose instead: a hung producer.** `fetchMethod` has no settle deadline, and |
| 60 | +the app pool sets no `statement_timeout` (`packages/db/db.ts` sets only `connect_timeout` / |
| 61 | +`idle_timeout`, neither of which bounds a query already in flight). Where a wedged read would hold |
| 62 | +every caller for the whole TTL, wrap `coalesceLocally` from `@/lib/concurrency/singleflight` around |
| 63 | +a read-through `LRUCache` instead — it evicts and rejects at its deadline. See |
| 64 | +`lib/api-key/byok-entitlement.ts`, and `lib/oauth/credential-service.ts` for the same shape. |
| 65 | + |
| 66 | +Do **not** build a house wrapper over `lru-cache`. Call sites differ in ways a thin helper cannot |
| 67 | +hold (synchronous memoization with `updateAgeOnGet` in `providers/client-cache.ts`, per-entry TTLs |
| 68 | +in `lib/auth/security-policy.ts`), so a wrapper covering the common case just adds a fourth pattern. |
| 69 | + |
| 70 | +## Cache the gate, never the credential |
| 71 | + |
| 72 | +Entitlements, plans, and policies tolerate bounded staleness **in the safe direction** — a lapsed |
| 73 | +organization keeping its own provider key for another minute costs a little metering and charges |
| 74 | +nobody wrongly. Key material does not: revocation has to be immediate, so |
| 75 | +`getBYOKKey` reads key rows fresh on every call and caches only the entitlement around them. |
| 76 | + |
| 77 | +An outage must not be cached as a negative answer. A resolver that maps a failed read to `false` |
| 78 | +makes an outage indistinguishable from a real lapse, so give it an `onError: 'throw'` option and |
| 79 | +write the cache only on the success path — see `resolveOrganizationPlan`. |
| 80 | + |
| 81 | +**Where a human is waiting, read fresh.** Keep two entry points rather than one cached function: |
| 82 | +the settings surfaces and management use cases must not tell an organization that just upgraded |
| 83 | +that it still lacks a plan, while the execution path underneath can serve from cache |
| 84 | +(`isOrganizationBYOKEntitled` vs `isOrganizationBYOKEntitledCached`). |
| 85 | + |
| 86 | +## React `cache()` does nothing in a worker |
| 87 | + |
| 88 | +`cache()` is request-scoped. Workflows run in Trigger.dev workers, which have no React request |
| 89 | +scope, so a `cache()`-wrapped gate that looks free on a settings page is uncached and per-block on |
| 90 | +the execution path. Anything reached from the executor needs a real cache — see |
| 91 | +`.claude/rules/sim-architecture.md`'s app/worker runtime boundary. |
| 92 | + |
| 93 | +## Invalidation |
| 94 | + |
| 95 | +Add a per-key invalidator only when the code that mutates the value runs in the **same process** |
| 96 | +that reads it. `invalidateSessionPolicyCache` works because the route writing the policy is the one |
| 97 | +serving the reads. An entitlement change arriving on a Stripe webhook lands in one process while |
| 98 | +the readers are per-worker, so an invalidator there would imply an immediacy it cannot deliver — |
| 99 | +the TTL is the real mechanism, and the absence of an invalidator should say so. |
0 commit comments