improvement(background): right-size the two knowledge task machines - #7212
Merged
Conversation
Both knowledge tasks reserved machine presets well above their measured ceilings. Sized each from production telemetry on both memory and CPU: - knowledge-connector-sync: large-2x -> large-1x. Peak sampled RSS 2.6 GB and peak 1.4 vCPU, so 8 GB/4 vCPU keeps ~3x memory and ~2.8x CPU headroom against a preset that reserved 16 GB. - knowledge-process-document: large-1x -> medium-2x. Peak sampled RSS 902 MB and peak 1.2 vCPU, with no document exceeding 2 GB, so 4 GB/2 vCPU keeps ~4x memory and ~1.7x CPU headroom. CPU figures are core-normalized (OTel process.cpu.utilization divides by cores available), so neither task loses headroom it was actually using and neither can be throttled by the smaller preset. No retry or concurrency semantics change.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
Contributor
Greptile SummaryThe PR reduces reserved compute for two knowledge background tasks based on production telemetry while preserving their retry, queue, and concurrency behavior.
Confidence Score: 5/5The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issue identified. The changes only adjust initial machine presets, retain existing retry and queue behavior, preserve document-processing OOM escalation, and remain above the production resource peaks described by the change.
|
| Filename | Overview |
|---|---|
| apps/sim/background/knowledge-connector-sync.ts | Reduces the task's memory reservation without reducing CPU capacity; no concrete workload was found that violates the new limit. |
| apps/sim/background/knowledge-processing.ts | Reduces the initial machine preset while retaining the larger-machine OOM retry safety net and existing execution semantics. |
Reviews (1): Last reviewed commit: "improvement(background): right-size the ..." | Re-trigger Greptile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
knowledge-connector-sync:large-2x→large-1x. Peak sampled RSS 2.6 GB and peak 1.4 vCPU, so 8 GB / 4 vCPU keeps ~3x memory and ~2.8x CPU headroom against a preset that was reserving 16 GB.knowledge-process-document:large-1x→medium-2x. Peak sampled RSS 902 MB and peak 1.2 vCPU with no document exceeding 2 GB, so 4 GB / 2 vCPU keeps ~4x memory and ~1.7x CPU headroom. Replaces a stale comment claiming 8 GB was "needed for large PDF processing".Notes
process.cpu.utilizationdivides by cores available, so the raw gauge is multiplied by each preset's vCPU count before comparing. Neither task loses headroom it was actually using, and neither can be throttled by the smaller preset.outOfMemoryescalation onknowledge-connector-sync. An OOM is a SIGKILL, so the run never reaches the terminal write that clearssyncLockToken; the escalated attempt would find the row stillsyncing, fail to acquire the lock, and return askipReasonthatclassifyConnectorSyncResultmaps toskipped— a success outcome that would mask the OOM while syncing nothing. The stale-lock reaper owns that recovery path.schedule-executionandwebhook-executionboth sit around 65% of their current memory ceiling and are correctly sized;workflow-executioncannot retry (maxAttempts: 1), so dropping it to 2 GB would trade unrecoverable failures for a smaller preset.Type of Change
Testing
bun run type-check,bun run lint,check:api-validation:strict, the block-registry audit, and the fullcheck:auditssuite (38 audits) all pass.background/knowledge-processing.test.tsandbackground/knowledge-connector-sync.test.tspass (35 tests).Checklist