Skip to content

Parallelize per-game feature extraction in IndexWorker#1141

Merged
aaylward merged 16 commits intomainfrom
parallel
Apr 8, 2026
Merged

Parallelize per-game feature extraction in IndexWorker#1141
aaylward merged 16 commits intomainfrom
parallel

Conversation

@aaylward
Copy link
Copy Markdown
Collaborator

@aaylward aaylward commented Apr 8, 2026

Summary

  • Submit each game's feature extraction to an injected fixed-size ExecutorService (default 4 threads, env INDEXER_EXTRACTION_THREADS) instead of processing games sequentially within a month
  • DB writes remain single-threaded and batched; failure isolation, status updates, and bullet filtering are preserved
  • MotifDetectors are stateless by contract — no per-task state required

Test plan

  • bazel test //domains/games/apis/one_d4/...
  • New IndexWorkerTest cases: concurrent extraction across pool threads, failing-extraction isolation, batch contains all games regardless of order

@cloudflare-workers-and-pages
Copy link
Copy Markdown

cloudflare-workers-and-pages bot commented Apr 8, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
1d4-web cc297e1 Commit Preview URL

Branch Preview URL
Apr 08 2026, 04:16 AM

@aaylward aaylward enabled auto-merge (squash) April 8, 2026 03:33
@aaylward aaylward disabled auto-merge April 8, 2026 03:34
@aaylward aaylward enabled auto-merge (squash) April 8, 2026 03:46
@aaylward aaylward merged commit 4f095bb into main Apr 8, 2026
12 checks passed
@aaylward aaylward deleted the parallel branch April 8, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant