Copycat L3 indexer with parallel processing and parent lookups

[nikooo777]

Summary

• Add L1 TX filtering with owner/tag support, offset loading, and block depth indexing (Rani)
• Add tests, refactor internals, improve logging, and fix operational issues (James)
• Add per-block item index with depth tracking and inventory mode
• Add parallel block processing with shared memory budget (configurable workers + byte-level throttling)
• Add parent containment index: track which block or bundle contains each item
• Add ~arweave@2.9/parent=<id> endpoint for parent lookups

How to use

Index blocks

Index a range of blocks at depth 3 (L1 TXs → L2 bundle items → L3 nested items):

curl "http://localhost:8005/~copycat@1.0/arweave?from=1890000&to=1889000&depth=3"

For long-running indexing, use the cron wrapper to avoid HTTP timeout killing the job:

curl "http://localhost:8005/~cron@1.0/once?cron-path=~copycat@1.0/arweave&from=-1&to=1862995&depth=3"

Query the inventory

See what was indexed per block, grouped by depth level:

curl "http://localhost:8005/~copycat@1.0/arweave?from=1890000&to=1889990&mode=inventory"

Example response:

{
  "1890000": {
    "depth": 3,
    "items": {
      "1": ["txid1", "txid2"],
      "2": ["bundleitem1", "bundleitem2"],
      "3": ["nesteditem1"]
    }
  }
}

Look up an item's parent

Find which block or bundle contains a given item:

curl "http://localhost:8005/~arweave@2.9/parent=CwxY--7bsqjtw2lneMUjkmYT9CWAYZvsmsO06dY232g"

Response:

{"parents": [{"type": "bundle", "id": "Rve4-grgOw8jXLVw3f6nUhQvlVn6AYm7wA4I5HeKW74"}]}

What gets indexed

The copycat indexer writes these entries to the index store:

• Offset index (<item-id> → codec + offset + length): maps each item to its location in the Arweave weave
• Block marker (block/<height>/depth → integer): records that a block was indexed and to what depth
• Block item index (block/<height>/items/<depth> → item IDs): lists items found at each depth level
• Parent index (parent/<item-id> → type + parent ref): maps each item to its containing block (height) or bundle (ID)

Configuration

Key	Default	Description
arweave_block_workers	3	Max concurrent blocks being processed
arweave_index_workers	1	Max concurrent TXs within a block
copycat_memory_budget	6 GB	Global memory pool for concurrent downloads
copycat_memory_cap	6 GB	Per-TX hard ceiling (skip if larger)

[speeddragon]

Review the PR, I will fix this later.

[speeddragon]

If we're indexing most up to date block, we should keep indexing and ignore this is_block_indexed.

[speeddragon]

Add force to force requests to be re-indexed in some edge cases.

[speeddragon]

Closed in favor of #903, which contain this and other work merged in a new branch.