Skip to content

Allow incremental ingestion without reprocessing entire history #35

@hotoke-no-Kami

Description

@hotoke-no-Kami

Overview

Enhance the ingestion workflow to support incremental data processing so that running ingestion does not reprocess the entire ledger history every time.

Acceptance Criteria

  • Tracks last processed ledger: System persists the last successfully processed ledger. Upon subsequent ingestion runs, only new ledgers are processed.
  • Idempotent execution: Running the ingestion multiple times without new data does not cause duplicate processing, errors, or side effects.
  • CLI command available: A command-line interface is available to trigger incremental ingestion, e.g., python -m astroml.ingestion.incremental (or similar, per project conventions).

Notes

  • Ensure the marker for the last processed ledger is robust and persists across restarts/crashes.
  • Handle initialization (if no previous state is found) by starting from the configured start ledger or from the earliest available in the database.
  • Update README with usage instructions for the new CLI.

Metadata

Metadata

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions