Skip to content

feat: migrate static data to SQLite and persist runtime timers/events across restarts#18

Merged
FaiThiX merged 6 commits intomasterfrom
feat/json-into-db
Mar 11, 2026
Merged

feat: migrate static data to SQLite and persist runtime timers/events across restarts#18
FaiThiX merged 6 commits intomasterfrom
feat/json-into-db

Conversation

@FaiThiX
Copy link
Owner

@FaiThiX FaiThiX commented Mar 11, 2026

Summary

This PR replaces large runtime-loaded JSON static files with SQLite and adds persistent runtime state for timers/event timestamps so data survives bot restarts.

Motivation

Large JSON files in src/staticFiles/ caused high memory usage because they were loaded into heap at runtime.
Additionally, timer/event state (e.g. DeepSea/Cargo/custom timers) was lost or reset incorrectly after restart.

Changes

  • Migrated static data storage from JSON files to a single SQLite database (staticData.sqlite).
  • Added/updated a manual build script to populate the DB from source JSON files (scripts/buildStaticFilesDb.js).
  • Moved raw static source files into a separate gitignored directory (no longer shipped in repo by default).
  • Added runtime persistence storage (runtimeData.sqlite) for server-scoped runtime state.
  • Persisted/restored:
  • custom in-game timers (timer add)
  • DeepSea timestamps/state
  • Cargo egress timers/state
  • Oil rig unlock timers
  • additional map-marker related event timestamps
  • Fixed restore timing by restoring persistent runtime state during first polling initialization.
  • Fixed DeepSea restart reset edge case by deriving active state from current marker set instead of fragile new/left marker transitions.
  • Improved Cargo restore robustness by restoring via ID and position fallback, including onItsWayOut state.

Result

  • Significant memory reduction after static JSON -> SQLite migration.
  • Runtime timers/events now survive bot restart and continue with correct remaining time.
  • Removed “temporary wrong initial time” after reconnect/first update for DeepSea/Cargo.

Breaking / Operational Notes

  • Raw static JSON files are no longer expected to be versioned/shipped in normal runtime.
  • Provide staticData.sqlite in deployment, or run the build script manually to generate it.
  • Existing users should verify DB paths and deployment includes the SQLite files.

Validation

  • Typecheck/tests passed (npm test -- --pretty false).
  • Manual restart tests for DeepSea, Cargo, OilRig unlock timers, and custom timers confirm persistence behavior.

@FaiThiX FaiThiX merged commit c2f4fe9 into master Mar 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant