Metadata rollback "no race" & File IO "watchdog"

[grecinto]

Background

• Blob Store: resilient and self-healing via regular transaction logs; no special handling needed.
• Registry (metadata store): needs stronger guarantees to meet business resiliency requirements.

Known failure modes

• Redis outage
  • All in-flight transactions roll back using both priority and regular logs.
  • Verified behavior; not a concern.
• App crash during registry sector write
  • A sector may be partially written, but a usable backup exists in the priority log.
  • SOP must detect stale priority logs and restore affected sectors before new transactions can overwrite them.
• Slow or degraded storage (network/HDD/SSD)
  • If sector lock blocks past the internal timeout (~30s), a race may occur:
    • A newer transaction restores+writes first.
    • The older one later “thinks” it succeeded and completes, corrupting the sector.
  • Newly created but stale priority logs (app/host killed) can also be overwritten by incoming transactions before they’re restored.

Current watchdog

• Background priority rollback processing.
• Registry update path that, on lock timeout, detects a stale sector and attempts a restore.
• Weakness: slow storage can extend beyond timeout, enabling the race above.

Goal

Eliminate race windows so there’s effectively 0% chance of sector corruption.

Proposed solution

1. Harden registry sector updates
   • Increase the sector lock timeout substantially (e.g., to ~3 minutes).
   • If it still times out:
     • Fail the current transaction.
     • Roll it back and emit a failover event.
   • Rationale: prefer fail-fast over risky takeover under severe slowness.
2. Proactive handling of new-but-stale priority logs
   • Detection:
     • On startup (and periodically), scan priority logs and build an index of affected logical IDs.
     • Keep the index in memory and refresh incrementally to avoid hot-path overhead.
   • Freeze gate:
     • Before transactions read/write affected sectors, consult the index.
     • If a sector is covered by a stale priority log, temporarily “freeze” its logical IDs via an L2 cache flag (similar to the active/passive toggler).
   • While frozen:
     • Restore from the priority log.
     • Unfreeze once the sector is consistent.
   • Cache/version sync:
     • Ensure L1/L2 versions are updated coherently post-restore so subsequent transactions see consistent state.

Notes

• Make freezing surgical (per logical ID/sector group), short‑lived (TTL), and restores idempotent.
• Keep background scanning bounded/incremental to avoid performance impact.
• Hard timeouts + failover protect correctness when storage is too slow for safe takeover.

[grecinto]

This had been fixed in racefix2 branch, being stabilized & cleaned. Watchdog is out to stay simple, lightweight, performant & predictable/better overall.