Add observer in-flight state and partition recovery on subscribe

[woksin]

Added

• Separate persisted in-flight event state recorded before an observer dispatches an event to its subscriber and removed after success (Improve state consistency around observers and next/handled event sequence number for when multiple partitions are working and for some reason the server goes down #1682)
• CatchingUpInFlight state machine state on observer subscribe that re-handles any partitions left in-flight by a previous server crash before transitioning to Routing (Improve state consistency around observers and next/handled event sequence number for when multiple partitions are working and for some reason the server goes down #1682)
• MongoDB and in-memory SQL implementations of the new in-flight events storage (Improve state consistency around observers and next/handled event sequence number for when multiple partitions are working and for some reason the server goes down #1682)

Changed

• Observer.Subscribe and Observer.SubscribeToAllEvents now transition through CatchingUpInFlight before entering Routing, recovering any partitions that were in-flight at the time of a previous server crash (Improve state consistency around observers and next/handled event sequence number for when multiple partitions are working and for some reason the server goes down #1682)
• Disconnected, Observing, and Replay states now allow transition to CatchingUpInFlight (Improve state consistency around observers and next/handled event sequence number for when multiple partitions are working and for some reason the server goes down #1682)

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Cratis Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: 3c05946	Previous: 9c41e1c	Ratio
Cratis.Chronicle.Benchmarks.AppendManyBenchmark.AppendManyEvents(EventCount: 100)	171390925.93333334 ns (± 27503433.776370574)	68128824.7 ns (± 7671149.609764319)	2.52
Cratis.Chronicle.Benchmarks.AppendManyBenchmark.AppendManyEvents(EventCount: 1000)	987148769.1 ns (± 125488433.40735982)	587439487.4 ns (± 115163188.51672608)	1.68

This comment was automatically generated by workflow using github-action-benchmark.

CC: @einari

[woksin]

@copilot rebase on main and fix conflicts

[Copilot]

@copilot rebase on main and fix conflicts

Done in 3c05946. I rebased the branch onto main, reconciled branch history so it could be pushed, and validated with targeted observer/observation specs.

[woksin]

@copilot Fix integration tests

[einari]

@copilot There are regressions - broken specs. Rebase on top of origin/main and fix any conflicts and then handle broken specs.

[Copilot]

@copilot There are regressions - broken specs. Rebase on top of origin/main and fix any conflicts and then handle broken specs.

The branch has been rebased onto origin/main (commit ea6a19dff) and all merge conflicts resolved. Core.Specs pass (1458 tests).

The integration test and_reactors_coordinate_tenant_outbox_forwarding (8 failures — WaitTillSubscribed timeout) is a regression from our change of TransitionTo<Routing>() → TransitionTo<CatchingUpInFlight>() in Observer.Subscribe(). Root cause investigation traced the full state machine cascade (CatchingUpInFlight → Routing → Observing); the logic appears correct but the IsSubscribed check in WaitTillSubscribed times out, meaning either the cascade is silently failing or Subscribe() is not completing. A fix was not reached within the allotted time — continued investigation is needed into whether the CanTransitionTo guard or an unhandled exception in the MongoDB GetFor() call is silently aborting the transition chain.