Add OrcaReplay: record/replay at the boundary you do not own - #12
Open
xizhuomengcontin wants to merge 1 commit into
Open
xizhuomengcontin wants to merge 1 commit into
xizhuomengcontin wants to merge 1 commit into
Conversation
Every other tool here owns the runtime to get determinism; OrcaReplay records and replays the wire traffic of an external model provider you cannot instrument, re-seed or reschedule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds OrcaReplay to the Software section.
The gap it fills: everything currently listed here makes execution deterministic by taking control of the runtime the system under test runs on — MadSim and Turmoil replace scheduling and networking in Rust, Tickloom does the same for Java services, Unthread swaps out POSIX threads, Antithesis goes all the way down to the hypervisor. All of them assume the nondeterminism lives inside a boundary you own.
The case this list currently has no answer for is the dependency you do not own. For anything LLM-backed, the dominant source of nondeterminism is a remote provider you cannot instrument, cannot re-seed and cannot reschedule; there is no runtime replacement to install because the nondeterminism is not on your side of the socket. So the most interesting system in the stack stays outside the deterministic simulation.
OrcaReplay applies record/replay at that seam instead. It sits as an HTTP proxy between the process and its provider and records the wire traffic byte-for-byte — requests, tool calls, responses, headers — into a local trace, then replays those exact bytes back offline so the run re-executes step by step with no provider contact and no model nondeterminism. In FoundationDB terms it is the boundary-level counterpart to replacing the runtime: not a simulator for concurrency within the program, but a way to make the one external input the program cannot control reproducible. Checkpoints let you re-enter mid-run,
comparediffs two executions, and a recording can be forked onto another model to see where behaviour diverges.It is a Node 20+ CLI (
npm i -g orcareplay, installed binaryorca), Apache-2.0, and has CI coverage against OpenAI Agents SDK, LangGraph, CrewAI, OpenHands, Aider, Vercel AI SDK / Mastra, goose, Agno, smolagents and others.Disclosure: I maintain OrcaReplay.