Blackbox OS is a research prototype for reliable LLM tool use in quant/data-science settings: stage-local tool partitions, expert process templates (SOPs), sandboxed code execution, and light validation guardrails. Full multi-agent βdeskβ composition remains future work.
-
Monolithic Routing Collapse: Injecting all available tool schemas into a single LLM prompt suffers from non-linear accuracy collapse beyond
$N \sim 60$ tools due to semantic conflation. - Geometric packing: Mean max nearest-neighbor cosine similarity (MiniLM) rises with catalog size N (Bare peak β0.51 at N=500 in our runs). Expert descriptions remain more separable than short bare text. We do not treat a fixed 0.50β0.56 band as a proven universal phase boundary.
- Stage-local tool exposure (K β€ 15 per node) is a practical design rule that kept multi-step E2E high in the live Data Scientist stress test; it is not a formal guarantee of βzero saturation.β
Even when correctly routed, LLMs fail to execute tools reliably under context pressure due to two distinct failure modes:
-
Schema Collapse: Bare prompts fail to adhere to JSON structures in up to
$73%$ of trials, omitting required keys or returning malformed outputs. - Arithmetic Collapse: When schemas are enforced (via structured outputs), LLMs frequently fail at floating-point calculations and statistical aggregation (mental math hallucinations).
- Mitigation: Enforcing expert Process Templates (SOPs) largely eliminates schema failures on atomic tasks, while delegating calculations to a Python Sandbox bypasses the mental math bottleneck, lifting atomic E2E success substantially.
- Noise gradient (L0βL5): Mild noise is often tolerated; adversarial injection (L5) collapses E2E for several models. An AST script-integrity guardrail recovers a substantial but incomplete fraction of L5 failures (e.g. ~0% β ~47% on DeepSeek V4 Flash and GPT-4o-mini)βnot 100%.
- Query variation matrix (2Γ2): Strong models stay high on many cells; combined novelty + ambiguity is model-dependent, not a single universal 70% collapse.
Rather than outputting direct text predictions, agents generate self-contained Python scripts executed in an isolated runtime environment. If execution fails (e.g., KeyError, ZeroDivisionError), the traceback is returned to the agent in a corrective feedback loop for automated repair.
+----------------+ Generates Script +-------------------+
| Agent Router | -------------------------> | Python Sandbox |
+----------------+ +-------------------+
^ |
| Traceback Error |
+-----------------------------------------------+
To bypass the routing cliff, tool catalogs are partitioned into isolated, domain-specific sub-graphs (e.g., Data Scientist, Quant Researcher, Quant Trader) ensuring no single node exposes more than
[ Root Gateway ]
β
βββββββββββββββββββΌββββββββββββββββββ
βΌ βΌ βΌ
[ Data Scientist ] [ Quant Researcher ] [ Quant Trader ]
(77 skills) (99 skills) (100 skills)
Evaluated on a 3-node sequential pipeline: lookahead_bias_audit $\to$ standard_scaler_apply $\to$ kelly_position_size.
| Model | Condition Mode | Routing Success | Execution Success | E2E Success |
|---|---|---|---|---|
| GPT-4o-mini | Direct Prompt Math | 100.0% | 50.0% | 50.0% |
| GPT-4o-mini | Python Sandbox | 100.0% | 94.4% | 94.4% |
| DeepSeek V4 Flash | Direct Prompt Math | 100.0% | 66.7% | 66.7% |
| DeepSeek V4 Flash | Python Sandbox | 100.0% | 100.0% | 100.0% |
Comparing Partitioned Expert SOP ($K \le 15$ tools/node) against Unpartitioned Bare Catalog ($N=77$ tools).
| Model | Configuration Mode | E2E Success | Direct Success | Loopback Recovery | Avg Loopbacks / Run |
|---|---|---|---|---|---|
| DeepSeek V4 Flash | Expert (SOP) | 100.0% | 100.0% | N/A | 0.00 |
| Bare (Flat |
63.3% | 53.3% | 21.4% | 0.87 | |
| GPT-4o-mini | Expert (SOP) | 100.0% | 100.0% | N/A | 0.00 |
| Bare (Flat |
46.7% | 46.7% | 0.0% | 1.07 | |
| Nemotron 550B | Expert (SOP) | 93.3% | 60.0% | 83.3% | 0.70 |
| Bare (Flat |
46.7% | 16.7% | 36.0% | 1.57 | |
| GPT-OSS 20B | Expert (SOP) | 93.3% | 86.7% | 50.0% | 0.23 |
| Bare (Flat |
76.7% | 66.7% | 30.0% | 0.57 |
Demonstrating how distractor packing density ($\bar{S}_{\text{NN}}$) drives routing collapse.
| Catalog Scale ( |
Bare NN Cosine | Expert NN Cosine | DeepSeek Flash Acc | GPT-4o-mini Acc | Nemotron 550B Acc | GPT-OSS 20B Acc |
|---|---|---|---|---|---|---|
| 15 | 0.3500 | 0.3559 | 100.0% | 100.0% | 80.0% | 93.3% |
| 30 | 0.4417 | 0.3854 | 100.0% | 100.0% | 86.7% | 86.7% |
| 60 | 0.3973 | 0.3621 | 100.0% | 100.0% | 80.0% | 100.0% |
| 100 | 0.3842 | 0.3768 | 100.0% | 93.3% | 100.0% | 93.3% |
| 200 | 0.4052 | 0.3683 | 100.0% | 100.0% | 93.3% | 93.3% |
| 500 | 0.5100 | 0.4124 | 100.0% | 86.7% | 93.3% | 46.7% |
| 1000 | 0.4732 | 0.4337 | 100.0% | 86.7% | 86.7% | 60.0% |
Evaluating selection error rate and JSON schema compliance under context pressure.
| Model | Metric (%) | N=5 (Bare / Expert) | N=15 (Bare / Expert) | N=60 (Bare / Expert) | N=200 (Bare / Expert) |
|---|---|---|---|---|---|
| DeepSeek Flash | Select Error | 0.0% / 0.0% | 0.0% / 0.0% | 0.0% / 0.0% | 0.0% / 0.0% |
| Schema Comp | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | |
| GPT-4o-mini | Select Error | 0.0% / 0.0% | 0.0% / 0.0% | 0.0% / 0.0% | 6.7% / 0.0% |
| Schema Comp | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | 100.0% / 100.0% | |
| GPT-OSS 20B | Select Error | 13.3% / 0.0% | 0.0% / 0.0% | 13.3% / 0.0% | 13.3% / 0.0% |
| Schema Comp | 86.7% / 100.0% | 100.0% / 100.0% | 86.7% / 100.0% | 86.7% / 100.0% |
βββ README.md # Project documentation
βββ LICENSE # MIT License
βββ images/ # Visualizations and publication plots
β βββ multi_role_stress_test_comparison.png
β βββ semantic_density_vs_N.png
β βββ attention_degradation_curve.png
βββ skill_experiment/ # Similarity analysis & distractor files
β βββ fillers_v4.json # 8,605 distractor skills
β βββ targets_v4.json # 58 target skills
βββ blackbox_os/
βββ state/
β βββ shared_state.py # Global blackboard state schema
β βββ validation_manager.py # Adaptive validation & sandbox runner
βββ roles_based_skills/ # Complete 276-skill catalog definitions
βββ roles/
βββ data_scientist/
β βββ workflows/ # Data Scientist SOPs & Orchestration
βββ quant_researcher/
β βββ workflows/ # Quant Researcher SOPs & Orchestration
βββ quant_trader/
βββ workflows/ # Quant Trader SOPs & Orchestration
git clone https://github.com/Jay846/Blackbox-OS.git
cd Blackbox-OS
pip install numpy matplotlib sentence-transformers pandas scikit-learn pytest langgraphexport OPENROUTER_API_KEY="your-openrouter-key"- Run Live Orchestrator Stress Test (Table 6):
python3 blackbox_os/roles/data_scientist/workflows/run_orchestrator_stress_test.py- Run Multi-Role Simulation Sweep (Figure 8):
python3 blackbox_os/run_multi_role_stress_test.py- Run Semantic Phase Transition Characterization (Table 7 / Figure 6):
python3 blackbox_os/roles/data_scientist/workflows/characterize_phase_transition.py --live --models deepseek-v4-flash gpt-4o-mini nvidia/nemotron-3-ultra-550b-a55b:free openai/gpt-oss-20b:free --trials 3- Run Attention Degradation & Schema Compliance Sweep (Table 8 / Figure 7):
python3 blackbox_os/roles/data_scientist/workflows/test_attention_degradation.py --models deepseek-v4-flash gpt-4o-mini nvidia/nemotron-3-ultra-550b-a55b:free openai/gpt-oss-20b:free --trials 3If you use Blackbox OS or its empirical benchmarks in your research, please cite our pre-print:
@article{salvi2026blackboxos,
title={Blackbox OS: A Reliable, Sandboxed Execution Runtime & Modular Orchestrator for LLM Agents at Scale},
author={Salvi, Jay},
journal={Zenodo Pre-print},
year={2026},
doi={10.5281/zenodo.21413144},
url={https://doi.org/10.5281/zenodo.21413144}
}




