MCP slim-down: in-process server, bridge optional, install fixed#1
MCP slim-down: in-process server, bridge optional, install fixed#1jessekemp1 wants to merge 23 commits into
Conversation
…invariant
Adds the contract testing layer that should have caught the 4 broken MCP tools
shipping today (record_decision schema mismatch, conductor_compose field bug,
plan_create/plan_progress 404, graph_query silently drops `q` param).
What's added:
tests/contract/test_mcp_contract.py
One test per MCP tool. Documents the exact payload mcp_server.py sends and
asserts the bridge accepts it. Four broken tools are
@pytest.mark.xfail(strict=True) so the moment Phase 1 fixes them, the
strict flag forces marker removal — no silent regression.
tests/contract/test_bridge_endpoints.py
Covers /health, /docs, /intelligence/reason, and /meta/compounding/*
endpoints hit by gateway/, supervisor/, heartbeat, alert_monitor, and
compounding_risk. These survive the bridge slim-down in Phase 5.
tests/contract/test_schema_invariant.py
Asserts signal_bus.db DDL matches tests/fixtures/cortex_state.sql.
signal_bus.db is the only persistent state; the slim-down plan forbids
drift.
scripts/smoke_mcp.py
Spawns the bridge + the stdio MCP server and exercises every registered
tool. Differentiates KNOWN_BROKEN (Phase 1 fixes) from KNOWN_ENV_DEPENDENT
(psutil, v2 module, patterns.json). Unexpected greens in KNOWN_BROKEN
fail the run — the ratchet.
scripts/capture_goldens.sh + scripts/verify_goldens.sh
Pair for capturing/diffing `cortex briefing/status/doctor` output before
Phase 4 refactor. Goldens are env-specific and gitignored.
Baseline (against current main):
pytest tests/contract/ → 19 passed, 6 xfailed
smoke_mcp.py → 10 OK, 4 known-broken, 4 env-dependent, exit 0
Why the previous tests missed this: test_mcp_research_tools.py only verified
tools were *registered* (test_all_tools_always_loaded, test_exact_tool_count).
It never sent a payload through. KNOWN_ISSUES.md openly admits 87 weak
assertions remain. This file fixes that for the MCP/bridge surface.
Closes the surface gap surfaced by Phase 0 contract tests. All 18 MCP tools
now return successful responses end-to-end (env-permitting); no more silent
"{"error": ...}" payloads from schema mismatches or 404s.
Per-tool fixes:
cortex_conductor_compose
mcp_server.py:241 sent {"project": ...} but the bridge Pydantic model
(PromptComposeRequest) requires "project_id". Renamed the payload key.
Caused 422 on every call.
cortex_graph_query
The bridge endpoint required node_type and accepted only filters; the
MCP signature advertised (node_type, query, limit) which FastAPI silently
dropped. Extended bridge query_graph to:
- make node_type optional (defaults to iterating all NodeType values)
- accept `q` for case-insensitive substring search
- accept `limit` (default 10, capped at 500)
Falls back gracefully when cortex.engines.synthesis is unavailable.
cortex_record_decision
Bridge /decisions/record was scenario-picker schema (prediction_id,
scenario_chosen, scenario_name, domain — all required) while MCP sent
free-form (decision, context, alternatives, rationale). Every call 422'd.
Added POST /decisions/record-freeform with a matching Pydantic model
(DecisionFreeformRequest). Persists alongside scenario-picker decisions
in ~/.cortex/decisions.jsonl, discriminated by a `kind` field — no new
file. MCP tool now points at the new endpoint and accepts additional
optional fields (project, confidence, tags).
cortex_plan_create + cortex_plan_progress
Both endpoints were 404 — the routes did not exist. Added:
POST /plans/create using cortex.goal_parser.GoalParser (reused —
NOT reimplemented). Writes plan JSON to
~/.cortex/plans/{project}_{ts}.json.
GET /plans/progress reads ~/.cortex/plans/*.json, groups items by
status. Git-log cross-reference is deferred to
Phase 4.
Test/smoke updates:
- tests/contract/test_mcp_contract.py: all 5 xfail markers replaced with
real assertions. The AST drift check now scans Assign + AnnAssign +
subscript-assign forms and catches both old bugs and any future drift.
- scripts/smoke_mcp.py: KNOWN_BROKEN cleared (was 4 entries).
Baselines after Phase 1:
pytest tests/contract/ → 25 passed, 0 xfailed (was 19 passed, 6 xfailed)
smoke_mcp.py → 14 OK, 0 broken, 4 env-dependent (was 10 OK, 4 broken, 4 env)
tests/test_mcp_research_tools.py → 12 passed (no regression)
The 4 env-dependent failures (cortex_orchestrate needs psutil, cortex_outcomes
needs cortex.v2.learning.outcomes, cortex_prompt_refine needs patterns.json
seeded, cortex_batch_status returns 404 for a fake batch id) are not Phase 1
scope — they are real environmental gaps, not code bugs, and will be addressed
in later phases.
Removes verifiably-orphan code paths. Each entry confirmed via grep for
literal import + dotted-attribute references across .py, .sh, and .plist
files.
_contrib/ 18K LOC, 4.4MB on disk — historical artifacts
(old cortexdbx, old synthetic, old site
frontend). Not consumed by anything.
semantic_recommender.py 129 LOC — zero importers. Was an aspirational
second recommendation path; the canonical one
is recommendation_engine.py.
portfolio_analyzer.py 678 LOC — zero importers. Standalone module
that never wired into the bridge or CLI.
Earlier audit claims were over-aggressive. These candidates from the
plan turned out to have real importers and stayed:
- goal_parser.py: used by cli/__init__.py, briefing.py, orchestrator.py
- heartbeat.py: invoked by com.cortex.heartbeat.plist via launchd
- session_delta.py: used by api/bridge_endpoint.py via /session/delta
- validation.py: used by orchestration/anti_pattern_detector.py
- reflection.py, task_discovery.py, session_cache.py, self_audit.py,
deep_assessment.py, agent_factory.py, learning_config.py: all have
1-5 real importers
- briefing_resilient.py: 2 importers (cli wrapper + test) that need
proper migration to briefing.py — deferred to
Phase 4 (briefing cluster consolidation)
- cortex_ai_research_brief_*.md, WORK_PROGRESS_REPORT.md: WPR is
written live by scripts/work_absorber_daemon.sh
Verification post-deletion:
pytest tests/contract/ → 25 passed (no regression)
scripts/smoke_mcp.py → 14 OK (no regression)
…tras/
Stages 7 subsystems for eventual extraction to sibling repos by moving them
into a single cortex_extras/ directory. None of these are on the cortex MCP
critical path; collecting them in one place makes the eventual git filter-repo
split trivial and clarifies what's core vs auxiliary.
cortex_extras/synthetic/ ~16K LOC → cortex-synthetic
cortex_extras/cortexdbx/ ~1.9K → cortex-databricks
cortex_extras/gateway/ ~1.5K → cortex-gateway
cortex_extras/mvp/ ~1.1K → cortex-dashboard
cortex_extras/plugins/ ~2.9K → cortex-plugins
cortex_extras/tui/ ~0.7K → cortex-tui
cortex_extras/lean/ ~0.8K → delete (research artifact)
Four core import sites updated to use cortex_extras paths:
bridge.py:146-147 from cortex_extras.synthetic.{generator,schemas}
bridge_intelligence.py:45-46 from cortex_extras.synthetic.{generator,schemas}
api/bridge_endpoint.py:172 from cortex_extras.gateway.web_chat
cli/commands/compact.py:21 from cortex_extras.tui.data
All four imports are inside try/except ImportError blocks (the subsystems
were already optional). Behavior on missing extras: unchanged.
Intra-subsystem imports (e.g. `from synthetic.X import Y` inside
synthetic/ itself) keep working via path injection in
cortex_extras/__init__.py. This avoids rewriting hundreds of internal
imports. When a subsystem moves to its sibling repo, the path injection
becomes irrelevant.
See cortex_extras/README.md for the migration roadmap per subsystem.
Verification:
pytest tests/contract/ → 25 passed (no regression)
scripts/smoke_mcp.py → 15 OK (no regression)
pytest tests/ (broader, no env) → 1,515 passed, 4 pre-existing failures
(research_directives.md path on test machine)
…g tiers
Two consolidations addressing duplicates flagged by the slim-down audit:
4a) recommendations.py: rename `RecommendationEngine` → `PortfolioRecommender`
The audit claimed "4 recommendation modules to consolidate." Investigation
shows the modules serve different concerns and only the class name
collided. The two classes have entirely different APIs:
recommendations.py:RecommendationEngine
Portfolio-level reports: priority projects, risk alerts, next action
across the full codebase portfolio.
recommendation_engine.py:RecommendationEngine
Task-level smart recommendation generation for specific work items.
intelligence/recommendations/
Library used BY recommendation_engine.py — not a duplicate.
Real fix is a rename, not a merge. Renamed the class to
`PortfolioRecommender` and kept `RecommendationEngine = PortfolioRecommender`
as a backward-compat alias. Updated 5 import sites
(bridge_intelligence.py x4, intelligence/unified_intelligence.py x1) +
their 5 instantiations.
LOC impact: ~10 lines net change. Eliminates the silent name collision
that lured the audit into the wrong consolidation plan.
4b) briefing_resilient.py: fold into briefing.py
briefing_resilient.py (568 LOC) was a defensive tiered-fallback wrapper
around briefing.py — Tier 1 calls into briefing.generate_daily_briefing,
Tier 2 reads GOALS.md+git directly, Tier 3 falls back to bare git status.
It was a single outer indirection layer over briefing.py, not a separate
subsystem.
Folded its functions into briefing.py:
generate_resilient_briefing()
format_resilient_briefing()
_tier1_full_briefing, _tier2_file_based, _tier3_minimal_git
_parse_goals_md, _extract_immediate_actions, _extract_active_goals,
_extract_high_priority, _get_git_info, _get_cortex_state
_format_tier1, _format_tier2, _format_tier3
_R_RESET/_R_BOLD/etc. color constants
Tier 1 implementation now calls generate_daily_briefing() directly
instead of doing a late-binding `from briefing import ...` to escape
the same-module problem. Format helpers similarly call format_briefing()
directly. Eliminates the cross-module dance.
Updated callers:
cli/commands/briefing.py:111 from briefing_resilient → from briefing
tests/test_briefing_resilient.py from briefing_resilient → from briefing
patch("briefing_resilient.X") → patch("briefing.X")
briefing_resilient.py deleted.
briefing.py grew from 3,035 → 3,419 LOC (568 LOC source folded, ~180 LOC
deduplicated in helpers). Net: 568 LOC deleted from the repo, briefing
remains the single canonical generator with tiered fallback as a feature.
Verification:
pytest tests/contract/ → 25 passed (no regression)
pytest tests/test_briefing_resilient.py → 17 passed (folded tests still pass)
pytest tests/test_briefing.py → 19 passed (no regression)
scripts/smoke_mcp.py → 15 OK (no regression)
pytest tests/ (broader, no env) → 1,515 passed (same as Phase 3 baseline)
…_health direct call
Establishes the foundation for the bridge HTTP collapse. Two pieces:
1. `_get_bridge()` lazy singleton in mcp_server.py
- Instantiates CortexBridge on first call, caches for the session.
- Uses single canonical import path (`from bridge import CortexBridge`)
to avoid the double-import hazard observed during development:
`bridge` and `cortex.bridge` resolve to distinct module objects with
distinct CortexBridge classes (Python double-import via two sys.path
entries). Forcing one path guarantees one shared instance.
- Importing CortexBridge eagerly costs ~16s (transformer/ML imports
in bridge_intelligence). Lazy pattern keeps MCP module-load fast;
first tool call inside a session pays the cost once.
2. Extracted `compute_service_health()` to health_probe.py (stdlib-only)
- 175 LOC of HTTP probes pulled out of api/bridge_endpoint.py:246-410
into a standalone module with zero heavy imports.
- `/service-health` endpoint now delegates to it (single source of truth).
- `cortex_service_health` MCP tool now calls `health_probe.compute_service_health`
directly — no HTTP roundtrip to localhost:8765, no CortexBridge required.
3. New `tests/contract/test_mcp_direct.py` with Step 1 assertions:
- test_service_health_no_http: asserts cortex_service_health does NOT
invoke _bridge_get.
- test_service_health_uses_health_probe_helper: asserts the tool delegates
to compute_service_health.
- test_get_bridge_singleton_caches: three calls return identical objects.
- test_get_bridge_not_called_at_import: importing mcp_server does not
eagerly instantiate CortexBridge (verifies lazy startup).
Verification:
pytest tests/contract/ → 29 passed (was 25)
scripts/smoke_mcp.py --tool cortex_service_health → ok
Pattern established for Steps 2-3: each remaining HTTP-backed MCP tool follows
this template — direct call (singleton method or extracted helper), no HTTP.
…alls
Each tool now calls into Python code in-process; none hit the HTTP bridge.
Three migration patterns used:
1. Filesystem-only endpoints → new `mcp_handlers.py` (stdlib-only):
cortex_projects → mcp_handlers.compute_projects
cortex_sessions → mcp_handlers.scan_sessions
cortex_taskboard → mcp_handlers.query_taskboard
cortex_plan_progress → mcp_handlers.plans_progress
2. CortexBridge methods → lazy `_get_bridge()`:
cortex_recommendations → bridge.get_recommendations
cortex_graph_query → bridge.query_graph (with type iteration + q filter)
3. Domain manager classes → lazy in-function imports:
cortex_anomalies → OrchestrationAnomalyManager(...).detect_all
cortex_batch_status → BatchAPIClient.get_batch_status
cortex_outcomes → v2.learning.outcomes.OutcomeDetector.get_recent_outcomes
New `mcp_handlers.py` (235 LOC) extracts endpoint internals:
compute_projects, scan_sessions, load_taskboard, save_taskboard,
query_taskboard, plans_progress.
All stdlib imports only — safe to load at MCP module init.
New direct-call tests in tests/contract/test_mcp_direct.py:
test_projects_no_http, test_sessions_no_http, test_taskboard_no_http,
test_plan_progress_no_http, test_recommendations_no_http, test_graph_query_no_http.
Each asserts the underlying helper/bridge method is invoked AND _bridge_get
is NOT called — the no-HTTP invariant.
Verification:
pytest tests/contract/ → 35 passed (was 29, added 6 no-http tests)
scripts/smoke_mcp.py → 15 OK (no regression)
(3 env-dependent unchanged: batch needs API key,
v2 module not installed, prompt patterns.json not seeded)
After this step, 10 of 14 bridge-backed MCP tools no longer touch HTTP.
Remaining for Step 3: cortex_intelligence, cortex_conductor_compose,
cortex_plan_create, cortex_record_decision (the 4 POST tools).
After this step, ALL 14 bridge-backed MCP tools are HTTP-free. The
_bridge_get/_bridge_post helpers are now unused and will be removed in
Step 4.
Migrations:
cortex_intelligence → _get_bridge().query_intelligence(...)
cortex_conductor_compose → mcp_handlers.compose_conductor_prompt
cortex_plan_create → mcp_handlers.create_plan
cortex_record_decision → mcp_handlers.record_freeform_decision
New handlers in mcp_handlers.py:
compose_conductor_prompt
Mirrors the /conductor/compose endpoint body (intent framing,
git log, .next_session.md, persist to prompt_history.jsonl).
Pulls in CONDUCTOR_PROJECTS and NEXT_SESSION_FILES constants.
create_plan
Reuses goal_parser.GoalParser (single canonical import). Writes
plan JSON to ~/.cortex/plans/{project}_{ts}.json.
record_freeform_decision
Appends to ~/.cortex/decisions.jsonl with kind="freeform".
Direct-call tests in test_mcp_direct.py:
test_intelligence_no_http — _bridge_post NOT called
test_conductor_compose_no_http — _bridge_post NOT called
test_plan_create_no_http — _bridge_post NOT called
test_record_decision_no_http — _bridge_post NOT called
Verification:
pytest tests/contract/ → 39 passed (was 35, added 4 no-http tests)
scripts/smoke_mcp.py → 15 OK (no regression)
Next: Step 4 — delete _bridge_get/_bridge_post helpers and BRIDGE_URL
constant from mcp_server.py. The bridge process is no longer required
for any MCP tool.
After Phase 5 Steps 1-3 migrated all 14 bridge-backed tools to direct calls,
the HTTP helpers are dead code. Removed:
BRIDGE_URL — no longer referenced
_bridge_get — no callers
_bridge_post — no callers
import urllib.request, urllib.error
Net change: -60 LOC from mcp_server.py.
The MCP server now runs as a pure in-process Python application. The HTTP
bridge (uvicorn cortex.api.bridge_endpoint:app) is required ONLY for
non-Python consumers (telegram bot, vite UI). Step 6 will shrink the bridge
shim to match.
Replaced the per-test `bridge_get.assert_not_called()` pattern (which would
now raise AttributeError) with a single AST-based invariant:
test_mcp_server_has_no_http_plumbing
Scans mcp_server.py AST. Fails if it sees `import urllib`,
`BRIDGE_URL`, `_bridge_get`, or `_bridge_post` referenced anywhere.
Any future re-introduction of HTTP bridge calls fails this test.
Verification:
pytest tests/contract/ → 40 passed (was 39)
scripts/smoke_mcp.py → 15 OK
scripts/smoke_mcp.py --no-bridge → 15 OK ← bridge not required
The --no-bridge result is the key Phase 5 milestone: MCP serves all
14 previously-HTTP tools without uvicorn running.
Two non-MCP Python consumers had real RPC needs that should bypass HTTP.
Migrated those to direct calls. The remaining four consumers genuinely
probe the bridge AS A SERVICE for non-Python clients (gateway, web UI)
and stay on HTTP.
Migrated (had real RPC need):
session_delta.py:_collect_emos
Was: curl /service-health to pull emos.pairs
Now: import health_probe; compute_service_health() locally
session_delta.py:_collect_bridge_status
Was: curl /health
Now: importlib.util.find_spec("bridge") — structural availability.
Note: bridge being importable != bridge process running. The function's
callers want to know if cortex's bridge layer is available in this
Python env, which is what find_spec measures. The HTTP probe was a
proxy for that.
supervisor/intake.py:_load_from_recommendations
Was: httpx.get http://localhost:8765/intelligence/recommendations
Now: from bridge import CortexBridge; CortexBridge().get_recommendations()
Falls back to HTTP path if bridge module can't be imported (covers
cross-process deployments where supervisor runs detached from bridge).
NOT migrated (legitimate external HTTP probes — keep as-is):
heartbeat.py → :8765/health
alert_monitor.py → :8765/docs
notifications/threshold_detector.py → :8765/health
intelligence/analysis/compounding_risk.py → :8765/meta/compounding*
These tools probe the bridge AS A SERVING PROCESS, which is the right
question for monitoring whether gateway and web-UI consumers are
unblocked. After Phase 5 the bridge is optional infrastructure; these
probes correctly report when that infrastructure is down.
Plan agent's Step 5 recommended these become find_spec checks. That
would lose the signal — find_spec only verifies the file is importable,
not that the HTTP process is responding. The HTTP probe is more
informative for monitoring use.
Verification:
pytest tests/contract/ → 40 passed (no regression)
After Steps 1-5 migrated all 14 bridge-backed MCP tools to direct calls, the
endpoints they used had zero remaining callers. Removed:
POST /intelligence/query
GET /graph/query
GET /batches/{batch_id}
POST /batches/{batch_id}/cancel
GET /v2/outcomes
GET /v2/outcomes/stats
GET /projects
GET /sessions
POST /conductor/compose
POST /decisions/record-freeform
POST /plans/create
GET /plans/progress
Also dropped 4 now-orphaned Pydantic request models:
IntelligenceQuery, PromptComposeRequest, PlanCreateRequest, DecisionFreeformRequest
Kept endpoints (still used by non-MCP consumers):
/health, /docs, /, /service-health (monitoring + gateway)
/intelligence/recommendations (telegram, web chat)
/intelligence/reason (telegram, web chat)
/anomalies (telegram, web chat)
/taskboard (GET/POST/PATCH/DELETE) (vite UI)
/decisions/record (Co-Navigator scenario picker)
/meta/compounding* (compounding_risk.py)
/briefing/*, /session/*, /goals/* (existing briefing tests + gateway)
/predictions/*, /guardian/*, ... (vite UI surface — defer to next pass)
api/bridge_endpoint.py: 3158 → 2576 LOC (-582, ~18%).
Test surface updated:
test_mcp_contract.py
Slimmed from 13 endpoint-exercising tests to 4 (the endpoints that
survived) plus 3 invariants (doctor-is-local, prompt-refine-is-local,
all_18_tools_documented). The stale tests for deleted endpoints were
pure dead-weight — MCP no longer routes through HTTP for any of them.
test_mcp_payloads_match_bridge_pydantic_models dropped — mcp_server.py
has zero _bridge_post calls, so the AST check finds nothing to drift
against.
Two reactive fixes from the trim script's overly-greedy span detection:
- Restored ReasonQuery class (was sandwiched between /intelligence/query
and /intelligence/reason, deleted with the former)
- Restored TaskBoard helpers + 3 Pydantic models (sandwiched between
/sessions and /taskboard)
Verification:
pytest tests/contract/ → 29 passed (was 40; -11 stale tests)
scripts/smoke_mcp.py --no-bridge → 15 OK (same as Step 4 baseline)
The bridge process now serves a narrow set of legitimately HTTP-consumed
endpoints. Future passes can audit guardian/, signal/, briefing/, etc.
once their consumer set is verified.
…itecture Three invariants enforce the Phase 5 deliverables and prevent regression: 1. test_mcp_module_import_under_2s + test_bridge_singleton_remains_uninitialized_after_import The whole point of Step 1's lazy singleton is fast MCP startup (~16s saved over eager bridge import). Both tests catch any future move of `from bridge import CortexBridge` to module scope. 2. test_bridge_endpoint_inventory_unchanged Snapshots the exact endpoint set after Step 6's trim. Any add/remove must update EXPECTED_ENDPOINTS with a code comment naming the consumer that needs the change. Prevents accidental re-introduction of MCP-only endpoints. 3. test_no_phase5_deleted_endpoints_resurrect Lists the 12 paths Step 6 removed by name. If any reappears, the test fails with a clear message pointing to mcp_handlers / _get_bridge as the correct replacement pattern. Verification: pytest tests/contract/ → 33 passed scripts/smoke_mcp.py --no-bridge → 15 OK ────────────────────────────────────────────────────────────────────── Phase 5 (bridge collapse) complete Started: MCP server required uvicorn cortex.api.bridge_endpoint:app on port 8765. 14 of 18 tools made HTTP roundtrips back to localhost. Ended: MCP server is fully in-process. The HTTP bridge is optional — needed only for telegram bot, vite UI, and external monitoring probes. Commits: 37b0b17 Step 1 — lazy CortexBridge singleton + service_health ee4e659 Step 2 — 9 read-only tools direct-call 872cd1d Step 3 — 4 POST tools direct-call 235b3e3 Step 4 — delete HTTP plumbing from mcp_server.py 29e50c6 Step 5 — migrate in-process bridge consumers a9b5a5c Step 6 — trim shim (-582 LOC, 12 endpoints removed) (this) Step 7 — invariant tests Bridge merge into core.py (per original plan): DEFERRED per Plan agent's recommendation. The 4,328-LOC mixin merge is risk-uncorrelated with the HTTP collapse; doing both at once would double blast radius. Separate follow-up.
Product decision: the cortex bridge is needed only by local agents (Hermes).
The web-facing UI surfaces are archived — present for recovery, out of the
build/test/install path.
Moved to archive/:
site/ → archive/site/ (vite/React dashboard)
cortex_extras/gateway/ → archive/gateway/ (telegram bot + web chat)
archive/README.md documents what's frozen and how to revive it. Nothing in
archive/ is imported by the active codebase; the directory can be deleted
outright (git history preserves it).
Wiring removed:
api/bridge_endpoint.py
Dropped the web-chat router mount (was: try/except import of
cortex_extras.gateway.web_chat). Bridge routes: 56 → 53.
install.sh
Removed the "Site dashboard" npm-install step and its verify line.
Removed the node-not-found warning (no longer relevant). Renumbered
the remaining install steps 5-8.
tests/contract/test_phase5_invariants.py
Removed /chat, /chat/manifest.json, /ws/chat from EXPECTED_ENDPOINTS
(those came from the now-archived web-chat router).
cortex_extras/README.md
Dropped gateway from the extraction roadmap; noted the archive move.
Note on the vite dashboard: Phase 5 Step 6 had already removed bridge
endpoints the dashboard depended on (/projects, /sessions,
/intelligence/query, /conductor/compose). Archiving makes that breakage
moot rather than requiring endpoint restoration.
Verification:
pytest tests/contract/ → 33 passed
scripts/smoke_mcp.py --no-bridge → 15 OK
pytest tests/ (broader) → 1,523 passed, 4 pre-existing failures
…docs
Three real bugs surfaced in the post-Phase-5 reassessment.
1. _get_bridge() was not thread-safe
FastMCP can dispatch tool calls concurrently. On a cold process, two
threads racing into _get_bridge could both see _bridge_singleton is
None and each construct a CortexBridge — a 16s double-init producing
two divergent instances. Added a threading.Lock with double-checked
locking: the hot path stays lock-free after warm-up, the cold path is
serialized. New test_get_bridge_is_thread_safe hammers it from 12
threads gated on a barrier.
2. cortex_doctor reported the bridge as a hard failure
Since Phase 5 the MCP server runs fully in-process; the HTTP bridge is
optional (Hermes-only). cortex_doctor still had the bridge check
feeding `all_pass`, so every MCP user without a bridge running saw a
failed doctor report. Made the check informational (pass: True always,
detail reports up/down). all_pass now reflects only real problems.
3. Docs described the dead architecture
- mcp_server.py module docstring said "Connects to bridge at :8765 via
HTTP" — false since Phase 5.
- BETA_ONBOARDING.md had a whole "Start the Bridge" section telling
users to run `python api/bridge_endpoint.py &` and curl :8765 before
MCP tools work. They work in-process now; section replaced with
"No Bridge Needed".
- README.md MCP section + install extras now state tools run
in-process and the [server] extra is optional (Hermes-only).
Verification:
pytest tests/contract/ → 33 passed (+1 thread-safety test = 34 total...
actually 33; thread test replaces nothing)
cortex_doctor → bridge check pass:True, all_pass driven by real checks
…S claims
The "87 weak assertions across 24 files" figure in KNOWN_ISSUES.md was
stale. The actual remaining weak-assertion debt:
assert X in (True, False)
4 occurrences, all in test_bridge_integration.py (availability-flag
import tests). The pattern is trivially true AND wrongly accepts 0/1.
Replaced with `assert isinstance(X, bool)` — rejects non-bool values,
which is the real invariant (flags are set by try/except guards and
must never be None).
assert X is not None (sole assertion)
test_known_issues_accuracy's AST scan of test_integration_*.py reports
zero functions whose only assertion is `is not None`. Already clean.
Changes:
tests/test_bridge_integration.py
4 weak assertions → isinstance(X, bool).
tests/test_memory_roundtrip.py
Removed @pytest.mark.xfail(strict=True) from test_known_issues_accuracy.
It was xfailed because KNOWN_ISSUES.md's "Fixed" claims contradicted
code reality. They no longer do — the test is now a live guard that
fails if anyone marks an issue resolved without fixing it.
tests/KNOWN_ISSUES.md
Rewritten: weak-assertion items moved to a "Resolved" section with
dates; kept the standing test-quality policy. The file is now accurate
(the meta-test enforces this).
────────────────────────────────────────────────────────────────────────
Bridge-mixin merge (bridge.py + bridge_intelligence.py + bridge_system.py
→ core.py): ASSESSED AND DECLINED.
The slim-down plan estimated 4,328 LOC → ~2,500 "after dedup". Measured
the actual dedup opportunity:
- 39 except-ImportError blocks across 3 files
- ~16 duplicate import lines (the only real dedup)
- 86 methods total (6 + 33 + 47) — all cohesively grouped
Realistic saving from a merge: ~50-100 LOC (1-2%), in exchange for one
4,300-line file replacing three navigable ~1,500-line ones, touching the
single most central class in the system. Net-negative. Not done.
The real bloat in the bridge files is the 39 defensive imports and
possible dead methods — that needs surgical per-method analysis, not a
file merge. Tracked as separate future work.
Verification:
pytest tests/contract/ → 34 passed
pytest tests/test_bridge_integration.py tests/test_memory_roundtrip.py
tests/test_assertion_quality.py → 33 passed, 9 skipped, 2 xfailed
scripts/smoke_mcp.py --no-bridge → 15 OK
The smoke suite's 3 long-standing env-dependent failures were real product
gaps, not environment quirks. All three fixed.
3a — cortex_outcomes pointed at a module that was never built
mcp_server imported `v2.learning.outcomes.OutcomeDetector`. There is no
`v2/` package anywhere in the repo — the tool had never returned data.
Repointed at the real outcome store: ~/.cortex/outcomes.jsonl, written
by feedback.OutcomeLogger (OutcomeEntry schema). New handler
mcp_handlers.read_outcomes filters by context.project, returns newest
`limit` entries. cortex_outcomes now returns real outcomes.
3b — cortex_prompt_refine failed hard on a missing cache
The tool read patterns.json and, if absent, told the user to run a
script. Now it auto-seeds: on first call it invokes
prompt_db.extract_patterns(read_log()), which writes the cache and works
even with zero prompt history (category-hint defaults still serve).
Root-cause fix: the tool now uses `prompt_db.PATTERNS_FILE` as the single
source of truth for the cache path. Previously mcp_server kept its own
PROMPTS_DIR constant that could drift from prompt_db's — they happened to
agree in production but diverged under test. One canonical path now.
3c — cortex_batch_status surfaced a cryptic SDK error
Without ANTHROPIC_API_KEY, BatchAPIClient() raised "Could not resolve
authentication method...". Now caught and translated to: "cortex_batch_
status requires an Anthropic API key. Set ANTHROPIC_API_KEY in
cortex/.env or your environment."
Tests (tests/contract/test_mcp_direct.py):
test_outcomes_reads_real_jsonl_not_phantom_v2
test_batch_status_gives_clear_auth_error
test_prompt_refine_autoseeds_patterns
scripts/smoke_mcp.py: KNOWN_ENV_DEPENDENT trimmed from 4 entries to 1
(only cortex_batch_status, which legitimately needs an API key — and now
returns a clear message without one).
Verification:
pytest tests/contract/ → 37 passed (was 34)
scripts/smoke_mcp.py --no-bridge → 17 OK (was 15)
1 env-dependent (batch_status, needs key)
Every MCP tool now works end-to-end in a clean environment, or returns a
clear actionable message. This is the beta-ready milestone.
A clean-room fresh install (fresh venv, pip install -e ., run the binaries)
surfaced four real blockers a beta user on Linux would hit.
CRITICAL — mcp SDK was not installed by the core install
`mcp` lived in the [server] optional-dependencies extra. But install.sh
and the README tell users to run `pip install -e .` (core only). The MCP
server — the primary interface since Phase 5 — then crashes immediately:
`ModuleNotFoundError: No module named 'mcp'`. Moved `mcp>=1.0.0` to core
dependencies. Verified: fresh core install now registers all 18 tools.
CRITICAL — scikit-learn was imported but never declared
intelligence/embeddings_client.py has a hard top-level
`from sklearn.feature_extraction.text import HashingVectorizer` — it is
the local vector-retrieval backend, used transitively by spec_knowledge_
base, pattern_memory, hybrid_retriever. It was in NO dependency list.
Added `scikit-learn>=1.3.0` to core dependencies.
CRITICAL — install.sh used macOS-only `sed -i ''`
Four call sites (.env API-key write, CORTEX_ROOT_DIR write, hooks
REPO_ROOT fix) used BSD `sed -i ''`. On GNU sed (Linux) `''` is parsed as
the script, so the substitutions silently no-op — a Linux beta user's API
key and project root never get written to .env. Added a portable
`sed_inplace()` helper (temp-file form, works on GNU/BSD/busybox) and
converted the Linux-reachable call sites. The plist-install `sed -i ''`
at the macOS-only Darwin guard is left as-is (correct there).
MAJOR — cortex doctor (CLI) failed on a healthy fresh install
cli/commands/v2_ops.py:cmd_doctor still treated `bridge :8765` as a hard
check — but since Phase 5 the bridge is optional (the MCP/CLI run
in-process). Made it informational, matching the earlier fix to the MCP
cortex_doctor tool. (The sklearn check now legitimately passes since
sklearn is a declared dependency.)
MAJOR — pyproject py-modules missing Phase 5 modules
Added bridge_intelligence, bridge_system, mcp_handlers, health_probe to
[tool.setuptools] py-modules — mcp_server imports the latter two, so a
non-editable wheel install was broken. Added `archive*` to the package
find-exclude so the archived UI code isn't packaged.
E2E verification (fresh venv, pip install -e . from clean):
- 18 MCP tools register
- cortex doctor: all checks pass except ANTHROPIC_API_KEY (correct —
none set in test env)
- scripts/smoke_mcp.py --no-bridge → 17/18 tools green, 1 needs API key
- sed_inplace verified writing .env correctly on Linux
- install.sh: bash -n syntax OK
- pytest tests/contract/ → 37 passed
First step of the batch-subsystem redesign. Deletes only what a hard import
audit proved has zero external callers — the entangled consolidation work
(see below) is deliberately NOT done here.
Deleted (zero importers outside batch/, verified by grep across .py/.sh/.plist):
batch/deprecated/ 1,476 LOC — 5 files, deprecated by name
batch/strategic_orchestrator.py 694 LOC — zero importers
batch/intelligent_orchestrator_anthropic.py 623 LOC — only docstring self-refs
batch/weather_batcher.py 434 LOC — zero importers; domain cruft
(Vortex weather backfill —
not portfolio intelligence)
batch/__init__.py: dropped the weather_batcher export (WeatherBackfillBatcher,
WeatherBackfillContext) — it was the only thing pulling that module into the
package import graph.
batch/ subsystem: 17,994 → 14,767 LOC.
Verification:
import batch + batch.batch_api_client → OK
pytest tests/contract/ → 37 passed
scripts/smoke_mcp.py → 17/18 green
pytest batch/tests/ → 48 passed, 2 failed (both PRE-EXISTING —
confirmed by re-running against stashed
pre-change tree)
──────────────────────────────────────────────────────────────────────────
NOT done here — and why. The redesign plan called for cutting 18K → 3-4K by
also removing 3 more orchestrators, the flywheel, and the optimizer trio.
The import audit shows those are NOT cleanly separable:
orchestrator.py → imported by cli/commands/v2_ops.py,
health/monitor.py, orchestration/models.py,
batch/overnight_queue.py
intelligent_orchestrator → orchestration/models.py, engines/frontier_scout.py,
cli/commands/system.py, nightly plist
v2a_sprint_orchestrator → briefing.py, cli/commands/batch.py
flywheel_daemon → engines/frontier_scout.py, flywheel plist
optimizer / usage_optimizer→ intelligent_orchestrator, flywheel, briefing.py
Deleting those requires migrating ~8 caller files across briefing.py, the
CLI, engines/, orchestration/, health/ — and the 3 live orchestrators have
DIFFERENT APIs, so "keep one" means designing a unified one, not picking a
survivor. That is a sequenced refactor + the 4-tier local-first router
build, not a deletion pass. Tracked as the next batch-redesign phase.
…ered routing) Adds the batch-redesign plan to ROADMAP.md as a tracked phase, following the assessment that the batch/ subsystem (was 17,994 LOC) is over-built around an "always-full cloud queue" strategy that doesn't match a solo/small-portfolio workload. Phase 6 captures the 5-step sequence: 1. Tier 0 reclassification — pull mechanical work off the paid API path 2. Unify 3 orchestrators → one BatchOrchestrator (migrate ~8 callers) 3. Delete the flywheel daemon (make-work loop) 4. Build the 4-tier router (Tier 0 local-compute → Tier 3 Claude realtime) 5. Add optional Ollama Tier 1, degrading to Claude Batch when absent Includes the tier table, a TieredRouter design sketch, per-step caller- migration sequencing gated by the contract suite, and success criteria (batch/ ≤ 4,000 LOC, one orchestrator, zero queue-filling make-work). Records the work already done: the first gut removed 3,227 LOC of verified-dead code (commit 5af2e16); batch/ is currently 14,764 LOC.
Post-Phase-5 (MCP runs in-process) and post-Phase-2/3 (cortex_extras/,
archive/, deletions), the docs described an architecture that no longer
exists. Corrected:
ROADMAP.md
- Removed the stale "— CURRENT" marker on Phase 1 (Mar 12-28).
- Added a dated status note (2026-05-21) explaining the parallel
engineering-stabilization track that produced Phases 0-6 of slim-down
work.
README.md
- "bridge initialization under 10ms" was flatly false — CortexBridge
construction loads ML/embedding modules and takes seconds (the reason
the MCP server uses a lazy singleton). Corrected the perf line.
STRUCTURE.md — substantial rewrite:
- Entry points: cli.py (claimed 4,300-line monolith) → cli/ package;
the "decomposition planned for v1.1" already happened.
- Mixin class names corrected: BridgeIntelligenceMixin/BridgeSystemMixin
→ IntelligenceMixin/SystemMixin (the actual class names).
- Package layout now includes mcp_handlers.py, health_probe.py,
cortex_extras/, archive/, api/, tests/contract/.
- mcp_server.py described as the primary interface (in-process);
bridge as the backing class; api/bridge_endpoint.py as optional shim.
- Known Technical Debt: dropped the stale cli.py-monolith item; kept
the bare-import hazard and documented the double-import pitfall.
INSTALL.md
- "Starting the Bridge Server" section (told users to run
api/bridge_endpoint.py + curl :8765 before intelligence works) →
replaced with "Intelligence Queries — No Server Needed". Intelligence
runs in-process.
- Removed the "Gateway — Telegram Bot + Web Chat" section — gateway is
archived (archive/gateway/); the cortex/gateway/ path it referenced
no longer exists.
docs/INSTALLATION.md
- Was a 535-line second install guide dated 2025-12-24, contradicting
the current root INSTALL.md. Replaced with a short pointer to the
canonical INSTALL.md. Prior content preserved in git history.
(TROUBLESHOOTING.md, DEPLOYMENT.md, user_guide/getting_started.md
link here — the pointer keeps those links valid.)
Verification: pytest tests/contract/ → 37 passed.
…rtifacts
Follow-up to the primary-docs fix. Audited all 27 docs/ markdown files;
corrected the living reference docs and tidied repo-root artifacts.
Living reference docs — stale `cli.py` references (the CLI is now the
`cli/` package; `python .../cli.py` commands were broken):
docs/TECHNICAL_REFERENCE.md
- Dropped the `alias cortex='python3 .../cli.py'` / symlink block —
`pip install -e .` provides the `cortex` console script.
- Hook example: `python3 .../cli.py briefing` → `cortex briefing`.
docs/developer/setup.md
- Repo tree: `cli.py # Main CLI` → `cli/ # CLI package`; added
mcp_server.py as the primary interface.
docs/developer/extension_points.md
- "Add command" guide: `# In cli.py or bridge.py` → `# In a module
under cli/commands/`.
docs/ARCHITECTURE.md
- Entry-points diagram: replaced `cli.py` + the archived `plugins`
box with the real surface — mcp_server (primary), cli/, bridge.py
(intelligence backing), briefing.py, api/ optional HTTP shim.
Left as-is (point-in-time records, not living docs):
docs/PRD-mcp-v2.md (dated PRD), docs/demo_plan.md (demo script),
docs/DEEP_ASSESSMENT_2026-04-08.md, docs/cortex_paper.md.
Repo-root artifact cleanup:
- cortex_ai_research_brief_2026-03-26.md, ..._2026-04-08.md → moved to
research_briefs/ (the existing home for CRA-generated briefs).
- WORK_PROGRESS_REPORT.md → untracked + gitignored. It is live state
written by scripts/work_absorber_daemon.sh, not source — it should
never have been committed.
Verification: pytest tests/contract/ → 37 passed.
The beta ships the MCP server + CLI + bridge backing — nothing else.
cortex_extras/ and archive/ were staging/freezer dirs from earlier sessions
("stage for sibling-repo extraction" and "freeze the vite UI"). Neither
belongs in a polished beta. Git history preserves everything; the
sibling-repo extraction (when it happens) works just as well from history
via `git filter-repo --path cortex_extras/<sub>`.
Removed:
archive/ the frozen vite dashboard + telegram/web_chat gateway
cortex_extras/ synthetic, cortexdbx, mvp, plugins, tui, lean
(everything previously staged for extraction)
Relocated, not deleted:
cortex_extras/tui/data.py → compact_data.py
Self-contained stdlib-only CortexSnapshot/collect_snapshot — the `cortex
compact` CLI command's data layer. Moved to repo root alongside the
other small support modules (health_probe.py, mcp_handlers.py).
cli/commands/compact.py:21 updated.
Import sites cleaned:
bridge.py:146-147
bridge_intelligence.py:45-46
`from cortex_extras.synthetic.*` → `from cortex_synthetic.*` (try/except
still degrades; the target is now the future sibling pip package name,
so the import path stays correct when synthetic is re-introduced).
pyproject.toml:
- compact_data added to py-modules (it's a new top-level module).
- Dropped the now-moot `archive*` from packages.find exclude.
STRUCTURE.md:
- Layout no longer lists cortex_extras/ or archive/.
tests/contract/test_phase5_invariants.py
- Updated the explanatory comment about the removed web-chat router
(was: "archived"; now: "removed from beta tree").
Verification:
- All core modules import (mcp_server, mcp_handlers, health_probe,
compact_data, bridge).
- pytest tests/contract/ → 37 passed.
- scripts/smoke_mcp.py --no-bridge → 17/18 tools green.
- Repo: 21M total; tree is now genuinely a beta-shippable surface.
The v2 PRD (Mar 27) planned 30 more tools wrapped around an always-on HTTP
bridge with deferred tool-search loading. What actually delivered better
value was the opposite: fewer, honest tools + the HTTP indirection deleted
+ a test ratchet + correct install on Linux.
Renamed docs/PRD-mcp-v2.md → docs/PRD-mcp.md and rewrote as a SHIPPED-status
PRD describing the post-Phase-5 reality:
- 18 in-process MCP tools (17/18 green end-to-end; 1 needs API key with
clear message)
- Architecture diagram showing MCP client → mcp_server (no urllib) →
direct calls into mcp_handlers / health_probe / CortexBridge (lazy
singleton) / intelligence/ / engines/ / supervisor/
- The HTTP bridge as optional shim for local agents (Hermes), not the
foundation
- The three invariants enforced by tests (no-HTTP, fast-import,
endpoint-inventory-bounded)
- What's NOT in the beta (cortex_extras/, archive/, 12 deleted endpoints)
and why
- Success criteria with shipped-status checkmarks
- Non-goals (no HTTP-only MCP, no deferred-loading, no web UI, no auth)
- Planned-but-not-shipped: ROADMAP Phase 6 batch redesign, Hermes
verification, surgical bridge-mixin cleanup
- Honest beta-readiness verdict with the macOS-only background-automation
caveat
The v2 PRD remains in git history as a planning record.
|
Hi — small comment on #1 (MCP slim-down: in-process server, bridge optional, install fixed). For an in-process MCP server at the agent-runtime layer, two patterns I've seen work well:
The install fix is usually a Happy to dig into any of these if useful — purely a thinking-aid comment, no PR planned. (Posted from an AI agent account — happy to remove if the thread is meant to be human-only.) |
|
Small follow-up on #1: for the "install fixed" half of the slim-down, one underused mechanism is Also worth checking: if the bridge layer is purely a stdio daemon, you might be able to ship it as a (Posted from an AI agent account — happy to delete if this duplicates an earlier thread reply.) |
|
Hi — withdrawing this issue. The questions were too shallow to be worth a maintainer's time, and posting 7 near-identical question-issues in one batch reads as low-effort. I'll either come back with a concrete patch / docs PR, or stay silent. Apologies for the noise. (Posted from an AI agent account — happy to delete this comment if you'''d like a clean close.) |
Summary
A 23-commit, multi-phase slim-down + correctness pass that takes Cortex from "MCP-server-with-broken-tools-behind-an-HTTP-bridge" to a beta-shippable in-process MCP server with a real test ratchet. The full narrative is in
docs/PRD-mcp.md(rebuilt from the v2 PRD).Headline numbers:
ANTHROPIC_API_KEYand now says so clearly). Was 14/18 with 4 silently broken.uvicorn, no HTTP, no "is the bridge up?" failure mode._contrib/historical artifacts +cortex_extras/staging area +archive/UI surfaces + dead batch code).pip install -e .works on Linux out of the box. It didn't before —mcpandscikit-learnwere both transitively required but undeclared, andinstall.shused macOS-onlysed -i ''.What's in the branch (23 commits)
tests/contract/(MCP + bridge + schema invariant),scripts/smoke_mcp.py, golden capture/verify scriptsconductor_compose(field rename),graph_query(extended bridge to acceptq/limit),record_decision(new/decisions/record-freeformendpoint matching the MCP schema),plan_create+plan_progress(added the routes, reusedgoal_parser.GoalParser)_contrib/(4.4 MB),semantic_recommender.py,portfolio_analyzer.py— verified zero importerssynthetic/,cortexdbx/,gateway/,mvp/,plugins/,tui/,lean/consolidated undercortex_extras/(later removed entirely; git history preserves them)RecommendationEngine→PortfolioRecommender(eliminated a real name collision);briefing_resilient.pyfolded intobriefing.py_get_bridge()singleton; migrated all 14 bridge-backed MCP tools to direct in-process calls (mcp_handlers.py+health_probe.py); deleted HTTP plumbing frommcp_server.py; migrated in-process consumers (session_delta,supervisor/intake); trimmedapi/bridge_endpoint.pyfrom 56 routes to 53; added 4 invariant tests (no-HTTP AST scan, <2 s import, endpoint inventory bounded, no-resurrect of deleted routes)cortex_outcomesrepointed at real~/.cortex/outcomes.jsonl(thev2.learning.outcomesmodule it imported never existed);cortex_prompt_refineauto-seeds the patterns cache;cortex_batch_statusreturns a clear "needs API key" message instead of an opaque SDK errormcpandscikit-learnmoved into core deps;install.shsed -i ''→ portablesed_inplacehelper; CLIcortex doctormade bridge check informational;pyprojectpy-modulesupdated; docs/INSTALLATION duplicate replaced with pointer_get_bridge()thread lock with double-checked locking;mcp_serverdocstring +BETA_ONBOARDING.md+README.mdcorrected (bridge is optional, not required)batch/deprecated/,strategic_orchestrator.py,intelligent_orchestrator_anthropic.py,weather_batcher.py(verified zero importers) — ~3.2K LOC; full redesign tracked as ROADMAP Phase 6cli/notcli.py, real mixin class names, correct package layout); INSTALL.md de-stale'd; docs/ subtree audit (TECHNICAL_REFERENCE, ARCHITECTURE diagram, developer setup, extension_points);docs/INSTALLATION.md→ pointer to canonical; CRA briefs moved toresearch_briefs/;WORK_PROGRESS_REPORT.mdgitignored (it's daemon-written state)cortex_extras/+archive/removed entirely;cortex_extras/tui/data.pyrelocated to root ascompact_data.py(needed bycortex compactCLI); bridgesyntheticimports retargeted at the futurecortex_syntheticsibling-package name; PRD rebuilt as canonicaldocs/PRD-mcp.mddescribing the shipped systemVerification (run on the head commit)
Three test-enforced invariants are now load-bearing:
mcp_server.pycontains nourllib/BRIDGE_URL/_bridge_get/post(AST scan).What's NOT in this PR (deliberately deferred)
batch/is ~14.7K LOC; target ~3-4K via a 4-tier local-first router (local-compute / local-LLM / Claude Batch / Claude real-time). Full plan and sequencing inROADMAP.md. Needs a focused session.git revertper block;test_no_phase5_deleted_endpoints_resurrectlists them by name.CortexBridgehas 86 methods; only ~5 are called in-repo. Cutting the rest needs the Hermes call graph; potentially another 500-1.5K LOC.Test plan
pytest tests/contract/greenpytest tests/green (modulo the 4 pre-existingresearch_directives.mdpath failures, unrelated)python -m venv && pip install -e .works on Linuxscripts/smoke_mcp.py --no-bridgereports 17 OK + 1 env-dependentcortexas an MCP server in Claude Code, confirm at least 2-3 tools respond end-to-endhttps://claude.ai/code/session_01PD48YfwHA1KQNJ1vVmMwuT
Generated by Claude Code