diff --git a/.investigations/jfr-mcp-diagnose-deadline/INDEX.md b/.investigations/jfr-mcp-diagnose-deadline/INDEX.md new file mode 100644 index 00000000..a34c1c34 --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/INDEX.md @@ -0,0 +1,19 @@ +# Investigation Index + +## Findings (confirmed) +- find-deadline-vs-real-cost | diagnose( full ) response 13-21s vs 15s deadline; fixed 15s->60s + property forwarding | [jfr-mcp, diagnose, timeout, e2e] + +## Hypotheses +(none open) + +## Dead ends +- dead-env-classification | Calling it 'environmental/flaky' without measuring | [classification] | REFUTED +- dead-heap-gc-logging | Heap, GC pressure and logging config as causes | [heap, gc, logging] | REFUTED + +## Evidence +- ev-run-timings | Three failing 15s runs + three green raised-deadline runs + heap-ablation runs | [timings, runs] +- ev-parse-bench | Warm full untyped parses of the 2.2MB test recording, agent-laden vs agent-free | [parser, bench, agents] +- ev-jfr-profile | jfr ExecutionSample attribution + GC view: parser internals on top, 6 GCs/26ms max | [jfr, profile, gc] + +## Questions +- q-parser-perf | Parser/analysis cost shape: hasConstantPool share, per-parse 14-thread daemon pools leaking until session close, agent tax on every test JVM | [parser, perf, follow-up] \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/STATE.md b/.investigations/jfr-mcp-diagnose-deadline/STATE.md new file mode 100644 index 00000000..83b8394a --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/STATE.md @@ -0,0 +1,34 @@ +# Current State + +## Active investigation +None — concluded 2026-10-03. Status `done` in meta.yaml; root cause found, fix merged in +`259a0da` (MCP transport tests: raise the response deadline to 60s). + +## What was concluded +`jfr_diagnose` (default `depth=full`, `includeAnalysis=true`) re-parses the whole recording for +each of its six sub-analyses; a warm full untyped parse of the 2.2 MB test recording measures +0.45–0.73 s (0.53–0.73 s under the test JVM's jacoco/mockito agents). ~10–12 passes ⇒ 13–21 s +real response time, vs the harness's 15 s deadline — the boundary is inside the natural range, +so the test failed whenever the box was loaded (including on pristine origin/main) and passed +under any raised deadline. Fix: default 60 s + forward `-Dmcp.test.timeout.ms`. + +## Open questions (for a successor) +- q-parser-perf — the diagnose cost shape is architectural: `MetadataField.hasConstantPool` + ≈ 15 % of CPU in the profile; each parse spawns a fresh fixed pool of + `availableProcessors()-2` unnamed daemon threads (`StreamingChunkParser` thread factory has no + naming/teardown until session close); agents tax every test JVM 15–40 % (paired measurements). + Any redesign is a planned-effort item, not a gate blocker. + +## Ruled out (don't re-investigate) +- Heap: 8 GB vs 2 GB — identical timings. +- GC pressure: JFR profile run — 6 collections / 45 s, longest pause 26 ms. +- Logging: one shared main/resources/logback.xml on both in-process and child paths; parser at WARN. +- Hang: three 120–180 s deadline runs all returned 13–21 s responses. +- Backend/wiring mismatch: both paths register the same `JafarMcpServer` tool specs and the same + jf shell-jafar backend (checked harness construction, `McpServer.sync`). + +## Fix that landed +`McpTransportHarness.RESPONSE_TIMEOUT_MS` 15_000 → 60_000; `:jfr-mcp:test` forwards +`-Dmcp.test.timeout.ms` (the harness failure message advertises it, the task used to ignore it). +R5 two-sided proof: 2000 ms fails the named test; default passes the whole class. +Case file: doc/agents/Verification.md under R6 ("the diagnose deadline between green and failure"). \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-jfr-profile.md b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-jfr-profile.md new file mode 100644 index 00000000..51ae2177 --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-jfr-profile.md @@ -0,0 +1,37 @@ +--- +id: ev-jfr-profile +type: evidence +created: 2026-10-03 +updated: 2026-10-03 +--- + +# JFR method profile + GC view of the focused class run + +Test JVM temporarily recorded with `-XX:StartFlightRecording=filename=/tmp/diag.jfr,settings=profile` +(run green, 18s-class). `jfr print --events jdk.ExecutionSample --json` → 4417 runnability samples. + +Deepest-frame (self) attribution — top 12: + +| samples | frame | +|---|---| +| 695 | io/jafar/parser/internal_api/metadata/MetadataField.hasConstantPool | +| 432 | io/jafar/parser/internal_api/RecordingStreamReader$BufferBackedRecordingStreamReader.readVarint | +| 278 | io/jafar/parser/ParsingUtils.parseLongSWAR | +| 274 | java/util/Objects.equals | +| 218 | io/jafar/parser/internal_api/RecordingStreamReader$BufferBackedRecordingStreamReader.readVarintSeq | +| 205 | java/util/HashMap.putVal | +| 168 | io/jafar/parser/internal_api/GenericValueReader.readValue | +| 166 | io/jafar/parser/internal_api/metadata/AbstractMetadataElement.getName | +| 140 | io/jafar/parser/internal_api/metadata/AbstractMetadataElement.getId | +| 114 | java/util/concurrent/ConcurrentHashMap.get | +| 114 | java/util/concurrent/ConcurrentHashMap.computeIfAbsent | +| 95 | io/jafar/parser/impl/MapValueBuilder.onLongValue | + +Also visible: `$jacocoInit` probes inside parser metadata classes (52 samples on +AbstractMetadataElement alone) — agent probes sit inside the hot loop. + +GC view: 6 collections over the 45s window, longest pause 26.3ms — GC pressure excluded. +Full stack (jstack during jfr_use): handler runs on Reactor `boundedElastic-1` parked in +`FutureTask.get` inside `StreamingChunkParser.parse:325`; chunk work on parser-pool threads +(unnamed `Thread-N` pairs — the custom thread factory in StreamingChunkParser does not name +threads; pool size = availableProcessors()-2 per parser instance). \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-parse-bench.md b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-parse-bench.md new file mode 100644 index 00000000..425eaf07 --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-parse-bench.md @@ -0,0 +1,26 @@ +--- +id: ev-parse-bench +type: evidence +created: 2026-10-03 +updated: 2026-10-03 +--- + +# Warm full untyped parse of the 2.2MB test recording (in-executor bench) + +Throwaway JUnit bench (`io.jafar.mcp.TmpParseBench`, deleted after the runs) driving +`ParsingContext.create().newUntypedParser(test-dd.jfr)` + `handle` + `run()` inside the real +Gradle test executor — no transport, no harness: + +| Config | runs (events=67657) | +|---|---| +| With agents (default test task: jacoco + mockito agents, `cores=12`) | 1.793s cold, 0.727s, 0.529s | +| Agents stripped (init script removed both `-javaagent` flags) | 1.980s cold, 0.588s, 0.449s | + +Agent cost at the parser level ≈ 15–25%; at the tool level (use/tsa/diagnose) the paired +test runs showed 15–40% — the analysis layer pays per-method-entry probes too, not just the +parse loop. + +Implication used for the fix: diagnose(full) ≈ six sequential sub-analyses, each re-running +whole-recording queries (USE fans out over ~4 event families, TSA ~4); ~10–12 parse-equivalents +× ~0.5s ⇒ 6–10s ideal CPU, 13–21s observed with agents + load. The 15s harness deadline was +inside the tool's natural range. \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-run-timings.md b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-run-timings.md new file mode 100644 index 00000000..47503acc --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/evidence/ev-run-timings.md @@ -0,0 +1,26 @@ +--- +id: ev-run-timings +type: evidence +created: 2026-10-03 +updated: 2026-10-03 +--- + +# Test-run timing series for McpJfrTransportTest.jfrDiagnoseReturnsReport + +All runs `:jfr-mcp:test --tests io.jafar.mcp.McpJfrTransportTest` unless noted; testcase wall +time from JUnit XML (includes jfr_open + harness teardown). + +| Run context | Deadline | Result | +|---|---|---| +| Branch-tip full suite (`:jfr-mcp:test`, 306 tests) | 15s | FAILED (only this test) — pre-rebase baseline | +| Focused, branch tip | 15s | FAILED at 17.8s testcase time (open ≈2.3s + 15s expiry) | +| Pristine origin/main worktree (`/tmp/jafar-main-wt`) | 15s | FAILED at 20.5s — same assertion, same message | +| Focused, `-Dmcp.test.timeout.ms=60000` (NOT forwarded at the time — no-op) | 15s | FAILED identically → forwarded-property gap discovered (R4) | +| Focused, `-Dmcp.test.timeout.ms=120000` (forwarding added) | 120s | green, 17.9s testcase | +| Focused under sampling, 180s | 180s | green, 20.9s (use 8.1s — box loaded: concurrent drydock-guided-tour suite observed with top) | +| Focused, agents stripped via init script | 180s | green, 13.1s (use 4.5s, tsa 2.3s, help 3.9s) | +| Heap ablation: `-Xmx8g` (2g default) | 15s | FAILED 18.4s — heap not the cause | +| Default after fix (60s) 2 | 60s | green, `:jfr-mcp:test --rerun-tasks` 404/404 | + +Evidence sources: /tmp/baseline_test.log, /tmp/main_diag_test.log, /tmp/diag_timeout.log, +/tmp/diag_120s.log, /tmp/diag_sampling_run.log, /tmp/noagents_run.log, /tmp/final_test.log. \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/meta.yaml b/.investigations/jfr-mcp-diagnose-deadline/meta.yaml new file mode 100644 index 00000000..1780a050 --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/meta.yaml @@ -0,0 +1,13 @@ +name: jfr-mcp-diagnose-deadline +description: > + McpJfrTransportTest.jfrDiagnoseReturnsReport failed on every run with no response within the + 15s harness deadline. Investigation established it was slow-not-hung (jfr_diagnose depth=full + re-parses the whole recording ~10-12x), ruled heap/GC/logging out with data, measured the + agent tax, and fixed it by raising the default deadline to 60s + forwarding the advertised + -Dmcp.test.timeout.ms. Parser perf cost profile left as an open follow-up question. +status: done +created: 2026-10-03 +updated: 2026-10-03 +completed: 2026-10-03 +last_commit: 292056b8350f1204a73bcad16d350650883d9005 +last_session: jafar-mcp-guidance-agent-workflows \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-env-classification.md b/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-env-classification.md new file mode 100644 index 00000000..2a43ae1a --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-env-classification.md @@ -0,0 +1,21 @@ +--- +id: dead-env-classification +type: deadend +depends_on: [ev-run-timings] +status: refuted +created: 2026-10-03 +updated: 2026-10-03 +--- + +# 'Environmental/flaky' — refuted by measurement + +The initial classification (recorded in the session report) treated the failure as a +pre-existing environmental issue with an honest NOT-VERIFIED note. The user pushed to +investigate; three cheap probes refuted it: +- A raised deadline made the SAME runs pass (3/3) — a stable failure, not a race. +- Pristine origin/main reproduced it identically — pre-existing, yes, but not environmental. +- Method profiling + an in-executor parse bench located the cost inside the tool's own + re-parse fan-out — a load-independent floor of ~6–10s that only *crosses* the 15s deadline + when the box is loaded. + +Lesson: 'environmental' means proven environmental; otherwise it is an unmeasured boundary. \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-heap-gc-logging.md b/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-heap-gc-logging.md new file mode 100644 index 00000000..91ed9d3f --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/nodes/dead-heap-gc-logging.md @@ -0,0 +1,24 @@ +--- +id: dead-heap-gc-logging +type: deadend +depends_on: [ev-jfr-profile, ev-run-timings] +status: refuted +tags: [heap, gc, logging] +created: 2026-10-03 +updated: 2026-10-03 +--- + +# Heap, GC and logging causes — refuted with data + +- Heap: `-Xmx8g` ablation produced identical failing timings (18.4s testcase at the 15s + deadline) vs the 2g baseline (17.9s) — no meaningful change (ev-run-timings). +- GC: JFR `profile` run of the focused class recorded 6 collections in 45s, longest pause + 26.3ms — nowhere near the seconds-scale cost being explained (ev-jfr-profile). +- Logging: one shared `jfr-mcp/src/main/resources/logback.xml` serves BOTH the in-process test + classpath and the shadow-jar child; `io.jafar.parser` is WARN in both, `io.jafar.mcp` DEBUG + (the DEBUG lines are tiny single lines per JSON-RPC message). A per-event DEBUG theory would + also have to explain why the child JVM is fast — it isn't per-event (ev-jfr-profile, + harness config). + +Each probe was one command; together they closed the space so the parser-attribution evidence +could carry the conclusion alone. \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/nodes/find-deadline-vs-real-cost.md b/.investigations/jfr-mcp-diagnose-deadline/nodes/find-deadline-vs-real-cost.md new file mode 100644 index 00000000..a12b6b53 --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/nodes/find-deadline-vs-real-cost.md @@ -0,0 +1,37 @@ +--- +id: find-deadline-vs-real-cost +type: finding +depends_on: [ev-run-timings, ev-parse-bench, ev-jfr-profile] +status: confirmed +tags: [jfr-mcp, diagnose, timeout, e2e, fix] +created: 2026-10-03 +updated: 2026-10-03 +--- + +# diagnose(full) response cost vs the 15s harness deadline + +## Reasoning chain +1. Failure signature: harness returns null on deadline — identical on branch tip AND pristine + origin/main (ev-run-timings) ⇒ not branch-induced. +2. Raised-deadline runs always completed 13–21s and were never above 30s ⇒ slow-not-hung. +3. Heap ablation (8 GB) unchanged ⇒ not memory; JFR GC view (6 collections, ≤26 ms pauses) ⇒ not GC; + shared logback config with parser at WARN both contexts ⇒ not logging overhead. +4. Method profile: cycles concentrate in raw parser internals (MetadataField.hasConstantPool 695, + readVarint 650, SWAR 278) — i.e. the recording being re-parsed, repeatedly (ev-jfr-profile). +5. In-executor bench: one warm full untyped parse 0.45–0.73 s (ev-parse-bench); diagnose(full) + = summary + exceptions + GC + hotmethods + USE(~4 queries) + TSA(~4 queries) ⇒ ~10–12 + parse-equivalents ⇒ 6–10 s ideal, 13–21 s with agents+load. +6. The 15 s deadline is below the tool's real cost ⇒ deterministic failure under load, flake + elsewhere. Sibling tools (use 4.5–8.1 s, tsa 2.3–3.2 s) fit under it; only diagnose crossed. + +## Fix (landed in 259a0da) +- McpTransportHarness.RESPONSE_TIMEOUT_MS 15_000 → 60_000 (3× worst measured; a hang still fails). +- :jfr-mcp:test forwards -Dmcp.test.timeout.ms (advertised in the harness failure message, until + now silently ignored by the build). +- Two-sided R5: 2000 ms → the named test FAILS again; 60 s default → class and full module green. + +## What this rules out +- 'Flaky/environmental' classification without measurement — the boundary flapped only because + load moved the tool's cost across the deadline; the tool itself was behaving correctly. +- Any production regression from #119/#120 in the diagnose path — the child-JVM e2e workflow + (fresh process, no agents) runs the same full diagnose inside a 10 s total. \ No newline at end of file diff --git a/.investigations/jfr-mcp-diagnose-deadline/nodes/q-parser-perf.md b/.investigations/jfr-mcp-diagnose-deadline/nodes/q-parser-perf.md new file mode 100644 index 00000000..4d3bac5b --- /dev/null +++ b/.investigations/jfr-mcp-diagnose-deadline/nodes/q-parser-perf.md @@ -0,0 +1,30 @@ +--- +id: q-parser-perf +type: question +status: open +related: [find-deadline-vs-real-cost, ev-parse-bench, ev-jfr-profile] +tags: [parser, perf, follow-up] +created: 2026-10-03 +updated: 2026-10-03 +--- + +# Parser/analysis cost follow-ups (data collected, redesign out of scope here) + +Open threads a future perf effort owns, with numbers already on hand: + +1. **`MetadataField.hasConstantPool` ≈ 15% of CPU** (695 of 4.4k samples in 45s) — metadata + lookups are hot in the parse loop; worth a look at caching the answer or restructuring the + call site (ev-jfr-profile). +2. **Per-parse executor churn**: each `JfrPathEvaluator` evaluation constructs a fresh + `UntypedJafarParser`/`StreamingChunkParser` with a fixed pool of + `availableProcessors()-2` (12 cores ⇒ 10 threads) unnamed daemon threads (factory in + `StreamingChunkParser` line ~29 does not name them; pools only tear down at session + `close()`). In long-lived MCP sessions that re-query, pools accumulate idle. Candidates: + shared/pooled executor + named threads (the repo's thread naming conventions), or a + reusable parser per session. +3. **Agent tax on every test JVM** (jacoco + mockito inline agents: 15–25% at parser, + 15–40% at tool level, paired measurements). CI pays this on the whole suite; consider + coverage scope trims if CI time matters (ev-parse-bench). + +None of these are correctness bugs and none block the e2e/eval gate; they are the follow-up +cost story the diagnose-deadline fix bought headroom for. \ No newline at end of file diff --git a/doc/agents/Verification.md b/doc/agents/Verification.md index ac729534..6099d857 100644 --- a/doc/agents/Verification.md +++ b/doc/agents/Verification.md @@ -145,6 +145,22 @@ diff /tmp/fail_base.txt /tmp/fail_now.txt && echo "zero new failures" Record the baseline **before** you start changing code. Report both numbers — total tests and the named failing set — so a reader can tell growth from regression. +> **Case file — the diagnose deadline between green and failure.** +> `McpJfrTransportTest.jfrDiagnoseReturnsReport` failed on every run while every sibling passed. +> Not flaky, not environmental, not a hang: `jfr_diagnose` (default `depth=full`, +> `includeAnalysis=true`) legitimately re-parses the whole recording for each of its six +> sub-analyses. A warm full untyped parse of the 2.2 MB test recording measures 0.45–0.73 s +> (0.53–0.73 s with the coverage/mocking agents attached — their probes show up directly in the +> profile as `$jacocoInit` in the parse loop; agents cost 15–40 % measured paired), and diagnose +> makes ~10–12 such passes: 13–21 s under load, versus a 15 s harness deadline. Heap (unchanged at +> 8 GB), GC (longest pause 26 ms) and logging (parser at WARN in both contexts) were ruled out with +> data before anything was touched; three green runs under a raised deadline settled slow-not-hung. +> The fix was one number (deadline 15 s → 60 s). Two habits to keep: **classify a slow test by +> measuring its actual response time under a raised deadline before calling it flaky** — three +> measurements beat three hypotheses — and **when a harness advertises an override in its own +> failure message (`-Dmcp.test.timeout.ms`, "raise if the machine is loaded"), forward it from the +> Gradle task**: advice that the build silently ignores is R3's fallback in disguise. + ## R7. One source of truth for any list two places must agree on If a list is duplicated, the copies will disagree, and the disagreement will be invisible until a diff --git a/doc/cli/JFRPath.md b/doc/cli/JFRPath.md index 0dacf58c..263103da 100644 --- a/doc/cli/JFRPath.md +++ b/doc/cli/JFRPath.md @@ -241,16 +241,18 @@ For list and array fields, control how filters apply to elements: **Syntax**: ``` -[any:[condition]] -[all:[condition]] -[none:[condition]] +[any: ] +[all: ] +[none: ] ``` +The condition applies when the field path resolves to a list; each element is tested in turn. + **Examples**: ``` -events/jdk.ExecutionSample[any:stackTrace/frames[matches(method/name/string, ".*Foo.*")]] -events/jdk.ExecutionSample[all:stackTrace/frames[lineNumber>0]] -events/jdk.ExecutionSample[none:stackTrace/frames[matches(method/name/string, ".*Test.*")]] +events/jdk.ExecutionSample[any:stackTrace/frames/method/name/string ~ ".*Foo.*"] +events/jdk.ExecutionSample[all:stackTrace/frames/lineNumber > 0] +events/jdk.ExecutionSample[none:stackTrace/frames/method/name/string ~ ".*Test.*"] ``` ### Interleaved Filters diff --git a/doc/plans/mcp-guidance-contract-and-agent-workflows-implementation.md b/doc/plans/mcp-guidance-contract-and-agent-workflows-implementation.md new file mode 100644 index 00000000..ef0295f8 --- /dev/null +++ b/doc/plans/mcp-guidance-contract-and-agent-workflows-implementation.md @@ -0,0 +1,518 @@ +# MCP guidance contract and agent workflows — implementation plan + +Status: implementation plan for the approved design in +[`mcp-guidance-contract-and-agent-workflows.md`](mcp-guidance-contract-and-agent-workflows.md). + +This is an execution plan, not a replacement for the approved design. Keep the design document +as the historical proposal; record implementation decisions and verification results in the +commit/PR. + +## Scope and boundaries + +Implement the approved work in two independently shippable gates: + +1. **Tier 0:** make the registered MCP guidance a reviewed, reproducible contract. +2. **Tier 1:** prove a real, locally built `--stdio` server can take one useful agent workflow for + JFR, HPROF, pprof, and OTLP inputs. + +Do not rename MCP tools, change their public input schemas, add a live LLM, call a hosted service, +test HTTP/SSE, or modify `btraceio/jafar-perf-box` in this repository. Tier 1 is the local shadow +JAR path only. The optional JBang path in the existing test remains a separately reported release +compatibility check, never a prerequisite for this change. + +The existing uncommitted guidance files and MCP source edits are the starting state. Do not discard +or overwrite unrelated work while implementing this plan. + +## Pre-flight: establish the baseline and the test net + +1. Inspect the current diff before editing. The expected Tier 0 surface is: + + - `jfr-mcp/src/test/java/io/jafar/mcp/GuidanceSurfaces.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/QueryExamples.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/GuidanceExampleQueriesTest.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/GuidanceToolReferencesTest.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/ToolDescriptionQualityTest.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/McpContractJson.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/McpContractExport.java` + - `jfr-mcp/src/test/java/io/jafar/mcp/ToolCatalogSnapshotTest.java` + - the relevant live guidance text, currently `JfrHelpProvider` and any changed family tool + descriptions under `jfr-mcp/src/main/java/io/jafar/mcp/{jfr,hdump,pprof,otlp}/`. + +2. Capture named failing tests before a fix. Do not compare only test counts: preserve the failing + test names and their messages. The first focused command is: + + ```bash + ./gradlew :jfr-mcp:test \ + --tests io.jafar.mcp.ToolCatalogSnapshotTest \ + --tests io.jafar.mcp.GuidanceExampleQueriesTest \ + --tests io.jafar.mcp.GuidanceToolReferencesTest \ + --tests io.jafar.mcp.ToolDescriptionQualityTest + ``` + + Before relying on this as a fast unit/module command, make the Gradle selection explicit: + `test.useJUnitPlatform { excludeTags 'e2e' }`. The tagged subprocess tests must never run from + ordinary `:jfr-mcp:test`, and ordinary tests must not require `shadowJar` as an incidental + dependency. The dedicated `endToEndTest` task is the sole local-test route that depends on + `shadowJar` and uses `includeTags 'e2e'`. + +3. If this command fails outside the known two contract gaps, stop Tier 0. Classify the unexpected + failures by name and repair the net/design discrepancy before adding Tier 1. Do not paper over a + parser failure by excluding an example from extraction, or paper over an unknown tool/resource + by relaxing the scanner. + +4. Confirm the Tier 1 starting net and wire locations before moving fixtures: + + - `jfr-mcp/src/test/java/io/jafar/mcp/McpEndToEndTest.java` is the current spawned-process + regression test and contains the JSON-RPC reader/handshake discipline to preserve. + - `jfr-mcp/src/test/java/io/jafar/mcp/McpTransportHarness.java` is in-process transport + coverage. It is not proof of the packaged process path. + - `hdump-parser/src/test/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java` is the + existing HPROF writer to promote; it must not be copied into `jfr-mcp`. + - `pprof-parser/src/testFixtures/java/io/jafar/pprof/MinimalPprofBuilder.java` and + `otlp-parser/src/testFixtures/java/io/jafar/otlp/MinimalOtlpBuilder.java` are the existing + deterministic profile writers to consume. + +Stop and update the plan/implementation approach if an actual tool requires a capability not in +the approved flow (for example a new required request field or an unavoidable external process). + +## Gate 1 — Tier 0 guidance contract + +### 1. Complete the live guidance before snapshotting it + +Files: + +- `jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrHelpProvider.java` +- any already-changed tool-description source in `JfrSessionTools`, `HdumpTools`, `PprofTools`, or + `OtlpTools` +- `doc/cli/JFRPath.md`, if the wording/query examples changed there must be aligned with the live + server's documentation contract. + +Actions: + +1. Update the actual JFR tool-selection help rendered by `JfrHelpProvider.getToolsHelp()` so + `jafar://help/tools` explicitly introduces `jfr_open` as the JFR entry point. The assertion in + `GuidanceToolReferencesTest.helpTextsMentionTheToolsTheyTeach` is intentionally precise; do not + weaken it or point it at metadata. +2. Keep each documented example syntactically accepted by its real parser. Correct source text or + schema descriptions, not the test's candidate extractor, when a shipped example is wrong. +3. Make the `jfr_*`, `hdump_*`, `pprof_*`, and `otlp_*` scanner resolve names against the live + `JafarMcpServer.createToolSpecifications()` list. Resource URI checks must resolve against the + live registered resource handlers, not a duplicated URI list. + +Gate command: + +```bash +./gradlew :jfr-mcp:test \ + --tests io.jafar.mcp.GuidanceExampleQueriesTest \ + --tests io.jafar.mcp.GuidanceToolReferencesTest \ + --tests io.jafar.mcp.ToolDescriptionQualityTest +``` + +Pass condition: all named tests pass and the output states a substantial parsed example corpus +(`GuidanceExampleQueriesTest` currently guards this at more than 200 examples). + +R5 proof: temporarily remove `jfr_open` from the rendered tools-help source (or make its returned +text omit that token) and run only +`--tests io.jafar.mcp.GuidanceToolReferencesTest`. It must fail the +`helpTextsMentionTheToolsTheyTeach` test. Restore the text and rerun it green. Also temporarily +change one known query example to an invalid expression and observe the named +`GuidanceExampleQueriesTest` failure; restore it immediately. Record both named failures in the +commit/PR, not merely that the test was run. + +Stop condition: if the parser rejects an example whose intended syntax is unclear, do not silently +delete it. Resolve whether the parser or documentation is the contract, then make the chosen +behavior explicit in the source/help text and its test. + +### 2. Make the snapshot and exported contract the same semantic artifact + +Files: + +- `jfr-mcp/src/test/java/io/jafar/mcp/GuidanceSurfaces.java` +- `jfr-mcp/src/test/java/io/jafar/mcp/McpContractJson.java` +- `jfr-mcp/src/test/java/io/jafar/mcp/McpContractExport.java` +- `jfr-mcp/src/test/java/io/jafar/mcp/ToolCatalogSnapshotTest.java` +- new `jfr-mcp/src/test/resources/mcp-contract.json` +- `jfr-mcp/build.gradle` (task comments/inputs only if needed to keep the contract task accurate) + +Actions: + +1. Add a package-private `GuidanceContractContext` (or equivalently named closeable helper) in + `jfr-mcp/src/test/java/io/jafar/mcp/`. Every snapshot/export rendering must create it *before* + constructing `JafarMcpServer`. It owns a unique test/build directory and sets both + `jafar.mcp.sessions.file=/sessions.json` and `user.home=/home`; it creates a fresh, + empty store, asserts both normalized paths are under its root, restores the original system + properties in `close`, and removes only its own directory. Protect tests that mutate the two + properties with JUnit's system-properties resource lock (or disable parallel execution for this + narrowly scoped context). This is required because `jafar://sessions` is dynamic content, not + static guidance metadata. +2. Keep `GuidanceSurfaces` as the single live-server inventory. It must collect: + - each registered tool's name, description, and real input schema; + - each prompt's name, description, declared arguments, and rendered messages from the registered + prompt handler; and + - each resource's URI, name, description, MIME type, and rendered contents from the registered + read handler. + Render `jafar://sessions` only from the clean context and include that deterministic empty + rendering in the snapshot; do not omit the resource merely because it is dynamic. Preserve + deterministic ordering at JSON rendering time (tool/prompt name and resource URI) so source + registration reordering is not a noisy contract change. +3. Extend `McpContractJson.render` to serialize the rendered prompt text and rendered resource + text, in addition to existing metadata. Treat values as text visible to a client; do not derive + them from duplicated literals. Keep the canonical mapper and parsed-tree comparison so formatting + alone does not change semantic equality. +4. Retain `McpContractExport` as the only external export path. Its `main` must itself enter the + clean `GuidanceContractContext` before rendering, rather than inheriting a developer's process + session state. It writes canonical JSON to a caller-supplied temporary path (or its default + build path); no second serializer and no shell copy step. The Gradle `exportMcpContract` task + may pass an output path, but must not be relied on to clear session state for correctness. +5. Add a focused export-versus-snapshot test, either in `ToolCatalogSnapshotTest` or a dedicated + `McpContractExportTest`. It must call `McpContractExport.main` with a JUnit temporary output + path, read that path, parse it, and compare it with the parsed committed + `src/test/resources/mcp-contract.json`. This proves the task's writer and the committed + baseline, not merely two calls to `McpContractJson.render`. +6. Add `GuidanceContractSessionIsolationTest` (or an equivalently focused test). It must seed a + valid stale `sessions.json` containing a distinctive alias in a candidate prior directory, then + render the snapshot/export via the clean contract context. Assert the rendered + `jafar://sessions` content is the exact empty-session form and does not contain that alias; + assert the context's normalized session/home paths are distinct from and nested under its owned + root. Run the same assertion through `McpContractExport.main`, not just an in-memory renderer. +7. Generate the initial snapshot only after the live guidance guards are green: + + ```bash + ./gradlew :jfr-mcp:test \ + --tests io.jafar.mcp.ToolCatalogSnapshotTest \ + -Dmcp.updateSnapshot=true + ``` + + Review `mcp-contract.json` as a public API diff. It must include all tool schemas, rendered + prompt messages, and resource contents. Never regenerate it blindly during ordinary test runs. +6. Rerun the normal snapshot test without the update switch. Its exact comparison must pass. + +Gate commands: + +```bash +./gradlew :jfr-mcp:test --tests io.jafar.mcp.ToolCatalogSnapshotTest +./gradlew :jfr-mcp:exportMcpContract +``` + +Inspect `jfr-mcp/build/mcp-contract/mcp-contract.json` as parsed JSON, compare it with the +committed snapshot, and confirm that it has rendered prompt/resource fields rather than metadata +only. The test is the authority for semantic equality; a byte diff is an optional readability +check. + +R5 proof: in a disposable working change, alter one rendered resource or prompt message while +leaving its metadata unchanged. `ToolCatalogSnapshotTest` must fail and identify the contract +change. Restore it. Separately break/export a deliberately different temporary payload (or point +the test at a modified temporary output) to prove the export-versus-snapshot assertion fails. +For the dynamic resource guard, deliberately make `GuidanceContractContext` reuse the seeded stale +store (or omit its clean-store override); `GuidanceContractSessionIsolationTest` must fail by +naming the leaked alias in `jafar://sessions`. Restore before proceeding. + +Stop conditions: + +- If the snapshot contains test-machine paths, timestamps, or non-deterministic session state, + stop. The empty `jafar://sessions` resource is required, but must be rendered through the clean + contract context; do not omit it or bless machine-specific JSON. +- If the exporter and snapshot require separate traversal/serialization code, stop and consolidate + through `McpContractExport` plus `McpContractJson` before accepting the baseline. +- If a contract diff changes an MCP tool/schema, consult the separate perf-box repository before + claiming compatibility. Do not claim its skills were updated from this repository unless that + work was actually authorized and completed. + +### 3. Tier 0 completion gate + +Run, in order: + +```bash +./gradlew :jfr-mcp:test \ + --tests io.jafar.mcp.ToolCatalogSnapshotTest \ + --tests io.jafar.mcp.GuidanceExampleQueriesTest \ + --tests io.jafar.mcp.GuidanceToolReferencesTest \ + --tests io.jafar.mcp.ToolDescriptionQualityTest +./gradlew :jfr-mcp:test +./gradlew :jfr-mcp:exportMcpContract +``` + +Do not start Tier 1 until all three commands pass. Report test totals and named failures, if any, +rather than a count-only comparison. If the complete module suite has standing recording-resource +failures, capture and compare the exact failing names against the pre-change baseline; stop for any +new named failure. + +## Gate 2 — promote the HPROF fixture without production leakage + +Files: + +- `hdump-parser/build.gradle` +- move `hdump-parser/src/test/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java` to + `hdump-parser/src/testFixtures/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java` +- HPROF parser tests that import the generator (update imports/source-set dependencies only as + needed) +- `jfr-mcp/build.gradle` + +Actions: + +1. Add the `java-test-fixtures` plugin to `hdump-parser`, matching `pprof-parser` and + `otlp-parser`. +2. Move—not copy—the synthetic generator to `src/testFixtures`. Keep the existing package so its + existing test imports remain stable. Add the JUnit dependency required by any `hdump-parser` + tests that continue to compile; test fixtures themselves must expose only what the generator + actually needs. +3. Exclude `testFixturesApiElements` and `testFixturesRuntimeElements` from Maven publication, as + the pprof/OTLP parser projects do, so the synthetic writer is a build-time testing capability, + not a production artifact. +4. Add `testImplementation(testFixtures(project(':hdump-parser')))` in `jfr-mcp/build.gradle`. + Do not add a custom HPROF writer to `jfr-mcp`. +5. Make a minimal fixture whose data is distinguishable from an empty/stale session. If the + existing minimal generator cannot encode a distinctive class/object count needed by the report + and query assertions, extend the promoted fixture once with named, deterministic content (for + example a unique class name and known instance count), then preserve its existing parser-test + methods. + +Gate commands: + +```bash +./gradlew :hdump-parser:test +./gradlew :jfr-mcp:test --tests io.jafar.mcp.HdumpHandlerTest +``` + +R5 proof: temporarily remove the `testFixtures(project(':hdump-parser'))` dependency or rename the +fixture import. The MCP E2E test compilation must fail, proving it consumes the promoted fixture; +restore it before recording a green test. Also change its expected distinctive value in the new E2E +test and observe the named test fail after the E2E tests have been added. + +Stop condition: if moving the fixture would force `hdump-parser` production code to depend on test +APIs or would publish fixture classes, stop and correct Gradle source-set/publication wiring first. + +## Gate 3 — Tier 1 subprocess workflow harness + +Files: + +- refactor/add under `jfr-mcp/src/test/java/io/jafar/mcp/`, preferably: + - migrate `McpEndToEndTest.java` to the shared child-process harness for its existing + progress-token regressions (it must no longer call its private `startServer` directly); + - add `McpAgentWorkflowEndToEndTest.java` for the four workflows; and + - add a package-private `McpStdioProcessHarness.java` for the shared local-JAR command, + JSON-RPC protocol, progress-notification collection, and owned-directory cleanup. +- `jfr-mcp/build.gradle` to explicitly exclude `e2e` from `test`, explicitly include it in + `endToEndTest`, and make only `endToEndTest` depend on `shadowJar`. + +Do not turn `McpTransportHarness` into the Tier 1 proof: it constructs `JafarMcpServer` in the test +JVM and cannot prove shadow-JAR command line, manifest, stdio, or child-session isolation. + +The normal tagged E2E suite uses only the locally built shadow JAR. Remove the existing +`mcp.e2e.use.jbang` branch from `McpEndToEndTest` and from the `endToEndTest` task's forwarded +properties, rather than leaving a branch that can bypass the required child JVM properties. A +published/JBang check, if separately desired later, is manual/release coverage outside this task +and must have its own documented isolation strategy. + +Selection proof (run after changing `jfr-mcp/build.gradle`): + +```bash +./gradlew :jfr-mcp:test --info +./gradlew :jfr-mcp:endToEndTest --tests io.jafar.mcp.McpEndToEndTest --info +``` + +The first invocation must show neither `McpEndToEndTest`/`McpAgentWorkflowEndToEndTest` execution +nor `:jfr-mcp:shadowJar`; the second must select the tagged existing class and run through +`:jfr-mcp:shadowJar`. Add an explicit Gradle-level assertion or a narrowly scoped task-selection +test if the log alone cannot establish this reliably in CI. R5: temporarily remove +`excludeTags 'e2e'` and observe a normal `:jfr-mcp:test` attempt to select the tagged class; then +restore the exclusion and prove it is absent. Separately remove `endToEndTest.dependsOn shadowJar` +in a disposable change and run from a clean `jfr-mcp/build/libs` so the selected E2E test fails for +the missing JAR; restore the dependency. + +### Harness design + +1. For each test launch a fresh local shadow JAR with: + + ```text + + -Djafar.mcp.sessions.file=/sessions.json + -Duser.home=/home + -jar + --stdio + ``` + + `unique-dir` must be created under `jfr-mcp/build/mcp-e2e/`, using a test-name-safe unique + suffix (UUID/counter). The test owns only that directory. The harness must normalize both paths + and assert `sessions.json` and `home` start with that unique directory before it starts the + process. This is a defensive double lock: an ignored explicit session property still points the + server only at the private `user.home`, never at the developer's `~/.jafar`. Both + `McpEndToEndTest` and `McpAgentWorkflowEndToEndTest` must instantiate this same harness; no + legacy child launch is exempt from path ownership, child stderr capture, shutdown, or cleanup. +2. Use JSON-RPC lines over child stdin/stdout. Send `initialize`, assert a successful response, then + send `notifications/initialized`. `tools/call` must send real `arguments`; for analyses that + report progress, attach a numeric `_meta.progressToken` and a stable test tool-use id. The reader + queues every JSON response/notification and waits by request id without losing out-of-order + progress messages. +3. A successful response means: matching JSON-RPC id, no top-level `error`, + `result.isError == false`, and parsed content containing the expected structured JSON. Retain + the full response in every assertion failure (R9). Never use `Process.isAlive()` or exit status + as the only assertion. +4. The message collector must additionally retain and select `notifications/progress` messages + emitted while a request is in flight. For every progress-capable analysis invoked with numeric + token `N`, assert at least one such notification, assert + `params.progressToken.isIntegralNumber()`, and assert `params.progressToken.asInt() == N` for + every notification associated with that call. Include the raw notification in failure output. + This asserts both JSON type and value, rather than merely proving that the server remained + alive after the historic string-token disconnect. +5. Shut down deterministically: send family close calls in each workflow, close stdin, allow a + bounded graceful wait, then forcibly destroy only the test-owned child as a cleanup fallback. + Stop the reader thread and retain stderr in a bounded buffer rather than discarding it, so an + E2E failure names the child error. In `finally`, recursively delete the exact unique directory + and assert `Files.notExists(uniqueDir)`. If cleanup fails, fail the test and leave the resolved + path in its message for inspection. Never recursively delete the broader `build/mcp-e2e` root. +6. Assert isolation behavior explicitly: each launch has a different canonical session path and + home; the server starts with no prior session; no configured path escapes its directory; and the + directory exists only while that harness is active. A second process/workflow must not see the + first process's aliases or sessions. Prove this with an initial `jafar://sessions` resource read + (or the equivalent live session listing) and a fresh-process assertion before any open call. +7. The harness needs no external service, downloaded recording, user state, API key, or network + access. If a test needs any of these, stop; construct a deterministic fixture instead. + +## Gate 4 — implement the four black-box workflows + +Put these in `McpAgentWorkflowEndToEndTest`, tagged `e2e`, so the existing +`./gradlew :jfr-mcp:endToEndTest` task runs them after `shadowJar`. + +### JFR workflow + +Fixture: the small repository-owned `demo/src/test/resources/test-dd.jfr` already selected by +`McpEndToEndTest.findTestJfr()` (with its existing explicit override only for local reproduction). + +Sequence: + +1. `jfr_open` with absolute path and alias; assert successful result has a numeric/id-like session + identifier and the expected path/alias evidence. +2. `jfr_summary`; assert successful structured recording summary, not just a success message. +3. `jfr_query` with a known-valid query suited to the fixture; assert the result fields the next + step uses and non-empty/fixture-valid evidence. Do not assert a fragile global count. +4. Call one guidance-recommended analysis (`jfr_diagnose` is preferred because it exercises numeric + progress token delivery; use `jfr_use` only if the small fixture cannot support diagnose's + contract). Attach a numeric progress token (for example `4`). Assert findings/content has the + documented shape, not prose alone, and assert every matching `notifications/progress` payload + carries integral token value `4` exactly. +5. `jfr_close`; then issue a further safe request (for example `jfr_help` or resource read) and + assert it receives a response, proving the child still accepts requests after analysis/close. + +### HPROF workflow + +Fixture: create it in the unique test directory with the promoted +`io.jafar.hdump.test.SyntheticHeapDumpGenerator`, using distinctive class/object content and a +known count. + +Sequence: + +1. `hdump_open` with alias; assert id/path/alias returned. +2. `hdump_summary`; assert the expected heap/class/object evidence for the synthetic dump. +3. `hdump_query` using a query accepted by HdumpPath and capable of returning the distinctive + fixture data; inspect `resultCount` and returned rows, not a prose success message. +4. `hdump_report`; assert a successful structured `findings` array or documented report fields and + at least one fixture-derived fact. If a deliberately minimal heap legitimately has no findings, + assert its stable structured empty result plus the summary/query evidence; do not manufacture a + fake leak merely to obtain a finding. +5. `hdump_close`, then a subsequent request and response as above. + +### pprof workflow + +Fixture: write a profile into the unique directory with `MinimalPprofBuilder`: choose a distinctive +function name (for example `tier1.pprof.UniqueCpuMethod`), a known sample type and value, plus a +label/query field supported by the parser. The exact query must be accepted by PprofPath before it +is used in the subprocess test. + +Sequence: + +1. `pprof_open`; assert session/profile metadata. +2. `pprof_summary`; assert known profile/sample evidence. +3. `pprof_query`; assert rows include the distinctive function/sample evidence and expected count + or value. +4. `pprof_use` (or the documented supported alternative only if the synthetic profile cannot + produce a meaningful USE response); attach a numeric progress token and assert successful + structured analysis, fixture-derived evidence where the tool's contract supplies it, and exact + integral token propagation in all progress notifications. +5. `pprof_close` and a later successful request. + +### OTLP workflow + +Fixture: write an OTLP profile into the unique directory with `MinimalOtlpBuilder`, with a unique +function/attribute and known sample value; do not reimplement protobuf encoding in `jfr-mcp`. + +Sequence: + +1. `otlp_open`; assert session/profile metadata. +2. `otlp_summary`; assert the known fixture characteristics. +3. `otlp_query`; assert the unique function/sample/attribute survives the actual stdio JSON. +4. `otlp_use`; attach a numeric progress token and assert successful structured analysis with + fixture-derived evidence where applicable plus exact integral token propagation in every + progress notification. +5. `otlp_close` followed by another response. + +For every workflow, capture and assert the actual decoded `result.content` JSON and relevant +`findings`, rows, session id, or summary fields. Payload assertions must distinguish the created +fixture from an empty fallback, a stale prior session, and a response from the wrong format family. +If the production response serializes JSON as text content, parse that text before asserting; +asserting only the surrounding MCP envelope is insufficient. + +R5 proof for the new safety net: + +1. Temporarily change each fixture's distinctive function/class/value while retaining the test's + old expected value. Run its named workflow test and observe a failure that identifies the + payload assertion; restore it. One proof per fixture family is required. +2. Temporarily remove the post-analysis/follow-up request or make the server-side analysis response + fail in a disposable change; the named JFR workflow must fail. Restore it. +3. Temporarily change the server-side progress token to a string or alter the harness expectation + from the sent integer (`4`) in a disposable change. The migrated `McpEndToEndTest` must fail on + the notification's token type/value assertion, not only because a process exits or a response + is missing. Restore it before proceeding. +4. Temporarily force the harness session path outside its unique directory. Its pre-launch path + assertion must fail before launching a child. Restore it. This is the isolation R5 proof. + +Stop conditions: + +- A workflow is not complete if it uses an in-process server, a mocked handler, a test-only MCP + registration, or a second protocol implementation that does not run `java -jar ... --stdio`. +- A test that passes with a synthetic file but does not inspect its distinctive data is not complete. +- A cleanup failure, path outside `build/mcp-e2e`, child process leak, or cross-workflow session + visibility is a Tier 1 blocker, not a tolerated test flake. +- If an analysis cannot operate on a minimal valid fixture, document the specific contract reason + and select the approved documented alternative; do not convert the workflow to per-tool smoke + testing or introduce a downloaded large recording. + +## Gate 5 — full validation and handoff + +Run formatting first, inspect its diff, then validate in this order: + +```bash +./gradlew spotlessApply +./gradlew spotlessCheck +./gradlew :hdump-parser:test +./gradlew :jfr-mcp:test \ + --tests io.jafar.mcp.ToolCatalogSnapshotTest \ + --tests io.jafar.mcp.GuidanceExampleQueriesTest \ + --tests io.jafar.mcp.GuidanceToolReferencesTest \ + --tests io.jafar.mcp.ToolDescriptionQualityTest +./gradlew :jfr-mcp:test +./gradlew :jfr-mcp:exportMcpContract +./gradlew :jfr-mcp:endToEndTest +``` + +Finally inspect: + +1. `jfr-mcp/build/mcp-contract/mcp-contract.json` parsed against the committed snapshot; +2. the exact subprocess command/isolated paths and captured JSON payloads for all four workflows; +3. `jfr-mcp/build/mcp-e2e/` for no leftover per-test directories; and +4. `git diff --check` plus the final diff, confirming no generated build file or user session file + is staged. + +If the broader repository-required release check is desired before PR, run the documented +`./gradlew test shadowJar` after the gates above. Report separately any tests not run and any +pre-existing downloaded-recording failure set by name. Do not describe the JBang/published path as +verified unless `-Dmcp.e2e.use.jbang=true` was deliberately run with a known published version. + +The commit/PR must state: + +- the Tier 0 snapshot fields now guarded and the explicit snapshot review mechanism; +- the four Tier 1 workflows and their fixture-specific payload evidence; +- the isolated child session/home and cleanup proof; +- each R5 temporary-break experiment and the named failing test it produced; and +- anything not verified (especially the optional published JBang route or unavailable large + recordings). diff --git a/doc/plans/mcp-guidance-contract-and-agent-workflows.md b/doc/plans/mcp-guidance-contract-and-agent-workflows.md new file mode 100644 index 00000000..e98db8cb --- /dev/null +++ b/doc/plans/mcp-guidance-contract-and-agent-workflows.md @@ -0,0 +1,135 @@ +# MCP guidance contract and scripted agent workflows + +Status: approved scope — Tier 0 completion and Tier 1 implementation. + +This plan closes the gap between a server that compiles and a client that can follow its guidance +through a complete analysis. It deliberately builds on the current `jfr-mcp` contract tests and +subprocess E2E task; it does not redesign the MCP tools or add an LLM. + +## Scope + +### Tier 0 — guidance is an executable contract + +Complete and validate the inherited guidance checks: + +- Commit `jfr-mcp/src/test/resources/mcp-contract.json`, rendered by the existing + `McpContractJson` / `McpContractExport` path. It is the reviewed baseline for registered tool + names, descriptions, input schemas, prompt definitions **and rendered prompt messages**, and + resource metadata **and rendered resource content**. The rendered text is part of the contract: + it is what an MCP client can actually follow, whereas metadata alone cannot expose stale playbook + instructions. +- Make the `jafar://help/tools` assertion true by ensuring the resource's actual tool-selection + text names the JFR entry point it teaches (`jfr_open`). Do not weaken the assertion merely to + make the test pass. +- Keep the four existing guards as the contract boundary: every documented query parses with its + real parser; every mentioned tool and resource exists; opening tools identify only their own + format; and the snapshot changes only after an explicit review. +- Preserve `exportMcpContract` as the single export path for external consumers such as + `btraceio/jafar-perf-box`. Its output and the committed snapshot must represent the same + live-server wiring. Add a test that writes a temporary export through `McpContractExport` and + semantically compares it with the committed snapshot; do not merely compare two calls to the + renderer. + +### Tier 1 — scripted ideal-agent workflows + +Add black-box tests that start the locally built shaded JAR with `--stdio`, perform MCP +initialization, and drive the shortest useful workflow for each supported input family: + +| Family | Scripted workflow | Required evidence | +|---|---|---| +| JFR | `jfr_open` -> `jfr_summary` -> `jfr_query` -> a guidance-recommended analysis -> `jfr_close` | Responses are successful, the result JSON has the fields the next step needs, and the process accepts a subsequent request. | +| HPROF | `hdump_open` -> `hdump_summary` -> `hdump_query` -> `hdump_report` -> `hdump_close` | A generated minimal valid dump is opened, query/report results have usable JSON content, and close succeeds. | +| pprof | `pprof_open` -> `pprof_summary` -> `pprof_query` -> `pprof_use` (or the documented alternative) -> `pprof_close` | The synthetic profile's known sample/method evidence survives the transport. | +| OTLP | `otlp_open` -> `otlp_summary` -> `otlp_query` -> `otlp_use` -> `otlp_close` | The synthetic profile's known sample/evidence survives the transport. | + +These are *agent workflows*, not one-test-per-tool coverage. Each must use the same JSON-RPC +requests a client sends, including the initialized notification and, where an analysis reports +progress, a numeric `_meta.progressToken`. Assertions inspect the response payload and confirm +that a later call still receives a response; an exit code alone is not evidence. + +Use deterministic, repository-owned fixtures only: the existing small JFR recording and minimal +pprof/OTLP files constructed during the test. For HPROF, promote the existing +`hdump-parser` `SyntheticHeapDumpGenerator` to its `java-test-fixtures` source set and consume it +from `jfr-mcp` with Gradle's `testFixtures(project(":hdump-parser"))`; do not add a second HPROF +encoder to `jfr-mcp`. Give fixture content distinctive names and known counts so assertions +establish that the opened file, rather than an empty fallback or stale session, produced the +result. + +Each launched child JVM receives a fresh, test-owned session file, for example +`-Djafar.mcp.sessions.file=/sessions.json`, and an isolated +`-Duser.home` under that same unique build directory. The subprocess harness creates the directory, +passes both properties before `-jar`, and removes it after shutdown. It asserts that the configured +session path is under the test's build directory, that no state is inherited from a prior workflow, +and that cleanup succeeded. This makes access to a real `~/.jafar` session file impossible even if +the explicit session property is accidentally ignored. + +## Affected components + +- `jfr-mcp` guidance sources, contract snapshot, and guidance tests (`GuidanceSurfaces`, + `ToolCatalogSnapshotTest`, `GuidanceExampleQueriesTest`, `GuidanceToolReferencesTest`, and + `ToolDescriptionQualityTest`). `McpContractJson` and `McpContractExport` expand to serialize + rendered prompt/resource text and gain an export-versus-snapshot test. +- `jfr-mcp` E2E test support and Gradle task. Reuse the request/response discipline in + `McpEndToEndTest`; factor shared subprocess JSON-RPC support only if it prevents divergent + protocol handling. The harness owns a unique child-home/session directory per process and proves + it is removed. The existing in-process `McpTransportHarness` remains transport-level coverage, + not Tier 1 proof. +- `hdump-parser` adds the `java-test-fixtures` capability and moves its existing synthetic HPROF + generator there; `jfr-mcp` consumes that fixture capability. Production tool APIs, tool names, + schemas, prompts, and runtime dependencies are not changed by Tier 1. + +## Compatibility and non-goals + +The public MCP surface remains backward compatible: no tool renames, parameter changes, response +shape changes, or new required client capabilities. A Tier 0 snapshot update is intentional only +when a reviewed guidance contract change warrants it; it is not regenerated blindly. + +Out of scope: hosted-model or agent evaluation, `btraceio/jafar-perf-box` changes, live JVM +capture, performance/load testing, HTTP/SSE coverage, broad handler refactors, and exhaustive +per-tool integration coverage. Tier 1 validates the normal stdio route a guided agent needs; the +existing unit and transport suites retain their respective roles. + +## Acceptance criteria + +1. `:jfr-mcp:test` passes with the committed contract snapshot present; changing a tool, + parameter, prompt (including rendered message), or resource (including rendered content) fails + `ToolCatalogSnapshotTest` until the reviewed update switch is used. +2. Every shipped guidance example is parsed by the matching JfrPath, HdumpPath, PprofPath, or + OtlpPath parser, and every named tool/resource resolves from the live server registrations. +3. `:jfr-mcp:exportMcpContract` writes the same logical JSON as the committed snapshot, proved by + a test that invokes the exporter to a temporary path and compares the parsed output. +4. A dedicated Tier 1 invocation builds the shadow JAR and passes all four complete workflows + against a newly launched server process. Each workflow fails if its expected tool call or + payload assertion is removed or deliberately broken (R5). +5. Every Tier 1 child process receives a distinct build-directory session path and isolated home; + the test asserts this configuration and removes its directory after shutdown. The task is + deterministic, requires no downloaded large recording, no running external service, and no paid + credential, and never reads or writes the user's MCP session state. + +## Verification + +Run the focused Tier 0 tests first, then the complete fast module suite and the subprocess tests: + +```bash +./gradlew :jfr-mcp:test --tests ToolCatalogSnapshotTest --tests GuidanceExampleQueriesTest --tests GuidanceToolReferencesTest --tests ToolDescriptionQualityTest +./gradlew :jfr-mcp:test +./gradlew :jfr-mcp:exportMcpContract +./gradlew :jfr-mcp:endToEndTest +``` + +For Tier 1, inspect the captured JSON in each assertion: it must show the expected session or +known fixture evidence and `isError: false`. Also inspect the child command/configuration and +assert its generated session path and home are both under `jfr-mcp/build/mcp-e2e/`, then assert +their removal after the process exits. Report any unverified published-JBang path separately; it is +useful release compatibility coverage but is not a prerequisite for local implementation. Before +committing a regression test, temporarily remove the behaviour or assertion it protects and record +the named failing test, then restore it (R5). Run `./gradlew spotlessApply` before final validation. + +## Security and cost boundary + +Tests start only the locally built server and feed it local synthetic/checked-in data. They make no +network calls, do not invoke an LLM, do not require API keys, and cannot access a user's persisted +MCP sessions because every child gets an explicit private session file and private home. File paths +are test-created or repository-owned. The tests must not convert a malformed/open failure into a +successful empty analysis: assertions verify the payload's fixture-specific evidence and actionable +error semantics where failure paths are tested. diff --git a/hdump-parser/build.gradle b/hdump-parser/build.gradle index 7be1fdf6..894b2072 100644 --- a/hdump-parser/build.gradle +++ b/hdump-parser/build.gradle @@ -1,9 +1,14 @@ plugins { id 'java-library' + id 'java-test-fixtures' id 'maven-publish' id 'com.vanniktech.maven.publish' } +// SyntheticHeapDumpGenerator is a build-time fixture, never a published parser capability. +components.java.withVariantsFromConfiguration(configurations.testFixturesApiElements) { skip() } +components.java.withVariantsFromConfiguration(configurations.testFixturesRuntimeElements) { skip() } + def libraryName = "hdump-parser" def component_version = project.hasProperty("jafar_version") ? project.jafar_version : rootProject.version diff --git a/hdump-parser/src/test/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java b/hdump-parser/src/testFixtures/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java similarity index 100% rename from hdump-parser/src/test/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java rename to hdump-parser/src/testFixtures/java/io/jafar/hdump/test/SyntheticHeapDumpGenerator.java diff --git a/jfr-mcp/build.gradle b/jfr-mcp/build.gradle index 2497c436..0eb2e4b4 100644 --- a/jfr-mcp/build.gradle +++ b/jfr-mcp/build.gradle @@ -29,6 +29,7 @@ dependencies { implementation project(':shell-core') implementation project(':parser') implementation project(':hdump-shell') + testImplementation(testFixtures(project(':hdump-parser'))) implementation project(':pprof-parser') implementation project(':pprof-shell') testImplementation(testFixtures(project(':pprof-parser'))) @@ -56,11 +57,19 @@ dependencies { } test { - useJUnitPlatform() + useJUnitPlatform { + excludeTags 'e2e' + } // Allocate sufficient heap for tests that load real JFR files jvmArgs '-Xmx2g', '-Xms512m' // Isolate session persistence from real ~/.jafar data during tests systemProperty 'jafar.mcp.sessions.file', "${System.getProperty('java.io.tmpdir')}/jafar-mcp-test-sessions.json" + // McpTransportHarness advertises -Dmcp.test.timeout.ms to raise its per-response deadline; + // forward it so that advice actually works. + systemProperties System.properties.subMap(['mcp.test.timeout.ms']) + if (System.getProperty('mcp.updateSnapshot') != null) { + systemProperty 'mcp.updateSnapshot', System.getProperty('mcp.updateSnapshot') + } // Stress tests are a separate concept from integration tests; always exclude from default `test`. exclude '**/JafarMcpStressTest.class' @@ -107,7 +116,6 @@ tasks.register('integrationTest', Test) { // End-to-end tests — spawn the MCP server subprocess and drive the real analysis workflow. // Run with: ./gradlew :jfr-mcp:endToEndTest -// To test the published release: ./gradlew :jfr-mcp:endToEndTest -Dmcp.e2e.use.jbang=true // To test a specific JFR file: ./gradlew :jfr-mcp:endToEndTest -Dmcp.e2e.jfr=/path/to/file.jfr tasks.register('endToEndTest', Test) { description = 'Spawns the MCP server as a subprocess and drives the real tool sequence' @@ -120,7 +128,7 @@ tasks.register('endToEndTest', Test) { includeTags 'e2e' } - systemProperties System.properties.subMap(['mcp.e2e.use.jbang', 'mcp.e2e.jbang.cmd', 'mcp.e2e.jfr']) + systemProperties System.properties.subMap(['mcp.e2e.jfr']) jvmArgs '-Xmx512m', '-Xms256m' @@ -138,6 +146,23 @@ tasks.register('endToEndTest', Test) { shouldRunAfter test } +// Writes the MCP guidance contract JSON (tool names, descriptions, input schemas, prompts, +// resources) so out-of-repo consumers can check against the in-development server. The jafar-perf +// plugin (btraceio/agent-plugins) consumes this instead of waiting for a published release. +// Run with: ./gradlew :jfr-mcp:exportMcpContract +// The output is byte-identical to the committed src/test/resources/mcp-contract.json snapshot. +tasks.register('exportMcpContract', JavaExec) { + description = 'Writes the MCP guidance contract JSON for out-of-repo consumers' + group = 'verification' + + classpath = sourceSets.test.runtimeClasspath + mainClass = 'io.jafar.mcp.McpContractExport' + args = [ layout.buildDirectory.file('mcp-contract/mcp-contract.json').get().asFile.absolutePath ] + + dependsOn testClasses + shouldRunAfter test +} + // Stress tests — exercise the full stdio pipeline under load. Exclude from default `test`. tasks.register('stressTest', Test) { description = 'Runs reliability stress tests over the in-process stdio harness' diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/bridge/StdioSseBridge.java b/jfr-mcp/src/main/java/io/jafar/mcp/bridge/StdioSseBridge.java index 34fdcb2f..26a21e79 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/bridge/StdioSseBridge.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/bridge/StdioSseBridge.java @@ -60,6 +60,8 @@ public interface DaemonSupplier { private static final int INTERNAL_ERROR = -32603; private static final int CONNECT_TIMEOUT_MS = 5_000; private static final int POST_TIMEOUT_MS = 30_000; + private static final int DELIVERY_ATTEMPTS = 3; + private static final long RECONNECT_RETRY_DELAY_MS = 25; private final DaemonSupplier daemon; private final InputStream stdin; @@ -168,10 +170,13 @@ private void handleClientLine(String line) { } } - /** Delivers one client message, reconnecting once if the daemon dropped the session. */ + /** + * Delivers one client message, retrying a bounded number of times if the daemon drops the + * session. + */ private synchronized boolean post(String line, String idKey) { IOException last = null; - for (int attempt = 0; attempt < 2; attempt++) { + for (int attempt = 0; attempt < DELIVERY_ATTEMPTS; attempt++) { if (attempt > 0 && idKey != null && !inFlight.containsKey(idKey)) { return false; // already failed to the client while we were reconnecting; don't answer twice } @@ -193,6 +198,15 @@ private synchronized boolean post(String line, String idKey) { if (connection != null) { dropConnection(connection, idKey); } + if (attempt + 1 < DELIVERY_ATTEMPTS) { + try { + Thread.sleep(RECONNECT_RETRY_DELAY_MS); + } catch (InterruptedException interrupted) { + Thread.currentThread().interrupt(); + fail(idKey, "interrupted while reconnecting to the jfr-mcp daemon"); + return false; + } + } } } fail( diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/hdump/HdumpTools.java b/jfr-mcp/src/main/java/io/jafar/mcp/hdump/HdumpTools.java index e0593e08..1ccefa8b 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/hdump/HdumpTools.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/hdump/HdumpTools.java @@ -101,7 +101,7 @@ public McpServerFeatures.SyncToolSpecification createHdumpOpenTool() { return new McpServerFeatures.SyncToolSpecification( buildTool( "hdump_open", - "Opens an HPROF heap dump file for analysis. " + "Opens an HPROF heap dump file (.hprof) for analysis. " + "Returns a session ID used by other hdump_* tools. " + "If no sessionId is supplied to other tools, the most recently opened session is used.", schema), @@ -817,9 +817,6 @@ private String getHdumpExamplesHelp() { ## Leak Detection ``` - # Run all leak detectors - checkLeaks() - # Thread-local leak detection checkLeaks(detector="threadlocal-leak") @@ -828,6 +825,8 @@ private String getHdumpExamplesHelp() { # Graph-based leak clusters clusters | sortBy(score desc) | head(10) + + # All detectors at once: hdump_report runs every analysis and ranks the findings ``` ## Collection Waste diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrHelpProvider.java b/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrHelpProvider.java index 740c273c..b81fecfb 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrHelpProvider.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrHelpProvider.java @@ -98,7 +98,7 @@ public String getFiltersHelp() { - `all:` - All elements match - `none:` - No elements match - Example: `events/jdk.ExecutionSample[any:stackTrace/frames[matches(method/type/name/string, ".*MyClass.*")]]` + Example: `events/jdk.ExecutionSample[any:stackTrace/frames/method/type/name/string ~ ".*MyClass.*"]` """; } @@ -400,6 +400,10 @@ public String getToolsHelp() { return """ # Choosing the Right jfr_* Tool + Start every JFR workflow with `jfr_open`: give it the recording path (and, when useful, an + alias), then pass the returned session id or alias to the tools below. Use `jfr_close` when + the investigation is finished. + ## CPU Profiling Tools ### jfr_hotmethods diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrSessionTools.java b/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrSessionTools.java index 59a7ea45..c2c1cabc 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrSessionTools.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/jfr/JfrSessionTools.java @@ -95,7 +95,7 @@ public McpServerFeatures.SyncToolSpecification createJfrOpenTool() { return new McpServerFeatures.SyncToolSpecification( buildTool( "jfr_open", - "Opens a JFR (Java Flight Recording) file for analysis. " + "Opens a JFR (Java Flight Recording) file (.jfr) for analysis. " + "Returns a session ID that can be used with other jfr_* tools. " + "If no session ID is provided to other tools, they use the most recently opened session.", schema), diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/otlp/OtlpTools.java b/jfr-mcp/src/main/java/io/jafar/mcp/otlp/OtlpTools.java index df88422c..34c4008e 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/otlp/OtlpTools.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/otlp/OtlpTools.java @@ -214,7 +214,7 @@ public McpServerFeatures.SyncToolSpecification createOtlpQueryTool() { "Executes an OtlpPath query against an open OTLP profile. " + "Query syntax: 'samples[predicate] | operator(args)'. " + "Examples: 'samples | count()', 'samples | top(10, cpu)', " - + "'samples | groupBy(thread, sum(cpu))', 'samples[thread=\\'main\\'] | head(5)'. " + + "'samples | groupBy(thread, sum(cpu))', 'samples[thread='main'] | head(5)'. " + "Use otlp_help for full query language reference.", schema), (exchange, args) -> handleOtlpQuery(args.arguments())); diff --git a/jfr-mcp/src/main/java/io/jafar/mcp/pprof/PprofTools.java b/jfr-mcp/src/main/java/io/jafar/mcp/pprof/PprofTools.java index 61b832f6..d79dcf2c 100644 --- a/jfr-mcp/src/main/java/io/jafar/mcp/pprof/PprofTools.java +++ b/jfr-mcp/src/main/java/io/jafar/mcp/pprof/PprofTools.java @@ -215,7 +215,7 @@ public McpServerFeatures.SyncToolSpecification createPprofQueryTool() { "Executes a PprofPath query against an open pprof profile. " + "Query syntax: 'samples[predicate] | operator(args)'. " + "Examples: 'samples | count()', 'samples | top(10, cpu)', " - + "'samples | groupBy(thread, sum(cpu))', 'samples[thread=\\'main\\'] | head(5)'. " + + "'samples | groupBy(thread, sum(cpu))', 'samples[thread='main'] | head(5)'. " + "Use pprof_help for full query language reference.", schema), (exchange, args) -> handlePprofQuery(args.arguments())); diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractContext.java b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractContext.java new file mode 100644 index 00000000..0176ecaf --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractContext.java @@ -0,0 +1,106 @@ +package io.jafar.mcp; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Comparator; + +/** + * Owns the only process state that can make a rendered guidance contract non-deterministic. + * + *

The sessions resource is deliberately part of the public contract. Rendering it against a + * developer's {@code ~/.jafar} state would leak paths and aliases into the snapshot, so callers + * must create this context before constructing {@link JafarMcpServer}. + */ +final class GuidanceContractContext implements AutoCloseable { + + private static final String SESSIONS_PROPERTY = "jafar.mcp.sessions.file"; + + private final Path root; + private final Path sessionsFile; + private final Path home; + private final String priorSessionsFile; + private final String priorUserHome; + + private GuidanceContractContext( + Path root, Path sessionsFile, Path home, String priorSessionsFile, String priorUserHome) { + this.root = root; + this.sessionsFile = sessionsFile; + this.home = home; + this.priorSessionsFile = priorSessionsFile; + this.priorUserHome = priorUserHome; + } + + static GuidanceContractContext create() throws IOException { + Path parent = Path.of("build", "mcp-contract").toAbsolutePath().normalize(); + Files.createDirectories(parent); + Path root = Files.createTempDirectory(parent, "guidance-").toRealPath(); + Path sessionsFile = root.resolve("sessions.json").normalize(); + Path home = root.resolve("home").normalize(); + assertOwned(root, sessionsFile, home); + Files.createDirectories(home); + Files.writeString(sessionsFile, "[]"); + + String priorSessionsFile = System.getProperty(SESSIONS_PROPERTY); + String priorUserHome = System.getProperty("user.home"); + System.setProperty(SESSIONS_PROPERTY, sessionsFile.toString()); + System.setProperty("user.home", home.toString()); + return new GuidanceContractContext(root, sessionsFile, home, priorSessionsFile, priorUserHome); + } + + Path root() { + return root; + } + + Path sessionsFile() { + return sessionsFile; + } + + Path home() { + return home; + } + + static void assertOwned(Path root, Path sessionsFile, Path home) { + Path normalizedRoot = root.toAbsolutePath().normalize(); + Path normalizedSessions = sessionsFile.toAbsolutePath().normalize(); + Path normalizedHome = home.toAbsolutePath().normalize(); + if (!normalizedSessions.startsWith(normalizedRoot) + || !normalizedHome.startsWith(normalizedRoot)) { + throw new IllegalArgumentException( + "contract session/home path escapes owned root: root=" + + normalizedRoot + + ", sessions=" + + normalizedSessions + + ", home=" + + normalizedHome); + } + } + + @Override + public void close() throws IOException { + restore(SESSIONS_PROPERTY, priorSessionsFile); + restore("user.home", priorUserHome); + try (var paths = Files.walk(root)) { + paths.sorted(Comparator.reverseOrder()).forEach(GuidanceContractContext::delete); + } + if (Files.exists(root)) { + throw new IOException("could not remove owned guidance-contract directory: " + root); + } + } + + private static void restore(String property, String priorValue) { + if (priorValue == null) { + System.clearProperty(property); + } else { + System.setProperty(property, priorValue); + } + } + + private static void delete(Path path) { + try { + Files.delete(path); + } catch (IOException e) { + throw new IllegalStateException("could not remove owned guidance-contract path: " + path, e); + } + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractSessionIsolationTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractSessionIsolationTest.java new file mode 100644 index 00000000..577f1420 --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceContractSessionIsolationTest.java @@ -0,0 +1,79 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.api.parallel.Resources; + +class GuidanceContractSessionIsolationTest { + + private static final String STALE_ALIAS = "must-not-appear-in-guidance-contract"; + private static final String EMPTY_SESSIONS = + """ + # Open sessions + + ## JFR recordings + _none open_ + + ## Heap dumps + _none open_ + + ## pprof profiles + _none open_ + + ## OTLP profiles + _none open_ + + A session id or alias can be passed as `sessionId` to any tool of the matching family, and named in cross-session operators such as `join(session=...)`. + + """; + + @Test + @ResourceLock(Resources.SYSTEM_PROPERTIES) + void renderingAndExportIgnoreStaleSessionState(@TempDir Path tempDir) throws Exception { + Path stale = tempDir.resolve("stale-sessions.json"); + Files.writeString( + stale, "[{\"id\":99,\"path\":\"/tmp/stale.jfr\",\"alias\":\"" + STALE_ALIAS + "\"}]"); + String originalSessions = System.getProperty("jafar.mcp.sessions.file"); + try { + System.setProperty("jafar.mcp.sessions.file", stale.toString()); + Path contractRoot; + String rendered; + try (GuidanceContractContext context = GuidanceContractContext.create()) { + contractRoot = context.root(); + assertTrue(context.sessionsFile().startsWith(contractRoot)); + assertTrue(context.home().startsWith(contractRoot)); + assertFalse(context.sessionsFile().equals(stale.toAbsolutePath().normalize())); + rendered = new GuidanceSurfaces().resourceTexts().get("jafar://sessions"); + assertEquals(EMPTY_SESSIONS, rendered); + assertFalse(rendered.contains(STALE_ALIAS)); + } + assertFalse(Files.exists(contractRoot), "context must remove only its owned root"); + + Path exported = tempDir.resolve("exported-contract.json"); + McpContractExport.main(new String[] {exported.toString()}); + String exportedJson = Files.readString(exported); + String exportedSessions = ""; + for (var resource : McpContractJson.parse(exportedJson).path("resources")) { + if ("jafar://sessions".equals(resource.path("uri").asString())) { + exportedSessions = resource.path("renderedText").asString(); + } + } + assertEquals(EMPTY_SESSIONS, exportedSessions); + assertFalse(exportedJson.contains(STALE_ALIAS)); + assertTrue(Files.exists(exported)); + } finally { + if (originalSessions == null) { + System.clearProperty("jafar.mcp.sessions.file"); + } else { + System.setProperty("jafar.mcp.sessions.file", originalSessions); + } + } + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceExampleQueriesTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceExampleQueriesTest.java new file mode 100644 index 00000000..5ec009aa --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceExampleQueriesTest.java @@ -0,0 +1,67 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.ArrayList; +import java.util.List; +import org.junit.jupiter.api.Test; + +/** + * Every query example in the server's guidance must parse with the real parser of its language. + * + *

An agent has no other teacher: the examples in tool descriptions, help topics and help + * resources are copied verbatim into tool calls. An example that does not parse is worse than no + * example — it teaches every agent that follows it to fail, and because the text looks plausible, + * the failure surfaces as an agent flailing, not as a Jafar bug. This is the guidance-level twin of + * R4 (documentation is code): the docs must run. + * + *

Extraction mirrors how a reader finds examples: fenced code lines, backtick spans, and bare + * prose (see {@link QueryExamples}). Anything that looks like a query must parse; placeholders and + * grammar notation are skipped. The extraction is deliberately generous — a candidate that parses + * is harmless even if it was English — because the test only fails on look-alikes the parser + * rejects. + * + *

This class found real bugs on its first run: {@code pprof_query} and {@code otlp_query} + * shipped examples with literal backslashes ({@code samples[thread=\'main\']}) from over-escaped + * Java string literals, the JfrPath filters help documented a nested-bracket {@code any:} form the + * parser rejects, and the heap help taught {@code checkLeaks()} — which does not parse and, had it + * parsed, would have silently returned rows unchanged. + */ +class GuidanceExampleQueriesTest { + + @Test + void everyDocumentedQueryExampleParses() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + List failures = new ArrayList<>(); + int checked = 0; + int finalChecked; + for (GuidanceSurfaces.Surface surface : surfaces.surfaces()) { + if (!surface.hasLanguage()) { + continue; + } + for (QueryExamples.Candidate candidate : + QueryExamples.extract(surface.name(), surface.text(), surface.language())) { + checked++; + try { + surface.language().parse(candidate.query()); + } catch (Exception | AssertionError e) { + failures.add( + String.format( + "%s (%s): %s%n parser said: %s", + surface.name(), candidate.origin(), candidate.query(), e.getMessage())); + } + } + } + finalChecked = checked; + assertTrue( + failures.isEmpty(), + () -> + failures.size() + + " of " + + finalChecked + + " documented query examples do not parse. An agent copying them gets a" + + " parse error — fix the example or the parser:%n%n" + + String.join("%n%n", failures)); + assertTrue(finalChecked > 200, "expected a substantial example corpus, got " + finalChecked); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceSurfaces.java b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceSurfaces.java new file mode 100644 index 00000000..cbc50bba --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceSurfaces.java @@ -0,0 +1,304 @@ +package io.jafar.mcp; + +import io.jafar.hdump.shell.hdumppath.HdumpPathParser; +import io.jafar.mcp.hdump.HdumpTools; +import io.jafar.mcp.jfr.JfrHelpProvider; +import io.jafar.otlp.shell.otlppath.OtlpPathParser; +import io.jafar.pprof.shell.pprofpath.PprofPathParser; +import io.jafar.shell.core.RequestScope; +import io.jafar.shell.jfrpath.JfrPathParser; +import io.modelcontextprotocol.common.McpTransportContext; +import io.modelcontextprotocol.server.McpAsyncServerExchange; +import io.modelcontextprotocol.server.McpServerFeatures; +import io.modelcontextprotocol.server.McpSyncServerExchange; +import io.modelcontextprotocol.spec.McpSchema; +import java.lang.reflect.Field; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * Every text surface a Jafar MCP client can read, collected from a live {@link JafarMcpServer}. + * + *

The server's guidance lives in more places than tool descriptions: help topics, MCP prompts, + * resources and input-schema property descriptions all name tools, parameters and example queries. + * A change to any of them can silently strand an agent (a renamed tool turns every skill or help + * line that still names the old one into confident instructions for a call that fails), so tests + * that guard the guidance must read all of it — from the same objects a live server registers, not + * from copies that can drift from the real wiring. + * + *

Surfaces are tagged with the query language (if any) their examples belong to, so + * example-extraction tests know which parser must accept them. Prompts and the sessions resource + * carry no query language: they are scanned only for tool-name references. + */ +final class GuidanceSurfaces { + + /** The query languages the four tool families document, each bound to its real parser. */ + enum QueryLanguage { + JFRPATH { + @Override + void parse(String query) throws Exception { + JfrPathParser.parse(query); + } + }, + HDUMPPATH { + @Override + void parse(String query) throws Exception { + HdumpPathParser.parse(query); + } + }, + PPROFPATH { + @Override + void parse(String query) throws Exception { + PprofPathParser.parse(query); + } + }, + OTLPPATH { + @Override + void parse(String query) throws Exception { + OtlpPathParser.parse(query); + } + }; + + /** Parses {@code query}, throwing when the parser rejects it. */ + abstract void parse(String query) throws Exception; + + static QueryLanguage forTool(String toolName) { + if (toolName.startsWith("jfr_")) return JFRPATH; + if (toolName.startsWith("hdump_")) return HDUMPPATH; + if (toolName.startsWith("pprof_")) return PPROFPATH; + if (toolName.startsWith("otlp_")) return OTLPPATH; + throw new IllegalArgumentException("no family for tool: " + toolName); + } + } + + /** One readable surface: where it came from, its text, and the query language it documents. */ + record Surface(String name, String text, QueryLanguage language) { + boolean hasLanguage() { + return language != null; + } + } + + private final JafarMcpServer server; + private final List tools; + + GuidanceSurfaces() { + this.server = new JafarMcpServer(); + this.tools = server.createToolSpecifications(); + } + + List tools() { + return tools; + } + + /** Names of every registered tool. */ + List toolNames() { + return tools.stream().map(t -> t.tool().name()).toList(); + } + + /** + * Every readable surface, in a stable order (tools, help, prompts, resources). Metadata + * descriptions of prompts and resources are surfaces too: a client's first view of them is the + * name and description, before any content is fetched. + */ + List surfaces() { + List out = new ArrayList<>(); + for (McpServerFeatures.SyncToolSpecification tool : tools) { + String name = tool.tool().name(); + out.add( + new Surface( + "tool:" + name + " description", + tool.tool().description(), + QueryLanguage.forTool(name))); + out.add( + new Surface("tool:" + name + " schema", schemaJson(tool), QueryLanguage.forTool(name))); + } + out.addAll(jfrHelpSurfaces()); + out.addAll(hdumpHelpSurfaces()); + out.addAll(pprofHelpSurfaces()); + out.addAll(otlpHelpSurfaces()); + out.addAll(promptSurfaces()); + out.addAll(resourceSurfaces()); + return out; + } + + private static String schemaJson(McpServerFeatures.SyncToolSpecification tool) { + try { + return io.modelcontextprotocol.json.McpJsonDefaults.getMapper() + .writeValueAsString(tool.tool().inputSchema()); + } catch (java.io.IOException e) { + throw new IllegalStateException("cannot serialize schema of " + tool.tool().name(), e); + } + } + + // ───────────────────────────────────────────────────────────────────────────── + // Help-topic surfaces (the same texts the *_help tools and jafar://help/* serve) + // ───────────────────────────────────────────────────────────────────────────── + + private List jfrHelpSurfaces() { + JfrHelpProvider help = new JfrHelpProvider(); + return List.of( + new Surface("help:jfr/overview", help.getOverviewHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/filters", help.getFiltersHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/pipeline", help.getPipelineHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/functions", help.getFunctionsHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/examples", help.getExamplesHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/event_types", help.getEventTypesHelp(), QueryLanguage.JFRPATH), + new Surface("help:jfr/tools", help.getToolsHelp(), QueryLanguage.JFRPATH)); + } + + private List hdumpHelpSurfaces() { + HdumpTools help = privateField(server, "hdumpTools"); + List out = new ArrayList<>(); + for (String topic : + List.of("overview", "roots", "filters", "operators", "examples", "patterns", "tools")) { + out.add(new Surface("help:hdump/" + topic, help.help(topic), QueryLanguage.HDUMPPATH)); + } + return out; + } + + private List pprofHelpSurfaces() { + return List.of( + new Surface("help:pprof", pprofTools().getPprofHelpText(), QueryLanguage.PPROFPATH)); + } + + private List otlpHelpSurfaces() { + return List.of(new Surface("help:otlp", otlpTools().getOtlpHelpText(), QueryLanguage.OTLPPATH)); + } + + // ───────────────────────────────────────────────────────────────────────────── + // Prompt and resource surfaces (rendered exactly as a client would receive them) + // ───────────────────────────────────────────────────────────────────────────── + + /** Prompt specifications, in registration order. */ + List promptSpecs() { + io.jafar.mcp.prompt.JafarPrompts prompts = privateField(server, "jafarPrompts"); + return prompts.createPromptSpecifications(); + } + + /** Resource specifications, in registration order. */ + List resourceSpecs() { + io.jafar.mcp.resource.JafarResources resources = privateField(server, "jafarResources"); + return resources.createResourceSpecifications(); + } + + /** + * The fixed client scope the contract is rendered in. + * + *

Since the server scopes session visibility to the MCP client session (see {@link + * RequestScope}), surfaces must be read as a caller, not as a raw thread. A single synthetic id + * keeps the rendered contract deterministic: it is never used to open anything, so every surface + * renders the fresh-client view. + */ + private static final String RENDER_SCOPE = "guidance-contract-render"; + + /** A real exchange carrying {@link #RENDER_SCOPE}, built the way the runtime does it. */ + private static McpSyncServerExchange renderExchange() { + return new McpSyncServerExchange( + new McpAsyncServerExchange(RENDER_SCOPE, null, null, null, McpTransportContext.EMPTY)); + } + + /** Prompt texts, rendered by invoking each prompt handler with empty arguments. */ + Map promptTexts() { + Map out = new LinkedHashMap<>(); + var exchange = renderExchange(); + for (var spec : promptSpecs()) { + String name = spec.prompt().name(); + RequestScope.clear(); + McpSchema.GetPromptResult result = + spec.promptHandler().apply(exchange, new McpSchema.GetPromptRequest(name, Map.of())); + StringBuilder text = new StringBuilder(); + for (var message : result.messages()) { + if (message.content() instanceof McpSchema.TextContent content) { + text.append(content.text()).append('\n'); + } + } + out.put(name, text.toString()); + } + return out; + } + + /** Resource contents keyed by URI, rendered by invoking each read handler. */ + Map resourceTexts() { + Map out = new LinkedHashMap<>(); + for (var spec : resourceSpecs()) { + String uri = spec.resource().uri(); + RequestScope.clear(); + McpSchema.ReadResourceResult result = + spec.readHandler().apply(renderExchange(), new McpSchema.ReadResourceRequest(uri)); + StringBuilder text = new StringBuilder(); + for (var content : result.contents()) { + if (content instanceof McpSchema.TextResourceContents textContent) { + text.append(textContent.text()).append('\n'); + } + } + out.put(uri, text.toString()); + } + return out; + } + + private List promptSurfaces() { + List out = new ArrayList<>(); + for (var spec : promptSpecs()) { + McpSchema.Prompt prompt = spec.prompt(); + StringBuilder meta = + new StringBuilder(prompt.description() == null ? "" : prompt.description()); + for (McpSchema.PromptArgument arg : prompt.arguments()) { + meta.append('\n').append(arg.name()).append(": ").append(arg.description()); + } + out.add(new Surface("prompt:" + prompt.name() + " description", meta.toString(), null)); + } + promptTexts().forEach((name, text) -> out.add(new Surface("prompt:" + name, text, null))); + return out; + } + + private List resourceSurfaces() { + List out = new ArrayList<>(); + for (var spec : resourceSpecs()) { + McpSchema.Resource resource = spec.resource(); + String meta = + (resource.name() == null ? "" : resource.name()) + + ": " + + (resource.description() == null ? "" : resource.description()); + out.add(new Surface("resource-meta:" + resource.uri(), meta, null)); + } + resourceTexts() + .forEach( + (uri, text) -> + out.add( + new Surface( + "resource:" + uri, + text, + switch (uri) { + case "jafar://help/jfrpath", "jafar://help/tools" -> + QueryLanguage.JFRPATH; + case "jafar://help/hdumppath" -> QueryLanguage.HDUMPPATH; + default -> null; + }))); + return out; + } + + // ───────────────────────────────────────────────────────────────────────────── + // Helpers + // ───────────────────────────────────────────────────────────────────────────── + + private io.jafar.mcp.pprof.PprofTools pprofTools() { + return privateField(server, "pprofTools"); + } + + private io.jafar.mcp.otlp.OtlpTools otlpTools() { + return privateField(server, "otlpTools"); + } + + @SuppressWarnings("unchecked") + private static T privateField(JafarMcpServer server, String name) { + try { + Field field = JafarMcpServer.class.getDeclaredField(name); + field.setAccessible(true); + return (T) field.get(server); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException("missing field " + name + " on JafarMcpServer", e); + } + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceToolReferencesTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceToolReferencesTest.java new file mode 100644 index 00000000..29b0596e --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/GuidanceToolReferencesTest.java @@ -0,0 +1,89 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.junit.jupiter.api.Test; + +/** + * Every tool name and resource URI the guidance mentions must actually exist. + * + *

Help topics, prompts, tool descriptions and resources name tools explicitly ("run + * jfr_diagnose", "hdump_report focus=leaks") because that is what makes them guidance rather than + * reference. A rename that leaves one mention behind turns that line into confident instructions + * for a call that fails — the same failure mode {@code doc/agents/Mcp.md} warns about for the + * out-of-repo plugin skills, caught here for every surface this repo ships. + * + *

The scan is over the full rendered text of every surface (see {@link GuidanceSurfaces}), so + * resource contents and prompt texts are covered too, not only tool descriptions. + */ +class GuidanceToolReferencesTest { + + /** + * Tool-name-looking tokens: a family prefix followed by at least one letter. Deliberately does + * not match {@code jfr_*} wildcards or {@code hdump_} alone, so prose saying "the jfr_* tools" is + * not a reference. + */ + private static final Pattern TOOL_REFERENCE = + Pattern.compile("\\b((?:jfr|hdump|pprof|otlp)_[a-z][a-z0-9_]*)"); + + /** Resource URIs mentioned anywhere in the guidance. */ + private static final Pattern RESOURCE_REFERENCE = Pattern.compile("\\b(jafar://[A-Za-z0-9/_-]+)"); + + @Test + void everyMentionedToolIsRegistered() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + Set registered = new HashSet<>(surfaces.toolNames()); + List dangling = new ArrayList<>(); + for (GuidanceSurfaces.Surface surface : surfaces.surfaces()) { + Matcher matcher = TOOL_REFERENCE.matcher(surface.text()); + while (matcher.find()) { + if (!registered.contains(matcher.group(1))) { + dangling.add(surface.name() + ": " + matcher.group(1)); + } + } + } + assertTrue( + dangling.isEmpty(), + () -> + "The guidance names tools that are not registered — an agent following it will call a" + + " tool that does not exist. Either register the tool or fix the text:%n%n" + + String.join("%n", dangling)); + } + + @Test + void everyMentionedResourceUriIsRegistered() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + Set registered = new HashSet<>(surfaces.resourceTexts().keySet()); + List dangling = new ArrayList<>(); + for (GuidanceSurfaces.Surface surface : surfaces.surfaces()) { + Matcher matcher = RESOURCE_REFERENCE.matcher(surface.text()); + while (matcher.find()) { + if (!registered.contains(matcher.group(1))) { + dangling.add(surface.name() + ": " + matcher.group(1)); + } + } + } + assertTrue( + dangling.isEmpty(), + () -> + "The guidance mentions resources the server does not serve:%n%n" + + String.join("%n", dangling)); + } + + @Test + void helpTextsMentionTheToolsTheyTeach() { + // A guard against help texts that drift into vague prose: each family's tool-selection + // guide must still name at least its family's open tool, or it no longer guides anything. + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + String toolsHelp = surfaces.resourceTexts().get("jafar://help/tools"); + assertTrue( + toolsHelp != null && toolsHelp.contains("jfr_open"), + "jafar://help/tools must mention jfr_open — it is the entry point it teaches"); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpAgentWorkflowEndToEndTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpAgentWorkflowEndToEndTest.java new file mode 100644 index 00000000..7043227b --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpAgentWorkflowEndToEndTest.java @@ -0,0 +1,204 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.jafar.hdump.test.SyntheticHeapDumpGenerator; +import io.jafar.otlp.MinimalOtlpBuilder; +import io.jafar.pprof.MinimalPprofBuilder; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import tools.jackson.databind.JsonNode; + +/** One useful black-box workflow per format through {@code java -jar ... --stdio}. */ +@Tag("e2e") +class McpAgentWorkflowEndToEndTest { + + private static final String[] WORKFLOW_ALIASES = { + "jfr-workflow", "hprof-workflow", "pprof-workflow", "otlp-workflow" + }; + + @Test + @Timeout(120) + void jfrWorkflowReadsFixtureAndKeepsTheServerUsable() throws Exception { + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("jfr-workflow")) { + mcp.assertFreshSessions(WORKFLOW_ALIASES); + Path fixture = McpEndToEndTest.findTestJfr(); + JsonNode open = + mcp.call("jfr_open", Map.of("path", fixture.toString(), "alias", "jfr-workflow")); + JsonNode openPayload = mcp.contentJson(open); + assertEquals(1, openPayload.path("id").asInt(), () -> "open=" + open); + assertEquals("jfr-workflow", openPayload.path("alias").asString(), () -> "open=" + open); + assertEquals(fixture.toString(), openPayload.path("path").asString(), () -> "open=" + open); + assertEquals(204, openPayload.path("availableTypes").asInt(), () -> "open=" + open); + assertEquals(2, openPayload.path("chunkCount").asInt(), () -> "open=" + open); + JsonNode summary = mcp.call("jfr_summary", Map.of("sessionId", "jfr-workflow")); + JsonNode summaryPayload = mcp.contentJson(summary); + assertEquals(1, summaryPayload.path("sessionId").asInt(), () -> "summary=" + summary); + assertEquals( + fixture.toString(), + summaryPayload.path("recordingPath").asString(), + () -> "summary=" + summary); + JsonNode query = + mcp.call( + "jfr_query", + Map.of( + "sessionId", + "jfr-workflow", + "query", + "events/jdk.JVMInformation | select(jvmVersion, jvmName)")); + JsonNode queryPayload = mcp.contentJson(query); + assertEquals(2, queryPayload.path("resultCount").asInt(), () -> "query=" + query); + assertEquals(2, queryPayload.path("results").size(), () -> "query=" + query); + assertEquals( + "OpenJDK 64-Bit Server VM", + queryPayload.at("/results/0/jvmName").asString(), + () -> "query=" + query); + JsonNode diagnose = mcp.call("jfr_diagnose", Map.of("sessionId", "jfr-workflow"), 4); + assertTrue(mcp.contentJson(diagnose).has("findings"), () -> "diagnose=" + diagnose); + mcp.assertProgress(4); + mcp.call("jfr_close", Map.of("sessionId", "jfr-workflow")); + mcp.call("jfr_help", Map.of("topic", "tools")); + } + } + + @Test + @Timeout(120) + void hprofWorkflowUsesPromotedSyntheticWriter() throws Exception { + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("hprof-workflow")) { + mcp.assertFreshSessions(WORKFLOW_ALIASES); + Path fixture = mcp.root().resolve("tier1-seven-objects.hprof"); + SyntheticHeapDumpGenerator.generateMinimalHeapDump(fixture, 7); + JsonNode open = + mcp.call("hdump_open", Map.of("path", fixture.toString(), "alias", "hprof-workflow")); + JsonNode openPayload = mcp.contentJson(open); + assertEquals(1, openPayload.path("id").asInt(), () -> "open=" + open); + assertEquals("hprof-workflow", openPayload.path("alias").asString(), () -> "open=" + open); + assertEquals(7, openPayload.path("objectCount").asInt(), () -> "open=" + open); + assertEquals(2, openPayload.path("classCount").asInt(), () -> "open=" + open); + JsonNode summary = mcp.call("hdump_summary", Map.of("sessionId", "hprof-workflow")); + JsonNode summaryPayload = mcp.contentJson(summary); + assertEquals(7, summaryPayload.path("objectCount").asInt(), () -> "summary=" + summary); + assertEquals(2, summaryPayload.path("classCount").asInt(), () -> "summary=" + summary); + JsonNode query = + mcp.call( + "hdump_query", + Map.of("sessionId", "hprof-workflow", "query", "classes | top(10, instanceCount)")); + JsonNode queryPayload = mcp.contentJson(query); + assertEquals(2, queryPayload.path("resultCount").asInt(), () -> "query=" + query); + assertEquals(2, queryPayload.path("results").size(), () -> "query=" + query); + assertTrue( + queryPayload.path("results").toString().contains("java.lang.Object") + && queryPayload.path("results").toString().contains("java.lang.String"), + () -> "query=" + query); + JsonNode report = + mcp.call("hdump_report", Map.of("sessionId", "hprof-workflow", "focus", "histogram")); + assertTrue(mcp.contentJson(report).has("findings"), () -> "report=" + report); + mcp.call("hdump_close", Map.of("sessionId", "hprof-workflow")); + mcp.call("jfr_help", Map.of("topic", "tools")); + } + } + + @Test + @Timeout(120) + void pprofWorkflowPreservesDistinctiveFunctionThroughStdio() throws Exception { + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("pprof-workflow")) { + mcp.assertFreshSessions(WORKFLOW_ALIASES); + Path fixture = pprofFixture(mcp.root()); + JsonNode open = + mcp.call("pprof_open", Map.of("path", fixture.toString(), "alias", "pprof-workflow")); + assertEquals(1, mcp.contentJson(open).path("id").asInt(), () -> "open=" + open); + JsonNode summary = mcp.call("pprof_summary", Map.of("sessionId", "pprof-workflow")); + assertTrue( + mcp.contentJson(summary).toString().contains("tier1.pprof.UniqueCpuMethod"), + () -> "summary=" + summary); + JsonNode query = + mcp.call( + "pprof_query", + Map.of("sessionId", "pprof-workflow", "query", "samples | top(10, cpu)")); + JsonNode queryPayload = mcp.contentJson(query); + assertEquals(1, queryPayload.path("resultCount").asInt(), () -> "query=" + query); + assertEquals(4242L, queryPayload.at("/results/0/cpu").asLong(), () -> "query=" + query); + assertEquals( + "tier1.pprof.UniqueCpuMethod", + queryPayload.at("/results/0/stackTrace/0/name").asString(), + () -> "query=" + query); + assertEquals( + 42, queryPayload.at("/results/0/stackTrace/0/line").asInt(), () -> "query=" + query); + JsonNode use = mcp.call("pprof_use", Map.of("sessionId", "pprof-workflow"), 5); + assertFalse(mcp.contentJson(use).isEmpty(), () -> "use=" + use); + mcp.assertProgress(5); + mcp.call("pprof_close", Map.of("sessionId", "pprof-workflow")); + mcp.call("jfr_help", Map.of("topic", "tools")); + } + } + + @Test + @Timeout(120) + void otlpWorkflowPreservesDistinctiveFunctionThroughStdio() throws Exception { + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("otlp-workflow")) { + mcp.assertFreshSessions(WORKFLOW_ALIASES); + Path fixture = otlpFixture(mcp.root()); + JsonNode open = + mcp.call("otlp_open", Map.of("path", fixture.toString(), "alias", "otlp-workflow")); + assertEquals(1, mcp.contentJson(open).path("id").asInt(), () -> "open=" + open); + JsonNode summary = mcp.call("otlp_summary", Map.of("sessionId", "otlp-workflow")); + assertTrue( + mcp.contentJson(summary).toString().contains("tier1.otlp.UniqueCpuMethod"), + () -> "summary=" + summary); + JsonNode query = + mcp.call( + "otlp_query", + Map.of("sessionId", "otlp-workflow", "query", "samples | top(10, cpu)")); + JsonNode queryPayload = mcp.contentJson(query); + assertEquals(1, queryPayload.path("resultCount").asInt(), () -> "query=" + query); + assertEquals(4343L, queryPayload.at("/results/0/cpu").asLong(), () -> "query=" + query); + assertEquals( + "tier1-main", queryPayload.at("/results/0/thread").asString(), () -> "query=" + query); + assertEquals( + "tier1.otlp.UniqueCpuMethod", + queryPayload.at("/results/0/stackTrace/0/name").asString(), + () -> "query=" + query); + assertEquals( + 42, queryPayload.at("/results/0/stackTrace/0/line").asInt(), () -> "query=" + query); + JsonNode use = mcp.call("otlp_use", Map.of("sessionId", "otlp-workflow"), 6); + assertFalse(mcp.contentJson(use).isEmpty(), () -> "use=" + use); + mcp.assertProgress(6); + mcp.call("otlp_close", Map.of("sessionId", "otlp-workflow")); + mcp.call("jfr_help", Map.of("topic", "tools")); + } + } + + private static Path pprofFixture(Path dir) throws Exception { + MinimalPprofBuilder builder = new MinimalPprofBuilder(); + int cpu = builder.addString("cpu"); + int nanos = builder.addString("nanoseconds"); + builder.addSampleType(cpu, nanos).setDurationNanos(1_000_000_000L); + long function = + builder.addFunction( + builder.addString("tier1.pprof.UniqueCpuMethod"), builder.addString("Tier1.java")); + long location = builder.addLocation(function, 42); + builder.addSample(List.of(location), List.of(4242L), List.of()); + return builder.write(dir); + } + + private static Path otlpFixture(Path dir) throws Exception { + MinimalOtlpBuilder builder = new MinimalOtlpBuilder(); + int cpu = builder.addString("cpu"); + int nanos = builder.addString("nanoseconds"); + builder.setSampleType(cpu, nanos).setDurationNanos(1_000_000_000L); + int function = + builder.addFunction( + builder.addString("tier1.otlp.UniqueCpuMethod"), builder.addString("Tier1.java")); + int location = builder.addLocation(function, 42); + int stack = builder.addStack(List.of(location)); + int attribute = builder.addAttribute(builder.addString("thread"), "tier1-main"); + builder.addSample(stack, List.of(attribute), List.of(4343L)); + return builder.write(dir); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpContractExport.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpContractExport.java new file mode 100644 index 00000000..0c88ff5c --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpContractExport.java @@ -0,0 +1,27 @@ +package io.jafar.mcp; + +import java.nio.file.Files; +import java.nio.file.Path; + +/** + * Writes the MCP guidance contract JSON to a path, for consumers outside the test suite. + * + *

Run via {@code ./gradlew :jfr-mcp:exportMcpContract}. The output is byte-identical to the + * committed snapshot ({@code src/test/resources/mcp-contract.json}), so the jafar-perf plugin in + * btraceio/agent-plugins (weekly Tool-drift check, scripts/check-tool-references.js) can validate + * its skills against the in-development server rather than waiting for a published release — the + * drift is caught before it ships, not by the weekly cron after. + */ +public final class McpContractExport { + + public static void main(String[] args) throws Exception { + Path out = args.length > 0 ? Path.of(args[0]) : Path.of("build/mcp-contract/mcp-contract.json"); + Files.createDirectories(out.toAbsolutePath().getParent()); + try (GuidanceContractContext ignored = GuidanceContractContext.create()) { + Files.writeString(out, McpContractJson.render(new GuidanceSurfaces())); + } + System.out.println("Wrote MCP guidance contract to " + out.toAbsolutePath()); + } + + private McpContractExport() {} +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpContractJson.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpContractJson.java new file mode 100644 index 00000000..316f0afa --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpContractJson.java @@ -0,0 +1,95 @@ +package io.jafar.mcp; + +import io.modelcontextprotocol.spec.McpSchema; +import java.util.ArrayList; +import java.util.Comparator; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; +import tools.jackson.databind.SerializationFeature; +import tools.jackson.databind.json.JsonMapper; +import tools.jackson.databind.node.ArrayNode; +import tools.jackson.databind.node.ObjectNode; + +/** + * Canonical JSON rendering of the server's guidance contract: every tool with its description and + * input schema, every prompt, every resource. + * + *

One rendering serves three consumers, so they can never disagree (one source of truth): the + * golden-snapshot test ({@code ToolCatalogSnapshotTest}) compares it against the committed {@code + * src/test/resources/mcp-contract.json}; a change there must be conscious, because the diff is + * exactly what every MCP client — and the agent-plugins jafar-perf plugin skills, which name tools + * and parameters explicitly — will experience. The {@code exportMcpContract} task writes the same + * JSON to the build directory so out-of-repo consumers (agent-plugins' tool-drift check) can diff + * their skills against the in-development server instead of waiting for a release. + * + *

Keys are sorted and tools are ordered by name, so cosmetic reordering in the Java sources does + * not churn the file — only real contract changes do. + */ +final class McpContractJson { + + private static final ObjectMapper MAPPER = + JsonMapper.builder().enable(SerializationFeature.ORDER_MAP_ENTRIES_BY_KEYS).build(); + + private McpContractJson() {} + + static String render(GuidanceSurfaces surfaces) { + ObjectNode root = MAPPER.createObjectNode(); + + var tools = new ArrayList<>(surfaces.tools()); + tools.sort(Comparator.comparing(t -> t.tool().name())); + ArrayNode toolsNode = root.putArray("tools"); + for (var tool : tools) { + ObjectNode toolNode = toolsNode.addObject(); + toolNode.put("name", tool.tool().name()); + toolNode.put("description", tool.tool().description()); + toolNode.set("inputSchema", MAPPER.readTree(schemaJson(tool))); + } + + var prompts = new ArrayList<>(surfaces.promptSpecs()); + prompts.sort(Comparator.comparing(p -> p.prompt().name())); + var promptTexts = surfaces.promptTexts(); + ArrayNode promptsNode = root.putArray("prompts"); + for (var prompt : prompts) { + ObjectNode promptNode = promptsNode.addObject(); + promptNode.put("name", prompt.prompt().name()); + promptNode.put("description", prompt.prompt().description()); + ArrayNode argsNode = promptNode.putArray("arguments"); + for (McpSchema.PromptArgument arg : prompt.prompt().arguments()) { + ObjectNode argNode = argsNode.addObject(); + argNode.put("name", arg.name()); + argNode.put("description", arg.description()); + argNode.put("required", Boolean.TRUE.equals(arg.required())); + } + promptNode.put("renderedText", promptTexts.get(prompt.prompt().name())); + } + + var resources = new ArrayList<>(surfaces.resourceSpecs()); + resources.sort(Comparator.comparing(r -> r.resource().uri())); + var resourceTexts = surfaces.resourceTexts(); + ArrayNode resourcesNode = root.putArray("resources"); + for (var resource : resources) { + ObjectNode resourceNode = resourcesNode.addObject(); + resourceNode.put("uri", resource.resource().uri()); + resourceNode.put("name", resource.resource().name()); + resourceNode.put("description", resource.resource().description()); + resourceNode.put("mimeType", resource.resource().mimeType()); + resourceNode.put("renderedText", resourceTexts.get(resource.resource().uri())); + } + + return MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(root); + } + + private static String schemaJson( + io.modelcontextprotocol.server.McpServerFeatures.SyncToolSpecification tool) { + try { + return io.modelcontextprotocol.json.McpJsonDefaults.getMapper() + .writeValueAsString(tool.tool().inputSchema()); + } catch (java.io.IOException e) { + throw new IllegalStateException("cannot serialize schema of " + tool.tool().name(), e); + } + } + + static JsonNode parse(String json) { + return MAPPER.readTree(json); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpEndToEndTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpEndToEndTest.java index 132ce959..ce71a19b 100644 --- a/jfr-mcp/src/test/java/io/jafar/mcp/McpEndToEndTest.java +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpEndToEndTest.java @@ -2,317 +2,71 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.junit.jupiter.api.Assumptions.assumeTrue; -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStreamReader; -import java.io.OutputStreamWriter; -import java.io.PrintWriter; -import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; -import java.util.ArrayList; -import java.util.List; import java.util.Map; -import java.util.concurrent.BlockingQueue; -import java.util.concurrent.LinkedBlockingQueue; -import java.util.concurrent.TimeUnit; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Timeout; import tools.jackson.databind.JsonNode; -import tools.jackson.databind.ObjectMapper; -import tools.jackson.databind.node.ObjectNode; -/** - * End-to-end test that spawns the MCP server as a subprocess — exactly as the Claude Code plugin - * launcher does — and drives the analysis workflow that previously caused the server to disconnect. - * - *

Run with: {@code ./gradlew :jfr-mcp:endToEndTest} - * - *

By default the test uses the locally-built shadow jar. Pass {@code -Dmcp.e2e.use.jbang=true} - * to use {@code jbang jfr-mcp@btraceio} instead (tests the published release against the same - * workflow). - */ +/** Regression coverage for the real local shadow-JAR stdio progress-token path. */ @Tag("e2e") class McpEndToEndTest { - private static final ObjectMapper MAPPER = new ObjectMapper(); - private static final long TOOL_TIMEOUT_MS = 30_000; - - // ───────────────────────────────────────────────────────────────────────────── - // Regression: wrong progressToken caused Claude Code to close stdin - // ───────────────────────────────────────────────────────────────────────────── - - /** - * Reproduces the "plugin disconnected" failure mode: jfr_open → jfr_query → jfr_diagnose. - * - *

Before the fix, jfr_diagnose sent progress notifications with a string token ("diagnose") - * while Claude Code had sent {@code progressToken: 4} (an integer). Claude Code's MCP client - * closed stdin within ~5 ms of receiving the mismatched notification, killing the server before - * it could return the diagnosis result. - */ @Test @Timeout(120) void serverSurvivesOpenQueryDiagnoseSequence() throws Exception { - Path jar = findShadowJar(); - Path jfr = findTestJfr(); - - Process proc = startServer(jar); - MessageCollector collector = new MessageCollector(proc.getInputStream()); - PrintWriter writer = writerFor(proc); - - try { - handshake(writer, collector); - - // Step 1: open the recording - sendCall(writer, 2, "jfr_open", Map.of("path", jfr.toAbsolutePath().toString()), null); - JsonNode openResp = collector.waitForId(2, TOOL_TIMEOUT_MS); - assertNotNull(openResp, "jfr_open must return a response"); - assertFalse(openResp.at("/result/isError").asBoolean(), "jfr_open must not error"); - assertTrue(proc.isAlive(), "server must be alive after jfr_open"); - - // Step 2: query JVM info (the intermediate step in the real workflow) - sendCall( - writer, - 3, - "jfr_query", - Map.of("query", "events/jdk.JVMInformation | select(jvmVersion, jvmName)"), - 3); - JsonNode queryResp = collector.waitForId(3, TOOL_TIMEOUT_MS); - assertNotNull(queryResp, "jfr_query must return a response"); - assertTrue(proc.isAlive(), "server must be alive after jfr_query"); - - // Step 3: jfr_diagnose — this is the call that previously killed the server. - // The request carries progressToken:4 (integer) to match Claude Code's real payload. - sendCall(writer, 4, "jfr_diagnose", Map.of(), 4); - JsonNode diagnoseResp = collector.waitForId(4, TOOL_TIMEOUT_MS); - - assertNotNull( - diagnoseResp, - "jfr_diagnose must return a response — server must not disconnect mid-session"); - assertFalse(diagnoseResp.has("error"), "jfr_diagnose must not be a JSON-RPC error"); + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("diagnose-progress")) { + JsonNode open = + mcp.call("jfr_open", Map.of("path", findTestJfr().toString(), "alias", "tier1-jfr")); + assertNotNull(mcp.contentJson(open).path("sessionId")); + + JsonNode query = + mcp.call( + "jfr_query", + Map.of("query", "events/jdk.JVMInformation | select(jvmVersion, jvmName)")); assertFalse( - diagnoseResp.at("/result/isError").asBoolean(), - "jfr_diagnose result.isError must be false"); - assertTrue( - proc.isAlive(), "server process must still be running after jfr_diagnose completes"); + mcp.contentJson(query).path("results").isMissingNode(), () -> "query payload=" + query); - } finally { - writer.close(); - proc.destroyForcibly(); - proc.waitFor(5, TimeUnit.SECONDS); - collector.stop(); + JsonNode diagnose = mcp.call("jfr_diagnose", Map.of(), 4); + assertFalse(mcp.contentJson(diagnose).isMissingNode(), () -> "diagnose payload=" + diagnose); + mcp.assertProgress(4); + mcp.call("jfr_close", Map.of("sessionId", "tier1-jfr")); + mcp.call("jfr_help", Map.of("topic", "tools")); } } - /** - * Regression: jfr_use sends multiple progress notifications (for CPU, memory, threads, I/O). - * After jfr_open + jfr_use the server must still respond to further calls. - */ @Test @Timeout(120) void serverSurvivesUseMethodAfterOpen() throws Exception { - Path jar = findShadowJar(); - Path jfr = findTestJfr(); - - Process proc = startServer(jar); - MessageCollector collector = new MessageCollector(proc.getInputStream()); - PrintWriter writer = writerFor(proc); - - try { - handshake(writer, collector); - - sendCall(writer, 2, "jfr_open", Map.of("path", jfr.toAbsolutePath().toString()), null); - assertNotNull(collector.waitForId(2, TOOL_TIMEOUT_MS), "jfr_open must respond"); - assertTrue(proc.isAlive(), "server must be alive after jfr_open"); - - sendCall(writer, 3, "jfr_use", Map.of(), 3); - JsonNode useResp = collector.waitForId(3, TOOL_TIMEOUT_MS); - assertNotNull(useResp, "jfr_use must return a response"); - assertFalse(useResp.has("error"), "jfr_use must not JSON-RPC error"); - assertTrue(proc.isAlive(), "server must be alive after jfr_use"); - - // Server must still handle a subsequent call - sendCall(writer, 4, "jfr_help", Map.of(), null); - JsonNode helpResp = collector.waitForId(4, TOOL_TIMEOUT_MS); - assertNotNull(helpResp, "jfr_help must respond after jfr_use"); - assertTrue(proc.isAlive(), "server must be alive after subsequent call"); - - } finally { - writer.close(); - proc.destroyForcibly(); - proc.waitFor(5, TimeUnit.SECONDS); - collector.stop(); - } - } - - // ───────────────────────────────────────────────────────────────────────────── - // Infrastructure helpers - // ───────────────────────────────────────────────────────────────────────────── - - private static Process startServer(Path jar) throws IOException { - // The child is a separate JVM, so the isolation Gradle gives the test JVM does not reach it: - // without this it reads and rewrites the developer's real ~/.jafar (persisted sessions). - String isolatedState = "-Djafar.state.dir=" + Files.createTempDirectory("jafar-e2e-state"); - List cmd; - if (Boolean.getBoolean("mcp.e2e.use.jbang")) { - String jbang = System.getProperty("mcp.e2e.jbang.cmd", "jbang"); - cmd = List.of(jbang, isolatedState, "jfr-mcp@btraceio", "--stdio"); - } else { - String java = ProcessHandle.current().info().command().orElse("java"); - cmd = List.of(java, isolatedState, "-jar", jar.toAbsolutePath().toString(), "--stdio"); - } - - ProcessBuilder pb = new ProcessBuilder(cmd); - pb.redirectError(ProcessBuilder.Redirect.DISCARD); - return pb.start(); - } - - private static PrintWriter writerFor(Process proc) { - return new PrintWriter( - new OutputStreamWriter(proc.getOutputStream(), StandardCharsets.UTF_8), true); - } - - private static void handshake(PrintWriter writer, MessageCollector collector) - throws InterruptedException { - writer.println( - "{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"initialize\"," - + "\"params\":{\"protocolVersion\":\"2025-06-18\"," - + "\"capabilities\":{},\"clientInfo\":{\"name\":\"e2e-test\",\"version\":\"1.0\"}}}"); - JsonNode resp = collector.waitForId(0, TOOL_TIMEOUT_MS); - assumeTrue(resp != null && !resp.has("error"), "MCP handshake failed: " + resp); - writer.println("{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\",\"params\":{}}"); - } - - /** - * Sends a {@code tools/call} request that mirrors Claude Code's real payload, including a {@code - * _meta.progressToken} when {@code progressToken} is non-null. - */ - private static void sendCall( - PrintWriter writer, int id, String tool, Map args, Integer progressToken) - throws Exception { - ObjectNode params = MAPPER.createObjectNode(); - params.put("name", tool); - params.set("arguments", MAPPER.valueToTree(args)); - if (progressToken != null) { - ObjectNode meta = MAPPER.createObjectNode(); - meta.put("claudecode/toolUseId", "toolu_e2e_" + id); - meta.put("progressToken", progressToken); - params.set("_meta", meta); + try (McpStdioProcessHarness mcp = new McpStdioProcessHarness("use-progress")) { + mcp.call("jfr_open", Map.of("path", findTestJfr().toString(), "alias", "tier1-use")); + mcp.call("jfr_use", Map.of(), 3); + mcp.assertProgress(3); + mcp.call("jfr_close", Map.of("sessionId", "tier1-use")); + mcp.call("jfr_help", Map.of("topic", "tools")); } - ObjectNode req = MAPPER.createObjectNode(); - req.put("jsonrpc", "2.0"); - req.put("id", id); - req.put("method", "tools/call"); - req.set("params", params); - writer.println(MAPPER.writeValueAsString(req)); } - private static Path findShadowJar() { - if (Boolean.getBoolean("mcp.e2e.use.jbang")) { - return Path.of("/dev/null"); // unused in jbang mode - } - Path libs = Paths.get("build/libs"); - if (!Files.isDirectory(libs)) { - libs = Paths.get("jfr-mcp/build/libs"); - } - try { - return Files.list(libs) - .filter(p -> p.getFileName().toString().matches("jfr-mcp-.*-all\\.jar")) - .max( - java.util.Comparator.comparingLong( - p -> { - try { - return Files.getLastModifiedTime(p).toMillis(); - } catch (IOException e) { - return 0L; - } - })) - .orElseGet( - () -> { - assumeTrue(false, "No jfr-mcp-*-all.jar found — run ./gradlew :jfr-mcp:shadowJar"); - return null; // unreachable - }); - } catch (IOException e) { - assumeTrue(false, "Could not list build/libs: " + e.getMessage()); - return null; // unreachable - } - } - - private static Path findTestJfr() { - // First try a custom override (useful for reproducing with specific recordings) + static Path findTestJfr() { String override = System.getProperty("mcp.e2e.jfr"); if (override != null) { - Path p = Path.of(override); - assumeTrue(Files.exists(p), "JFR file from mcp.e2e.jfr not found: " + override); - return p; - } - // Fall back to the repo's small test recording - Path p = Paths.get("../demo/src/test/resources/test-dd.jfr").normalize(); - if (!Files.exists(p)) { - p = Paths.get("demo/src/test/resources/test-dd.jfr").normalize(); - } - assumeTrue(Files.exists(p), "test-dd.jfr not found — ensure the demo module is present"); - return p; - } - - // ───────────────────────────────────────────────────────────────────────────── - // Background message reader - // ───────────────────────────────────────────────────────────────────────────── - - static final class MessageCollector { - - private final BlockingQueue messages = new LinkedBlockingQueue<>(); - private final Thread thread; - - MessageCollector(java.io.InputStream in) { - thread = - new Thread( - () -> { - try (BufferedReader reader = - new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - if (line.isBlank()) continue; - try { - messages.put(MAPPER.readTree(line)); - } catch (InterruptedException ie) { - Thread.currentThread().interrupt(); - break; - } catch (Exception ignored) { - } - } - } catch (Exception ignored) { - } - }, - "e2e-reader"); - thread.setDaemon(true); - thread.start(); - } - - JsonNode waitForId(int id, long timeoutMs) throws InterruptedException { - long deadline = System.currentTimeMillis() + timeoutMs; - List stash = new ArrayList<>(); - try { - while (System.currentTimeMillis() < deadline) { - long remaining = deadline - System.currentTimeMillis(); - JsonNode msg = messages.poll(remaining, TimeUnit.MILLISECONDS); - if (msg == null) break; - if (msg.has("id") && msg.get("id").asInt() == id) return msg; - stash.add(msg); - } - return null; - } finally { - messages.addAll(stash); + Path supplied = Path.of(override).toAbsolutePath().normalize(); + if (!Files.exists(supplied)) { + throw new IllegalArgumentException("JFR file from mcp.e2e.jfr not found: " + supplied); } + return supplied; } - - void stop() { - thread.interrupt(); + Path fixture = Paths.get("../demo/src/test/resources/test-dd.jfr").normalize(); + if (!Files.exists(fixture)) { + fixture = Paths.get("demo/src/test/resources/test-dd.jfr").normalize(); + } + if (!Files.exists(fixture)) { + throw new IllegalStateException("test-dd.jfr not found — ensure the demo module is present"); } + return fixture.toAbsolutePath(); } } diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpStdioProcessHarness.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpStdioProcessHarness.java new file mode 100644 index 00000000..c52d7324 --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpStdioProcessHarness.java @@ -0,0 +1,318 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.BufferedReader; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStreamWriter; +import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Map; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; +import tools.jackson.databind.node.ObjectNode; + +/** A real local shadow-JAR MCP client with per-test process and persistence isolation. */ +final class McpStdioProcessHarness implements AutoCloseable { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + private static final long TIMEOUT_MS = 30_000; + + private final Path root; + private final Path sessionsFile; + private final Path home; + private final Path stateDir; + private final Process process; + private final PrintWriter stdin; + private final BlockingQueue pending = new LinkedBlockingQueue<>(); + private final List observed = java.util.Collections.synchronizedList(new ArrayList<>()); + private final ByteArrayOutputStream stderr = new ByteArrayOutputStream(); + private final Thread stdoutReader; + private final Thread stderrReader; + private int nextId = 1; + + McpStdioProcessHarness(String testName) throws Exception { + Path parent = Path.of("build", "mcp-e2e").toAbsolutePath().normalize(); + Files.createDirectories(parent); + root = Files.createTempDirectory(parent, testName.replaceAll("[^A-Za-z0-9_.-]", "_") + "-"); + sessionsFile = root.resolve("sessions.json").normalize(); + home = root.resolve("home").normalize(); + stateDir = root.resolve("state").normalize(); + assertOwned(root, sessionsFile, home, stateDir); + Files.createDirectories(home); + Files.createDirectories(stateDir); + + String java = ProcessHandle.current().info().command().orElse("java"); + process = + new ProcessBuilder( + java, + "-Djafar.mcp.sessions.file=" + sessionsFile, + "-Duser.home=" + home, + // The child is a separate JVM, so the Gradle test JVM's isolation does not reach + // it. Belt and braces: an explicit state dir for daemon files (port marker, auth + // token, persisted sessions) next to the redirected home that is its default. + "-Djafar.state.dir=" + stateDir, + "-jar", + findShadowJar().toString(), + "--stdio") + .start(); + stdin = + new PrintWriter( + new OutputStreamWriter(process.getOutputStream(), StandardCharsets.UTF_8), true); + stdoutReader = new Thread(() -> readStdout(process.getInputStream()), "mcp-e2e-stdout"); + stderrReader = new Thread(() -> copyStderr(process.getErrorStream()), "mcp-e2e-stderr"); + stdoutReader.setDaemon(true); + stderrReader.setDaemon(true); + stdoutReader.start(); + stderrReader.start(); + initialize(); + } + + Path root() { + return root; + } + + Path sessionsFile() { + return sessionsFile; + } + + Path home() { + return home; + } + + Path stateDir() { + return stateDir; + } + + JsonNode call(String tool, Map arguments) throws Exception { + return call(tool, arguments, null); + } + + JsonNode call(String tool, Map arguments, Integer progressToken) + throws Exception { + int id = nextId++; + ObjectNode params = MAPPER.createObjectNode(); + params.put("name", tool); + params.set("arguments", MAPPER.valueToTree(arguments)); + if (progressToken != null) { + ObjectNode meta = params.putObject("_meta"); + meta.put("claudecode/toolUseId", "toolu_tier1_" + id); + meta.put("progressToken", progressToken); + } + send(id, "tools/call", params); + JsonNode response = awaitId(id); + assertNotNull(response, () -> tool + " did not respond; stderr:\n" + stderrText()); + assertFalse( + response.has("error"), + () -> tool + " JSON-RPC error: " + response + "\nstderr:\n" + stderrText()); + assertFalse( + response.at("/result/isError").asBoolean(), () -> tool + " tool error: " + response); + return response; + } + + JsonNode readResource(String uri) throws Exception { + int id = nextId++; + ObjectNode params = MAPPER.createObjectNode(); + params.put("uri", uri); + send(id, "resources/read", params); + JsonNode response = awaitId(id); + assertNotNull(response, () -> "resources/read did not respond; stderr:\n" + stderrText()); + assertFalse(response.has("error"), () -> "resources/read error: " + response); + return response; + } + + /** + * Proves this child starts with no inherited session state before a workflow opens its fixture. + * + *

The per-child JVM properties are defensive containment; this resource read proves the server + * actually honoured them rather than merely receiving private paths on its command line. + */ + void assertFreshSessions(String... aliasesThatMustBeAbsent) throws Exception { + assertOwned(root, sessionsFile, home, stateDir); + assertTrue(Files.exists(root), () -> "E2E root disappeared before session check: " + root); + JsonNode response = readResource("jafar://sessions"); + String sessions = contentText(response); + assertTrue( + sessions.contains("## JFR recordings\n_none open_") + && sessions.contains("## Heap dumps\n_none open_") + && sessions.contains("## pprof profiles\n_none open_") + && sessions.contains("## OTLP profiles\n_none open_"), + () -> "fresh child has inherited open sessions: " + sessions); + for (String alias : aliasesThatMustBeAbsent) { + assertFalse( + sessions.contains(alias), + () -> "fresh child leaked alias '" + alias + "' through jafar://sessions: " + sessions); + } + } + + JsonNode contentJson(JsonNode response) throws IOException { + JsonNode content = response.path("result").path("content"); + assertTrue( + content.isArray() && !content.isEmpty(), () -> "missing MCP result content: " + response); + String text = content.get(0).path("text").asString(); + assertFalse(text.isBlank(), () -> "empty MCP text payload: " + response); + return MAPPER.readTree(text); + } + + String contentText(JsonNode response) { + JsonNode contents = response.path("result").path("contents"); + assertTrue( + contents.isArray() && !contents.isEmpty(), + () -> "resources/read returned no content: " + response); + String text = contents.get(0).path("text").asString(); + assertFalse(text.isBlank(), () -> "resources/read returned empty text content: " + response); + return text; + } + + void assertProgress(int expectedToken) { + List progress = new ArrayList<>(); + synchronized (observed) { + for (JsonNode message : observed) { + if (message.has("method") + && "notifications/progress".equals(message.path("method").asString())) { + progress.add(message); + } + } + } + assertFalse( + progress.isEmpty(), + () -> "expected progress token " + expectedToken + "; observed=" + observed); + for (JsonNode notification : progress) { + JsonNode token = notification.at("/params/progressToken"); + assertTrue( + token.isIntegralNumber(), () -> "progress token must be integral: " + notification); + assertEquals(expectedToken, token.asInt(), () -> "wrong progress token: " + notification); + } + } + + private void initialize() throws Exception { + ObjectNode params = MAPPER.createObjectNode(); + params.put("protocolVersion", "2025-06-18"); + params.putObject("capabilities"); + params.putObject("clientInfo").put("name", "jafar-tier1-e2e").put("version", "1.0"); + send(0, "initialize", params); + JsonNode response = awaitId(0); + assertNotNull(response, () -> "MCP handshake failed; stderr:\n" + stderrText()); + assertFalse( + response.has("error"), + () -> "MCP handshake error: " + response + "\nstderr:\n" + stderrText()); + stdin.println("{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\",\"params\":{}}"); + } + + private void send(int id, String method, JsonNode params) throws IOException { + ObjectNode request = MAPPER.createObjectNode(); + request.put("jsonrpc", "2.0"); + request.put("id", id); + request.put("method", method); + request.set("params", params); + stdin.println(MAPPER.writeValueAsString(request)); + assertFalse(stdin.checkError(), () -> "could not write MCP request; stderr:\n" + stderrText()); + } + + private JsonNode awaitId(int id) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(TIMEOUT_MS); + List stashed = new ArrayList<>(); + try { + while (System.nanoTime() < deadline) { + JsonNode message = pending.poll(deadline - System.nanoTime(), TimeUnit.NANOSECONDS); + if (message == null) { + return null; + } + if (message.has("id") && message.path("id").asInt(Integer.MIN_VALUE) == id) { + return message; + } + stashed.add(message); + } + return null; + } finally { + pending.addAll(stashed); + } + } + + private void readStdout(java.io.InputStream stream) { + try (BufferedReader reader = + new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8))) { + for (String line; (line = reader.readLine()) != null; ) { + if (!line.isBlank()) { + JsonNode message = MAPPER.readTree(line); + observed.add(message); + pending.put(message); + } + } + } catch (Exception ignored) { + // Assertion failures include stderr and the messages received before the stream ended. + } + } + + private void copyStderr(java.io.InputStream stream) { + try (stream) { + stream.transferTo(stderr); + } catch (IOException ignored) { + } + } + + private String stderrText() { + return stderr.toString(StandardCharsets.UTF_8); + } + + private static Path findShadowJar() throws IOException { + Path candidate = Paths.get("build/libs"); + if (!Files.isDirectory(candidate)) { + candidate = Paths.get("jfr-mcp/build/libs"); + } + final Path libs = candidate; + try (var jars = Files.list(libs)) { + return jars.filter(path -> path.getFileName().toString().matches("jfr-mcp-.*-all\\.jar")) + .max(Comparator.comparing(Path::toString)) + .orElseThrow(() -> new IOException("no jfr-mcp shadow JAR in " + libs)); + } + } + + private static void assertOwned(Path root, Path sessionsFile, Path home, Path stateDir) { + Path normalizedRoot = root.toAbsolutePath().normalize(); + assertTrue(sessionsFile.toAbsolutePath().normalize().startsWith(normalizedRoot)); + assertTrue(home.toAbsolutePath().normalize().startsWith(normalizedRoot)); + assertTrue(stateDir.toAbsolutePath().normalize().startsWith(normalizedRoot)); + } + + @Override + public void close() throws Exception { + stdin.close(); + if (!process.waitFor(5, TimeUnit.SECONDS)) { + process.destroy(); + if (!process.waitFor(5, TimeUnit.SECONDS)) { + process.destroyForcibly(); + process.waitFor(5, TimeUnit.SECONDS); + } + } + stdoutReader.interrupt(); + stderrReader.interrupt(); + try (var paths = Files.walk(root)) { + paths + .sorted(Comparator.reverseOrder()) + .forEach( + path -> { + try { + Files.delete(path); + } catch (IOException e) { + throw new IllegalStateException("cannot remove E2E-owned path: " + path, e); + } + }); + } + assertFalse(Files.exists(root), () -> "E2E cleanup left owned directory: " + root); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/McpTransportHarness.java b/jfr-mcp/src/test/java/io/jafar/mcp/McpTransportHarness.java index 8e8a4cbd..2a6ed806 100644 --- a/jfr-mcp/src/test/java/io/jafar/mcp/McpTransportHarness.java +++ b/jfr-mcp/src/test/java/io/jafar/mcp/McpTransportHarness.java @@ -34,7 +34,7 @@ */ final class McpTransportHarness implements AutoCloseable { - static final long RESPONSE_TIMEOUT_MS = Long.getLong("mcp.test.timeout.ms", 15_000); + static final long RESPONSE_TIMEOUT_MS = Long.getLong("mcp.test.timeout.ms", 60_000); private static final Logger logger = LoggerFactory.getLogger(McpTransportHarness.class); private static final ObjectMapper MAPPER = new ObjectMapper(); diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/QueryExamples.java b/jfr-mcp/src/test/java/io/jafar/mcp/QueryExamples.java new file mode 100644 index 00000000..b2e2b747 --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/QueryExamples.java @@ -0,0 +1,332 @@ +package io.jafar.mcp; + +import java.util.ArrayList; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +/** + * Extracts query-looking examples from a guidance surface, the way an agent reads them. + * + *

Examples appear in three shapes: fenced code lines, backtick spans, and bare prose ("e.g., + * events/jdk.ExecutionSample | count()"). Anything a reader could reasonably copy as a query must + * parse — an example that does not is worse than no example, because it teaches every agent that + * follows it to fail. + * + *

Extraction is deliberately generous: a candidate that parses is harmless even if it was + * English ("objects" in a sentence), while the test only fails on candidates that look + * like queries but are rejected by the parser. Two kinds of look-alikes are skipped: placeholder + * grammar ({@code events/}) and optional-argument notation ({@code + * checkLeaks([detector="name"])}), which are syntax descriptions, not runnable queries. + */ +final class QueryExamples { + + record Candidate(String surface, String query, String origin) {} + + /** + * Roots that begin a runnable query per language, for the bare-text scan. JfrPath roots carry + * their slash so English words like "events" do not anchor; the sampling languages anchor on + * {@code samples} because that is their only root. + */ + private static final Map> INLINE_ROOTS = + Map.of( + GuidanceSurfaces.QueryLanguage.JFRPATH, + List.of("events/", "metadata/", "constants/"), + GuidanceSurfaces.QueryLanguage.HDUMPPATH, + List.of("objects", "classes", "gcroots", "clusters", "duplicates", "ages", "checkLeaks"), + GuidanceSurfaces.QueryLanguage.PPROFPATH, + List.of("samples"), + GuidanceSurfaces.QueryLanguage.OTLPPATH, + List.of("samples")); + + /** + * Roots accepted for fenced lines and backtick spans, where the whole span is the example. Adds + * bare JfrPath roots (e.g. {@code chunks}) that are too generic to anchor a prose scan for. + */ + private static final Map> DELIMITED_ROOTS = + Map.of( + GuidanceSurfaces.QueryLanguage.JFRPATH, + List.of("events/", "metadata/", "constants/", "chunks"), + GuidanceSurfaces.QueryLanguage.HDUMPPATH, + List.of("objects", "classes", "gcroots", "clusters", "duplicates", "ages", "checkLeaks"), + GuidanceSurfaces.QueryLanguage.PPROFPATH, + List.of("samples"), + GuidanceSurfaces.QueryLanguage.OTLPPATH, + List.of("samples")); + + /** Placeholder grammar like {@code } — a syntax description, not a query. */ + private static final Pattern PLACEHOLDER = Pattern.compile("<[A-Za-z_][A-Za-z0-9_ ]*>"); + + /** + * Grammar notation like {@code samples[predicate]} — a bracket naming what goes there rather than + * a runnable filter. Real filters always carry an operator or quoted value inside the brackets; + * type brackets like {@code int[]} are empty and never match this. + */ + private static final Pattern GRAMMAR_BRACKET = Pattern.compile("\\[[A-Za-z_][A-Za-z0-9_]*\\]"); + + private static final Pattern BACKTICK_SPAN = Pattern.compile("`([^`\\n]+)`"); + + private QueryExamples() {} + + /** All query candidates in {@code text}, deduplicated, in encounter order. */ + static List extract( + String surface, String text, GuidanceSurfaces.QueryLanguage language) { + Set out = new LinkedHashSet<>(); + for (String line : fencedLines(text)) { + String candidate = trimExample(line); + if (starts(candidate, DELIMITED_ROOTS.get(language))) { + addIfRunnable(out, surface, candidate, "fenced", language); + } + } + Matcher spans = BACKTICK_SPAN.matcher(text); + while (spans.find()) { + String candidate = trimExample(spans.group(1)); + if (starts(candidate, DELIMITED_ROOTS.get(language))) { + addIfRunnable(out, surface, candidate, "backtick", language); + } + } + scanInline(out, surface, text, language); + return List.copyOf(out); + } + + // ───────────────────────────────────────────────────────────────────────────── + // Fenced blocks + // ───────────────────────────────────────────────────────────────────────────── + + private static List fencedLines(String text) { + List lines = new ArrayList<>(); + boolean inFence = false; + for (String rawLine : text.split("\n", -1)) { + String line = rawLine.trim(); + if (line.startsWith("```")) { + inFence = !inFence; + continue; + } + if (inFence) { + // Annotated example lines end with two-plus spaces of prose ("samples[...] AND"); + // the queries themselves never contain a double space. + int annotation = line.indexOf(" "); + lines.add(annotation > 0 ? line.substring(0, annotation).trim() : line); + } + } + return lines; + } + + // ───────────────────────────────────────────────────────────────────────────── + // Bare-text scan + // ───────────────────────────────────────────────────────────────────────────── + + private static void scanInline( + Set out, String surface, String text, GuidanceSurfaces.QueryLanguage language) { + List roots = INLINE_ROOTS.get(language); + int from = 0; + while (from < text.length()) { + int start = findRoot(text, from, roots); + if (start < 0) { + return; + } + int end = scanQueryEnd(text, start, roots); + if (end < 0) { + from = start + 1; + continue; + } + String candidate = trimExample(text.substring(start, end)); + // A bare root word in prose is English ("objects and classes"); inline candidates must + // carry query structure — a path, filter, argument or pipeline — to count as examples. + boolean hasStructure = candidate.length() > rootLength(candidate, roots); + if (hasStructure) { + addIfRunnable(out, surface, candidate, "inline", language); + } + from = Math.max(end, start + 1); + } + } + + private static int findRoot(String text, int from, List roots) { + int best = -1; + for (String root : roots) { + int idx = text.indexOf(root, from); + if (idx >= 0 && (best < 0 || idx < best)) { + best = idx; + } + } + return best; + } + + private static int rootLength(String candidate, List roots) { + for (String root : roots) { + if (candidate.startsWith(root)) { + return root.length(); + } + } + return 0; + } + + /** + * Consumes a query-shaped substring starting at {@code start}: root, path segments, filters, root + * arguments and pipeline stages. Quotes protect their contents; stops at anything a query cannot + * continue with (a comma, a sentence period, a closing quote, a line break). + * + *

Returns {@code -1} when what follows the root cannot begin a query (a path segment that is + * missing after a slash — grammar notation such as {@code events/}), because the + * anchored text is not an example at all. + */ + private static int scanQueryEnd(String text, int start, List roots) { + int i = start + rootLength(text.substring(start), roots); + while (i < text.length()) { + char c = text.charAt(i); + if (c == '/') { + int segment = consumeSegment(text, i + 1); + if (segment == i + 1) { + return -1; // "root/" with nothing after: grammar fragment, not an example + } + i = segment; + } else if (c == '[' || c == '(') { + int balanced = consumeBalanced(text, i); + if (balanced < 0) { + break; // unbalanced bracket on this line: prose, not a query + } + i = balanced; + } else if (Character.isWhitespace(c)) { + int j = i; + while (j < text.length() && Character.isWhitespace(text.charAt(j))) { + j++; + } + if (j < text.length() && text.charAt(j) == '|' && pipelineOperatorFollows(text, j + 1)) { + i = j; + } else { + break; + } + } else if (c == '|' && pipelineOperatorFollows(text, i + 1)) { + i = consumePipelineStage(text, i + 1); + } else { + break; + } + } + return i; + } + + /** + * A pipeline stage must start with a bare operator name — otherwise the pipe is a table column + * separator or prose, and the candidate must end before it. + */ + private static boolean pipelineOperatorFollows(String text, int i) { + while (i < text.length() && Character.isWhitespace(text.charAt(i))) { + i++; + } + return i < text.length() && (Character.isLetter(text.charAt(i)) || text.charAt(i) == '_'); + } + + private static int consumeSegment(String text, int i) { + while (i < text.length() && isSegmentChar(text.charAt(i))) { + i++; + } + return i; + } + + private static boolean isSegmentChar(char c) { + return Character.isLetterOrDigit(c) || c == '_' || c == '.' || c == '$' || c == '*' || c == '?'; + } + + private static int consumePipelineStage(String text, int i) { + while (i < text.length() && Character.isWhitespace(text.charAt(i))) { + i++; + } + i = consumeSegment(text, i); + if (i < text.length() && text.charAt(i) == '(') { + i = consumeBalanced(text, i); + } + return i; + } + + /** + * Consumes balanced brackets or parentheses; quotes protect their contents. Returns -1 when the + * brackets do not close before the end of the line — a query never spans lines, so the anchor was + * prose. + */ + private static int consumeBalanced(String text, int i) { + char open = text.charAt(i); + char close = open == '[' ? ']' : ')'; + int depth = 0; + int endOfLine = text.indexOf('\n', i); + int limit = endOfLine < 0 ? text.length() : endOfLine; + while (i < limit) { + char c = text.charAt(i); + if (c == '"' || c == '\'') { + i = consumeQuoted(text, i, c); + continue; + } + if (c == open) { + depth++; + } else if (c == close) { + depth--; + if (depth == 0) { + return i + 1; + } + } + i++; + } + return -1; + } + + private static int consumeQuoted(String text, int i, char quote) { + i++; + while (i < text.length() && text.charAt(i) != quote) { + i++; + } + return Math.min(i + 1, text.length()); + } + + // ───────────────────────────────────────────────────────────────────────────── + // Candidate admission + // ───────────────────────────────────────────────────────────────────────────── + + private static boolean starts(String candidate, List roots) { + if (candidate == null || candidate.isBlank()) { + return false; + } + return roots.stream().anyMatch(candidate::startsWith); + } + + private static void addIfRunnable( + Set out, + String surface, + String candidate, + String origin, + GuidanceSurfaces.QueryLanguage language) { + if (candidate.isBlank() || PLACEHOLDER.matcher(candidate).find()) { + return; + } + if (GRAMMAR_BRACKET.matcher(candidate).find()) { + return; + } + // Optional-argument notation, e.g. checkLeaks([detector="name"]): "[" directly after "(" + // is grammar, not a runnable query. + if (candidate.contains("([")) { + return; + } + out.add(new Candidate(surface, candidate, origin)); + } + + /** Strips list markers and trailing sentence punctuation from a delimited example line. */ + private static String trimExample(String line) { + String s = line.trim(); + while (s.startsWith("-") || s.startsWith("*") || s.startsWith("#")) { + s = s.substring(1).trim(); + } + // Trailing sentence punctuation. Closing brackets and parens are always part of a query + // (objects/int[], count()) and are never stripped. + while (!s.isEmpty() + && (s.endsWith(".") + || s.endsWith(",") + || s.endsWith(";") + || s.endsWith(":") + || s.endsWith("\"") + || s.endsWith("`"))) { + s = s.substring(0, s.length() - 1).trim(); + } + return s; + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/ToolCatalogSnapshotTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/ToolCatalogSnapshotTest.java new file mode 100644 index 00000000..0ec61b40 --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/ToolCatalogSnapshotTest.java @@ -0,0 +1,110 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.api.parallel.ResourceLock; +import org.junit.jupiter.api.parallel.Resources; + +/** + * Pins the full MCP guidance contract — tool names, descriptions, input schemas, prompts and + * resources — to the committed {@code src/test/resources/mcp-contract.json}. + * + *

Tool descriptions are prompts: they are the only thing an agent routes by, and a change to one + * changes agent behaviour in ways no unit test notices. The snapshot turns every such change into a + * visible diff that must be consciously accepted, the same discipline applied to prompt text + * anywhere else. It is also the artifact the weekly Tool-drift check of the jafar-perf plugin + * (btraceio/agent-plugins, scripts/check-tool-references.js) diffs its skills against. + * + *

To accept a change, review the diff, update any skill file in btraceio/agent-plugins + * (plugins/jafar-perf) that names the changed tool or parameter, then run: + * + * 

./gradlew :jfr-mcp:test --tests ToolCatalogSnapshotTest -Dmcp.updateSnapshot=true
+ */ +class ToolCatalogSnapshotTest { + + private static final Path SNAPSHOT = Path.of("src/test/resources/mcp-contract.json"); + + @Test + @ResourceLock(Resources.SYSTEM_PROPERTIES) + void guidanceContractMatchesCommittedSnapshot() throws Exception { + String current; + try (GuidanceContractContext ignored = GuidanceContractContext.create()) { + current = McpContractJson.render(new GuidanceSurfaces()); + } + + if (Boolean.getBoolean("mcp.updateSnapshot")) { + String committed = Files.exists(SNAPSHOT) ? Files.readString(SNAPSHOT) : ""; + if (!current.equals(committed)) { + Files.createDirectories(SNAPSHOT.getParent()); + Files.writeString(SNAPSHOT, current); + System.out.println("Updated " + SNAPSHOT + " — review the diff before committing."); + } + return; + } + + assertTrue( + Files.exists(SNAPSHOT), + "snapshot missing: " + SNAPSHOT + " — run this test with -Dmcp.updateSnapshot=true"); + String committed = Files.readString(SNAPSHOT); + + assertEquals( + McpContractJson.parse(committed), + McpContractJson.parse(current), + () -> + "The MCP guidance contract changed. Every MCP client and the agent-plugins jafar-perf" + + " skills experience this diff. Review it, update any jafar-perf skill that names the" + + " changed tool or parameter, then accept it with" + + " -Dmcp.updateSnapshot=true.\n" + + previewDiff(committed, current)); + } + + @Test + @ResourceLock(Resources.SYSTEM_PROPERTIES) + void exportedContractMatchesCommittedSnapshot(@TempDir Path tempDir) throws Exception { + Path output = tempDir.resolve("mcp-contract.json"); + McpContractExport.main(new String[] {output.toString()}); + + assertEquals( + McpContractJson.parse(Files.readString(SNAPSHOT)), + McpContractJson.parse(Files.readString(output)), + "exportMcpContract must write the same semantic artifact as the committed snapshot"); + } + + private static String previewDiff(String committed, String current) { + // The full contract is large; surface WHICH entries differ (tools, prompts, resources) so the + // failure message names them without scrolling through the whole JSON. + var committedByKind = new java.util.HashMap(); + var committedTree = McpContractJson.parse(committed); + for (var kind : java.util.List.of("tools", "prompts", "resources")) { + var key = "tools".equals(kind) ? "name" : ("resources".equals(kind) ? "uri" : "name"); + for (var entry : committedTree.path(kind)) { + committedByKind.put(kind + ":" + entry.path(key).asString(), entry.toString()); + } + } + var changed = new java.util.ArrayList(); + var currentTree = McpContractJson.parse(current); + for (var kind : java.util.List.of("tools", "prompts", "resources")) { + var key = "tools".equals(kind) ? "name" : ("resources".equals(kind) ? "uri" : "name"); + for (var entry : currentTree.path(kind)) { + String name = kind + ":" + entry.path(key).asString(); + String old = committedByKind.remove(name); + if (old == null) { + changed.add(name + " (added)"); + } else if (!entry.toString().equals(old)) { + changed.add(name + " (modified)"); + } + } + } + for (String name : new java.util.TreeSet<>(committedByKind.keySet())) { + changed.add(name + " (removed)"); + } + return changed.isEmpty() + ? "no top-level entry differs by toString comparison — compare raw JSON" + : "changed entries: " + String.join(", ", changed); + } +} diff --git a/jfr-mcp/src/test/java/io/jafar/mcp/ToolDescriptionQualityTest.java b/jfr-mcp/src/test/java/io/jafar/mcp/ToolDescriptionQualityTest.java new file mode 100644 index 00000000..2da53b73 --- /dev/null +++ b/jfr-mcp/src/test/java/io/jafar/mcp/ToolDescriptionQualityTest.java @@ -0,0 +1,208 @@ +package io.jafar.mcp; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +import io.modelcontextprotocol.spec.McpSchema; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.junit.jupiter.api.Test; + +/** + * Contracts the guidance silently relies on: descriptions an agent routes by, and error messages an + * agent recovers by. + * + *

An agent picking a tool sees only its name and description. When {@code hdump_open} does not + * say it takes a {@code .hprof} — or when {@code jfr_open}'s description could plausibly cover a + * heap dump — routing becomes guesswork, and a wrong guess costs a failed call plus a recovery + * round-trip. The routing assertions keep the four families disambiguated by extension. + * + *

Recovery is guided by error text. The no-session errors are the load-bearing case: a client + * that calls an analysis tool before opening a file is the most common first-session mistake, and + * the only thing that turns it into a successful second call is an error that names the tool to + * call ("Use jfr_open first"). That contract is pinned here so a refactor of the session registries + * cannot quietly replace it with a bare "no session". + */ +class ToolDescriptionQualityTest { + + private static final Map> FAMILY_EXTENSIONS = + Map.of( + "jfr", List.of(".jfr"), + "hdump", List.of(".hprof"), + "pprof", List.of(".pprof", ".pb.gz"), + "otlp", List.of(".otlp")); + + /** Tools whose input-schema parameter surface is pinned, with the expected property names. */ + private static final Map> EXPECTED_PROPERTIES = + Map.of( + "jfr_open", Set.of("path", "alias"), + "hdump_open", Set.of("path", "alias"), + "pprof_open", Set.of("path", "alias"), + "otlp_open", Set.of("path", "alias"), + "jfr_query", Set.of("query", "sessionId", "limit"), + "hdump_query", Set.of("query", "sessionId", "limit"), + "pprof_query", Set.of("query", "sessionId", "limit"), + "otlp_query", Set.of("query", "sessionId", "limit")); + + @Test + void everyToolHasASubstantialDescription() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + List problems = new java.util.ArrayList<>(); + for (var tool : surfaces.tools()) { + String name = tool.tool().name(); + String description = tool.tool().description(); + if (description == null || description.isBlank()) { + problems.add(name + ": blank description"); + continue; + } + // Too short gives an agent nothing to route by; the ceiling only catches accidental + // wall-of-text regressions, since long descriptions burn client context for every call. + if (description.length() < 40) { + problems.add( + name + ": description is only " + description.length() + " chars: " + description); + } + if (description.length() > 4000) { + problems.add(name + ": description is " + description.length() + " chars — too long"); + } + } + assertTrue( + problems.isEmpty(), () -> "tool description problems:%n" + String.join("%n", problems)); + } + + @Test + void openToolsNameTheirFileExtensionsAndNoOthers() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + List problems = new java.util.ArrayList<>(); + for (var tool : surfaces.tools()) { + String name = tool.tool().name(); + String description = tool.tool().description(); + if (description == null) { + description = ""; + } + String ownFamily = familyOf(name); + // Only the *_open tools route by extension; asserting on all descriptions would forbid a + // cross-family tool from legitimately mentioning another format (e.g. correlation). + if (!name.endsWith("_open")) { + continue; + } + for (String extension : FAMILY_EXTENSIONS.get(ownFamily)) { + if (!description.contains(extension)) { + problems.add(name + " does not mention its extension " + extension); + } + } + for (Map.Entry> rival : FAMILY_EXTENSIONS.entrySet()) { + if (rival.getKey().equals(ownFamily)) { + continue; + } + for (String extension : rival.getValue()) { + if (description.contains(extension)) { + problems.add( + name + + " mentions " + + extension + + " (" + + rival.getKey() + + " family) — routing must stay unambiguous"); + } + } + } + } + assertTrue(problems.isEmpty(), () -> "routing problems:%n" + String.join("%n", problems)); + } + + @Test + void coreToolParameterSurfacesMatchTheirSchemas() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + List problems = new java.util.ArrayList<>(); + for (var tool : surfaces.tools()) { + String name = tool.tool().name(); + Set expected = EXPECTED_PROPERTIES.get(name); + if (expected == null) { + continue; + } + Set actual = schemaProperties(tool); + if (!expected.equals(actual)) { + problems.add(name + ": schema properties " + actual + " != expected " + expected); + } + } + assertTrue( + problems.isEmpty(), + () -> + "tool parameter surface drifted from the pinned contract (update the pin deliberately" + + " when adding a parameter, and remember the agent-plugins jafar-perf skills name parameters" + + " too):%n" + + String.join("%n", problems)); + } + + @Test + void noSessionErrorsNameTheToolToCallInstead() { + GuidanceSurfaces surfaces = new GuidanceSurfaces(); + List problems = new java.util.ArrayList<>(); + for (String tool : List.of("jfr_query", "hdump_query", "pprof_query", "otlp_query")) { + McpSchema.CallToolResult result = call(surfaces, tool, Map.of("query", "events | count()")); + String text = errorText(result); + String remedy = familyOf(tool) + "_open"; + if (!Boolean.TRUE.equals(result.isError()) || text == null) { + problems.add(tool + " with no session open must return an error"); + } else if (!text.contains(remedy)) { + problems.add( + tool + " no-session error must name " + remedy + " (the call that fixes it): " + text); + } + } + assertTrue( + problems.isEmpty(), () -> "recovery-guidance problems:%n" + String.join("%n", problems)); + } + + // ───────────────────────────────────────────────────────────────────────────── + // Helpers + // ───────────────────────────────────────────────────────────────────────────── + + private static String familyOf(String toolName) { + return toolName.substring(0, toolName.indexOf('_')); + } + + private static McpSchema.CallToolResult call( + GuidanceSurfaces surfaces, String tool, Map args) { + for (var spec : surfaces.tools()) { + if (spec.tool().name().equals(tool)) { + Map safeArgs = new LinkedHashMap<>(args); + return spec.callHandler().apply(null, new McpSchema.CallToolRequest(tool, safeArgs)); + } + } + throw new IllegalArgumentException("no such tool: " + tool); + } + + private static String errorText(McpSchema.CallToolResult result) { + var content = result.content(); + if (content == null || content.isEmpty()) { + return null; + } + return content.get(0) instanceof McpSchema.TextContent text ? text.text() : null; + } + + private static Set schemaProperties( + io.modelcontextprotocol.server.McpServerFeatures.SyncToolSpecification tool) { + Map properties; + try { + @SuppressWarnings("unchecked") + Map parsed = + (Map) + new tools.jackson.databind.ObjectMapper() + .readValue( + io.modelcontextprotocol.json.McpJsonDefaults.getMapper() + .writeValueAsString(tool.tool().inputSchema()), + Map.class); + properties = parsed; + } catch (Exception e) { + throw new IllegalStateException("cannot read schema of " + tool.tool().name(), e); + } + Set names = new HashSet<>(); + Object props = properties.get("properties"); + if (props instanceof Map map) { + map.keySet().forEach(k -> names.add(String.valueOf(k))); + } + return names; + } +} diff --git a/jfr-mcp/src/test/resources/mcp-contract.json b/jfr-mcp/src/test/resources/mcp-contract.json new file mode 100644 index 00000000..b3582c6c --- /dev/null +++ b/jfr-mcp/src/test/resources/mcp-contract.json @@ -0,0 +1,802 @@ +{ + "tools" : [ { + "name" : "hdump_close", + "description" : "Closes one or all open heap dump sessions.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias to close; omit to close the current session" + }, + "closeAll" : { + "type" : "boolean", + "description" : "If true, close all of your open heap dump sessions and any shared ones restored after a restart (never another client's)" + } + } + } + }, { + "name" : "hdump_help", + "description" : "HdumpPath query language documentation. Topics: overview, roots, filters, operators, examples, patterns, tools. Call hdump_help(topic=\"tools\") to see all hdump_* MCP tools.", + "inputSchema" : { + "type" : "object", + "properties" : { + "topic" : { + "type" : "string", + "description" : "Help topic: overview, roots, filters, operators, examples, patterns, tools. Omit for full reference.", + "enum" : [ "overview", "roots", "filters", "operators", "examples", "patterns", "tools" ] + } + } + } + }, { + "name" : "hdump_open", + "description" : "Opens an HPROF heap dump file (.hprof) for analysis. Returns a session ID used by other hdump_* tools. If no sessionId is supplied to other tools, the most recently opened session is used.", + "inputSchema" : { + "type" : "object", + "properties" : { + "path" : { + "type" : "string", + "description" : "Absolute path to the HPROF heap dump file" + }, + "alias" : { + "type" : "string", + "description" : "Optional human-readable alias for the session" + } + }, + "required" : [ "path" ] + } + }, { + "name" : "hdump_query", + "description" : "Executes an HdumpPath query against an open heap dump session.\n\nQUERY SYNTAX\n [/][] [| ]*\n\nROOT TYPES\n objects — heap object instances\n fields: id, class, shallow, retained, arrayLength, stringValue\n classes — class metadata\n fields: id, name, simpleName, instanceCount, instanceSize, superClass, isArray\n gcroots — GC root references\n fields: type, objectId, object, shallow, retained\n types: JNI_GLOBAL, JNI_LOCAL, JAVA_FRAME, NATIVE_STACK, STICKY_CLASS,\n THREAD_BLOCK, MONITOR_USED, THREAD_OBJ\n clusters — graph-based leak clusters (expensive first time)\n fields: id, objectCount, retainedSize, score, dominantClass, anchorType\n duplicates — structurally-identical subgraphs; depth=N controls fingerprint depth (default 3)\n fields: id, rootClass, copies, uniqueSize, wastedBytes, depth, nodeCount\n ages — objects enriched with estimated age score\n fields: all object fields + estimatedAge, ageBucket, ageSignals\n\nTYPE SPECIFIERS (after /)\n objects/java.lang.String\n objects/java.util.* (glob)\n objects/instanceof/java.util.Map (subtypes)\n objects/int[] or objects/[I (arrays)\n\nPREDICATES [field op value]\n ops: = != > >= < <= ~(regex)\n boolean: and, or, not, ()\n size units: K KB M MB G GB\n functions: contains(f,\"s\") startsWith(f,\"p\") between(f,lo,hi) exists(f)\n examples: [shallow > 1MB] [class = \"java.lang.String\"]\n\nPIPELINE OPERATORS (chained with |)\n top(n [,field] [,asc|desc]) sortBy(field [asc|desc])\n filter(predicate) head(n) tail(n)\n groupBy(field [,agg=count|sum|avg|min|max] [,value=expr])\n count sum(field) stats(field) select(f1, f2 as alias, ...) distinct(field)\n waste() — Map/List capacity waste: capacity, size, wastedBytes\n cacheStats() — Map cache stats: entryCount, fillRatio, costPerEntry, isLruMode\n checkLeaks([detector=\"name\"] [,minSize=N])\n — detectors: threadlocal-leak classloader-leak duplicate-strings\n growing-collections listener-leak finalizer-queue\n pathToRoot() — trace object to nearest GC root\n retentionPaths() — merged class-level retention paths\n retainedBreakdown([depth=N]) — dominator subtree by class\n dominators([groupBy=\"class\"|\"package\"] [,minRetained=size])\n threadOwner() — enrich with ownerThread, ownership\n dominatedSize() — for THREAD_OBJ roots: threadName, dominated, dominatedCount\n estimateAge() — estimatedAge, ageBucket, ageSignals\n whatif() — simulate removal: freedBytes, freedPct, remainingRetained\n join(session=id|alias [,root=\"eventType\", by=field])\n — heap diff (two hprof) or JFR correlation\n\nNOTE: retained-size queries trigger approximate retained-size computation on first use.\nFor large heaps (>500K objects) this may take several minutes.\n\nCOMMON PATTERNS\n Top 10 memory hogs: objects | top(10, retained)\n Duplicate waste: duplicates | sortBy(wastedBytes desc) | top(20)\n Leak clusters: clusters | sortBy(score desc) | head(10)\n String duplicates: checkLeaks(detector=\"duplicate-strings\")\n Thread memory: gcroots/THREAD_OBJ | dominatedSize() | sortBy(dominated desc)\n Collection waste: objects/instanceof/java.util.HashMap | waste() | top(20, wastedBytes)\n Heap diff: classes | join(session=1) | sortBy(instanceCountDelta desc)\n Retention paths: objects[retained > 100MB] | retentionPaths()", + "inputSchema" : { + "type" : "object", + "properties" : { + "query" : { + "type" : "string", + "description" : "HdumpPath query string" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum rows to return (default: 100)" + } + }, + "required" : [ "query" ] + } + }, { + "name" : "hdump_report", + "description" : "Full heap health report. Triggers retained-size computation, leak detection, and duplicate subgraph analysis. May take several minutes on large heaps. Returns severity-ranked findings (CRITICAL > WARNING > INFO) with suggested queries. Use 'focus' to limit analyses: leaks, waste, duplicates, histogram.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "focus" : { + "type" : "string", + "description" : "Comma-separated subset of analyses to run: leaks, waste, duplicates, histogram. Omit to run all." + }, + "format" : { + "type" : "string", + "description" : "Output format: text (default) or markdown", + "enum" : [ "text", "markdown" ] + } + } + } + }, { + "name" : "hdump_summary", + "description" : "Quick overview of an open heap dump. Does NOT trigger retained-size computation. Returns object/class counts, heap size, top classes by instance count, and GC root types.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + } + } + } + }, { + "name" : "jfr_callgraph", + "description" : "Generates a call graph showing caller-callee relationships from stack traces. Unlike flamegraph (which preserves full paths), this shows which methods call which, revealing convergence points where multiple callers invoke the same method. DOT format can be visualized with graphviz. JSON format includes node and edge data.", + "inputSchema" : { + "type" : "object", + "properties" : { + "eventType" : { + "type" : "string", + "description" : "Event type to analyze (e.g., jdk.ExecutionSample, jdk.ObjectAllocationSample)" + }, + "format" : { + "type" : "string", + "description" : "Output format: dot (graphviz) or json", + "enum" : [ "dot", "json" ], + "default" : "dot" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "minWeight" : { + "type" : "integer", + "description" : "Minimum edge weight to include (default: 1)" + } + }, + "required" : [ "eventType" ] + } + }, { + "name" : "jfr_close", + "description" : "Closes a JFR recording session and releases resources. Options: provide sessionId for specific session, closeAll=true for all of your sessions, or neither to close the current session.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias to close" + }, + "closeAll" : { + "type" : "boolean", + "description" : "Close all of your open sessions and any shared ones restored after a restart (never another client's)" + } + } + } + }, { + "name" : "jfr_compare", + "description" : "Compares a candidate JFR recording against a baseline and reports what changed: event rates per second, GC and exception rates, and per-frame CPU self-time shares. Normalises for recording duration and sampling rate, flags changes below the noise floor as insignificant, and warns when the two recordings are not comparable. Use for before/after regression checks; open both recordings with jfr_open first.", + "inputSchema" : { + "type" : "object", + "properties" : { + "baselineSessionId" : { + "type" : "string", + "description" : "Session ID or alias of the baseline (the 'before' recording). Required." + }, + "candidateSessionId" : { + "type" : "string", + "description" : "Session ID or alias of the candidate (the 'after' recording). Defaults to the current session." + }, + "eventType" : { + "type" : "string", + "description" : "Execution sample event type. Auto-detected per recording when omitted." + }, + "minDeltaPct" : { + "type" : "number", + "description" : "Noise floor in percentage points for per-frame changes (default: 1.0)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of changed frames to return (default: 25)" + } + }, + "required" : [ "baselineSessionId" ] + } + }, { + "name" : "jfr_diagnose", + "description" : "Diagnoses performance issues in a JFR recording by running the appropriate analyses and merging their results. Covers exception rates, GC pressure, CPU hotspots, resource bottlenecks (USE) and thread states (TSA), and returns severity-ranked structured findings plus the capability gaps that limit what this recording can answer. Use this as the first step on an unfamiliar recording; pass depth=quick to skip the USE and TSA passes on very large files.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "includeAnalysis" : { + "type" : "boolean", + "description" : "Include full analysis results from triggered tools (default: true)" + }, + "depth" : { + "type" : "string", + "enum" : [ "quick", "full" ], + "description" : "quick = summary-derived thresholds only; full (default) also runs the USE and TSA analyses in-process and merges their findings" + } + } + } + }, { + "name" : "jfr_exceptions", + "description" : "Analyzes exception events in a JFR recording. Extracts exception types from stack traces, groups by exception class, and identifies throw sites. Works with both JDK exception events (jdk.JavaExceptionThrow) and profiler exception samples (datadog.ExceptionSample). Returns exception type counts, throw site locations, and patterns.", + "inputSchema" : { + "type" : "object", + "properties" : { + "eventType" : { + "type" : "string", + "description" : "Exception event type (e.g., datadog.ExceptionSample, jdk.JavaExceptionThrow). Auto-detects if not specified." + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "minCount" : { + "type" : "integer", + "description" : "Minimum exception count to include in results (default: 1)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of exception types to return (default: 50)" + } + } + } + }, { + "name" : "jfr_flamegraph", + "description" : "Generates aggregated stack trace data for flamegraph-style analysis. Returns stack paths with sample counts in folded or tree format. Use direction=bottom-up to see hot methods (where time is spent), or direction=top-down to see call paths from entry points. Folded format is semicolon-separated paths compatible with standard flamegraph tools.", + "inputSchema" : { + "type" : "object", + "properties" : { + "eventType" : { + "type" : "string", + "description" : "Event type to analyze (e.g., jdk.ExecutionSample, jdk.ObjectAllocationSample)" + }, + "direction" : { + "type" : "string", + "description" : "Stack direction: bottom-up (hot methods at root) or top-down (entry points at root)", + "enum" : [ "bottom-up", "top-down" ], + "default" : "bottom-up" + }, + "format" : { + "type" : "string", + "description" : "Output format: folded (semicolon-separated) or tree (JSON)", + "enum" : [ "folded", "tree" ], + "default" : "folded" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "minSamples" : { + "type" : "integer", + "description" : "Minimum sample count to include in output (default: 1)" + }, + "maxDepth" : { + "type" : "integer", + "description" : "Maximum stack depth to include (default: unlimited)" + } + }, + "required" : [ "eventType" ] + } + }, { + "name" : "jfr_help", + "description" : "Returns JfrPath query language documentation. Call without arguments for overview, or specify a topic for detailed help. Topics: overview, filters, pipeline, functions, examples, event_types, tools.", + "inputSchema" : { + "type" : "object", + "properties" : { + "topic" : { + "type" : "string", + "description" : "Help topic: overview, filters, pipeline, functions, examples, event_types, tools", + "enum" : [ "overview", "filters", "pipeline", "functions", "examples", "event_types", "tools" ] + } + } + } + }, { + "name" : "jfr_hotmethods", + "description" : "Returns the hottest methods (leaf frames) from CPU profiling samples. Simpler and more compact than full flamegraph - just shows which methods are consuming CPU. Useful for quick CPU hotspot identification.", + "inputSchema" : { + "type" : "object", + "properties" : { + "eventType" : { + "type" : "string", + "description" : "Execution sample event type (e.g., datadog.ExecutionSample, jdk.ExecutionSample). Auto-detects if not specified." + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of methods to return (default: 20)" + }, + "includeNative" : { + "type" : "boolean", + "description" : "Include native/VM methods (default: true)" + } + } + } + }, { + "name" : "jfr_list_types", + "description" : "Lists all event types available in a JFR recording. By default returns type names from metadata (fast). Use scan=true to get actual event counts (scans the entire recording). Categories: jdk.* (JDK events), jdk.jfr.* (JFR infrastructure), custom app events.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "filter" : { + "type" : "string", + "description" : "Filter for event type names (case-insensitive)" + }, + "scan" : { + "type" : "boolean", + "description" : "Scan recording to get accurate event counts (may be slow for large files)" + } + } + } + }, { + "name" : "jfr_open", + "description" : "Opens a JFR (Java Flight Recording) file (.jfr) for analysis. Returns a session ID that can be used with other jfr_* tools. If no session ID is provided to other tools, they use the most recently opened session.", + "inputSchema" : { + "type" : "object", + "properties" : { + "path" : { + "type" : "string", + "description" : "Absolute path to the JFR recording file" + }, + "alias" : { + "type" : "string", + "description" : "Optional alias for the session" + } + }, + "required" : [ "path" ] + } + }, { + "name" : "jfr_query", + "description" : "Executes a JfrPath query against a JFR recording. JfrPath forms: events/, events/[filter], events/ | pipeline. Common types: jdk.ExecutionSample, jdk.GCPhasePause, jdk.MonitorEnter, jdk.FileRead. Pipeline ops: count(), groupBy(field), top(n), select(f1,f2), stats(field). Examples: events/jdk.ExecutionSample | count(), events/jdk.GCPhasePause | top(10)", + "inputSchema" : { + "type" : "object", + "properties" : { + "query" : { + "type" : "string", + "description" : "JfrPath query expression" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum results to return (default: 100)" + } + }, + "required" : [ "query" ] + } + }, { + "name" : "jfr_stackprofile", + "description" : "Returns structured stack profiling data with time-series distribution and per-thread breakdown for each frame. Unlike jfr_flamegraph (which returns aggregated stack paths for visualization), this tool returns machine-readable JSON with: (1) raw time-bucket arrays showing how each method's samples distribute over the recording duration — use this to detect bursty vs steady hotspots and N+1 query patterns; (2) per-thread sample counts revealing thread affinity; (3) numeric percentage fields and a derived category (normal/hotspot/steady-hotspot). Choose jfr_stackprofile when you need to programmatically analyze CPU behavior over time or across threads. Choose jfr_flamegraph when you need aggregated call-path data for visualization or simple hotspot listing.", + "inputSchema" : { + "type" : "object", + "properties" : { + "eventType" : { + "type" : "string", + "description" : "Execution sample event type (e.g., jdk.ExecutionSample, datadog.ExecutionSample). Auto-detects if not specified." + }, + "direction" : { + "type" : "string", + "description" : "Stack direction: top-down (entry points first) or bottom-up (hot methods first)", + "enum" : [ "top-down", "bottom-up" ], + "default" : "top-down" + }, + "buckets" : { + "type" : "integer", + "description" : "Number of time buckets for temporal distribution (default: 10)" + }, + "minPct" : { + "type" : "number", + "description" : "Minimum percentage threshold to include a frame (default: 1.0)" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of frames to return (default: 200)" + } + } + } + }, { + "name" : "jfr_summary", + "description" : "Provides a quick overview of a JFR recording including duration, event counts, and key highlights like GC statistics, exception rates, and top CPU consumers. Useful for getting oriented with a new recording before deeper analysis.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + } + } + } + }, { + "name" : "jfr_tsa", + "description" : "Analyzes JFR recording using Thread State Analysis (TSA) methodology. Shows how threads spend their time across different states (RUNNABLE, WAITING, BLOCKED, etc.). Identifies problematic threads and correlates blocking states with contended locks/monitors.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "startTime" : { + "type" : "integer", + "description" : "Start time in nanoseconds from recording start (optional)" + }, + "endTime" : { + "type" : "integer", + "description" : "End time in nanoseconds from recording start (optional)" + }, + "topThreads" : { + "type" : "integer", + "description" : "Number of top threads to analyze per state (default: 10)" + }, + "minSamples" : { + "type" : "integer", + "description" : "Minimum samples for a thread to be included (default: 5)" + }, + "correlateBlocking" : { + "type" : "boolean", + "description" : "Correlate blocking states with lock/monitor events (default: true)" + }, + "includeInsights" : { + "type" : "boolean", + "description" : "Include actionable insights and recommendations (default: true)" + } + } + } + }, { + "name" : "jfr_use", + "description" : "Analyzes JFR recording using Brendan Gregg's USE Method (Utilization, Saturation, Errors). Examines CPU, Memory, Threads/Locks, and I/O resources to identify bottlenecks. Returns metrics for utilization (how busy), saturation (queued work), and errors for each resource.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias (uses current if not specified)" + }, + "startTime" : { + "type" : "integer", + "description" : "Start time in nanoseconds from recording start (optional)" + }, + "endTime" : { + "type" : "integer", + "description" : "End time in nanoseconds from recording start (optional)" + }, + "resources" : { + "type" : "array", + "items" : { + "type" : "string", + "enum" : [ "cpu", "memory", "threads", "io", "all" ] + }, + "description" : "Which resources to analyze (default: all)" + }, + "includeInsights" : { + "type" : "boolean", + "description" : "Include actionable insights and recommendations (default: true)" + } + } + } + }, { + "name" : "otlp_close", + "description" : "Closes an otlp session and releases its resources.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias to close. Omit to close the current session." + } + } + } + }, { + "name" : "otlp_flamegraph", + "description" : "Produces folded stack profile data suitable for flame graph visualization. Returns rows of {stack: 'root;parent;leaf', : N} sorted by value descending. Stack frames are separated by ';' with root frame first.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "valueField" : { + "type" : "string", + "description" : "Value type to aggregate (e.g. 'cpu'). Defaults to first sample type." + }, + "filter" : { + "type" : "string", + "description" : "Optional filter predicate, e.g. \"thread='main'\"" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of stack entries to return (default: 50)" + } + } + } + }, { + "name" : "otlp_help", + "description" : "Returns OtlpPath query language documentation and examples.", + "inputSchema" : { + "type" : "object", + "properties" : { } + } + }, { + "name" : "otlp_open", + "description" : "Opens an OpenTelemetry profiling file (.otlp) for analysis. Returns a session ID and profile metadata (sample type, duration, counts). Use the returned session ID with other otlp_* tools.", + "inputSchema" : { + "type" : "object", + "properties" : { + "path" : { + "type" : "string", + "description" : "Absolute path to the OTLP profiles file (.otlp)" + }, + "alias" : { + "type" : "string", + "description" : "Optional alias for the session" + } + }, + "required" : [ "path" ] + } + }, { + "name" : "otlp_query", + "description" : "Executes an OtlpPath query against an open OTLP profile. Query syntax: 'samples[predicate] | operator(args)'. Examples: 'samples | count()', 'samples | top(10, cpu)', 'samples | groupBy(thread, sum(cpu))', 'samples[thread='main'] | head(5)'. Use otlp_help for full query language reference.", + "inputSchema" : { + "type" : "object", + "properties" : { + "query" : { + "type" : "string", + "description" : "OtlpPath query string, e.g. 'samples | top(10, cpu)'" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of results to return (default: 100)" + } + }, + "required" : [ "query" ] + } + }, { + "name" : "otlp_summary", + "description" : "Returns a quick overview of an OTLP profile: sample type, counts, duration, and top functions by the primary value type.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + } + } + } + }, { + "name" : "otlp_use", + "description" : "Analyzes an OTLP profile using Brendan Gregg's USE Method (Utilization, Saturation, Errors). CPU: utilization stats and top consuming functions. Threads: distribution across threads to identify serial bottlenecks. Errors: heuristic scan of hot function names for error-related patterns.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "resources" : { + "type" : "array", + "items" : { + "type" : "string", + "enum" : [ "cpu", "threads", "errors" ] + }, + "description" : "Resources to analyze. Defaults to all: cpu, threads, errors." + } + } + } + }, { + "name" : "pprof_close", + "description" : "Closes a pprof session and releases its resources.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias to close. Omit to close the current session." + } + } + } + }, { + "name" : "pprof_flamegraph", + "description" : "Produces folded stack profile data suitable for flame graph visualization. Returns rows of {stack: 'root;parent;leaf', : N} sorted by value descending. Stack frames are separated by ';' with root frame first.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "valueField" : { + "type" : "string", + "description" : "Value type to aggregate (e.g. 'cpu', 'alloc_space'). Defaults to first sample type." + }, + "filter" : { + "type" : "string", + "description" : "Optional filter predicate, e.g. \"thread='main'\"" + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of stack entries to return (default: 50)" + } + } + } + }, { + "name" : "pprof_help", + "description" : "Returns PprofPath query language documentation and examples.", + "inputSchema" : { + "type" : "object", + "properties" : { } + } + }, { + "name" : "pprof_hotmethods", + "description" : "Identifies the hottest methods by direct (leaf-frame) cost. Groups samples by the leaf function name and sums the chosen value type. Shows what each function is directly responsible for (not including callees). Use pprof_flamegraph for inclusive (total) cost breakdown.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "topN" : { + "type" : "integer", + "description" : "Number of top methods to return (default: 20)" + }, + "valueField" : { + "type" : "string", + "description" : "Value type to rank by (e.g. 'cpu', 'alloc_objects'). Defaults to first sample type." + } + } + } + }, { + "name" : "pprof_open", + "description" : "Opens a pprof profile file (.pb.gz or .pprof) for analysis. Returns a session ID and profile metadata (sample types, duration, counts). Use the returned session ID with other pprof_* tools.", + "inputSchema" : { + "type" : "object", + "properties" : { + "path" : { + "type" : "string", + "description" : "Absolute path to the pprof profile file (.pb.gz or .pprof)" + }, + "alias" : { + "type" : "string", + "description" : "Optional alias for the session" + } + }, + "required" : [ "path" ] + } + }, { + "name" : "pprof_query", + "description" : "Executes a PprofPath query against an open pprof profile. Query syntax: 'samples[predicate] | operator(args)'. Examples: 'samples | count()', 'samples | top(10, cpu)', 'samples | groupBy(thread, sum(cpu))', 'samples[thread='main'] | head(5)'. Use pprof_help for full query language reference.", + "inputSchema" : { + "type" : "object", + "properties" : { + "query" : { + "type" : "string", + "description" : "PprofPath query string, e.g. 'samples | top(10, cpu)'" + }, + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "limit" : { + "type" : "integer", + "description" : "Maximum number of results to return (default: 100)" + } + }, + "required" : [ "query" ] + } + }, { + "name" : "pprof_summary", + "description" : "Returns a quick overview of a pprof profile: sample types, counts, duration, and top functions by the primary value type.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + } + } + } + }, { + "name" : "pprof_tsa", + "description" : "Thread State Analysis for pprof profiles. Shows how threads distribute their samples and infers thread states (RUNNING, WAITING, IO_BLOCKED, LOCK_BLOCKED) from leaf function names. Useful for identifying serial bottlenecks, blocking patterns, and I/O-heavy threads. Note: state inference is heuristic since pprof does not record explicit thread states.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "topThreads" : { + "type" : "integer", + "description" : "Number of top threads to profile in detail (default: 10)" + }, + "includeInsights" : { + "type" : "boolean", + "description" : "Whether to include generated insights (default: true)" + } + } + } + }, { + "name" : "pprof_use", + "description" : "Analyzes a pprof profile using Brendan Gregg's USE Method (Utilization, Saturation, Errors). CPU: utilization stats and top consuming functions. Memory: saturation analysis if allocation sample types are present. Threads: distribution across threads to identify serial bottlenecks. Errors: heuristic scan of hot function names for error-related patterns.", + "inputSchema" : { + "type" : "object", + "properties" : { + "sessionId" : { + "type" : "string", + "description" : "Session ID or alias. Omit to use the current session." + }, + "resources" : { + "type" : "array", + "items" : { + "type" : "string", + "enum" : [ "cpu", "memory", "threads", "errors" ] + }, + "description" : "Resources to analyze. Defaults to all: cpu, memory, threads, errors." + } + } + } + } ], + "prompts" : [ { + "name" : "compare", + "description" : "Decide whether a candidate recording regressed against a baseline, and attribute the change.", + "arguments" : [ { + "name" : "baseline", + "description" : "Path to the baseline recording", + "required" : false + }, { + "name" : "candidate", + "description" : "Path to the candidate recording", + "required" : false + } ], + "renderedText" : "Compare the baseline (baseline) against the candidate (candidate).\n\n1. jfr_open both, giving each a clear alias.\n2. Call jfr_compare with baselineSessionId and candidateSessionId.\n3. Read the `comparability` block before anything else. If it reports different execution-sample event types, very different durations, or low sample counts, say so in your answer and temper every conclusion accordingly.\n4. Treat `frames` as shares of samples, not wall-clock time. A frame growing from 3% to 9% of samples means the profile shifted; it is evidence of a slowdown only together with a rate or duration change.\n5. Changes below `minDeltaPct` are withheld deliberately. Do not go looking for smaller ones and present them as findings.\n\nConclude with either a named regression and the frame or metric that carries it, or an explicit \"no regression above the noise floor\". Never claim an improvement without the measurement that shows it.\n\n" + }, { + "name" : "latency", + "description" : "Investigate response-time problems that are not CPU-bound: contention, parking, queue saturation and blocking I/O.", + "arguments" : [ { + "name" : "path", + "description" : "Path to the JFR recording", + "required" : false + } ], + "renderedText" : "Investigate latency in the recording.\n\nLatency is usually waiting, and waiting produces no execution samples — so a healthy\nflamegraph proves nothing here.\n\n1. jfr_tsa with correlateBlocking=true. Read stateDistribution first: if most time is RUNNABLE this is a CPU problem, and you should switch to hot-method analysis.\n2. jfr_use resources=all. Look at insights.bottlenecks, and treat queue_saturation as first-class: work waiting in an executor queue cannot be recovered by making methods faster.\n3. Separate jdk.JavaMonitorEnter (blocked acquiring) from jdk.JavaMonitorWait (waiting on a condition) — they mean different things. Rank monitors by summed duration relative to the recording's wall clock, never by event count.\n4. For the code that contends, correlate samples with the wait window on the same thread using decorateByTime(jdk.JavaMonitorWait, fields=monitorClass,duration).\n5. Check jdk.ThreadPark grouped by parked class. A pool parked on its own queue is idle and healthy; a request thread parked on a future or a connection pool is the bug.\n\nRemember JFR's monitor events have a duration threshold, so absence of events is not absence of contention. Report time-overlap correlations as \"concurrent with\", never as proof of cause.\n\n" + }, { + "name" : "leak-hunt", + "description" : "Find unintended retention in a heap dump, and attribute it to the code that allocated it.", + "arguments" : [ { + "name" : "dump", + "description" : "Path to the .hprof heap dump", + "required" : false + }, { + "name" : "recording", + "description" : "Optional JFR recording from the same interval", + "required" : false + } ], + "renderedText" : "Hunt for a memory leak in the heap dump.\n\n1. hdump_open, then hdump_summary to orient.\n2. hdump_report focus=leaks. Work the findings from highest severity with the largest retainedSize.\n3. Rank by retained size, not shallow size: hdump_query \"classes | sortBy(retained desc) | top(20)\". A large char[] or byte[] population is normal; its dominator is the finding.\n4. Try the named detectors for known patterns (threadlocal-leak, classloader-leak, growing-collections, listener-leak, duplicate-strings, finalizer-queue), then `clusters` for patterns nobody wrote a detector for.\n5. Prove retention with a path to a GC root — pathToRoot() per object, or retentionPaths() merged at class level. A leak claim without a root path is a guess. The field named in that path is the fix.\n\nDistinguish a leak from intended retention: a cache that is configured to be large is working as designed, and the finding is then about its sizing, not a bug.\n\n" + }, { + "name" : "triage", + "description" : "Establish what an unfamiliar JFR recording, profile or heap dump contains and which investigation to run next.", + "arguments" : [ { + "name" : "path", + "description" : "Absolute path to the artifact to analyse", + "required" : false + } ], + "renderedText" : "Triage the recording.\n\n1. Open it with the tool matching its type: jfr_open for .jfr, hdump_open for .hprof, pprof_open for .pprof/.pb.gz, otlp_open for .otlp.\n2. Run the summary tool (jfr_summary / hdump_summary / ...) and read the event or object mix before forming any hypothesis.\n3. For JFR, run jfr_diagnose. It returns severity-ranked findings, the analyses it ran, and capabilityGaps. Read capabilityGaps first: a negative result about something the recording never captured is not a negative result.\n4. Establish the recording's duration and convert every count you plan to quote into a rate. Absolute counts across recordings of different lengths are not comparable.\n5. Route to the specific investigation the findings point at — CPU, latency and contention, GC and allocation, or heap retention — rather than running everything.\n\nReport findings ranked by impact. For each one give the tool call that produced it, the numbers, your interpretation, and a confidence level. Do not quote a number you did not measure in this session, and label sampled data as sampled.\n\n" + } ], + "resources" : [ { + "uri" : "jafar://help/hdumppath", + "name" : "HdumpPath reference", + "description" : "Query language for hdump_query: roots, predicates and heap analysis operators.", + "mimeType" : "text/markdown", + "renderedText" : "# HdumpPath Query Language\n\nHdumpPath is a path-based query language for heap dump analysis. Queries select a\nroot data set, optionally filter by type and predicates, then apply a pipeline of\ntransformation operators.\n\n## Query Structure\n```\n[/][] [| ]*\n```\n\n## Roots\n- `objects` — all heap object instances\n- `classes` — class metadata (no retained-size needed)\n- `gcroots` — GC root references\n- `clusters` — graph-based leak clusters (triggers retained-size on first use)\n- `duplicates` — structurally-identical subgraphs (cached after first run)\n- `ages` — objects annotated with estimated age\n\n## Quick Examples\n```\nobjects | top(10, retained)\nclasses | top(20, instanceCount)\nobjects/java.lang.String | count\nobjects/instanceof/java.util.Map | waste() | top(10, wastedBytes)\nduplicates | sortBy(wastedBytes desc) | top(20)\ngcroots/THREAD_OBJ | dominatedSize() | sortBy(dominated desc)\ncheckLeaks(detector=\"duplicate-strings\")\nclusters | sortBy(score desc) | head(10)\n```\n\n## Performance Notes\nOperators that compute retained sizes (retained, pathToRoot, retentionPaths,\ndominators, waste, cacheStats, checkLeaks) trigger approximate retained-size\ncomputation on first use. For large heaps (>500K objects) this may take minutes.\nResults are cached; subsequent queries on the same session are fast.\n\n" + }, { + "uri" : "jafar://help/jfrpath", + "name" : "JfrPath reference", + "description" : "Query language for jfr_query: roots, filters, units, pipeline operators and correlation.", + "mimeType" : "text/markdown", + "renderedText" : "# JfrPath Query Language\n\nJfrPath is a path-based query language for navigating and querying JFR files.\n\n## Basic Syntax\n```\nevents/[filters] | pipeline_operator\n```\n\n## Query Roots\n- `events/` - Access event data (e.g., events/jdk.ExecutionSample)\n- `metadata/` - Access type metadata\n- `chunks` - Access chunk information\n- `constants/` - Access constant pool entries\n\n## Field Navigation\n- Simple: `events/jdk.FileRead/path`\n- Nested: `events/jdk.ExecutionSample/stackTrace/frames`\n- Array index: `events/jdk.ExecutionSample/stackTrace/frames/0`\n\n## Quick Examples\n```\nevents/jdk.ExecutionSample | count()\nevents/jdk.FileRead[bytes>1000] | top(10)\nevents/jdk.GCPhasePause | stats(duration)\nevents/jdk.ExecutionSample | groupBy(eventThread/javaName)\n```\n\n## Help Topics\nCall jfr_help with topic parameter for detailed documentation:\n- filters: Filter syntax and operators\n- pipeline: Aggregation operators (count, groupBy, top, stats, etc.)\n- functions: Built-in functions for filters and select\n- examples: Common query patterns\n- event_types: Common JDK event types\n- tools: When to use each jfr_* tool (flamegraph vs stackprofile vs hotmethods, etc.)\n\n\n# JfrPath Filters\n\nFilters use square brackets `[...]` to constrain results.\n\n## Comparison Operators\n- `=` : Equal\n- `!=` : Not equal\n- `>` : Greater than\n- `>=` : Greater than or equal\n- `<` : Less than\n- `<=` : Less than or equal\n- `~` : Regex match\n\n## Examples\n```\nevents/jdk.FileRead[bytes>1000]\nevents/jdk.ExecutionSample[eventThread/javaName=\"main\"]\nevents/jdk.FileRead[path~\"/tmp/.*\"]\nevents/jdk.GCPhasePause[duration>10000000]\n```\n\n## Boolean Expressions\n```\nevents/jdk.FileRead[bytes>1000 and path~\"/tmp/.*\"]\nevents/jdk.ExecutionSample[eventThread/javaName=\"main\" or eventThread/javaName=\"worker\"]\nevents/jdk.FileRead[not (bytes<100)]\n```\n\n## Filter Functions\n- `contains(field, \"substr\")` - String contains\n- `startsWith(field, \"prefix\")` - String starts with\n- `endsWith(field, \"suffix\")` - String ends with\n- `matches(field, \"regex\")` - Regex match\n- `exists(field)` - Field is not null\n- `empty(field)` - String/list is empty\n- `between(field, min, max)` - Numeric range\n- `len(field)` - Length (use in comparisons)\n\n## Duration Values\nDurations can be specified with units:\n- `10ms` - 10 milliseconds\n- `1s` - 1 second\n- `500us` - 500 microseconds\n- `100ns` - 100 nanoseconds\n\nExample: `events/jdk.GCPhasePause[duration>10ms]`\n\n## List Matching\n- `any:` (default) - Any element matches\n- `all:` - All elements match\n- `none:` - No elements match\n\nExample: `events/jdk.ExecutionSample[any:stackTrace/frames/method/type/name/string ~ \".*MyClass.*\"]`\n\n\n# JfrPath Pipeline Operators\n\nPipeline operators transform or aggregate results. Append with `|`:\n\n## count()\nCount matching events.\n```\nevents/jdk.ExecutionSample | count()\n→ { \"count\": 12345 }\n```\n\n## top(n[, by=field, asc=false])\nReturn top N results sorted by field.\n```\nevents/jdk.FileRead | top(10, by=bytes)\nevents/jdk.GCPhasePause | top(5, by=duration)\nevents/jdk.FileRead | top(10, by=bytes, asc=true) # smallest first\n```\n\n## groupBy(field[, agg=count|sum|avg|min|max, value=path, sortBy=key|value, asc=false])\nGroup by field and aggregate.\n```\nevents/jdk.ExecutionSample | groupBy(eventThread/javaName)\nevents/jdk.FileRead | groupBy(path, agg=sum, value=bytes)\nevents/jdk.ExecutionSample | groupBy(eventThread/javaName, sortBy=value)\n```\n\n## stats([field])\nCompute min, max, avg, stddev, count.\n```\nevents/jdk.FileRead | stats(bytes)\nevents/jdk.GCPhasePause | stats(duration)\n→ { \"min\": 100, \"max\": 50000, \"avg\": 5000, \"stddev\": 2500, \"count\": 200 }\n```\n\n## sum([field])\nSum numeric values.\n```\nevents/jdk.FileRead | sum(bytes)\n→ { \"sum\": 1234567, \"count\": 100 }\n```\n\n## quantiles(q1, q2, ...[, path=field])\nCompute percentiles (0.0 to 1.0).\n```\nevents/jdk.GCPhasePause | quantiles(0.5, 0.9, 0.99, path=duration)\n→ { \"p50\": 1000, \"p90\": 5000, \"p99\": 20000 }\n```\n\n## select(field1, field2, expr as alias, ...)\nProject specific fields or compute expressions.\n```\nevents/jdk.FileRead | select(path, bytes)\nevents/jdk.FileRead | select(path, bytes / 1024 as kb)\nevents/jdk.ExecutionSample | select(eventThread/javaName as thread, startTime)\n```\n\n## sortBy(field[, asc=false])\nSort results by field.\n```\nevents/jdk.FileRead | select(path, bytes) | sortBy(bytes)\n```\n\n## decorateByTime(eventType, fields=f1,f2[, threadPath=..., decoratorThreadPath=...])\nJoin with time-overlapping events on same thread.\n```\nevents/jdk.ExecutionSample | decorateByTime(jdk.JavaMonitorWait, fields=monitorClass,duration)\n```\n\n## decorateByKey(eventType, key=path, decoratorKey=path, fields=f1,f2)\nJoin using correlation keys.\n```\nevents/jdk.ExecutionSample | decorateByKey(RequestStart, key=eventThread/javaThreadId, decoratorKey=thread/javaThreadId, fields=requestId)\n```\n\n\n# JfrPath Functions\n\n## Filter Functions\n\n### String Functions\n- `contains(field, \"substr\")` - Check if contains substring\n- `startsWith(field, \"prefix\")` - Check if starts with prefix\n- `endsWith(field, \"suffix\")` - Check if ends with suffix\n- `matches(field, \"regex\")` - Check if matches regex\n- `matches(field, \"regex\", \"i\")` - Case-insensitive regex\n\n### Existence Functions\n- `exists(field)` - Field is present and not null\n- `empty(field)` - String or list is empty\n\n### Numeric Functions\n- `between(field, min, max)` - Value in range (inclusive)\n- `len(field)` - Length of string or array\n\n## Select Expression Functions\n\n### Conditional\n- `if(condition, trueValue, falseValue)` - Conditional expression\n\n### String Functions\n- `upper(string)` - Convert to uppercase\n- `lower(string)` - Convert to lowercase\n- `substring(string, start[, length])` - Extract substring\n- `length(string)` - String length\n- `coalesce(v1, v2, ...)` - First non-null value\n\n### Arithmetic\n- `+`, `-`, `*`, `/` - Basic arithmetic\n- Parentheses for grouping: `(bytes * count) / 1024`\n\n### String Templates\nUse `${...}` for interpolation:\n```\nevents/jdk.FileRead | select(\"${path}: ${bytes / 1024} KB\" as info)\n```\n\n## Transform Operators (after |)\n- `| len()` - String/list length\n- `| uppercase()` - Convert to uppercase\n- `| lowercase()` - Convert to lowercase\n- `| trim()` - Trim whitespace\n- `| abs()` - Absolute value\n- `| round()` - Round to integer\n- `| floor()` - Round down\n- `| ceil()` - Round up\n\n\n# JfrPath Query Examples\n\n## CPU/Execution Analysis\n```\n# Count execution samples\nevents/jdk.ExecutionSample | count()\n\n# Samples by thread\nevents/jdk.ExecutionSample | groupBy(eventThread/javaName)\n\n# Top threads by sample count\nevents/jdk.ExecutionSample | groupBy(eventThread/javaName, sortBy=value) | top(10)\n\n# Samples for specific thread\nevents/jdk.ExecutionSample[eventThread/javaName=\"main\"] | count()\n```\n\n## GC Analysis\n```\n# GC pause statistics\nevents/jdk.GCPhasePause | stats(duration)\n\n# Longest GC pauses\nevents/jdk.GCPhasePause | top(10, by=duration)\n\n# GC pauses over 10ms\nevents/jdk.GCPhasePause[duration>10ms] | count()\n\n# Heap usage after GC\nevents/jdk.GCHeapSummary[when/when=\"After GC\"] | select(heapUsed, heapSpace/committedSize)\n```\n\n## I/O Analysis\n```\n# File read statistics\nevents/jdk.FileRead | stats(bytes)\n\n# Largest file reads\nevents/jdk.FileRead | top(10, by=bytes)\n\n# Reads by file path\nevents/jdk.FileRead | groupBy(path, agg=sum, value=bytes)\n\n# Slow file operations (>10ms)\nevents/jdk.FileRead[duration>10ms] | top(10)\n\n# Socket reads by host\nevents/jdk.SocketRead | groupBy(host)\n```\n\n## Lock Contention\n```\n# Monitor enter events\nevents/jdk.JavaMonitorEnter | count()\n\n# Longest lock waits\nevents/jdk.JavaMonitorEnter | top(10, by=duration)\n\n# Contention by monitor class\nevents/jdk.JavaMonitorEnter | groupBy(monitorClass)\n\n# Lock waits over 1ms\nevents/jdk.JavaMonitorEnter[duration>1ms] | count()\n```\n\n## Memory Analysis\n```\n# Allocation samples\nevents/jdk.ObjectAllocationSample | count()\n\n# Allocations by class\nevents/jdk.ObjectAllocationSample | groupBy(objectClass/name)\n\n# Large allocations\nevents/jdk.ObjectAllocationOutsideTLAB | top(10, by=allocationSize)\n```\n\n## Thread Analysis\n```\n# Thread starts\nevents/jdk.ThreadStart | count()\n\n# Thread CPU load\nevents/jdk.ThreadCPULoad | top(10, by=user)\n\n# Thread sleep events\nevents/jdk.ThreadSleep | stats(time)\n```\n\n## Correlation/Decoration\n```\n# Samples during lock waits\nevents/jdk.ExecutionSample | decorateByTime(jdk.JavaMonitorWait, fields=monitorClass,duration)\n\n# Group samples by lock class\nevents/jdk.ExecutionSample | decorateByTime(jdk.JavaMonitorEnter, fields=monitorClass) | groupBy($decorator.monitorClass)\n```\n\n" + }, { + "uri" : "jafar://help/tools", + "name" : "Choosing the right tool", + "description" : "Which analysis tool answers which question, and when to prefer one over another.", + "mimeType" : "text/markdown", + "renderedText" : "# Choosing the Right jfr_* Tool\n\nStart every JFR workflow with `jfr_open`: give it the recording path (and, when useful, an\nalias), then pass the returned session id or alias to the tools below. Use `jfr_close` when\nthe investigation is finished.\n\n## CPU Profiling Tools\n\n### jfr_hotmethods\nBest for: Quick identification of which methods consume the most CPU.\nReturns: Flat ranked list of leaf methods with sample counts and percentages.\nUse when: You need a fast answer to \"where is CPU time going?\" without call-path context.\n\n### jfr_flamegraph\nBest for: Understanding call paths and how code reaches hot methods.\nReturns: Aggregated stack traces in folded (semicolon-separated) or tree (JSON) format.\nUse when: You need to see the full call hierarchy — which callers lead to a hotspot,\nor which entry points fan out into expensive subtrees. Output is designed for\nvisualization tools or structural call-path reasoning.\n\n### jfr_stackprofile\nBest for: Detecting temporal patterns and thread affinity in CPU hotspots.\nReturns: Structured JSON with per-frame time-bucket arrays, per-thread sample counts,\nnumeric percentages, and a derived category (normal / hotspot / steady-hotspot).\nUse when: You need to programmatically analyze how a method's CPU usage changes over\nthe recording duration (bursty vs steady), identify N+1 query patterns (steady hotspots),\nor determine which threads are contributing to a hotspot. The time-bucket arrays let you\ndetect intermittent load spikes that would be invisible in aggregated flamegraph data.\n\n### Decision Guide\n```\nNeed a quick \"top CPU consumers\" list? → jfr_hotmethods\nNeed to understand call paths / callers? → jfr_flamegraph\nNeed temporal patterns or thread breakdown? → jfr_stackprofile\nNeed all of the above for deep investigation? → Start with jfr_stackprofile,\n then jfr_flamegraph for call paths\n```\n\n## Other Specialized Tools\n\n### jfr_summary\nRecording overview: duration, event counts, JVM info. Start here for orientation.\n\n### jfr_diagnose\nAutomated performance triage. Runs multiple checks and returns findings + recommendations.\n\n### jfr_use\nUSE Method analysis (Utilization, Saturation, Errors) for CPU, memory, threads, and I/O.\n\n### jfr_tsa\nThread State Analysis. Breaks down what threads are doing: running, blocked, waiting, I/O.\n\n### jfr_exceptions\nException pattern analysis: types, frequencies, and common throw sites.\n\n### jfr_callgraph\nCall graph between methods. Shows caller→callee relationships with sample weights.\n\n### jfr_query\nGeneral-purpose JfrPath queries for anything not covered by specialized tools.\nUse jfr_help(topic=\"pipeline\") and jfr_help(topic=\"filters\") for query syntax.\n\n" + }, { + "uri" : "jafar://sessions", + "name" : "Open sessions", + "description" : "Recordings, heap dumps and profiles currently open, with their ids and aliases.", + "mimeType" : "text/markdown", + "renderedText" : "# Open sessions\n\n## JFR recordings\n_none open_\n\n## Heap dumps\n_none open_\n\n## pprof profiles\n_none open_\n\n## OTLP profiles\n_none open_\n\nA session id or alias can be passed as `sessionId` to any tool of the matching family, and named in cross-session operators such as `join(session=...)`.\n\n" + } ] +} \ No newline at end of file diff --git a/llm-openai/src/test/java/io/jafar/shell/llm/openai/OpenAiCompatibleBackendTest.java b/llm-openai/src/test/java/io/jafar/shell/llm/openai/OpenAiCompatibleBackendTest.java index 0242ff1c..68755454 100644 --- a/llm-openai/src/test/java/io/jafar/shell/llm/openai/OpenAiCompatibleBackendTest.java +++ b/llm-openai/src/test/java/io/jafar/shell/llm/openai/OpenAiCompatibleBackendTest.java @@ -82,7 +82,10 @@ void stopServer() { private LlmConfig config(Map extra) { Map settings = new HashMap<>(extra); settings.putIfAbsent("llm.base-url", baseUrl); - return new LlmConfig(settings::get); + // No settings-file fallback: a developer machine with a real ~/.config/jafar/llm.properties + // (llm.model=…) must not leak into these assertions — the model then overrides the profile + // default and the test fails everywhere except on machines without the file. + return new LlmConfig(settings::get, java.util.Optional::empty); } private static LlmRequest request() { diff --git a/shell-core/src/main/java/io/jafar/shell/core/llm/LlmConfig.java b/shell-core/src/main/java/io/jafar/shell/core/llm/LlmConfig.java index bb836266..c1da4648 100644 --- a/shell-core/src/main/java/io/jafar/shell/core/llm/LlmConfig.java +++ b/shell-core/src/main/java/io/jafar/shell/core/llm/LlmConfig.java @@ -85,8 +85,13 @@ public LlmConfig(Function lookup) { this(lookup, LlmSettingsFile::find); } - /** Package-private seam: lets a test supply a settings file without setting an env var. */ - LlmConfig( + /** + * Creates a config with an explicitly supplied settings-file source. + * + *

This seam lets integrations and tests control settings-file discovery without mutating + * process-wide environment variables. + */ + public LlmConfig( Function lookup, java.util.function.Supplier> settingsFile) { this.lookup = lookup == null ? name -> null : lookup;