Repository navigation
Add MCP guidance contract and agent workflow E2Es - #121
Merged
Merged
Conversation
…tests Tier 0 renders the guidance contract — every tool description/input schema, every prompt with its rendered message text, every resource with its content — to a committed snapshot (jfr-mcp/src/test/resources/mcp-contract.json), checked against the live server wiring and against a run through McpContractExport. Guidance evals parse every documented query example with the matching real parser, resolve every mentioned tool/resource from the live registrations, and enforce per-family tool isolation. Tier 1 drives the shortest useful workflow per input family (JFR, HPROF, pprof, OTLP) through java -jar ... --stdio in a fresh child JVM, via a new McpStdioProcessHarness: per-test session file and user home under build/mcp-e2e, asserted owned and removed on shutdown. The hprof workflow consumes the promoted hdump-parser SyntheticHeapDumpGenerator via java-test-fixtures. Also: JFRPath filter docs corrected (list-match example), opening-tool descriptions name their file extension, and jafar://help/tools teaches the JFR entry point it recommends. Verified per the plan doc acceptance criteria; see the implementation doc for the R5 fail proofs recorded per guard.
#119 scoped MCP resource reads to the calling MCP session and #120 renamed the plugin consumer, so the guidance tests and e2e harness needed re-anchoring: - GuidanceSurfaces renders prompts/resources through a real McpSyncServerExchange with a fixed synthetic client id (the idiom SessionsResourceOwnershipTest already uses) instead of a null exchange that the scoped jafar://sessions read handler NPEs on. - mcp-contract.json refreshed through the reviewed-update switch (-Dmcp.updateSnapshot=true); full JSON diff verified the delta is exactly #119's three closeAll ownership wording changes. - ToolCatalogSnapshotTest's drift preview was self-broken (two %s placeholders against one argument, which turned every real drift into an opaque MissingFormatArgumentException) and listed only tools; it now builds the message without format traps and covers tools, prompts and resources. Proved against a deliberately stale snapshot (R5): the message named the changed entries. - McpStdioProcessHarness folds in #119's jafar.state.dir isolation next to the redirected user.home, now that the child spawn lives there instead of the deleted McpEndToEndTest.startServer. - Stale btraceio/jafar-perf-box references point at btraceio/agent-plugins (plugins/jafar-perf); scripts/check-tool-references.js remains the external drift check this snapshot feeds after a release. Verified: ToolCatalogSnapshotTest, GuidanceExampleQueriesTest, GuidanceToolReferencesTest, GuidanceContractSessionIsolationTest, ToolDescriptionQualityTest green; :jfr-mcp:endToEndTest 6/6; spotlessCheck green. Failure sets compared by name (R6) against the pre-rebase baseline.
McpJfrTransportTest.jfrDiagnoseReturnsReport failed on every run with "did not respond": the harness's 15s per-response deadline sits below the real cost of jfr_diagnose (default depth=full), which re-parses the whole recording for each of its six sub-analyses. Measured, not guessed: warm full untyped parse of the 2.2MB test recording = 0.45-0.73s; diagnose makes ~10-12 such passes, 13-21s under load (three green runs at 120-180s, never above 30s; heap at 8GB unchanged, GC pauses <=26ms, logging ruled out — see the case file). The 15s boundary flipped depending on machine load, which is why pristine origin/main reproduced it too. Fixes: - McpTransportHarness.RESPONSE_TIMEOUT_MS default 15_000 -> 60_000. A real hang still fails the test within a minute; every legitimate response the profile showed fits comfortably. - :jfr-mcp:test forwards -Dmcp.test.timeout.ms. The harness's own failure message advertises raising it, but the task silently ignored the property — the advertised advice did nothing. - Case file recorded in doc/agents/Verification.md under R6 (measure a slow test's real response time before calling it flaky; forward the properties a harness advertises). R5 proof: -Dmcp.test.timeout.ms=2000 fails the named test again; with the default deadline the focused class and the full module run green. Verified: :jfr-mcp:test --rerun-tasks = 404 testcases, 0 failures; :jfr-mcp:endToEndTest = 6/6; spotlessCheck green.
Memory graph (.investigations/jfr-mcp-diagnose-deadline/, marked done) from the investigation behind 259a0da: run timing series, JFR method+GC profile, in-executor parse bench, the refuted environment/heap/GC/logging hypotheses, and the open parser-cost follow-ups with measurements on hand.
Contributor
Combined JUnit Test Report
HTML Test ReportsRun artifacts: https://github.com/btraceio/jafar/actions/runs/37159262749
|
The JDK 21 CI run exposed a reconnect race where a request received an internal error after the SSE session dropped. Keep delivery bounded, but allow two short retries for transient disconnects. The original failure was observed in StdioSseBridgeTest.reconnectsAndReplaysTheHandshakeAfterTheDaemonDropsTheSession().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
:jfr-mcp:testguidance/contract tests:jfr-mcp:endToEndTest --tests io.jafar.mcp.McpAgentWorkflowEndToEndTest(4/4):jfr-mcp:endToEndTest --tests io.jafar.mcp.McpEndToEndTest(2/2):llm-openai:test --tests io.jafar.shell.llm.openai.OpenAiCompatibleBackendTestspotlessApplyNo hosted LLM provider is contacted by these tests.