Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion internal/agent/runtime.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
"github.com/labstack/fanout/internal/toolerror"
)

const systemPrompt = `You are Fanout's observability assistant. When a view is attached to your reply it is the picture: never draw diagrams, trees, or charts in text, never use code fences to draw boxes, arrows, or trees, and never add a table or list that restates what an attached view already shows; your prose adds only what the view omits. Use get_observability_overview first for broad health questions, get_intelligence_snapshot for the latest precomputed anomalies and recurring log patterns, get_service_topology for direct dependency edges, get_service_dependencies for bounded upstream or downstream reachability from a service, get_service_performance for activity/latency/endpoints/comparisons, inspect_trace for trace or root-cause inspection, and search_logs for log questions. Treat structured outputs as authoritative. You build and change the user's dashboards. Build one whenever the user asks for an overview, asks why something is slow, failing or changing, asks to compare, break down or track telemetry, or asks for anything they would want to look at again; answer a single factual question with a view instead. To build one, read get_telemetry_schema, draft a complete spec of panels that answer the request, run preview_panels, fix every invalid panel, replace or explain every empty one, and only then call create_dashboard. Cover every part of the request: when a part has no data, keep its panel and say why in the panel description instead of dropping it. Title each panel with exactly what it measures. Prefer a few precise panels over many vague ones: headline stats first, then the time series that explain them, then a table of the worst offenders. After saving, reply in two or three sentences with what the dashboard shows and what stands out. Use filter values exactly as the schema lists them. To change a dashboard, get_dashboard first and use edit_dashboard so unrelated panels stay as they are; replace only when the user asks for a redesign. State the time window you used, distinguish missing data from healthy behavior, and never invent services, metrics, or causal claims. Keep answers concise because attached views provide interactive details. Never expose implementation details to the user: do not mention protocol names, tool names, schemas, query IDs, data-source names, storage engines, providers, or internal execution steps. Refer to attached interactive content simply as a view.` + dashboardAnalysisGuidance
const systemPrompt = `You are Fanout's observability assistant. When a view is attached to your reply it is the picture: never draw diagrams, trees, or charts in text, never use code fences to draw boxes, arrows, or trees, and never add a table or list that restates what an attached view already shows; your prose adds only what the view omits. Use get_observability_overview first for broad health questions, get_intelligence_snapshot for the latest precomputed anomalies and recurring log patterns, get_service_topology for direct dependency edges, get_service_dependencies for bounded upstream or downstream reachability from a service, get_service_performance for activity/latency/operations/comparisons, inspect_trace for trace or root-cause inspection, and search_logs for log questions. Treat structured outputs as authoritative. You build and change the user's dashboards. Build one whenever the user asks for an overview, asks why something is slow, failing or changing, asks to compare, break down or track telemetry, or asks for anything they would want to look at again; answer a single factual question with a view instead. To build one, read get_telemetry_schema, draft a complete spec of panels that answer the request, run preview_panels, fix every invalid panel, replace or explain every empty one, and only then call create_dashboard. Cover every part of the request: when a part has no data, keep its panel and say why in the panel description instead of dropping it. Title each panel with exactly what it measures. Prefer a few precise panels over many vague ones: headline stats first, then the time series that explain them, then a table of the worst offenders. After saving, reply in two or three sentences with what the dashboard shows and what stands out. Use filter values exactly as the schema lists them. To change a dashboard, get_dashboard first and use edit_dashboard so unrelated panels stay as they are; replace only when the user asks for a redesign. State the time window you used, distinguish missing data from healthy behavior, and never invent services, metrics, or causal claims. Keep answers concise because attached views provide interactive details. Never expose implementation details to the user: do not mention protocol names, tool names, schemas, query IDs, data-source names, storage engines, providers, or internal execution steps. Refer to attached interactive content simply as a view.` + dashboardAnalysisGuidance

const dashboardAnalysisGuidance = ` For analysis dashboards, use only the types the question needs; do not fill a dashboard with all of them. Use heatmap for latency changes, histogram for distributions, scatter for relationships, state_timeline for threshold states, logs for events, log_patterns for repeated messages, traces for slow or erroring traces, service_map for dependencies, and health for service health. Use drill for span or log evidence; keep checked filters. Include annotations for change investigations; deploys and detector findings do not prove causes. Use a deploy split for scoped before/since comparisons; retain missing-deploy explanations. A definition, explanation, or single fact is an answer intent; do not create or replace a dashboard for it. Preserve every requested facet and explain absent telemetry without inventing it. Saved panels must keep working as new telemetry arrives: never filter on trace, span or request IDs or other values copied from one result; show recent evidence with a traces panel and drill to its logs. In replies, never name schema fields to the user. Explain in chat is answer intent: use the observed absolute panel window and resolved variables, explain errors and suggest corrections without creating, editing, replacing or restoring dashboards.`

Expand Down
4 changes: 2 additions & 2 deletions internal/mcp/fragments.go
Original file line number Diff line number Diff line change
Expand Up @@ -197,8 +197,8 @@ func fragmentPreset(kind, service, namespace, severity, search string, limit int
add("latency", "p95 latency", "timeseries", "spans", "p95(duration_ms)", "ms", nil, "auto")
add("errors", "Error rate", "timeseries", "spans", "error_rate()", "percent", nil, "auto")
add("requests", "Request rate", "timeseries", "spans", "rate()", "per_second", nil, "auto")
add("endpoints", "Slow endpoints", "table", "spans", "p95(duration_ms)", "ms", []string{"http_route"}, "")
d.Panels[len(d.Panels)-1].Query.Where = append(d.Panels[len(d.Panels)-1].Query.Where, "http_route <> ''")
add("operations", "Slow operations", "table", "spans", "p95(duration_ms)", "ms", []string{"operation"}, "")
d.Panels[len(d.Panels)-1].Query.Where = append(d.Panels[len(d.Panels)-1].Query.Where, "kind IN ('SPAN_KIND_SERVER','SPAN_KIND_CONSUMER')")
case "logs":
if severity != "" {
filters = append(filters, "severity = "+literal(severity))
Expand Down
14 changes: 7 additions & 7 deletions internal/mcp/fragments_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ func TestFragmentVarsRoundTripAndAppHelpers(t *testing.T) {
}
}

func TestFragmentPresetsExecutedSummariesAndEndpointScope(t *testing.T) {
func TestFragmentPresetsExecutedSummariesAndOperationScope(t *testing.T) {
s := panelServerFixture(t, true)
for _, name := range []string{"overview", "performance", "topology", "logs", "trace"} {
content, f, err := s.presetFragment(t.Context(), QueryInput{Namespace: "shop"}, name, "checkout", "ERROR", "timeout")
Expand All @@ -261,25 +261,25 @@ func TestFragmentPresetsExecutedSummariesAndEndpointScope(t *testing.T) {
if r.Status == panel.StatusError {
t.Fatalf("%s: %+v", name, r)
}
if r.ID == "endpoints" {
if r.ID == "operations" {
if r.Frame == nil {
t.Fatal("missing endpoints frame")
t.Fatal("missing operations frame")
}
cart := false
for c, col := range r.Frame.Columns {
if col.Name == "http_route" {
if col.Name == "operation" {
for _, value := range r.Frame.Values[c] {
if value == "/cart" {
if value == "GET cart" {
cart = true
}
if value == "" {
t.Fatal("empty route included")
t.Fatal("unexpected empty operation")
}
}
}
}
if !cart {
t.Fatal("missing /cart endpoint")
t.Fatal("missing GET cart operation")
}
}
}
Expand Down
102 changes: 102 additions & 0 deletions internal/mcp/performance_operations_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
package mcp

import (
"fmt"
"reflect"
"testing"
"time"

"github.com/labstack/fanout/internal/config"
"github.com/labstack/fanout/internal/panel"
"github.com/labstack/fanout/internal/query"
"github.com/labstack/fanout/internal/telemetry"
telemetrystore "github.com/labstack/fanout/internal/telemetry/store"
)

func TestPerformanceSlowOperationsAcrossProtocols(t *testing.T) {
cfg := config.Config{DataDir: t.TempDir(), DuckDBMemory: "256MB", DuckDBThreads: 2, DuckDBMaxConns: 4, RollupInterval: time.Hour}
repo, err := telemetrystore.Open(cfg.TelemetryDir())
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = repo.Close() })
duck, err := query.NewDuck(t.Context(), cfg, repo)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = duck.Close() })
from := time.Date(2026, 10, 1, 12, 0, 0, 0, time.UTC)
to := from.Add(time.Hour)
cases := []struct{ protocol, service, kind, slow, fast, route string }{
{"grpc", "checkout", "SPAN_KIND_SERVER", "CheckoutService/PlaceOrder", "CheckoutService/GetCart", ""},
{"kafka", "fraud-detection", "SPAN_KIND_CONSUMER", "orders process", "refunds process", ""},
{"http", "frontend", "SPAN_KIND_SERVER", "GET /cart", "GET /health", "/cart"},
}
var spans []telemetry.Span
add := func(namespace, service, operation, kind, route string, ms float64) {
n := from.Add(time.Minute).UnixNano()
id := fmt.Sprintf("span-%d", len(spans))
spans = append(spans, telemetry.Span{
Namespace: namespace, ServiceName: service, TraceID: id, SpanID: id,
Name: operation, Kind: kind, HTTPRoute: route, DurationMS: ms,
StartUnixNanos: n, EndUnixNanos: n + int64(ms*1e6), IngestedAt: n,
})
}
for _, tc := range cases {
for _, ms := range []float64{100, 300} {
add("shop", tc.service, tc.slow, tc.kind, tc.route, ms)
}
for _, ms := range []float64{20, 40} {
add("shop", tc.service, tc.fast, tc.kind, tc.route, ms)
}
for _, kind := range []string{"SPAN_KIND_CLIENT", "SPAN_KIND_INTERNAL", "SPAN_KIND_PRODUCER"} {
// Excluded kinds must neither create groups nor inflate a handled
// operation's p95. Even an HTTP route must not admit these spans.
add("shop", tc.service, "excluded "+kind, kind, "/outbound", 10000)
add("shop", tc.service, tc.fast, kind, tc.route, 10000)
}
add("other", tc.service, "other namespace", tc.kind, tc.route, 10000)
}
if err := repo.Commit(t.Context(), telemetrystore.Batch{ID: "operations", Spans: spans}); err != nil {
t.Fatal(err)
}
s := &Server{panels: panel.NewExecutor(duck, 30)}
for _, tc := range cases {
t.Run(tc.protocol, func(t *testing.T) {
for _, limit := range []int{0, 1} {
_, out, err := s.performance(t.Context(), nil, PerformanceInput{Service: tc.service, Namespace: "shop", From: &from, To: &to, Limit: limit})
if err != nil {
t.Fatal(err)
}
p := out.Dashboard.Panels[3]
if p.ID != "operations" || p.Title != "Slow operations" || p.Viz != "table" || p.Unit != "ms" || p.Drill != "traces" || !reflect.DeepEqual(p.Query.By, []string{"operation"}) {
t.Fatalf("operations panel = %+v", p)
}
wantLimit := 400
if limit > 0 {
wantLimit = limit
}
if p.Query.Limit != wantLimit {
t.Fatalf("limit = %d, want %d", p.Query.Limit, wantLimit)
}
r := out.Results[3]
wantOperations := []any{tc.slow, tc.fast}
wantP95 := []float64{290, 39}
if limit > 0 {
wantOperations, wantP95 = wantOperations[:limit], wantP95[:limit]
}
if r.ID != "operations" || r.Status != panel.StatusOK || r.Frame == nil || r.Frame.Rows != len(wantOperations) {
t.Fatalf("operations result = %+v", r)
}
if len(r.Frame.Columns) != 2 || r.Frame.Columns[0].Name != "operation" || r.Frame.Columns[1].Name != "p95" || !reflect.DeepEqual(r.Frame.Values[0], wantOperations) {
t.Fatalf("operations frame = %+v, want %v / %v", r.Frame, wantOperations, wantP95)
}
for i, want := range wantP95 {
if got := numeric(r.Frame.Values[1][i]); got != want {
t.Errorf("%s p95 = %v, want %v", wantOperations[i], got, want)
}
}
}
})
}
}
6 changes: 3 additions & 3 deletions internal/mcp/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const ProtocolVersion = "2026-07-28"

const staticCatalogTTLMs = 5 * 60 * 1000

const serverInstructions = "Start with get_observability_overview for system health, use get_intelligence_snapshot for the latest precomputed anomalies and log patterns, get_service_topology for direct dependency edges (use this preset once for a service map; do not repeat the same map with query_telemetry), get_service_dependencies for bounded upstream or downstream reachability from a service, get_service_performance for latency and errors, inspect_trace for one trace, and search_logs for application events. Treat schema, timestamps, provenance, and bounded time windows as authoritative. Dashboard tools are scoped to the authenticated user. For a custom chart in chat, use query_telemetry with one v1 panel; every chart view returns a dashboard fragment rendered by the shared panel renderer. To build a dashboard, read get_telemetry_schema, draft panels, run preview_panels until every panel is ok or deliberately empty, then create_dashboard. To change one, get_dashboard first and use edit_dashboard; replace only for a redesign the user asked for."
const serverInstructions = "Start with get_observability_overview for system health, use get_intelligence_snapshot for the latest precomputed anomalies and log patterns, get_service_topology for direct dependency edges (use this preset once for a service map; do not repeat the same map with query_telemetry), get_service_dependencies for bounded upstream or downstream reachability from a service, get_service_performance for latency, errors and slow operations across HTTP, gRPC and consumers, inspect_trace for one trace, and search_logs for application events. Treat schema, timestamps, provenance, and bounded time windows as authoritative. Dashboard tools are scoped to the authenticated user. For a custom chart in chat, use query_telemetry with one v1 panel; every chart view returns a dashboard fragment rendered by the shared panel renderer. To build a dashboard, read get_telemetry_schema, draft panels, run preview_panels until every panel is ok or deliberately empty, then create_dashboard. To change one, get_dashboard first and use edit_dashboard; replace only for a redesign the user asked for."

type Observability interface {
Dependencies(context.Context, observability.Scope, observability.DependencyOptions) (observability.Result[observability.Dependencies], error)
Expand Down Expand Up @@ -62,7 +62,7 @@ type PerformanceInput struct {
Window string `json:"window,omitempty" jsonschema:"Time window such as 15m, 1h, 24h, 168h, or 720h; defaults to 1h"`
Namespace string `json:"namespace,omitempty" jsonschema:"OpenTelemetry service namespace; empty queries all namespaces"`
Service string `json:"service,omitempty" jsonschema:"Optional exact OpenTelemetry service name; omit for the whole system"`
Limit int `json:"limit,omitempty" jsonschema:"Maximum endpoints to return, from 1 to 500"`
Limit int `json:"limit,omitempty" jsonschema:"Maximum operations to return, from 1 to 500"`
}

type TraceInput struct {
Expand Down Expand Up @@ -349,7 +349,7 @@ func (s *Server) registerTools() {
fragmentTool(s, &mcp.Tool{
Name: "get_service_performance",
Title: "Service performance explorer",
Description: "Display separate latency, error rate and request rate panels plus slow endpoints for one service or the system.",
Description: "Display separate latency, error rate and request rate panels plus slow operations (HTTP, gRPC and consumers) for one service or the system.",
Annotations: readOnly,
Meta: appToolMeta(panelsAppURI),
}, s.performance)
Expand Down
11 changes: 9 additions & 2 deletions internal/panel/diagnose.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ const maxDiagnosedFilters = 8
// diagnose explains an empty structured panel. It counts the signal in the
// window, then reruns the count with each of the first maxDiagnosedFilters
// filters removed in turn. The filter whose removal recovers the most rows is
// named, with the values seen for its field when it was an equality. Ties
// keep the author's filter order.
// named in the context of the active combination, with the values seen for its
// field when it was an equality. Ties keep the author's filter order.
func (e *Executor) diagnose(ctx context.Context, p *Panel, filters []Filter, scope Scope) string {
if p.Query == nil {
return "The query returned no rows for this time range."
Expand Down Expand Up @@ -78,6 +78,13 @@ func (e *Executor) diagnose(ctx context.Context, p *Panel, filters []Filter, sco
f := active[best]
others := append(append([]Filter{}, active[:best]...), active[best+1:]...)
message := fmt.Sprintf("No %s match %s. Without that filter, %d do.", sig.name, f.Source, recovered)
if len(active) > 1 {
var sources []string
for _, filter := range active {
sources = append(sources, "("+filter.Source+")")
}
message = fmt.Sprintf("No %s match the combination of filters: %s. Without %s, %d match the remaining filters.", sig.name, strings.Join(sources, " AND "), f.Source, recovered)
}
if f.EqField != "" {
if seen := e.topValues(ctx, sig, f.EqField, others, scope); seen != "" {
message += " Values seen: " + seen + "."
Expand Down
Loading
Loading