SWIP-16 Support LLM-as-Judge on Top of GenAI Observability

[peachisai]

• If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
• Update the CHANGES log.

[wu-sheng]

This looks dangerous, could you ref to existing app log?
I feel we have a uuid kind of thing?

[peachisai]

This looks dangerous, could you ref to existing app log? I feel we have a uuid kind of thing?

Fixed

[wu-sheng]

Thanks for the work here — the module structure, the PPM sampling and pairing the records with a MAL metric are all the right shape.

I want to focus this round entirely on the query API, because query-protocol is a published contract for the UI and it is much cheaper to change now than after release. Five things, roughly in priority order. Nothing below is about code style.

1. The UI cannot render the page SWIP-16 specifies

GenAIEvaluationRecord returns only opaque ids — no serviceName, no providerName, no modelName, no operationName. To display "openai / gpt-4o" the UI would have to base64-decode SkyWalking's internal ID encoding client-side, which no other surface asks of it.

The house convention is the opposite — type Log returns both, and LogQueryService enriches the names server-side via IDManager.ServiceID.analysisId(...):

type Log {
    serviceName: String
    serviceId: ID
    serviceInstanceName: String
    serviceInstanceId: ID
    endpointName: String
    endpointId: ID
    ...
}

Worse, this is not fixable later without a data migration: AIEvaluationContext already carries serviceName, serviceInstanceName and operationName, and persistResults writes none of them. Note also that serviceId here is the GenAI provider and serviceInstanceId is the model — the calling application is not stored at all, so "show me judge scores for my chatbot service" cannot be asked now or later.

Please add serviceName/serviceInstanceName to the type (enriched in the query service, zero storage cost), and persist providerName, modelName, operationName and the caller's service name as real columns.

2. The chart → records drill-down is not expressible

The whole reason to pair a record list with a metric is that a user clicks a dip in the score chart and gets the responses behind it. gen_ai_model_evaluation_score_ppm aggregates by provider_name / model_name / task_name. The record query can filter by none of the three. There is no query that answers "show me the responses behind this point".

(Separately, SWIP-16 names the metric gen_ai_evaluation_score_ppm, but the rule emits gen_ai_model_evaluation_score_ppm — metricPrefix: gen_ai_model_evaluation + name: score_ppm. Worth fixing in the doc.)

3. The primary filters do not work on Elasticsearch, and the trace drill-down silently returns nothing on BanyanDB

Column.storageOnly() is documented as "The column is just saved, never used as a query condition", and StorageEsInstaller.createMapping stamps "index": false for it, plus "doc_values": false unless @ElasticSearch.EnableDocValues is present.

In GenAIEvaluationRecord these are all storageOnly = true: service_id (L71), service_instance_id (L76), segment_id (L80), span_id (L83), value_type (L93), value (L96), reason (L104), judge_model (L107). Yet GenAIEvaluationRecordQueryEsDAO term-queries service_id (L95), service_instance_id (L98), segment_id (L105), span_id (L108), and any whitelisted tag key (L117).

segment_id, span_id, value_type, value, reason and judge_model have no EnableDocValues, so they get index:false, doc_values:false and a term query on them cannot work on any ES version. service_id/service_instance_id keep doc values, so behaviour there depends on the ES version — which is its own problem, since we support ES 7/8/9 and OpenSearch and the same GraphQL document would behave differently across them.

On BanyanDB the drill-down fails a different way:

query.and(eq(GenAIEvaluationRecord.SPAN_ID, (long) relatedTrace.getSpanId()));  // L89

That binds an int64 against a tag registered from a String column (ES and JDBC both use String.valueOf(...)). It returns an empty list, not an error — indistinguishable to the user from "the judge said nothing about this span".

Suggested fix: drop storageOnly from every column a DAO filters on (service_id, service_instance_id, segment_id, span_id, value_type, judge_model — the same set AbstractLogRecord keeps indexed), keep it only on the 4096-char value and reason and remove those two from QUERYABLE_TAG_KEYS, and derive the whitelist from one shared constant so the annotation and the whitelist cannot drift apart again.

4. "The 20 worst-scoring responses" cannot be asked

value is a String for all four value types, and queryOrder: Order is only a direction — all three DAOs hard-wire the sort column to evaluation_time. So neither "rank the worst outputs" nor "scores below 0.5" is expressible, and that is the primary triage entry point for judge data.

We already have precedent for both halves — enum QueryOrder { BY_START_TIME, BY_DURATION } in trace.graphqls, and RecordCondition{topN, order} in record.graphqls. Suggest a real numeric score column (indexed, @BanyanDB.EnableSort), scoreValue: Float on the type, minScore/maxScore on the condition, and an explicit sort field. Please also give BanyanDB's OrderBy an explicit column — today it passes a bare new AbstractQuery.OrderBy(Sort.DESC) and only matches evaluation_time by virtue of @BanyanDB.TimestampColumn, which is an accidental coupling.

5. The filter dropdowns cannot be populated

Task names and level bands are operator config in ai-evaluation.yml, and nothing in the API exposes them — so the "filter by task / filter by level" controls SWIP-16 promises can only be built by hardcoding today's defaults, which breaks the moment an operator renames a band or adds a task.

They are also reachable only through the generic tags list, keyed by snake_case storage column names (evaluation_level) that do not match the camelCase response fields (evaluationLevel) — and a wrong key behaves three different ways: empty page on ES and JDBC, ErrTagNotDefined on BanyanDB. A UI cannot write one error path for that, and an empty page for a typo is a silent wrong answer.

Every other family that offers tags ships the autocomplete companions for exactly this reason — queryLogTagAutocompleteKeys/Values, queryTraceTagAutocompleteKeys/Values, queryAlarmTagAutocompleteKeys/Values. Please promote taskName, evaluationLevel and judgeModel to typed condition fields (the way alarm.graphqls promoted ruleNames), and either add the autocomplete queries or expose the configured tasks/bands directly.

---

Two smaller notes while the API is open: supportGenAIEvaluationRecordQueryByKeywords: Boolean! returns the DAO interface default false, is overridden by no storage, and the condition has no keyword field to pair with it — please either implement keyword search over the judge's explanation (mirroring LogQueryEsDAO) or drop the flag rather than ship a public non-null field that can never become meaningful without a breaking rename. And errorReason on the result wrapper is never set by anything, while the one real error path throws.

Lastly — nothing in the tree executes queryGenAIEvaluationRecord: no UT, no storage IT, no e2e. That is almost certainly why the three backends disagree. One e2e case per storage that writes an evaluation and reads it back with a relatedTrace drill-down and a task/level filter would have caught 3 and most of 5.