OLS-2987: Define agentic telemetry data model

[blublinsky]

Summary
Delivers the schema definition document for the agentic telemetry data model. Defines four tiers of telemetry (step metrics, proposal summary, transcript, audit trail), data classification, and extensibility principles.

Key design decisions
Nested structure: Proposal summary contains steps[] array with per-step metrics derived from Result CRs
Tiers are independent: Each tier works without the others; no breaking changes when adding higher tiers
Transcript is agent-owned: The operator does not collect prompt/tool data — the agent logs it
Audit trail extends Proposal Summary: Not a separate record — adds identity/authorization columns
Sensitivity gates export: Non-sensitive ships by default; medium requires opt-in; high requires explicit configuration
finalPhase instead of "outcome" — maps directly to operator's DerivePhase() terminology

Open questions (requiring BU input)
Which Proposal Summary fields does the product need for analytics?
Dataverse table structure — reuse OLS transcript schema or new?
Export cadence — real-time or batch?
Retention policy?
Which compliance regulations apply?

[openshift-ci-robot]

@blublinsky: This pull request references OLS-2987 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary
Delivers the schema definition document for the agentic telemetry data model. Defines four tiers of telemetry (step metrics, proposal summary, transcript, audit trail), data classification, and extensibility principles.

Key design decisions
Nested structure: Proposal summary contains steps[] array with per-step metrics derived from Result CRs
Tiers are independent: Each tier works without the others; no breaking changes when adding higher tiers
Transcript is agent-owned: The operator does not collect prompt/tool data — the agent logs it
Audit trail extends Proposal Summary: Not a separate record — adds identity/authorization columns
Sensitivity gates export: Non-sensitive ships by default; medium requires opt-in; high requires explicit configuration
finalPhase instead of "outcome" — maps directly to operator's DerivePhase() terminology

Open questions (requiring BU input)
Which Proposal Summary fields does the product need for analytics?
Dataverse table structure — reuse OLS transcript schema or new?
Export cadence — real-time or batch?
Retention policy?
Which compliance regulations apply?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign raptorsun for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

📝 Walkthrough

Summary by CodeRabbit

• Documentation
  • Added comprehensive telemetry data model specification defining metrics collection across four tiers: step metrics, proposal summaries, transcripts, and audit trails. Includes field schemas, storage guidance, and export policies with privacy and compliance controls.

Walkthrough

A new spec file .ai/spec/what/telemetry.md defines the Agentic Telemetry Data Model. It establishes four tiers (Step metrics, Proposal summary, Transcript, Audit trail), specifies field-level schemas and Result CR storage locations, defines a data classification matrix with sensitivity-gated export rules, declares extensibility constraints, and lists open questions.

Changes

Agentic Telemetry Data Model Spec

Layer / File(s)	Summary
Overview and core per-step/proposal tiers .ai/spec/what/telemetry.md	Introduces four telemetry tiers. Defines StepMetrics schema (latency, tokens, cost, model/provider, tool call count) stored on Result CR status objects. Specifies the Proposal Summary aggregate shape with nested per-step fields, an example JSON payload, and an open questions subsection on computation/storage/export.
Transcript and Audit Trail tiers .ai/spec/what/telemetry.md	Documents that transcript capture is the agent's responsibility via stdout/pod logs (not CRDs), with per-tool-call structure and opt-in export due to sensitivity. Defines the Audit Trail extension adding identity/authorization/compliance fields to the Proposal Summary record; clarifies behavioral metrics are derived, not collected directly.
Data classification, extensibility, and open questions .ai/spec/what/telemetry.md	Adds sensitivity classification matrix: non-sensitive shipped by default, medium opt-in, high requires explicit configuration. Specifies redaction for failure reasons and strong restrictions on prompt content, reasoning traces, and tool outputs. Declares additive-only extensibility with exporters decoupled from storage. Lists BU open questions on analytics fields, Dataverse schema reuse, export cadence, retention policy, and compliance regulations.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly summarizes the main change: defining an agentic telemetry data model spec.
Description check	✅ Passed	The description covers the key design decisions, data tiers, and open questions relevant to the telemetry schema being introduced.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

.ai/spec/what/telemetry.md (1)

245-245: ⚡ Quick win

Clarify the analytics question with concrete product intent.

The phrasing "make the system better" is vague. Specify what analytics outcomes or KPIs you're targeting (e.g., reliability metrics, cost optimization, bottleneck identification, etc.).

💡 Suggested revision

-1. Which Proposal Summary fields does product actually need for "make the system better" analytics?
+1. Which Proposal Summary fields does product actually need for analytics on system reliability, cost, and performance optimization?

Or a more generic version:

-1. Which Proposal Summary fields does product actually need for "make the system better" analytics?
+1. Which Proposal Summary fields does product actually need to improve observability and inform system optimizations?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.ai/spec/what/telemetry.md at line 245, Replace the vague phrase "make the
system better" in the question at the specified location with concrete,
measurable analytics outcomes and KPIs. Specify what product actually intends to
measure or optimize for, such as reliability metrics, cost optimization,
bottleneck identification, performance analysis, or other specific product
objectives. This clarification will help define which Proposal Summary fields
are truly needed for the analytics goals.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.ai/spec/what/telemetry.md:
- Line 245: Replace the vague phrase "make the system better" in the question at
the specified location with concrete, measurable analytics outcomes and KPIs.
Specify what product actually intends to measure or optimize for, such as
reliability metrics, cost optimization, bottleneck identification, performance
analysis, or other specific product objectives. This clarification will help
define which Proposal Summary fields are truly needed for the analytics goals.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5d401695-728e-4ee5-9b3d-bfee00a9a307

📥 Commits

Reviewing files that changed from the base of the PR and between 4a22a7e and cc1255a.

📒 Files selected for processing (1)

• .ai/spec/what/telemetry.md

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

• openshift/lightspeed-agentic-sandbox (manual)