[Discussion][Observability 1/2] Recording Agent Traces in the Event Log

[joeyutong]

This proposal is the first in a two-part Flink Agents observability series:

1. Recording Agent Traces in the Event Log (this proposal)
2. Built-in Operational Metrics for Flink Agents (follow-up)

The two proposals serve complementary purposes. Agent Trace reconstructs what happened during a single run, while operational metrics describe the aggregate health and behavior of many runs.

1. Context and Existing Work

Flink Agents currently provides three main observability surfaces: metrics, Event Log, and EventListener. They provide aggregate signals, persisted business Event records, and callback hooks, but do not capture enough context to reconstruct an agent run as a causal trace.

This proposal builds on the gap identified in Discussion #710 and focuses on recording. Related community work includes:

Community work	Existing progress	Relationship to this proposal
Discussion #710	Identifies the lack of per-run causal traces and separates recording from reconstruction and visualization.	Provides the overall problem statement; this proposal addresses the recording side.
PR #923 / Issue #841	Adds minimal lineage from an emitted business Event to its source Event and emitting Action.	Provides business Event lineage. This proposal adds concrete Action execution identity and Execution Events for Action, LLM, Parser, and Tool boundaries.
Discussion #876 and PR #887	Extend the business Event taxonomy with typed Memory Events and an AgentRunBeginEvent carrying a short-term-memory snapshot.	This proposal retains them as business Events and applies the same trace context to them as to other logged Events.

To close the remaining recording gap, this proposal combines two complementary forms of context:

Agent Trace = business Event causality + execution hierarchy

2. Motivating Example

Consider an Action that calls an LLM and parses the response into a structured output:

InputEvent
  -> chat_model_action
       -> LLM call: success
       -> structured-output parser: failed

The Event Log may contain the original input, a chat request, a chat response, and an Action failure. Reconstructing the complete run requires both lineage and execution information.

Run identity and business Event lineage answer:

• Which records belong to the same run?
• Which Event triggered an Action, and which Action emitted a later Event?

Execution information answers:

• Was the failure caused by the model call or by parsing its response?
• Which concrete invocation of chat_model_action contained those operations?
• How long did the LLM call and Parser execution take independently?

Event lineage alone cannot represent the LLM and Parser boundaries because those operations are not business Events. A complete trace therefore needs both forms of context.

3. Goals and Non-goals

Goals

• Associate Event Log records with one run, business key, and Agent.
• Integrate existing business Event lineage with concrete Action execution records.
• Represent Action, LLM, Parser, and Tool execution boundaries.
• Reconstruct parent-child relationships between nested executions.
• Record execution outcomes and stable failure categories.
• Preserve the original business Event payload.
• Keep Java and Python Event Log records semantically aligned.
• Extend the existing Event Log instead of introducing an independent tracing backend.

Non-goals

• Trace query APIs or storage indexing.
• Trace reconstruction or visualization.
• OpenTelemetry export.
• Evaluation datasets or experiment tracking.
• Dashboards, alerts, or other product-level capabilities.
• Provider-internal attempt tracing for retries.
• Session-level tracing across multiple inputs.

4. Proposed Trace Model

The existing Event Log model records a business Event together with its occurrence context:

Existing EventLogRecord
  = Event
  + EventContext

This model records which business Event occurred and when it was processed. It does not identify the run or execution to which the record belongs, and it cannot represent runtime work that is not a business Event.

The proposed model extends the record as follows:

Proposed EventLogRecord
  = Event or Execution Event
  + EventContext
  + ExecutionTraceContext (when Trace persistence is enabled)

The proposal leaves the responsibilities of Event and EventContext unchanged. It adds Execution Events for runtime lifecycle boundaries and ExecutionTraceContext for run identity and execution hierarchy. Together, business Events describe workflow causality, while Execution Events describe the runtime work performed at each Action node.

Abstraction	Responsibility
Event	Describes what happened through an Event id, type, and attributes.
Execution Event	Describes the lifecycle and result of one execution.
EventContext	Carries occurrence metadata, currently the Event type and timestamp.
ExecutionTraceContext	Associates the record with a run, Action scope, execution hierarchy, and execution entity.
EventLogRecord	Combines an Event, its occurrence context, and its trace context into one persistent record.

4.1 Business Event and Execution Event

Business Events and Execution Events have distinct semantics:

• A business Event represents a workflow occurrence and retains its original payload. It continues to participate in Event routing, EventListener callbacks, and existing business Event metrics.
• An Execution Event represents the lifecycle of one execution. It records started, finished, failed, or reused and is used only for runtime observation.

An Execution Event is a runtime recording abstraction, not a new public Event base class. The built-in lifecycle reporting path sends it directly to observability consumers without automatically submitting it to EventRouter; it therefore does not trigger Actions, notify EventListener, or contribute to business Event metrics. This integration boundary does not prohibit a future explicit routing API.

Trace information remains outside user-owned Event attributes. A typed business Event, including a Memory Event or AgentRunBeginEvent, can receive the same run and execution context when written without changing its payload schema.

4.2 Trace Relationships

In this proposal, a run is one processing instance initiated by an input for one business key. It is not a user session and does not span multiple inputs. The serialized field is named inputRunId to make this scope explicit.

An execution is one concrete framework-observed invocation of an Action, LLM, Parser, or Tool. Invoking the same entity twice within one run creates two distinct executions. A framework retry that invokes an entity again therefore creates another execution; retries performed internally by a provider remain outside this model.

The trace combines two relationships:

1. Business Events connect Action executions into a causal graph.
2. Parent execution identifiers connect operations performed inside an Action into an execution tree.

flowchart TB
    IN["Input Event"] -->|"triggers"| A1["Action execution A"]
    A1 -->|"contains"| L1["LLM execution"]
    A1 -->|"contains"| P1["Parser execution"]
    A1 -->|"emits"| E1["Business Event E"]
    E1 -->|"triggers"| A2["Action execution B"]
    A2 -->|"contains"| T1["Tool execution 1"]
    A2 -->|"contains"| T2["Tool execution 2"]

Loading

For business Event causality:

• Existing business Event lineage identifies the source Event and the Action that emitted a later Event.
• A business Event emitted by an Action carries that Action execution's executionId, linking the lineage to a concrete execution.
• Action executions connected by business Events are siblings rather than parent and child executions.

For execution hierarchy:

• The run groups all trace records and serves as the trace root; it is not itself an execution.
• LLM, Parser, and Tool executions are children of the Action execution in which they run.

Together, business Event records reconstruct the workflow graph, while Execution Events and their parent identifiers reconstruct the work performed within each Action execution.

5. Event Log Record

The serialized representation remains a flat JSONL record so that commonly queried fields can be indexed and aggregated without parsing nested envelopes. The in-memory model preserves the separation between Event, EventContext, and ExecutionTraceContext.

Field group	Fields	Source	Purpose
Event occurrence context	timestamp	EventContext	Records when the Event occurrence was processed.
Event	eventId, eventType, eventAttributes	Event or Execution Event	Preserve the Event identity, semantics, and payload.
Run identity	inputRunId, businessKey, agentName	ExecutionTraceContext	Group records from one input-processing run.
Execution hierarchy	executionId, parentExecutionId	ExecutionTraceContext	Identify concrete executions and reconstruct containment.
Execution entity	entityType, entityName, entityMetadata	ExecutionTraceContext	Identify the primary execution entity and carry its supplemental metadata.
Execution outcome	status, problemCategory	Execution Event attributes	Describe the lifecycle result and stable failure category of an Execution Event.

EventContext mirrors Event.getType() as part of its occurrence contract, and the serializer writes that value once as eventType. For Execution Events, status and problemCategory are promoted from Event attributes to top-level fields during serialization.

entityType and entityName identify the primary execution node, such as an Action, LLM, Parser, or Tool invocation. entityMetadata contains supplemental attributes for that node; it neither introduces another execution node nor defines a parent-child relationship.

parentExecutionId represents execution containment. Business Event lineage remains responsible for cross-Action causality, while a business Event emitted by an Action carries the same executionId as that Action to connect the two models.

For example, a structured-output parsing failure may be written as:

{
  "timestamp": "2026-07-02T01:23:45.678Z",
  "inputRunId": "run-1",
  "businessKey": "order-1",
  "agentName": "order-agent",
  "executionId": "parser-execution-1",
  "parentExecutionId": "action-execution-1",
  "entityType": "parser",
  "entityName": "structured_output",
  "eventId": "event-17",
  "eventType": "_execution_failed_event",
  "status": "failed",
  "problemCategory": "model_output_parse_error",
  "eventAttributes": {
    "errorType": "JsonProcessingException",
    "errorMessage": "Unable to parse the model response"
  }
}

eventId identifies a logical Event occurrence; it does not imply that the corresponding physical log record is written exactly once.

6. Recording Boundaries

Run Context

When the runtime starts processing a new input, it creates a run context containing:

• inputRunId
• businessKey
• agentName

The same context is propagated to business Events and executions created during the run. If a checkpointed ActionTask is restored, its existing run and execution context is restored with it rather than recreated.

Action

The runtime records one Action execution for each concrete invocation:

• started before invoking the Action.
• finished after successful completion.
• failed when an unhandled Action exception terminates the invocation.
• reused when ActionStateStore returns an already completed Action result without re-executing the Action.

Action-level failures use action_execution_failed; the runtime does not infer a more specific category from a child failure. LLM, Parser, and Tool executions record their own precise failure categories.

LLM

An LLM execution represents one framework invocation of ChatModel. If framework retry logic calls the model again, the retry creates another LLM execution. Retries hidden inside the provider or connection implementation are not separately observed.

• A returned model response completes the current LLM execution successfully before structured-output parsing begins.
• A model-call failure completes the current LLM execution with model_call_failed, even when the framework subsequently retries it.

Parser

Each structured-output parsing attempt has its own Parser execution so that consumers can distinguish between:

• A model call that failed.
• A model call that succeeded but returned an unparseable response.

Parser failures use model_output_parse_error. Recording the Parser boundary must not change the existing retry or ignore behavior of structured-output processing.

Tool

A Tool execution represents one concrete tool invocation. If a ToolRequestEvent requests multiple calls, each invocation produces a separate Tool execution.

• A successful Tool response records success.
• A Tool error response keeps the original ToolResponseEvent payload but records the Tool execution as failed.
• A thrown Tool exception records tool_call_failed and preserves the underlying error type and message.

Tool calls, MCP Servers, and Skills have different roles in the trace model:

Concept	Execution node	Trace representation
Tool call	Yes	Each invocation is one Tool execution.
MCP Tool call	Yes, as a Tool execution	The MCP Server is the resource exposing the Tool, not another execution level.
MCP Server	No	Recorded as resource provenance on its MCP Tool executions.
Skill	No	Represents instructions and resources; only an explicit load_skill invocation is an execution.

For example, an MCP Tool call has one execution node:

Action execution
  -> Tool execution: search
       metadata:
       - tool type: MCP
       - MCP Server: search_server

In the serialized record, entityType and entityName identify the Tool execution as the primary entity. Its Tool kind and MCP Server provenance are stored in entityMetadata, not represented as child execution nodes:

{
  "entityType": "tool",
  "entityName": "search",
  "entityMetadata": {
    "toolType": "mcp",
    "mcpServer": "search_server"
  }
}

The MCP Server therefore has no separate execution id or lifecycle Events. Its metadata can still be used to query or aggregate related MCP Tool executions.

load_skill is a built-in Tool that reads the requested Skill's SKILL.md content or a specific resource and returns it to the model. Its invocation is represented as a Tool execution, with the Skill name and optional resource path stored as metadata. The runtime does not propagate a Skill execution context to subsequent Tool calls, so the trace does not attribute those calls to the loaded Skill.

Retry and Replay

Retry and replay do not have dedicated Event types. A framework retry appears as another LLM or Parser execution under the same Action. Provider-internal retries are not visible. Runtime replay may instead reproduce Event Log records as described in the recovery contract below.

7. Runtime Integration and Compatibility

Business Events and built-in execution lifecycle reporting follow separate semantic paths but share the same Event Log writer:

flowchart TD
    BE["Business Event"] --> ER["EventRouter"]
    ER --> EL["EventListener"]
    ER --> BM["Business Event metrics"]
    ER --> AR["Action routing"]
    ER --> W["Event Log writer"]

    EE["Execution Event"] --> EM["Execution metrics"]
    EE -. "trace enabled" .-> ES["ExecutionEventSink"]
    ES --> W
    W --> LOG["Event Log"]

Loading

This separation preserves existing externally observable behavior:

• Business Events retain their existing EventRouter behavior.
• Built-in lifecycle reporting does not submit Execution Events to EventRouter, so they do not trigger Actions or notify user EventListener implementations.
• Both paths use the same writer lifecycle and output format.
• Event Log writes remain best-effort and do not affect the outcome of an Action.
• Existing EventLogger implementations can continue to handle Event and EventContext without consuming trace context.

Trace persistence is controlled by event-log.trace.enabled, which defaults to false:

• When disabled, business Events continue to be logged without ExecutionTraceContext, while Execution Events are not persisted.
• When enabled, business Events include trace context and Execution Events are written through the shared Event Log writer.
• The option affects only Event Log persistence. In-process execution reporting remains active so that built-in metrics retain the same semantics.

New records use a normalized, flat JSON format with camelCase field names. Built-in Event type values retain their existing snake_case names, such as _execution_started_event. The deserializer continues to support the previous nested format. Legacy records have no run or execution context and therefore cannot form a complete trace.

When Trace persistence is enabled, explicit execution boundaries add lifecycle records, so log volume grows with the number of recorded Action, LLM, Parser, and Tool executions. The proposal uses the existing Event Log path instead of introducing a separate tracing exporter or recording pipeline.

8. Recovery and Replay Semantics

The Event Log is best-effort and does not provide exactly-once delivery. Run identity follows Flink's restore-versus-replay boundary:

• A checkpointed ActionTask carries its ExecutionTraceContext in Flink state. Restoring that task preserves its existing run and execution identities.
• If the source replays an input that was not represented by a restored ActionTask, the runtime processes it as a new run with new run and execution identities.
• Each resulting run retains an internally consistent execution hierarchy.

ActionStateStore adds a separate reuse path:

• A completed Action can be reused without re-executing user code.
• The current Action execution records reused instead of started and finished.
• Persisted output Events retain their Event ids and are logged under that reused Action execution, allowing consumers to present them as outputs replayed from durable state.

9. Prototype Status

The model has been prototyped across the Java and Python runtime paths. The prototype validates:

• A shared Event Log schema for business Events and Execution Events.
• Action, LLM, Parser, and Tool lifecycle records, including success and failure paths.
• Distinct LLM and Parser executions for framework-level retry attempts.
• Business Event lineage alongside nested executions and multiple Tool calls from one request.
• MCP Server and explicit Skill metadata.
• Optional Trace persistence that does not disable in-process execution metrics.
• Java and Python semantic alignment through conformance tests and end-to-end probes.

The purpose of this discussion is to review the model and contracts before submitting the implementation upstream.

10. Summary

Business Event lineage is necessary for Agent Trace, but it is not sufficient. A complete trace must also represent runtime executions that are not business Events.

This proposal extends the existing Event Log with:

• Run identity and business key association.
• Integration with existing business Event lineage.
• Action, LLM, Parser, and Tool execution lifecycle records.
• Parent-child execution relationships.
• Stable execution outcomes and failure categories.
• A shared recording format for Java and Python.

The result is a framework-native recording model that allows consumers to reconstruct a run without changing business Event payloads or introducing a separate tracing system.

[Zhuoxi2000]

Thanks for writing this up. the model reads well. One thing that might be worth mentioning explicitly is that the fields line up pretty closely with the OpenTelemetry GenAI semantic conventions.

Since OTel export is listed as a non-goal here, I read that as a possible follow-up rather than a “never,” so calling out the alignment could make that future path clearer.

Concretely:

This proposal	OTel GenAI
input_run_id	trace_id
execution_id / parent_execution_id	span_id / parent_span_id
entity_type + entity_name	gen_ai.operation.name + gen_ai.tool.name / gen_ai.agent.name
business_key	gen_ai.conversation.id
status / problem_category	span status + error.type

Two things might be worth deciding while the contracts are still open:

1. ID shape. OTel trace_id / span_id are fixed-width, 128-bit / 64-bit IDs. If these remain opaque strings, an exporter can hash them, but that hash needs to be stable across restarts and replays, so the ID derivation still matters.

2. Run identity. The doc says a run “creates or restores a run context,” but doesn’t define how input_run_id is derived. In [Feature] Add lineage fields to the event log for per-run trace reconstruction #841, we seemed to converge on deriving it from stable runtime coordinates rather than inputEvent.id, since Java creates a fresh UUID when wrapping the event and that isn’t replay-stable. I think the same replay-stability concern applies here as well.

Happy to take on the OTel export follow-up once this lands, if helpful.

[joeyutong]

Thanks for pointing this out and for offering to help. I agree that OpenTelemetry export should be a supported follow-up, and I believe the proposed recording model provides the information needed to implement it: an input run defines the Trace boundary, the execution hierarchy provides the Span topology, and lifecycle Events provide execution timing and outcomes. The exact GenAI attribute mapping can be worked out as part of the exporter design.

For ID shape, I would prefer to keep the framework IDs opaque rather than constrain them to the OTel representation. An exporter can derive valid fixed-width Trace and Span IDs from the native identities.

The current input_run_id represents one runtime processing instance. It is preserved when an existing ActionTask is restored from Flink state, while a source-replayed input creates a new run and would therefore produce a new OTel Trace. I have not yet identified a Trace or OTel export requirement that requires the ID to remain stable across source replay; a concrete use case would help determine whether that semantic should change.

An OTel exporter follow-up would be very welcome once the recording contract is settled.

[Zhuoxi2000]

Thanks! The checkpoint-aligned run identity plus status="reused" for replayed actions resolves my replay-stability concern. I also agree that keeping Execution Events out of Action routing is the right call, and event-log.trace.enabled defaulting to off makes sense given the log-volume concern @weiqingy raised.

Since this covers trace recording, I’d be happy to pick up the export side as a follow-up: mapping Execution Events to the OpenTelemetry GenAI semantic conventions, including invoke_agent / execute_tool / chat spans and gen_ai.usage.* token attributes, gated behind config in the same way.

I’ve implemented the inverse mapping, OTel GenAI to storage, in a side project, so I have a concrete attribute table to start from. Happy to open a [DISCUSS] with the full mapping if that would be useful for Observability 2/2.

[weiqingy]

Thanks for the detailed write-up. The split between Business Events and Execution Events reads right to me. A few questions on the recording contract before implementation, mostly around run identity, execution lifecycle, and replay boundaries.

Run identity. The proposal says the runtime creates or restores a run context, but doesn't define how input_run_id is derived. This reinforces the replay-stability concern already raised on the thread, with a second failure mode on the Python side: in #841 the worry was that inputEvent.id isn't stable enough. Java may mint a fresh UUID when wrapping an input on replay, while Python's content-derived IDs can collapse distinct inputs with identical payloads. Would it make sense to derive input_run_id from stable runtime coordinates, such as job/agent identity + business key + per-key input sequence, or another explicitly replay-aware source?

Execution counting. §4.2 and §6 seem to point in slightly different directions: §4.2 says invoking the same entity twice creates two distinct executions, while §6 says retry attempts don't count as separate LLM executions. In the structured-output path, the model call and parse may be retried together, so a call that succeeds after parse retries could read as one LLM execution but several Parser executions under the same Action. Would it be worth stating the retry-collapsing rule once at the execution level, so Parser and LLM counting are consistent, for example one execution per logical call carrying an attempt count?

Boundary safety. Since Execution Events reuse the existing Event structure, and the _ prefix today marks types that route normally (_input_event, _chat_request_event, etc.), would it be worth making "Execution Events never route" an explicit invariant backed by a focused test? That would keep a future call path from accidentally treating an _execution_* record as a routable business Event.

Operational cost. Explicit Action/LLM/Parser/Tool lifecycle records can multiply Event Log volume noticeably for tool-heavy agents. Would the first implementation include a recording switch or sampling option, or at least clear guidance on when to expect log growth and whether recording is on by default?

Durable replay. For ActionStateStore replay, a reused Action records a reused Execution Event, but its persisted output Events continue through the current run without the original ExecutionTraceContext. That's honestly noted as a limitation. Could it make trace reconstruction misleading in durable mode, where a replayed output looks freshly produced? Would it be feasible to store the original trace context with the completed Action, or, failing that, mark replayed output Events as "replayed from prior execution" so consumers don't infer a fresh causal production?

[joeyutong]

Thanks for raising these questions. They help clarify several boundaries that were not explicit enough in the proposal.

Run identity. At present, input_run_id identifies one runtime processing instance. It is preserved when an existing ActionTask is restored from Flink state, but a source-replayed input receives a new ID, so the current behavior follows Flink's resume-versus-replay boundary and is partly checkpoint-dependent. After reviewing #841, I have not yet identified a Trace requirement that necessarily requires replay stability. Do you have a concrete case where input_run_id must remain stable across source replay?

Execution counting. Good catch. The current wording and prototype are not fully aligned: the LLM lifecycle currently wraps the retry loop, while each Parser invocation is recorded separately. I think the cleaner contract is one execution per framework-observed invocation. Each retry that calls the model again would create a new LLM execution, and each parse attempt would remain a separate Parser execution; provider-internal retries would remain out of scope. A separate retry Event or attempt count is unnecessary because the repeated executions are already visible under the same Action.

Boundary safety. The prototype intentionally keeps built-in Execution Event reporting outside EventRouter, so enabling observability does not implicitly trigger Actions, notify user listeners, or otherwise change workflow behavior. However, making “Execution Events never route” a permanent restriction may be unnecessarily strong. I would define and test the narrower contract that built-in lifecycle reporting does not automatically submit these Events to EventRouter, while leaving room for an explicit integration in the future.

Operational cost. A coarse-grained Trace recording option seems appropriate for the initial implementation. I propose an event-log.trace.enabled option that controls whether trace context and Execution Events are persisted to the Event Log, while in-process lifecycle observation remains active for metrics. I would default it to disabled to avoid increasing Event Log volume for existing jobs. Sampling can be considered later, but it should be decided per run rather than per Event so that recorded Traces remain complete.

Durable replay. The current recording should not make a replayed output look freshly produced. When a completed ActionState is reused, the runtime records the current Action execution with status=reused rather than started or finished, and the persisted output Events are associated with that same reused execution while retaining their original Event IDs. A Trace consumer can therefore represent them as outputs replayed from durable state.

[weiqingy]

Thanks, this resolves what I raised. Execution counting, boundary safety, and the trace-enabled default all read settled to me.

Run identity. You're right on the general case. A replayed record with no restored Action state is genuinely a second execution, so a fresh run id describes what happened.

The one case I land on is replay combined with durable-execution reuse, since ActionState survives a restart in an external store. The replayed input gets a fresh run id and its Action is recorded reused, while every nested LLM and Tool execution for that same work was already flushed under the previous run id. Your point about persisted output Events keeping their original IDs does give a way to bridge the two, which I had underrated, though it goes through the business-event stream rather than the trace itself.

So my question is narrower than run-id stability: would it be worth the reused execution carrying the identity of what it reuses, either the original execution_id or input_run_id? Today it carries only status. That keeps your resume-versus-replay rule as is, and for an OTel exporter it is the difference between two disconnected traces and a link.

Since the implementation is up in #924, I have left my remaining comments there and will keep further detail on the PR.