[Discussion] Agent observability: tracing & evaluation beyond metrics + event log

[addu390]

Context

Flink Agents today ships three observability surfaces:

1. Metrics: event/action counters, token usage per model, plus a custom-metrics API, via Flink metric reporter.
2. Event log: structured records per event, with log levels.
3. EventListener: per-event callback, configured via event-listeners.

That covers aggregates (metrics), audit records (event log), and hooks (listener). What it doesn't cover is reconstructing a single run as a causal tree:

InputEvent
 └─ action: classify
      └─ ChatRequest → ChatResponse
 └─ action: tool_use
      └─ ToolRequest → ToolResponse
      └─ ChatRequest → ChatResponse
 └─ OutputEvent

That tree shape is what LangSmith (and Langfuse, Phoenix, etc) render for debugging, or where MLflow slots into for batch eval runs (one run = one trace, with metrics, prompts, and outputs tracked across versions of the agent). There is also OpenTelemetry GenAI semantic conventions. Makes debugging and battle-testing non-trivial agent workflows tractable.

Scope

Not proposing this as a default for production streaming jobs. Full-fidelity per-event tracing at streaming QPS is too heavy, metrics + event log stay the right production defaults.

The question is whether the framework should provide first-class support for tracing where it actually pays off:

• Local authoring loop
• CI / batch eval and replay
• Staging mini-cluster runs
• Canaried / sampled production

In all four, you want a single run rendered end-to-end. Today you piece it together from event-log records.

Open question

Is this worth the framework solving? Curious if others hit this in their own dev loop.

[xintongsong]

Hi @addu390,

+1 on this. Being able to reconstruct and visualize the trace of a single run is really helpful for observing and understanding agent behavior, and it's exactly the gap that neither metrics nor the event log fills today.

I'd suggest splitting this into two fairly independent problems:

• Recording: keep enough information in the event log to reconstruct the causal tree.
• Reconstruction & visualization: rebuild and render a single run from that information.

Once you split it this way, the overhead concern mostly lands on the first part, and I think that part is actually pretty light. We already record every event in the event log. To reconstruct the causal tree, we basically just need one extra field per event: which action emitted the event. With that, the whole run can be rebuilt from the event log.

The reverse edges (which actions an event triggered) don't even need to be recorded explicitly. They can be derived from the action trigger rules plus timestamps. So the increment on the recording side is small, and I think it can be kept separate from the concern about full-fidelity tracing being too heavy at streaming QPS.

The second part can be fully on-demand rather than always-on. We only run it when needed, e.g. local debugging, CI/eval, or staging. The rendering form doesn't have to be a tree either; PlantUML or something else would work too, and we can discuss that separately. As long as the recording side captures the necessary info, there's a lot of flexibility in what we do with it later.

[addu390]

Hi @xintongsong. Agreed on the split.

The overhead (additional CoP) concern I raised was coming from also wanting to be OTel-compatible on the recording side, so existing AI observability tools can pick up our runs out of the box, e.g. how LangSmith does it: trace-with-opentelemetry.

But as you pointed out, we already track almost everything needed, a few additional fields on the event log would be enough to render the full run trace. If those land as the log, the overhead concern goes away.

Any hard takes on OTel compatibility and to use GenAI semantic conventions, or keep it framework-native for now?

[xintongsong]

Hi @addu390,

I don't have a strong opinion on OTel compatibility itself; it mostly comes down to whether there's clear demand from users.

On the implementation side, OTel could plug in as just another logger for the event log. The event log already writes through pluggable loggers (currently SLF4J and FILE), so OTel could sit alongside them as a third option. For users it would just be picking an output backend among otel / slf4j / file, which means the same data isn't recorded twice and runtime overhead shouldn't change much. One caveat: wiring OTel in might require some changes to the current plugin interface.

I slightly lean toward waiting until there's clear demand before building the OTel logger, but I'm also okay with you or anyone else picking this up now, as long as it doesn't affect existing functionality or performance.

[addu390]

@xintongsong Sounds good, I don't have a strong take either. I was experimenting to potentially migrate a few workloads to flink-agents and tracing did come across as bit of a gap.

A good starter would be to get the trace-tree per run, right within the logs. I'll create an issue for that? and down to pick that up too.

[xintongsong]

Sounds great. Please feel free to move ahead.

[rosemarYuan]

Event Lineage: Reconstructing a Trace Tree Based on Event Causality

TL;DR: I suggest splitting Agent observability into two independent phases. The first phase adds only two framework-managed causal fields to each Event to build a minimal Event Trace Tree. A later phase can introduce a complete record model 
that includes a run ID, execution levels, lifecycle information, and failure details.  

Based on Discussion #710, #841, and #886, this proposal focuses only on reconstructing the runtime causal path between business 
Events and Actions. This problem is independently useful and can be addressed through a smaller, lowerimpact change than the 
complete trace model proposed in Discussion #900.

Goals

The goal of this design is to rebuild a Trace Tree from the Event Log, with an InputEvent as the root, as shown below:

InputEvent E0
├─ [Action: action_a]  // branch a
│  └─ ChatRequestEvent E1
│     └─ [Action: chat_model_action]
│        └─ ChatResponseEvent E2
│           └─ [Action: output_a]
│              └─ OutputEvent E3
└─ [Action: action_b]  // branch b
   └─ ChatRequestEvent E4
      └─ [Action: chat_model_action]
         └─ ChatResponseEvent E5
            └─ [Action: output_b]
               └─ OutputEvent E6

This makes it easy for users to see which Event triggered an Action and which later Events were output by that Action.

EventLineage 

To rebuild the Trace Tree above, the Event Log must record each relationship in the path：

Event → triggered Action → new Event emitted by the Action

Consider the following causal chain:E1 → [A1] → E2 → [A2] → E3

Using E2 as the reference point, the lineage can be recorded in either direction:

• Forward: E2 records that it triggered A2 and that A2 emitted E3.
• Backward: E2 records that it was emitted by A1, which was triggered by E1.

The forward approach requires updating E2 after A2 completes, because E3 is not yet known when E2 is first recorded.

The backward approach requires no later update. When A1 emits E2, the framework already knows E1, A1, and E2. Therefore, the first 
phase uses the backward approach.

Therefore, the first phase uses the second approach. Each Event emitted by an Action records:

• the ID of its direct causal parent Event;
• the name of the Action that emitted it.

Following these references backward is sufficient to reconstruct the complete lineage tree for the available Event records.

API

Event already records an id. This design only adds two fields to Event:

Field	Type	Meaning
upstream_event_id	UUID	ID of the Event consumed by the Action that emitted the current Event
upstream_action_name	String	Name of the Action that emitted the current Event

These two fields are maintained automatically by the framework. Users do not need to set them when creating an Event, and they are not part of the business attributes. The Event Log will continue to serialize the complete Event, so the fields will appear directly in 
the event object without adding a new log record model.

The Event Log continues to serialize the complete Event object. No new top-level Event Log record model is introduced in this phase.

An Event returned by the Event Log could therefore look like:

{
  "timestamp": "2026-07-13T10:00:00.000Z",
  "eventType": "tool_request",
  "event": {
    "eventType": "tool_request",
    "id": "f9cbb921-664a-4acd-b1a0-050fa8857657",
    "upstreamEventId": "b438dd06-b619-4763-bfd7-8eb7388f01ac",
    "upstreamActionName": "chat_model_action",
    "attributes": {}
  }
}

Design

Lineage Design

ActionTask already stores both the Event that triggered the current Action and the Action being run:

actionTask.event   = the Event that triggered the current Action
actionTask.action  = the Action currently being run

Therefore, when processing an Event output by ActionTask, the framework can directly assign:

event.upstream_event_id    = actionTask.event.id
event.upstream_action_name = actionTask.action.name

Both values are available from the existing ActionTask, so no separate context object is needed.

Why a Run ID Is Not Required in the First Phase

Each non-root Event directly references its parent Event by ID. Each connected component in the resulting parent-child graph 
therefore forms one independently reconstructable lineage tree.

For a complete log, following parent references is sufficient to associate every descendant Event with its root.
This approach has limitations:

• If the log only contains part of a run, the missing records cannot be recovered.
• Without a run ID, locating every record for one run may require scanning or indexing Event IDs.
• A run ID will still be valuable for the complete execution record model proposed in Discussion [Discussion][Observability 1/2] Recording Agent Traces in the Event Log #900.

These limitations are acceptable for the first-phase goal of reconstructing direct Event–Action causality.

Trace Tree Reconstruction

A local reader or script can rebuild and display the Event lineage tree from a saved JSONL Event Log. The reconstruction process is:

1. Read EventLogRecord entries from the JSONL file;
2. Build a node index by event_id;
3. Treat records with an empty upstream_event_id as root nodes;
4. Connect parent and child nodes by upstream_event_id;
5. Use upstream_action_name to create or reuse Action nodes, then attach the current Event under the matching Action node.

The Trace Tree is derived on the reading side. It does not add any new fields to the Event Log. For example, the script renders an 
InputEvent -> MiddleEvent -> OutputEvent lineage as:

Trace Tree 1:
  _input_event (dad5ed00-80e2-4746-8bdb-f126bad504b5)
    [Action: action1]
      MiddleEvent (39361629-4f1d-4b62-b734-a48b181cb6e0)
        [Action: action2]
          _output_event (f452ce08-c672-4c9c-841f-d81d15a900c5)

The visualization provides two basic output formats: a text tree and Trace Tree JSON. The vertical order in the UI is only for easier 
reading. Action nodes are virtual nodes derived on the reading side; they do not represent separate Action log records.  

Thanks again to @addu390, @zxs1633079383, for the initial exploration of this topic and for giving me the opportunity to continue the design. With #900 proposing a more complete execution-based model for Event relationships, lifecycle information, and failure 
details, I believe the lineage layer here can remain intentionally minimal and focus only on the causal link needed to reconstruct the 
Event–Action–Event path.I would really appreciate your feedback on whether this simplified design and the proposed boundary 
between the two phases make sense.

[weiqingy]

Thanks for writing this up — I like the backward-edge direction. Because the framework already knows the triggering Event and the emitting Action when an Action emits an Event, recording upstream_event_id + upstream_action_name can reconstruct the Event → Action → Event tree without later-update writes. The two-field first phase reads sensible to me. A few questions, mostly around field placement and Java/Python reconstruction behavior.

Where the fields live. One question is whether Event itself is the right carrier. Even outside attributes, Event is still the public business-event object: it flows through routing, listeners, action state, and cross-language serde. #900 moves trace data toward a separate ExecutionTraceContext / log-record envelope, so putting lineage on Event risks creating a second metadata location that #900 later has to reconcile.

There is a real pull the other way, though: if the fields ride on the persisted Event, they survive ActionStateStore replay for free. A reused Action re-emits its persisted output Events verbatim, so the causal edge comes back intact, whereas #900 §8 notes that separate trace context is not stored with completed Actions and replayed outputs cannot link back to the original execution. So this feels like a genuine trade-off rather than a simple "Event is the wrong home" question. Where do you see it landing: lineage on the log record/context like #900, or on the Event to preserve the replay property?

Python identity. Two Python-specific details seem worth pinning down. First, Python Event.id is content-derived from event content excluding id, so adding framework-managed fields to the Event model could change the id unless those fields are explicitly excluded. That invariant seems important if the fields stay on Event.

Second, because Python ids are already content-derived, two runs triggered by the same input payload can get the same root id. With rootless reconstruction — index by event_id, empty upstream_event_id means root — those distinct runs can collapse into one tree, while Java's per-occurrence UUIDs keep them apart. Does that make "no run id" more than a locating limitation for Python? How should same-payload roots stay distinct?

Ordering and partial logs. With only upstream_event_id + upstream_action_name, multiple sibling Events from one invocation, retries/replay, or repeated invocations of the same Action from the same source Event may be hard to order or tell apart. Partial logs also cannot recover a root. Would a minimal invocation id or emit index earn its place in this phase, or should those cases be explicitly deferred to #900?

[rosemarYuan]

Thanks for raising these questions. They help clarify several boundaries that were not explicit enough in the proposal.

Field placement. I would keep upstream_event_id and upstream_action_name on the Event. I consider them framework-managed, intrinsic metadata of an Event occurrence, similar to its id and type, rather than optional execution-tracing context. This keeps the Event–Action–Event relationship available in the standard Event Log even when the richer tracing in #900 is disabled.

#900 serves a different purpose: its ExecutionTraceContext describes a concrete execution, including run identity, invocation hierarchy, lifecycle, and status. Event lineage remains the source of truth for the Event’s immediate causal origin, while #900 provides finer-grained, execution-based relationships and context.

Python identity. I agree that the current content-derived Python Event ID creates both problems. I plan to align Python with Java by generating a new UUID for each Event occurrence.

The ID will remain unchanged after the Event is created, so attaching framework-managed lineage will not affect it. Events with identical payloads will also receive different IDs and remain separate nodes during reconstruction.

Ordering and replay. We considered adding emitIndex, but sibling Events share a causal parent rather than a causal relationship with one another. The minimal tree therefore does not guarantee their order. Preserving it across Python async execution and replay would add complexity without a clear current use case.

Similarly, distinguishing repeated invocations, retries, and reused results requires execution identity. Those semantics belong to #900. For partial logs, the reader can report a missing parent, but neither a run ID nor an ordering field can recover records that are no longer available.

Proposed boundary. The two-field proposal provides an independently usable Event–Action–Event causal tree. #900 provides the complete execution trace, including run and invocation identity, execution hierarchy, lifecycle, reuse, and failures.

Does this distinction address your concerns about field placement, Python identity, replay, and ordering?

[weiqingy]

Thanks, this addresses all four, and the boundary you draw between lineage and #900 reads right to me.

On field placement, correcting one thing in my earlier comment: the state path does not preserve the causal edge on its own. ActionStateUtil.generateKey builds the key from the key, the sequence number, a hash of event.getAttributes(), and the Action, so the trigger Event's id never enters it. On a replay the trigger is a fresh occurrence with a new id, so the persisted children would still point at the original occurrence rather than the replayed one. That does not change where the fields belong, but it is the re-stamp on the reuse path that keeps the edge intact, not the placement itself.

Python identity and ordering are both settled from my side. Per-occurrence UUIDs close both halves of the identity question, and deferring sibling ordering to #900 sounds good.

I have put the rest of my comments on the PR (#923) so this does not fork into two threads.