Skip to content

feat: add metrics to BroadcastExec - #895

Open
raghavyadav01 wants to merge 1 commit into
datafusion-contrib:mainfrom
raghavyadav01:feat/broadcast-exec-metrics
Open

raghavyadav01 wants to merge 1 commit into
datafusion-contrib:mainfrom
raghavyadav01:feat/broadcast-exec-metrics

Conversation

@raghavyadav01

@raghavyadav01 raghavyadav01 commented Nov 16, 2026 •

Copy link
Copy Markdown

Closes #848.

BroadcastExec had no metrics, so in EXPLAIN ANALYZE and the collected task metrics it showed up as metrics=[].

It now reports:

  • output_rows, output_bytes, output_batches for each output partition. A batch read by 3 consumer tasks is counted 3 times, since that's what the node outputs.
  • elapsed_compute: time spent buffering input batches (not tied to an output partition, since each input partition is buffered once for all consumers) plus time spent by each consumer reading from the queue.
  • max_mem_used: peak bytes buffered across input partitions while waiting for slower consumers. It goes up when a batch is pushed and down when every reader has read it and it's evicted, using the same sizes as the existing memory reservations.

Tests: test_metrics_collection_in_broadcast_join checks a broadcast join over the metrics collection path (output rows and bytes are exactly 3x the build side, and the peak is below what the consumers read), and broadcast_exec_tracks_buffered_memory checks the buffered bytes go back to 0 once both consumers finish.

Record output rows, bytes and batches for each output partition, time
spent pushing and reading batches as elapsed_compute, and the peak bytes
buffered while waiting for slower consumers as max_mem_used.

Signed-off-by: Raghav Yadav <raghavmnnit@gmail.com>
@raghavyadav01
raghavyadav01 force-pushed the feat/broadcast-exec-metrics branch 2 times, most recently from cd5f547 to e1d5526 Compare November 18, 2026 23:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add basic metrics to BroadcastExec

2 participants