Skip to content

s3-dump: expose per-target flush results so consumers can label metrics by target #80

Description

@jaronoff97

Problem

S3Dump.flush() returns a single aggregate FlushResult — it sums every sealed batch across all configured targets into one struct. Consumers that record flush metrics (uploaded/failed objects, records, bytes, backlog) therefore cannot break them down by target.

We want a target label on the s3-dump flush metrics in edge, but there's no truthful way to produce it from the current API:

  • flush() aggregates across targets; there's no per-target flush call.
  • The per-target upload happens inside uploadOne(... snapshots[sealed_batch.target] ...), so the consumer can't intercept it to attribute a target name.

Suggested change

flush() already iterates sealed batches that each carry a .target index (src/extensions/s3_dump.zig, the upload loop) — it just sums them. Bucket the counters by target instead and return per-target results, e.g.:

pub const TargetFlushResult = struct {
    name: []const u8,
    objects_uploaded: u32 = 0,
    objects_failed: u32 = 0,
    objects_requeued: u32 = 0,
    records_uploaded: u64 = 0,
    records_failed: u64 = 0,
    records_dropped: u64 = 0,
    bytes_uploaded: u64 = 0,
    bytes_failed: u64 = 0,
    backlog_bytes: usize = 0,
};

Add targets: []const TargetFlushResult to FlushResult (keep the existing aggregate fields for backward compatibility). The slice can be allocated in the flush arena; consumers copy what they need before it's freed, or it's owned like the rest of the result.

records_dropped is currently a single S3Dump-wide atomic drained per flush — deciding whether/how to attribute drops per target (delivery-time drops know their slot; backlog-cap drops may not) is part of this issue.

Consumer context

edge wires s3-dump for Datadog logs and records these aggregate metrics today:
edge_s3_dump_{flushes,objects_uploaded,objects_failed,records_uploaded,records_dropped,bytes_uploaded}_total + edge_s3_dump_backlog_bytes. Once per-target results land we'll add a target label sourced from TargetFlushResult.name.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions