fix: delete Prometheus series for removed streams

[bookcherry]

Summary

• implement optional metric deletion for Prometheus counters and gauges
• remove summary and histogram series with DeleteLabelValues
• ensure deleted streams no longer leave labelled Prometheus series behind

Dependency

Depends on redpanda-data/benthos#474, which adds stream-scoped metric lifecycle cleanup.

Manual integration verification

This Connect branch and the Benthos PR branch were combined in a temporary Go workspace so the running process used both changes. Redpanda Connect was started in streams mode with file watching enabled, the HTTP API listening on port 4195, and the Prometheus exporter enabled.

The observability configuration was:

http:
  enabled: true
  address: 127.0.0.1:4195

metrics:
  prometheus: {}

logger:
  level: INFO

The watched stream continuously generated messages and emitted a custom counter through a metric processor.

Stream deletion

Before deleting stream trigger-g, the Prometheus endpoint exposed 22 series labelled with that stream ID:

curl -s localhost:4195/metrics | grep -c "stream=\"trigger-g\""
# 22

After deleting the stream configuration, the watcher logged successful removal:

Stream trigger-g config deleted, attempting to remove stream.
Removed stream trigger-g.

The metrics endpoint was queried again after removal completed:

curl -s localhost:4195/metrics | grep -c "stream=\"trigger-g\""
# 0

The test was repeated with verify-a, verify-b, and verify-c. Each stream exposed 22 series while active. After all three configurations were deleted, no series remained for any of those IDs.

Stream update

Stream foo initially emitted stream_old_metric. Its configuration was changed in place to emit stream_new_metric.

After the watcher applied the update:

stream_old_metric series: 0
stream_new_metric series: 1

This verifies both that the old Prometheus series is removed and that cleanup does not remove the replacement stream series.

Same-ID recreation

Stream foo was deleted, and the endpoint was checked until the number of series labelled stream="foo" reached 0. The same stream ID was then recreated with a different metric name.

The new series appeared normally. After deleting the recreated stream, the series count returned to 0.

Pre-fix comparison

The identical procedure was run against Benthos 70c0dfc70 and Connect f5203c9a5, before these changes. Deleting four streams left 22 stale series per stream, or 88 stale series in total. With the patched branches, the stale-series count was 0.

Automated verification

Prometheus exporter tests:

go test -race ./internal/impl/prometheus

The same test suite was also run while resolving the Benthos dependency to the companion PR branch.

The tests cover deletion of:

• counters
• gauges
• summaries
• histograms

Related issue: redpanda-data/benthos#275.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}