Repository navigation
fix(sink): set the producer batching delay once - #166
Merged
Merged
Conversation
Vector writes batch.timeout_secs as queue.buffering.max.ms. The production profile writes linger.ms. librdkafka reads both as one setting. rdkafka applies a client's options in hash order. Over 30 starts of Vector 0.61.0 on the generated sink, 13 gave 100 ms and 17 gave 1000 ms. Vector writes queue.buffering.max.ms whenever a batch block is present. The generator moves a delay named by any librdkafka layer into batch.timeout_secs and drops both spellings from librdkafka_options. The highest layer that names the delay wins. With this change, all 30 starts gave 100 ms. On the bus transport, sink.batch.timeout_secs is ignored when a layer sets the delay. The supervisor warns when sink.batch.timeout_secs is not 1. Fix four comments and messages that did not match the code: - the filebeat pipeline header, which called the file verbatim and byte-identical - the Helm key in the disk buffer warning - the metric name in the scrape failure comment - the count of other Rust apps in the deployment comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vector writes batch.timeout_secs as queue.buffering.max.ms. The production profile writes linger.ms. librdkafka reads both as one setting. rdkafka applies a client's options in hash order.
Over 30 starts of Vector 0.61.0 on the generated sink, 13 gave 100 ms and 17 gave 1000 ms.
Vector writes queue.buffering.max.ms whenever a batch block is present. The generator moves a delay named by any librdkafka layer into batch.timeout_secs and drops both spellings from librdkafka_options. The highest layer that names the delay wins. With this change, all 30 starts gave 100 ms.
On the bus transport, sink.batch.timeout_secs is ignored when a layer sets the delay. The supervisor warns when sink.batch.timeout_secs is not 1.
Fix four comments and messages that did not match the code: