[talwgx]

Every event the Fluent Bit outputs write now reaches Fluent Bit as a valid record and is sent, including events with no timestamp and events larger than 16 KB. Two changes, both in this repository:

• Events with no timestamp. The Fluent Bit outputs (pipelines/run/output/event/fluentbit/*.yaml) and the process output example write each event as [ <time> , {"text":...} ], the form Fluent Bit's stdin input reads. The time came from dateFormat("+%s.%N"), which writes nothing for an event with no timestamp, so the record was [ , {...} ]. That is not JSON, and Fluent Bit's stdin input discards it together with every other record in the same read. The time field is now timestamped ? dateFormat("+%s.%N") : now() / 1000: an event with no timestamp carries the second it is written, as Fluent Bit itself times a record that has none with the time it reads it. Events with a timestamp are written as before, byte for byte.
• Records larger than 16 KB. fluentbit-10x-input.conf now sets Buffer_Size 16M on the stdin input. With the default 16k, a record that does not fit fills the buffer, the next read asks for 0 bytes, Fluent Bit 5.1.5 takes the 0 as the end of its input and shuts down, and that record and every record after it are not sent. A multi-line event (a stack trace, a group of JSON lines) is often larger than 16 KB.

The config test workflow also runs when these files change.

Checked

Fluent Bit 5.1.5 in a container, the engine 1.1.153 on the JVM as the Receiver with stdin input, Fluent Bit's Splunk output over TLS to a stand-in HEC that stores every request. The Splunk file used is splunk.yaml with the fix of #120 (the shipped file stops Fluent Bit at startup), with and without this change; the bytes the engine wrote to Fluent Bit's stdin were copied to a file.

	records written	with no time	delivered to the HEC
OpenTelemetry sample, 5,000 lines (3,723 events), shipped files	470 (then Fluent Bit stopped reading and the engine stopped writing)	282	0
same, 16M buffer only	3,723	637	2,986
same, time fallback only (an engine build that applies it), 16k buffer	470, all valid	0	68: Fluent Bit stopped at record 69, the first over 16 KB (71,777 bytes the largest)
same, this change	3,723, all valid	0	3,723; the 5,000 input lines come back at the HEC
Windows Security, 662 events	662, byte-identical to the shipped fields'	0	662
CloudTrail, 8,100 events	8,100, byte-identical to the shipped fields'	0	8,100

On the OpenTelemetry sample, 3,083 of the 3,086 records with a timestamp are byte-identical to the shipped fields'. The other 3 carry only a time of day (20:12:46.393), which the engine dates by the clock of the run; the two runs were on either side of 20:12 UTC, so those 3 differ by exactly one day, with or without this change.

node --test tests/: 30 of 30. The new tests/fluentbit-record.test.mjs evaluates each output's fields with stubs of the engine functions and parses the record as Fluent Bit reads it; on main 8 of its 15 tests fail.

Not changed

• Events with a timestamp: their time is written as before. That time is +%s.%N of the engine's epoch, which is <seconds>.0 for a millisecond timestamp (the milliseconds are not sent) and has no leading zeros in the fraction for a nanosecond timestamp (41.022663733 is sent as 41.22663733). That is in the engine's formatter and is not changed here.
• Records still follow each other with no newline between them; Fluent Bit's stdin input reads them that way.

🤖 Generated with Claude Code