feat(logs): add syslog format detection, ISO timestamp parser, and observedTimestamp fallback#1807
Merged
Merged
Conversation
…servedTimestamp fallback - Add router route for non-standard RFC 3164 with ISO 8601 timestamps (VMware ESXi/vSAN) - Add regex parser to extract timestamp, hostname, priority from ISO format messages - Add attributes.log.syslog.format label (rfc5424, rfc3164, rfc3164_iso8601, unknown) - Add transform/syslog_observed_timestamp_fallback to use observedTimestamp when no timestamp could be parsed (fixes 1970 epoch timestamps for unknown format logs) - Add comments explaining the different syslog format routes
timojohlo
force-pushed
the
feat/syslog-iso-parser-and-format-attribute
branch
from
July 10, 2026 10:59
b9f0ed0 to
d9d42d3
Compare
timojohlo
marked this pull request as ready for review
July 10, 2026 11:13
Contributor
There was a problem hiding this comment.
Pull request overview
Adds enhanced syslog handling in the logs chart to correctly classify and timestamp non-standard RFC3164-like syslog messages that use ISO 8601 timestamps (notably VMware ESXi/vSAN), and introduces an observed-timestamp fallback to avoid epoch timestamps when parsing fails.
Changes:
- Added syslog routing + parsing for ISO 8601 “RFC3164-like” messages and set
attributes.log.syslog.formatfor TCP/TLS syslog ingestion. - Added
transform/syslog_observed_timestamp_fallbackand wired it into syslog pipelines to avoid@timestamp = 1970-01-01T00:00:00Zwhen parsing fails. - Bumped chart and plugin definition versions.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
logs/charts/templates/_syslog-config.tpl |
Adds syslog format routing, ISO regex parsing, and format labeling; wires observed-timestamp fallback into syslog pipelines. |
logs/charts/templates/_syslog-audit-filter-config.tpl |
Defines the new observed-timestamp fallback transform processor. |
logs/charts/Chart.yaml |
Helm chart version bump to 0.4.13. |
logs/plugindefinition.yaml |
Plugin/chart version reference bump to 0.15.13 / 0.4.13. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
… to UDP receiver - Change timestamp layout from 'Z' to 'Z07:00' to parse both UTC (Z) and offset (+HH:MM) timestamps in the ISO 8601 regex parser - Add attributes.log.syslog.format=rfc3164 to the UDP syslog receiver for consistent format labeling across all syslog paths Addresses Copilot review feedback on PR #1807.
joluc
approved these changes
Jul 15, 2026
Signed-off-by: Timo <32620814+timojohlo@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
priority,timestamp,hostname, andmessagefrom these ISO-format messages, fixing logs that previously ended up with@timestamp = 1970-01-01T00:00:00Zattributes.log.syslog.formatlabel to all syslog logs with values:rfc5424,rfc3164,rfc3164_iso8601, orunknowntransform/syslog_observed_timestamp_fallbackprocessor that usesobservedTimestampas fallback when no timestamp could be parsed from the log body (unknown format / garbage messages)Problem
Some syslog senders emit messages in a non-standard format:
This doesn't match RFC 5424 (missing version number) or RFC 3164 (ISO timestamp instead of BSD
Mmm dd HH:MM:SS), so these messages fell through to the default route with no timestamp parsing, resulting in@timestamp = 1970-01-01T00:00:00Z.Additionally, some messages arrive with no syslog header at all (e.g. continuation lines from multi-line messages), which also end up at epoch time.
Changes
_syslog-config.tpl_syslog-audit-filter-config.tpltransform/syslog_observed_timestamp_fallbackprocessorChart.yaml0.4.12→0.4.13plugindefinition.yaml0.15.12→0.15.13/ chart ref0.4.12→0.4.13Testing
Deployed and validated on a QA cluster. Both external-collector pods started cleanly with 0 errors and 0 restarts. Confirmed:
log.syslog.formatattribute is populated on all syslog logsobservedTimestampinstead of epoch