Reporting enhancements: forensic mode, StaleMARC, VERP, DB suppressions#324
Open
thegushi wants to merge 22 commits into
Open
Conversation
…steddomainproject#52) RFC 7489 requires a <version> element as the first child of <feedback> and an <envelope_from> element in <identifiers>. Both pieces of data were already available; they just weren't being emitted. envelope_from is omitted for null reverse-path messages (empty env_domain).
…domainproject#52) RFC 7489 requires <scope> in SPF auth_results and <fo> in policy_published. Neither was being captured or reported. - Add mctx_spfmode to dmarcf_msgctx and capture SPF origin (MAILFROM vs HELO) in all three SPF code paths; write spf_scope to the history file alongside the existing spf line - Call opendmarc_policy_fetch_fo() after the DMARC policy query and write fo bitmap to the history file - Add spf_scope TINYINT to messages table and fo TINYINT to requests table in schema.mysql - Update opendmarc-import.in to parse spf_scope and fo and store in DB; update the requests UPDATE to persist fo - Update opendmarc-reports.in to SELECT and emit <scope> (mfrom/helo) in SPF auth_results and convert the fo bitmap to a colon-separated RFC 7489 string for <fo> in policy_published - Add startup warnings with ALTER TABLE commands for existing installs missing the new columns
…ssue trusteddomainproject#230) Use LEFT JOIN for selectors in the signatures query so that signatures stored with selector_id=0 (from import of history files where the selector was empty) are still returned. Treat a NULL selector name as an empty string rather than dropping the row.
…sue trusteddomainproject#217) Initialize arc, arc_policy, align_dkim, align_spf to safe sentinel values in opendmarc-import so old history files without these fields can be imported cleanly. Also add DEFAULT clauses to schema.mysql for these columns so strict mode accepts the CREATE TABLE.
…omparison (issue trusteddomainproject#210) DATE(messages.date) >= DATE(FROM_UNIXTIME(?)) truncates to calendar day in the MySQL server's local timezone, causing messages near day boundaries to be included in the wrong report when the server timezone differs from the reporting timezone. Replace with direct timestamp comparisons, which are timezone-agnostic and consistent with the non-daybound query path.
…riaDB hang (issue trusteddomainproject#196) DBI passes untyped parameters as strings, which can trigger MDEV-27242 in MariaDB causing queries to hang when comparing integer columns to string values. Add bind_param with SQL_INTEGER for the two domain selection queries that were still using execute() with untyped args.
…main (issue trusteddomainproject#270) RFC 7489 appendix C requires policy_published.domain to be the domain at which the DMARC record was found. For subdomains inheriting policy from the organizational domain, opendmarc-reports was emitting the from domain (e.g. subdomain.example.com) rather than the domain where the record was actually located (e.g. example.com). The milter already captures the correct value via opendmarc_policy_fetch_utilized_domain() and stores it in messages.policy_domain. This change adds a query to look up that value per reporting window and use it in the XML output, falling back to the from domain if no rows are found.
…rusteddomainproject#269) Adds --smtp-username, --smtp-password, and --smtp-ssl options to opendmarc-reports for sites that relay through an SMTP server requiring authentication. Unauthenticated submission to a local MTA remains the default. opendmarc-run is updated with commented-out SMTPUSER, SMTPPASSWD, and SMTPSSL variables that are passed through when set.
…rusteddomainproject#269) Allows specifying a CA bundle for SMTP connections to servers using a private or self-signed certificate. Without this, Net::SMTP verifies against the system CA bundle, which fails for internal relay hosts.
…ddomainproject#202) When RequiredHeaders rejects a message, the reason was logged but the SMTP response was a generic 550 5.7.1. Call dmarcf_setreply() with the specific error string so the client sees e.g. "not exactly one Date field" rather than a Postfix default message.
…sue trusteddomainproject#25) Adds --report-bcc to opendmarc-reports, which adds the address as an SMTP envelope recipient and a Bcc: header. opendmarc-run gains a commented-out REPORTBCC variable passed through when set.
…s7.1) Adds support for rua= URIs with http:// or https:// schemes. Reports are submitted via HTTP POST with Content-Type application/gzip using LWP::UserAgent. The existing mailto: path is unchanged. Unsupported schemes are still logged and skipped.
…marc-reports - Add --forensic mode: reads a pre-formed AFRF message from stdin and injects it via SMTP, replacing sendmail -t as the milter ReportCommand - Add --stale-check/--stale-def-deny/--stale-server: query the StaleMARC RBL before sending to skip known-stale RUA/RUF addresses - Add --verp: encode recipient into MAIL FROM so bounces identify the failing address; forensic mode sends one transaction per recipient - Add suppressions table: local DB-backed skip list checked at startup; designed to be populated from VERP bounce processing - Remove Switch module dependency; replace all switch/case blocks with if/elsif chains (Switch is deprecated and not in Perl core since 5.14) - Fix opendmarc-import INSERT IGNORE: re-importing an already-seen history file is now silent and idempotent instead of producing duplicate errors - Update opendmarc-reports man page: add ENVIRONMENT and REPORTING SUPPRESSIONS sections, document all new options - Update db/README.schema: document suppressions table and add it to the schema upgrade instructions
…notes - Add --config file reading to opendmarc-reports; defaults to @sysconfdir@/opendmarc/opendmarc-reports.conf, loaded before GetOptions so CLI args always win. Applies to both aggregate and forensic modes so SMTP credentials are never exposed in opendmarc.conf ReportCommand. - Add contrib/opendmarc-reports.conf.sample documenting all options - Add SUFFIX variable to opendmarc-run for alternate backend variants (e.g. SUFFIX=-pg for PostgreSQL scripts) - Source /etc/opendmarc/opendmarc-run.conf in opendmarc-run if present, surviving package upgrades - Add contrib/opendmarc-run.conf.sample - Replace LAST_INSERT_ID() SQL calls with DBI->last_insert_id() for portability across database backends - Add comments flagging all MySQL-specific SQL constructs with their SQLite and PostgreSQL equivalents to guide future variant scripts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch builds on
fix/issue-52-rfc-complianceand adds a set ofreporting enhancements to
opendmarc-reports.Changes
Forensic (RUF) report delivery (
--forensic)Adds a
--forensicmode that reads a pre-formed AFRF message from stdinand injects it via SMTP. This makes
opendmarc-reportsa drop-inreplacement for
sendmail -tas the milter'sReportCommand, using thesame SMTP configuration as aggregate reports.
StaleMARC RBL integration (
--stale-check)Before sending each report, optionally query the StaleMARC RBL
(stalemarc.measurement.network) to skip addresses known to have delivery
problems. Supports a default-deny mode for stricter filtering. Fails
open on lookup errors.
VERP envelope senders (
--verp)Encodes the recipient address into the SMTP MAIL FROM so that bounces
identify the specific failing address. In forensic mode, VERP causes
one SMTP transaction per recipient.
Database suppression table
Adds a
suppressionstable checked at startup. Entries suppressreporting to specific addresses or domains. Designed to be populated
from VERP bounce processing, closing the loop between delivery failures
and future sending decisions.
Switch module removal
Replaces all
switch/caseblocks withif/elsifchains.Switchis deprecated and has not been in Perl core since 5.14.
opendmarc-import idempotency
Changes the messages
INSERTtoINSERT IGNOREand checksrows()to detect skipped duplicates. Re-importing an already-seen history file
is now silent rather than producing duplicate key errors.
Documentation
Updates the
opendmarc-reportsman page with a newENVIRONMENTsectionand a
REPORTING SUPPRESSIONSsection. Adds thesuppressionstableand upgrade instructions to
db/README.schema.