Skip to content

FEATURE_ANALYSIS.md

Latest commit

 

History

History
549 lines (483 loc) · 24.7 KB

FEATURE_ANALYSIS.md

File metadata and controls

549 lines (483 loc) · 24.7 KB

GoMail Feature Implementation Analysis

Last Updated: April 21, 2026
Analysis Scope: Comprehensive code review of all major GoMail features

1. SMTP Protocol Features

EHLO/HELO Support

✅ YES - Fully implemented

  • Location: smtp/session.go handleEHLO() (line 125), handleHELO() (line 178)
  • Details: Advertises capabilities including SIZE, 8BITMIME, ENHANCEDSTATUSCODES, PIPELINING, DSN, SMTPUTF8, CHUNKING, and STARTTLS (if TLS available)
  • Status: Production ready

8BITMIME Extension

✅ YES - Fully implemented

  • Location: smtp/session.go
  • Details: Advertised in EHLO response and supported by parser
  • Status: Production ready

SMTPUTF8 Support (RFC 6531)

✅ YES - Fully implemented

  • Location: smtp/session.go (advertised), smtp/session.go handleMAIL() (parameter detection)
  • Details: UTF8 parameter checked in MAIL FROM command, stored in session
  • Status: Production ready

CHUNKING/BDAT (RFC 3030)

✅ YES - Fully implemented

  • Location: smtp/session.go handleBDAT() function
  • Details: Full BDAT command implementation with chunk size validation and LAST qualifier handling
  • Size enforcement: SIZE limits enforced per chunk (line 370)
  • Status: Production ready

PIPELINING

✅ YES - Advertised, partial buffering

  • Location: smtp/session.go (advertised in EHLO)
  • Details: Advertised to clients; command parsing loop handles sequential commands
  • Limitation: Commands processed sequentially (line 108-130), not truly buffered for pipelining
  • Status: Advertised but clients should not rely on full pipelining support

DSN Extension (RFC 3461)

✅ YES - Fully implemented

  • Location: smtp/session.go (advertised), smtp/session.go MAIL handling
  • Details:
    • MAIL FROM: Parses DSN RET, ENVID parameters → parseDSNMailParams()
    • RCPT TO: Parses NOTIFY, ORCPT parameters → parseDSNRcptParams()
    • Delivery reports generated on failure (see Delivery/Queue section)
  • Status: Production ready

SIZE Extension (RFC 1870)

✅ YES - Fully implemented with enforcement

  • Location: smtp/session.go (advertised), smtp/session.go and smtp/session.go (enforced)
  • Details:
    • Advertised limit in EHLO response based on config (maxSize)
    • DATA command enforces size limit during message reception
    • BDAT enforces size limit per chunk
  • Error: 552 "Message exceeds maximum size"
  • Status: Production ready

STARTTLS (RFC 3207)

✅ YES - Fully implemented for inbound and outbound

  • Inbound: smtp/session.go handleSTARTTLS()
    • Advertised in EHLO if TLS config provided
    • TLS handshake performed, connection upgraded
  • Outbound: smtp/outbound.go in deliverToHost()
    • STARTTLS attempted if supported by server
    • Can be made mandatory via MTA-STS policy
  • Status: Production ready

2. Authentication

DKIM Signing (RFC 6376)

✅ YES - Fully implemented

  • Location: auth/dkim.go (signing), delivery/queue.go (applied)
  • Key Features:
    • Per-domain DKIM signers loaded from database
    • Algorithms supported: RSA-2048, Ed25519
    • Header canonicalization: relaxed
    • Public key generation and storage in DB
    • Body hash computed per RFC 6376 §3.7
  • Signing location: delivery/queue.go calls signer.Sign()
  • Database: DKIM keys stored in domains table (dkim_selector, dkim_algorithm, dkim_private_key, dkim_public_key)
  • Status: Production ready

DKIM Verification (RFC 6376)

✅ YES - Fully implemented

  • Location: auth/dkim.go VerifyDKIM() (line 230), auth/dkim_lookup.go LookupDKIMPublicKey()
  • Key Features:
    • DNS lookup for selector._domainkey.domain TXT records
    • Signature verification with RSA-2048 and Ed25519
    • Body hash verification
    • Header list validation
    • Proper error handling (temperror vs permerror)
  • Called from: smtp/inbound.go during message processing
  • Status: Production ready

SPF Verification (RFC 7208)

✅ YES - Fully RF-compliant implementation

  • Location: auth/spf.go CheckSPF() (line 26)
  • Mechanisms Supported:
    • ✅ all - Universal match (line 182)
    • ✅ a - A record lookup with optional CIDR (line 187)
    • ✅ mx - MX record lookup with optional CIDR (line 191)
    • ✅ ip4/ip6 - IP4 and IP6 CIDR blocks (line 196, 199)
    • ✅ ptr - Reverse DNS validation (deprecated but implemented, line 221)
    • ✅ exists - Macro expansion DNS query (line 201)
    • ✅ include - Recursive include (RFC 7208 §5.2, line 169)
    • ✅ redirect - Redirect mechanism (line 167-169)
  • Modifiers Supported:
    • ✅ exp= - Explanation text fetching (line 192 in results)
    • ✅ Macro expansion (RFC 7208 §7)
  • RFC Compliance:
    • DNS lookup limit: 10 (enforced line 79)
    • Void lookup limit: 2 (RFC 7208 §4.6.4)
    • Depth tracking for includes/redirects
    • NXDOMAIN handling per RFC
  • Status: Production ready

DMARC Verification (RFC 7489)

✅ YES - Fully compliant implementation

  • Location: auth/dmarc.go CheckDMARC()
  • Policy Evaluation:
    • ✅ Policy lookup (lookupDMARC())
    • ✅ Alignment checking (relaxed and strict modes)
    • ✅ SPF alignment validation
    • ✅ DKIM alignment validation
    • ✅ Subdomain policies (sp= tag)
  • Report Addresses:
    • ✅ rua= (aggregate report recipients)
    • ✅ ruf= (forensic report recipients)
  • Policy Sampling:
    • ✅ pct= percentage enforcement (line 50-60)
    • ✅ Random sampling to limit policy application
  • Dispositions: none, quarantine, reject
  • Called from: smtp/inbound.go in message processing
  • Status: Production ready

ARC Signing (RFC 8617)

✅ YES - Fully implemented

  • Location: auth/arc.go
  • Components Implemented:
    • ✅ ARCAuthenticationResults() - Builds authentication results header
    • ✅ ARCMessageSignature() - Signs message with ARC-Message-Signature header
    • ✅ ARCSeal() - Creates ARC-Seal header
  • Key Features:
    • Instance tracking (i= tag)
    • Algorithms: RSA-sha256, Ed25519-sha256
    • Canonicalization: relaxed/relaxed
    • Base64-encoded signatures
    • Timestamp handling (t= tag)
  • Applied in: delivery/worker.go via addARCHeaders()
  • Status: Production ready

ARC Verification (RFC 8617)

✅ YES - Fully implemented

  • Location: auth/arc.go
  • Functions:
    • ✅ ValidateArcChainAtInstance() - Validates ARC chain (line 603)
    • ✅ verifyARCMessageSignature() - Verifies message signature (line 362)
    • ✅ verifyARCSeal() - Verifies seal signature (line 496)
  • Key Features:
    • Chain validation up to specific instance
    • Result status: pass/fail
    • Cryptographic verification with RSA and Ed25519
  • Called from: smtp/inbound.go message processing
  • Status: Production ready

3. Delivery/Queue

Queue Persistence

✅ YES - Fully implemented with SQLite

  • Location: store/messages.go (line 497+) and store/schema.sql
  • Table: outbound_queue
  • Operations:
    • ✅ EnqueueMessage() - Add to queue (line 505)
    • ✅ GetPendingQueue() - Retrieve ready entries (line 518)
    • ✅ UpdateQueueEntry() - Update status/retry info (line 547)
  • Status Fields: pending, sending, sent, failed
  • DSN Tracking: DSN flags stored per entry (DSN_notify, DSN_ret, DSN_envid, DSN_sent)
  • Status: Production ready

Retry with Backoff (RFC 5321)

✅ YES - Exponential backoff implementation

  • Location: delivery/retry.go RetrySchedule.NextRetry()
  • Algorithm:
    • Uses configured retry intervals (from config)
    • After exhausting intervals, applies exponential backoff: lastInterval × 2^(attempt - configLength)
    • Maximum retry time capped at 48 hours
    • Respects SMTP 4xx temporary error codes
  • Configuration: delivery.retry_intervals in config
  • Called from: delivery/worker.go when temporary failure detected
  • Status: Production ready

Worker Pool (Concurrent Delivery)

✅ YES - Configurable worker pool

  • Location: delivery/worker.go Pool struct
  • Features:
    • ✅ Configurable number of workers (config: delivery.queue_workers)
    • ✅ Mutex-protected claim mechanism to prevent duplicate processing
    • ✅ Per-worker retry schedule
    • ✅ Polling-based processing every 10 seconds
  • Pool:Start() launches workers (delivery.queue_workers count)
  • Status: Production ready

Stale Entry Recovery

✅ YES - Automatic recovery implemented

  • Location: store/messages.go RecoverStaleQueueEntries() (line 560)
  • Logic: Finds entries stuck in "sending" status for > 30 minutes, resets to "pending"
  • Called from: delivery/worker.go Pool.Start() (line 39) on startup
  • SQL: Queries for entries where status='sending' AND age > 30 minutes
  • Status: Production ready

DSN Generation (RFC 3464)

✅ YES - DSN bounce notification system

  • Location: delivery/worker.go sendDSNReport() (line 462)
  • Functionality:
    • ✅ Only sends DSN if NOTIFY parameter was set in MAIL FROM
    • ✅ Filters for FAILURE notifications
    • ✅ Constructs RFC 3464 multipart DSN message
    • ✅ Extracts SMTP code from error message
    • ✅ Records DSN sent status
  • Message Disposition:
    • Permanent failure (5xx) → Mark failed, send DSN immediately
    • Temporary failure + max attempts reached → Send DSN
    • Temporary failure with retries available → Schedule retry
  • Related Code: reporting/dsn.go for DSN report generation
  • Status: Production ready

4. Outbound Security

MTA-STS Policy Fetching (RFC 8461)

✅ YES - Fully compliant implementation

  • Location: mta_sts/fetcher.go FetchPolicy()
  • Process:
    • DNS TXT lookup: _mta-sts.<domain> for policy ID
    • HTTPS fetch: https://mta-sts.<domain>/.well-known/mta-sts.txt
    • Caching with expiration (respects max_age)
    • Policy format parsing (mode, MX hosts, etc.)
  • Policy Modes: enforce, testing, none
  • Caching: policyCache with TTL based on max_age from policy
  • Status: Production ready

MTA-STS Enforcement (RFC 8461)

✅ YES - Policy enforcement in outbound delivery

  • Location: smtp/outbound.go SendMail() (line 54-85)
  • Enforcement Logic:
    • ✅ Fetches policy for recipient domain
    • ✅ In enforce mode: validates MX host in policy, fails if not listed, requires TLS
    • ✅ In testing mode: logs violations but allows delivery
    • ✅ In none mode: no enforcement
    • ✅ Skips hosts not in policy during enforce mode
  • Failure Propagation: TLS failures recorded for TLS-RPT
  • Status: Production ready

TLS Enforcement per Domain (RFC 8689)

✅ YES - RequireTLS flag per domain

  • Location: store/models.go Domain.RequireTLS field
  • Database Column: domains.require_tls
  • Used in: delivery/worker.go
    • Retrieves domain config
    • Passes requireTLS to smtp.SendMail()
  • Effect: If true, delivery fails if TLS cannot be established
  • Status: Production ready

DANE Verification (RFC 6698, 6125)

❌ NO - Not implemented

  • Location: tls/dane.go contains GenerateTLSARecord() and GenerateTLSARecordFromDER() for record generation only
  • Missing: No TLSA record lookup, verification, or enforcement in smtp/outbound.go
  • Current Status: Operators can generate TLSA records but verification is not performed
  • Workaround: TLS verification relies on certificate chain validation only
  • Note: This is a known limitation for enhanced security but basic opportunistic TLS still works

5. Inbound Security

PTR Verification (Forward-Confirmed Reverse DNS)

✅ YES - Fully implemented

  • Location: dns/ptr.go VerifyPTR()
  • Process:
    • Reverse DNS lookup (net.LookupAddr())
    • Forward-confirm: resolve hostname back to same IP
    • Returns: (hostname, isValid, error)
  • Called from: smtp/inbound.go in handleConnection()
  • Caching: DNS results cached with TTL
  • Logged: Results logged in smtp/inbound.go
  • Usage: Stored in session, available in message processing context
  • Status: Production ready

Greylisting (RFC 6647)

✅ YES - Per-domain configurable greylisting with smart cleanup

  • Location: store/schema.sql (greylisting table), store/messages.go (CheckGreylist(), CleanupGreylisting()), smtp/session.go (handleRCPT integration), web/handlers/admin.go, templates/admin_domain_edit.html
  • Features:
    • ✅ Per-domain enable/disable toggle (default: OFF)
    • ✅ Configurable delay per domain (5-480 minutes, default 15)
    • ✅ Sender triplet tracking: (remote_IP, sender_email, recipient_email, recipient_domain)
    • ✅ Three-state lifecycle: NEW (reject 450) → DELAYING (reject 421) → WHITELISTED (accept 250)
    • ✅ Admin panel UI with checkbox and delay input
  • Database:
    • Table: greylisting with fields: id, recipient_domain, remote_ip, sender_email, recipient_email, first_seen, whitelisted_at, rejected_count
    • Indexes on domain, triplet, and first_seen for fast lookups
    • UNIQUE constraint on (recipient_domain, remote_ip, sender_email, recipient_email)
  • SMTP Integration: Validated in handleRCPT() after account verification
    • New triplets: Send 450 "Mailbox temporarily unavailable", record entry
    • Delaying triplets: Send 421 "Service temporarily unavailable" until delay expires
    • Whitelisted triplets: Continue with normal acceptance (250)
  • Cleanup Strategy: Daily cleanup job removes rejected entries older than 30 days (where whitelisted_at IS NULL). Whitelisted senders remain in database permanently, preventing re-rejection after inactivity.
  • Logging: Consistent [smtp] greylisting: prefix for monitoring
  • Status: Production ready

Rate Limiting by IP

✅ YES - Per-IP rate limiting on connections and messages

  • Location: smtp/ratelimit.go RateLimiter struct
  • Limits:
    • ✅ Connections per minute (configurable)
    • ✅ Messages per minute (configurable)
  • Implementation:
    • Sliding window with 1-minute buckets
    • Timestamp tracking per IP
    • Automatic cleanup of stale entries every 5 minutes
  • Functions:
    • AllowConnection() - Gate new connections
    • AllowMessage() - Gate new messages (called in smtp/inbound.go for BDAT/DATA)
  • Configuration: smtp.rate_limit.connections_per_minute, smtp.rate_limit.messages_per_minute
  • Status: Production ready

Max Connections

✅ YES - Semaphore-based connection limit

  • Location: smtp/inbound.go connSemaphore channel
  • Implementation:
    • Semaphore with capacity cfg.SMTP.MaxConnections
    • Each connection acquires slot on accept, releases on close
    • Excess connections rejected
  • Configuration: smtp.max_connections in config
  • Logging: Rate limit rejections logged (line 89)
  • Status: Production ready

Tarpitting

✅ YES - Per-domain configurable progressive delays with exponential backoff

  • Location: store/schema.sql, store/messages.go, smtp/session.go, web/handlers/admin.go, templates/admin_tarpitting.html
  • Features:
    • ✅ Per-domain enable/disable toggle (default: OFF)
    • ✅ Configurable max delay per domain (1-30 seconds, default 8)
    • ✅ Remote IP tracking with failure count
    • ✅ Exponential backoff delays: 0s, 1s, 2s, 4s, 8s, 16s, ..., up to configured max
    • ✅ One-hour timeout resets failure counter
    • ✅ Whitelist functionality to bypass delays for legitimate sources
    • ✅ Admin panel UI with dashboard and per-domain control
  • Database:
    • Table: tarpitting with fields: id, recipient_domain, remote_ip, failure_count, last_invalid_command, first_failure, last_failure, whitelisted_at, notes
    • Indexes on domain, remote_ip, and first_failure for fast lookups
    • UNIQUE constraint on (recipient_domain, remote_ip)
  • SMTP Integration: Applied on invalid SMTP commands
    • Calculates delay using calculateTarpittingDelay(failureCount, maxDelay)
    • Applies delay via time.Sleep()
    • Increments failure counter on invalid command
  • Delay Calculation:
    • 1st failure: 0s (free pass)
    • 2nd failure: min(1s, maxDelay)
    • 3rd failure: min(2s, maxDelay)
    • 4th failure: min(4s, maxDelay)
    • 5th+ failures: Continues exponential backoff capped at maxDelay
    • Example: 30s max with 11 failures = min(512s, 30s) = 30s delay
  • Admin Control: Checkbox to enable/disable + number input (1-30 seconds) for max delay configuration per domain
  • Logging: Consistent [smtp] tarpitting: prefix for monitoring
  • Status: Production ready

6. Web UI & Compose

Attachment Upload

✅ YES - Multipart form with 25MB limit

  • Location: web/handlers/compose.go Send() method (line 68+)
  • Features:
    • ✅ r.ParseMultipartForm(25 * 1024 * 1024) - 25MB limit
    • ✅ File attachment handling via form value
    • ✅ Attachment save to disk via parser.SaveAttachments()
    • ✅ Database records created in attachments table
  • Storage Path: cfg.Store.AttachmentsPath (typically data/attachments/)
  • Validation: Files stored and linked to message ID
  • Status: Production ready

Attachment Download

✅ YES - Secure file serving with MIME type handling

  • Location: web/handlers/message.go DownloadAttachment() (line 313)
  • Features:
    • ✅ Attachment ID lookup in database
    • ✅ Path traversal protection (uses filename from DB)
    • ✅ Proper Content-Disposition header
    • ✅ MIME type handling
    • ✅ File existence verification
  • Response Headers: Content-Disposition: attachment; filename="..."
  • Status: Production ready

Plain Text Compose

✅ YES - Plain text textarea for message body

  • Location: templates/compose.html line 52
  • Features:
    • ✅ <textarea> element for plain text body
    • ✅ Optional pre-fill from reply context
    • ✅ Form value: name="body"
  • Status: Production ready

Reply/Forward

✅ YES - Both reply and forward implemented

  • Reply: web/handlers/compose.go
    • URL parameter: ?reply=true&to=...&subject=
    • Prefill logic in template
  • Forward: web/handlers/forward.go
    • Dedicated handler for forwarding messages
    • ForwardPage() for form display
    • Original message included in forwarded message
    • Subject prefixed with "Fwd:"
  • Status: Production ready

7. Reporting

DMARC Report Generation (RFC 7489)

✅ YES - RFC-compliant XML report generation

  • Location: reporting/dmarc.go GenerateDMARCAggregateReport()
  • Features:
    • ✅ XML structure per RFC 7489 Appendix A
    • ✅ Report metadata (org, email, date range)
    • ✅ Policy published section
    • ✅ Per-record authentication results (SPF, DKIM)
    • ✅ Policy evaluation (pass/fail/quarantine/reject)
    • ✅ Source IP grouping
  • Feedback Tracking: store/dmarc_feedback.go stores feedback records
  • Status: Production ready

DMARC Report Sending

✅ YES - Weekly scheduled report distribution

  • Location: reporting/scheduler.go
  • Features:
    • ✅ Weekly scheduler (Sunday 00:00 UTC)
    • ✅ Configurable enable/disable: config.dmarc.send_reports
    • ✅ Manual trigger: SendReportsNow() function
    • ✅ Reports sent via email to rua= addresses
    • ✅ GZIP compression before sending
    • ✅ Base64 encoding
  • Configuration: dmarc.send_reports boolean in config.json
  • Status: Production ready

TLS-RPT Report Generation (RFC 8460)

✅ YES - RFC-compliant JSON report generation

  • Location: reporting/tlsrpt.go GenerateTLSReport()
  • Features:
    • ✅ JSON structure per RFC 8460
    • ✅ Date range specification
    • ✅ Failure categorization by type
    • ✅ MTA and IP tracking (if available)
    • ✅ Aggregated failure counts
  • Failure Types Tracked: TLS negotiation failures, certificate issues, etc.
  • Data Source: store/tls_failures.go tracks TLS failures
  • Status: Production ready

TLS-RPT Report Sending

✅ YES - Weekly scheduled report distribution

  • Location: reporting/scheduler.go sendTLSRPTReportForDomain()
  • Features:
    • ✅ Weekly scheduler (integrated with DMARC reports)
    • ✅ DNS lookup for TLS-RPT addresses: _smtp._tls.<domain> TXT record
    • ✅ Extraction of rua= email addresses
    • ✅ Report delivery via queue
  • Report Format: JSON per RFC 8460
  • Manual Trigger: Via SendReportsNow()
  • Status: Production ready

Summary Table

Feature Status Notes
EHLO/HELO ✅ YES Production ready
8BITMIME ✅ YES Production ready
SMTPUTF8 ✅ YES Production ready
CHUNKING/BDAT ✅ YES Production ready
PIPELINING ⚠️ PARTIAL Advertised but not fully buffered
DSN ✅ YES Production ready
SIZE ✅ YES Production ready
STARTTLS ✅ YES Production ready (inbound & outbound)
DKIM Signing ✅ YES Per-domain, RSA & Ed25519
DKIM Verification ✅ YES DNS lookup & verification
SPF ✅ YES Full RFC 7208 compliance
DMARC ✅ YES Full RFC 7489 compliance
ARC Signing ✅ YES RFC 8617 compliant
ARC Verification ✅ YES RFC 8617 compliant
Queue Persistence ✅ YES SQLite based
Retry with Backoff ✅ YES Exponential backoff
Worker Pool ✅ YES Configurable workers
Stale Recovery ✅ YES Auto-recovery on startup
DSN Generation ✅ YES RFC 3464 compliant
MTA-STS Fetching ✅ YES RFC 8461 compliant
MTA-STS Enforcement ✅ YES enforce/testing/none modes
TLS per Domain ✅ YES RequireTLS flag
DANE Verification ❌ NO Records can be generated but not verified
PTR Verification ✅ YES Forward-confirmed
Rate Limiting ✅ YES Per-IP limits
Max Connections ✅ YES Semaphore based
Tarpitting ✅ YES Per-domain configurable with exponential backoff
Greylisting ✅ YES Per-domain configurable
Attachment Upload ✅ YES 25MB limit
Attachment Download ✅ YES Secure serving
Plain Text Compose ✅ YES Textarea input
Reply/Forward ✅ YES Both implemented
DMARC Reports ✅ YES Weekly distribution
TLS-RPT Reports ✅ YES Weekly distribution

Known Limitations & Gaps

  1. DANE Verification Not Implemented - TLSA records can be generated but not verified during outbound delivery. Current TLS relies on certificate chain validation only.

  2. PIPELINING Partial - Advertised but command processing is sequential, not buffered. Clients should not exploit pipelining.

  3. No Forward Secrecy Tracking - TLS sessions not analyzed for forward secrecy support.

  4. Limited Policy Enforcement - MTA-STS policy validation could be enhanced with more granular error handling.

Recommendations for Enhancement

  1. Implement DANE Verification - Add TLSA record lookup and verification in outbound delivery for enhanced security
  2. Enhance PIPELINING - Implement command buffering to fully support pipelined SMTP
  3. Add TLS-ED Certificate Support - Track Ed25519 certificate usage in TLS-RPT
  4. Implement BIMI - Consider adding Brand Indicators for Message Identification support

Testing Notes

All features have been verified in code. For production deployment:

  • Use config.dev.json for testing (plaintext SMTP)
  • Test DKIM/SPF/DMARC with NIST DANE test validator
  • Verify TLS certificates with Let's Encrypt ACME integration (configured in tls/)
  • Monitor delivery queue with go test ./delivery for retry logic