Add Real-Time Mode (RTM) sub-second latency streaming demo

[jiteshsoni]

Summary

Companion code for the blog post Unlocking Sub-Second Latency with Spark Structured Streaming Real-Time Mode.

Demonstrates Spark Real-Time Mode (RTM) for achieving ~5ms to ~300ms latency (depending on workload complexity) in stateless streaming pipelines with a Kafka-to-Kafka guardrail pattern.

Contents

File	Description
rtm_stateless_guardrail.py	Main notebook - RTM guardrail pipeline with dynamic topic routing
cluster_config.json	Cluster configuration with RTM settings (no autoscaling, no Photon)
test_rtm_guardrail.py	Local Python tests for validation logic (no Kafka required)
produce_test_data.py	Test data producer for sending sample Ethereum block events
README.md	Comprehensive documentation with best practices and troubleshooting

Key Features

• Real-Time Mode trigger: .trigger(realTime="5 minutes") for sub-second latency
• Native Spark SQL: Pattern detection using F.rlike() (RTM-compatible, no Python UDFs)
• Databricks Secrets: No hardcoded credentials - uses dbutils.secrets.get()
• Sensitive data detection: Email, SSN, Credit Card, JWT, AWS Keys, Private Keys
• Validation rules: HIGH_GAS_USAGE, HIGH_TX_COUNT, EMPTY_BLOCK, ZERO_MINER
• Dynamic topic routing: ALLOW events → *-allowed, QUARANTINE events → *-quarantine
• Production configs: RocksDB state store, changelog checkpointing, stable checkpoint paths

RTM Requirements (Critical)

Requirement	Status
DBR 16.4+	✅ Required
Dedicated clusters	✅ Required (no serverless/shared)
Autoscaling	❌ Must be DISABLED
Photon	❌ NOT supported
Output mode	✅ update mode required
maxOffsetsPerTrigger	❌ NOT compatible with RTM

Best Practices Implemented

• ✅ Stable checkpoint paths (no UUID in production)
• ✅ RocksDB state store with changelog checkpointing (for future-proofing)
• ✅ Native Spark SQL operators (no Python UDFs for better performance)
• ✅ Kafka timeout configs for production stability
• ✅ Consumer group ID for monitoring
• ✅ Secrets management via Databricks secrets
• ✅ At-least-once delivery semantics (Kafka sink)

Testing Completed

Environment:

• Cluster: rtm-guardrail-demo (e2-demo-field-eng workspace)
• Runtime: DBR 16.4 LTS
• Kafka: Redpanda Serverless (SCRAM-SHA-256 auth, SASL_SSL)

Test Data:

• ✅ Sent 7 test ethereum blocks with various validation triggers
• ✅ Verified 2 clean blocks routed to ALLOWED topic
• ✅ Verified 5 blocks with issues routed to QUARANTINE topic

Validation Results:

• ✅ HIGH_GAS_USAGE detection (block 1000002)
• ✅ EMPTY_BLOCK detection (block 1000003)
• ✅ PII_EMAIL detection (block 1000004)
• ✅ ZERO_MINER detection (block 1000006)
• ✅ HIGH_TX_COUNT detection (block 1000007)

Performance:

• ✅ RTM streaming with sub-second latency confirmed
• ✅ Dynamic topic routing working correctly
• ✅ All validation rules triggering as expected

References

• Databricks RTM Trigger Documentation
• Databricks RTM Blog Post
• Blog Post: Sub-Second Latency with RTM