Webhook-to-CRM reliability boundary

Independent technical work sample — not client work. The webhook is fictional, the CRM is simulated, and this repository makes no production, customer, revenue, or reliability-result claim.

This standard-library Python sample demonstrates a narrow integration boundary: authenticate the exact webhook body, validate and normalize it, establish event identity, classify simulated CRM responses, and leave an inspectable audit trail without making an outbound network request. It runs as both a local CLI and an HMAC-protected preview API.

Deployed API · Verification and handoff · Runtime evidence · Alert drill · Production gaps

GET /api is the public health route. POST /api requires a private HMAC signing secret; no production credential or payload belongs in this sample.

Buyer, hard part, and operator recovery

The intended reviewer is a SaaS or agency engineer, technical lead, or operations owner investigating duplicate records, lost events, retry storms, or invisible integration failures. Connecting two APIs is not the hard part. The hard part is proving who sent an event, deciding whether it is new or conflicting, preventing duplicate side effects across restarts, bounding retries, and giving an operator enough durable evidence to recover safely.

For the local CLI, an operator investigates the SQLite event and append-only audit rows, distinguishes retry_exhausted from permanent failure or payload conflict, corrects the upstream cause, and explicitly supplies --replay-exhausted for the original event. The sample never retries forever or silently changes an event identity. The preview API deliberately uses an in-memory ledger scoped to one request and does not claim cross-request durability.

What it proves

raw local JSON bytes + HMAC-SHA256 signature
  -> constant-time signature digest comparison
  -> validation and normalization
  -> deterministic idempotency key + payload fingerprint
  -> atomic SQLite claim / duplicate / conflict decision
  -> bounded simulated HTTP retry decisions (500 and 429)
  -> success, permanent failure, exhaustion, or explicit replay state
  -> recursively redacted audit output

• HMAC-SHA256 authenticity verification occurs before JSON parsing. The expected and supplied digests are compared with hmac.compare_digest.
• SQLite stores one durable row per idempotency key and append-only audit rows. Each event decision and its audit writes share a BEGIN IMMEDIATE transaction.
• An exact successful redelivery is suppressed. Reusing an event ID with changed normalized content is a non-retryable conflict.
• Simulated HTTP 500 and 429 responses produce deterministic retry evidence. Retry-After and exponential-backoff values are recorded but never slept.
• Retry exhaustion is durable. A later process cannot try again unless the operator supplies --replay-exhausted.
• Audit values under secret, signature, token, password, cookie, authorization, and API-key field names are replaced with [REDACTED] before SQLite storage.
• Runtime code makes no outbound network calls and has no third-party dependency.
• The preview API fails closed without its platform secret, limits bodies to 64 KiB and batches to 10 events, emits payload-free structured logs bound to safe deployment metadata, and exposes no CRM write path.
• A separately authenticated Vercel Cron route emits one daily, side-effect-free runtime heartbeat. It is synthetic monitoring—not customer traffic, uptime, alert-response, adoption, or CRM-delivery evidence.
• One distinct authenticated production heartbeat was observed inside the registered daily schedule window after the operator smoke, without another operator trigger from this audit. The timing and configuration support scheduler attribution, but the runtime log does not expose cryptographic caller identity. See docs/RUNTIME_EVIDENCE.md.
• A separate public-health workflow passed one healthy control, created one controlled alert issue, produced an owner-visible GitHub notification, and received an authorized operator triage and closure. This is a synthetic drill, not a real incident or uptime claim. See docs/ALERT_DRILL.md.

Quick start

Requires Python 3.11 or later. The commands below use a public, demo-only secret and the matching signature for the checked-in sample bytes. Never reuse either in a real system.

export WEBHOOK_SIGNING_SECRET='demo-only-secret'
export DEMO_SIGNATURE='sha256=55f6559c5de036ead57e1147e6c40cf5b04b16b26add32dd3a867a380e17ad6d'

python3 demo.py \
  --input samples/webhook.json \
  --signature "$DEMO_SIGNATURE"

The default ledger is in memory so the quick start is repeatable and leaves no file. Supply --ledger PATH to prove state across separate CLI processes:

python3 demo.py \
  --input samples/webhook.json \
  --signature "$DEMO_SIGNATURE" \
  --ledger /tmp/webhook-crm-demo.sqlite3

Run the same command again to observe duplicate_suppressed. Use a new ledger path for each independent scenario.

Scenarios

Scenario	Simulated result	Final behavior
success	CRM dry-run success	would_write
transient-500-once	HTTP 500, then success	one backoff record, then would_write
rate-limit-once	HTTP 429 with Retry-After: 4, then success	rate-limit evidence, then would_write
transient-always	HTTP 500 at every attempt	retry_exhausted
permanent	invalid CRM mapping	one non-retryable failure

Example:

python3 demo.py \
  --input samples/webhook.json \
  --signature "$DEMO_SIGNATURE" \
  --scenario rate-limit-once

Intentional failure scenarios return exit code 1. Authentication, argument, JSON, and validation failures return 2. Success and duplicate suppression return 0. --max-attempts accepts values from 1 through 10.

Verify

python3 -m unittest discover -s tests -v
python3 -m compileall -q api web_api.py demo.py reliability.py tests

The suite covers valid, invalid, and missing signatures; deterministic identity; duplicates and conflicts; HTTP 500 and 429 retry paths; permanent failure; retry exhaustion; explicit replay; restart safety; recursive secret redaction; SQLite integrity; and execution under a Python audit hook that rejects socket events.

The API contracts additionally cover content type, body and batch limits, HMAC rejection, bounded query parameters, validation errors, security headers, safe internal errors, request-scoped duplicate handling, payload-free logs, fail-closed cron authentication, and sanitized commit-bound runtime metadata. See docs/VERIFICATION.md for reviewer reproduction and handoff checks.

Repository map

• demo.py — local CLI boundary and structured exit behavior.
• web_api.py — testable request validation, authentication, response, and log contract for the preview API.
• api/index.py — thin Vercel HTTP adapter.
• api/monitor.py — authenticated, side-effect-free Vercel Cron heartbeat adapter.
• vercel.json — once-daily UTC heartbeat schedule.
• reliability.py — authentication, normalization, retry policy, SQLite ledger, audit redaction, and deterministic CRM simulation.
• tests/test_demo.py — black-box CLI contracts plus focused ledger checks.
• samples/webhook.json — synthetic input only.
• docs/ARCHITECTURE.md — component and transaction boundaries.
• docs/RELIABILITY_MODEL.md — states, invariants, and retry decisions.
• docs/DEMO_SCRIPT.md — repeatable walkthrough commands.
• docs/VERIFICATION.md — checks and what each one actually proves.
• docs/PRODUCTION_GAPS.md — explicit limits before real deployment.
• docs/DEBUGGING_CASE_STUDY.md — synthetic failure investigation.
• docs/ALERT_DRILL.md — controlled health-alert receipt and response evidence.

Scope boundary

This is not a production connector. The preview endpoint accepts HTTPS and uses platform-managed secret injection and runtime logs. Its daily heartbeat can prove only a synthetic invocation at one deployed commit. The project still has no outbound HTTP client, queue, provider SDK, real CRM schema, durable cross-request ledger, timestamps or leases for distributed workers, database encryption, retention job, alerting, or SLO. See docs/PRODUCTION_GAPS.md before adapting it.

Relevant next step

Use this proof for a bounded Integration Reliability Audit: map one real event path, verify signature/idempotency/retry assumptions, reproduce one failure, and define the smallest safe fix. The next action is to provide one sanitized event contract and one observed failure mode—not production credentials.

Truthful portfolio wording

Independent technical work sample: webhook-to-CRM reliability boundary. Built a standard-library Python dry run that verifies HMAC-SHA256 signatures, normalizes synthetic webhook data, records atomic SQLite idempotency and audit decisions, suppresses duplicates, detects event-ID conflicts, and exposes bounded HTTP 500/429 retry and explicit replay behavior. It does not connect to a real CRM and does not represent client work or measured production outcomes.