Your AI app passed its demo. There's nothing testing the parts that fail in production. Before you scale it, wrap a harness around it.
Two free things in this repo:
- A 40-point harness-readiness checklist. Run it against your system. Everything you can't tick is a failure class with no gate in front of it.
- 5 generator skills that build the missing gates around your codebase, in any language, by running them through an AI coding agent (Claude Code / Cursor).
Plus real case studies, including a 4-week production trace outage on a system I run, where the health check stayed green the whole time.
All free. No paywall, no email wall, nothing to buy. It's markdown you run through an agent against your own repo. The harness it builds is yours and lives in your repo. Nothing phones home.
A test suite is something you write up front. A harness is something you wrap around code that already ships, to catch the failures you'd otherwise only find in production.
Most vibe-coded AI apps have neither. They've got a green /health check and a prayer. A green health check tells you the container is up. It doesn't tell you traces are landing, messages are sending, the agent stayed inside its budget, or that today's model still answers the way last week's did. Those break quietly, and the quiet is the whole problem. You find out from a customer.
This won't make you adopt my harness. The skills read your repo and write gates in your stack, and you commit them. There's no framework to install and no runtime.
Your demo works. That's not the question. The question is what happens the first time it quietly stops working, and how long before you notice.
Tick what's true. Everything unticked is a failure class with no gate in front of it.
- 1. You have a written list of how this system has actually failed: real incidents, dead-letter rows, support tickets. Not a guessed list.
- 2. Each known failure class maps to one gate that would refuse it before deploy. An unmapped failure has no gate.
- 3. You can name the single highest-blast-radius failure (revenue, or customer-facing) and point to the test that covers it.
- 4. You separate existence checks from arrival checks. A green
/healthisn't "traces are landing," "messages are sending," or "jobs are draining." - 5. Silent failures have a freshness watchdog (last-trace timestamp, dead-letter count, age of last successful job), not just an up/down ping.
- 6. You know which parts are AI/LLM (non-deterministic) and which are deterministic, and you test them differently.
- 7. Every "it works" claim is backed by a command a stranger can run and get the same answer from.
- 8. The harness reads each service's real lint/test/build/deploy commands from one declarative file (a registry), not hardcoded per script.
- 9. Every gate reads that registry through one injectable seam (an env var or config path), so you can point it at a throwaway fixture.
- 10. Gates run hermetically. No gate needs network or production access to be tested.
- 11. LLM calls are pinned to an exact model version, never a floating alias (
gpt-4o,claude-3-5-sonnet-latestdrift under you silently). - 12. Tests are deterministic, or explicitly marked stochastic. A stochastic check you run once isn't really a test.
- 13. Time, randomness, and IDs are injectable, so you can reproduce a failing run exactly.
- 14. You have at least one versioned golden dataset (inputs + expected behavior) committed to the repo as code.
- 15. Golden cases come from real production failures and support queues, not synthetic generation.
- 16. There's a holdout split you never tune against.
- 17. Deterministic graders (string / JSON / exit-code) cover everything checkable without judgment. You use LLM-as-judge only for genuinely subjective criteria.
- 18. Any LLM-as-judge is fenced (user data delimited as untrusted), pinned to a version, returns a structured verdict, and defaults to FAIL on empty or garbled output.
- 19. The eval gates on an absolute pass-rate floor and a regression delta against a baseline score committed to the repo.
- 20. There's a known-failing case in the suite, asserted to fail, so you know the grader can actually fail.
- 21. verify: deterministic checks (lint / type / unit) run from the registry's real commands. No "unified test" that doesn't match how each service actually tests.
- 22. eval: the golden suite runs and gates on floor + regression on every change to AI behavior.
- 23. e2e: the real end-to-end path is default-deny. You have to opt in (
--confirm); the literal default is off. - 24. deploy: only SHAs reachable from
origindeploy (ancestry check), never the working tree. Dry-run is the default; shipping is opt-in. - 25. deploy: fails closed when offline or when a required tool is missing.
gitleaksnot installed means refuse, not skip. - 26. deploy: a secret scan runs and refuses on any hit, before anything ships.
- 27. agent-safety (if you run agents): a kill-switch exit code, a cost/attempt ceiling, bounded retries, and an always-valid-JSON trace of what the agent did.
- 28. post-deploy: a "healthy but broken" check that confirms the new code actually serves correct responses, not just that the container came up.
- 29. Every gate has a self-test that builds a throwaway fixture, points the seam at it, and asserts the exit code and output.
- 30. Stateful gates test every exit path (pass / fail / kill / ceiling), not just the happy one.
- 31. The whole self-test suite runs with one command and prints "N passed, 0 failed."
- 32. Any gate without a self-test is documented as untested, not claimed as a control.
- 33. Deterministic gates run on every commit (pre-commit hook or CI), not when someone remembers.
- 34. The full eval runs on PR / merge and blocks the merge on failure.
- 35. A weekly canary re-runs the eval against production behavior, to catch drift no PR introduced (model updates, data shift).
- 36. Production failures auto-nominate new golden cases, so the loop closes.
- 37. The harness itself is in version control with a real commit history. If your deploy gate checks SHAs, the harness can't live in an untracked folder.
- 38. Migrations have a gate, or at least an explicit acknowledged reminder, before deploy. Not silent.
- 39. The harness fails closed by default, everywhere. An unhandled state refuses; it never passes by accident.
- 40. You've run the whole harness end to end at least once and watched it catch a real problem you didn't already know about.
- You ticked all 40.
- You watched the harness refuse on a real failure (item 40), not just pass.
- The harness is committed (item 37), so its own claims are reproducible.
If you can't tick all 40, you don't have a harness yet. You've got a health check and optimism. That's fine for a demo and expensive in production.
These build the missing gates around your codebase. They're in 5-free-skills/. Drop them into .claude/skills/ and run with /skill <name> in Claude Code, or paste into Cursor. Each one reads your repo and writes working files in your language.
| Skill | What it builds |
|---|---|
5-free-skills/detect-failure-classes.md |
Mines your logs, dead-letter tables, and incident history into a ranked list of failure classes the harness has to gate. Start here. |
5-free-skills/find-seams.md |
Finds the one injectable seam (env var / config path) that lets every gate be tested with no prod access. |
5-free-skills/scaffold-golden-suite.md |
Generates a versioned golden dataset, a deterministic grader, and a regression baseline from your real production failures. |
5-free-skills/build-deploy-gate.md |
Generates a fail-closed deploy gate: deploys only origin-reachable SHAs, dry-run by default, refuses on secret-scan hits or when offline. |
5-free-skills/selftest-the-harness.md |
Generates hermetic self-tests for the gates you built, so the harness proves itself instead of claiming things. |
| Case Study | What went wrong |
|---|---|
free-case-studies/connectivitycx-otlp-404.md |
A production LLM agent's tracing was dead for ~4 weeks. Health was green the whole time. One wrong character in a URL. |
free-case-studies/air-canada-chatbot.md |
Air Canada's chatbot invented a refund policy. A tribunal held the airline liable. No eval, no guardrail. |
More to come, one at a time.
Not a SaaS. Not a framework you npm install. Nothing phones home.
These are static markdown files. You run them through an agent against your own repo, the agent writes gates in your stack, you commit them. No dashboard, no runtime, no dependency on me.
It's also not a substitute for real testing discipline if you're in a regulated domain (health, finance). It covers the failure classes that show up constantly in AI apps shipped straight from a working demo. It doesn't cover everything.
I'm Rishabh Kumar (@rispectrum). I build and operate LLM systems in production — production RAG, autonomous email/voice agents, multi-agent orchestration. The 4-week trace outage in the case study is from a system I run (milo-agent, a multi-tenant SMS agent), not a reconstruction. The harness patterns in these skills — the seam, the fail-closed deploy gate, the freshness watchdog — are the ones I actually use.
These skills generate a harness. What they can't do is tell your team which failure classes actually matter for your system, or pressure-test the harness you end up with. That judgment is the part I do for teams.
If you've shipped an AI feature from a working demo and want a second set of eyes on what fails silently before you scale, that's the work.
I take on a few harness reviews a quarter: I run this checklist against your system and tell you which of the 40 you're actually missing, and why.
→ rishabh.vaaiv@gmail.com · rsbk.cc · @rispectrum
If this saved you from scaling something that fails silently, star the repo and send it to whoever shipped an AI feature straight from the demo.