diff --git a/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg b/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg new file mode 100644 index 00000000000..834e3c61a18 --- /dev/null +++ b/content/blog/2026/11/let-there-be-chaos/chaos-exploration.svg @@ -0,0 +1,317 @@ + + +Ten minutes of chaos: what the AI tried on the shop, and what each route did +10 experiments chosen by a local model (qwen3.6:35b-mlx), one at a time, within a 10 minute budget + + +0 min + +1 min + +2 min + +3 min + +4 min + +5 min + +6 min + +7 min + +8 min + +9 min + +10 min +when +experiment 1: PASS, 0:26 to 1:07 + +experiment 2: PASS, 1:11 to 1:52 + +experiment 3: FAIL, 1:57 to 2:04 + +experiment 4: PASS, 2:20 to 3:01 + +experiment 5: PASS, 3:06 to 3:48 + +experiment 6: INCONCLUSIVE, 3:53 to 5:24 + +experiment 7: PASS, 5:29 to 7:00 + +experiment 8: PASS, 7:07 to 7:48 + +experiment 9: PASS, 7:54 to 8:35 + +experiment 10: PASS, 8:40 to 9:22 + + +report + +✓ +#1 +PASS +payment answers +503, 25% + +✓ +#2 +PASS +shipping route +fails, 20% + +✗ +#3 +FAIL +payment answers +503, 25% + +✓ +#4 +PASS +payment times +out (5s), 10% + +✓ +#5 +PASS +payment gets +duplicates, 10% + +? +#6 +UNDECIDED +payment down, +dead letter fails + +✓ +#7 +PASS +b +the same, +waits longer + +✓ +#8 +PASS +parked file +write fails, 25% + +✓ +#9 +PASS +a +payment times +out (5s), 10% + +✓ +#10 +PASS +c +payment route +fails, 5% +orders +experiment 1, orders: 41 completed, 0 failed, 11 redeliveries + +41 ok +11 retried + +experiment 2, orders: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 3, orders: 6 completed, 0 failed, 3 redeliveries + +6 ok +3 retried + +experiment 4, orders: 32 completed, 0 failed, 2 redeliveries + +32 ok +2 retried + +experiment 5, orders: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 6, orders: 90 completed, 0 failed, 118 redeliveries, 58 lost at the dead letter channel + + +58 lost +at dead letter + +experiment 7, orders: 90 completed, 0 failed, 117 redeliveries, 58 lost at the dead letter channel + + +58 lost +at dead letter + +experiment 8, orders: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 9, orders: 36 completed, 0 failed, 1 redeliveries + +36 ok +1 retried + +experiment 10, orders: 41 completed, 0 failed, 0 redeliveries + +41 ok + +refunds + +no traffic + +no traffic +experiment 3, refunds: 15 completed, 5 failed, 0 redeliveries + + +5 failed +gone + + +no traffic +experiment 5, refunds: 20 completed, 0 failed, 0 redeliveries + +20 ok + + +no traffic + +no traffic + +no traffic +experiment 9, refunds: 18 completed, 2 failed, 0 redeliveries + + +2 failed +gone + + +no traffic +shipping +experiment 1, shipping: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 2, shipping: 41 completed, 0 failed, 5 redeliveries + +41 ok +5 retried + +experiment 3, shipping: 6 completed, 0 failed, 0 redeliveries + +6 ok + +experiment 4, shipping: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 5, shipping: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 6, shipping: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 7, shipping: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 8, shipping: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 9, shipping: 36 completed, 0 failed, 0 redeliveries + +36 ok + +experiment 10, shipping: 41 completed, 0 failed, 0 redeliveries + +41 ok + +parked + +no traffic + +no traffic + +no traffic + +no traffic + +no traffic + +no traffic + +no traffic +experiment 8, parked: 20 completed, 0 failed, 0 redeliveries + +20 ok + + +no traffic + +no traffic +payment +experiment 1, payment: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 2, payment: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 3, payment: 21 completed, 0 failed, 0 redeliveries + +21 ok + +experiment 4, payment: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 5, payment: 69 completed, 0 failed, 0 redeliveries + +69 ok + +experiment 6, payment: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 7, payment: 32 completed, 0 failed, 0 redeliveries + +32 ok + +experiment 8, payment: 41 completed, 0 failed, 0 redeliveries + +41 ok + +experiment 9, payment: 54 completed, 0 failed, 0 redeliveries + +54 ok + +experiment 10, payment: 41 completed, 0 failed, 0 redeliveries + +41 ok + + + + +Same fault, two routes. +Orders retries through its dead letter channel and loses none; refunds has no error handling and loses 5 of 20. + +failed: the message is gone + +lost at the dead letter channel +ok: completed (retried: redelivered and then completed) +Read afterwards: +a +The expectation measured time, so the 2 failed refunds did not fail the experiment (the AI saw them, but put them down to an earlier experiment). +b +Passed in this run; it fails since CAMEL-26789, as an experiment now counts what the dead letter channel lost. +c +The fault never fired: the rule matched nothing, so the PASS says nothing. + \ No newline at end of file diff --git a/content/blog/2026/11/let-there-be-chaos/featured.jpg b/content/blog/2026/11/let-there-be-chaos/featured.jpg new file mode 100644 index 00000000000..832ae4a21f8 Binary files /dev/null and b/content/blog/2026/11/let-there-be-chaos/featured.jpg differ diff --git a/content/blog/2026/11/let-there-be-chaos/index.md b/content/blog/2026/11/let-there-be-chaos/index.md new file mode 100644 index 00000000000..73dddfa1652 --- /dev/null +++ b/content/blog/2026/11/let-there-be-chaos/index.md @@ -0,0 +1,165 @@ +--- +title: "Chaos in Camel 4.24, part 2: let there be chaos, an AI that breaks your routes on purpose" +date: 2026-11-22 +draft: false +authors: [davsclaus] +categories: ["Camel", "Tooling", "AI"] +keywords: ["apache camel", "chaos", "chaos engineering", "ai", "camel cli", "mcp", "local llm", "resilience", "camel 4.24"] +preview: "camel-chaos lets you add faults to a running integration. Now you can hand the whole thing to an AI: give it ten minutes, and it tries faults by itself, measures what held and what broke, and comes back with a report and scenarios you can replay." +--- + +In [part 1](/blog/2026/11/camel-chaos/) you added the faults yourself: "fail a quarter of the payment calls", +"fail the dead letter channel too", and then you read the report. That works, but it means you have to know what to +try. With Camel 4.24 you can hand that part to an AI as well: + +```shell +camel cmd chaos-explore shop --budget=10m +``` + +Let there be chaos. For ten minutes the AI of `camel ask` explores how the integration copes with faults, by +itself. It reads what the routes do, picks a fault to try, says what should hold, runs the experiment, reads the +verdict, and goes on. At the end it writes a report: what held, what broke and why, and a hint for each finding. The +integration keeps running; the AI never changes your routes. + +![Let there be chaos](let-there-be-chaos.gif) + +## The shop + +The shop is the one from the [chaos example](https://github.com/apache/camel-jbang-examples/tree/main/fail-well/chaos): +an order every second, charged at a payment service over HTTP, handed to shipping, with a dead letter channel that +parks what could not be charged. We added one route: refunds arrive on Kafka and are paid back at the payment +service. The refunds route has no error handling of its own. That is the kind of gap you want found before +production. Kafka is stubbed (`camel run --stub=kafka`), so no broker is needed and nothing real is touched. + +```shell +camel run * --chaos --stub=kafka --name=shop +camel cmd chaos-explore shop --budget=10m +``` + +The model is qwen3.6 (35B), running locally on a laptop. No cloud, no API key. + +## Ten minutes, in one picture + +Here is the whole run. Along the top, when each experiment ran within the budget; below, one column per experiment, +with what it did to each route: + +![Ten minutes of chaos: ten experiments, and what each did to each route](chaos-exploration.svg) + +Look at columns 1 and 3. It is the same fault both times: the payment service answers 503 to a quarter of the calls. +The orders retried 11 calls through their dead letter channel, and every order got through. The refunds have no +retry: 5 of the 20 failed, and they are gone. + +The AI did not stop at the verdict. It looked up the errors of the integration by itself and explained what it saw: + +> The refunds route failed with `HttpOperationFailedException` - it has no error handling for the HTTP call to +> payment. [...] This is a real issue: the refunds route needs better error handling (like an onException with +> redelivery or a dead letter channel). + +That is the finding, the evidence and the hint, in the AI's own words, without anybody telling it where to look. + +In another run it went for the nastier case: the payment service down, and the dead letter channel failing too. +The dead letter channel then handles the orders it could not park, so they neither fail nor complete: they are just +gone. That run reported "when payment returned 503 and the dead letter channel also failed, 2 messages were lost at +the dead letter channel" (columns 6 and 7 above are the same experiment, before Camel counted those losses; more on +that below). + +## The journal + +Every experiment goes into the journal of the exploration, with the hypothesis the AI wrote for it and the evidence +for the verdict. These are the ten experiments of the run, as the AI phrased them: + +| # | Hypothesis | Expected | Verdict | Evidence | +|---|---|---|---|---| +| 1 | When payment returns 503, orders should retry and not lose messages | orders: failed ≤ 0 | PASS | 0 failed, 11 retried | +| 2 | When shipping fails, orders should not lose the message handed off via seda | orders→shipping: lost ≤ 0 | PASS | 0 lost (39 in orders, 39 in shipping) | +| 3 | When payment returns 503, refunds should handle errors gracefully | refunds: failed ≤ 0 | **FAIL** | **5 failed** | +| 4 | When payment times out, orders should retry and complete within reasonable time | orders: max time ≤ 30s | PASS | max 5.5s of 31 orders | +| 5 | When payment receives duplicates, refunds should handle them safely | refunds: failed ≤ 0 | PASS | 0 failed | +| 6 | When payment fails and the dead letter channel also fails, orders should not lose messages entirely | orders: failed ≤ 0 | undecided | the window was longer than the wait | +| 7 | The same, again with a shorter window | orders: failed ≤ 0 | PASS | 0 failed (58 lost at the dead letter channel) | +| 8 | When the file write fails, parked should handle the error gracefully | parked: failed ≤ 0 | PASS | 0 failed, 7 handled | +| 9 | When payment times out, refunds should complete within reasonable time | refunds: max time ≤ 30s | PASS | max 5s (2 failed) | +| 10 | When the payment route fails, HTTP clients should see proper error responses | payment: failed ≤ 0 | PASS | the fault never fired | + +The AI kept to its budget: with 38 seconds left it said that was not enough for another experiment, and finished. + +## One experiment per call + +The trick that makes this work with a small local model is that each experiment is a single tool call. The model +does not juggle rules, traffic and timers; it says what it wants to know: + +```json +{ + "hypothesis": "When payment returns 503, refunds should handle errors gracefully", + "rule": "endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=30s expect=refunds:failed<=0 within=40s", + "feed": "refunds", "body": "{\"refundId\": \"R-1\"}", "repeat": 20, "interval": "100..300" +} +``` + +Camel adds the rule, sends the 20 refunds to the stubbed Kafka topic, waits until the expectation is decided, removes +the rule, and answers with the verdict, the counts per route and the traffic. You can do exactly the same by hand, +without any AI: + +```shell +camel cmd chaos-explore-start --budget=10m +camel cmd chaos-experiment --hypothesis='a failed refund is retried, not lost' \ + --rule='endpoint=http://localhost:8080/payment error status=503 rate=0.25 until=30s expect=refunds:failed<=0 within=40s' \ + --feed=refunds --body='{"refundId": "R-1"}' --repeat=20 --interval=100..300 +camel cmd chaos-explore-finish +``` + +## Safe by construction + +You say yes once, by starting the exploration, and Camel holds the line from there, whatever the model does: + +- Chaos only runs in development (profile `dev`), never with `prod`. +- The exploration has a deadline (10 minutes by default, 2 hours at most). Every rule ends at the deadline at the + latest, and at the deadline all rules are removed, even if the AI has crashed or lost its context. +- Remote systems are stubbed, so the experiments do not reach your real Kafka or payment provider. +- The AI only adds faults and sends test messages to the routes; it does not edit your project. + +## What you keep + +Everything ends up in `test/chaos`, which `camel run` and `camel export` leave alone: + +- `test/chaos/reports/explore-