Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion bots/blue-liquidation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,10 @@ executable quote from the top venue — `0x` / `1inch` / `lifi` (one rate-limite
route-bound fixed sell amount; LiFi routes keyless) — falling through to the next venue on failure.
A free oracle-based route-quality check (against the full-path oracle reference) rejects any route
more than `MAX_ROUTE_IMPACT_BPS` below it. Quotes are made only for the small liquidatable set; a
per-`(id, borrower)` exponential backoff suppresses repeated failures.
per-`(id, borrower)` exponential backoff suppresses repeated failures — including a send the chain
declined with an execution revert, which is a deliberate divergence from `bots/midnight-liquidation`
(blue's liquidation incentive is static, so a shortfall on this block does predict the next one; see
[TIB-2026-08-28](../../docs/decisions/TIB-2026-08-28-midnight-send-shortfall-classification.md)).

### Simulation

Expand Down
2 changes: 1 addition & 1 deletion bots/blue-liquidation/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ import { SETTLED_COOLDOWN_BLOCKS } from './constants'
import { createGraphqlCandidateSource, discoverCandidates } from './discovery/borrowers'
import { encodeLiquidationExec } from './execution/encode-call'
import { composeQuoting } from './quotes'
import { revertReason } from './revert.utils'
import { runTick } from './runner/tick'
import { readBlueLiquidationLens } from './state/lens.sol'
import { createMarketParamsResolver, multicallIdToMarketParams } from './state/market-params'
import { revertReason } from './tx-error'

async function main() {
const config = loadConfig()
Expand Down
3 changes: 3 additions & 0 deletions bots/blue-liquidation/src/runner/tick.ts
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,9 @@ export async function runTick(deps: {
// block from re-quoting, re-simulating and re-sending it — reaching this line at all means any
// earlier entry had already expired, so leaving it untouched suppresses nothing. A queue-wide
// refusal says nothing about the position, so it records nothing.
// Blue keeps backoff on every rejected send, including an execution revert, because its
// liquidation incentive is static — unlike midnight, which exempts that case. See
// docs/decisions/TIB-2026-08-28-midnight-send-shortfall-classification.md.
if (outcome.reason === 'send_failed') backoff.record(label, chainHead)
}
}
Expand Down
13 changes: 12 additions & 1 deletion bots/blue-liquidation/test/runner/tick.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,18 @@ describe('runTick', () => {
// would re-quote, re-simulate and re-send.
const { counters, backoff } = await runWith({
seedBackoffAt: 1n,
submitOutcome: { sent: false, reason: 'send_failed' }
submitOutcome: { sent: false, reason: 'send_failed', executionRevert: false }
})
expect(counters).toMatchObject({ ok: 1, submitted: 0, notSent: 1 })
expect(backoff.shouldSkip(LABEL, 101n)).toBe(true)
})

it("keeps backoff on an execution-reverted send: blue's incentive is static", async () => {
// Pins the divergence documented at the backoff.record call in src/runner/tick.ts: midnight
// exempts this case, blue must not.
const { counters, backoff } = await runWith({
seedBackoffAt: 1n,
submitOutcome: { sent: false, reason: 'send_failed', executionRevert: true }
})
expect(counters).toMatchObject({ ok: 1, submitted: 0, notSent: 1 })
expect(backoff.shouldSkip(LABEL, 101n)).toBe(true)
Expand Down
40 changes: 38 additions & 2 deletions bots/midnight-liquidation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Environment variables:
| `HEADROOM_FLOOR_BPS` | no | `3` | **Lower bound** on swap execution cost — the cheapest route you would ever expect, NOT a typical cost. A seize-exact plan's whole margin is the incentive `(lif - 1)/lif`, so a plan below this floor cannot fund its own repay by any route and is skipped as `plan.skipped` / `insufficient_headroom` before it costs a quote, a simulation or a gas estimate. Post-maturity the incentive ramps from zero over an hour, so this acts as a pure time gate: `3` suppresses roughly the first 25s on a 4.4%-maxLif tier. Set it too high and it blinds the earliest, most contested part of a maturity. `0` disables the gate. |
| `MIN_SURPLUS_BPS` | no | `0` | Surplus over break-even a quoted route's **expected** output must clear before the bot spends a simulation on it, in bps of the plan's contract-derived repay. `0` is pure break-even: both sides then come from the contract's own formula with no tuned value, so the gate can only reject plans that would have reverted anyway. It gates the expected output only — the min-out actually encoded in the swap calldata stays at break-even — so raising it buys margin against a route that underperforms its quote, not against oracle drift between simulation and inclusion. |
| `PENDLE_SLIPPAGE_BPS` | no | `50` | Slippage for the Pendle PT → underlying unwrap hop (before the downstream venue sells). |
| `BACKOFF_BASE_BLOCKS` / `BACKOFF_MAX_BLOCKS` | no | `2` / `64` | Exponential per-position cooldown (in blocks) after a failed quote/simulate, bounding API + RPC usage under a backlog. |
| `BACKOFF_BASE_BLOCKS` / `BACKOFF_MAX_BLOCKS` | no | `2` / `64` | Exponential per-position cooldown (in blocks) after a failed quote/simulate, bounding API + RPC usage under a backlog. An economic refusal (`floor_unmet`, an unprofitable quote) never arms it, and neither does a send the chain itself declined — see [Broadcast And Pending Queue](#broadcast-and-pending-queue). |
| `POSITION_LIQUIDATION_COOLDOWN_MS` | no | `0` | Opt-in per-position cooldown (ms) after a failed liquidation attempt; `0` disables it (re-attempt every tick). |
| `BETTERSTACK_SOURCE_TOKEN` / `BETTERSTACK_INGESTING_HOST` | no | — | Opt-in log shipping; when both are set the bot's in-process loglayer transport ships structured logs to BetterStack (inert otherwise). |
| `BETTERSTACK_HEARTBEAT_URL` | no | — | Optional Better Stack Uptime heartbeat URL, pinged every minute; failures only log a warning and never interrupt liquidations. |
Expand Down Expand Up @@ -511,7 +511,7 @@ The bot computes the oracle-priced reference output for free (no extra API call)
route more than `MAX_ROUTE_IMPACT_BPS` below it (`quote.route_quality_failed`). Quote failures (no
route, timeout, rate-limited, API error) log `quote.failed`; once every ranked venue is exhausted the
position is backed off — an exponential per-position cooldown that bounds API + RPC usage when many
positions fail (the rate-limit defense). A successful submit clears the backoff.
positions fail (the rate-limit defense). A successful submit clears the backoff; nothing else does.

If no venue is enabled (bad-debt-only mode) or the collateral is on `EXCLUDE_COLLATERALS`, the tick
logs `config.no_swap_path` and skips the candidate (no API call, no backoff). Pure bad-debt
Expand Down Expand Up @@ -547,6 +547,42 @@ While a label is pending, later ticks skip that position. On each block the queu
logs confirmed or reverted transactions, and fee-bumps stuck transactions until either they confirm,
hit the fee ceiling, or exhaust bump attempts.

A queue answer that broadcast nothing is classified three ways, counted on `tick.end` as
`sendRefused` / `sendReverted` / `sendRejected` (which sum to `notSent`):

- **refused** — the queue declined before reaching the send (aborted-send latch, failed nonce sync,
nonce hole). Queue-wide, so it is held against no position.
- **execution-reverted** — the node rejected this position's own transaction with an on-chain
execution revert (`tx.submit_failed`, `executionRevert: true`, plus the 4-byte `selector` when the
payload carried one). **This does not extend the position's suppression window**, and it also
exempts the position from an entry a sibling candidate armed, whichever order the two ran in — an
execution revert, unlike a broadcast, does not stop the next-ranked sibling from being tried.
Post-maturity the LIF ramps over an hour, so a min-out shortfall says nothing about the next
attempt; backing off sampled that ramp
exponentially, which is what turned a maturity into 4–9 minutes per position on 2026-08-28. The
position is instead retried as fast as the tick can drain, with skipped heights coalesced. An
opted-in `POSITION_LIQUIDATION_COOLDOWN_MS` window is deliberately left armed: it is a flat
operator throttle, not a ramp sampler, so lifting it is an operator's call.
- **rejected** — the send machinery failed (nonce, funds, RPC). Nothing was learned about the plan,
so the position backs off exactly as before.

Because the execution-reverted case carries no throttle at all, an unbroken streak of
execution-reverted sends on one position is tracked and reported as `send.revert_streak` (warn) on the
one send that first takes it past 15 minutes, carrying the revert count, the streak duration, the last
selector, and whether that selector stayed constant across the streak. Only that crossing is logged:
with no throttle on the path, warning on every later revert would ship a line per tick — two, when
both of a position's siblings revert — for as long as it stays stuck. A constant selector over a long
streak points at a structural fault — an expired route deadline, malformed aggregator calldata,
an estimator/provider discrepancy, a gate that keeps closing — rather than an incentive that has yet
to catch up. The threshold is a duration and not an attempt count on purpose: the incentive ramps on
wall clock, so attempts-to-clear is `clearing_time / sweep_period` and shrinks every time the bot
gets faster, which a count threshold would have to be recalibrated against. The streak only reports;
it never suppresses.

`bots/blue-liquidation` deliberately diverges here and keeps backoff on every rejected send,
execution reverts included: its liquidation incentive is static rather than ramping, so a declined
send there really is evidence about the next block.

Queue state is in-memory. On restart, chain truth wins: the bot rediscovers live candidates and the
signer nonce cursor starts from the pending chain nonce. If the initial raw broadcast fails after a
nonce is claimed but before a hash is returned, the signer rolls the cursor back and the queue aborts
Expand Down
7 changes: 6 additions & 1 deletion bots/midnight-liquidation/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,10 @@ import { createListedMarketFilter, createUnionListedMarketFilter } from './disco
import { createTokenPriceSource } from './discovery/token-prices'
import { encodeLiquidationExec } from './execution/encode-call'
import { composeQuoting } from './quotes'
import { revertReason } from './revert.utils'
import { createRevertStreakStore } from './runner/revert-streak'
import { runTick } from './runner/tick'
import { readMidnightLiquidationLens } from './state/lens.sol'
import { revertReason } from './tx-error'

async function main() {
const config = loadConfig()
Expand Down Expand Up @@ -220,6 +221,9 @@ async function main() {
// Opt-in per-position cooldown (default disabled): one in-memory store for the process lifetime,
// complementary to `backoff` (see POSITION_LIQUIDATION_COOLDOWN_MS).
const cooldown = createCooldownStore({ cooldownMs: config.positionCooldownMs })
// Telemetry only: an execution-reverted send is exempt from backoff, so this is what reports a
// position whose sends keep being declined — see `createRevertStreakStore` for the threshold.
const revertStreaks = createRevertStreakStore()

// The exec calldata for one liquidation — the same bytes the simulate gate checks and the queue
// broadcasts, so a sim-ok plan and its broadcast can't drift.
Expand Down Expand Up @@ -372,6 +376,7 @@ async function main() {
},
backoff,
cooldown,
revertStreaks,
inflightLabels: () => queue.inflightLabels(),
usdValueOf: tokenPrices.usdValueOf,
// Phase A.5's probe seam. `refresh` is staleness-gated and `select` is a pure cache lookup, so
Expand Down
101 changes: 101 additions & 0 deletions bots/midnight-liquidation/src/runner/revert-streak.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
import type { Hex } from 'viem'

/**
* How long an unbroken execution-revert streak may run before {@link RevertStreak.escalate} reports it.
*
* A DURATION rather than an attempt count, because post-maturity LIF ramps on wall-clock: a position
* becomes fundable at a moment, not after N tries, and attempts-to-clear is `clearing_time /
* sweep_period` — the sweep period being exactly what shrinks as the bot gets faster. A count
* threshold would need recalibrating on every latency win and would start firing on healthy positions.
* 15 minutes sits past the 4–9 minutes positions actually took to clear on 2026-08-28 and well inside
* the 60-minute `TIME_TO_MAX_LIF` ramp.
*/
export const REVERT_STREAK_ESCALATE_MS = 15 * 60_000

/** What {@link RevertStreakStore.record} learned about the streak the just-recorded revert extends. */
export type RevertStreak = {
/** Consecutive execution-reverted sends, this one included. */
count: number
/** Wall-clock ms from the streak's FIRST execution-reverted send to this one; `0` on the first. */
durationMs: number
/** The 4-byte selector this send reverted with, absent when the payload carried none. */
selector: Hex | undefined
/**
* True while every send in the streak reported the same selector — much stronger evidence of a
* structural fault (a closed gate, malformed calldata, an estimator discrepancy) than a mixed
* streak, which reads as ordinary min-out shortfalls against whichever pool the route hit.
*/
selectorConstant: boolean
/**
* Where this revert sits against the store's threshold ({@link REVERT_STREAK_ESCALATE_MS}):
* `crossed` on the one revert that first runs past it, `ongoing` on every revert after that. A
* reporter must fire on `crossed` alone — `ongoing` repeats for as long as the position stays
* stuck, which on a per-block sweep is unbounded.
*/
escalate: 'below' | 'crossed' | 'ongoing'
}

/**
* Per-position tracker of consecutive execution-reverted sends, keyed by the `${id}:${borrower}`
* label. The backstop that makes running an execution-reverted send with NO retry throttle
* defensible: a min-out shortfall clears as the LIF ramps, but a persistent estimator-only failure —
* an expired route deadline, malformed aggregator calldata, a gate that keeps closing — can pass the
* simulation and fail the send's gas estimate indefinitely, which would otherwise burn quotes forever
* without progressing. Both calls run at `latest`; they diverge because they are issued by DIFFERENT
* clients over their own `failover` transport pairs, so they can observe different heads, different
* provider-side estimator behaviour, and pool state that moved in between. It only reports; it never
* suppresses.
*
* In-memory only, like the shared `Backoff` and `CooldownStore` — chain truth wins on restart. Entries
* for a position that recovers to non-liquidatable are never re-checked and linger until process exit:
* the same accepted, bounded leak `createBackoff` documents at its canonical home.
*/
export type RevertStreakStore = {
/** Extends `label`'s streak with one execution-reverted send and returns its state. */
record: (label: string, selector?: Hex) => RevertStreak
/** Ends `label`'s streak — a broadcast, or a send failure the chain did not decline. */
reset: (label: string) => void
}

type Entry = {
count: number
startedAt: number
selector: Hex | undefined
constant: boolean
escalated: boolean
}

/**
* Lives in this bot rather than `@repo/bot-kit` because the threshold is calibrated against one bot's
* incentive shape — a wall-clock LIF ramp — and no second consumer exists yet.
*/
export const createRevertStreakStore = (
opts: { escalateAfterMs?: number; now?: () => number } = {}
): RevertStreakStore => {
const escalateAfterMs = opts.escalateAfterMs ?? REVERT_STREAK_ESCALATE_MS
const now = opts.now ?? (() => Date.now())
const streaks = new Map<string, Entry>()

return {
record: (label, selector) => {
const at = now()
const previous = streaks.get(label)
const count = (previous?.count ?? 0) + 1
const startedAt = previous?.startedAt ?? at
Comment thread
haydenshively marked this conversation as resolved.
const constant = previous ? previous.constant && previous.selector === selector : true
Comment thread
haydenshively marked this conversation as resolved.
Comment thread
haydenshively marked this conversation as resolved.
const durationMs = at - startedAt
const past = durationMs > escalateAfterMs
streaks.set(label, { count, startedAt, selector, constant, escalated: past })
return {
count,
durationMs,
selector,
selectorConstant: constant,
escalate: past ? (previous?.escalated ? 'ongoing' : 'crossed') : 'below'
}
},
reset: label => {
streaks.delete(label)
}
}
}
Loading
Loading