tulip: optional admission-gate for run_tool(), works across all 4 framework adapters

[fede-kamel]

Addresses #381.

What this adds

An optional admission gate for ToolkitCore.run_tool(): every real call is classified and weighed against a policy before it reaches Stripe.

sequenceDiagram
    participant Agent as Agent / LLM
    participant Mixin as GovernedToolkitMixin
    participant Gate as tulip.control.admit()
    participant Stripe as mcp.stripe.com

    Agent->>Mixin: run_tool(method, args)
    Mixin->>Gate: admit(classified action, policy)
    alt allowed
        Gate->>Stripe: perform() -- the real call
        Stripe-->>Agent: result
    else held / denied
        Gate--xStripe: never called
        Gate-->>Agent: AdmissionError
    end
    Gate->>Gate: record decision (hash-chained audit trail)

Loading

class GovernedStripeAgentToolkit(GovernedToolkitMixin, StripeAgentToolkit) is the entire integration, and it composes identically across all four framework adapters (langchain, openai, crewai, strands) since the override point is ToolkitCore itself. Nothing changes for anyone who doesn't opt in — tulip-agents isn't in the core pyproject.toml, only examples/tulip/'s.

Results, not claims

examples/tulip/eval_dataset.py — 62 real Stripe API operations pulled live from stripe_api_search, across payments, customers, subscriptions, disputes, invoices, coupons, and more. examples/tulip/eval_models.py scores three classifiers against it, same policy and action text:

classifier	correct	missed real risk	over-cautious
classify() (this PR's rules)	62/62	0	0
Claude Sonnet 4.5	51/62	0	11
clusiana-admit-v4 (tulip's model-based classifier)	53/62	0	9

The metric that matters for a gate isn't raw accuracy, it's missed real risk — a case that should've required confirmation but got silently let through. Zero, for all three, on every genuinely high-risk case in the dataset. Every disagreement was in the safe direction: extra confirmation asked on a harmless config write.

Read classify()'s 62/62 as the deterministic floor this PR actually gates with, not a fair win over the two model rows: it was hand-patched against 6 of these exact cases (below), so it's grading against its own answer key. Sonnet and Clusiana got zero tuning on this dataset and still caught every real risk cold — that's the more informative number here, and Clusiana (a small model Tulip runs locally) beat a frontier model doing the same zero-shot task.

Full detail, the six real bugs the rules were patched against, and the live end-to-end run: examples/tulip/VERIFICATION.md.

Also optional: AI-based escalation

classify() stays a fixed, dependency-free rule engine — no model server required to install this package. GovernedToolkitMixin accepts an optional advisory callable that can escalate a rule-classified-low-risk call over real inference, never soften the reverse, fails safe on any error. examples/tulip/clusiana_advisory.py is a working example.

Two unrelated findings

• pyproject.toml pins mcp>=1.0.0 unbounded — mcp 2.0.0 renamed streamablehttp_client, breaking a fresh install's import in shared/mcp_client.py. Not fixed here; pinned mcp<2.0.0 locally to build/test.
• test_mcp_client.py's two most consequential cases are skipped ("Requires mocking MCP SDK internals"); this PR's own tests do that mocking instead.

Who we are

Tulip (tulip-agents, Apache-2.0, pre-GA v2.4.0) — flagging that plainly.

The ask

Whether this is a real gap from where you sit, whether the marker set is the right line, and whether an optional dependency like this is wanted at all.

[cla-assistant]

All committers have signed the CLA.

[fede-kamel]

Update: the two tools #381 names first were classified low-risk

Re-read the issue against the marker set and found a seventh bug — the worst one so far, because it's exactly the case the issue opens with.

Every entry in _HIGH_RISK_MARKERS is a reversal or destruction verb: refund, cancel, dispute, finalize, delete, void, payout. All of them describe money leaving or a record being destroyed. None of them describe money arriving. So:

call	before	now
create_payment_intent (initiates a real charge on a card)	low-risk, executed	high-risk, held
create_checkout_session (stands up a live payable page)	low-risk, executed	high-risk, held
stripe_api_write + PostPaymentIntents	low-risk, executed	high-risk, held
stripe_api_write + PostCheckoutSessions	low-risk, executed	high-risk, held
list_payment_intents, GetCheckoutSessions	low-risk	low-risk (unchanged)

create_payment_intent and create_checkout_session are the first two entries in this issue's list. The gate opened for both.

Why the dataset didn't catch it

Worth stating plainly, because it's the more useful finding. eval_dataset.py's labeling rule scoped risk to money moving out — so no charge-initiating write was ever a case in the 62. The two payment-link writes were labeled False, and the file's own docstring flagged that as the arguable call at the time. It was simply the wrong call: a payment link is the same live payment surface as a Checkout Session. Both are now True.

A ground-truth set built from one framing can only find bugs inside that framing. Six rounds of hardening against real operation ids, a live server, and two model classifiers all missed this, because all three were scored against a dataset that had already decided what "risky" meant. Reading this issue's own list of tools is what surfaced it — which is a point in favour of require_approval lists like the one in the issue body over any classifier, mine included.

The fix

_initiates_payment() requires a payment-surface stem (paymentintent, checkoutsession, paymentlink) and a creating verb (create_ on a named tool, Post on a dispatcher operation id), matched on a separator-stripped copy so both tool shapes collapse to the same stem. Stems alone would have swept in every read of those resources and made the gate unusable.

Re-scored: 62/62 still correct on the live-pulled cases, with only those two payment-link labels moved and no read affected. Plus REGRESSION_CASES — 10 cases built from the five tools this issue names, in both tool shapes, held separate from CASES because they're hand-written rather than live-pulled and that provenance claim should stay clean. 30 tests, ruff clean, CI green.

The model rows in the results table are left at their original values and marked as predating this fix rather than silently restated — they'd need re-scoring against live endpoints to be comparable.

Also: the audit trail now carries the Stripe object id

admit() writes its decision record before awaiting the real call, so the trail proved that a charge was authorized but never which charge. On ALLOW there's now a second record with the id and object Stripe returned, hash-chained immediately after the decision. That id is the only key joining a governance decision to the financial record a dispute or an audit reconciles against months later. An unparseable Stripe response costs that record's detail, never the caller's result.

One thing this deliberately does not do

The example policy in the issue body also asks for spend_limit: $500/day and rate_limit: 5/hour. Those are out of scope here and now stated as such in the README. Both need durable cross-call state — a spend ledger, a rolling window — that a per-call classifier has no business owning. ControlPolicy is the seam they'd attach to. They're also probably the control a Stripe user asks for first, so flagging the gap rather than implying the gate is complete.

On the proxy comparison

Worth noting one structural difference from a proxy-based approach, since it came up in this thread. Because this composes into ToolkitCore.run_tool() via MRO, admit() and the real Stripe call share a single call frame — the arguments the gate weighed are the arguments dispatched, by construction. A proxy sitting in front of the toolkit can't see what the runtime does after an approval returns, which is why that shape needs proposal/dispatch digest comparison to close the same hole. Different threat models: this defends against a misbehaving model, a proxy defends against a compromised process. Both are real; they aren't substitutes.

Still the same ask as the original post: whether this is a gap worth closing in-tree, and whether the risk line is drawn where you'd draw it.

[fede-kamel]

Closing to keep my open PR queue focused. Happy to reopen if there is interest in an admission-gate hook for run_tool().