Skip to content

feat: add Jev (TypeSafe System One) as an optional classifier backend - #48

Open
jauderho wants to merge 1 commit into
czottmann:mainfrom
jauderho:main
Open

jauderho wants to merge 1 commit into
czottmann:mainfrom
jauderho:main

Conversation

@jauderho

Copy link
Copy Markdown

What this is

A proposal, opened for feedback rather than as a finished change.

It adds TypeSafe's Jev as an additional backend for the tool_call classifier. Jev answers typed questions with calibrated probabilities instead of emitting decision JSON, so the allow/block decision moves out of prompt parsing and into code.

Nothing changes without configuration: autoMode.classifierProvider defaults to "pi" and that path is untouched.

How it works

  • State: the exact action, environment, loaded project instructions, and the classifier transcript. Rule lists stay out.
  • Questions: one Noul per hard_deny / soft_deny / allow rule, plus an explicit-authorization Noul and a severity Score. Jev evaluates them in parallel in one call, so the fan-out is close to free.
  • Combination: in code, mirroring CLASSIFIER_SYSTEM_PROMPT semantics. Probabilities in a [reviewThreshold, threshold) band are unresolved and escalate to structured review, or block in "jev" mode.

Only steps 13–14 of the enforcement order change. Permission rules, the deterministic hard-deny checks, path gates, and protected paths all still run first and are untouched.

Gating

Jev is inert unless three conditions hold: OMP ≥ 18.2.4 exports TypeSafeJudge, a typesafe credential resolves, and config opts in. A miss runs the existing classifier and reports why through /automode config and the status line — it never blocks. Stock Pi is unaffected; the runtime check is a feature detect, so a future Pi judgment API needs one branch.

Auth, transport, retries, and credential rotation are all OMP's (TypeSafeJudge + ctx.modelRegistry.resolver("typesafe", …)). No new runtime dependency, and this extension never sees the key.

Points I'd particularly like your view on

  • Not using OMP's resolveJudge(). It falls back to the tiny/smol chat chain on failure, which would silently demote the safety classifier to an unvalidated model. Instead jev.onFailure chooses between the user's own configured classifier and failing closed. Reasonable, or too clever?
  • "auto" vs "jev-prefilter" are currently synonyms. "auto" exists to express "the credential is the switch", but it may just be dead surface.
  • Default thresholds (docs/configuration.md) are principled but not measured against real traffic. They are documented as starting points. Happy to hold the defaults behind a stronger warning if you'd rather.
  • Jev's injection posture. TypeSafe documents that jev-1.13 does not treat state as hostile. That is why "jev-prefilter" is the recommended mode and why the caveat sits next to the standalone mode in the docs rather than being glossed.

Notes

  • 26 new tests in tests/jev.test.ts; tsc --noEmit clean.
  • I could not run npm test locally (no node on this machine) — the suite passes under bun's node:test runner, 324/326, with the one pre-existing permissions.test.ts backslash failure that also fails on a clean checkout there. CI is the real check.
  • ADR-003 records the design decisions.

Happy to split this up, change defaults, or drop parts of it. Feedback welcome.

Add TypeSafe Jev as an optional backend for the tool_call classifier.
The default provider stays "pi", so behavior does not change without
configuration.

Jev needs Oh My Pi 18.2.4 or newer and a TypeSafe credential. If either
is absent, auto mode runs the existing classifier and reports why.

- Ask one typed question per policy rule. Combine answers in code.
- Escalate an unresolved review band to structured review.
- Apply jev.onFailure after the gate: fall back, or fail closed.
- Show the gate state in the status line and /automode status.

Signed-off-by: Jauder Ho <jauderho@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant