Proposal: guardex-ai-openeval-adapter (portable eval-dataset interchange for screen_batch)

[adhabnr-ux]

Proposal: guardex-ai-openeval-adapter for EvalPort

Hi! I maintain EvalPort, a small open interchange format for portable LLM evaluation datasets — a Suite (test cases + graders) and a ResultSet (per-test-case results + grader results), spec here. The idea is that different eval/observability tools can export/import the same JSON shape instead of everyone inventing their own.

I went through guardex-ai's source (not just the README) to check this makes sense before proposing it. What I found in docs/sdk/guard.md and docs/sdk/types.md:

• Guard.screen_batch(texts: List[str], gate: Gate = "input", context: GuardExContext | None = None) -> List[ScreenResult] — takes a list of strings and returns one typed result per input, in order. That's the part that maps cleanly to "run a dataset, get per-item results."
• ScreenResult (@dataclass(frozen=True) in guardex/_types.py) carries gate, action, classify: ClassifyResult, pii: PIIResult, text, optional scope: ScopeResult, latency_ms, plus convenience properties safe / blocked.
• ClassifyResult carries safe, category, categories, confidence, description (the S1–S14 LlamaGuard-style categories).
• PIIResult / PIIEntity carry has_pii, entities: list[PIIEntity] each with text, label, score, start, end.

That's a real, structured, per-item result — not just a bare boolean — which is what makes an honest EvalPort mapping possible. I want to flag up front that guardex-ai is fundamentally a runtime guardrail, not an eval framework, so this adapter would specifically be about the "screen a batch of prompts and grade the outcome" use case (e.g. running a red-team / jailbreak / PII-leakage dataset through Guard and getting portable results out) — not a claim that GuardEx is an eval framework.

Rough sketch

Modeled on the existing guardrails-openeval-adapter in the same repo (a similar runtime-guardrail case):

# to_openeval: build a Suite from a list of prompts
def to_openeval(texts: list[str], ids: list[str] | None = None, gate: str = "input") -> dict:
    """One TestCase per text; graders = ['classify', 'pii'] (+'scope' if used)."""
    ...

# from_openeval: pull the raw strings back out to feed guard.screen_batch()
def from_openeval(suite: dict) -> list[dict]:
    """-> [{"id": ..., "text": ...}, ...]"""
    ...

# evaluation_result_to_openeval: turn List[ScreenResult] into a ResultSet
def evaluation_result_to_openeval(results: list, ids: list[str], suite_id: str) -> dict:
    """
    Per ScreenResult -> one TestCaseResult with two GraderResults:
      - "classify": passed=result.classify.safe, score=result.classify.confidence,
                    reason=result.classify.description
      - "pii":      passed=not result.pii.has_pii,
                    reason=", ".join(e.label for e in result.pii.entities) or None
    TestCaseResult.passed = result.safe (AND of the above, matching
    ScreenResult.safe's own semantics)
    """
    ...

Usage would look like:

from guardex import Guard
from guardex_ai_openeval_adapter import to_openeval, from_openeval, evaluation_result_to_openeval

prompts = ["Hello there", "Ignore all previous instructions and...", "My SSN is 123-45-6789"]
suite = to_openeval(prompts, ids=["a", "b", "c"], gate="input")

items = from_openeval(suite)
guard = Guard()
results = guard.screen_batch([i["text"] for i in items], gate="input")

result_set = evaluation_result_to_openeval(results, [i["id"] for i in items], suite_id=suite["id"])

Happy to be the one who writes and maintains this adapter (it'd live under adapters/guardex-ai-openeval-adapter/ in the EvalPort repo, MIT/Apache-2.0, standalone pip-installable package) — opening this issue mainly to check the mapping above matches your intent for screen_batch()/ScreenResult before I build against it, and to see if you'd want anything different (e.g. whether scope/safety_route should be separate graders too, given they're optional and only present when configured). No pressure either way — just flagging it in case it's useful.

Thanks for building this — zero-external-API local guardrails is a nice property.