Repository navigation
Python: [Feature]: Document a third-party Python evaluator that preserves evidence metadata #8217
Description
Context: Agent Framework 1.17.0 and Hermes Rubric 1.2.1 on Python 3.10. Filing this before code or a PR, per the contribution guidance.
Description
Agent Framework already defines an experimental, provider-neutral Python Evaluator protocol and structured EvalResults, EvalItemResult, and EvalScoreResult types. I could not find a third-party example showing how to preserve additional provenance—evidence citations, assessed coverage, and a run receipt—without reducing the result to an unexplained number.
Proposed scope
A thin optional-dependency example or integration implementing the current protocol exactly:
async def evaluate(
self,
items: Sequence[EvalItem],
*,
eval_name: str,
) -> EvalResults:
...It would invoke Hermes Rubric through assess_async(..., rubric=caller_rubric, backend=caller_backend), then map:
- aggregate score to
EvalScoreResult.score; - per-dimension scores to
RubricScore; - evidence citations, coverage, limitations, and receipt to
EvalScoreResult.sampleorEvalItemResult.metadata, as maintainers prefer; - Hermes
AssessmentErrorto itemstatus="error"witherror_codeanderror_message, with no numeric fallback.
It would provide no default rubric, backend, threshold, or pass predicate. Hermes FeedbackPolicy, if supplied, would only generate deterministic post-assessment feedback; it would not be treated as a pass/fail policy.
Decision needed
EvalScoreResult.passed may be None, while documented EvalItemResult.status values are pass, fail, or error. Before implementation, should this adapter:
- require a caller-supplied pass predicate so item status is defined, or
- remain an external example until maintainers specify the supported non-gating mapping?
If wanted, should it live under python/samples or remain an external package referencing the protocol?
Tests
- pinned imports with Agent Framework 1.17.0 and Hermes Rubric 1.2.1 on Python 3.10;
- evidence, dimensions, coverage, and receipt preserved;
- partial coverage remains explicit;
- malformed/provider failures become explicit errors without numeric fallback;
- missing optional dependency fails clearly.
Limitations
This is one evaluator option, not a Foundry replacement or parity claim. Results depend on the caller’s rubric and backend and remain advisory. Python only; no adoption, certification, correctness, or safety claim.
I will not open a PR until a maintainer confirms the route and the required item-status mapping.
Metadata
Metadata
Type
Projects
- Status
Activity
[-][Feature]: Document a third-party Python evaluator that preserves evidence metadata[/-][+]Python: [Feature]: Document a third-party Python evaluator that preserves evidence metadata[/+]eavanvalkenburg commentedon Oct 27, 2026
/plan Investigate the supported contract and appropriate ownership route for this proposal before implementation.
Please assess current
main, rather than assuming the reported package versions still define the contract:Evaluatorcan preserve aggregate and dimension scores, evidence citations, coverage, limitations and receipts using existingEvalScoreResult.sampleandEvalItemResult.metadata. Identify any actual framework gap separately from vendor-specific mapping.passed=Noneis available, but documented item statuses are pass/fail/error. Compare requiring an explicit caller-supplied pass predicate with a genuinely provider-neutral non-gating result contract. Explain effects on item helpers, aggregate counts/rates, partial results and error handling; do not silently classify an ungated assessment as passed or invent an undocumented status.This requests investigation and a proposed plan, not acceptance of a vendor integration, a new core result status, or a particular sample location. Please identify decisions requiring explicit maintainer agreement before an implementing PR. No implementation or timing commitment is made.