Skip to content

Fix/108 stable fallback grading - #115

Merged
Irahan2 merged 5 commits into
mainfrom
fix/108-stable-fallback-grading
Sep 27, 2026
Merged

Irahan2 merged 5 commits into
mainfrom
fix/108-stable-fallback-grading

Conversation

@Irahan2

@Irahan2 Irahan2 commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

The full-text search returns the same rows for the same question but the grader's list over them moves between runs, and sometimes it drops the category row and the answer is "Nie wiem". Now a full-text row whose title holds the grader's anchor stays even if the list left it out (a context that just mentions it doesn't count), the grader can still add rows but can't drop those. A code like R2 doesn't pin anything since after #106 every row the search returns has it, so the R2 zero verdict from the issue still abstains. Same if the grader names the whole noun phrase as the entity, I kept the row side strict like in #109. benchmarks/run_grader_stability.py replays the grader N times over the same rows; they're rebuilt from the issues, so they should be refreshed with --capture against the real graph.

Fixes #108

The full-text search returns the same rows for the same question and the
grader's list over them varied run to run, down to an empty one on a
question the graph answers (#108). A row whose own title or context holds
the anchor is now kept whatever the list says. The grader can still add
rows, and without such a row its list decides alone, as before.

related doesn't count, and neither does an anchor that every row already
holds: a lone lowercase word, or a code, since the search requires every
code in the question (#106).
Runs grade_context repeatedly on one saved set of rows and reports what the
grader listed next to what the run kept, so a grading change can be
measured on identical input. --capture swaps the rebuilt rows for real ones
from Neo4j.
A failed call or an unreadable reply keeps every row, so a replay against a model that
is out of quota printed "1 distinct set(s), 0 empty", which reads as a perfectly stable
grader. Each case now counts its runs without a verdict, and the script exits non-zero
when no run got one. Also lists the new test file in CLAUDE.md's tree.

@omakszewska omakszewska left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the pin also applies when the grader's verdict was correct, not only when it's empty. union(held) adds every row whose context mentions the entity. For "Kto jest dziekanem Wydziału X?" the grader would keep only the dean, but after the pin the run also keeps every course, club and study programme that mentions that faculty.

The #108 failure is the empty verdict, and in your replay the category row alone already carried all the criteria in related. Would it be enough to pin on title only (FULLTEXT_OWN_TEXT_COLUMNS = ("title",)), or only when kept is empty? Both should still fix the "Nie wiem" case. Or is there a case where pinning on context is needed?

A context that mentions the entity is not the entity. For "Kto jest dziekanem
Wydzialu X?" every course, club and study programme at the faculty names it in its
context, so the pin overruled a grader that rightly kept only the dean. The title is
the entity's name, and the entity's own row still lists what sits under it in related,
so the #108 "Nie wiem" case is still fixed.

The pin still adds to a non-empty list: a list that drops the category row and keeps
one criterion would otherwise answer that criterion as the whole list.
@Irahan2

Irahan2 commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator Author

Yeah you're right, I tried the dean case and the pin brought back the faculty, the course, the club and the programme next to the dean. Went with title only in 1543e35, now it just adds the faculty node itself. I couldn't find a case that needs context, the category row's related already has the criteria.

I kept it as a union instead of only when kept is empty though. If the grader drops the category and keeps one criterion, only-when-empty wouldn't fire and the answer would list that one criterion like it's the whole list. With title only the category row comes back there too.
@omakszewska

@Dominik-Galus Dominik-Galus left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems ok

@Irahan2
Irahan2 added this pull request to the merge queue Sep 27, 2026
Merged via the queue into main with commit d368cf3 Sep 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The grader's verdict on the same fallback rows varies run to run

3 participants