Repository navigation
Fix/108 stable fallback grading - #115
Conversation
The full-text search returns the same rows for the same question and the grader's list over them varied run to run, down to an empty one on a question the graph answers (#108). A row whose own title or context holds the anchor is now kept whatever the list says. The grader can still add rows, and without such a row its list decides alone, as before. related doesn't count, and neither does an anchor that every row already holds: a lone lowercase word, or a code, since the search requires every code in the question (#106).
Runs grade_context repeatedly on one saved set of rows and reports what the grader listed next to what the run kept, so a grading change can be measured on identical input. --capture swaps the rebuilt rows for real ones from Neo4j.
A failed call or an unreadable reply keeps every row, so a replay against a model that is out of quota printed "1 distinct set(s), 0 empty", which reads as a perfectly stable grader. Each case now counts its runs without a verdict, and the script exits non-zero when no run got one. Also lists the new test file in CLAUDE.md's tree.
omakszewska
left a comment
There was a problem hiding this comment.
I think the pin also applies when the grader's verdict was correct, not only when it's empty. union(held) adds every row whose context mentions the entity. For "Kto jest dziekanem Wydziału X?" the grader would keep only the dean, but after the pin the run also keeps every course, club and study programme that mentions that faculty.
The #108 failure is the empty verdict, and in your replay the category row alone already carried all the criteria in related. Would it be enough to pin on title only (FULLTEXT_OWN_TEXT_COLUMNS = ("title",)), or only when kept is empty? Both should still fix the "Nie wiem" case. Or is there a case where pinning on context is needed?
A context that mentions the entity is not the entity. For "Kto jest dziekanem Wydzialu X?" every course, club and study programme at the faculty names it in its context, so the pin overruled a grader that rightly kept only the dean. The title is the entity's name, and the entity's own row still lists what sits under it in related, so the #108 "Nie wiem" case is still fixed. The pin still adds to a non-empty list: a list that drops the category row and keeps one criterion would otherwise answer that criterion as the whole list.
|
Yeah you're right, I tried the dean case and the pin brought back the faculty, the course, the club and the programme next to the dean. Went with title only in 1543e35, now it just adds the faculty node itself. I couldn't find a case that needs context, the category row's related already has the criteria. I kept it as a union instead of only when kept is empty though. If the grader drops the category and keeps one criterion, only-when-empty wouldn't fire and the answer would list that one criterion like it's the whole list. With title only the category row comes back there too. |
The full-text search returns the same rows for the same question but the grader's list over them moves between runs, and sometimes it drops the category row and the answer is "Nie wiem". Now a full-text row whose title holds the grader's anchor stays even if the list left it out (a context that just mentions it doesn't count), the grader can still add rows but can't drop those. A code like R2 doesn't pin anything since after #106 every row the search returns has it, so the R2 zero verdict from the issue still abstains. Same if the grader names the whole noun phrase as the entity, I kept the row side strict like in #109. benchmarks/run_grader_stability.py replays the grader N times over the same rows; they're rebuilt from the issues, so they should be refreshed with --capture against the real graph.
Fixes #108