Skip to content

fix(persona): 逐命题核对收录资格、频率强度不得升级、单句截断按焦点拆分 (PE-23) - #141

Open
leafliber wants to merge 8 commits into
mainfrom
fix/persona-claim-eligibility
Open

leafliber wants to merge 8 commits into
mainfrom
fix/persona-claim-eligibility

Conversation

@leafliber

@leafliber leafliber commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

PE-23:persona 生成与检查的三类通用修复。只改提示词和确定性检查,没有调用真实模型;真实评测和判分不在本 PR 内。之后的提交按规划者五轮审查意见修改。

提示词

  • persona_check_v37(SHA-256 0e33db0b7676d971ef3d4a2940edc1e13a3a9aca26e01b89929d030f349091b4,18699 字节):逐命题收录资格与 claim_checks;频率强度并入 quantifier_checks(source_instances、强度序、只认原话里被肯定的频率词、只有实例时写成有计数的有界经历)。比 v36 多约 2.6 KB;source_quantifier 要求只填一个词,长依据句的材料字节余量相应减少(按决定不改,评测时观察)。
  • persona_generate_v24(SHA-256 eab1d23d1b9c283da2b1d6a2a20037c36814fe343442ac53f7cf4dc9510ffbcc):保留当次认识时省去随附的未来打算;不把有限次数写成“都/总是/每次”。
  • 两份提示词都已去掉贴近公开 dev 样本的措辞。旧提示词文件全部保留。

1. 逐命题收录资格

  • PLAN_MARKERS(将来、打算、承诺的时间词和动词,封闭语法类,同 整理:保留限定词并精确校验结构化主体引用(含 GLM 评测) #107 先例):以后 今后 往后 日后 将来 下次 下回 下一次 下一回 改天 届时 到时 到时候 将要 将会 打算 准备 计划 预备 决定 承诺 保证 答应 发誓 立志 争取 尽量 力争 务必。单字“会、要”和希望、愿意这类词不作标记。正反问句(打不打算)不算标记。
  • 只有请求里带 plan_markers 的焦点必须返回 claim_checks;模型多返回的不带标记的分句不要求。缺失、格式不符或漏项先重问一次,仍缺按不合格处理。
  • plan 一律不合格;任一项不合格,整句走既有修复/删句路径,不被其他命题覆盖。负结论也进 _adverse_field_verdicts。

2. 频率与断言强度

  • FREQUENCY_STRENGTH:偶尔<有时<经常<通常/习惯<总是/都/每次/从不,五级封闭词表,含每天、天天、每晚、每周、每月、每年、全都;单字“常、总”和“一般”有歧义,不加。与 SCOPED_QUANTIFIERS 共用 quantifiers/quantifier_checks,频率项另含 source_instances;非频率量词多给的这个字段忽略。
  • 候选与来源共用 _focus_quantifiers:被否定的频率词(不经常、不是每次、从没……每次)不算。
  • “都/全都”一律按频率/全称核对(宁可多删句,不漏拦),只有三种写法按分配义豁免:
    • 紧挨在“都”前(最多隔一个副词)的复数人称代词或人群词:我们、你们、他们、她们、它们、咱们、大家、大伙、各位、俩、两人、两个人、几个人。
    • 紧挨在“都”前的固定形状:[这/那]+两个以上或“几”+(个人|人或物量词本、门、位、件、首、张、篇、份、部、只、条、样+至多两字名词,前面可有一个新/老/小/大/好/旧)+至多一个人称代词+至多一个副词,中间不允许其他字(“两个人独处时”“两首歌时”“这两本书里”都照常核对)。“一”不算,分句里有“每”也不算。
    • 紧挨在“都”前的“X和/跟/与/同Y”,X、Y 都是人称代词(含“对方”)或本次材料中主体的名字/别名(只供代码使用,不发给模型)。
    • 其余(这些天、每个晚上、这些歌、每个人、任何人、“接龙和私聊”“平时和周末”“跟人吵架”等)都照常核对。紧贴另一个量词(每次都)或在同一分句里跟着已计数的全称词(每次排练我都、所有事情都)的“都”是关联词,不另计。
  • source_quantifier 只能是词表中的一个明确频率词,来源里单独的“都”(这些歌我都听过、每个人我都问了)不算依据;填整段或多个词时按结构错误重问一次,仍不合格按无依据处理。频率依据只取原话;当前正文只能收窄,没有摘录才用正文。修复替换句复查时同样核对。
  • 来源原话肯定了频率词:候选不能更强。否则只有“都”可以写成有界经历,焦点里每个被肯定的“都”都要有自己紧挨着的计数(“这次我都明白了自己从来都太急”的第二个“都”照常核对):紧挨在“都”前(只隔主语代词或副词)的“这/那+数词或几+次/回/场/遍/趟”(省去数词的“那次”按一次计),同一分句不能有自从、从、之后、以后、过后、往后、后、起、以来、开始,次数不超过 source_instances。总是、每次、一直等其他频率词没有来源频率词时一律不通过,任何计数都不能代替;“那次直播里我一直耐心解释”也因此被拦(已接受的保守取舍)。
  • source_quantifier 不合格触发重问时,同一焦点其他量词项已算出的不利结论随重问保留。

3. 截断后的恢复

  • _check_focus_split:只剩一句的窗口截断时按焦点二分,子请求带完整句子与全部绑定来源,直到单焦点;合并后缺焦点结论不能通过。子请求截止时间扣掉 60 秒保留时间;任一子请求失败(超时及其超时重试、429、传输错误、仍截断)都退回原来的截断,按终端截断/保留当前版处理,不抛子请求的错误。
  • 恢复前先估算:恢复用时(重试按一次、拆分按焦点数,各以被截断请求的用时计)+之后仍要运行的窗口数(含程度判定,以及同一窗口按句或按焦点拆分后还没跑的兄弟子批次)× 本阶段已见最长窗口用时+60 秒保留,超过剩余时间就不恢复,直接按终端截断。手写审查的重试另算其余审查页和随后检查的估计窗口数。
  • 按句拆分的子批失败(超时、429、传输错误、仍截断)也退回原来的截断,按终端截断处理,不抛子批的错误。
  • 所有回退只针对超时、429、传输错误、仍截断、无法解析,以及这一个请求自身的 item_error、content_rejection;配置、密钥、账户错误,用途暂停、每日 token 上限、维护调用额度照原路径处理(尝试记为跳过或失败,不变成被拒候选)。
  • 不能再拆的请求若截断回复为空、成段空白或未闭合 JSON,原样重试一次:
    • 不占超时重试额度,自身截止时间不超过阶段截止时间减 60 秒保留,预算条件同上。
    • 超时、429、传输错误、仍截断或仍无法解析时,一律回退到原来的截断结果,走原有的终端截断/保留当前版路径,不变成阶段失败。程度判定和手写审查同样处理。
  • 计数:
    • 保持 truncations、splits、split_results、terminal_truncations 和 retained_incomplete_checkpoints_v2 的口径。
    • 新增 recovered_truncations、focus_splits、focus_split_results、output_retries、output_retry_results。
    • 被截断的窗口(事实窗口、程度判定或手写审查页)只要最终检查完整,不论经重试、结构重问还是拆分,其中的截断都记为已恢复;只有最终仍不完整才计终端截断和不完整检查点。
    • 手写审查有自己的 output_limit_diagnostics,并计入评测汇总。

评测代码(报告计数部分)

persona_evaluation.py 只改计数:

  • _check_output_limit_metrics 汇总新字段(含手写审查和 output_retry_errors)。
  • _model_call_outcomes 按窗口是否最终完整判断截断调用是否已恢复(旧记录仍认按句拆分的恢复)。
  • 因超时失败的重试留下的那条超时调用,在保留当前版的不完整检查点里记为“容忍”而非“已恢复”:每条截断记录带调用顺序号(truncation_events,跨阶段按顺序号去重),与检查点里按顺序排列的截断调用一一对应;只有记录写明重试因超时失败的那条截断,其紧随的同用途超时调用才被容忍,每个失败重试至多一条,按焦点拆分因超时失败时,只容忍记录在该次拆分失败子请求名下的那几次超时调用(含它自己的超时重试;超时与截断共用调用顺序号),只在它与下一条截断调用之间找;按句拆分的兄弟子批超时不容忍。数目对不上时一条都不容忍,仍占每轮不完整额度;calls 新增 tolerated_timed_out_calls。其他超时口径不变(例如拆分子请求的普通超时即使靠超时重试恢复,在不完整检查点里仍判无效)。

已知限制

  • 多句批次里单行的字段重问或补查被截断时,仍直接记终端截断,不先拆成单句(按规划者决定本次不改,偏保守)。
  • 计划标记收窄后,“说会先……”这类句子不强制逐命题核对,依赖模型整体判断。
  • source_instances 仍采信模型填写。
  • 有界计数要求“这/那”;不带的裸计数不算有界。
  • “X和Y都”只认代词(含“对方”)和材料中主体的名字/别名;其他并列一律按频率词,偏保守。
  • 首个窗口的恢复预算以被截断请求的用时估计后续窗口。
  • 失败重试因 429 或传输错误留下的调用不在容忍范围内,仍按现有口径计入无效调用。
  • “所有/全部+场合名词+都”只走比较范围核对(“所有”的范围),不另按频率强度核对。
  • 拆分超时容忍的 tolerated_as 标注按顺序关联,可能对不上具体是哪一次调用(容忍的条数是对的)。
  • persona_check_v37 已 18699 字节,长依据句的单次请求离 32768 字节上限更近。
  • tests/test_persona_streaming.py::test_future_deadline_closes_stream_worker_and_preserves_shared_client 时间敏感:在未改动的 origin/main 上也会失败,本机时过时不过;与本 PR 无关,由规划者另开任务。

不变

手写审查规则、发布流程、DEFAULT_PUBLISH_MODE=all_manual、评分说明和评测语料都没有改;没有新增模型用途。

测试

新增 test_persona_claim_eligibility.py、test_persona_frequency_strength.py、test_persona_focus_split.py,每条修改都有通过和拦截两个方向。persona_fakes 的协议假模型补齐 claim_checks 与 source_instances(仍不判断资格或强度);几处既有测试按新版本号、按焦点拆分的调用次数与新计数字段更新。

本次(0025f86)结果:3.13 6118 passed、1 failed;3.12(--locked --isolated)6118 passed、1 failed;两边失败的都是上面那个 streaming 测试。uv build 通过。

- Claim eligibility: check v37 asks each substantive claim for its own
  eligibility; a closed class of future/intention/commitment markers
  (PLAN_MARKERS) obliges focus-level claim_checks. Code binds the verdicts:
  any ineligible claim or any plan rejects the sentence through the existing
  repair/delete path; missing or malformed verdicts count as ineligible.
- Frequency strength: FREQUENCY_STRENGTH ranks a closed class of frequency
  and universal words (偶尔<有时<经常<通常/习惯<总是/都/每次/从不) inside the
  existing quantifier_checks contract, plus source_instances. A candidate
  may not outrank its source word; instances alone support only a bounded
  experience (那两次……). Hedges never lower a rank.
- Output limits: a truncated one-sentence window is split by focus with the
  whole sentence and sources kept; an unsplittable request whose truncated
  reply is an unclosed JSON object or blank padding is retried once while
  the stage can still hold it. Truncation records note their handling; new
  focus_splits/output_retries counts reach the evaluation report.
- Prompts: persona_check_v37, persona_generate_v24; older files kept.
- Retry: an empty, blank or unclosed truncated reply of an unsplittable
  request is asked once more, without the timeout allowance and with a
  deadline that leaves a 60 s stage reserve. Any failure of the retry
  restores the original truncation (terminal / current kept), never a stage
  failure; degree and handwritten review included. A focus split needs the
  same room in the stage, else the truncation is terminal.
- Counting: truncations of a window that ends complete (retry, structure
  reask or split) are recovered; only an incomplete window counts terminal.
  Handwritten review gets its own diagnostics, summed in the report.
- Frequency: candidate and source share one reading; negated words and a
  distributive 都 (adjacent quantifier, plural pronoun, bounded set) are no
  claim or evidence. Evidence comes from utterances only. A bounded
  experience needs 这/那 + count + 次/回/场/遍/天 before the word, not opening
  a period, within source_instances. Adds 每天 天天 每晚 每周 每月 每年 全都.
- Claims: PLAN_MARKERS drop 会 要 想要 希望 但愿 愿意; only marked requested
  foci owe claim_checks; a missing or incomplete contract is asked again
  once, then rejected.
- Prompts v37/v24 edited in place (unmerged, never run formally).
- Recovery budget: a retry or focus split runs only if its estimate (one
  request, or one per focus, as long as the truncated one) plus the windows
  still to run (each as the longest seen in the stage, degree included; the
  handwritten review also counts its remaining pages and the check after it)
  plus the 60 s reserve fits. Halves end before the reserve; any failed half
  (timeouts after their retry, 429, transport, truncation) restores the
  original truncation as terminal instead of raising.
- 都: read from everything before it in its clause. Sets of times or
  occasions keep it a frequency claim; a universal phrase (每次排练, 每个人,
  任何人) takes it as its correlative; bounded sets of people or things,
  including a coordination with names or pronouns right before it, make it
  distributive; 一样 is no numeral and classifier.
- Bounded counts: 那次 counts one; any period opener between the count and the
  word unbounds it.
- source_quantifier must be one closed-class word: otherwise asked again once,
  then read as no frequency evidence.
- Evaluation: a failed output retry's own timeout is tolerated (not recovered)
  at a retained checkpoint, only as the next same-purpose call after the
  truncation and one per failed retry; retry errors are reported.
- Prompt check v37: source_quantifier names one word.
- 都/全都 is a frequency or universal claim except in three closed forms:
  a plural pronoun or group word right before it; a numeral (or 这/那 +
  numeral) with a closed person-or-thing classifier before it in its clause;
  X 和/跟/与/同 Y right before it, X and Y pronouns or names/aliases of subjects
  in this material (code-only, not sent to the checker). The occasion, set and
  text-coordination heuristics are removed.
- Only 都 can be bounded: 这/那 + count + 次/回/场/遍/趟 right before it, only a
  subject pronoun or adverb between, no period word anywhere in the clause.
  总是/每次/一直 and the rest always need a source frequency word.
- A recovery leaves room for unrun sibling sub-batches of a split; a failed
  sentence-split child restores the original truncation as terminal.
- A reask for one bad source word keeps the other items' adverse verdicts.
- Evaluation: truncation records carry their call order; only the call right
  after the truncation whose record says its retry failed by timeout is
  tolerated, records deduplicated across phases, none if counts disagree.
- Prompts v37/v24 state the narrower bounded rule.
Counted groups need two or more and no each-clause; each 都 needs its own
count; a bare source 都 is no evidence; pauses are never absorbed by a
fallback; a failed focus split's own timeouts are tolerated by record;
subject names never reach a model request.
- A counted group is one fixed shape right before 都: 这/那? + two or more (or
  几) + 个人, or a person/thing classifier with a noun of at most two
  characters (one adjective may precede it, no time, place or action
  character), then at most one pronoun and one adverb. Anything else between
  the count and 都 makes it a frequency claim.
- item_error and content_rejection of the one request are contained by a retry
  or split fallback; configuration, key, account, pause, usage limit and call
  budget errors keep their own path.
- Check prompt v37: a bare 都 of the source is no frequency evidence.
- COUNTED_GROUP: a count right after 第 or another numeral is no group, so an
  ordinal (第二本书我都) stays a frequency claim.
- A bare 每 of the source with a later 都 in its clause (每遇到新朋友我都先问)
  is explicit universal evidence; 每 is a legal source_quantifier and asks for
  no reask. A lone 都, a 每 without its 都 or a negated 每 is still none.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant