Repository navigation
feat(renderer): --verify-model for a per-pass reflection model #2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
7a0f3e2
c31a293
4538348
a294437
224acdd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,7 +16,7 @@ | |
| base URL that render.py used (release-only distribution model). | ||
|
|
||
| Placeholders (all __UPPER__): | ||
| __PROVIDER__ __GOOSE_MODEL__ __LANGUAGE__ __API_KEY_NAME__ | ||
| __PROVIDER__ __GOOSE_MODEL__ __GOOSE_VERIFY_MODEL__ __LANGUAGE__ __API_KEY_NAME__ | ||
| __VERIFY_PROFILE__ __VERIFY_MODE__ (verification gate, on/shadow only) | ||
| __STYLE_ASSET__ (asset basename, e.g. instructions.graded.md; the | ||
| rendered config downloads it at CI runtime and substitutes | ||
|
|
@@ -28,7 +28,9 @@ | |
| off the #[verify:begin]/#[verify:end] marked region is REMOVED | ||
| The reflection instructions (instructions.reflection.md asset) are | ||
| downloaded at CI runtime with the same pattern as inline_threads.py, | ||
| so a single model/provider/config serves both passes. | ||
| so a single model/provider/config serves both passes unless | ||
| --verify-model overrides __GOOSE_VERIFY_MODEL__ (the reflection | ||
| pass rewrites the config before its first goose invocation). | ||
| """ | ||
| import argparse | ||
| import os | ||
|
|
@@ -191,6 +193,11 @@ def main(): | |
| ap.add_argument("platform", choices=list(SOURCES)) | ||
| ap.add_argument("--provider", required=True) | ||
| ap.add_argument("--model", required=True) | ||
| ap.add_argument("--verify-model", default=None, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 SUGGESTION —
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied — a294437: 두 층위에서 반영했습니다. ① argparse help에 "With --verification off the value is unused" 추가, ② 렌더러가 github + |
||
| help="model for the verification (reflection) pass; " | ||
| "defaults to --model (single-model behavior). " | ||
| "GitHub workflows only — rejected elsewhere. " | ||
| "With --verification off the value is unused") | ||
| ap.add_argument("--style", required=True, choices=["graded-review", "changes-summary"]) | ||
| ap.add_argument("--language", default="Korean") | ||
| ap.add_argument("--local", action="store_true", | ||
|
|
@@ -210,6 +217,36 @@ def main(): | |
| print(f"FAIL: unknown provider {args.provider}") | ||
| return 2 | ||
|
|
||
| if args.verify_model and args.platform != "github": | ||
| # The __GOOSE_VERIFY_MODEL__ switch only exists in the GitHub workflow | ||
| # template; on other platforms the flag would silently no-op and the | ||
| # reflection pass would keep the review model. Fail loud instead. | ||
| print("FAIL: --verify-model is only supported on the github platform") | ||
| return 2 | ||
| if args.verify_model and args.verification == "off": | ||
| # --verification off strips the whole verify region including the | ||
| # model switch, so --verify-model would be dead input. Warn (not | ||
| # fail): the render is still valid, the user just gave a no-op value. | ||
| print("WARN: --verify-model has no effect with --verification off") | ||
|
Comment on lines
+220
to
+230
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 WARNING —
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied — 224acdd (WARN 휴리스틱 채택): 제안대로 prefix 기반 경고를 render 시점에 추가했습니다. |
||
| # Cheap prefix heuristic: warn when the verify model id looks like it | ||
| # belongs to a different provider than the one configured. The rewritten | ||
| # config keeps the REVIEW provider, so a mismatched model id fails at | ||
| # runtime — and the reflection gate fails open (review still posts). | ||
| # This is a heuristic, not a catalog check: unknown prefixes pass. | ||
| if args.verify_model and args.verify_model != args.model: | ||
| _pfx = {"fireworks-ai": "accounts/fireworks/models/", | ||
| "openai": ["gpt-", "o1", "o3", "chatgpt"], | ||
| "anthropic": ["claude-"], | ||
| "openrouter": [], | ||
| "ollama_cloud": []} | ||
| _own = _pfx.get(provider, []) | ||
| _own_list = _own if isinstance(_own, list) else [_own] | ||
| if _own_list and not any(args.verify_model.startswith(p) for p in _own_list): | ||
| _foreign = [name for name, pfx in _pfx.items() | ||
| if name != provider and any(args.verify_model.startswith(p) for p in (pfx if isinstance(pfx, list) else [pfx]))] | ||
|
Comment on lines
+236
to
+246
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 WARNING — |
||
| if _foreign: | ||
| print(f"WARN: --verify-model looks like a {_foreign[0]} model but the config keeps provider {provider} — the reflection pass will fail auth if the provider does not serve it") | ||
|
|
||
| spec = SOURCES[args.platform] | ||
| if args.local: | ||
| # Development mode: render straight from the working-tree templates | ||
|
|
@@ -239,6 +276,7 @@ def main(): | |
| ("__RECIPES_BASE__", release_base()), | ||
| ("__PROVIDER__", provider), | ||
| ("__GOOSE_MODEL__", args.model), | ||
| ("__GOOSE_VERIFY_MODEL__", args.verify_model or args.model), | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 WARNING —
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied — 4538348 (guard 방식 채택, 템플릿 포팅은 보류):
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 SUGGESTION —
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied — a294437 (문서화): 재작성된 config가 리뷰 provider를 유지하므로 "동일 provider의 모델만 지정" 요건을 레시피 Step 4 가이드에 명시했습니다. 프로바이더 간 모델 검증은 렌더러가 모델 카탈로그를 갖고 있지 않아 정적 검증이 불가능하므로, 사용자 오류 시 런타임 실패 + 문서화로 대응하는 게 적절하다고 판단했습니다. |
||
| ("__API_KEY_NAME__", API_KEY_NAMES[provider]), | ||
| ("__VERIFY_PROFILE__", args.verify_profile), | ||
| ("__VERIFY_MODE__", verify_mode), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -169,6 +169,15 @@ jobs: | |
| python3 verify_findings.py hunks --diff pr.diff --findings findings.json --out hunks.txt | ||
| cat hunks.txt | ||
| } >> reflection_input.txt | ||
| # Verify-model switch (render.py --verify-model): swap ONLY the | ||
| # GOOSE_MODEL line for the reflection + retry invocations. Runs | ||
| # only here — findings exist. (A head-move after this point is | ||
| # tolerated: the post step's live guard skips a stale review.) | ||
| # When --verify-model is unset the substituted value equals the | ||
| # review model and this sed is a value no-op. sed (not a full | ||
| # rewrite) so keys added to Configure goose by future releases | ||
| # survive into the reflection pass. | ||
| sed -i.bak 's|^GOOSE_MODEL: .*|GOOSE_MODEL: __GOOSE_VERIFY_MODEL__|' ~/.config/goose/config.yaml && rm -f ~/.config/goose/config.yaml.bak | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 WARNING — There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟢 SUGGESTION — |
||
| goose run --instructions reflection_input.txt > reflect_raw.txt | ||
| if python3 verify_findings.py reflect-parse --raw reflect_raw.txt --out scores.json; then | ||
| python3 verify_findings.py merge --body body.md --scores scores.json \ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 WARNING —
codegoose-setup.yaml:172[P2] 레시피가 Step 2 명령에--verify-model {{ goose_verify_model }}을 추가하라고 지시하는데, 이 지시가 플랫폼 구분 없이 공통으로 기술되어 있다. 그러나scripts/render.py:219는--verify-model을 GitHub에서만 허용하고 나머지(GitLab/Gitea/TeamCity)에서는return 2로 실패시킨다. 따라서 해당 플랫폼을 선택하면서 검증 모델을 입력한 사용자는 레시피 지시를 그대로 따르다가 렌더 단계에서 반드시 실패한다. 실패는 명시적(소리 내며 fail)이라 조용한 오류는 아니지만, 레시피 프롬프트는 "GitHub에서만 유효"라는 비고를 추가해 비-GitHub 사용자의 동작 경로를 문서화할 필요가 있다(레시피는 렌더러 제약의 Source of Truth여야 하므로).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Applied — a294437: Step 4 가이드에 GITHUB ONLY 제약을 명시했습니다 — 비-github 플랫폼에서 검증 모델이 설정되면 렌더러가 exit 2로 거부함을 레시피가 먼저 알려주고, 행동 경로도 제시합니다("제약을 보고하고 플래그 없이 재실행"). 레시피가 렌더러 제약의 source of truth 역할을 하도록 정렬했습니다.