Refs #406: Warn when live submission gate hits GitHub caps

[adliebe]

Summary

• Add explicit live-mode safety caps for GitHub PR and issue collection in scripts/submission_quality_gate.py.
• Warn when gh pr list or gh issue list returns exactly the cap, because similar-PR and bounty-discovery checks may be incomplete.
• Surface that warning in the evaluated checks as source_completeness so contributors see the uncertainty instead of treating the gate as fully sourced.

Refs #406.

Evidence

The live submission quality gate depends on bounded gh pr list and gh issue list calls. Before this change, hitting the limit looked the same as a complete collection, so a crowded repo could silently miss similar open PRs or bounty rows while still producing a normal-looking result.

This is distinct from the current API-focused submission-gate work in PR #478 and queue-health API-capacity work in PR #474. This change covers the GitHub collection boundary used by the live gate.

Live bounty preflight for internal bounty 66 / issue #406:

• API status: open
• awards remaining: 15
• active attempts: none

Open-PR duplicate search found no existing PR for gh pr list reached, gh issue list reached, similar-open-PR checks may be incomplete, or bounty discovery may be incomplete.

Validation

• .\.venv\Scripts\python.exe -m pytest tests\test_submission_quality_gate.py::test_submission_quality_gate_warns_when_live_collections_hit_safety_caps -q -> 1 passed
• .\.venv\Scripts\python.exe -m pytest tests\test_submission_quality_gate.py -q -> 24 passed
• .\.venv\Scripts\python.exe -m pytest -q -> 415 passed
• .\.venv\Scripts\python.exe -m ruff check . -> All checks passed
• .\.venv\Scripts\python.exe -m ruff format --check . -> 79 files already formatted
• .\.venv\Scripts\python.exe -m mypy app scripts\submission_quality_gate.py -> success, no issues found
• .\.venv\Scripts\python.exe scripts\docs_smoke.py -> docs smoke ok
• git diff --check -> clean
• changed-file secret scan -> no matches

Summary by CodeRabbit

• New Features

  • Submission quality gate now includes source completeness checks that warn when GitHub pull request or issue data collection reaches safety limits.

• Tests

  • Added test validating source completeness warnings when GitHub data collection reaches safety caps.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b1c33215-cdd4-44be-b6ae-143459f75921

📥 Commits

Reviewing files that changed from the base of the PR and between d8532d4 and c61f3fd.

📒 Files selected for processing (2)

• scripts/submission_quality_gate.py
• tests/test_submission_quality_gate.py

---

📝 Walkthrough

Walkthrough

The PR adds safety-cap constants for GitHub PR and issue list retrieval, wires them into the live-context loader to detect when fetched list sizes reach capacity, and surfaces those detection warnings through the submission evaluator as source_completeness checks. A new test validates the complete flow from mocked GitHub CLI responses through evaluation output.

Changes

Safety caps and source completeness warnings

Layer / File(s)	Summary
Safety cap constants and live-context detection scripts/submission_quality_gate.py	Module-level GH_PR_SAFETY_CAP and GH_ISSUE_SAFETY_CAP constants replace hardcoded limits in gh pr list and gh issue list calls. Conditional logic appends load warnings to load_warnings when the fetched PR or issue list sizes meet or exceed these caps.
Evaluation integration and test coverage scripts/submission_quality_gate.py, tests/test_submission_quality_gate.py	evaluate_submission reads data["load_warning"] and creates a source_completeness warning check when present. New test mocks GitHub CLI responses to hit both safety caps and validates that evaluation returns warn with source_completeness reporting both PR and issue cap triggers.

Possibly related PRs

• ramimbo/mergework#325: Established the live-mode submission quality gate evaluated by _load_live_context and evaluate_submission; this PR extends that gate with safety-cap detection and source-completeness warnings.

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically references the main change: adding a warning when the live submission gate hits GitHub API caps.
Description check	✅ Passed	The PR description is comprehensive, covering summary, evidence of the problem, related context, and validation results against the template checklist.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Mergework Public Artifact Hygiene	✅ Passed	PR is technical (safety caps for GitHub API). No investment, price, cash-out, payout claims, or security details in code or docs. MRWK correctly described as native ledger coin.
Bounty Pr Focus	✅ Passed	Diff matches stated files, adds safety caps, integrates load_warnings in live context, adds source_completeness check, includes test with full validations of the feature.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[GHX5T-SOL]

Reviewed current head c61f3fd7733e0e637d7ccf8468f96fd01b8db3a3.

The change is focused and matches the live-gate failure mode: the GitHub PR and issue list limits are now explicit safety caps, and evaluate_submission() surfaces cap hits as a source_completeness warning before the rest of the bounty checks run. The existing unavailable-GitHub path still returns the prior live-data warning, while the new regression covers both PR and issue cap hits plus the evaluated warning check.

Validation run locally:

• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --extra dev python -m pytest tests/test_submission_quality_gate.py::test_submission_quality_gate_warns_when_live_collections_hit_safety_caps -q -> 1 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --extra dev python -m pytest tests/test_submission_quality_gate.py -q -> 24 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --extra dev python -m pytest -q -> 415 passed
• uv run --extra dev ruff check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py -> passed
• uv run --extra dev ruff format --check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py -> 2 files already formatted
• uv run --extra dev python -m mypy app scripts/submission_quality_gate.py -> success, 31 source files
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 uv run --extra dev python scripts/docs_smoke.py -> docs smoke ok
• git diff --check origin/main...HEAD -> clean
• git diff --no-ext-diff origin/main..HEAD | gitleaks stdin --no-banner --redact --exit-code 1 -> no leaks found

[tolga-tom-nook]

Reviewed current head c61f3fd7733e0e637d7ccf8468f96fd01b8db3a3 for Bounty #447. No blockers found.

Evidence:

• Inspected scripts/submission_quality_gate.py and tests/test_submission_quality_gate.py.
• Verified the live-mode PR/issue collection limits are now explicit safety caps (101 / 201) so equality means the gh result may be truncated rather than complete.
• Verified evaluate_submission() surfaces load_warning as a source_completeness warning, so incomplete live context is visible in the gate output instead of silently passing duplicate/payability checks.
• Confirmed the new regression test exercises both capped PR and issue collections plus API bounty capacity/attempt loading.

Commands run locally:

• ./.venv/bin/python -m pytest tests/test_submission_quality_gate.py -q → 24 passed
• ./.venv/bin/python -m pytest -q → 415 passed
• ./.venv/bin/python -m ruff check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py → passed
• ./.venv/bin/python -m ruff format --check scripts/submission_quality_gate.py tests/test_submission_quality_gate.py → passed
• ./.venv/bin/python -m mypy scripts/submission_quality_gate.py → passed

Verdict: approve. The change is focused, preserves advisory degradation on live-data uncertainty, and adds regression coverage for the cap boundary that matters to the submission gate.

[IsaacOdeimor]

Reviewed PR #493 for the live submission gate cap-warning behavior.

Evidence checked:

• Inspected scripts/submission_quality_gate.py: introduces GH_PR_SAFETY_CAP = 101 and GH_ISSUE_SAFETY_CAP = 201, uses those caps in gh pr list / gh issue list, and appends load_warning messages when returned collections hit the cap.
• Verified evaluate_submission() surfaces load_warning as a source_completeness warning, so incomplete live-source coverage affects the final check output instead of being silent.
• Inspected tests/test_submission_quality_gate.py: the new test stubs PR and issue lists exactly at the safety caps, verifies both warning messages appear in data["load_warning"], and verifies the warning is included in result["checks"].

Local verification on the PR branch:

• python -m pytest tests/test_submission_quality_gate.py -q -> 24 passed.
• python -m py_compile scripts/submission_quality_gate.py -> passed.

No blocker found. The change makes capped live data visibly incomplete while preserving existing evaluation flow.