Skip to content

[AGT-04] GitHubEventResolver — concrete event-to-claim resolution adapter#8519

Open
an0mium wants to merge 6 commits into
mainfrom
vision-incubator/agt-04-github-event-resolver
Open

[AGT-04] GitHubEventResolver — concrete event-to-claim resolution adapter#8519
an0mium wants to merge 6 commits into
mainfrom
vision-incubator/agt-04-github-event-resolver

Conversation

@an0mium

@an0mium an0mium commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Slice

Adds GitHubEventResolver — the concrete resolution adapter that converts pre-fetched GitHub webhook event payloads (PR merge, issue close, CI check/workflow) into StakeableClaim resolutions. This is sub-deliverable 2 of AGT-04; sub-deliverable 1 (StakeableClaim schema and GithubResolutionAdapterStub) is already on main.

Gating

What this adds

Symbol Purpose
GitHubEventPayload (frozen dataclass) Minimal webhook event representation: event_type, action, target_ref, merged, conclusion, raw
ResolutionResult (frozen dataclass) Outcome of a resolution attempt: claim_id, resolved, resolution_value, evidence
GitHubEventResolver can_resolve(claim, event) → bool + resolve_from_event(claim, event) → ResolutionResult; dispatches to per-type resolvers for PR_MERGE, ISSUE_CLOSE, CI_PASS

All logic is deterministic; no live GitHub API calls. The adapter consumes pre-fetched payloads, making it testable offline and safe to compose with the existing InMemoryStakeableClaimStore.

Tests

  • TestCanResolve (7 tests) — matching/mismatching event types, target refs, DEPENDENCY_RELEASE unsupported type
  • TestFlagGate (2 tests) — resolve_from_event raises when flag off; can_resolve is flag-free
  • TestPRMergeResolution (5 tests) — merged→YES, closed-no-merge→NO, non-terminal action, already-resolved skip, target-ref mismatch
  • TestIssueCloseResolution (3 tests) — closed→YES, reopened/labeled non-terminal
  • TestCIPassResolution (5 tests) — success/failure/cancelled check_run, success workflow_run, queued non-terminal
  • TestResolverWithStore (2 tests) — end-to-end roundtrips through InMemoryStakeableClaimStore

Validation

  • pytest tests/prediction/test_github_event_resolver.py --noconftest24 passed
  • Syntax: all three files parse without errors (ast.parse)
  • Ruff / mypy: no new dependencies introduced; frozen dataclasses with explicit type annotations

Out of scope

  • Live GitHub API polling (sub-deliverable 3; will use PyGithub or REST client behind an adapter interface)
  • Durable JSONL/PostgreSQL claim store (covered by agt-04-jsonl-claim-store branch)
  • AGT-05 reputation wiring — ResolutionResult is the feed point; wiring waits for AGT-05

Generated by Claude Code

Concrete resolution adapter that turns pre-fetched GitHub event payloads
(PR merge, issue close, CI pass/fail) into StakeableClaim resolutions.
No live GitHub API calls; all logic is deterministic and testable offline.

Advances issue #6065 (AGT-04), sub-deliverable 2 — GitHub event resolution.
Flag-gated: ARAGORA_PREDICTION_MARKETS_ENABLED (default OFF).

- aragora/prediction/github_event_resolver.py: GitHubEventPayload dataclass,
  ResolutionResult dataclass, GitHubEventResolver with can_resolve() and
  resolve_from_event() dispatching to per-type resolvers
- tests/prediction/test_github_event_resolver.py: 19 tests covering flag
  gate, can_resolve semantics, all three question types, non-terminal
  actions, already-resolved skip, and a full roundtrip with InMemoryStore
- aragora/prediction/__init__.py: export new symbols
@an0mium an0mium added the vision-layer label Jun 20, 2026 — with Claude
CI lint check reported 1 file would be reformatted; apply it now.
@github-actions

Copy link
Copy Markdown
Contributor

Aragora Code Review

Advisory-only review. No issues found.

scarmani and others added 2 commits June 30, 2026 11:13
prediction module now has py_files=3, test_file_count=2 after adding
github_event_resolver.py and its test file. Also picks up concurrent
main-branch drift (evaluation moved to integrated, metrics added to
deprecated, Python file count 4219→4251, test counts updated).
@scarmani

Copy link
Copy Markdown
Collaborator

Claude independent model review

Reviewer: claude (anthropic) — independent adversarial model review via the Aragora Claude reviewer, grounded on the exact PR head.
Head: 3216f48 (3216f48), committed 2026-06-30T16:19:14Z.
PR: #8519.
Model family: claude

Verdict: PASS

  • [P3] aragora/prediction/github_event_resolver.py _resolve_ci_pass (~L191-206): a single check_run with conclusion="success" resolves the whole CI_PASS claim as pass. A repo has many check_runs; one passing check ≠ all CI green. Correctness depends entirely on the caller pre-aggregating and setting target_ref granularity — worth a docstring caveat, since check_run and workflow_run have very different aggregation semantics yet share one resolver.
  • [P3] github_event_resolver.py resolve_from_event (~L96): no claim.expiry check — an event arriving after expiry still resolves an expired market. If expiry isn't enforced elsewhere, this lets stale events settle closed markets.
  • [P3] github_event_resolver.py _resolve_issue_close (~L178-189): any issues/closed event yields resolution_value=True, ignoring state_reason (not_planned). For a claim meaning "issue gets fixed," a not-planned close is a false-positive YES.
  • [P3] docs/METRICS.md: trailing whitespace appended after "...regardless of threshold." (L52), and the "Unique permission strings" command now renders \\1 (was \1) inside a code span — copy-pasting the doc'd sed gives a broken backreference. Both look like regeneration artifacts; harmless but cosmetic regressions in a generated file.

Note (non-finding): the metrics deprecated entry sets test_file_count: 0 despite tests/metrics/ containing ~10 files, but this matches the existing operations deprecated convention (also 0), so it appears intentional rather than a defect.

dogfood: yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants