Agent Observer · 巡天智能体 — competition platform (challenge v3)

Event website and evaluation backend for the GOSIM "Agent Observer" hackathon. The site is a static Vue 3 app on GitHub Pages; everything stateful lives in one Supabase project; participant agents are executed by a Python worker (GitHub Actions by default) against the challenge v3 environment delivered by the science team (example3: solar observing calendar, tile geometry with lunar quality, directional weather with hidden events and uncertain forecasts, temporary observation requests, challenge-score-v3, JSON-Lines agent protocol with one global wall clock).

web/                 Vue 3 + Vite + Tailwind 4 + supabase-js site (GitHub Pages)
supabase/migrations  Postgres schema, RLS, RPCs, storage policies (v3 columns in 20260910001000_challenge_v3.sql)
supabase/functions   leaderboard (public JSON); scoring runs in the worker
challenge/           the vendored v3 environment (unchanged modules, relative imports) + scenario_builder + replay generator + reference scenario + tests
worker/              evaluation worker: sandboxed agent runs (challenge_runner.py), scoring, replay upload, scenario seeding/admin CLI
starter_kit/         what participants download: agent/, challenge/ copy, dev-reference + demo-week scenarios, run_baseline / run_demo_week launchers, local_runner, make_scenario, pack_agent, sac_submit, SKILL.md
tests/               pytest: runner sandbox, starter kit, platform integration on an embedded Postgres + real PostgREST harness; hosted_smoke.py for the live project
docs/                competition-format.md (which submission route each phase uses and where it is implemented), example3-analysis-brief.md (engineering brief on the v3 package)
legacy/fastapi/      first self-hosted version (reference only)

Competition mechanics (as implemented)

Topic	Behaviour
Scenario	directory config/*.json + outputs/reference/*.csv; stored in bucket scenarios/<slug>/...; per-file visibility flags (weather_public, forecasts_public, events_public); global_wallclock_seconds per scenario
Practice	scenarios fully public (incl. weather_events.csv) so local scoring reproduces the platform; results files (decisions.csv) and agent packages both accepted
Online competition	hidden weather/forecasts/events; agent packages only; the platform runs the agent through participant-agent-protocol-v1 with the scenario's global wall clock; score = mean over the phase's scenarios
Agent package	zip with minimal_agent.py/agent.py/main.py at the root, optional requirements.txt (installed into a per-run venv), optional .env (model keys; loaded into the agent environment only, never logged); Python 3.12; network allowed (LLM APIs)
Isolation	agent runs in its own directory with a scrubbed environment, rlimits, process-group kill at the cutoff; the scenario directory is never mounted; docker mode (SAC_SANDBOX_MODE=docker) adds a read-only container
Scoring	challenge/scoring_core.py (unchanged from the science team) re-scores the committed decisions.csv; report v3 with breakdown, per-segment audit and input checksums
Outcomes	survey_complete, global_wallclock_expired, agent_error, agent_initialization_error are all scored on what was committed plus terminal penalties (package semantics); only unreadable packages/files are invalid
Artifacts	per evaluation: report.json, decisions.csv, agent.log, workflow_result.json, decision_replay.html (organizer-style replay, generated by challenge/replay.py) in bucket results/<team>/sub-<id>/<scenario>/
Leaderboard	best scored submission per team; columns total, base science, program bonus, request reward, penalties, tiles, REQUIRED missing

Deploy (organizers)

1. Supabase: apply supabase/migrations/*.sql in order (SQL editor or supabase db push); Auth site URL and redirect URLs; auto-confirm sign-ups or configure SMTP.
2. Seed and promote the first admin (needs the service role key): SUPABASE_URL=… SUPABASE_SERVICE_ROLE_KEY=… python -m worker.main seed, python -m worker.main promote-admin you@org. Default seed: demo-week (7 nights, public, the copy shipped in the starter kit), dev-reference (180 nights, public), dev-fortnight (14 nights, public), eval-a/eval-b (30 nights, hidden, 3600 s wall clock) and phases practice/online. Which phase accepts which submission route is recorded in docs/competition-format.md.
3. More scenarios: python -m worker.main gen-scenario --slug eval-c --seed 777 --days 30 --start-date 2026-10-05 --wallclock 3600 --hidden-weather --hidden-forecasts or add-scenario --root <dir> for a directory produced by the science team. Scenarios are validated by the authoritative scorer and checksummed before upload.
4. Edge function: supabase functions deploy leaderboard --no-verify-jwt --use-api.
5. Worker: .github/workflows/worker.yml keeps one evaluator alive on a GitHub-hosted runner (python -m worker.main run --max-seconds 19800, polling every 5 s, secrets SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY) and re-dispatches itself with the workflow token before it ends; the cron is only a backstop. It publishes a heartbeat (site_settings.worker_heartbeat) that the submission page shows with the queue position. Wall clocks of 1–2 h per scenario mean extra runners are advisable for the online phase: python -m worker.main run anywhere with Python 3.12, or the Docker image (worker/Dockerfile).
6. Website: GitHub Pages via .github/workflows/deploy-pages.yml (repo variables VITE_SUPABASE_URL, VITE_SUPABASE_ANON_KEY, VITE_SITE_URL, VITE_BASE_PATH).

Tests

.venv/bin/pytest challenge/tests tests/test_challenge_runner.py tests/test_starter_kit.py   # environment, sandbox, kit
SAC_POSTGREST_BIN=/path/to/postgrest .venv/bin/pytest tests/supabase                        # RLS/RPC/worker on embedded Postgres + PostgREST
cd web && npm ci && npm run build && cd .. && .venv/bin/pytest tests/e2e_web                # browser e2e against the harness

SUPABASE_URL=… SUPABASE_ANON_KEY=… SUPABASE_SERVICE_ROLE_KEY=… python tests/hosted_smoke.py # live project, self-cleaning
python tests/hosted_agent_smoke.py --dispatch                                                # queue a minimal-agent zip on the hosted project and run it through the GitHub Actions worker
python tests/live_e2e.py                                                                     # browser walk-through of the production site (public pages, storage policy, submit, replay, admin)
python tests/live_cli_participant.py                                                         # command-line participant journey with the downloaded kit (fetch, run, score, pack, submit, mistakes)

tests/e2e_web/ is four files: test_site.py walks the happy path (register → team → results file → agent package → leaderboard → admin), test_site_v2.py / test_site_v3.py cover the console, replay and v3 report, and test_site_v4.py runs five rounds over the same loop from other angles — sign-up validation and duplicate e-mail, team membership gating a submission, what each phase accepts (kind, scenario visibility, extension, daily limit), a package with no entry script plus the hidden-scenario file list, and both languages, mobile navigation and prefers-reduced-motion.

The harness needs pgserver, psycopg, playwright (plus playwright install chromium) and a postgrest binary. On macOS the release binary links against Homebrew's libpq; the copy inside pgserver works instead:

export SAC_POSTGREST_BIN=/path/to/postgrest SAC_NODE_BIN=$(dirname "$(command -v node)")
export DYLD_LIBRARY_PATH=$PWD/.venv/lib/python3.12/site-packages/pgserver/pginstall/lib

Known deviations from the science team's package

• challenge/run_challenge.py: the agent transport writes non-blocking; the original blocking os.write of a message larger than the pipe buffer (the first decision snapshot is ~200 KB) could hang until the agent read it, defeating the global wall clock. Everything else in challenge/ is the delivered code with relative imports and a root-parameterised project_paths.py.
• challenge/scenario_builder.py clamps time_limited_window_days and the forecast horizon for short scenarios.
• challenge/replay.py regenerates the delivered decision_replay.html (its generator was not in the package).