Add compact conversational memory QA task

[hr-jin]

Summary

Submission paths (new tasks): task-submissions/haoran/1-x-1
Category / Implementation or Optimization / CPU or GPU: Category 1 / Implementation / CPU
Authors and coauthors: Haoran Jin

Add a conversational-memory question-answering task over 67 ICSI meetings. The agent receives the complete transcripts and 30 public question–answer examples, then builds a compact memory, an offline retriever, and an answerer. Evaluation uses 118 disjoint questions about the same history.

The submission provides two executable scripts. Retrieval runs offline and returns at most ten memory records. Answering receives only the question and those records and may call the configured generation API. The original transcripts are unavailable to both programs during evaluation.

The verifier checks trajectory compliance, retrieved-evidence relevance, and answer correctness. A question earns a point only when both evidence and answer checks pass. The primary metric is EvidenceGroundedAnswerAccuracy, with evidence-hit rate and answer accuracy reported separately.

Validation evidence

Validated against upstream f02b98f.

Check	Result
Submission checker	0 errors
Release checker	0 errors, 0 warnings
Task-local tests	33 passed
Repository tests	83 passed
Pi trajectory tests	21 passed
Public asset verification	All 4 files verified
Shell syntax and diff checks	Passed

Commands:

python scripts/check_submission.py task-submissions/haoran/1-x-1 --base "$(git rev-parse upstream/main)"
python scripts/check_release.py
python -m unittest discover -s task-submissions/haoran/1-x-1/tests -p 'test_*.py'
python -m unittest discover -s scripts/tests -p 'test_*.py'
python -m unittest scripts.test_pi_trajectory
python scripts/download_assets.py --task-path task-submissions/haoran/1-x-1 --verify-only
bash -n task-submissions/haoran/1-x-1/tests/test.sh
git diff --check

Task-specific tests cover valid and invalid outputs, compliance-gate handling, stage isolation, joint scoring, and evidence provenance. A disposable-container smoke test also exercised the actual script interfaces, filesystem restrictions, and API transport with local model test doubles.

An official Quick Start run used Pi 0.85.1 + deepseek/deepseek-flash, with thinking=xhigh. Both images built, and Harbor completed one trial with no execution errors.

Metric	Result
Evidence hits	80/118 — 67.80%
Correct answers	25/118 — 21.19%
Evidence and answer both correct	24/118 — 20.34%
Total retrieval time	40.061 seconds
Total answering time	160.226 seconds

After clarifying the compliance judge’s yes/no score semantics, a separate audit of the same submission and complete trajectory passed. The finalizer produced reward 0.2033898305, reusing the original evidence and answer judgments.

Checks not completed:

• The full runtime negative-submission sweep and independent judge calibration have not been completed; validation currently includes the task-local cases, container smoke test, and coding-agent trial described above.
• The paid end-to-end trial was not repeated after the final documentation-only cleanup.
• The merge-ready check remains pending final task promotion and official asset migration.

Contributor checklist

• All new tasks use exactly one contributor first-name namespace; temporary ordinals are category-unique and never reused after promotion; no direct formal additions or guessed final IDs.
• Actual authors in task.toml; commit email associated with my GitHub account.
• Allow edits from maintainers; promotion stays in this same PR.
• No secrets, downloaded inputs/models, job outputs, or author-only solutions.
• Static package/submission checks and applicable runtime checks recorded.
• Asset provenance, redistribution rights, licenses and pinned development SHA documented.

Maintainer merge checklist (new tasks)

• Design, verifier isolation and runtime evidence reviewed; exceptions explicitly approved.
• Updated to latest main; every final number assigned serially without collision.
• Separate pure git mv and reviewed finalization commits per task, in this PR.
• Per-task HF community PR or maintainer mirror merged into official dataset; manifest, SOURCES and license updated, all old assets preserved; official SHA pinned in assets.json.
• Repository inventories/docs updated; full checks and merge-ready gate pass; every task is promoted and no submission task.toml remains.
• Merge commit only (no squash/rebase); contributor history retained.