Add compact conversational memory QA task

task-submissions/haoran/1-x-1/.gitignore

@@ -0,0 +1,4 @@
+/data/
+/models/
+__pycache__/
+*.py[cod]

task-submissions/haoran/1-x-1/README.md

@@ -0,0 +1,176 @@
+# Compact Conversational Memory Question Answering
+
+Build a compact memory and a retrieval-augmented system that answers questions about meeting transcripts.
+
+**Task:** `task-1-x-1` · **Mode:** Implementation · **Metric:** EvidenceGroundedAnswerAccuracy
+
+## Overview
+
+This task covers 67 meetings from three ICSI series: Bmr (29), Bro (23) and
+Bed (15). The history contains 53,600 utterances and 719,788 whitespace-delimited
+words, including interruptions, references to earlier discussions, corrections
+and changing proposals.
+
+The agent receives the complete history during development and builds a
+finished memory, a retriever and an answerer. Evaluation withholds the original
+transcripts from all three submitted programs. Each answer must be supported by the
+records retrieved from the compact memory.
+
+## What This Task Tests
+
+- Preserving useful information from conversations within a storage budget.
+- Retrieving question-relevant evidence from a locally prepared memory.
+- Producing answers grounded in the retrieved records.
+- Packaging retrieval and answering for separate execution environments.
+
+## Task Setup
+
+### Provided Assets
+
+| Asset | Purpose |
+| --- | --- |
+| `data/history.jsonl` | Complete transcripts of 67 meetings |
+| `data/validation/queries.jsonl` | 30 public development questions |
+| `data/validation/golden_answers.jsonl` | Public reference answers and required facts |
+| `data/validation/evidence.jsonl` | Transcript excerpts supporting the public answers |
+| `environment/docs/` | Installed runtime and permitted API resources |
+
+The download script restores the task's `data/` directory, mounted read-only
+under `/task/data`. The 118 held-out questions concern the same history and are
+disjoint from the public examples. Their answers and 1,344 supporting excerpts
+are packaged under `tests/data/` and excluded from the agent environment.
+Evidence provenance is checked against the supplied transcripts.
+
+### Fixed Components and Allowed Changes
+
+The history, submission interfaces and resource limits are fixed. Memory
+format, information selection, indexing, retrieval and answer generation are
+implementation choices. The task starts without a starter implementation.
+
+Memory construction and retrieval must use local computation, including during
+development. Only the answerer may call the configured generation API, subject
+to the [resource policy](environment/docs/available_resources.md).
+
+### Environment and Resource Limits
+
+The CPU Python 3.12 environment provides 8 CPUs, 8 GiB memory, 8 GiB storage
+and no GPU. The agent has 120 minutes; the Harbor verifier phase has 90 minutes.
+Across the question set, retrieval has 150 seconds and answering 1,800 seconds.
+Each answer permits at most two API calls, 2,000 output tokens per call and
+2,000 Unicode characters in the final text.
+
+`memory.json` is readable UTF-8 JSON and may occupy at most 183,981 bytes,
+5% of the 3,679,621 dialogue-text bytes. Scripts are outside this memory budget
+and contain corpus-independent code.
+
+## Submission Contract
+
+The agent submits exactly four files: `memory.json`, `build_index.sh`, `search.sh`
+and `answer.sh`. Each script is self-contained and executable.
+
+The verifier builds an index from `memory.json` offline, with a 300-second
+limit. For each question, offline search reads the index and returns a JSON
+array of at most ten strings. The answerer receives the question and that array
+and returns plain text. Only the answerer can call the configured generation
+model, through a task-provided transport.
+
+Each stage runs unprivileged with a fresh working directory and a separate file
+allowlist. The builder reads the memory; search reads the generated index; the
+answerer reads only the current retrieved strings. The original history,
+hidden evaluation data and grading files are not available to these programs.
+Only the four submitted files are transferred; the index is created at runtime.
+See [instruction.md](instruction.md) for flags and output formats.
+
+## Evaluation
+
+### Search Quality
+
+A question scores one only when both an evidence hit and answer correctness
+are accepted. The evidence judge compares retrieved text with supporting
+transcript excerpts, accepting faithful paraphrases. At least one
+question-relevant fact must be preserved. Relevant passages are first extracted
+without access to the reference evidence. The verifier checks their origin in
+the recalled strings before comparing them with the reference. Positive
+judgments identify the passages supporting the matched facts. A further check,
+without the reference evidence, verifies that the cited passages support those
+facts, without requiring a complete answer. The answer judge checks the final
+answer against its reference and required facts.
+
+These are separate judgments: the evidence judge does not see the submitted
+answer, and the answer judge does not see the retrieved records. Each uses
+three votes with majority voting; positive evidence votes also require the
+reference-blind support check. EvidenceGroundedAnswerAccuracy is the mean
+of the joint outcomes over the 118 held-out questions; the report also includes
+`evidence_accuracy` and `answer_accuracy`.
+
+### Correctness and Resource Gates
+
+All three entry points, output formats, execution limits and the memory budget must
+pass. Invalid output or a failed submission process invalidates the submission.
+Empty retrieval is an evidence miss. A malformed judge response or model
+service failure invalidates the measurement.
+
+### Integrity Checks and Final Reward
+
+A separate trajectory and file audit checks task and resource compliance.
+It runs under its own user, with read-only access to the submission and a copy
+of the trajectory. Its model relay holds provider credentials outside the
+auditing process. The judge cannot read hidden labels or write final scores.
+A violation sets the final reward to zero; an incomplete audit is an
+infrastructure failure. The audit uses `deepseek-flash` through pinned RewardKit
+0.1.7; evidence and answer grading use independently configured judge settings.
+
+```text
+reward = mean(evidence_hit AND answer_correct), if the compliance audit passes
+score = 100 * reward
+```
+
+## Running This Task
+
+From the repository root, follow the [quick start](../../../docs/quickstart.md)
+to prepare the runtime and Docker. The [evaluation guide](../../../docs/evaluation.md)
+explains agent and verifier configuration; the [asset guide](../../../docs/assets.md)
+covers downloads and checksums.
+
+Configure `ANSWER_API_KEY`, `ANSWER_API_BASE_URL` and `ANSWER_MODEL` for the
+submitted answerer, `ANSWER_JUDGE_*` for evidence and answer grading, and
+`VERIFIER_OPENAI_*` for the trajectory audit. These settings default to empty
+values in the task configuration. The verifier's network allowlist permits
+`api.deepseek.com`; using another provider requires a matching task configuration
+change. Credentials must remain outside the task package.
+
+```bash
+python scripts/download_assets.py --task-path task-submissions/haoran/1-x-1
+bash scripts/run_task.sh --task-path task-submissions/haoran/1-x-1 --model "YOUR_AGENT_MODEL"
+```
+
+Replace `YOUR_AGENT_MODEL` with the configured model. Add `--dry-run` to inspect
+the launch command without starting an evaluation. Task-local checks can be run
+with `python -m unittest discover -s task-submissions/haoran/1-x-1/tests -p 'test_*.py'`.
+
+## Task Files
+
+| File or directory | What to read it for |
+| --- | --- |
+| [instruction.md](instruction.md) | Complete agent-facing specification and executable contract |
+| [task.toml](task.toml) | Task identity, artifacts, resource limits and environment variables |
+| [assets.json](assets.json) | Fixed asset paths, immutable revisions and checksums |
+| [Environment guide](environment/docs/environment.md) | Installed runtime and task environment |
+| [Environment configuration](environment/docker-compose.yaml) | Read-only data mounts |
+| [Verifier](tests/) | Execution, output validation and scoring |
+| [Resource policy](environment/docs/available_resources.md) | Permitted API calls and credential handling |
+| [Corpus license](ICSI_LICENSE.html) | ICSI redistribution terms and attribution |
+
+## Data Provenance and Limitations
+
+ICSI stands for **International Computer Science Institute**. The
+[ICSI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/icsi/download/) contains
+recorded research meetings and human transcripts; see Janin et al., *The ICSI
+Meeting Corpus*, ICASSP 2003. The original license notice is included.
+The 148 QA examples were authored for this task and are not original ICSI
+annotations. They have transcript-linked evidence but no independent human
+certification. Semantic grading remains subject to model variability.
+
+Public assets are pinned in [assets.json](assets.json) to the
+[development dataset](https://huggingface.co/datasets/hrjinbb12345/search-swe-development/tree/7fe4b0bfb7699cb393bc5b11835c47ffeb13aaec).
+Official asset migration and final task numbering remain maintainer steps.

task-submissions/haoran/1-x-1/assets.json

@@ -0,0 +1,49 @@
+{
+  "schema_version": 1,
+  "files": [
+    {
+      "path": "data/history.jsonl",
+      "size_bytes": 12155329,
+      "sha256": "d71fbbc8f233145e155131bd01b6c104a9a5c578379cb66bfa4c8c2bba54cb52",
+      "source": {
+        "repo_id": "hrjinbb12345/search-swe-development",
+        "repo_type": "dataset",
+        "revision": "7fe4b0bfb7699cb393bc5b11835c47ffeb13aaec",
+        "filename": "development/task-1-x-1/history.jsonl"
+      }
+    },
+    {
+      "path": "data/validation/evidence.jsonl",
+      "size_bytes": 74062,
+      "sha256": "d7419b529598c2c392fbe346b5045789a20ecb763fb336fa7140a793d5befbf7",
+      "source": {
+        "repo_id": "hrjinbb12345/search-swe-development",
+        "repo_type": "dataset",
+        "revision": "7fe4b0bfb7699cb393bc5b11835c47ffeb13aaec",
+        "filename": "development/task-1-x-1/validation/evidence.jsonl"
+      }
+    },
+    {
+      "path": "data/validation/golden_answers.jsonl",
+      "size_bytes": 11271,
+      "sha256": "d6d2588018a278a86d9fa9a2e7f358b3b243a8de026d97a1338ee636575fbec6",
+      "source": {
+        "repo_id": "hrjinbb12345/search-swe-development",
+        "repo_type": "dataset",
+        "revision": "7fe4b0bfb7699cb393bc5b11835c47ffeb13aaec",
+        "filename": "development/task-1-x-1/validation/golden_answers.jsonl"
+      }
+    },
+    {
+      "path": "data/validation/queries.jsonl",
+      "size_bytes": 6950,
+      "sha256": "1bc181d04d065275401a5972c710912ea6c77375b10389652fc83ffff83afaa1",
+      "source": {
+        "repo_id": "hrjinbb12345/search-swe-development",
+        "repo_type": "dataset",
+        "revision": "7fe4b0bfb7699cb393bc5b11835c47ffeb13aaec",
+        "filename": "development/task-1-x-1/validation/queries.jsonl"
+      }
+    }
+  ]
+}

task-submissions/haoran/1-x-1/environment/Dockerfile

@@ -0,0 +1,21 @@
+FROM docker.io/hanhainebula/search-swe-base:cpu-py3.12-1.0.0
+
+ARG SEARCH_SWE_CODEX_VERSION=0.147.0
+ARG SEARCH_SWE_CLAUDE_CODE_VERSION=2.1.273
+ARG SEARCH_SWE_PI_VERSION=0.85.1
+RUN npm install --global --ignore-scripts \
+        --registry=https://registry.npmmirror.com \
+        "@openai/codex@${SEARCH_SWE_CODEX_VERSION}" \
+        "@earendil-works/pi-coding-agent@${SEARCH_SWE_PI_VERSION}" \
+    && npm install --global \
+        --registry=https://registry.npmmirror.com \
+        "@anthropic-ai/claude-code@${SEARCH_SWE_CLAUDE_CODE_VERSION}" \
+    && codex --version | grep -Fx "codex-cli ${SEARCH_SWE_CODEX_VERSION}" \
+    && test "$(pi --version)" = "${SEARCH_SWE_PI_VERSION}" \
+    && test "$(claude --version)" = "${SEARCH_SWE_CLAUDE_CODE_VERSION} (Claude Code)"
+RUN claude --help | grep -F "(low, medium, high, xhigh, max)" >/dev/null
+
+RUN useradd --create-home --uid 10000 --user-group agentdev \
+    && mkdir -p /app \
+    && chown agentdev:agentdev /app
+WORKDIR /app

task-submissions/haoran/1-x-1/environment/docker-compose.yaml

@@ -0,0 +1,21 @@
+services:
+  main:
+    volumes:
+      - type: bind
+        source: ../data/history.jsonl
+        target: /task/data/history.jsonl
+        read_only: true
+        bind:
+          create_host_path: false
+      - type: bind
+        source: ../data/validation
+        target: /task/data/validation
+        read_only: true
+        bind:
+          create_host_path: false
+      - type: bind
+        source: ./docs
+        target: /task/docs
+        read_only: true
+        bind:
+          create_host_path: false

task-submissions/haoran/1-x-1/environment/docs/available_resources.md

@@ -0,0 +1,79 @@
+# Available Resources
+
+## Answering API configuration
+
+During development, use the following runtime environment variables to test your answerer:
+
+| Variable | Meaning |
+| --- | --- |
+| `ANSWER_API_KEY` | Credential for the answering API |
+| `ANSWER_API_BASE_URL` | OpenAI-compatible API base URL; the chat endpoint is this URL plus `/chat/completions` |
+| `ANSWER_MODEL` | The only model your answerer may call |
+
+Harbor injects these values into the development environment for local answerer tests. During evaluation, the submitted answerer receives only `ANSWER_MODEL` and the task-provided transport; the verifier retains the real API key and upstream URL. Read them as environment variables; do not source a `.env` file inside the container or hard-code a key, URL, or model. For direct development calls, check that the variables are set without printing the key:
+
+```python
+import os
+for name in ("ANSWER_API_KEY", "ANSWER_API_BASE_URL", "ANSWER_MODEL"):
+    if not os.environ.get(name):
+        raise RuntimeError(f"Missing runtime setting: {name}")
+```
+
+Only the configured endpoint and model are allowed. Do not use other providers, external search or answer services, or benchmark-answer datasets. Credentials must not be written into submitted code, memory, indexes, prompts, or logs. The model running your coding session is separate from this API resource.
+
+## Permitted API use
+
+Only the answerer may call the configured API, using the current question and the records returned by the submitted retriever. This restriction applies during development and evaluation. Do not call helper APIs for memory construction, interpretation of the raw corpus, summarization, indexing, embedding or ranking. Local computation and installed libraries are allowed.
+
+The allowed API host is `api.deepseek.com`. Use only the endpoint and model named by the injected settings. The coding agent's model access and verifier judge credentials are separate resources.
+
+At evaluation time, index construction and retrieval have no network access or API credentials. The answerer can read only its corpus-independent code, runtime dependencies and the current question's retrieved records. The original history, full memory and previous requests are unavailable to it.
+
+## Calling the API from your answerer
+
+Use the task-provided Chat Completions transport in your submitted `answer.sh`. It forwards your request to `ANSWER_API_BASE_URL` using `ANSWER_API_KEY`; you supply the prompts and parse the response. Direct network access is disabled during evaluation, so this transport is the permitted API route.
+
+```python
+import importlib.util
+import os
+
+spec = importlib.util.spec_from_file_location("task_llm", os.environ["TASK_LLM_CLIENT"])
+api = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(api)
+response = api.chat_completion(
+    messages=[{"role": "user", "content": prompt_from_current_query_and_retrieved_memories}],
+    model=os.environ["ANSWER_MODEL"],
+    temperature=0,
+    max_tokens=1000,
+)
+text = response["choices"][0]["message"]["content"]
+```
+
+During evaluation, the runtime supplies `TASK_LLM_CLIENT` and `TASK_LLM_FD` to `answer.sh`. Preserve `TASK_LLM_FD` if launching another process to implement the answerer. Neither `ANSWER_API_KEY` nor `ANSWER_API_BASE_URL` is passed to submitted programs during evaluation. Do not require them when `TASK_LLM_CLIENT` is present.
+
+### Development self-tests
+
+The development environment supplies `ANSWER_API_KEY`, `ANSWER_API_BASE_URL`, and `ANSWER_MODEL`, but no verifier transport descriptor. Your answerer should use the transport above when `TASK_LLM_CLIENT` is present. Otherwise, during development only, it may send the same request directly to the configured endpoint:
+
+```python
+import requests
+response = requests.post(
+    os.environ["ANSWER_API_BASE_URL"].rstrip("/") + "/chat/completions",
+    headers={"Authorization": "Bearer " + os.environ["ANSWER_API_KEY"]},
+    json=dict(request_body, model=os.environ["ANSWER_MODEL"]),
+    timeout=45,
+    allow_redirects=False,
+)
+response.raise_for_status()
+result = response.json()
+```
+
+`request_body` is your answerer's Chat Completions request, built only from the current question and the retriever's output. Use the same call and token limits in both modes. The direct route is unavailable during evaluation; do not fall back to another endpoint or model when a call fails.
+
+Run your retriever on a public question, then pass that exact output file to your answerer using the interfaces in the task instruction, after building an index from `memory.json`. Keep test outputs outside `/app`.
+
+Allowed request options are `model`, `messages`, `temperature`, `top_p`, `max_tokens`, `response_format`, and `thinking`. Each request may contain 1–32 messages, each with string `role` and `content` fields; roles are `system`, `user`, or `assistant`. Serialized requests must fit within 128 KiB including the newline. Each question permits up to two API calls and each call up to 2,000 output tokens. Streaming, tool calls, and alternate endpoints are unsupported. Provider-specific options should be used only if supported by the configured endpoint.
+
+The answerer must use only the current query and retrieved evidence. Its prompts and script must not contain corpus-specific facts. Do not reuse information from previous questions. When the evidence is insufficient, say so in plain text.
+
+Using an API outside the answering component, accessing unprovided evidence, or otherwise bypassing the memory and retrieval pipeline is a task violation and sets the entire score to zero.

task-submissions/haoran/1-x-1/environment/docs/environment.md

@@ -0,0 +1,15 @@
+# CPU Docker Environment
+
+This image provides a Conda-managed Python 3.12 environment with CPU-only PyTorch and the common search, embedding, indexing, document-processing, media, HTTP, and service packages used by the tasks. Installed Python packages include `torch`, `torchvision`, `torchcodec`, `numpy`, `transformers`, `sentence-transformers`, `FlagEmbedding`, `deepspeed`, `faiss-cpu`, `bm25s`, `rank-bm25`, `pyserini`, `hnswlib`, `qdrant-client`, `docling`, `marker-pdf`, `pdf2image`, `pypdfium2`, `CairoSVG`, `av`, `imageio`, `imageio-ffmpeg`, `tiktoken`, `fastapi`, `uvicorn`, `python-multipart`, `requests`, `aiohttp`, `openai`, and `pydantic-settings`.
+
+The task Python interpreter and its installed packages are available at `/opt/conda/bin/python`. Use `/opt/conda/bin/python` and `/opt/conda/bin/pip` when invoking Python or installing packages.
+
+The image also includes JDK 21, FFmpeg, Poppler utilities, Cairo, Git, curl, `jq`, `build-essential`, `ca-certificates`, `libffi`, `libgomp`, `netbase`, `netcat`, `procps`, `tzdata`, `unzip`, and the related system runtime libraries.
+
+## Task-specific execution
+
+The runtime provides 8 CPUs, 8 GiB RAM, 8 GiB storage and no GPU. Numerical libraries default to one thread per process during development to avoid excessive threading on shared hosts. Development runs as `agentdev`; `/app` and your home directory are writable. Task data and system libraries are read-only.
+
+`/app` starts empty. Submit `memory.json`, `build_index.sh`, `search.sh` and `answer.sh` there. Only declared artifacts are transferred to the separate verifier; development home files, extra installed packages and running services are not transferred. No local model weights are provided under `/opt/models`.
+
+Evaluation runs index construction, retrieval and answering as an unprivileged user with read-only submission files and separate filesystem permissions. Use the supplied output directory for runtime scratch files. The task instruction defines each stage's inputs and limits; [available_resources.md](available_resources.md) documents the answering API transport.