feat: 完成 M05 论文检索、相关性筛选与 OA 全文本地化

.ai-team/TASK.md

@@ -1,61 +1,120 @@
 # Current Task
 
-- ID: `AR-003`
-- Title: `Simplify the assignment plan to module level`
-- Status: `done`
-- Owner: `zzg`
-- Next owner: `user/team`
+- ID: `AR-005`
+- Title: `Finish practical M05 search-to-reader pipeline`
+- Status: `active`
+- Owner: `wdwd200`
+- Next owner: `wdwd200`
 
 ## Goal
 
-根据用户反馈，把任务拆解入口从 122 个子任务收敛为模块级分配。保留此前详细版作为附录，但当前团队先按模块认领负责人、边界、主要产出、依赖和验收目标。
+Improve practical paper-search precision and later materialize legal OA full text locally for Reader.
+
+## Stage 1 scope
+
+Implement lightweight semantic relevance selection after multi-source search and deduplication:
+
+1. Send every deduplicated connector candidate to the optional LLM in batches of 8 using the original user research intent and only bounded title/abstract context.
+2. Strictly validate every batch response and retain valid scores 2/3; reject scores 0/1.
+3. If LLM screening is unavailable or a batch fails, retain candidates fail-open and use lexical relevance only for deterministic fallback ordering.
+4. Build a seed-prioritized Reader workset capped at 20 papers. Persist relevant overflow candidates as deferred bibliographic records without OA/materialization in the current run.
+5. Persist, resolve OA, materialize legal full text, and hand off only the current Reader workset.
+
+Stage 1 does not change query rewriting, Reader/Writer, OA providers, agents, embeddings, BM25, vector stores, rerankers, MCP, or the `PaperRecord` contract.
+
+## Stage 2 scope
+
+Materialize only legally available OA PDF full text after persistence and OA resolution:
+
+1. Preserve explicitly supplied seed papers after normal deduplication, bypass semantic rejection, and prioritize them in the current Reader workset.
+2. Materialize only `open_access` records with an HTTP(S) `fulltext_url` into `paper-projects/<project_id>/02_literature/full_text/`.
+3. Stream bounded downloads to a temporary file, validate them as PDFs readable by the existing Reader dependency, atomically rename on success, and upsert the same `PaperRecord.full_text_path`.
+4. Preserve paper and OA records on every materialization failure, record a diagnostic, and continue processing remaining papers.
+
+Stage 2 does not change Reader, bypass access controls, perform browser automation, add OCR, introduce PDF architecture, or run live Unpaywall testing.
+
+## Query rewrite scope
+
+Use the existing optional LLM to generate exactly three validated, complementary scholarly retrieval queries: core topic, academic terminology/synonym, and a faithful complementary angle. The original `ResearchState.idea` remains the authoritative research intent and is passed separately to LLM relevance screening.
+
+On unavailable, failed, invalid, duplicate, or near-identical LLM output, use three conservative deterministic query variants derived only from the original idea. Do not restore unconditional review or benchmark query suffixes.
 
 ## Acceptance scenarios
 
-- [x] `docs/AutoResearch_任务拆解书.md` 只作为模块级分配入口，覆盖 M00–M17。
-- [x] 每个模块说明目标、边界、主要交付物、前置模块、负责人建议和当前状态。
-- [x] 文档提供模块负责人验收模板和第一批模块分配建议。
-- [x] 原 122 子任务版本保留为 `docs/AutoResearch_任务拆解书_详细版.md`，不丢失后续拆解素材。
-- [x] README、Project-to-Act 和 VibeCollab 任务状态保持一致，不改变五业务 Agent 和证据门禁边界。
+- [x] Offline tests cover all-candidate batch semantic screening, strict validation, 0/1 rejection, 2/3 retention, failed-batch fail-open behavior, original-intent context, and dedupe-before-screening.
+- [x] Offline tests cover lexical-only fallback ordering, deterministic fallback retention, score priority/tie-breaking, bounded Reader worksets, deferred persistence, and deferred OA/materialization exclusion.
+- [x] Existing OA tests continue passing.
+- [x] Full requested checks pass.
+- [x] Multiple explicit seed papers bypass semantic screening while connector duplicates are removed.
+- [x] Offline materialization tests cover successful PDF storage and persistence, Reader path consumption, failures, size limits, unsupported responses, ineligible OA records, repeat execution, and search integration.
+- [x] Materializer diagnostics retain the exception type and safe bounded reasons; streamed downloads without `Content-Length` still enforce the byte limit.
+- [x] Offline query-rewrite tests cover exactly-three valid LLM output, distinctness, failure/invalid/duplicate fallbacks, original-intent preservation, rewritten lexical ranking, and semantic screening context.
 
 ## Invariants
 
-- 当前入口不要求团队一次性拆到函数、接口或工单；模块负责人后续再自行细化。
-- 详细版是参考材料，不代表所有子任务都已实现或已验收。
-- 真实论文试点、生产数据库、Web UI、灾备和性能仍保持原有“外部阻塞/待开发”状态。
-- 不提交 token、密钥、私有来源、完整论文、原始提示或虚构外部结果。
+- Preserve exactly five business agents and existing service architecture.
+- LLM use remains optional and environment-configured; offline tests require no API key.
+- OA enrichment remains after final relevance retention and continues to use existing providers and persistence.
+- Full-text materialization only follows an OA record with an HTTP(S) full-text URL, and only persists a Reader-readable PDF after atomic local completion.
+- Search-query rewriting improves retrieval only; it never overwrites the original research idea used to decide semantic relevance.
+- Lexical relevance is fallback ordering only and never gates candidates when semantic screening is unavailable or a batch fails.
+- Do not claim scientific truth, evidence validity, or paper conclusions from relevance screening.
 
 ## Decisions
 
-- 使用 M00–M17 作为稳定模块 ID，先按模块分配负责人。
-- 保留详细拆解版，避免未来重新整理时丢失依赖和验收素材。
-- `.project-to-act` 继续作为长期项目事实源；`.ai-team/TASK.md` 只记录本次短周期交付。
+- Keep relevance selection inside `PaperSearchService` as focused private methods, with the existing `LLMService` injected by `AutoResearchApplication`.
+- Batch semantic screening uses a constant batch size of 8 and validates exact requested-ID coverage with integer scores 0 through 3.
+- Rank successful score-3 candidates before score-2 candidates; tie-break by DOI or normalized title. Place failed-batch candidates afterward using lexical score and stable identity.
+- Treat seed papers as explicit selections: deduplicate them first, exclude connector duplicates, bypass semantic screening, and prioritize them in the 20-paper workset.
+- Persist all deduplicated candidates as bibliographic records, but invoke evidence/OA/materialization only for the current Reader workset; relevant overflow remains deferred.
+- The existing `papers.search_completed` event retains `paper_ids` and also records `reader_workset_paper_ids` and `deferred_paper_ids` so deferred papers are distinguishable from semantically rejected records.
+- Use `FullTextMaterializer` after OA resolution. It stores deterministic PDF filenames under the existing `02_literature` project folder and upserts the existing paper record only after validation and atomic rename.
+- Full-text materialization includes short `ValueError` reasons in diagnostics and remains bounded to 160 characters; other exception diagnostics remain type-only to avoid leaking transport details.
+- `OrchestratorAgent` validates exactly three normalized-distinct, non-near-identical LLM queries. `PaperSearchAgent` passes `ResearchState.idea` as the semantic relevance context while lexical selection uses `search_queries`.
+- The opt-in live smoke script generates a unique UTC timestamp-plus-random project ID unless `--project-id` is supplied; explicit project reuse is refused when its normal full-text directory already contains PDFs.
+- Live E2E verification is stricter than production fail-open behavior: it requires real query rewrite, no semantic-screening fallback or failed batches, and at least one local-PDF Reader success.
 
 ## Completed
 
-- 将原 `docs/AutoResearch_任务拆解书.md` 移为 `docs/AutoResearch_任务拆解书_详细版.md`。
-- 新建模块级 `docs/AutoResearch_任务拆解书.md`，包含 18 个模块、执行阶段、负责人模板和首批分配建议。
-- 保留 README 任务书入口，并将 AR-003 的状态、范围和下一步写入协作账本。
+- AR-004 OA discovery is present and must remain unchanged in behavior for retained papers.
+- Existing search connectors, dedupe, persistence, evidence, knowledge, and Reader handoff are available.
+- Stage 1 seed handling now preserves all unique user-supplied papers and prevents a matching connector record from duplicating them.
+- Stage 2 now materializes qualifying OA PDFs locally and exposes the resulting path through the existing `PaperRecord.full_text_path` contract.
+- Stage-2 cleanup now reports safe download failure reasons and validates the no-content-length streaming byte cap.
+- Scholarly query rewriting now replaces the former fixed review/benchmark expansion without changing `ResearchState` or contracts.
+- All deduplicated connector candidates now receive batched semantic screening when available; lexical ordering is fail-open fallback only.
+- Relevant overflow candidates are retained as bibliographic records and deferred from OA/materialization when the Reader workset is full.
+- Search-completed events now explicitly identify both the current Reader workset and deferred paper IDs.
+- Live smoke tooling now prevents stale local PDFs from counting as a new run and treats any semantic batch failure as an incomplete E2E verification.
+- Manual live OA-only verification passed for DOI `10.3389/frai.2024.1293084`: Unpaywall returned `open_access` with `cc-by`, a 733733-byte PDF was materialized locally, and `PaperReaderService` consumed it with 10 locators.
+- Manual live E2E verification passed for "How can retrieval augmented generation reduce hallucinations in large language models?": a configured OpenAI-compatible LLM generated exactly three queries, real scholarly connectors returned 17 observed/deduplicated candidates, and real semantic screening completed with zero batch failures.
+- The live E2E run placed 10 papers in the Reader workset with zero deferred; 8 resolved `open_access`, 2 had no open full text found, 4 PDFs were materialized locally, Reader consumed all 4 with zero failures, and the full search-to-Reader smoke result succeeded.
+
+## Live Verification Limitation
+
+Some legitimate OA records return HTML landing pages rather than direct PDF responses. `FullTextMaterializer` intentionally rejects HTML and does not scrape landing pages, automate browsers, or bypass access controls. Therefore `open_access` does not guarantee local PDF materialization.
 
 ## Pending
 
-- 用户/团队需要为模块负责人模板填写实际 owner、分支、目标日期和 evidence ID。
-- 模块负责人确认后，再选择需要展开的模块，不默认展开全部模块。
-- 真实论文试点仍等待 idea、领域、目标 venue、授权全文和实验资源边界。
+- Opt-in live M05 smoke tooling was manually executed successfully by the user; do not rerun it during offline verification.
+- Real email/API credentials are intentionally not stored in source, task files, fixtures, records, events, or reports.
 
 ## Next step
 
-先认领 M01–M04、M05–M07、M08、M09–M10、M11–M12、M13–M15/M17、M16 这 7 个模块包；完成模块边界确认后，再只对当前瓶颈模块做子任务拆解。
+Prepare the combined AR-004 + AR-005 M05 pull request; do not execute live network modes or commit/push as part of this request.
 
 ## Verification
 
-- [x] Project-to-Act `--check` — managed 配置有效，唯一账本保持不变。
-- [x] 模块文档结构检查 — M00–M17 全部存在，详细版文件可访问。
-- [x] `node .ai-team/check.mjs --base main` — AR-003 acceptance 5/5，任务范围和变更同步有效。
-- [x] Project-to-Act `--validate` — managed 配置 valid，issues 为空。
+- [x] `.venv\Scripts\python.exe -m pytest tests/test_relevance_selection.py -q` - 14 passed
+- [x] `.venv\Scripts\python.exe -m pytest tests/test_query_rewrite.py tests/test_relevance_selection.py tests/test_open_access_service.py tests/test_full_text_service.py -q` - 27 passed
+- [x] `.venv\Scripts\python.exe -m pytest tests/test_live_m05_smoke.py -q` - 8 passed; no network calls
+- [x] `.venv\Scripts\python.exe -m pytest -q` - 53 passed
+- [x] `.venv\Scripts\python.exe -m pytest -W error -q -p no:cacheprovider` - 53 passed
+- [x] `.venv\Scripts\python.exe -m ruff check src tests scripts` - all checks passed
+- [x] `.venv\Scripts\autoresearch.exe doctor` - healthy, five agents
+- [x] `node .ai-team/check.mjs --base upstream/main` - valid; 8/8 functional progress
+- [x] `git diff --check` - no whitespace errors; Windows line-ending warnings only
 
 ## Handoff note
 
-- From: `zzg`
-- To: `user/team`
-- Summary: 当前分配入口已经降为模块级；详细拆解保留在附录。先分模块负责人，后续只对真正需要的模块继续细化。
+AR-005 stages 1, 2, query rewriting, bounded worksets, OA PDF materialization, and opt-in live smoke verification are complete. Live credentials remain local and unrecorded.

.gitignore

@@ -20,6 +20,9 @@ dist/
 var/
 .langgraph_api/
 
+# Opt-in live M05 smoke artifacts remain local for manual inspection.
+/paper-projects/live-m05-smoke-*/
+
 # Local reference repository; not part of the AutoResearch source tree
 DeepReason-Agents-Framework/