Concept / transcript submission intake (provenance + dedup + recusal)

[arkh-node]

Splitting this out of #15 as you suggested. Grounded in what's already on main (benchmark_proposals, constructs).

Intake. A public form that accepts a concept or a text transcript and turns it into one or more benchmark_proposals. Fields: title, content (paste/upload), source_type (symposium / experiment / dialogue / paper), provenance (author, venue, date, link), submitter, consent. benchmark_proposals already has submitterName/submitterEmail — this generalizes that into submitter + source so an extracted proposal keeps a pointer to where it came from.

Dedup. On submit, key content by content_hash; same doc twice → one row. (Semantic-similarity dedup later; hash first.)

Extraction. A pass over the content emits candidate proposals with source = <submission id>, proposer = submitter. Categories stay your existing red-team set (prisoner / life raft / sycophancy / deception) — no separate "construct" abstraction.

Recusal. Once proposals get scored/ranked (#15), a submitter doesn't score their own — rater == proposer drops out, ours included. No rating store on main yet, so this is the rule to bake in when that lands.

Open question: a canonical set of categories to freeze as the seed, or grow them from submissions?

[fractastical]

grow them from submissions I think. We can have a category that is for uncategorized parts (which is how most of it started).

[fractastical]

@a-rybnikov have we tested the text insertion of benchmarks or related parsing? I would like to send out but feedback would be appreciated.

[arkh-node]

On it — walked the intake/parsing path (runAcademicEvaluation → evaluateContribution). Fetch side looks solid (SSRF guard, timeout, size cap). Sent you the details and a concrete plan by email; PR with tests + a small parse-status flag to follow shortly.

[fractastical]

ok can we put testing process here or show what tests have been run?

[arkh-node]

The parsing tests are in #31 — server/evaluationParse.test.ts covers valid JSON, fenced JSON, prose (no JSON), string-typed score, empty, and oversized inputs, asserting the new parseOk status so an intake/parse failure is visible and retriable instead of silently defaulting to a score of 1. All pass. The parseOk flag + the extracted evaluateContribution parser are in that PR: #31

[arkh-node]

Here's the actual test run for the parsing (#31), reproducible locally.

Targeted suite — npx vitest run server/evaluationParse.test.ts --reporter=verbose:

✓ clean JSON: score + summary parse, parseOk true
✓ fenced ```json block is stripped and parsed
✓ prose (non-JSON) → defaults to 1 but parseOk FALSE (distinguishes from a real 1)
✓ string-typed score '8' is NOT accepted → 1, parseOk FALSE (the #22 gap)
✓ missing score → 1, parseOk FALSE
✓ out-of-range score is clamped to 1..10, parseOk true
✓ float score is rounded
✓ empty / whitespace input → 1, parseOk FALSE
✓ summary falls back to raw content, capped at 2000 chars
✓ stripJsonFence handles bare and fenced input

Test Files  1 passed (1)
     Tests  10 passed (10)

Full suite — npx vitest run: Test Files 5 passed | 3 skipped · Tests 44 passed | 11 skipped | 0 failed. The 11 skipped are the DB-backed suites (storage.run, storage.proposals) that skip when no Postgres is configured — not failures.

Re: the intake/parsing question — benchmark/prose text that isn't JSON now surfaces parseOk: false (+ a console.warn) instead of silently scoring 1; a string-typed score like "8" is flagged too. Score/summary behaviour is unchanged — this only makes a format failure visible & retriable. Happy to add a persisted parse-status column or a provider JSON-mode retry as a follow-up.