Skip to content

A Data Source's uploaded file as a node's input: docs, harness shape, two tests - #14

Merged
makseq merged 1 commit into
mainfrom
feat/data-source-input
Oct 13, 2026
Merged

makseq merged 1 commit into
mainfrom
feat/data-source-input

Conversation

@makseq

@makseq makseq commented Oct 13, 2026

Copy link
Copy Markdown
Member

Why

The orchestrator now hands an external node wired directly after a Data Source that Data Source's uploaded file as its input (HumanSignal/orchestrator#319, live in production since 2026-10-13 00:25 UTC). Until now such a node always started with "inputs": []. This repository is the document the product points every node author at, so it has to say so — and the harness has to send the job shape production sends.

What

  • docs/PROTOCOL.md §2.2 — the uploaded-file input: one object on input, which file, the cleaned stored name as relpath, the kind (last extension of that name, lower-cased; file when there is none), upstream_execution_id null whenever no execution id is known, limits (uploaded files only, external nodes only, one file per run, 500 MB default cap on the declared upload size). Contract version 1 is unchanged: no field was added.
  • docs/OPERATIONS.md — the wiring paragraph (word for word what the product's generated prompt and in-product guide say), a new section Giving the node a file you uploaded, when a required kind fails at once (a Data Source file) and when only after retries (an upstream step), four troubleshooting rows.
  • docs/AUTHORING.md, docs/README.md, docs/CONFORMANCE.md — a third way to get input, "at most one port", checklist items for kind, provenance and file names.
  • conformance/job.py — writes what the launcher writes (upstream_execution_id on every object, "prefix_digest": null on a file port, organization_id); a job can carry a real port kind and a null producing execution.
  • Two tests — the reference node copies one binary zip input with no producing execution; this copy of the contract accepts null provenance and refuses what the real parser refuses. node.py needed no change.

Verification

  • Full suite with real Docker and the verbatim citation check against the merged orchestrator commit 9fdafaa8: 143 passed (141 before; the two new tests).
  • Both new tests were broken on purpose in a scratch copy and went red for the intended reason.
  • In production: an uploaded file reached the hosted reference node, which downloaded it and returned identical bytes (same sha256).
  • Reviewed by Codex (gpt-6-astra, xhigh) and Grok (grok-4.7): two inexact statements about which file a retry reads and about dot-files were found and fixed; second pass clean.

Not in this PR: provenance blocks, pinned commits and test counts — refreshed next, from a re-measured run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XQDS15iYfpfCaTY6LoDx9e

… two tests

The orchestrator (PR HumanSignal/orchestrator#319, live in production since
2026-10-13) now hands an external node wired directly after a Data Source that
Data Source's uploaded file: ONE object on the `input` port, pinned by sha256
and size like any input. Contract version 1 is unchanged.

Docs
- PROTOCOL 2.2: where the port's kind comes from (first upstream artifact, or
  the last extension of an upload's cleaned stored name), null
  `upstream_execution_id` whenever no execution id is known, the uploaded-file
  input and its limits, the cleaned relpath, the default 500 MB cap on the
  declared upload size.
- OPERATIONS: the wiring paragraph (same words as the product's generated
  prompt and in-product guide), a new section "Giving the node a file you
  uploaded", when a required kind fails at once and when after retries, four
  troubleshooting rows about a node wired after a Data Source.
- AUTHORING, README matrix, CONFORMANCE: the third way to get input; "at most
  one port"; checklist items for kind, provenance and file names.

Harness
- conformance/job.py writes what the launcher writes: `upstream_execution_id`
  on every object, `"prefix_digest": null` on a file port, `organization_id`;
  a job can carry a real port kind and a null producing execution.

Tests (143 in total, all green; citations verified against orchestrator
9fdafaa8)
- the reference node copies ONE binary input of kind `zip` whose producing
  execution is unknown;
- this copy of the contract accepts a null `upstream_execution_id` and refuses
  the values the real parser refuses.

Provenance blocks, pinned commits and test counts are refreshed separately
from a re-measured run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQDS15iYfpfCaTY6LoDx9e
@makseq
makseq merged commit a9ca5b8 into main Oct 13, 2026
1 check passed
@makseq
makseq deleted the feat/data-source-input branch October 13, 2026 00:39
makseq added a commit that referenced this pull request Oct 13, 2026
Brings the branch up to main: the documentation alignment (#13) and the Data
Source input (#14). Three files conflicted; everything else merged by itself
and was read through afterwards.

Conflicts:
- README.md: the opening paragraph. Main's dated verification statement is kept
  word for word; this branch had only bumped a test count in the paragraph main
  replaced, and the counts are refreshed separately from a re-measured run. The
  two search-facts bullets in the list below it are kept.
- docs/AUTHORING.md, "node.py and this skeleton", two places. Main's new
  paragraph about the input check stays, and the sentence under it now counts
  "Three things" because this branch adds a third. That third bullet (the
  report document is inventoried after its upload) is kept, followed by main's
  rewritten paragraph about the `report` port in place of the older one.
- docs/PROTOCOL.md, section 4.4. Main had only dropped the line numbers from a
  citation inside the paragraph "Nothing in the orchestrator ever reads
  result.json", which this branch replaces: the replacement stays, and it
  already cites by file and function name. The background note that opens the
  section is shortened to what is true in main's text: it no longer refers to
  line numbers (main's document has none) or says that every other section is
  measured against the commit the preamble names (the Data Source passages are
  not).

Merged without conflict and checked by reading: CONFORMANCE-BASELINE.md,
conformance/README.md, docs/OPERATIONS.md, docs/README.md, node.py and
tests/test_harness_self.py. The search-facts tests need no change for main's
harness: the new Job parameters have defaults.

Provenance blocks, pinned commits and test counts are left as each side wrote
them; they are refreshed separately from a re-measured run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQDS15iYfpfCaTY6LoDx9e
makseq added a commit that referenced this pull request Oct 13, 2026
)

An external node may now put an optional `facts` list into result.json —
[{"key": "task", "value": "282139672"}, {"key": "input", "value": "tasks.json"}] —
and the orchestrator records each entry in its search-facts table when it
collects the run (at marker acceptance, on any outcome, under the 1 MiB
ceiling), so the run is found by task:…, input:…, or a bare id.

node.py: one `input` fact per copied input (the envelope's own name); the
orchestrator's rules applied at write time (claim(): key grammar, value ≤512
chars, no control characters or lone surrogates, cap 10,000; skips counted in
summary.facts_not_claimed); the report written on the failure path before the
marker (marker strictly last; a refused upload is retried and never falsely
inventoried — RESULT_UPLOADED); not attempted on the cancel path (a stopping
node that abandons in-flight connections cannot open a new one — one plain
line says so).

Conformance harness: validate_result shape-checks `facts` NON-fatally (facts
are never load-bearing): the orchestrator's rules mirrored constant for
constant — key ^[a-z][a-z0-9_]{0,63}\Z, value str|int non-bool ≤512 free of
control characters and lone surrogates, meta absent or a flat object (≤32 keys
of 1–64 chars, scalar values) — reported as recommendations.

Docs: PROTOCOL.md (the facts key, the flat meta rule, when it is read),
AUTHORING.md ("Make the run findable" + checklist), OPERATIONS.md limits,
README + capability matrix, conformance README, CONFORMANCE-BASELINE counts.

Companion of HumanSignal/orchestrator "feat(external): search facts for
external nodes".

Brought up to date on 2026-10-13, after the orchestrator half (#296) was
deployed: merged with main a9ca5b8 (#13, #14 — the Data Source input);
four statements corrected against the platform as deployed (reserved search
prefixes; no facts after an operator's Cancel of running work; the `input`
fact of a Data Source upload carries the cleaned name; one pointer); and
provenance refreshed from a re-measured run against orchestrator 86461801:
153 tests green with the verbatim citation check (152 + the one tolerated
skip without orchestrator sources), groups 62/67/24, bases 83/45/25,
38 citations / 90 fragments, mutation verifier 12/12. Three sentences that
had gone stale since 2026-09-10 were fixed (registration is under
"Organization"; the setup command no longer says there is no UI; a pointer).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AuqWuKD5dik1RMBRkcpbAj
Claude-Session: https://claude.ai/code/session_01XQDS15iYfpfCaTY6LoDx9e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant