Skip to content

[Experiment/index-time] Bind every cell to its intended and executed strict route #168

Description

@samchon

Problem

The real-provider and cold-index matrices do not bind every result to both the
strict route they intended to prove and the producer that actually served it.
The measurement can preserve a fallback as useful evidence, but its report and
summary must not turn that fallback into a successful measurement of a
different provider.

The complete index-time run 33377586947 on PR #164 head b26fe40 provides
three independent witnesses:

Project Intended owner Actual result Current interpretation
Excalidraw ttscgraph static fallback correctly marked non-semantic, but the report binds provisioned 0.25.0 while the fixture-resolved binary is 0.22.0
Gson javac-graph scip-java summary reports a 2.1x strict-provider ratio
Redis clangd-snapshot scip-clang summary reports a 0.9x strict-provider ratio

index-time-summary.mjs currently defines “strict provider served” as any
lsp/hybrid servedBy value that does not contain “no strict provider”. A
fallback provider therefore satisfies the predicate even when the primary
owner failed. The report's strict boolean records that strict providers were
enabled, not that the intended owner served.

The Excalidraw artifact separately proves a toolchain-binding gap. Its manifest
records workspace ttscgraph 0.25.0 and its SHA-256, while the pinned fixture
declares ttsc 0.22.0. Project-local resolution wins and the log says that the
executed binary returned a legacy full dump. The report is therefore bound to
provisioned bytes that did not produce the cell.

The real-provider artifacts have a related evidence omission. They record the
provider's declared fact list and observed edge counts, but not the real dump's
15 coverage states or unresolved reason counts. A real producer can therefore
run while the artifact cannot show which families it called complete, partial,
or unsupported.

Consequence surface

The index-time workflow is a measurement rather than a merge gate. A provider
failure should still produce and upload a report; making the job red would lose
the evidence this issue exists to preserve.

Approach

Give a cell separate structured fields for intent and outcome. Derive the
intended primary owner from the canonical provider registry, not from corpus
names or summary regexes. Preserve the actual provider provenance and producer
version from the completed dump in a bounded machine-readable side channel, so
the report can distinguish the route, tool, and version that served.

Update the Excalidraw source fixture to a protocol-v1 ttsc release and pin the
new fixture commit through the benchmark corpus and question manifest. The
fixture's own project-local tool and the provisioned manifest must describe the
same executable generation.

Add compact coverage and unresolved summaries to real-provider experiment
artifacts. Keep full unresolved sites in the graph dump; the experiment needs
counts by family/state/reason, not another unbounded payload.

Acceptance and verification

  • A completed strict-intent cell records the expected primary provider, actual
    provider provenance, producer identity/version, and an explicit
    served/fallback/static/unknown verdict.
  • Gson through scip-java and Redis through scip-clang are reported as
    primary-provider failures and never receive strict-provider speed ratios.
  • A same-run fallback remains publishable evidence and the measurement job
    remains successful while the summary emits an explicit GitHub warning.
  • Excalidraw resolves a protocol-v1 project-local ttscgraph; the actual
    provenance agrees with the toolchain evidence and the cell is not static.
  • Experiment artifacts record all 15 coverage families and unresolved counts
    by stable reason for the exact real generation.
  • Report validation rejects a reversed intent, a mismatched expected owner, and
    claimed toolchain/provenance that cannot describe the served producer.
  • Deterministic positive, fallback, static, timeout, missing-provenance, and
    historical-result cases pass without fixture-name branches.
  • pnpm --filter @samchon/graph-benchmark test, focused experiment-contract
    tests, pnpm build, pnpm test, and pnpm coverage pass.

Coordination

The next complete index-time matrix must also exercise the provider fixes under
#73 and #74. Do not publish tracked result JSON or chart assets from a partial
matrix, and do not run a paid agent benchmark for this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions