Skip to content

Draft: prove completeness before context reduction - #30

Draft
stpku wants to merge 5 commits into
feat/task-context-tc1-planning-real-v0.1from
feat/task-context-provider-truncation-proof-v0.1
Draft

stpku wants to merge 5 commits into
feat/task-context-tc1-planning-real-v0.1from
feat/task-context-provider-truncation-proof-v0.1

Conversation

@stpku

@stpku stpku commented Aug 27, 2026 •

Copy link
Copy Markdown
Owner

Stack relationship

This PR is stacked on feat/task-context-tc1-planning-real-v0.1 / PR #21. It was temporarily retargeted to main for combined exact-head CI and has now been restored to the stacked base.

Goal

Turn the Phoenix ArcGIS 2,000-record truncation failure into a small, offline-replayable benchmark proof.

The proof answers one narrow question:

Can a smaller bounded provider response be treated as Task Context reduction before source completeness is proven?

Recorded real evidence

Ordinary bounded responses initially looked attractive:

growth broad   2,000 features   4,573,559 bytes
growth task    2,000 features   2,485,291 bytes

A naive comparison would advertise about 45.66% fewer bytes for task scope.

But both responses hit the observed 2,000-record ceiling. The later IDs-first complete retrieval for the same broad joined representation proved:

complete broad features    118,190
unique planning units           471
pages                            119
network bytes            172,127,458
complete                       true

So the ordinary broad response contained only about 1.69% of the complete feature cardinality and undercounted it by about 98.31%.

Method

Freeze a benchmark-level rule:

Completeness Before Reduction

scope
  -> provider object-ID set
  -> complete paged retrieval
  -> prove retrieved IDs == source IDs
  -> only then compare context burden

Implemented

  • compact structured fixture containing the already recorded real diagnostic evidence;
  • dependency-free offline analyzer;
  • tests that preserve the attractive naive reduction but reject it as scoreable;
  • fail-closed countertests for incomplete completeness evidence;
  • one-command offline replay CLI;
  • method report under docs/reference/.

Replay:

python -m benchmarks.tc1_real.spatial_planning.run_provider_truncation_proof

Important boundary

This PR does not make joined layer 2 a valid scoring baseline. The existing planning diagnostic separately proved it is the wrong representation because it repeats planning-unit geometry across Pop/Emp records. The formal planning experiment continues to use base layer 4 + table 13.

The order is therefore:

choose task-appropriate representation
-> prove provider completeness
-> prove semantic/entity coverage
-> only then measure context reduction

Exact-head validation

Exact head a5764d1317f59b8d6cb091ab4fd5deb9d4da1a26 passed combined main CI run #242:

workflow  geotask-core
status    completed / success

Core impact

GeoTask Core code delta      0
public schema delta          0
live provider dependency     0

This is benchmark/method evidence only.

@stpku
stpku changed the base branch from feat/task-context-tc1-planning-real-v0.1 to main August 27, 2026 06:46
@stpku
stpku changed the base branch from main to feat/task-context-tc1-planning-real-v0.1 August 27, 2026 06:57
@stpku
stpku force-pushed the feat/task-context-provider-truncation-proof-v0.1 branch from 9dd42b3 to a5764d1 Compare August 27, 2026 07:13

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant