Repository navigation
Phase 4 Step 11: complete reports and CLI milestone - #1
Merged
Merged
Conversation
Implement physical CSV/JSONL parsing, byte-snapshot inventory and SHA-256, explicit local input paths, and a lazily imported optional Parquet reader. Preserve raw values and locations; do not implement canonical normalization, provenance joins, observability classification, metrics, lineage, or reports. The Theory Owner approved the narrow Step 2 checker/test allowlist exception. Keep all later-stage modules docstring-only and add negative boundary probes. Core local suite: 167 passed. Optional real-PyArrow tests were explicitly attempted: 3 blocked by ModuleNotFoundError (PyArrow unavailable locally). Do not treat optional presence coverage as passed; no tests were xfailed or silently skipped. No workflows, dependency manifests, schemas, or hero files were changed. Stop after Step 2; do not merge into main.
…ndaries Implement PR-003 mapping declarations and 14 approved transforms, ordered row-local execution, immutable declaration snapshots, separate extras, strict JSON parsing, and content-safe mapping diagnostics. Synchronize the two phase-boundary checks under the Theory Owner's approved ongoing rule: only currently authorized modules, functions, and imports are allowed. Retain 30 docstring-only protected modules and negative boundary tests. Add PR-003 tests to the existing security workflow without removing checks. Local results before environment reset: 318 tests passed; PR-003 subset 145 passed; security subset 179 passed. Wheel and sdist builds and installed package smoke checks passed. Hosted verification remains required. Step 2 optional real-PyArrow verification remains unresolved, not counted as passed. Do not merge into main. Stop before Step 4; no metrics or analytics.
Implement row-local canonical field/type/enum checks, exact composite identity, CSV lexical null handling, explicit timezone normalization, immutable output snapshots, and stable presentation ordering. Preserve unknown/null/absent states and duplicate parent declarations; no joins, coverage, parent resolution, generation derivation, observability, metrics, reports, or simulation. Update phase-boundary checkers under the standing authorization and PR-009 / PR-016 placeholder tests under the specifically approved Step 4 exception. Retain all earlier negative boundary checks and 27 protected placeholder modules. Local candidate subset: 317 passed; local wheel build and structural checks passed before execution-environment reset. Final full hosted regression is required, including the additional header-only table checks. Do not claim the partial local checkout as the full repository regression. Real PyArrow presence verification remains outstanding. Stop before Step 5; do not merge into main.
Implement one-to-one composite identity attachment, missing-row preservation, required-field assessments, and three explicitly scoped validation fractions. Keep missing/null/unknown and source/grounding/review declarations distinct. Reject duplicates, unmatched rows and invalid present values; retain required field failures and explicit strict-warning promotions without hiding coverage. Reuse Step 4 field validators without weakening strict canonical validation. Keep shared batch/timestamp metadata separate and flag explicit conflicts. No provenance analytics, closure bounds, parent resolution, generation inference, observability classification, report, or simulation is implemented. Update exact checker definitions under the standing Phase 2 authorization. No new executable module, dependency, schema, workflow, or hero change. Local isolated candidate validation: 120 passed. Full hosted checkout regression is required; do not mislabel the local partial harness as the whole repository. Real optional PyArrow presence verification remains outstanding from Step 2. Stop before Step 6 and do not merge into main.
…ration candidate Land the prior Step 6 local candidate before Step 7. Preserve Step 5 behavior and all 27 protected placeholder modules. Add explicit chronology evidence, immediate parent validation, bounded generation declaration checks, and 144 targeted cases. No graph analysis, roots, depth, ancestry, observability, reports or simulation. Update exact traceability gates under the standing Phase 2 maintenance approval. Full hosted checkout regression required after landing; optional PyArrow presence verification remains outstanding. Do not merge into main.
… reads Add PR-017 base-relative resolution, traversal and symlink containment, remote and device syntax rejection, regular-file checks, optional per-file byte limits, strict UTF-8 loading and content-safe diagnostics. Existing table loaders never automatically open reference fields. Keep all prior validation implementations. Require a trusted stable directory tree; document POSIX no-follow descriptor hardening and Windows checked-path fallback without claiming an OS sandbox. No analysis, graph, observability, report, new dependency or run-config fields. Preserve the 27 later-layer placeholders and strengthen exact boundary checks. Add 74 targeted content cases plus six boundary injection cases; extend Security Boundary to execute the PR-017 tests. Local partial harness: 74 passed; full hosted checkout regression remains the acceptance gate. No skip/xfail. Stop before Step 8.
Implement PR-010 maximum Level 0-5 and the seven independent PR-011 input capabilities. Retain validation errors, named coverage denominators, partial provenance/lineage limits, representation compatibility and explicit scenario parameter eligibility. Hero classification reaches Level 4 without metrics. No data loading, state assignment, graph traversal, simulation, report, public CLI or new run-config schema. Model evidence presence is not certification. Preserve prior model methods and exact later-phase protections; add 12 new checker mutation cases. Real optional PyArrow verification remains outstanding. Local isolated PR-010 regression: 65 passed. Full current checkout regression and platform validation require hosted CI. Stop before Step 9; do not merge main.
Add validate_bundle to orchestrate existing ingestion, mapping, canonical normalization, provenance attachment, chronology, immediate parent and generation validation, optional explicit local-content reads, and observability eligibility. Preserve missing/null/unknown distinctions, declared identity and input bytes. Retain incomplete provenance and family-specific failures as errors, not passing analysis. No metric, graph traversal, simulation, report or CLI implementation. Add internal bundle handoff and row mapping evidence, with physical line metadata. Keep every existing validator function unchanged. Restrict adjacent-layer imports and file-reading calls to precise invocation sites in the phase checker. Add complete Hero and synthetic integration coverage and negative boundary tests. Local syntax/static checks only; full hosted checkout regression is required. Optional real PyArrow presence validation remains outstanding from Step 2. Stop before Step 10. Do not merge main.
…delivery Synchronize input-layer documentation and add complete core/platform, real-extra, security, Hero-contract and clean installed-wheel gates. Retain JUnit and raw logs, verify exact approved Phase 0 hashes, unchanged Step 9 package/schema/Hero bytes, and assemble a checksummed tracked-source archive plus wheel and sdist. No package implementation, dependency/version declaration, analytical metric, general graph, report renderer, simulation, public CLI or main merge is changed. Completion records remain CANDIDATE pending actual hosted gate results. Phase 3 is not authorized.
…tion request Document actual core and real-Parquet outcomes, the pre-existing draft-state VALIDATION_PLAN.md mismatch, and the two-line restoration required. Do not alter the draft file, approved manifest, strict gate or package implementation. Milestone remains BLOCKED BY DECISION; no completed archive or Phase 3 claim.
… owner authorization Restore VALIDATION_PLAN.md byte-for-byte from the approved Phase 1 archive. Retain the original PHASE_0_APPROVAL.md digest and every substantive requirement. The only document changes are the two previously identified approval-state markers. Allow only this exact before/after SHA-256 transition in the Step 10 diff gate; reject alternate paths, additions, deletions, mutated bytes and unchanged draft. Preserve all original tests and add eight narrow restoration-gate cases. No src, schemas, Hero, dependency, version, permissions or runtime changes. Rerun the complete Phase 2 gates before finalizing milestone records. Do not merge main, publish a release, or begin Phase 3.
…pass Document the accepted a9e4b5c input-only implementation and exact baseline restoration. Record 1159 core passes, 1162 with real PyArrow, successful four-platform CI, Security/Hero workflows, wheel/sdist checks, isolated installed-wheel validation, and byte-verified repository archive. Preserve initial failure history and limits. Only three milestone Markdown records change. No package, schemas, examples, tests, CI, dependency, version, tag, registry publication or main merge changes. Regenerate the delivery artifact from this documentation commit and rerun gates. Phase 3 is not authorized.
…ontracts Adopt the exact approved PHASE_3_PLAN and P3-D01 through P3-D10. Add immutable typed calculation declarations, unavailable metadata and independent rational oracle targets only. Preserve all existing Phase 2 runtime definitions and all inherited test identities. Parameterize current-step audits, compare to the immutable Phase 2 file/test trees, and retain complete core and real-Parquet evidence. Build only an intermediate Step 1 candidate; no Phase 3 completion or publication. No metric, state assignment, RNG execution, report, graph or Step 2 behavior is implemented. Do not merge main. Stop after Step 1 verification.
The first full core and real-Parquet suites passed, but the independent Hero job still called the default historical Phase 2 diff gate and correctly failed. Select --phase 3 --step 1 explicitly and retain every Hero structural check. Strengthen the existing workflow boundary test to reject default --diff calls. No test identities removed and no runtime, authority, schema, fixture or metric change. Rerun all hosted checks and rebuild the intermediate candidate. Stop before Step 2.
Add pure topic/label selection, immutable assignments and explicit missing-state policies with collision checks, selected scope, exclusions and named coverage. Preserve native missing/null/unknown/empty distinctions and source record bytes. No metric, content hash, sampler, graph, report or new CLI is implemented. Retain all accepted Phase 2 and Step 1 test identities and frozen source anchors. Advance exact current-stage gates, workflow scope and installed input smoke. Apply only the Theory Owner approved one-expression contract-test exception. Local full core suite passed 1429 tests; require full hosted core/real-Parquet, four-platform CI and candidate build/installed checks on this exact commit. Stop before Step 3. Do not merge main or publish.
Implement explicit exact_utf8_v1 in-memory content states and exact groups, duplicate record counts and duplicate group counts with scoped observed-fact metadata. Preserve all source records and input evidence. LOCAL_REF requires explicitly supplied text; paths are never read or hashed by this layer. Reject artificial digest collisions between unequal bytes, unsupported modes, missing content and implicit version pooling. No semantic or origin inference. Retain the Phase 2, Step 1 and Step 2 checkpoints and all inherited test identities. Advance only the exact Step 3 module, call, arithmetic, file and workflow gates. Keep the historical Step 2 contract-test exception byte-exact and unchanged. Local full core regression passed 1575 tests on the exact tree. Require full hosted core, real Parquet, security, Hero and packaged-install verification. Stop before Step 4; no main merge, final phase delivery, tag or publication.
Implement T1 F-001 through F-004 with explicit representation/scope metadata, strict count/probability validation, deterministic ordering and separate opt-in weighted companions. Preserve zero-mass states, excluded records and original provenance denominators. No clipping, normalization, inference or sampling. Retain all accepted Phase 2 and Phase 3 Step 1-3 source/test anchors. Maintain historical step permission tests against their original scopes, while admitting only the current reviewed distribution module. Require full hosted core, real-Parquet, security, Hero, build and installed-wheel verification. Local execution was interrupted by environment resets; final acceptance must use this exact committed tree and retained workflow artifacts. No full-phase completion, main merge, publication or Step 5 work is authorized.
Recomputed from the exact reviewed runtime blob 79deeb2. Replace the malformed 67-character literal with its actual 64-character SHA-256. Runtime source and all tests are unchanged; strict mutation rejection remains required. Re-run complete acceptance; stop before Step 5.
Request full-field dumps on Python 3.13+, keep older full-field behavior, and retain the empty-type-params-only compatibility rule. The reviewed runtime source and mathematical tests remain unchanged. Require all original mutation rejections and rerun complete acceptance. No Step 5 or publication.
Add explicitly scoped PR-005 source counts/shares, PR-004 confidence and missing-row facts, T3 direct grounding input basis, and opt-in weighted source companions. Preserve Phase 2 validation evidence and all prior runtime modules. No closure intervals, ancestry, reports, simulation, main merge or publication. Record accepted Step 4 checkpoint, exact Step 5 file permissions and test migrations. Retain all 1714 inherited core identities; local candidate suite: 1888 passed. Require full hosted core, real-Parquet, security, Hero, build and installed-wheel acceptance. Stop before Step 6.
Add explicit T3 F-009/F-010 lower and upper bounds and interval width, with a separately evidence-checked dataset adapter. Preserve coverage, confidence disclosure, missingness and original input errors. No lineage, midpoint, threshold, simulation, report or automatic input-pipeline execution. Retain prior runtime modules, authority hashes and inherited test identities. Update only approved Step 6 files and gates. Local core regression: 1987 passed; require complete hosted core, real-Parquet, security, Hero, build and installed-wheel acceptance. Stop before Step 7; no main merge or publication.
Add T2 explicit observed-tail selection, stable rarity ranks, tail support and record share, and separately invoked F-014 one-step closed-resampling probability. Retain declared scope, representation, unavailable reasons and simulation evidence; disclose numerical underflow. No random sampling, reopening, graph, risk narrative, report or automatic validation-pipeline execution. Preserve all prior runtime files, authority hashes and inherited test identities. Exact local core regression: 2117 passed. Require complete hosted matrix, real-Parquet, security, Hero, build and installed-wheel gates. Stop before Step 8. No main merge or publication.
Add T1 F-015 fixed-size expected diversity and separately invoked PCG64 closed categorical sampled paths. Preserve explicit distribution, representation, scope, seed, replicate and method evidence, disclose bounded input round-off corrections and numerical underflow, and enforce work limits before allocation. Retain all earlier runtime sources and historical tests. No external-reference loss, reopening, general graph, pair comparison, report, automatic validation-pipeline simulation, main merge or publication. Local complete regression: 2268 passed. Require full hosted matrix, actual Parquet, security, Hero, build and installed-wheel validation. Stop before Step 9.
Add directed declaration checks and pure F-005/F-006/F-018 comparisons for one explicitly selected ordered pair. Retain original and harmonized distributions, missingness, weighting and numerical trace metadata; disclose many-to-one coarsening. Reject inferred chronology, ambiguous maps, incompatible bases and forged summary evidence. All preexisting single-scope definitions, frozen specifications, schemas, Hero inputs, math oracles and dependency declarations are preserved. Local final full core regression: 2404 passed; authority and boundary checks passed. Require complete hosted matrix, real Parquet, Security, Hero, build and installed-wheel validation before acceptance. No longitudinal dispatcher, lineage/model deltas, report, audit CLI, main merge or publication. Stop before Step 10.
…rformance Keep all 40 runtime modules byte-identical to accepted Step 9. Add test-only mapped-input/Hero composition, execute all 20 unchanged mathematical oracles, and measure 100000 synthetic records plus the Hero calculation path with disclosed timing and traced-memory scope. Preserve historical test identities, approval hashes, schemas, Hero, dependencies and version. Require executed golden and performance evidence, exact current-step file permissions, full hosted matrix, real Parquet, Security, Hero, build and installed-wheel checks. Local complete core regression: 2472 passed; no failure or skip. No runtime repair, report orchestration, lineage, reopening, main merge, publication or Phase 3 completion. Stop before Step 11.
Synchronize dev2 documentation and exact stage guards, preserve historical tests under approved maintenance exceptions, and verify current/minimum-compatible dependency profiles. Final acceptance remains pending all workflow roles.
Record all twelve successful jobs, preserved baseline identities, dependency profiles, reviews, limitations and verified artifacts. Pin the accepted implementation and require all workflow roles to rerun on this finalization commit. No runtime, formula, oracle or dependency change.
Source tree verified identical to locally tested candidate 5aa9cd6d06bfffec9cc280a52277aa9152a8627e. Published through the connected GitHub account after explicit user approval.
Freeze immutable reports, 90 analytical field registrations and the local JSON Schema. Preserve inherited kernels and introduce explicit Step 2 gates. Local validation: 2728 core, 2728 minimum-dependency and 2731 real-Parquet tests passed; isolated builds and installed contract checks passed. Source tree matches reviewed local candidate c3ce887. Approved Phase 4 Step 2 publication to the existing phase4-reports-cli branch only.
Apply the explicitly approved report assembly repair, retain diagnostic and privacy semantics, and add complete Hero and 100k measurements with bounded instrumentation. Consolidate current scope verification around the accepted Git baseline and use focused push CI; full compatibility and release gates remain available at candidates.
DavidWallstructurallaw
changed the title
DavidWallstructurallaw
marked this pull request as ready for review
October 2, 2026 06:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 4 is accepted at development version
0.1.0.dev3. This PR completes the reports and CLI milestone, updates public usage documentation, and records verified behavior and measured limits.The accepted code candidate is
ceb0cf702284dbdbe50b14722026ec26fbcc21ba, tree05a691710f8622b1f816e5a1e26b80d8b7e58069. All 12 required jobs passed:CI and delivery, Hero, and security completed successfully on that exact candidate. Every matrix profile retained the actual 100,000-record audit.
Earlier failed and cancelled candidates remain recorded. Historical preconditions were bound to existing accepted fixtures without removing assertions. A demonstrated 75-minute job-budget shortfall was corrected to 105 minutes; the 100k subprocess limit remains 1,800 seconds.
Final administrative head
1db3b1a460c233242ff37fbe45b8fac74d6101efchanges only the completion, validation and architecture reports. The other 243 Git tree entries retain identical mode, type and blob identities. Focused document, scope, specification and traceability checks passed. Its[skip ci]commit reuses the accepted candidate's results under the user's verification governance policy; it does not claim another matrix run.Completion, validation, and architecture review describe the accepted scope and limitations.
This PR remains a draft and unmerged. Skipped required checks on the administrative head may remain pending. No merge-ready claim, tag, package publication or Phase 5 authorization is made.