Unify mapped reads, conversions and output storage

Cargo.toml

@@ -7,12 +7,12 @@ version = "0.1.59"
 edition = "2024"
 
 [workspace.dependencies]
-rustymath = { path = ".", version = ">=0.1.56, <1", default-features = false }
+rustymath = { path = ".", version = ">=0.1.59, <1", default-features = false }
 rustymath-buffer = { path = "buffer", version = ">=0.1.39, <1" }
 pyo3 = ">=0.29"
 pyo3-build-config = ">=0.29"
-rustymath-core = { path = "core", version = ">=0.1.56, <1", default-features = false }
-rustymath-kernels = { path = "kernels", version = ">=0.1.56, <1" }
+rustymath-core = { path = "core", version = ">=0.1.59, <1", default-features = false }
+rustymath-kernels = { path = "kernels", version = ">=0.1.59, <1" }
 rustymath-timing = { path = "timing", version = ">=0.1.56, <1" }
 half = { version = ">=2.7.1, <3", features = ["num-traits"] }
 hashbrown = ">=0.15, <1"

DEV.md

@@ -8,9 +8,9 @@ Rustymath drops precision that numerical code doesn't need, for speed and a smal
 
 Keep the documented accuracy bounds that callers use. `nbs/accuracy.qmd` lists every exception in one place. A few ulps more, for real speed or a simpler implementation, become a documented exception there, not a reason to keep slower code. Check a change that alters arithmetic against those contracts, not against bit-identical results. Don't add sign canonicalization, compensated arithmetic or defensive passes without an agreed need. Each shortcut that drops precision is a hypothesis that real uses don't need that precision. When a real use needs digits that a shortcut loses, return that operation to the conservative method, or create an optional parameter.
 
-Rustymath and basedpl have no outside users yet, so treat neither as published. Machinery is shared when it has two users: basedpl and rustymath's Python API. Changing rustymath to give them shared machinery is encouraged. Separate machinery in basedpl and rustymath for the same task is an error, unless Jeremy has asked for them to differ there.
+Rustymath is BPL's mathematics and array core, with a useful thin Python interface over the same algorithms. Generic numerical and array algorithms belong here; parsing, language/function semantics, fusion, JIT, batching and Program construction belong to BPL. Choose ownership by the work an algorithm performs, not by whether BPL currently calls a particular convenience wrapper. Consolidate duplicate general algorithms in RM, retaining the better implementation from either project. Rustymath and basedpl have no outside users yet; neither API is treated as published. Discuss public Python removals before making them.
 
-Development and planning run on Syd1 under `/home/ubuntu/aai-ws/{basedpl,rustymath}`. Both projects' `meta/` directories are maintained there and stay outside Git. Start at `../basedpl/meta/open.md` for the shared repair process. Mac copies are snapshots; never sync them over newer Syd1 work.
+Read `../basedpl/meta/open.md` for the current working host, authoritative source, builds, plans and evidence. Both projects' `meta/` directories stay outside Git.
 
 ## Commands and builds
 
@@ -25,7 +25,7 @@ nbdev-test
 nbdev-preview
 ```
 
-Use `cargo develop` to rebuild and install the Python extension after Rust changes. It builds the `release` profile, which development, tests and timings all use; `fastws.cargo.run_tests(root, ...)` builds it too. Bare `cargo test` uses Cargo's built-in `test` profile, which has `release`'s settings but builds again; pass `--release`. Cargo tests compile missing code but do not install it. An already-running Python process retains its loaded extension. `cargo nextest run --cargo-profile release` runs each test in its own process and every test binary at once. `.config/nextest.toml` reports a test as slow at 5 s, stops it at 30 s, and runs `speed` last with nothing beside it. Its `untimed` and `speed` profiles split the speed test from the rest. basedpl's default test command, `fastws.cargo.run_tests([basedpl, rustymath], profiles=('untimed', 'timed', 'speed'))`, runs each profile in turn, at once in each repository that defines it, so `speed` runs after basedpl's timings, with nothing beside it.
+Use `cargo develop` to rebuild and install the Python extension after Rust changes. It builds the `release` profile, which development, tests and timings all use; `fastws.cargo.run_tests(root, ...)` builds it too. Bare `cargo test` uses Cargo's built-in `test` profile, which has `release`'s settings but builds again; pass `--release`. Cargo tests compile missing code but do not install it. An already-running Python process retains its loaded extension. `cargo nextest run --cargo-profile release` runs each test in its own process and every test binary at once. `.config/nextest.toml` reports a test as slow at 5 s, stops it at 30 s, and runs `speed` last with nothing beside it. Its `untimed` and `speed` profiles split the speed test from the rest. The shared default command is `fastws.cargo.run_tests([basedpl, rustymath])`. It uses each repository's default nextest profile, including timings and counts. Explicit profile selection is for focused diagnostics.
 
 `cargo nextest run --cargo-profile release` and `pytest -q` include the speed tests, `tests/speed.rs` and `tests/test_speed.py`. Their ordinary runs time each case beside builds and other tests, under a shared hold on the timing lock and on one thread. They also check each case's recorded counts of instructions and allocated bytes for each item, which don't depend on what else the machine runs. A case over its budget is timed again under an exclusive hold where no one else holds the lock, and is reported `unconfirmed`, without failing, where someone does. `RUSTYMATH_COSTS=1` makes either test a calibration run, as "Speed measurements" describes. Use one test binary for a focused check. Run `pytest -q` at the end of a piece of work. Before reporting a failure in untouched code, rebuild with the same profile and rerun it. Builds have no execution timeout. Use the documented managed runners with a 30-second test execution budget. Cargo serializes builds in the same target directory.
 
@@ -85,7 +85,7 @@ A generic function compiles once for each combination of types that callers give
 
 ## Code map
 
-The dependency layers are `rustymath-core` → `rustymath-kernels` → the public `rustymath` facade; the facade also uses core directly. Core owns formulas, number facts, storage and geometry together, with no dependency on executable kernels. Kernels owns concrete typed execution. Existing public numerical paths are facade reexports, not duplicate implementations. Files below are relative to `core/src/` unless qualified.
+The dependency layers are `rustymath-core` → `rustymath-kernels` → the public `rustymath` facade; the facade also uses core directly. Core owns formulas, number facts, storage and geometry, without depending on executable kernels. Kernels owns concrete typed execution. The facade explicitly exports math/array capabilities, scalar formulas and their public traits/macros; kernel-only helpers stay in the lower crates. BPL owns language evaluation, Program/trace/fusion/JIT policy and diagnostic labels. Files below are relative to `core/src/` unless qualified.
 
 | Files | Responsibility |
 |---|---|
@@ -96,13 +96,14 @@ The dependency layers are `rustymath-core` → `rustymath-kernels` → the publi
 | `slice.rs`, `parallel.rs` | Scalar/slice object interfaces, output storage and elementwise scheduling. |
 | `reduce.rs`, `stats.rs` | Folds, scans, axis traversal and statistical summaries. |
 | `facts.rs` | What a planner knows about each function object before it reads an item: the `Range` of its results, from its arguments' ranges by interval rules, whether any result fails, and which argument a constant keeps (`Facts`). Also the one table of which reduction folds each function and which objects take characters, which rustymath, its Python crate and basedpl each expand. |
-| `structure.rs` | Structural selection plans. `structure.rs` also holds each structural form's shape rule and the steps that read its result (`take_steps`, `rotate_steps`, `select_steps` and the rest), written once over the `Lengths` trait: `Known` computes on numbers, and `Bounded` on a trace's length variables, recording each decision for later runs to check (`evaluate`). Each concrete plan calls them with `Known`. A recording calls them with `Bounded` for a template, `Steps<Traced>`, which each run fills at its lengths (`Steps::at`). |
+| `structure.rs` | Concrete shape rules and positional plans for indexing, take/drop, rotation, windows, transpose and catenation. Shape functions take `usize` dimensions; `take_steps`, `rotate_steps` and related functions build the `Steps` that read each result. `Walk` and `Slide` recognize shared structural geometry. Symbolic lengths, trace decisions and Program construction belong to BPL. |
 | `arith.rs`, `integer.rs`, `convert.rs` | Wrapped/checked arithmetic, integer functions and vectorizable conversions. |
 | `distribution.rs`, `random.rs` | Checked distributions, function objects, sampling, rolls and deals. |
 | `number_theory.rs` | Primality, factorization, prime enumeration/counting and factorial. |
 | `field.rs`, `exact.rs`, `linalg/exact.rs`, `polynomial.rs` | Generic field algorithms, rational functions, exact matrices and polynomials. |
 | `search.rs`, `tolerant.rs`, `gather.rs` | Exact/tolerant search and classification, stable sorting, partial selection, comparisons and copying through conversions. |
 | `error.rs`, `layout.rs`, `array.rs` | Checked array errors; shapes, axis keys, names and size limits; and the borrowed `ArrayView` over source bindings. |
+| `events.rs` | Mathematical routes and one generic capture transport (`Record`, schema-owned rows, `Run`, `Sink`). A producer owns its vocabulary and typed row construction; BPL's tables and language routes live in BPL `src/events.rs`. |
 | `optimize.rs` | Minimization by five methods behind one `Objective` trait, and solving behind one `Mapping` trait: Brent's root finder and Levenberg–Marquardt for nonlinear equations, and CG, MINRES and GMRES for linear ones. |
 | `linalg.rs`, `linalg/scalar.rs`, `linalg/factor/` | Packed products, scalar operations and decompositions. |
 | `signal.rs` | Correlations and convolutions, which slide a kernel along leading axes through `linalg`'s mapped products, and overlap-add. |
@@ -250,7 +251,7 @@ The native SVD, Hermitian and general solvers compile everywhere. On macOS the S
 
 ## Python boundary
 
-A change that adds a user-facing public Rust function also adds its Python binding, following the existing API's names, argument order and dtypes. Every binding follows the conventions of the Rust core that BasedPL shares. Related bindings agree with each other. A function with NumPy's name needn't behave as NumPy's does. `remainder` and `floor_divide` round through the comparison tolerance where one is set, set functions keep first-occurrence order, orderings put NaN last, and `diagonal` and `trace` take the last two axes, as the matrix functions do.
+Expose useful mathematical and array capabilities through a thin Python interface to the shared Rust algorithms. A public Rust helper does not require a matching Python binding. Keep Python argument decoding, defaults, scalar-return policy and result wrappers in `py/src/` or `python/rustymath/`; keep generic mathematics and array algorithms in RM even when a Python convenience is their only direct caller. Related bindings share the core's names, argument order and dtypes. A function with NumPy's name needn't behave as NumPy's does. `remainder` and `floor_divide` round through the comparison tolerance where one is set, set functions keep first-occurrence order, orderings put NaN last, and `diagonal` and `trace` take the last two axes, as the matrix functions do.
 
 Python operations are eager. `python/rustymath/create.py` gives fill constructors their names/defaults over one native fill primitive, and names the dtype facts that Rust returns. Sequence generation stays in `create`. Structural functions and bracket indexing in `py/src/structure.rs` delegate selector decoding to `buffer/src/selection.rs`, shared with BasedPL, for the selection and writeback paths. Brackets plan through `structure::plan_parts`, the planner that BasedPL's `⌷` and Python indexing share, with broadcasting, as BasedPL's `.[]` does: index arrays pair by broadcasting, by arithmetic's rule for names. A selector with one axis name selects along the axis with that name (`structure::addressed`). A selection by strides, padding or rotations is a view that shares storage, and a selection by index arrays copies its items (`Array::selected`). `reshape` keeps a view's mapping, which reads the same items in order under any layout. A view whose mapping isn't affine, such as padded windows or a reshape of windows, keeps the items that its first whole read gathers, and later reads use them (`ArrayView::keeps`). An affine view, such as a slice or an affine reshape of one, keeps nothing, and a reader that can't use its strides gathers it again. A small selection that doesn't compose with a view's mapping lists its offsets, and a larger one gathers the view first (`agreement::small_read`, which BasedPL's reads of pending arrays share). `take`, `pad` and `windows` read positions past an end by `structure::Edge`'s modes, which BasedPL's `•edge` names: `take` through `Edge::positions`, and `pad` and padded windows through `Steps::padded`. `structure::Windows` plans windows for `windows` as it does for BasedPL's `↕`. A `fill=` is a `Fill` checked at the array's dtype (`Fill::new`), which the selection's mapping reads (`Mapping::filled`). Assignment detaches shared storage, preserving independent values. `rm.at` combines through the same selection with `Array::combine_at` and its function's `Scatter` kernels, sharing `Array::assign`'s checks. A view retains its whole base until `compact()` copies its logical items, beside any items it has gathered, and even when it is a contiguous subrange. Methods call the same binding functions as package-level operations. `python/rustymath/structure.py` composes conveniences such as trace and repeated differences. Differences run subtraction kernels on two mappings of the input, not gathered copies. `python/rustymath/random.py` subclasses the native generator for distribution conveniences and multivariate-normal sampling. `py/src/random.rs` owns its RNG. Sampling, rolls and deals stay in the shared core.