diff --git a/ABI-FFI-README.md b/ABI-FFI-README.adoc similarity index 74% rename from ABI-FFI-README.md rename to ABI-FFI-README.adoc index b2a1605..963311c 100644 --- a/ABI-FFI-README.md +++ b/ABI-FFI-README.adoc @@ -1,19 +1,22 @@ -{{~ Aditionally delete this line and fill out the template below ~}} +\{\{~ Aditionally delete this line and fill out the template below ~}} -# VAE_NORMALIZER ABI/FFI Documentation +== VAE_NORMALIZER ABI/FFI Documentation -## Overview +=== Overview -This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: +This library follows the *Hyperpolymath RSR Standard* for ABI and FFI +design: -- **ABI (Application Binary Interface)** defined in **Idris2** with formal proofs -- **FFI (Foreign Function Interface)** implemented in **Zig** for C compatibility -- **Generated C headers** bridge Idris2 ABI to Zig FFI -- **Any language** can call through standard C ABI +* *ABI (Application Binary Interface)* defined in *Idris2* with formal +proofs +* *FFI (Foreign Function Interface)* implemented in *Zig* for C +compatibility +* *Generated C headers* bridge Idris2 ABI to Zig FFI +* *Any language* can call through standard C ABI -## Architecture +=== Architecture -``` +.... ┌─────────────────────────────────────────────┐ │ ABI Definitions (Idris2) │ │ src/abi/ │ @@ -45,11 +48,11 @@ This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: │ Any Language via C ABI │ │ - Rust, ReScript, Julia, Python, etc. │ └─────────────────────────────────────────────┘ -``` +.... -## Directory Structure +=== Directory Structure -``` +.... vae_normalizer/ ├── src/ │ ├── abi/ # ABI definitions (Idris2) @@ -77,15 +80,17 @@ vae_normalizer/ ├── rust/ ├── rescript/ └── julia/ -``` +.... -## Why Idris2 for ABI? +=== Why Idris2 for ABI? -### 1. **Formal Verification** +==== 1. *Formal Verification* -Idris2's dependent types allow proving properties about the ABI at compile-time: +Idris2’s dependent types allow proving properties about the ABI at +compile-time: -```idris +[source,idris] +---- -- Prove struct size is correct public export exampleStructSize : HasSize ExampleStruct 16 @@ -97,13 +102,14 @@ fieldAligned : Divides 8 (offsetOf ExampleStruct.field) -- Prove ABI is platform-compatible public export abiCompatible : Compatible (ABI 1) (ABI 2) -``` +---- -### 2. **Type Safety** +==== 2. *Type Safety* Encode invariants that C/Zig cannot express: -```idris +[source,idris] +---- -- Non-null pointer guaranteed at type level data Handle : Type where MkHandle : (ptr : Bits64) -> {auto 0 nonNull : So (ptr /= 0)} -> Handle @@ -111,13 +117,14 @@ data Handle : Type where -- Array with length proof data Buffer : (n : Nat) -> Type where MkBuffer : Vect n Byte -> Buffer n -``` +---- -### 3. **Platform Abstraction** +==== 3. *Platform Abstraction* Platform-specific types with compile-time selection: -```idris +[source,idris] +---- CInt : Platform -> Type CInt Linux = Bits32 CInt Windows = Bits32 @@ -125,13 +132,14 @@ CInt Windows = Bits32 CSize : Platform -> Type CSize Linux = Bits64 CSize Windows = Bits64 -``` +---- -### 4. **Safe Evolution** +==== 4. *Safe Evolution* Prove that new ABI versions are backward-compatible: -```idris +[source,idris] +---- -- Compiler enforces compatibility abiUpgrade : ABI 1 -> ABI 2 abiUpgrade old = MkABI2 { @@ -140,71 +148,78 @@ abiUpgrade old = MkABI2 { -- Can add new fields new_features = defaults } -``` +---- -## Why Zig for FFI? +=== Why Zig for FFI? -### 1. **C ABI Compatibility** +==== 1. *C ABI Compatibility* Zig exports C-compatible functions naturally: -```zig +[source,zig] +---- export fn library_function(param: i32) i32 { return param * 2; } -``` +---- -### 2. **Memory Safety** +==== 2. *Memory Safety* Compile-time safety without runtime overhead: -```zig +[source,zig] +---- // Null check enforced at compile time const handle = init() orelse return error.InitFailed; defer free(handle); -``` +---- -### 3. **Cross-Compilation** +==== 3. *Cross-Compilation* Built-in cross-compilation to any platform: -```bash +[source,bash] +---- zig build -Dtarget=x86_64-linux zig build -Dtarget=aarch64-macos zig build -Dtarget=x86_64-windows -``` +---- -### 4. **Zero Dependencies** +==== 4. *Zero Dependencies* No runtime, no libc required (unless explicitly needed): -```zig +[source,zig] +---- // Minimal binary size pub const lib = @import("std"); // Only includes what you use -``` +---- -## Building +=== Building -### Build FFI Library +==== Build FFI Library -```bash +[source,bash] +---- cd ffi/zig zig build # Build debug zig build -Doptimize=ReleaseFast # Build optimized zig build test # Run tests -``` +---- -### Generate C Header from Idris2 ABI +==== Generate C Header from Idris2 ABI -```bash +[source,bash] +---- cd src/abi idris2 --cg c-header Types.idr -o ../../generated/abi/vae_normalizer.h -``` +---- -### Cross-Compile +==== Cross-Compile -```bash +[source,bash] +---- cd ffi/zig # Linux x86_64 @@ -215,13 +230,14 @@ zig build -Dtarget=aarch64-macos # Windows x86_64 zig build -Dtarget=x86_64-windows -``` +---- -## Usage +=== Usage -### From C +==== From C -```c +[source,c] +---- #include "vae_normalizer.h" int main() { @@ -237,16 +253,19 @@ int main() { vae_normalizer_free(handle); return 0; } -``` +---- Compile with: -```bash + +[source,bash] +---- gcc -o example example.c -lvae_normalizer -L./zig-out/lib -``` +---- -### From Idris2 +==== From Idris2 -```idris +[source,idris] +---- import VAE_NORMALIZER.ABI.Foreign main : IO () @@ -259,11 +278,12 @@ main = do free handle putStrLn "Success" -``` +---- -### From Rust +==== From Rust -```rust +[source,rust] +---- #[link(name = "vae_normalizer")] extern "C" { fn vae_normalizer_init() -> *mut std::ffi::c_void; @@ -282,11 +302,12 @@ fn main() { vae_normalizer_free(handle); } } -``` +---- -### From Julia +==== From Julia -```julia +[source,julia] +---- const libvae_normalizer = "libvae_normalizer" function init() @@ -312,27 +333,30 @@ try finally cleanup(handle) end -``` +---- -## Testing +=== Testing -### Unit Tests (Zig) +==== Unit Tests (Zig) -```bash +[source,bash] +---- cd ffi/zig zig build test -``` +---- -### Integration Tests +==== Integration Tests -```bash +[source,bash] +---- cd ffi/zig zig build test-integration -``` +---- -### ABI Verification (Idris2) +==== ABI Verification (Idris2) -```idris +[source,idris] +---- -- Compile-time verification %runElab verifyABI @@ -342,44 +366,44 @@ main = do verifyLayoutsCorrect verifyAlignmentsCorrect putStrLn "ABI verification passed" -``` +---- -## Contributing +=== Contributing When modifying the ABI/FFI: -1. **Update ABI first** (`src/abi/*.idr`) - - Modify type definitions - - Update proofs - - Ensure backward compatibility - -2. **Generate C header** - ```bash - idris2 --cg c-header src/abi/Types.idr -o generated/abi/vae_normalizer.h - ``` - -3. **Update FFI implementation** (`ffi/zig/src/main.zig`) - - Implement new functions - - Match ABI types exactly - -4. **Add tests** - - Unit tests in Zig - - Integration tests - - ABI verification tests - -5. **Update documentation** - - Function signatures - - Usage examples - - Migration guide (if breaking changes) - -## License +[arabic] +. *Update ABI first* (`+src/abi/*.idr+`) +* Modify type definitions +* Update proofs +* Ensure backward compatibility +. *Generate C header* ++ +[source,bash] +---- +idris2 --cg c-header src/abi/Types.idr -o generated/abi/vae_normalizer.h +---- +. *Update FFI implementation* (`+ffi/zig/src/main.zig+`) +* Implement new functions +* Match ABI types exactly +. *Add tests* +* Unit tests in Zig +* Integration tests +* ABI verification tests +. *Update documentation* +* Function signatures +* Usage examples +* Migration guide (if breaking changes) + +=== License MPL-2.0 -## See Also +=== See Also -- [Idris2 Documentation](https://idris2.readthedocs.io) -- [Zig Documentation](https://ziglang.org/documentation/master/) -- [Rhodium Standard Repositories](https://github.com/hyperpolymath/rhodium-standard-repositories) -- [FFI Migration Guide](../ffi-migration-guide.md) -- [ABI Migration Guide](../abi-migration-guide.md) \ No newline at end of file +* https://idris2.readthedocs.io[Idris2 Documentation] +* https://ziglang.org/documentation/master/[Zig Documentation] +* https://github.com/hyperpolymath/rhodium-standard-repositories[Rhodium +Standard Repositories] +* link:../ffi-migration-guide.md[FFI Migration Guide] +* link:../abi-migration-guide.md[ABI Migration Guide] diff --git a/ARCHITECTURE.adoc b/ARCHITECTURE.adoc new file mode 100644 index 0000000..1c0a7a6 --- /dev/null +++ b/ARCHITECTURE.adoc @@ -0,0 +1,48 @@ +== Architecture + +=== Overview + +This repository follows a modular, maintainable architecture designed +for clarity, scalability, and long-term sustainability. + +=== Directory Structure + +.... +. +├── src/ # Source code +├── tests/ # Test suites +├── docs/ # Documentation +├── scripts/ # Utility scripts +├── config/ # Configuration files +├── LICENSE # License file +├── LICENSES/ # Full license texts +└── README.adoc # Project documentation +.... + +=== Design Principles + +* *Separation of Concerns*: Each module has a single responsibility +* *Testability*: Code is written to be easily testable +* *Documentation*: All public APIs are documented +* *Configuration*: Environment-specific settings are externalized + +=== Dependencies + +* External dependencies are minimized and clearly declared +* Version pinning is used for reproducibility + +=== Security Considerations + +* Sensitive data is never committed to the repository +* Secrets are managed through environment variables or secure vaults +* Regular dependency audits are performed + +=== Maintainability + +* Code follows consistent style guidelines +* Pull requests require review and CI checks +* Issues and discussions are tracked transparently + +''''' + +_Last updated: 2026-07-18_ diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index 607e3d8..0000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,47 +0,0 @@ -# Architecture - -## Overview - -This repository follows a modular, maintainable architecture designed for clarity, scalability, and long-term sustainability. - -## Directory Structure - -``` -. -├── src/ # Source code -├── tests/ # Test suites -├── docs/ # Documentation -├── scripts/ # Utility scripts -├── config/ # Configuration files -├── LICENSE # License file -├── LICENSES/ # Full license texts -└── README.adoc # Project documentation -``` - -## Design Principles - -- **Separation of Concerns**: Each module has a single responsibility -- **Testability**: Code is written to be easily testable -- **Documentation**: All public APIs are documented -- **Configuration**: Environment-specific settings are externalized - -## Dependencies - -- External dependencies are minimized and clearly declared -- Version pinning is used for reproducibility - -## Security Considerations - -- Sensitive data is never committed to the repository -- Secrets are managed through environment variables or secure vaults -- Regular dependency audits are performed - -## Maintainability - -- Code follows consistent style guidelines -- Pull requests require review and CI checks -- Issues and discussions are tracked transparently - ---- - -*Last updated: 2026-07-18* diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index 40266f2..c495d09 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -1,102 +1,74 @@ - - - -= Changelog - -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -== [Unreleased] - -=== Added -- RSR (Rhodium Standard Repository) compliance -- Full documentation suite -- **Diff-based compression**: Store VAE images as diffs to reduce dataset size by ~50% - - New `compress` subcommand to convert VAE images to diffs - - New `decompress` subcommand to reconstruct full VAE directory - - New `reconstruct` subcommand to reconstruct single VAE image - - Julia `CompressedVAEDataset` for on-the-fly VAE reconstruction - - Diff encoding: `diff = VAE - Original + 128` (offset for signed values) -- **Contrastive learning model** for VAE artifact detection (`contrastive_model.jl`) - - CNN encoder with ResNet-style residual blocks (embed_dim=256) - - Projection head for contrastive learning (proj_dim=128) - - Multiple loss functions: NT-Xent, Supervised Contrastive, Triplet, Contrastive - - Two-phase training: contrastive pre-training + classifier fine-tuning - - Binary classifier for original vs VAE discrimination - - Embedding extraction for visualization (t-SNE, UMAP compatible) - - Evaluation metrics: accuracy, precision, recall, F1, confusion matrix - - GPU support via CUDA.jl (optional) - - Justfile recipes: `train`, `train-compressed`, `evaluate`, `embed`, `train-full` - -== [1.0.0] - 2024-01-01 - -=== Added - -==== Core Features -- SHAKE256 (d=256) cryptographic checksums for all image files -- Train/test/validation/calibration splits (70/15/10/5 ratio) -- Both random and stratified split generation -- Size-based stratification with configurable bins - -==== Metadata & Configuration -- CUE metadata generation with Dublin Core compliance -- Nickel configuration schema -- SPDX headers on all source files - -==== Formal Verification -- Isabelle/HOL proofs for split properties - - Disjointness (no overlap between splits) - - Exhaustiveness (all images assigned) - - Ratio correctness (within 1% tolerance) - - Original-VAE bijection preservation - -==== CLI -- `normalize` subcommand for dataset processing -- `verify` subcommand for output validation -- `stats` subcommand for statistics display -- `hash` subcommand for single-file checksums -- Comprehensive help and examples - -==== Infrastructure -- Nix flake for reproducible builds -- Podman container with Chainguard Wolfi base -- Justfile task runner -- Julia utilities for Flux.jl training +== Changelog + +All notable changes to `+zerostep+` will be documented in this file. + +This file is generated from conventional commits by the +https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml[`+changelog-reusable.yml+`] +workflow (`+hyperpolymath/standards#206+`). Adopt the workflow in this +repo’s CI to keep this file in sync automatically — see +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+templates/cliff.toml+`] +for the canonical config. + +The format follows https://keepachangelog.com/en/1.1.0/[Keep a +Changelog]; this project aims to follow +https://semver.org/spec/v2.0.0.html[Semantic Versioning]. + +=== [Unreleased] + +==== Added + +* feat(crg): add crg-grade and crg-badge justfile recipes +* feat: add stapeln.toml layer-based container definitionfrom existing +Containerfile to stapeln format.Chainguard base, security hardening, +SBOM generation.-Authored-By: Claude Opus 4.6 (1M context) +noreply@anthropic.com +* feat: deploy UX Manifesto infrastructure +* feat: add CLADE.a2ml — clade taxonomy declaration +* feat(ci): enable Hypatia scanning + +==== Fixed + +* fix(ci): bump a2ml/k9-validate-action pins to canonical (#32) +* fix(ci): sync hypatia-scan.yml to canonical (#31) +* fix(ci): build Hypatia escript from repo root (estate dogfood drift) +* fix(ci): hypatia-scan workdir ($\{\{ env.HOME }} resolves empty) (#30) +* fix(ci): bump erlef/setup-beam SHA for ubuntu24 runner support (#23) +* fix(ci): repair YAML block-scalar in workflow-linter Check Permissions +step (#26) +* fix(ci): move secret-scanner Cargo.toml gate from job-level if: to +step-level (#28) +* fix(src/metadata.rs): remove unused std::fs::File import (#25) +* fix(codeql): switch language matrix to '`actions`' (no JS/TS in repo) +(#24) +* fix(ci): Resolve workflow-linter self-matching and metadata issues + +==== Changed + +* refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) ==== Documentation -- README with usage examples -- QUICKSTART guide -- RSR compliance documentation -- Security policy -- Contributing guidelines (TPCF) -- Governance model -=== Security -- Memory-safe Rust implementation -- No unsafe code blocks -- FIPS 202 compliant cryptography -- Non-root container execution -- Supply chain security (SPDX, pinned deps) +* docs: record tech-debt audit findings (2026-05-26) (#43) +* docs: substantive CRG C annotation (EXPLAINME.adoc) +* docs: add EXPLAINME.adoc — prove-it file backing README claims +* docs: add checkpoint files for state tracking -== Migration Guide +==== CI -=== From v0.x to v1.0.0 +* ci(rust): convert rust-ci.yml to thin wrapper (standards#174) (#40) +* ci: redistribute concurrency-cancel guard to read-only check workflows +(#34) +* ci(dependabot): restore cargo PR limit so security PRs flow (#16) +* ci(secret-scanner): drop duplicate –fail from trufflehog extra_args +(#15) +* ci: bump actions/upload-artifact SHA to current v4 (#14) -This is the initial release. No migration needed. +=== Pre-history -== Version History +Prior commits to this file’s introduction are recorded in git history +but not formally classified into Keep-a-Changelog sections. To backfill, +run `+git cliff -o CHANGELOG.md+` locally using the canonical +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+cliff.toml+`] +— this is one-shot mechanical work. -| Version | Release Date | Status | -|---------|--------------|--------| -| 1.0.0 | 2024-01-01 | Current | - -== Deprecation Policy - -- Deprecated features announced one minor version before removal -- Migration guides provided for breaking changes -- LTS branches may be created for critical security fixes - -[Unreleased]: https://huggingface.co/datasets/joshuajewell/VAEDecodedImages-SDXL/compare/v1.0.0...HEAD -[1.0.0]: https://huggingface.co/datasets/joshuajewell/VAEDecodedImages-SDXL/releases/tag/v1.0.0 +''''' diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 6ea2187..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,67 +0,0 @@ - - -# Changelog - -All notable changes to `zerostep` will be documented in this file. - -This file is generated from conventional commits by the -[`changelog-reusable.yml`](https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml) -workflow (`hyperpolymath/standards#206`). Adopt the workflow in this repo's CI to keep this file in sync automatically — see -[`templates/cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) -for the canonical config. - -The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); -this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -### Added - -- feat(crg): add crg-grade and crg-badge justfile recipes -- feat: add stapeln.toml layer-based container definition\n\nConverted from existing Containerfile to stapeln format.\nIncludes Chainguard base, security hardening, SBOM generation.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) -- feat: deploy UX Manifesto infrastructure -- feat: add CLADE.a2ml — clade taxonomy declaration -- feat(ci): enable Hypatia scanning - -### Fixed - -- fix(ci): bump a2ml/k9-validate-action pins to canonical (#32) -- fix(ci): sync hypatia-scan.yml to canonical (#31) -- fix(ci): build Hypatia escript from repo root (estate dogfood drift) -- fix(ci): hypatia-scan workdir (${{ env.HOME }} resolves empty) (#30) -- fix(ci): bump erlef/setup-beam SHA for ubuntu24 runner support (#23) -- fix(ci): repair YAML block-scalar in workflow-linter Check Permissions step (#26) -- fix(ci): move secret-scanner Cargo.toml gate from job-level if: to step-level (#28) -- fix(src/metadata.rs): remove unused std::fs::File import (#25) -- fix(codeql): switch language matrix to 'actions' (no JS/TS in repo) (#24) -- fix(ci): Resolve workflow-linter self-matching and metadata issues - -### Changed - -- refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) - -### Documentation - -- docs: record tech-debt audit findings (2026-05-26) (#43) -- docs: substantive CRG C annotation (EXPLAINME.adoc) -- docs: add EXPLAINME.adoc — prove-it file backing README claims -- docs: add checkpoint files for state tracking - -### CI - -- ci(rust): convert rust-ci.yml to thin wrapper (standards#174) (#40) -- ci: redistribute concurrency-cancel guard to read-only check workflows (#34) -- ci(dependabot): restore cargo PR limit so security PRs flow (#16) -- ci(secret-scanner): drop duplicate --fail from trufflehog extra_args (#15) -- ci: bump actions/upload-artifact SHA to current v4 (#14) - -## Pre-history - -Prior commits to this file's introduction are recorded in git history but not formally classified into Keep-a-Changelog sections. To backfill, run `git cliff -o CHANGELOG.md` locally using the canonical [`cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) — this is one-shot mechanical work. - ---- - - diff --git a/CODE_OF_CONDUCT.adoc b/CODE_OF_CONDUCT.adoc new file mode 100644 index 0000000..e49f54f --- /dev/null +++ b/CODE_OF_CONDUCT.adoc @@ -0,0 +1,134 @@ +== Contributor Covenant Code of Conduct + +=== Our Pledge + +We as members, contributors, and leaders pledge to make participation in +our community a harassment-free experience for everyone, regardless of +age, body size, visible or invisible disability, ethnicity, sex +characteristics, gender identity and expression, level of experience, +education, socio-economic status, nationality, personal appearance, +race, caste, color, religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, +welcoming, diverse, inclusive, and healthy community. + +=== Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our +mistakes, and learning from the experience +* Focusing on what is best not just for us as individuals, but for the +overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or +advances of any kind +* Trolling, insulting or derogatory comments, and personal or political +attacks +* Public or private harassment +* Publishing others’ private information, such as a physical or email +address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a +professional setting + +=== Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our +standards of acceptable behavior and will take appropriate and fair +corrective action in response to any behavior that they deem +inappropriate, threatening, offensive, or harmful. + +Community leaders have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other +contributions that are not aligned to this Code of Conduct, and will +communicate reasons for moderation decisions when appropriate. + +=== Scope + +This Code of Conduct applies within all community spaces, and also +applies when an individual is officially representing the community in +public spaces. Examples of representing our community include using an +official email address, posting via an official social media account, or +acting as an appointed representative at an online or offline event. + +=== Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may +be reported to the community leaders responsible for enforcement listed +in MAINTAINERS.md. + +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security +of the reporter of any incident. + +=== Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in +determining the consequences for any action they deem in violation of +this Code of Conduct: + +==== 1. Correction + +*Community Impact*: Use of inappropriate language or other behavior +deemed unprofessional or unwelcome in the community. + +*Consequence*: A private, written warning from community leaders, +providing clarity around the nature of the violation and an explanation +of why the behavior was inappropriate. A public apology may be +requested. + +==== 2. Warning + +*Community Impact*: A violation through a single incident or series of +actions. + +*Consequence*: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, for a specified period of +time. This includes avoiding interactions in community spaces as well as +external channels like social media. Violating these terms may lead to a +temporary or permanent ban. + +==== 3. Temporary Ban + +*Community Impact*: A serious violation of community standards, +including sustained inappropriate behavior. + +*Consequence*: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No +public or private interaction with the people involved, including +unsolicited interaction with those enforcing the Code of Conduct, is +allowed during this period. Violating these terms may lead to a +permanent ban. + +==== 4. Permanent Ban + +*Community Impact*: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of +individuals. + +*Consequence*: A permanent ban from any sort of public interaction +within the community. + +=== Attribution + +This Code of Conduct is adapted from the +https://www.contributor-covenant.org[Contributor Covenant], version 2.1, +available at +https://www.contributor-covenant.org/version/2/1/code_of_conduct.html. + +Community Impact Guidelines were inspired by +https://github.com/mozilla/diversity[Mozilla’s code of conduct +enforcement ladder]. + +For answers to common questions about this code of conduct, see the FAQ +at https://www.contributor-covenant.org/faq. Translations are available +at https://www.contributor-covenant.org/translations. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index 78159af..0000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,133 +0,0 @@ -# Contributor Covenant Code of Conduct - -## Our Pledge - -We as members, contributors, and leaders pledge to make participation in our -community a harassment-free experience for everyone, regardless of age, body -size, visible or invisible disability, ethnicity, sex characteristics, gender -identity and expression, level of experience, education, socio-economic status, -nationality, personal appearance, race, caste, color, religion, or sexual -identity and orientation. - -We pledge to act and interact in ways that contribute to an open, welcoming, -diverse, inclusive, and healthy community. - -## Our Standards - -Examples of behavior that contributes to a positive environment for our -community include: - -* Demonstrating empathy and kindness toward other people -* Being respectful of differing opinions, viewpoints, and experiences -* Giving and gracefully accepting constructive feedback -* Accepting responsibility and apologizing to those affected by our mistakes, - and learning from the experience -* Focusing on what is best not just for us as individuals, but for the overall - community - -Examples of unacceptable behavior include: - -* The use of sexualized language or imagery, and sexual attention or advances of - any kind -* Trolling, insulting or derogatory comments, and personal or political attacks -* Public or private harassment -* Publishing others' private information, such as a physical or email address, - without their explicit permission -* Other conduct which could reasonably be considered inappropriate in a - professional setting - -## Enforcement Responsibilities - -Community leaders are responsible for clarifying and enforcing our standards of -acceptable behavior and will take appropriate and fair corrective action in -response to any behavior that they deem inappropriate, threatening, offensive, -or harmful. - -Community leaders have the right and responsibility to remove, edit, or reject -comments, commits, code, wiki edits, issues, and other contributions that are -not aligned to this Code of Conduct, and will communicate reasons for moderation -decisions when appropriate. - -## Scope - -This Code of Conduct applies within all community spaces, and also applies when -an individual is officially representing the community in public spaces. -Examples of representing our community include using an official email address, -posting via an official social media account, or acting as an appointed -representative at an online or offline event. - -## Enforcement - -Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported to the community leaders responsible for enforcement listed in -MAINTAINERS.md. - -All complaints will be reviewed and investigated promptly and fairly. - -All community leaders are obligated to respect the privacy and security of the -reporter of any incident. - -## Enforcement Guidelines - -Community leaders will follow these Community Impact Guidelines in determining -the consequences for any action they deem in violation of this Code of Conduct: - -### 1. Correction - -**Community Impact**: Use of inappropriate language or other behavior deemed -unprofessional or unwelcome in the community. - -**Consequence**: A private, written warning from community leaders, providing -clarity around the nature of the violation and an explanation of why the -behavior was inappropriate. A public apology may be requested. - -### 2. Warning - -**Community Impact**: A violation through a single incident or series of -actions. - -**Consequence**: A warning with consequences for continued behavior. No -interaction with the people involved, including unsolicited interaction with -those enforcing the Code of Conduct, for a specified period of time. This -includes avoiding interactions in community spaces as well as external channels -like social media. Violating these terms may lead to a temporary or permanent -ban. - -### 3. Temporary Ban - -**Community Impact**: A serious violation of community standards, including -sustained inappropriate behavior. - -**Consequence**: A temporary ban from any sort of interaction or public -communication with the community for a specified period of time. No public or -private interaction with the people involved, including unsolicited interaction -with those enforcing the Code of Conduct, is allowed during this period. -Violating these terms may lead to a permanent ban. - -### 4. Permanent Ban - -**Community Impact**: Demonstrating a pattern of violation of community -standards, including sustained inappropriate behavior, harassment of an -individual, or aggression toward or disparagement of classes of individuals. - -**Consequence**: A permanent ban from any sort of public interaction within the -community. - -## Attribution - -This Code of Conduct is adapted from the [Contributor Covenant][homepage], -version 2.1, available at -[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. - -Community Impact Guidelines were inspired by -[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. - -For answers to common questions about this code of conduct, see the FAQ at -[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at -[https://www.contributor-covenant.org/translations][translations]. - -[homepage]: https://www.contributor-covenant.org -[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html -[Mozilla CoC]: https://github.com/mozilla/diversity -[FAQ]: https://www.contributor-covenant.org/faq -[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.adoc b/CONTRIBUTING.adoc new file mode 100644 index 0000000..4ffb055 --- /dev/null +++ b/CONTRIBUTING.adoc @@ -0,0 +1,108 @@ +== Clone the repository + +git clone https://github.com/hyperpolymath/zerostep.git cd zerostep + +== Using Nix (recommended for reproducibility) + +nix develop + +== Or using toolbox/distrobox + +toolbox create zerostep-dev toolbox enter zerostep-dev # Install +dependencies manually + +== Verify setup + +just check # or: cargo check / mix compile / etc. just test # Run test +suite + +.... + +### Repository Structure +.... + +zerostep/ ├── src/ # Source code (Perimeter 1-2) ├── lib/ # Library code +(Perimeter 1-2) ├── extensions/ # Extensions (Perimeter 2) ├── plugins/ +# Plugins (Perimeter 2) ├── tools/ # Tooling (Perimeter 2) ├── docs/ # +Documentation (Perimeter 3) │ ├── architecture/ # ADRs, specs (Perimeter +2) │ └── proposals/ # RFCs (Perimeter 3) ├── examples/ # Examples +(Perimeter 3) ├── spec/ # Spec tests (Perimeter 3) ├── tests/ # Test +suite (Perimeter 2-3) ├── .well-known/ # Protocol files (Perimeter 1-3) +├── .github/ # GitHub config (Perimeter 1) │ ├── ISSUE_TEMPLATE/ │ └── +workflows/ ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md # +This file ├── GOVERNANCE.md ├── LICENSE ├── MAINTAINERS.md ├── +README.adoc ├── SECURITY.md ├── flake.nix # Nix flake (Perimeter 1) └── +Justfile # Task runner (Perimeter 1) + +.... + +--- + +## How to Contribute + +### Reporting Bugs + +**Before reporting**: +1. Search existing issues +2. Check if it's already fixed in `main` +3. Determine which perimeter the bug affects + +**When reporting**: + +Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include: + +- Clear, descriptive title +- Environment details (OS, versions, toolchain) +- Steps to reproduce +- Expected vs actual behaviour +- Logs, screenshots, or minimal reproduction + +### Suggesting Features + +**Before suggesting**: +1. Check the [roadmap](ROADMAP.md) if available +2. Search existing issues and discussions +3. Consider which perimeter the feature belongs to + +**When suggesting**: + +Use the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and include: + +- Problem statement (what pain point does this solve?) +- Proposed solution +- Alternatives considered +- Which perimeter this affects + +### Your First Contribution + +Look for issues labelled: + +- [`good first issue`](https://github.com/hyperpolymath/zerostep/labels/good%20first%20issue) — Simple Perimeter 3 tasks +- [`help wanted`](https://github.com/hyperpolymath/zerostep/labels/help%20wanted) — Community help needed +- [`documentation`](https://github.com/hyperpolymath/zerostep/labels/documentation) — Docs improvements +- [`perimeter-3`](https://github.com/hyperpolymath/zerostep/labels/perimeter-3) — Community sandbox scope + +--- + +## Development Workflow + +### Branch Naming +.... + +docs/short-description # Documentation (P3) test/what-added # Test +additions (P3) feat/short-description # New features (P2) +fix/issue-number-description # Bug fixes (P2) refactor/what-changed # +Code improvements (P2) security/what-fixed # Security fixes (P1-2) + +.... + +### Commit Messages + +We follow [Conventional Commits](https://www.conventionalcommits.org/): +.... + +(): + +{empty}[optional body] + +{empty}[optional footer] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 4f9d673..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,116 +0,0 @@ -# Clone the repository -git clone https://github.com/hyperpolymath/zerostep.git -cd zerostep - -# Using Nix (recommended for reproducibility) -nix develop - -# Or using toolbox/distrobox -toolbox create zerostep-dev -toolbox enter zerostep-dev -# Install dependencies manually - -# Verify setup -just check # or: cargo check / mix compile / etc. -just test # Run test suite -``` - -### Repository Structure -``` -zerostep/ -├── src/ # Source code (Perimeter 1-2) -├── lib/ # Library code (Perimeter 1-2) -├── extensions/ # Extensions (Perimeter 2) -├── plugins/ # Plugins (Perimeter 2) -├── tools/ # Tooling (Perimeter 2) -├── docs/ # Documentation (Perimeter 3) -│ ├── architecture/ # ADRs, specs (Perimeter 2) -│ └── proposals/ # RFCs (Perimeter 3) -├── examples/ # Examples (Perimeter 3) -├── spec/ # Spec tests (Perimeter 3) -├── tests/ # Test suite (Perimeter 2-3) -├── .well-known/ # Protocol files (Perimeter 1-3) -├── .github/ # GitHub config (Perimeter 1) -│ ├── ISSUE_TEMPLATE/ -│ └── workflows/ -├── CHANGELOG.md -├── CODE_OF_CONDUCT.md -├── CONTRIBUTING.md # This file -├── GOVERNANCE.md -├── LICENSE -├── MAINTAINERS.md -├── README.adoc -├── SECURITY.md -├── flake.nix # Nix flake (Perimeter 1) -└── Justfile # Task runner (Perimeter 1) -``` - ---- - -## How to Contribute - -### Reporting Bugs - -**Before reporting**: -1. Search existing issues -2. Check if it's already fixed in `main` -3. Determine which perimeter the bug affects - -**When reporting**: - -Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include: - -- Clear, descriptive title -- Environment details (OS, versions, toolchain) -- Steps to reproduce -- Expected vs actual behaviour -- Logs, screenshots, or minimal reproduction - -### Suggesting Features - -**Before suggesting**: -1. Check the [roadmap](ROADMAP.md) if available -2. Search existing issues and discussions -3. Consider which perimeter the feature belongs to - -**When suggesting**: - -Use the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and include: - -- Problem statement (what pain point does this solve?) -- Proposed solution -- Alternatives considered -- Which perimeter this affects - -### Your First Contribution - -Look for issues labelled: - -- [`good first issue`](https://github.com/hyperpolymath/zerostep/labels/good%20first%20issue) — Simple Perimeter 3 tasks -- [`help wanted`](https://github.com/hyperpolymath/zerostep/labels/help%20wanted) — Community help needed -- [`documentation`](https://github.com/hyperpolymath/zerostep/labels/documentation) — Docs improvements -- [`perimeter-3`](https://github.com/hyperpolymath/zerostep/labels/perimeter-3) — Community sandbox scope - ---- - -## Development Workflow - -### Branch Naming -``` -docs/short-description # Documentation (P3) -test/what-added # Test additions (P3) -feat/short-description # New features (P2) -fix/issue-number-description # Bug fixes (P2) -refactor/what-changed # Code improvements (P2) -security/what-fixed # Security fixes (P1-2) -``` - -### Commit Messages - -We follow [Conventional Commits](https://www.conventionalcommits.org/): -``` -(): - -[optional body] - -[optional footer] diff --git a/DCO.adoc b/DCO.adoc new file mode 100644 index 0000000..c13bb3f --- /dev/null +++ b/DCO.adoc @@ -0,0 +1,73 @@ +== Developer Certificate of Origin + +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +''''' + +=== Developer’s Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +*(a)* The contribution was created in whole or in part by me and I have +the right to submit it under the open source license indicated in the +file; or + +*(b)* The contribution is based upon previous work that, to the best of +my knowledge, is covered under an appropriate open source license and I +have the right under that license to submit that work with +modifications, whether created in whole or in part by me, under the same +open source license (unless I am permitted to submit under a different +license), as indicated in the file; or + +*(c)* The contribution was provided directly to me by some other person +who certified (a), (b) or (c) and I have not modified it. + +*(d)* I understand and agree that this project and the contribution are +public and that a record of the contribution (including all personal +information I submit with it, including my sign-off) is maintained +indefinitely and may be redistributed consistent with this project or +the open source license(s) involved. + +''''' + +=== How to Sign Off + +To sign off your commits, add the following line to your commit message: + +.... +Signed-off-by: Your Name +.... + +You can automate this with: + +[source,bash] +---- +git commit -s -m "Your commit message" +---- + +Or configure git to always sign off: + +[source,bash] +---- +git config --local format.signoff true +---- + +''''' + +=== Why DCO? + +The Developer Certificate of Origin ensures: + +[arabic] +. *Provenance*: Clear chain of authorship +. *Legal clarity*: Contributors affirm they have rights to contribute +. *Community trust*: Transparent contribution process +. *License compliance*: Contributions match project licenses + +This aligns with RSR (Rhodium Standard Repository) accountability +requirements. diff --git a/DCO.md b/DCO.md deleted file mode 100644 index 9973f94..0000000 --- a/DCO.md +++ /dev/null @@ -1,74 +0,0 @@ - - - -# Developer Certificate of Origin - -Version 1.1 - -Copyright (C) 2004, 2006 The Linux Foundation and its contributors. - -Everyone is permitted to copy and distribute verbatim copies of this -license document, but changing it is not allowed. - ---- - -## Developer's Certificate of Origin 1.1 - -By making a contribution to this project, I certify that: - -**(a)** The contribution was created in whole or in part by me and I -have the right to submit it under the open source license -indicated in the file; or - -**(b)** The contribution is based upon previous work that, to the best -of my knowledge, is covered under an appropriate open source -license and I have the right under that license to submit that -work with modifications, whether created in whole or in part -by me, under the same open source license (unless I am -permitted to submit under a different license), as indicated -in the file; or - -**(c)** The contribution was provided directly to me by some other -person who certified (a), (b) or (c) and I have not modified -it. - -**(d)** I understand and agree that this project and the contribution -are public and that a record of the contribution (including all -personal information I submit with it, including my sign-off) is -maintained indefinitely and may be redistributed consistent with -this project or the open source license(s) involved. - ---- - -## How to Sign Off - -To sign off your commits, add the following line to your commit message: - -``` -Signed-off-by: Your Name -``` - -You can automate this with: - -```bash -git commit -s -m "Your commit message" -``` - -Or configure git to always sign off: - -```bash -git config --local format.signoff true -``` - ---- - -## Why DCO? - -The Developer Certificate of Origin ensures: - -1. **Provenance**: Clear chain of authorship -2. **Legal clarity**: Contributors affirm they have rights to contribute -3. **Community trust**: Transparent contribution process -4. **License compliance**: Contributions match project licenses - -This aligns with RSR (Rhodium Standard Repository) accountability requirements. diff --git a/GOVERNANCE.adoc b/GOVERNANCE.adoc new file mode 100644 index 0000000..9b836fb --- /dev/null +++ b/GOVERNANCE.adoc @@ -0,0 +1,60 @@ +== Governance + +=== Overview + +This project is governed by the following principles and structures to +ensure transparent, inclusive, and effective decision-making. + +=== Roles and Responsibilities + +==== Maintainers + +Maintainers are responsible for: - Reviewing and merging pull requests - +Managing releases and versioning - Ensuring code quality and standards - +Triaging issues and bug reports - Community engagement and support + +==== Contributors + +Contributors are expected to: - Follow the code of conduct - Submit +well-documented pull requests - Write tests for new functionality - +Maintain existing tests - Update documentation as needed + +=== Decision Making + +==== Minor Changes + +* Can be made by any maintainer +* Include bug fixes, documentation updates, dependency updates + +==== Major Changes + +* Require discussion in issues or pull requests +* Include new features, architectural changes, API changes +* Need approval from at least 2 maintainers + +==== Breaking Changes + +* Require RFC (Request for Comments) process +* Need approval from majority of maintainers +* Must include migration guide + +=== Code of Conduct + +All participants are expected to follow our Code of Conduct. Violations +can be reported to the maintainers. + +=== Communication + +* *Issues*: For bug reports and feature requests +* *Discussions*: For questions and general discussion +* *Pull Requests*: For code contributions + +=== Licensing + +All contributions are made under the terms of the repository’s LICENSE +file. By submitting a pull request, you agree to license your +contributions accordingly. + +''''' + +_Last updated: 2026-07-18_ diff --git a/GOVERNANCE.md b/GOVERNANCE.md deleted file mode 100644 index e27364c..0000000 --- a/GOVERNANCE.md +++ /dev/null @@ -1,60 +0,0 @@ -# Governance - -## Overview - -This project is governed by the following principles and structures to ensure transparent, inclusive, and effective decision-making. - -## Roles and Responsibilities - -### Maintainers - -Maintainers are responsible for: -- Reviewing and merging pull requests -- Managing releases and versioning -- Ensuring code quality and standards -- Triaging issues and bug reports -- Community engagement and support - -### Contributors - -Contributors are expected to: -- Follow the code of conduct -- Submit well-documented pull requests -- Write tests for new functionality -- Maintain existing tests -- Update documentation as needed - -## Decision Making - -### Minor Changes -- Can be made by any maintainer -- Include bug fixes, documentation updates, dependency updates - -### Major Changes -- Require discussion in issues or pull requests -- Include new features, architectural changes, API changes -- Need approval from at least 2 maintainers - -### Breaking Changes -- Require RFC (Request for Comments) process -- Need approval from majority of maintainers -- Must include migration guide - -## Code of Conduct - -All participants are expected to follow our Code of Conduct. Violations can be reported to the maintainers. - -## Communication - -- **Issues**: For bug reports and feature requests -- **Discussions**: For questions and general discussion -- **Pull Requests**: For code contributions - -## Licensing - -All contributions are made under the terms of the repository's LICENSE file. -By submitting a pull request, you agree to license your contributions accordingly. - ---- - -*Last updated: 2026-07-18* diff --git a/MAINTAINERS.adoc b/MAINTAINERS.adoc new file mode 100644 index 0000000..81df1c0 --- /dev/null +++ b/MAINTAINERS.adoc @@ -0,0 +1,73 @@ +== Maintainers + +This file lists the maintainers of the VAE Dataset Normalizer project. + +=== Current Maintainers + +==== BDFL (Benevolent Dictator For Life) + +[width="100%",cols="22%,21%,25%,32%",options="header",] +|=== +|Name |Role |Since |Contact +|Joshua Jewell |BDFL, Lead Maintainer |2024-01-01 |Via repository issues +|=== + +==== Core Maintainers + +[cols=",,,",options="header",] +|=== +|Name |Role |Since |Focus Area +|Joshua Jewell |Lead |2024-01-01 |All areas +|=== + +==== Expert Contributors + +[cols=",,,",options="header",] +|=== +|Name |Role |Since |Focus Area +|(Open for contributions) | | | +|=== + +=== Becoming a Maintainer + +See CONTRIBUTING.adoc for the Tri-Perimeter Contribution Framework +(TPCF). + +==== Requirements + +[arabic] +. *Sustained contribution*: 6+ months of active involvement +. *Technical competence*: Demonstrated understanding of codebase +. *Community trust*: Positive interactions, Code of Conduct adherence +. *Security awareness*: Understanding of security practices +. *Availability*: Commitment to reasonable response times + +==== Process + +[arabic] +. Nomination by existing maintainer +. Discussion among current maintainers +. Vote (requires unanimous approval) +. Onboarding and access provisioning + +=== Emeritus Maintainers + +Former maintainers who have stepped down: + +[cols=",,,",options="header",] +|=== +|Name |Role |Period |Reason +|(None yet) | | | +|=== + +=== Contact + +* *General*: Open an issue on the repository +* *Security*: See SECURITY.md +* *Governance*: See GOVERNANCE.adoc + +=== Acknowledgments + +We thank all contributors, past and present, for their work on this +project. See the commit history and release notes for full contributor +recognition. diff --git a/MAINTAINERS.md b/MAINTAINERS.md deleted file mode 100644 index 2f25bc2..0000000 --- a/MAINTAINERS.md +++ /dev/null @@ -1,61 +0,0 @@ -# Maintainers - -This file lists the maintainers of the VAE Dataset Normalizer project. - -## Current Maintainers - -### BDFL (Benevolent Dictator For Life) - -| Name | Role | Since | Contact | -|------|------|-------|---------| -| Joshua Jewell | BDFL, Lead Maintainer | 2024-01-01 | Via repository issues | - -### Core Maintainers - -| Name | Role | Since | Focus Area | -|------|------|-------|------------| -| Joshua Jewell | Lead | 2024-01-01 | All areas | - -### Expert Contributors - -| Name | Role | Since | Focus Area | -|------|------|-------|------------| -| (Open for contributions) | | | | - -## Becoming a Maintainer - -See CONTRIBUTING.adoc for the Tri-Perimeter Contribution Framework (TPCF). - -### Requirements - -1. **Sustained contribution**: 6+ months of active involvement -2. **Technical competence**: Demonstrated understanding of codebase -3. **Community trust**: Positive interactions, Code of Conduct adherence -4. **Security awareness**: Understanding of security practices -5. **Availability**: Commitment to reasonable response times - -### Process - -1. Nomination by existing maintainer -2. Discussion among current maintainers -3. Vote (requires unanimous approval) -4. Onboarding and access provisioning - -## Emeritus Maintainers - -Former maintainers who have stepped down: - -| Name | Role | Period | Reason | -|------|------|--------|--------| -| (None yet) | | | | - -## Contact - -- **General**: Open an issue on the repository -- **Security**: See SECURITY.md -- **Governance**: See GOVERNANCE.adoc - -## Acknowledgments - -We thank all contributors, past and present, for their work on this project. -See the commit history and release notes for full contributor recognition. diff --git a/PROOF-NEEDS.adoc b/PROOF-NEEDS.adoc new file mode 100644 index 0000000..fefe7e4 --- /dev/null +++ b/PROOF-NEEDS.adoc @@ -0,0 +1,46 @@ +== PROOF-NEEDS.md + +=== Current State + +* *LOC*: ~3,610 +* *Languages*: Rust, Julia, Idris2, Zig +* *Existing ABI proofs*: `+src/abi/*.idr+` (template-level) +* *Dangerous patterns*: None detected + +=== What Needs Proving + +==== VAE Dataset Normalization (Julia core) + +* `+contrastive_model.jl+`, `+julia_utils.jl+` — data processing +pipeline +* Prove: normalization is invertible (or document when it is lossy) +* Prove: contrastive model training preserves dataset statistical +properties + +==== Metadata Handling (src/metadata.rs) + +* Dataset metadata management +* Prove: metadata correctly describes the normalized dataset + +==== Fuzz Target (fuzz/fuzz_targets/fuzz_input.rs) + +* Fuzzing exists — formal proofs of normalization properties would be +stronger + +=== Recommended Prover + +* *Lean4* with Mathlib for numerical/statistical properties +* *Idris2* for ABI layer and metadata invariants + +=== Priority + +*LOW* — Dataset preprocessing tool. Normalization correctness matters +for ML pipeline reproducibility but is not safety-critical. The small +codebase limits the scope of potential errors. + +=== Template ABI Cleanup (2026-03-29) + +Template ABI removed – was creating false impression of formal +verification. The removed files (Types.idr, Layout.idr, Foreign.idr) +contained only RSR template scaffolding with unresolved +\{\{PROJECT}}/\{\{AUTHOR}} placeholders and no domain-specific proofs. diff --git a/PROOF-NEEDS.md b/PROOF-NEEDS.md deleted file mode 100644 index a8d4c54..0000000 --- a/PROOF-NEEDS.md +++ /dev/null @@ -1,38 +0,0 @@ -# PROOF-NEEDS.md - - -## Current State - -- **LOC**: ~3,610 -- **Languages**: Rust, Julia, Idris2, Zig -- **Existing ABI proofs**: `src/abi/*.idr` (template-level) -- **Dangerous patterns**: None detected - -## What Needs Proving - -### VAE Dataset Normalization (Julia core) -- `contrastive_model.jl`, `julia_utils.jl` — data processing pipeline -- Prove: normalization is invertible (or document when it is lossy) -- Prove: contrastive model training preserves dataset statistical properties - -### Metadata Handling (src/metadata.rs) -- Dataset metadata management -- Prove: metadata correctly describes the normalized dataset - -### Fuzz Target (fuzz/fuzz_targets/fuzz_input.rs) -- Fuzzing exists — formal proofs of normalization properties would be stronger - -## Recommended Prover - -- **Lean4** with Mathlib for numerical/statistical properties -- **Idris2** for ABI layer and metadata invariants - -## Priority - -**LOW** — Dataset preprocessing tool. Normalization correctness matters for ML pipeline reproducibility but is not safety-critical. The small codebase limits the scope of potential errors. - -## Template ABI Cleanup (2026-03-29) - -Template ABI removed -- was creating false impression of formal verification. -The removed files (Types.idr, Layout.idr, Foreign.idr) contained only RSR template -scaffolding with unresolved {{PROJECT}}/{{AUTHOR}} placeholders and no domain-specific proofs. diff --git a/QUICKSTART.md b/QUICKSTART.adoc similarity index 72% rename from QUICKSTART.md rename to QUICKSTART.adoc index a63540d..165a6f2 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.adoc @@ -1,13 +1,11 @@ - - - -# Quick Start Guide +== Quick Start Guide Get up and running with vae-normalizer in 5 minutes. -## 1. Check Dependencies +=== 1. Check Dependencies -```bash +[source,bash] +---- # Required rustc --version # Need 1.70+ cargo --version @@ -16,23 +14,27 @@ cargo --version cue version nickel --version julia --version -``` +---- -## 2. Build +=== 2. Build -```bash +[source,bash] +---- cd vae-normalizer cargo build --release -``` +---- Or with just: -```bash + +[source,bash] +---- just build -``` +---- -## 3. Download Dataset +=== 3. Download Dataset -```bash +[source,bash] +---- # Option A: Hugging Face CLI huggingface-cli download joshuajewell/VAEDecodedImages-SDXL \ --local-dir ~/vae-dataset \ @@ -40,40 +42,47 @@ huggingface-cli download joshuajewell/VAEDecodedImages-SDXL \ # Option B: Git git clone https://huggingface.co/datasets/joshuajewell/VAEDecodedImages-SDXL ~/vae-dataset -``` +---- -## 4. Normalize +=== 4. Normalize Fast mode (no checksums): -```bash + +[source,bash] +---- ./target/release/vae-normalizer normalize \ -d ~/vae-dataset \ -o ~/vae-normalized \ --skip-checksums -``` +---- Full mode with SHAKE256: -```bash + +[source,bash] +---- ./target/release/vae-normalizer normalize \ -d ~/vae-dataset \ -o ~/vae-normalized -``` +---- -## 5. Verify +=== 5. Verify -```bash +[source,bash] +---- ./target/release/vae-normalizer verify -o ~/vae-normalized -``` +---- -## 6. View Statistics +=== 6. View Statistics -```bash +[source,bash] +---- ./target/release/vae-normalizer stats -o ~/vae-normalized -``` +---- -## 7. Train a Model (Julia) +=== 7. Train a Model (Julia) -```julia +[source,julia] +---- # Install dependencies using Pkg Pkg.add(["Flux", "CSV", "DataFrames", "Images", "FileIO"]) @@ -125,13 +134,14 @@ for epoch in 1:10 acc = mean([accuracy(model, x, y) for (x, y) in test_loader]) println("Epoch $epoch: accuracy = $(round(acc * 100, digits=1))%") end -``` +---- -## Common Tasks +=== Common Tasks -### Compare Random vs Stratified Splits +==== Compare Random vs Stratified Splits -```julia +[source,julia] +---- include("julia_utils.jl") using .VAEDatasetUtils @@ -141,44 +151,50 @@ strat_train = VAEDetectorDataset("splits/stratified_train.txt", "manifest.csv", println("Random train: $(length(random_train)) samples") println("Stratified train: $(length(strat_train)) samples") -``` +---- -### Validate CUE Schema +==== Validate CUE Schema -```bash +[source,bash] +---- cue vet metadata_schema.cue ~/vae-normalized/metadata.cue -``` +---- -### Run Isabelle Proofs +==== Run Isabelle Proofs -```bash +[source,bash] +---- isabelle build -d . -b VAEDataset_Splits -``` +---- -### Hash a Single File +==== Hash a Single File -```bash +[source,bash] +---- ./target/release/vae-normalizer hash ~/vae-dataset/Original/image001.png -``` +---- + +=== Troubleshooting -## Troubleshooting +*Build fails with "`missing feature`"* -**Build fails with "missing feature"** -```bash +[source,bash] +---- rustup update cargo clean && cargo build --release -``` +---- + +*"`No matching image pairs found`"* - Check dataset structure: needs +`+Original/+` and `+VAE/+` subdirectories - Image filenames must match +(same stem, any extension) -**"No matching image pairs found"** -- Check dataset structure: needs `Original/` and `VAE/` subdirectories -- Image filenames must match (same stem, any extension) +*Out of memory during checksumming* - Use `+--skip-checksums+` for large +datasets - Or process in batches -**Out of memory during checksumming** -- Use `--skip-checksums` for large datasets -- Or process in batches +*Julia: "`Images package not found`"* -**Julia: "Images package not found"** -```julia +[source,julia] +---- using Pkg Pkg.add("Images") -``` +---- diff --git a/REVERSIBILITY.adoc b/REVERSIBILITY.adoc new file mode 100644 index 0000000..a006ec6 --- /dev/null +++ b/REVERSIBILITY.adoc @@ -0,0 +1,124 @@ +== Reversibility Policy + +This document describes how operations in vae-normalizer can be undone +or reverted. + +=== Core Principle + +*Every operation should be reversible.* This project follows the +principle that users should never be trapped by their choices. + +=== Operation Reversibility + +==== Dataset Normalization (`+normalize+`) + +[cols=",,",options="header",] +|=== +|Operation |Reversible? |How to Revert +|Generate splits |Yes |Delete output directory +|Compute checksums |Yes |Delete manifest.csv +|Generate metadata |Yes |Delete metadata.cue +|Create output directory |Yes |`+rm -rf output/+` +|=== + +*Note*: The normalize command never modifies the source dataset. All +outputs are written to a separate output directory. + +==== Verification (`+verify+`) + +[cols=",,",options="header",] +|=== +|Operation |Reversible? |How to Revert +|Read manifest |Yes (read-only) |N/A +|Check file existence |Yes (read-only) |N/A +|Verify checksums |Yes (read-only) |N/A +|=== + +*Note*: The verify command is entirely read-only and makes no changes. + +==== Statistics (`+stats+`) + +[cols=",,",options="header",] +|=== +|Operation |Reversible? |How to Revert +|Read manifest |Yes (read-only) |N/A +|Display stats |Yes (read-only) |N/A +|=== + +*Note*: The stats command is entirely read-only. + +==== Hash (`+hash+`) + +[cols=",,",options="header",] +|=== +|Operation |Reversible? |How to Revert +|Compute hash |Yes (read-only) |N/A +|=== + +*Note*: The hash command is entirely read-only. + +=== Non-Destructive Defaults + +[arabic] +. *No overwrites*: The tool will not overwrite existing output without +`+--force+` +. *No source modification*: Source dataset is never modified +. *Explicit confirmation*: Destructive operations require explicit flags +. *Dry-run available*: Use `+--dry-run+` to preview operations + +=== Recovery Procedures + +==== Lost Output Directory + +If the output directory is accidentally deleted: + +[source,bash] +---- +# Regenerate everything +vae-normalizer normalize -d /path/to/dataset -o /path/to/output +---- + +The tool is deterministic with the same seed, so regeneration produces +identical results. + +==== Corrupted Manifest + +If manifest.csv becomes corrupted: + +[source,bash] +---- +# Regenerate with same parameters +vae-normalizer normalize -d /path/to/dataset -o /path/to/output --seed 42 +---- + +==== Version Control + +All outputs are text-based and suitable for version control: + +* `+manifest.csv+` - Track in git for audit trail +* `+splits/*.txt+` - Track for reproducibility +* `+metadata.cue+` - Track for provenance + +=== Git Safety + +This project integrates with git safely: + +* Never force-pushes +* Never rewrites history without explicit request +* Always allows reverting commits +* Maintains full audit trail + +=== Formal Guarantees + +The Isabelle/HOL proofs in `+VAEDataset_Splits.thy+` guarantee: + +[arabic] +. *Determinism*: Same inputs → same outputs (with same seed) +. *Completeness*: All images accounted for +. *Disjointness*: No data leakage between splits +. *Bijection*: Original ↔ VAE mapping preserved + +=== Contact + +Questions about reversibility? Open an issue with the `+reversibility+` +label. diff --git a/REVERSIBILITY.md b/REVERSIBILITY.md deleted file mode 100644 index 018ebfd..0000000 --- a/REVERSIBILITY.md +++ /dev/null @@ -1,109 +0,0 @@ - - - -# Reversibility Policy - -This document describes how operations in vae-normalizer can be undone or reverted. - -## Core Principle - -**Every operation should be reversible.** This project follows the principle that users should never be trapped by their choices. - -## Operation Reversibility - -### Dataset Normalization (`normalize`) - -| Operation | Reversible? | How to Revert | -|-----------|-------------|---------------| -| Generate splits | Yes | Delete output directory | -| Compute checksums | Yes | Delete manifest.csv | -| Generate metadata | Yes | Delete metadata.cue | -| Create output directory | Yes | `rm -rf output/` | - -**Note**: The normalize command never modifies the source dataset. All outputs are written to a separate output directory. - -### Verification (`verify`) - -| Operation | Reversible? | How to Revert | -|-----------|-------------|---------------| -| Read manifest | Yes (read-only) | N/A | -| Check file existence | Yes (read-only) | N/A | -| Verify checksums | Yes (read-only) | N/A | - -**Note**: The verify command is entirely read-only and makes no changes. - -### Statistics (`stats`) - -| Operation | Reversible? | How to Revert | -|-----------|-------------|---------------| -| Read manifest | Yes (read-only) | N/A | -| Display stats | Yes (read-only) | N/A | - -**Note**: The stats command is entirely read-only. - -### Hash (`hash`) - -| Operation | Reversible? | How to Revert | -|-----------|-------------|---------------| -| Compute hash | Yes (read-only) | N/A | - -**Note**: The hash command is entirely read-only. - -## Non-Destructive Defaults - -1. **No overwrites**: The tool will not overwrite existing output without `--force` -2. **No source modification**: Source dataset is never modified -3. **Explicit confirmation**: Destructive operations require explicit flags -4. **Dry-run available**: Use `--dry-run` to preview operations - -## Recovery Procedures - -### Lost Output Directory - -If the output directory is accidentally deleted: - -```bash -# Regenerate everything -vae-normalizer normalize -d /path/to/dataset -o /path/to/output -``` - -The tool is deterministic with the same seed, so regeneration produces identical results. - -### Corrupted Manifest - -If manifest.csv becomes corrupted: - -```bash -# Regenerate with same parameters -vae-normalizer normalize -d /path/to/dataset -o /path/to/output --seed 42 -``` - -### Version Control - -All outputs are text-based and suitable for version control: - -- `manifest.csv` - Track in git for audit trail -- `splits/*.txt` - Track for reproducibility -- `metadata.cue` - Track for provenance - -## Git Safety - -This project integrates with git safely: - -- Never force-pushes -- Never rewrites history without explicit request -- Always allows reverting commits -- Maintains full audit trail - -## Formal Guarantees - -The Isabelle/HOL proofs in `VAEDataset_Splits.thy` guarantee: - -1. **Determinism**: Same inputs → same outputs (with same seed) -2. **Completeness**: All images accounted for -3. **Disjointness**: No data leakage between splits -4. **Bijection**: Original ↔ VAE mapping preserved - -## Contact - -Questions about reversibility? Open an issue with the `reversibility` label. diff --git a/SECURITY.adoc b/SECURITY.adoc new file mode 100644 index 0000000..e0cf581 --- /dev/null +++ b/SECURITY.adoc @@ -0,0 +1,100 @@ +== Security Policy + +=== Supported Versions + +[cols=",",options="header",] +|=== +|Version |Supported +|1.x.x |:white_check_mark: +|< 1.0 |:x: +|=== + +=== Reporting a Vulnerability + +We take security seriously. If you discover a security vulnerability, +please report it responsibly. + +==== Reporting Channel + +* *Preferred*: Open a security advisory on the repository +* *Alternative*: Email security concerns to the maintainers listed in +MAINTAINERS.md +* *Do NOT*: Open public issues for security vulnerabilities + +==== Response SLA + +* *Acknowledgement*: Within 24 hours +* *Initial Assessment*: Within 72 hours +* *Fix Timeline*: Depends on severity +** Critical: 7 days +** High: 14 days +** Medium: 30 days +** Low: 90 days + +==== What to Include + +[arabic] +. Description of the vulnerability +. Steps to reproduce +. Potential impact assessment +. Suggested fix (if available) +. Your contact information + +==== Security Measures + +This project implements the following security practices: + +===== Type Safety + +* Written in Rust with compile-time type checking +* No unsafe code blocks without explicit justification +* Memory safety guaranteed by ownership model + +===== Cryptographic Integrity + +* SHAKE256 (d=256) for file checksums +* FIPS 202 compliant implementation +* No custom cryptography + +===== Supply Chain Security + +* SPDX headers on all source files +* Pinned dependencies (no floating versions) +* Regular dependency audits via `+cargo audit+` +* SBOM generation available + +===== Container Security + +* Chainguard Wolfi base images (minimal attack surface) +* Non-root container execution +* No privileged operations required +* Podman (rootless) preferred over Docker + +==== Security Headers (if web-facing) + +Not applicable - this is a CLI tool. + +==== Disclosure Policy + +We follow coordinated disclosure: + +[arabic] +. Reporter submits vulnerability +. We acknowledge within 24 hours +. We assess and develop fix +. Reporter is credited (unless anonymity requested) +. Public disclosure after fix is released + +==== Security Audit History + +[cols=",,,",options="header",] +|=== +|Date |Auditor |Scope |Findings +|2024-01-01 |Self-audit |Full codebase |N/A +|=== + +==== Hall of Fame + +Security researchers who have responsibly disclosed vulnerabilities: + +* (None yet - be the first!) diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 9309963..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,88 +0,0 @@ -# Security Policy - -## Supported Versions - -| Version | Supported | -| ------- | ------------------ | -| 1.x.x | :white_check_mark: | -| < 1.0 | :x: | - -## Reporting a Vulnerability - -We take security seriously. If you discover a security vulnerability, please report it responsibly. - -### Reporting Channel - -- **Preferred**: Open a security advisory on the repository -- **Alternative**: Email security concerns to the maintainers listed in MAINTAINERS.md -- **Do NOT**: Open public issues for security vulnerabilities - -### Response SLA - -- **Acknowledgement**: Within 24 hours -- **Initial Assessment**: Within 72 hours -- **Fix Timeline**: Depends on severity - - Critical: 7 days - - High: 14 days - - Medium: 30 days - - Low: 90 days - -### What to Include - -1. Description of the vulnerability -2. Steps to reproduce -3. Potential impact assessment -4. Suggested fix (if available) -5. Your contact information - -### Security Measures - -This project implements the following security practices: - -#### Type Safety -- Written in Rust with compile-time type checking -- No unsafe code blocks without explicit justification -- Memory safety guaranteed by ownership model - -#### Cryptographic Integrity -- SHAKE256 (d=256) for file checksums -- FIPS 202 compliant implementation -- No custom cryptography - -#### Supply Chain Security -- SPDX headers on all source files -- Pinned dependencies (no floating versions) -- Regular dependency audits via `cargo audit` -- SBOM generation available - -#### Container Security -- Chainguard Wolfi base images (minimal attack surface) -- Non-root container execution -- No privileged operations required -- Podman (rootless) preferred over Docker - -### Security Headers (if web-facing) - -Not applicable - this is a CLI tool. - -### Disclosure Policy - -We follow coordinated disclosure: - -1. Reporter submits vulnerability -2. We acknowledge within 24 hours -3. We assess and develop fix -4. Reporter is credited (unless anonymity requested) -5. Public disclosure after fix is released - -### Security Audit History - -| Date | Auditor | Scope | Findings | -| ---------- | ---------- | ------------------ | -------- | -| 2024-01-01 | Self-audit | Full codebase | N/A | - -### Hall of Fame - -Security researchers who have responsibly disclosed vulnerabilities: - -- (None yet - be the first!) diff --git a/SUPPORT.adoc b/SUPPORT.adoc new file mode 100644 index 0000000..7fc1c0d --- /dev/null +++ b/SUPPORT.adoc @@ -0,0 +1,108 @@ +== Support + +This document describes how to get help with the VAE Dataset Normalizer. + +=== Getting Help + +==== Documentation + +Start with the documentation: + +* README.adoc - Overview and quick start +* QUICKSTART.md - Step-by-step guide +* CONTRIBUTING.adoc - Contribution guidelines + +==== Issue Tracker + +For bugs and feature requests, use the issue tracker: + +* *GitLab*: https://gitlab.com/hyperpolymath/zerostep/-/issues + +Before opening an issue: + +[arabic] +. Search existing issues to avoid duplicates +. Include reproduction steps for bugs +. Provide system information (OS, Rust version, etc.) + +==== Security Issues + +For security vulnerabilities, *do not* open a public issue. + +See SECURITY.md for responsible disclosure procedures. + +=== Support Channels + +[cols=",,",options="header",] +|=== +|Channel |Response Time |Use For +|Issue tracker |1-7 days |Bugs, features, documentation +|Email |7-14 days |Private inquiries +|Security email |48 hours |Vulnerabilities +|=== + +=== What We Support + +==== Supported Versions + +[cols=",,",options="header",] +|=== +|Version |Status |Support +|1.x |Current |Full +|0.x |Legacy |Security only +|=== + +==== Supported Platforms + +* *Rust*: 1.70+ +* *Julia*: 1.9+ +* *Isabelle*: 2023+ +* *OS*: Linux, macOS, Windows (via WSL2) +* *Containers*: Podman with Chainguard Wolfi + +=== Self-Help Resources + +==== Common Issues + +*Build fails with missing dependencies*: + +[source,bash] +---- +# Check all dependencies +just check-deps + +# Install with Nix (recommended) +nix develop +---- + +*Checksum verification fails*: + +[source,bash] +---- +# Verify dataset integrity +vae-normalizer verify -o /path/to/output --checksums -d /path/to/dataset +---- + +*Julia package errors*: + +[source,bash] +---- +# Reinstall Julia dependencies +just julia-setup +---- + +==== Debug Mode + +Enable verbose output for troubleshooting: + +[source,bash] +---- +vae-normalizer -v normalize -d /path/to/dataset -o /path/to/output +---- + +=== Community + +This is a research project maintained by volunteers. Please be patient +and respectful in all interactions. + +See CODE_OF_CONDUCT.md for community guidelines. diff --git a/SUPPORT.md b/SUPPORT.md deleted file mode 100644 index 12042f6..0000000 --- a/SUPPORT.md +++ /dev/null @@ -1,99 +0,0 @@ - - - -# Support - -This document describes how to get help with the VAE Dataset Normalizer. - -## Getting Help - -### Documentation - -Start with the documentation: - -- [README.adoc](README.adoc) - Overview and quick start -- [QUICKSTART.md](QUICKSTART.md) - Step-by-step guide -- [CONTRIBUTING.adoc](CONTRIBUTING.adoc) - Contribution guidelines - -### Issue Tracker - -For bugs and feature requests, use the issue tracker: - -- **GitLab**: https://gitlab.com/hyperpolymath/zerostep/-/issues - -Before opening an issue: - -1. Search existing issues to avoid duplicates -2. Include reproduction steps for bugs -3. Provide system information (OS, Rust version, etc.) - -### Security Issues - -For security vulnerabilities, **do not** open a public issue. - -See [SECURITY.md](SECURITY.md) for responsible disclosure procedures. - -## Support Channels - -| Channel | Response Time | Use For | -|---------|---------------|---------| -| Issue tracker | 1-7 days | Bugs, features, documentation | -| Email | 7-14 days | Private inquiries | -| Security email | 48 hours | Vulnerabilities | - -## What We Support - -### Supported Versions - -| Version | Status | Support | -|---------|--------|---------| -| 1.x | Current | Full | -| 0.x | Legacy | Security only | - -### Supported Platforms - -- **Rust**: 1.70+ -- **Julia**: 1.9+ -- **Isabelle**: 2023+ -- **OS**: Linux, macOS, Windows (via WSL2) -- **Containers**: Podman with Chainguard Wolfi - -## Self-Help Resources - -### Common Issues - -**Build fails with missing dependencies**: -```bash -# Check all dependencies -just check-deps - -# Install with Nix (recommended) -nix develop -``` - -**Checksum verification fails**: -```bash -# Verify dataset integrity -vae-normalizer verify -o /path/to/output --checksums -d /path/to/dataset -``` - -**Julia package errors**: -```bash -# Reinstall Julia dependencies -just julia-setup -``` - -### Debug Mode - -Enable verbose output for troubleshooting: - -```bash -vae-normalizer -v normalize -d /path/to/dataset -o /path/to/output -``` - -## Community - -This is a research project maintained by volunteers. Please be patient -and respectful in all interactions. - -See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for community guidelines. diff --git a/TEST-NEEDS.adoc b/TEST-NEEDS.adoc new file mode 100644 index 0000000..db2fd5d --- /dev/null +++ b/TEST-NEEDS.adoc @@ -0,0 +1,67 @@ +== TEST-NEEDS: zerostep + +=== CRG Grade: C — ACHIEVED 2026-04-04 + +=== Current State + +[width="100%",cols="40%,26%,34%",options="header",] +|=== +|Category |Count |Details +|*Source modules* |7 |Rust (main, metadata), Julia (contrastive_model, +julia_utils), 3 Idris2 ABI + +|*Unit tests* |5 |test/runtests.jl (5 @test) + +|*Integration tests* |0 |None + +|*E2E tests* |0 |None + +|*Benchmarks* |0 |None + +|*Fuzz tests* |1 |fuzz/fuzz_targets/fuzz_input.rs +|=== + +=== What’s Missing + +==== P2P Tests + +* [ ] No tests for Rust <-> Julia interop +* [ ] No tests for contrastive model feeding into metadata processing + +==== E2E Tests + +* [ ] No test running zerostep binary end-to-end +* [ ] No test for the contrastive learning pipeline + +==== Aspect Tests + +* [ ] *Security*: No input validation tests +* [ ] *Performance*: ML contrastive model with 0 benchmarks +* [ ] *Concurrency*: No tests for parallel model execution +* [ ] *Error handling*: No tests for malformed input data, model +failures + +==== Build & Execution + +* [ ] No Rust compilation test +* [ ] No Idris2 ABI compilation test +* [ ] Nickel config (config.ncl) untested + +==== Benchmarks Needed + +* [ ] Contrastive model training time +* [ ] Inference latency +* [ ] Metadata processing throughput + +==== Self-Tests + +* [ ] No self-diagnostic mode + +=== FLAGGED ISSUES + +* *5 @test for 7 source modules* = barely tested +* *ML project with 0 benchmarks* – performance is the entire point +* *Fuzz target exists* (fuzz_input.rs) – rare positive +* *Julia model code with no Julia-specific tests* for the model itself + +=== Priority: P1 (HIGH) diff --git a/TEST-NEEDS.md b/TEST-NEEDS.md deleted file mode 100644 index 695a8de..0000000 --- a/TEST-NEEDS.md +++ /dev/null @@ -1,51 +0,0 @@ -# TEST-NEEDS: zerostep - -## CRG Grade: C — ACHIEVED 2026-04-04 - -## Current State - -| Category | Count | Details | -|----------|-------|---------| -| **Source modules** | 7 | Rust (main, metadata), Julia (contrastive_model, julia_utils), 3 Idris2 ABI | -| **Unit tests** | 5 | test/runtests.jl (5 @test) | -| **Integration tests** | 0 | None | -| **E2E tests** | 0 | None | -| **Benchmarks** | 0 | None | -| **Fuzz tests** | 1 | fuzz/fuzz_targets/fuzz_input.rs | - -## What's Missing - -### P2P Tests -- [ ] No tests for Rust <-> Julia interop -- [ ] No tests for contrastive model feeding into metadata processing - -### E2E Tests -- [ ] No test running zerostep binary end-to-end -- [ ] No test for the contrastive learning pipeline - -### Aspect Tests -- [ ] **Security**: No input validation tests -- [ ] **Performance**: ML contrastive model with 0 benchmarks -- [ ] **Concurrency**: No tests for parallel model execution -- [ ] **Error handling**: No tests for malformed input data, model failures - -### Build & Execution -- [ ] No Rust compilation test -- [ ] No Idris2 ABI compilation test -- [ ] Nickel config (config.ncl) untested - -### Benchmarks Needed -- [ ] Contrastive model training time -- [ ] Inference latency -- [ ] Metadata processing throughput - -### Self-Tests -- [ ] No self-diagnostic mode - -## FLAGGED ISSUES -- **5 @test for 7 source modules** = barely tested -- **ML project with 0 benchmarks** -- performance is the entire point -- **Fuzz target exists** (fuzz_input.rs) -- rare positive -- **Julia model code with no Julia-specific tests** for the model itself - -## Priority: P1 (HIGH) diff --git a/TOPOLOGY.md b/TOPOLOGY.adoc similarity index 89% rename from TOPOLOGY.md rename to TOPOLOGY.adoc index 8e56dd2..1942393 100644 --- a/TOPOLOGY.md +++ b/TOPOLOGY.adoc @@ -1,12 +1,8 @@ - - - +== VAE Dataset Normalizer (zerostep) — Project Topology -# VAE Dataset Normalizer (zerostep) — Project Topology +=== System Architecture -## System Architecture - -``` +.... ┌─────────────────────────────────────────┐ │ ML RESEARCHER │ │ (CLI / Julia Notebook) │ @@ -46,11 +42,11 @@ │ Justfile Automation .machine_readable/ │ │ Nix / Wolfi 0-AI-MANIFEST.a2ml │ └─────────────────────────────────────────┘ -``` +.... -## Completion Dashboard +=== Completion Dashboard -``` +.... COMPONENT STATUS NOTES ───────────────────────────────── ────────────────── ───────────────────────────────── CORE NORMALIZATION @@ -71,25 +67,26 @@ REPO INFRASTRUCTURE ───────────────────────────────────────────────────────────────────────────── OVERALL: ██████████ 100% Production-ready ML infrastructure -``` +.... -## Key Dependencies +=== Key Dependencies -``` +.... Raw Dataset ──────► Checksum Engine ────► Diff Comp ──────► Dataset Splits │ │ │ │ ▼ ▼ ▼ ▼ CUE Schema ───────► Nickel Config ─────► Isabelle Proof ───► Julia Model -``` +.... -## Update Protocol +=== Update Protocol This file is maintained by both humans and AI agents. When updating: -1. **After completing a component**: Change its bar and percentage -2. **After adding a component**: Add a new row in the appropriate section -3. **After architectural changes**: Update the ASCII diagram -4. **Date**: Update the `Last updated` comment at the top of this file +[arabic] +. *After completing a component*: Change its bar and percentage +. *After adding a component*: Add a new row in the appropriate section +. *After architectural changes*: Update the ASCII diagram +. *Date*: Update the `+Last updated+` comment at the top of this file -Progress bars use: `█` (filled) and `░` (empty), 10 characters wide. -Percentages: 0%, 10%, 20%, ... 100% (in 10% increments). +Progress bars use: `+█+` (filled) and `+░+` (empty), 10 characters wide. +Percentages: 0%, 10%, 20%, … 100% (in 10% increments). diff --git a/docs/tech-debt-2026-05-26.adoc b/docs/tech-debt-2026-05-26.adoc new file mode 100644 index 0000000..0c4317f --- /dev/null +++ b/docs/tech-debt-2026-05-26.adoc @@ -0,0 +1,71 @@ +== Tech-Debt Audit — zerostep — 2026-05-26 + +*Source:* estate-wide automated scan 2026-05-26. *Companion:* +https://github.com/hyperpolymath/standards/tree/main/docs/audits[`+hyperpolymath/standards+` +2026-05-26-estate-*-debt audits]. *Combined severity:* `+MEDIUM+`. + +This file records the _raw findings_ — it does not by itself fix the +debt. Each section ends with a '`Recommended next move`' line; closing +the debt is follow-up work. + +=== 1. Proof debt + +No proof-bearing files (`+*.v+`, `+*.lean+`, `+*.agda+`, `+*.idr+`, +`+*.idr2+`, `+*.fst+`, `+*.dfy+`, `+*.tla+`, `+*.ads+`, `+*.adb+`) found +in this repo. + +*Recommended next move:* none. + +=== 2. Licence debt + +[cols=",",options="header",] +|=== +|Field |Value +|LICENSE file |`+LICENSE+` +|SPDX header |`+MPL-2.0+` +|Manifest licence |`+MIT+` +|Body classifier |`+Palimp-MPL-2.0+` +|Severity |`+ok+` +|=== + +*Recommended next move:* none for licence. + +=== 3. Documentation debt + +[cols=",",options="header",] +|=== +|Field |Value +|README lines |352 +|`+docs/+` files |1 +|`+docs/+` LoC |36 +|CHANGELOG.md |N +|CONTRIBUTING.md |Y +|CODE_OF_CONDUCT.md |Y +|SECURITY.md |Y +|Severity |`+MEDIUM+` +|=== + +*Recommended next move:* introduce a `+docs/+` directory. The README at +352 lines has likely grown to do the work of `+docs/+` — split it into a +thin README + `+docs/architecture.md+`, `+docs/usage.md+`, etc. +Heavy-wiki exemplars to copy from: `+affinescript+`, `+boj-server+`, +`+echidna+`, `+hypatia+`. + +Additionally: *CHANGELOG.md is missing.* 65% of estate repos lack one — +adopting a CHANGELOG (or auto-generating via `+git-cliff+`) is a +recommended estate-wide follow-up. + +=== Cross-references + +* Estate proof-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md+` +* Estate licence-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md+` +* Estate documentation-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md+` + +''''' + +🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). +This file is informational — closing the debt is follow-up work owned by +the maintainer. diff --git a/docs/tech-debt-2026-05-26.md b/docs/tech-debt-2026-05-26.md deleted file mode 100644 index cd922b4..0000000 --- a/docs/tech-debt-2026-05-26.md +++ /dev/null @@ -1,57 +0,0 @@ - - -# Tech-Debt Audit — zerostep — 2026-05-26 - -**Source:** estate-wide automated scan 2026-05-26. -**Companion:** [`hyperpolymath/standards` 2026-05-26-estate-*-debt audits](https://github.com/hyperpolymath/standards/tree/main/docs/audits). -**Combined severity:** `MEDIUM`. - -This file records the *raw findings* — it does not by itself fix the debt. Each section ends with a 'Recommended next move' line; closing the debt is follow-up work. - -## 1. Proof debt - -No proof-bearing files (`*.v`, `*.lean`, `*.agda`, `*.idr`, `*.idr2`, `*.fst`, `*.dfy`, `*.tla`, `*.ads`, `*.adb`) found in this repo. - -**Recommended next move:** none. - -## 2. Licence debt - -| Field | Value | -|---|---| -| LICENSE file | `LICENSE` | -| SPDX header | `MPL-2.0` | -| Manifest licence | `MIT` | -| Body classifier | `Palimp-MPL-2.0` | -| Severity | `ok` | - -**Recommended next move:** none for licence. - -## 3. Documentation debt - -| Field | Value | -|---|---| -| README lines | 352 | -| `docs/` files | 1 | -| `docs/` LoC | 36 | -| CHANGELOG.md | N | -| CONTRIBUTING.md | Y | -| CODE_OF_CONDUCT.md | Y | -| SECURITY.md | Y | -| Severity | `MEDIUM` | - -**Recommended next move:** introduce a `docs/` directory. The README at 352 lines has likely grown to do the work of `docs/` — split it into a thin README + `docs/architecture.md`, `docs/usage.md`, etc. Heavy-wiki exemplars to copy from: `affinescript`, `boj-server`, `echidna`, `hypatia`. - -Additionally: **CHANGELOG.md is missing.** 65% of estate repos lack one — adopting a CHANGELOG (or auto-generating via `git-cliff`) is a recommended estate-wide follow-up. - -## Cross-references - -- Estate proof-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md` -- Estate licence-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md` -- Estate documentation-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md` - ---- - -🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). This file is informational — closing the debt is follow-up work owned by the maintainer. diff --git a/llm-warmup-dev.adoc b/llm-warmup-dev.adoc new file mode 100644 index 0000000..86bbd67 --- /dev/null +++ b/llm-warmup-dev.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — zerostep (Developer) + +=== What is zerostep? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-dev.md b/llm-warmup-dev.md deleted file mode 100644 index 272cf90..0000000 --- a/llm-warmup-dev.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — zerostep (Developer) - -## What is zerostep? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.adoc b/llm-warmup-user.adoc new file mode 100644 index 0000000..a82dfcb --- /dev/null +++ b/llm-warmup-user.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — zerostep (User) + +=== What is zerostep? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.md b/llm-warmup-user.md deleted file mode 100644 index 3112301..0000000 --- a/llm-warmup-user.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — zerostep (User) - -## What is zerostep? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture diff --git a/vae-normalizer/DCO.adoc b/vae-normalizer/DCO.adoc new file mode 100644 index 0000000..c13bb3f --- /dev/null +++ b/vae-normalizer/DCO.adoc @@ -0,0 +1,73 @@ +== Developer Certificate of Origin + +Version 1.1 + +Copyright (C) 2004, 2006 The Linux Foundation and its contributors. + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +''''' + +=== Developer’s Certificate of Origin 1.1 + +By making a contribution to this project, I certify that: + +*(a)* The contribution was created in whole or in part by me and I have +the right to submit it under the open source license indicated in the +file; or + +*(b)* The contribution is based upon previous work that, to the best of +my knowledge, is covered under an appropriate open source license and I +have the right under that license to submit that work with +modifications, whether created in whole or in part by me, under the same +open source license (unless I am permitted to submit under a different +license), as indicated in the file; or + +*(c)* The contribution was provided directly to me by some other person +who certified (a), (b) or (c) and I have not modified it. + +*(d)* I understand and agree that this project and the contribution are +public and that a record of the contribution (including all personal +information I submit with it, including my sign-off) is maintained +indefinitely and may be redistributed consistent with this project or +the open source license(s) involved. + +''''' + +=== How to Sign Off + +To sign off your commits, add the following line to your commit message: + +.... +Signed-off-by: Your Name +.... + +You can automate this with: + +[source,bash] +---- +git commit -s -m "Your commit message" +---- + +Or configure git to always sign off: + +[source,bash] +---- +git config --local format.signoff true +---- + +''''' + +=== Why DCO? + +The Developer Certificate of Origin ensures: + +[arabic] +. *Provenance*: Clear chain of authorship +. *Legal clarity*: Contributors affirm they have rights to contribute +. *Community trust*: Transparent contribution process +. *License compliance*: Contributions match project licenses + +This aligns with RSR (Rhodium Standard Repository) accountability +requirements. diff --git a/vae-normalizer/DCO.md b/vae-normalizer/DCO.md deleted file mode 100644 index 9973f94..0000000 --- a/vae-normalizer/DCO.md +++ /dev/null @@ -1,74 +0,0 @@ - - - -# Developer Certificate of Origin - -Version 1.1 - -Copyright (C) 2004, 2006 The Linux Foundation and its contributors. - -Everyone is permitted to copy and distribute verbatim copies of this -license document, but changing it is not allowed. - ---- - -## Developer's Certificate of Origin 1.1 - -By making a contribution to this project, I certify that: - -**(a)** The contribution was created in whole or in part by me and I -have the right to submit it under the open source license -indicated in the file; or - -**(b)** The contribution is based upon previous work that, to the best -of my knowledge, is covered under an appropriate open source -license and I have the right under that license to submit that -work with modifications, whether created in whole or in part -by me, under the same open source license (unless I am -permitted to submit under a different license), as indicated -in the file; or - -**(c)** The contribution was provided directly to me by some other -person who certified (a), (b) or (c) and I have not modified -it. - -**(d)** I understand and agree that this project and the contribution -are public and that a record of the contribution (including all -personal information I submit with it, including my sign-off) is -maintained indefinitely and may be redistributed consistent with -this project or the open source license(s) involved. - ---- - -## How to Sign Off - -To sign off your commits, add the following line to your commit message: - -``` -Signed-off-by: Your Name -``` - -You can automate this with: - -```bash -git commit -s -m "Your commit message" -``` - -Or configure git to always sign off: - -```bash -git config --local format.signoff true -``` - ---- - -## Why DCO? - -The Developer Certificate of Origin ensures: - -1. **Provenance**: Clear chain of authorship -2. **Legal clarity**: Contributors affirm they have rights to contribute -3. **Community trust**: Transparent contribution process -4. **License compliance**: Contributions match project licenses - -This aligns with RSR (Rhodium Standard Repository) accountability requirements. diff --git a/vae-normalizer/SUPPORT.adoc b/vae-normalizer/SUPPORT.adoc new file mode 100644 index 0000000..7fc1c0d --- /dev/null +++ b/vae-normalizer/SUPPORT.adoc @@ -0,0 +1,108 @@ +== Support + +This document describes how to get help with the VAE Dataset Normalizer. + +=== Getting Help + +==== Documentation + +Start with the documentation: + +* README.adoc - Overview and quick start +* QUICKSTART.md - Step-by-step guide +* CONTRIBUTING.adoc - Contribution guidelines + +==== Issue Tracker + +For bugs and feature requests, use the issue tracker: + +* *GitLab*: https://gitlab.com/hyperpolymath/zerostep/-/issues + +Before opening an issue: + +[arabic] +. Search existing issues to avoid duplicates +. Include reproduction steps for bugs +. Provide system information (OS, Rust version, etc.) + +==== Security Issues + +For security vulnerabilities, *do not* open a public issue. + +See SECURITY.md for responsible disclosure procedures. + +=== Support Channels + +[cols=",,",options="header",] +|=== +|Channel |Response Time |Use For +|Issue tracker |1-7 days |Bugs, features, documentation +|Email |7-14 days |Private inquiries +|Security email |48 hours |Vulnerabilities +|=== + +=== What We Support + +==== Supported Versions + +[cols=",,",options="header",] +|=== +|Version |Status |Support +|1.x |Current |Full +|0.x |Legacy |Security only +|=== + +==== Supported Platforms + +* *Rust*: 1.70+ +* *Julia*: 1.9+ +* *Isabelle*: 2023+ +* *OS*: Linux, macOS, Windows (via WSL2) +* *Containers*: Podman with Chainguard Wolfi + +=== Self-Help Resources + +==== Common Issues + +*Build fails with missing dependencies*: + +[source,bash] +---- +# Check all dependencies +just check-deps + +# Install with Nix (recommended) +nix develop +---- + +*Checksum verification fails*: + +[source,bash] +---- +# Verify dataset integrity +vae-normalizer verify -o /path/to/output --checksums -d /path/to/dataset +---- + +*Julia package errors*: + +[source,bash] +---- +# Reinstall Julia dependencies +just julia-setup +---- + +==== Debug Mode + +Enable verbose output for troubleshooting: + +[source,bash] +---- +vae-normalizer -v normalize -d /path/to/dataset -o /path/to/output +---- + +=== Community + +This is a research project maintained by volunteers. Please be patient +and respectful in all interactions. + +See CODE_OF_CONDUCT.md for community guidelines. diff --git a/vae-normalizer/SUPPORT.md b/vae-normalizer/SUPPORT.md deleted file mode 100644 index 12042f6..0000000 --- a/vae-normalizer/SUPPORT.md +++ /dev/null @@ -1,99 +0,0 @@ - - - -# Support - -This document describes how to get help with the VAE Dataset Normalizer. - -## Getting Help - -### Documentation - -Start with the documentation: - -- [README.adoc](README.adoc) - Overview and quick start -- [QUICKSTART.md](QUICKSTART.md) - Step-by-step guide -- [CONTRIBUTING.adoc](CONTRIBUTING.adoc) - Contribution guidelines - -### Issue Tracker - -For bugs and feature requests, use the issue tracker: - -- **GitLab**: https://gitlab.com/hyperpolymath/zerostep/-/issues - -Before opening an issue: - -1. Search existing issues to avoid duplicates -2. Include reproduction steps for bugs -3. Provide system information (OS, Rust version, etc.) - -### Security Issues - -For security vulnerabilities, **do not** open a public issue. - -See [SECURITY.md](SECURITY.md) for responsible disclosure procedures. - -## Support Channels - -| Channel | Response Time | Use For | -|---------|---------------|---------| -| Issue tracker | 1-7 days | Bugs, features, documentation | -| Email | 7-14 days | Private inquiries | -| Security email | 48 hours | Vulnerabilities | - -## What We Support - -### Supported Versions - -| Version | Status | Support | -|---------|--------|---------| -| 1.x | Current | Full | -| 0.x | Legacy | Security only | - -### Supported Platforms - -- **Rust**: 1.70+ -- **Julia**: 1.9+ -- **Isabelle**: 2023+ -- **OS**: Linux, macOS, Windows (via WSL2) -- **Containers**: Podman with Chainguard Wolfi - -## Self-Help Resources - -### Common Issues - -**Build fails with missing dependencies**: -```bash -# Check all dependencies -just check-deps - -# Install with Nix (recommended) -nix develop -``` - -**Checksum verification fails**: -```bash -# Verify dataset integrity -vae-normalizer verify -o /path/to/output --checksums -d /path/to/dataset -``` - -**Julia package errors**: -```bash -# Reinstall Julia dependencies -just julia-setup -``` - -### Debug Mode - -Enable verbose output for troubleshooting: - -```bash -vae-normalizer -v normalize -d /path/to/dataset -o /path/to/output -``` - -## Community - -This is a research project maintained by volunteers. Please be patient -and respectful in all interactions. - -See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) for community guidelines.