diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc new file mode 100644 index 0000000..b6a8dde --- /dev/null +++ b/CHANGELOG.adoc @@ -0,0 +1,80 @@ +== Changelog + +All notable changes to `+nexia-list+` will be documented in this file. + +This file is generated from conventional commits by the +https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml[`+changelog-reusable.yml+`] +workflow (`+hyperpolymath/standards#206+`). Adopt the workflow in this +repo’s CI to keep this file in sync automatically — see +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+templates/cliff.toml+`] +for the canonical config. + +The format follows https://keepachangelog.com/en/1.1.0/[Keep a +Changelog]; this project aims to follow +https://semver.org/spec/v2.0.0.html[Semantic Versioning]. + +=== [Unreleased] + +==== Added + +* feat(crg): add crg-grade and crg-badge justfile recipes +* feat: add stapeln.toml container definition +* feat: add UX Justfile with doctor, tour, help-me, assail recipes +* feat: deploy UX Manifesto infrastructure +* feat(desktop): replace Tauri with Gossamer — gossamer-rs backend +* feat: Gossamer migration — RuntimeBridge, gossamer.conf.json, +Tauri→Gossamer conversion +* feat: add CLADE.a2ml — clade taxonomy declaration +* feat: add mirror.yml workflow for GitLab/Bitbucket mirroring +* feat: add AI Gatekeeper Protocol manifest +* feat(ci): enable Hypatia scanning + +==== Fixed + +* fix(ci): bump a2ml/k9-validate-action pins to canonical (standards#85) +(#7) +* fix(ci): sync hypatia-scan.yml to canonical (kill cd-scanner build +drift) (#6) +* fix(ci): build Hypatia escript from repo root (estate dogfood drift) +* fix(ci): repair YAML block-scalar in workflow-linter Check Permissions +step (#4) +* fix(scorecard): enforce granular permissions and add fuzzing +placeholder +* fix(ci): Resolve workflow-linter self-matching and metadata issues +* fix: RSR compliance — fix SPDX headers, resolve placeholders, rewrite +stale SCM +* fix: remove duplicate SCM files from root + +==== Changed + +* refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) + +==== Documentation + +* docs: add post-audit status report for M5 sweep +* docs: add TEST-NEEDS.md (CRG C) +* docs: add EXPLAINME.adoc — prove-it file backing README claims +* docs: update SCM files with project information +* docs: add CODE_OF_CONDUCT.md +* docs: add CONTRIBUTING.md +* docs: add checkpoint files for state tracking +* docs: update license from AGPL to PMPL + +==== CI + +* ci: redistribute concurrency-cancel guard to read-only check workflows +(#9) +* ci: bump actions/upload-artifact SHA to current v4 (#3) +* ci: SHA-pin hyperpolymath validate-actions in dogfood-gate +* ci: wire hypatia-scan.yml to query own Dependabot alerts +* ci: deploy dogfood-gate, fix hypatia-scan, add pre-commit hooks + +=== Pre-history + +Prior commits to this file’s introduction are recorded in git history +but not formally classified into Keep-a-Changelog sections. To backfill, +run `+git cliff -o CHANGELOG.md+` locally using the canonical +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+cliff.toml+`] +— this is one-shot mechanical work. + +''''' diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 078c31d..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,74 +0,0 @@ - - -# Changelog - -All notable changes to `nexia-list` will be documented in this file. - -This file is generated from conventional commits by the -[`changelog-reusable.yml`](https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml) -workflow (`hyperpolymath/standards#206`). Adopt the workflow in this repo's CI to keep this file in sync automatically — see -[`templates/cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) -for the canonical config. - -The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); -this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -### Added - -- feat(crg): add crg-grade and crg-badge justfile recipes -- feat: add stapeln.toml container definition -- feat: add UX Justfile with doctor, tour, help-me, assail recipes -- feat: deploy UX Manifesto infrastructure -- feat(desktop): replace Tauri with Gossamer — gossamer-rs backend -- feat: Gossamer migration — RuntimeBridge, gossamer.conf.json, Tauri→Gossamer conversion -- feat: add CLADE.a2ml — clade taxonomy declaration -- feat: add mirror.yml workflow for GitLab/Bitbucket mirroring -- feat: add AI Gatekeeper Protocol manifest -- feat(ci): enable Hypatia scanning - -### Fixed - -- fix(ci): bump a2ml/k9-validate-action pins to canonical (standards#85) (#7) -- fix(ci): sync hypatia-scan.yml to canonical (kill cd-scanner build drift) (#6) -- fix(ci): build Hypatia escript from repo root (estate dogfood drift) -- fix(ci): repair YAML block-scalar in workflow-linter Check Permissions step (#4) -- fix(scorecard): enforce granular permissions and add fuzzing placeholder -- fix(ci): Resolve workflow-linter self-matching and metadata issues -- fix: RSR compliance — fix SPDX headers, resolve placeholders, rewrite stale SCM -- fix: remove duplicate SCM files from root - -### Changed - -- refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) - -### Documentation - -- docs: add post-audit status report for M5 sweep -- docs: add TEST-NEEDS.md (CRG C) -- docs: add EXPLAINME.adoc — prove-it file backing README claims -- docs: update SCM files with project information -- docs: add CODE_OF_CONDUCT.md -- docs: add CONTRIBUTING.md -- docs: add checkpoint files for state tracking -- docs: update license from AGPL to PMPL - -### CI - -- ci: redistribute concurrency-cancel guard to read-only check workflows (#9) -- ci: bump actions/upload-artifact SHA to current v4 (#3) -- ci: SHA-pin hyperpolymath validate-actions in dogfood-gate -- ci: wire hypatia-scan.yml to query own Dependabot alerts -- ci: deploy dogfood-gate, fix hypatia-scan, add pre-commit hooks - -## Pre-history - -Prior commits to this file's introduction are recorded in git history but not formally classified into Keep-a-Changelog sections. To backfill, run `git cliff -o CHANGELOG.md` locally using the canonical [`cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) — this is one-shot mechanical work. - ---- - - diff --git a/CODE_OF_CONDUCT.adoc b/CODE_OF_CONDUCT.adoc new file mode 100644 index 0000000..18135e4 --- /dev/null +++ b/CODE_OF_CONDUCT.adoc @@ -0,0 +1,339 @@ +== Code of Conduct + +=== Our Pledge + +We as members, contributors, and leaders pledge to make participation in +Nexia-List a harassment-free experience for everyone, regardless of age, +body size, visible or invisible disability, ethnicity, sex +characteristics, gender identity and expression, level of experience, +education, socio-economic status, nationality, personal appearance, +race, caste, colour, religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, +welcoming, diverse, inclusive, and healthy community. + +We recognise that a thriving open source community requires +*psychological safety* — an environment where people can contribute, ask +questions, make mistakes, and learn without fear of ridicule or +retaliation. + +''''' + +=== Our Standards + +==== Expected Behaviour + +The following behaviours contribute to a positive environment: + +*Communication* - Using welcoming and inclusive language - Being +respectful of differing viewpoints and experiences - Giving and +gracefully accepting constructive feedback - Assuming good intent while +addressing impact - Communicating clearly and patiently, especially with +newcomers + +*Collaboration* - Focusing on what is best for the community - Showing +empathy and kindness toward other community members - Being +collaborative rather than competitive - Mentoring and supporting less +experienced contributors - Celebrating others’ contributions and +successes + +*Professionalism* - Accepting responsibility and apologising to those +affected by our mistakes - Learning from the experience and avoiding +repetition - Respecting others’ time and attention - Staying on topic in +project spaces - Following project guidelines and conventions + +*Accessibility* - Using plain language and avoiding unnecessary jargon - +Providing alt text for images and transcripts for audio/video - Being +patient with those using assistive technologies - Accommodating +different communication styles and needs - Recognising that not everyone +communicates the same way + +==== Unacceptable Behaviour + +The following behaviours are considered harassment and are unacceptable: + +*Harassment* - The use of sexualised language or imagery, and sexual +attention or advances of any kind - Trolling, insulting or derogatory +comments, and personal or political attacks - Public or private +harassment - Deliberate intimidation, stalking, or following (online or +in-person) - Unwelcome physical contact or simulated physical contact +(e.g., emoji) - Sustained disruption of talks, events, or online +discussions + +*Discrimination* - Discriminatory jokes and language - Posting or +threatening to post others’ personally identifying information +("`doxing`") - Advocating for, or encouraging, any of the above +behaviour - Microaggressions — subtle, often unintentional, +discriminatory comments or actions + +*Professional Misconduct* - Publishing others’ private information +without explicit permission - Misrepresenting affiliation or +contributions - Plagiarism or claiming credit for others’ work - +Retaliating against anyone who reports a Code of Conduct violation - +Other conduct which could reasonably be considered inappropriate in a +professional setting + +==== Grey Areas + +Some situations require judgement. When uncertain: + +* *Intent vs Impact*: Good intentions do not excuse harmful impact. +Focus on making things right. +* *Power Dynamics*: Those with more power (maintainers, employers, +experienced contributors) must be especially mindful of their impact. +* *Cultural Differences*: What’s acceptable varies by culture. When in +doubt, err on the side of caution and ask. +* *Humour*: Jokes at others’ expense are rarely funny to everyone. Punch +up, not down. + +''''' + +=== Scope + +This Code of Conduct applies within all community spaces, including: + +*Online Spaces* - Repository discussions, issues, and pull/merge +requests - Project chat channels (Matrix, Discord, Slack, IRC) - Mailing +lists and forums - Social media when representing the project - Video +calls and virtual meetings + +*In-Person Spaces* - Conferences, meetups, and events - Workshops and +training sessions - Any gathering where you represent the project + +*Representation* This Code of Conduct also applies when an individual is +officially representing the community in public spaces. Examples +include: + +* Using an official project email address +* Posting via an official social media account +* Acting as an appointed representative at an event +* Speaking on behalf of the project + +''''' + +=== Enforcement + +==== Reporting + +If you experience or witness unacceptable behaviour, or have any other +concerns, please report it as soon as possible. + +*How to Report* + +[width="99%",cols="30%,33%,37%",options="header",] +|=== +|Method |Details |Best For +|*Email* |j.d.a.jewell@open.ac.uk |Detailed reports, sensitive matters + +|*Private Message* |Contact any maintainer directly |Quick questions, +minor issues + +|*Anonymous Form* |[Link to form if available] |When you need anonymity +|=== + +*What to Include* + +* Your contact information (unless anonymous) +* Names/usernames of those involved +* Description of what happened +* When and where it occurred +* Any witnesses +* Any supporting evidence (screenshots, links) +* How you would like us to respond (if you have a preference) + +*What Happens Next* + +[arabic] +. You will receive acknowledgment within *48 hours* +. The Nexia-List Maintainers will review the report +. We may ask for additional information +. We will determine appropriate action +. We will inform you of the outcome (respecting others’ privacy) + +==== Confidentiality + +All reports will be handled with discretion: + +* Reporter identity is protected by default +* Details are shared only with those who need to know +* We will ask before naming you in any communication +* Anonymous reports are accepted and investigated + +==== Conflicts of Interest + +If a Nexia-List Maintainers member is involved in an incident: + +* They will recuse themselves from the process +* Another maintainer or external party will handle the report +* We will disclose any potential conflicts + +''''' + +=== Enforcement Guidelines + +The Nexia-List Maintainers will follow these guidelines in determining +consequences: + +==== 1. Correction + +*Community Impact*: Use of inappropriate language or other behaviour +deemed unprofessional or unwelcome. + +*Consequence*: A private, written warning providing clarity around the +nature of the violation and an explanation of why the behaviour was +inappropriate. A public apology may be requested. + +*Duration*: Immediate + +==== 2. Warning + +*Community Impact*: A violation through a single incident or series of +actions. + +*Consequence*: A warning with consequences for continued behaviour. No +interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, for a specified period. This +includes avoiding interactions in community spaces as well as external +channels like social media. Violating these terms may lead to a +temporary or permanent ban. + +*Duration*: 1-4 weeks + +==== 3. Temporary Ban + +*Community Impact*: A serious violation of community standards, +including sustained inappropriate behaviour. + +*Consequence*: A temporary ban from any sort of interaction or public +communication with the community for a specified period. No public or +private interaction with the people involved, including unsolicited +interaction with those enforcing the Code of Conduct, is allowed during +this period. Violating these terms may lead to a permanent ban. + +*Duration*: 1-6 months + +==== 4. Permanent Ban + +*Community Impact*: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behaviour, harassment of an +individual, or aggression toward or disparagement of classes of +individuals. + +*Consequence*: A permanent ban from any sort of public interaction +within the community. + +*Duration*: Permanent (with appeal rights after 12 months) + +==== Enforcement Across Perimeters + +For contributors with elevated access (Perimeter 2 or 1): + +[cols=",",options="header",] +|=== +|Level |Additional Consequence +|Correction |Noted in contributor record +|Warning |Access privileges may be temporarily reduced +|Temporary Ban |Access reduced to Perimeter 3 for ban duration +|Permanent Ban |All access revoked +|=== + +''''' + +=== Appeals + +If you believe an enforcement decision was made in error: + +[arabic] +. *Wait 7 days* after the decision (cooling-off period) +. *Email* j.d.a.jewell@open.ac.uk with subject line "`Appeal: [Original +Report ID]`" +. *Explain* why you believe the decision should be reconsidered +. *Provide* any new information not previously available + +*Appeals Process* + +* Appeals are reviewed by a different Nexia-List Maintainers member than +the original +* You will receive a response within 14 days +* The appeals decision is final +* You may only appeal once per incident + +*Grounds for Appeal* + +* Procedural errors in the original investigation +* New evidence not previously available +* Disproportionate response to the violation +* Misunderstanding of facts + +''''' + +=== Supporting Those Who Report + +We are committed to supporting those who report violations: + +*We Will* - Believe and take all reports seriously - Respect your +privacy and confidentiality preferences - Keep you informed of progress +(if you wish) - Take steps to protect you from retaliation - Provide +resources if you need support + +*We Will Not* - Require you to confront the person directly - Dismiss +reports without investigation - Reveal your identity without consent - +Tolerate retaliation against reporters - Rush you to make decisions + +''''' + +=== Prevention + +Beyond enforcement, we actively work to prevent issues: + +*Onboarding* - All contributors are expected to read this Code of +Conduct - Perimeter 2 applicants must confirm they’ve read and +understood it - Maintainers receive additional training on enforcement + +*Culture* - We model the behaviour we expect - We intervene early when +we see potential issues - We thank people for positive contributions - +We create opportunities for diverse voices + +*Review* - This Code of Conduct is reviewed annually - Community +feedback is welcomed - Changes are communicated clearly + +''''' + +=== Acknowledgments + +This Code of Conduct is adapted from: + +* https://www.contributor-covenant.org/[Contributor Covenant], version +2.1 +* https://www.djangoproject.com/conduct/[Django Code of Conduct] +* https://www.rust-lang.org/policies/code-of-conduct[Rust Code of +Conduct] +* https://www.python.org/psf/conduct/[Python Community Code of Conduct] + +We thank these communities for their leadership in creating welcoming +spaces. + +''''' + +=== Questions? + +If you have questions about this Code of Conduct: + +* Open a +https://github.com/hyperpolymath/nexia-list/discussions[Discussion] (for +general questions) +* Email j.d.a.jewell@open.ac.uk (for private questions) +* Contact any maintainer directly + +''''' + +=== Summary + +*Be kind. Be respectful. Be collaborative.* + +We’re all here because we care about this project. Let’s make it a place +where everyone can do their best work. + +''''' + +Last updated: 2026 · Based on Contributor Covenant 2.1 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index d0a3a4b..0000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,308 +0,0 @@ - -# Code of Conduct - -## Our Pledge - -We as members, contributors, and leaders pledge to make participation in Nexia-List a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, colour, religion, or sexual identity and orientation. - -We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. - -We recognise that a thriving open source community requires **psychological safety** — an environment where people can contribute, ask questions, make mistakes, and learn without fear of ridicule or retaliation. - ---- - -## Our Standards - -### Expected Behaviour - -The following behaviours contribute to a positive environment: - -**Communication** -- Using welcoming and inclusive language -- Being respectful of differing viewpoints and experiences -- Giving and gracefully accepting constructive feedback -- Assuming good intent while addressing impact -- Communicating clearly and patiently, especially with newcomers - -**Collaboration** -- Focusing on what is best for the community -- Showing empathy and kindness toward other community members -- Being collaborative rather than competitive -- Mentoring and supporting less experienced contributors -- Celebrating others' contributions and successes - -**Professionalism** -- Accepting responsibility and apologising to those affected by our mistakes -- Learning from the experience and avoiding repetition -- Respecting others' time and attention -- Staying on topic in project spaces -- Following project guidelines and conventions - -**Accessibility** -- Using plain language and avoiding unnecessary jargon -- Providing alt text for images and transcripts for audio/video -- Being patient with those using assistive technologies -- Accommodating different communication styles and needs -- Recognising that not everyone communicates the same way - -### Unacceptable Behaviour - -The following behaviours are considered harassment and are unacceptable: - -**Harassment** -- The use of sexualised language or imagery, and sexual attention or advances of any kind -- Trolling, insulting or derogatory comments, and personal or political attacks -- Public or private harassment -- Deliberate intimidation, stalking, or following (online or in-person) -- Unwelcome physical contact or simulated physical contact (e.g., emoji) -- Sustained disruption of talks, events, or online discussions - -**Discrimination** -- Discriminatory jokes and language -- Posting or threatening to post others' personally identifying information ("doxing") -- Advocating for, or encouraging, any of the above behaviour -- Microaggressions — subtle, often unintentional, discriminatory comments or actions - -**Professional Misconduct** -- Publishing others' private information without explicit permission -- Misrepresenting affiliation or contributions -- Plagiarism or claiming credit for others' work -- Retaliating against anyone who reports a Code of Conduct violation -- Other conduct which could reasonably be considered inappropriate in a professional setting - -### Grey Areas - -Some situations require judgement. When uncertain: - -- **Intent vs Impact**: Good intentions do not excuse harmful impact. Focus on making things right. -- **Power Dynamics**: Those with more power (maintainers, employers, experienced contributors) must be especially mindful of their impact. -- **Cultural Differences**: What's acceptable varies by culture. When in doubt, err on the side of caution and ask. -- **Humour**: Jokes at others' expense are rarely funny to everyone. Punch up, not down. - ---- - -## Scope - -This Code of Conduct applies within all community spaces, including: - -**Online Spaces** -- Repository discussions, issues, and pull/merge requests -- Project chat channels (Matrix, Discord, Slack, IRC) -- Mailing lists and forums -- Social media when representing the project -- Video calls and virtual meetings - -**In-Person Spaces** -- Conferences, meetups, and events -- Workshops and training sessions -- Any gathering where you represent the project - -**Representation** -This Code of Conduct also applies when an individual is officially representing the community in public spaces. Examples include: - -- Using an official project email address -- Posting via an official social media account -- Acting as an appointed representative at an event -- Speaking on behalf of the project - ---- - -## Enforcement - -### Reporting - -If you experience or witness unacceptable behaviour, or have any other concerns, please report it as soon as possible. - -**How to Report** - -| Method | Details | Best For | -|--------|---------|----------| -| **Email** | j.d.a.jewell@open.ac.uk | Detailed reports, sensitive matters | -| **Private Message** | Contact any maintainer directly | Quick questions, minor issues | -| **Anonymous Form** | [Link to form if available] | When you need anonymity | - -**What to Include** - -- Your contact information (unless anonymous) -- Names/usernames of those involved -- Description of what happened -- When and where it occurred -- Any witnesses -- Any supporting evidence (screenshots, links) -- How you would like us to respond (if you have a preference) - -**What Happens Next** - -1. You will receive acknowledgment within **48 hours** -2. The Nexia-List Maintainers will review the report -3. We may ask for additional information -4. We will determine appropriate action -5. We will inform you of the outcome (respecting others' privacy) - -### Confidentiality - -All reports will be handled with discretion: - -- Reporter identity is protected by default -- Details are shared only with those who need to know -- We will ask before naming you in any communication -- Anonymous reports are accepted and investigated - -### Conflicts of Interest - -If a Nexia-List Maintainers member is involved in an incident: - -- They will recuse themselves from the process -- Another maintainer or external party will handle the report -- We will disclose any potential conflicts - ---- - -## Enforcement Guidelines - -The Nexia-List Maintainers will follow these guidelines in determining consequences: - -### 1. Correction - -**Community Impact**: Use of inappropriate language or other behaviour deemed unprofessional or unwelcome. - -**Consequence**: A private, written warning providing clarity around the nature of the violation and an explanation of why the behaviour was inappropriate. A public apology may be requested. - -**Duration**: Immediate - -### 2. Warning - -**Community Impact**: A violation through a single incident or series of actions. - -**Consequence**: A warning with consequences for continued behaviour. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. - -**Duration**: 1-4 weeks - -### 3. Temporary Ban - -**Community Impact**: A serious violation of community standards, including sustained inappropriate behaviour. - -**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. - -**Duration**: 1-6 months - -### 4. Permanent Ban - -**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behaviour, harassment of an individual, or aggression toward or disparagement of classes of individuals. - -**Consequence**: A permanent ban from any sort of public interaction within the community. - -**Duration**: Permanent (with appeal rights after 12 months) - -### Enforcement Across Perimeters - -For contributors with elevated access (Perimeter 2 or 1): - -| Level | Additional Consequence | -|-------|----------------------| -| Correction | Noted in contributor record | -| Warning | Access privileges may be temporarily reduced | -| Temporary Ban | Access reduced to Perimeter 3 for ban duration | -| Permanent Ban | All access revoked | - ---- - -## Appeals - -If you believe an enforcement decision was made in error: - -1. **Wait 7 days** after the decision (cooling-off period) -2. **Email** j.d.a.jewell@open.ac.uk with subject line "Appeal: [Original Report ID]" -3. **Explain** why you believe the decision should be reconsidered -4. **Provide** any new information not previously available - -**Appeals Process** - -- Appeals are reviewed by a different Nexia-List Maintainers member than the original -- You will receive a response within 14 days -- The appeals decision is final -- You may only appeal once per incident - -**Grounds for Appeal** - -- Procedural errors in the original investigation -- New evidence not previously available -- Disproportionate response to the violation -- Misunderstanding of facts - ---- - -## Supporting Those Who Report - -We are committed to supporting those who report violations: - -**We Will** -- Believe and take all reports seriously -- Respect your privacy and confidentiality preferences -- Keep you informed of progress (if you wish) -- Take steps to protect you from retaliation -- Provide resources if you need support - -**We Will Not** -- Require you to confront the person directly -- Dismiss reports without investigation -- Reveal your identity without consent -- Tolerate retaliation against reporters -- Rush you to make decisions - ---- - -## Prevention - -Beyond enforcement, we actively work to prevent issues: - -**Onboarding** -- All contributors are expected to read this Code of Conduct -- Perimeter 2 applicants must confirm they've read and understood it -- Maintainers receive additional training on enforcement - -**Culture** -- We model the behaviour we expect -- We intervene early when we see potential issues -- We thank people for positive contributions -- We create opportunities for diverse voices - -**Review** -- This Code of Conduct is reviewed annually -- Community feedback is welcomed -- Changes are communicated clearly - ---- - -## Acknowledgments - -This Code of Conduct is adapted from: - -- [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1 -- [Django Code of Conduct](https://www.djangoproject.com/conduct/) -- [Rust Code of Conduct](https://www.rust-lang.org/policies/code-of-conduct) -- [Python Community Code of Conduct](https://www.python.org/psf/conduct/) - -We thank these communities for their leadership in creating welcoming spaces. - ---- - -## Questions? - -If you have questions about this Code of Conduct: - -- Open a [Discussion](https://github.com/hyperpolymath/nexia-list/discussions) (for general questions) -- Email j.d.a.jewell@open.ac.uk (for private questions) -- Contact any maintainer directly - ---- - -## Summary - -**Be kind. Be respectful. Be collaborative.** - -We're all here because we care about this project. Let's make it a place where everyone can do their best work. - ---- - -Last updated: 2026 · Based on Contributor Covenant 2.1 diff --git a/CONTRIBUTING.adoc b/CONTRIBUTING.adoc new file mode 100644 index 0000000..79112ac --- /dev/null +++ b/CONTRIBUTING.adoc @@ -0,0 +1,151 @@ +== Contributing to Nexia-List + +=== Development Setup + +Prerequisites: https://bun.sh/[Bun] 1.3+ and +https://www.rust-lang.org/tools/install[Rust] stable (plus the +`+wasm32-unknown-unknown+` target for WASM builds). Bun is the only JS +toolchain — do not use npm/deno/yarn/pnpm. A Guix environment is +provided via `+guix.scm+` (`+guix shell+`) if you prefer reproducible +shells. + +[source,bash] +---- +# Clone the repository +git clone https://github.com/hyperpolymath/nexia-list.git +cd nexia-list + +# Install dependencies +bun install --frozen-lockfile + +# Run the development server (http://localhost:5173) +bun run dev + +# Build (ReScript + web bundle) +bun run build + +# Verify setup +bun run lint +bun run test # Rust core tests + UI tests +---- + +Equivalent `+just+` recipes exist: `+just setup+`, `+just build+`, +`+just test+`, `+just run+`, `+just check+`. + +==== Repository Structure + +.... +nexia-list/ +├── core/ # Rust core — notes, backlinks, search, JSON storage +├── ui/ # ReScript TEA-style UI (@rescript/react) +├── scripts/ # Bun build/dev scripts +├── web/ # Browser entry + bundle output (dist/) +├── desktop/ # OPTIONAL Gossamer shell (external sibling checkout; +│ # not built in this repo's CI) +├── docs/ # ADRs, reports +│ └── adr/ # Architecture decision records +├── tests/ # Cross-cutting tests +├── .well-known/ # Protocol files (ai.txt, security.txt, humans.txt) +├── .machine_readable/ # Contractiles, STATE/META/ECOSYSTEM checkpoints, +│ # and governance metadata (see below) +├── .github/ # GitHub config and workflows +├── CHANGELOG.md +├── CODE_OF_CONDUCT.md +├── CONTRIBUTING.md # This file +├── LICENSE +├── MAINTAINERS.adoc +├── README.adoc +├── ROADMAP.adoc +├── SECURITY.md +├── package.json # Bun tasks and dependencies +├── bun.lock # Exact JavaScript dependency graph +└── Justfile # Task runner recipes +.... + +Governance and invariants are machine-readable: see +link:.machine_readable/[`+.machine_readable/+`] (in particular +`+MUST.contractile+` and `+INTENT.contractile+`) and +link:0-AI-MANIFEST.a2ml[`+0-AI-MANIFEST.a2ml+`]. + +''''' + +=== How to Contribute + +==== Reporting Bugs + +*Before reporting*: 1. Search existing issues 2. Check if it’s already +fixed in `+main+` 3. Determine which perimeter the bug affects + +*When reporting*: + +Use the link:.github/ISSUE_TEMPLATE/bug_report.md[bug report template] +and include: + +* Clear, descriptive title +* Environment details (OS, versions, toolchain) +* Steps to reproduce +* Expected vs actual behaviour +* Logs, screenshots, or minimal reproduction + +==== Suggesting Features + +*Before suggesting*: 1. Check the link:ROADMAP.adoc[roadmap] 2. Search +existing issues and discussions 3. Consider which perimeter the feature +belongs to + +*When suggesting*: + +Use the link:.github/ISSUE_TEMPLATE/feature_request.md[feature request +template] and include: + +* Problem statement (what pain point does this solve?) +* Proposed solution +* Alternatives considered +* Which perimeter this affects + +==== Your First Contribution + +Look for issues labelled: + +* https://github.com/hyperpolymath/nexia-list/labels/good%20first%20issue[`+good first issue+`] +— Simple Perimeter 3 tasks +* https://github.com/hyperpolymath/nexia-list/labels/help%20wanted[`+help wanted+`] +— Community help needed +* https://github.com/hyperpolymath/nexia-list/labels/documentation[`+documentation+`] +— Docs improvements +* https://github.com/hyperpolymath/nexia-list/labels/perimeter-3[`+perimeter-3+`] +— Community sandbox scope + +''''' + +=== Development Workflow + +==== Branch Naming + +.... +docs/short-description # Documentation (P3) +test/what-added # Test additions (P3) +feat/short-description # New features (P2) +fix/issue-number-description # Bug fixes (P2) +refactor/what-changed # Code improvements (P2) +security/what-fixed # Security fixes (P1-2) +.... + +==== Commit Messages + +We follow https://www.conventionalcommits.org/[Conventional Commits]: + +.... +(): + +[optional body] + +[optional footer] +.... + +''''' + +=== Questions? + +See MAINTAINERS.adoc for who to contact, and SECURITY.md for reporting +vulnerabilities. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index df101ad..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,142 +0,0 @@ - -# Contributing to Nexia-List - -## Development Setup - -Prerequisites: [Bun](https://bun.sh/) 1.3+ and [Rust](https://www.rust-lang.org/tools/install) -stable (plus the `wasm32-unknown-unknown` target for WASM builds). Bun is the -only JS toolchain — do not use npm/deno/yarn/pnpm. A Guix environment is -provided via `guix.scm` (`guix shell`) if you prefer reproducible shells. - -```bash -# Clone the repository -git clone https://github.com/hyperpolymath/nexia-list.git -cd nexia-list - -# Install dependencies -bun install --frozen-lockfile - -# Run the development server (http://localhost:5173) -bun run dev - -# Build (ReScript + web bundle) -bun run build - -# Verify setup -bun run lint -bun run test # Rust core tests + UI tests -``` - -Equivalent `just` recipes exist: `just setup`, `just build`, `just test`, -`just run`, `just check`. - -### Repository Structure -``` -nexia-list/ -├── core/ # Rust core — notes, backlinks, search, JSON storage -├── ui/ # ReScript TEA-style UI (@rescript/react) -├── scripts/ # Bun build/dev scripts -├── web/ # Browser entry + bundle output (dist/) -├── desktop/ # OPTIONAL Gossamer shell (external sibling checkout; -│ # not built in this repo's CI) -├── docs/ # ADRs, reports -│ └── adr/ # Architecture decision records -├── tests/ # Cross-cutting tests -├── .well-known/ # Protocol files (ai.txt, security.txt, humans.txt) -├── .machine_readable/ # Contractiles, STATE/META/ECOSYSTEM checkpoints, -│ # and governance metadata (see below) -├── .github/ # GitHub config and workflows -├── CHANGELOG.md -├── CODE_OF_CONDUCT.md -├── CONTRIBUTING.md # This file -├── LICENSE -├── MAINTAINERS.adoc -├── README.adoc -├── ROADMAP.adoc -├── SECURITY.md -├── package.json # Bun tasks and dependencies -├── bun.lock # Exact JavaScript dependency graph -└── Justfile # Task runner recipes -``` - -Governance and invariants are machine-readable: see -[`.machine_readable/`](.machine_readable/) (in particular `MUST.contractile` -and `INTENT.contractile`) and [`0-AI-MANIFEST.a2ml`](0-AI-MANIFEST.a2ml). - ---- - -## How to Contribute - -### Reporting Bugs - -**Before reporting**: -1. Search existing issues -2. Check if it's already fixed in `main` -3. Determine which perimeter the bug affects - -**When reporting**: - -Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include: - -- Clear, descriptive title -- Environment details (OS, versions, toolchain) -- Steps to reproduce -- Expected vs actual behaviour -- Logs, screenshots, or minimal reproduction - -### Suggesting Features - -**Before suggesting**: -1. Check the [roadmap](ROADMAP.adoc) -2. Search existing issues and discussions -3. Consider which perimeter the feature belongs to - -**When suggesting**: - -Use the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and include: - -- Problem statement (what pain point does this solve?) -- Proposed solution -- Alternatives considered -- Which perimeter this affects - -### Your First Contribution - -Look for issues labelled: - -- [`good first issue`](https://github.com/hyperpolymath/nexia-list/labels/good%20first%20issue) — Simple Perimeter 3 tasks -- [`help wanted`](https://github.com/hyperpolymath/nexia-list/labels/help%20wanted) — Community help needed -- [`documentation`](https://github.com/hyperpolymath/nexia-list/labels/documentation) — Docs improvements -- [`perimeter-3`](https://github.com/hyperpolymath/nexia-list/labels/perimeter-3) — Community sandbox scope - ---- - -## Development Workflow - -### Branch Naming -``` -docs/short-description # Documentation (P3) -test/what-added # Test additions (P3) -feat/short-description # New features (P2) -fix/issue-number-description # Bug fixes (P2) -refactor/what-changed # Code improvements (P2) -security/what-fixed # Security fixes (P1-2) -``` - -### Commit Messages - -We follow [Conventional Commits](https://www.conventionalcommits.org/): -``` -(): - -[optional body] - -[optional footer] -``` - ---- - -## Questions? - -See [MAINTAINERS.adoc](MAINTAINERS.adoc) for who to contact, and -[SECURITY.md](SECURITY.md) for reporting vulnerabilities. diff --git a/GOVERNANCE.adoc b/GOVERNANCE.adoc new file mode 100644 index 0000000..9b836fb --- /dev/null +++ b/GOVERNANCE.adoc @@ -0,0 +1,60 @@ +== Governance + +=== Overview + +This project is governed by the following principles and structures to +ensure transparent, inclusive, and effective decision-making. + +=== Roles and Responsibilities + +==== Maintainers + +Maintainers are responsible for: - Reviewing and merging pull requests - +Managing releases and versioning - Ensuring code quality and standards - +Triaging issues and bug reports - Community engagement and support + +==== Contributors + +Contributors are expected to: - Follow the code of conduct - Submit +well-documented pull requests - Write tests for new functionality - +Maintain existing tests - Update documentation as needed + +=== Decision Making + +==== Minor Changes + +* Can be made by any maintainer +* Include bug fixes, documentation updates, dependency updates + +==== Major Changes + +* Require discussion in issues or pull requests +* Include new features, architectural changes, API changes +* Need approval from at least 2 maintainers + +==== Breaking Changes + +* Require RFC (Request for Comments) process +* Need approval from majority of maintainers +* Must include migration guide + +=== Code of Conduct + +All participants are expected to follow our Code of Conduct. Violations +can be reported to the maintainers. + +=== Communication + +* *Issues*: For bug reports and feature requests +* *Discussions*: For questions and general discussion +* *Pull Requests*: For code contributions + +=== Licensing + +All contributions are made under the terms of the repository’s LICENSE +file. By submitting a pull request, you agree to license your +contributions accordingly. + +''''' + +_Last updated: 2026-07-18_ diff --git a/GOVERNANCE.md b/GOVERNANCE.md deleted file mode 100644 index e27364c..0000000 --- a/GOVERNANCE.md +++ /dev/null @@ -1,60 +0,0 @@ -# Governance - -## Overview - -This project is governed by the following principles and structures to ensure transparent, inclusive, and effective decision-making. - -## Roles and Responsibilities - -### Maintainers - -Maintainers are responsible for: -- Reviewing and merging pull requests -- Managing releases and versioning -- Ensuring code quality and standards -- Triaging issues and bug reports -- Community engagement and support - -### Contributors - -Contributors are expected to: -- Follow the code of conduct -- Submit well-documented pull requests -- Write tests for new functionality -- Maintain existing tests -- Update documentation as needed - -## Decision Making - -### Minor Changes -- Can be made by any maintainer -- Include bug fixes, documentation updates, dependency updates - -### Major Changes -- Require discussion in issues or pull requests -- Include new features, architectural changes, API changes -- Need approval from at least 2 maintainers - -### Breaking Changes -- Require RFC (Request for Comments) process -- Need approval from majority of maintainers -- Must include migration guide - -## Code of Conduct - -All participants are expected to follow our Code of Conduct. Violations can be reported to the maintainers. - -## Communication - -- **Issues**: For bug reports and feature requests -- **Discussions**: For questions and general discussion -- **Pull Requests**: For code contributions - -## Licensing - -All contributions are made under the terms of the repository's LICENSE file. -By submitting a pull request, you agree to license your contributions accordingly. - ---- - -*Last updated: 2026-07-18* diff --git a/READINESS.adoc b/READINESS.adoc new file mode 100644 index 0000000..2d6d519 --- /dev/null +++ b/READINESS.adoc @@ -0,0 +1,68 @@ +== Readiness + +*Current Grade:* D + +____ +*Re-audit due — this grade is probably stale.* Every criterion listed +under _Path to C_ below now appears to be met (see the checklist). The +grade is assigned by audit, not by this file, so it is left at D pending +a maintainer re-audit rather than self-promoted. `+just crg-badge+` +publishes this letter, so the badge is understating the project until +that happens. +____ + +Graded per the hyperpolymath +https://github.com/hyperpolymath/standards/tree/main/component-readiness-grades[Component +Readiness Grades] standard. Grade assigned in the +link:docs/reports/audit/audit-2026-04-15-post.md[2026-04-15 post-audit +report] (promoted from E/X). + +=== What D means here + +[width="100%",cols="50%,50%",options="header",] +|=== +|Aspect |Status +|Builds from source |Yes — `+bun run build+` (ReScript + Bun bundle); +Rust core builds and its *91 tests* pass (82 unit + 3 exchange + 2 +golden + 2 property + 2 doc) + +|Lockfiles |Yes — `+bun.lock+`, `+Cargo.lock+` + +|CI |Estate governance/scanning *plus product CI*: `+rust-ci.yml+` (fmt, +clippy `+--all-targets --features wasm+`, tests, wasm32 build) and +`+ui-ci.yml+` (ReScript, wasm, Bun tests/bundle, Biome lint) — both on +every PR and SHA-pinned + +|Tests |Rust core (91) *and* UI (11, via `+bun run test:ui+`), including +TEA↔WASM and complete ReScript↔WASM facade contract tests. No +browser-level integration test yet + +|Docs |Truth-reset 2026-07-02; TOPOLOGY re-derived 2026-07-17 after the +λδ merges (#35, #36, #43) had gone unrecorded + +|Known debt |unwrap/expect calls in core and desktop; unsafe `+get+` in +View.res (see audit report) +|=== + +=== Path to C + +All three appear *met* as of 2026-07-17 — a re-audit should confirm and +regrade: + +* [x] Product CI running on every PR (Rust build+test, ReScript build, +lint) — `+rust-ci.yml+` + `+ui-ci.yml+`, both `+on: pull_request+` +* [x] Tests beyond the core crate: UI unit tests exercised in CI — 10 +tests via `+ui-ci.yml+` +* [x] WASM bridge built and smoke-tested in CI — `+rust-ci.yml+` builds +wasm32 + generates bindings; `+ui-ci.yml+` runs the +`+TEA update delegates to the wasm core+` contract test against the real +bundle + +=== Path to B + +* Integration tests covering the UI → WASM core → persistence path (the +UI → WASM half is covered; the IndexedDB persistence half needs a +browser-level test) +* Canvas interaction coverage (pan/zoom, note creation, drag — all now +implemented; geometry is unit-tested, direct interaction is not) +* Known-debt items from the audit resolved or explicitly waived diff --git a/READINESS.md b/READINESS.md deleted file mode 100644 index 36e59ad..0000000 --- a/READINESS.md +++ /dev/null @@ -1,44 +0,0 @@ - - - - -# Readiness - -**Current Grade:** D - -> **Re-audit due — this grade is probably stale.** Every criterion listed under -> *Path to C* below now appears to be met (see the checklist). The grade is -> assigned by audit, not by this file, so it is left at D pending a maintainer -> re-audit rather than self-promoted. `just crg-badge` publishes this letter, so -> the badge is understating the project until that happens. - -Graded per the hyperpolymath -[Component Readiness Grades](https://github.com/hyperpolymath/standards/tree/main/component-readiness-grades) -standard. Grade assigned in the -[2026-04-15 post-audit report](docs/reports/audit/audit-2026-04-15-post.md) -(promoted from E/X). - -## What D means here - -| Aspect | Status | -| --- | --- | -| Builds from source | Yes — `bun run build` (ReScript + Bun bundle); Rust core builds and its **91 tests** pass (82 unit + 3 exchange + 2 golden + 2 property + 2 doc) | -| Lockfiles | Yes — `bun.lock`, `Cargo.lock` | -| CI | Estate governance/scanning **plus product CI**: `rust-ci.yml` (fmt, clippy `--all-targets --features wasm`, tests, wasm32 build) and `ui-ci.yml` (ReScript, wasm, Bun tests/bundle, Biome lint) — both on every PR and SHA-pinned | -| Tests | Rust core (91) **and** UI (11, via `bun run test:ui`), including TEA↔WASM and complete ReScript↔WASM facade contract tests. No browser-level integration test yet | -| Docs | Truth-reset 2026-07-02; TOPOLOGY re-derived 2026-07-17 after the λδ merges (#35, #36, #43) had gone unrecorded | -| Known debt | unwrap/expect calls in core and desktop; unsafe `get` in View.res (see audit report) | - -## Path to C - -All three appear **met** as of 2026-07-17 — a re-audit should confirm and regrade: - -- [x] Product CI running on every PR (Rust build+test, ReScript build, lint) — `rust-ci.yml` + `ui-ci.yml`, both `on: pull_request` -- [x] Tests beyond the core crate: UI unit tests exercised in CI — 10 tests via `ui-ci.yml` -- [x] WASM bridge built and smoke-tested in CI — `rust-ci.yml` builds wasm32 + generates bindings; `ui-ci.yml` runs the `TEA update delegates to the wasm core` contract test against the real bundle - -## Path to B - -- Integration tests covering the UI → WASM core → persistence path (the UI → WASM half is covered; the IndexedDB persistence half needs a browser-level test) -- Canvas interaction coverage (pan/zoom, note creation, drag — all now implemented; geometry is unit-tested, direct interaction is not) -- Known-debt items from the audit resolved or explicitly waived diff --git a/SECURITY.adoc b/SECURITY.adoc new file mode 100644 index 0000000..4ca940c --- /dev/null +++ b/SECURITY.adoc @@ -0,0 +1,32 @@ +== Security Policy + +=== Reporting a Vulnerability + +If you discover a security vulnerability, please report it responsibly. + +*Contact:* j.d.a.jewell@open.ac.uk + +Please include: - Description of the vulnerability - Steps to reproduce +- Potential impact - Suggested fix (if any) + +=== Response Timeline + +* *Acknowledgment:* Within 48 hours +* *Assessment:* Within 7 days +* *Fix/Mitigation:* As soon as possible, depending on severity + +=== Supported Versions + +Only the latest version is supported with security updates. + +=== Scope + +This policy covers the nexia-list repository and its published +artifacts: - Rust crates (`+nexia-core+`, `+nexia-desktop+`) - ReScript +UI package - Tauri application binaries + +=== Disclosure Policy + +We follow coordinated disclosure. Please do not publicly disclose +vulnerabilities until a fix has been released or 90 days have passed +since the initial report. diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 46a4741..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,36 +0,0 @@ - -# Security Policy - -## Reporting a Vulnerability - -If you discover a security vulnerability, please report it responsibly. - -**Contact:** j.d.a.jewell@open.ac.uk - -Please include: -- Description of the vulnerability -- Steps to reproduce -- Potential impact -- Suggested fix (if any) - -## Response Timeline - -- **Acknowledgment:** Within 48 hours -- **Assessment:** Within 7 days -- **Fix/Mitigation:** As soon as possible, depending on severity - -## Supported Versions - -Only the latest version is supported with security updates. - -## Scope - -This policy covers the nexia-list repository and its published artifacts: -- Rust crates (`nexia-core`, `nexia-desktop`) -- ReScript UI package -- Tauri application binaries - -## Disclosure Policy - -We follow coordinated disclosure. Please do not publicly disclose vulnerabilities -until a fix has been released or 90 days have passed since the initial report. diff --git a/TEST-NEEDS.adoc b/TEST-NEEDS.adoc new file mode 100644 index 0000000..659b6fc --- /dev/null +++ b/TEST-NEEDS.adoc @@ -0,0 +1,71 @@ +== TEST-NEEDS.md — nexia-list + +=== CRG Grade: D (see READINESS.md; last audited 2026-04-15, test matrix updated 2026-08-01) + +=== Current Test State + +[width="100%",cols="34%,22%,22%,22%",options="header",] +|=== +|Category |Where |Count |Notes +|Rust unit tests |`+core/src/*.rs+` (`+#[cfg(test)]+`) |85 |domain +model, storage, exchange, λδ kernel/host, including generated +determinism, macro non-capture, and formula non-mutation properties + +|Rust exchange properties |`+core/tests/exchange.rs+` |3 |total +import/parser and Markdown topology round-trip + +|Golden contract (Rust) |`+core/tests/golden.rs+` |2 |on-disk JSON +format, shared fixture `+tests/fixtures/notebook.golden.json+` + +|λδ conformance (Rust/WASM) |shared JSON fixture + Rust/JS consumers |1 +Rust + 1 JS |same pure-language vectors and printed results on both +sides of the WASM boundary + +|Graph/serde properties |`+core/tests/invariants.rs+` (proptest) |3 +|arbitrary-byte parser totality; invariants after every generated +transition, including removed IDs; complete semantic serde round-trip +after index rebuild + +|Golden contract (JS/wasm) |`+ui/tests/contract.test.js+` |2 |same +fixture decoded through the wasm bindings; camelCase view shape; +snake_case disk format + +|TEA update tests |`+ui/tests/UpdateTests.res+` + `+update.test.js+` |1 +suite (~25 assertions) |CRUD/link/search/zoom/delete-guard against the +real wasm core +|=== + +=== What’s Covered + +* [x] Core unit tests +* [x] Property-based tests (backlink invariant, serde roundtrip) +* [x] Cross-language contract tests (Rust ⇄ wasm/JS golden fixture) +* [x] UI update-function tests against the wasm core +* [x] CI/CD test automation (`+rust-ci.yml+`, `+ui-ci.yml+`) +* [x] Executable proof ledger with explicit claim boundaries +(`+docs/verification/proof-baseline-0.md+`) + +=== Still Missing (for CRG C/B) + +* [ ] Browser E2E in CI (Astral smoke flows: +create/edit/link/reload-restore, keyboard-only session) — verified +manually, not yet a CI job +* [ ] Real fuzz target (`+fuzz_load_notebook+`: arbitrary bytes → +`+serde_json::from_str::+` must never panic) replacing +`+tests/fuzz/placeholder.txt+` +* [ ] 10k-note performance benchmark (search <100 ms target) +* [ ] Accessibility gate (axe-core in E2E) +* [x] Shared Rust/WASM λδ success-vector corpus +* [ ] Extend the shared λδ corpus to errors, budgets, and formula +contexts +* [ ] Canonical serialization, if byte-identical output becomes a +requirement + +=== Run Tests + +[source,bash] +---- +bun run test # cargo test (core) + bun test (UI/contract) +bun run test:rust # Rust only +bun run test:ui # UI/contract only (needs build:res + build:wasm first) +---- diff --git a/TEST-NEEDS.md b/TEST-NEEDS.md deleted file mode 100644 index afaab6f..0000000 --- a/TEST-NEEDS.md +++ /dev/null @@ -1,47 +0,0 @@ - -# TEST-NEEDS.md — nexia-list - -## CRG Grade: D (see READINESS.md; last audited 2026-04-15, test matrix updated 2026-08-01) - -## Current Test State - -| Category | Where | Count | Notes | -|----------|-------|-------|-------| -| Rust unit tests | `core/src/*.rs` (`#[cfg(test)]`) | 85 | domain model, storage, exchange, λδ kernel/host, including generated determinism, macro non-capture, and formula non-mutation properties | -| Rust exchange properties | `core/tests/exchange.rs` | 3 | total import/parser and Markdown topology round-trip | -| Golden contract (Rust) | `core/tests/golden.rs` | 2 | on-disk JSON format, shared fixture `tests/fixtures/notebook.golden.json` | -| λδ conformance (Rust/WASM) | shared JSON fixture + Rust/JS consumers | 1 Rust + 1 JS | same pure-language vectors and printed results on both sides of the WASM boundary | -| Graph/serde properties | `core/tests/invariants.rs` (proptest) | 3 | arbitrary-byte parser totality; invariants after every generated transition, including removed IDs; complete semantic serde round-trip after index rebuild | -| Golden contract (JS/wasm) | `ui/tests/contract.test.js` | 2 | same fixture decoded through the wasm bindings; camelCase view shape; snake_case disk format | -| TEA update tests | `ui/tests/UpdateTests.res` + `update.test.js` | 1 suite (~25 assertions) | CRUD/link/search/zoom/delete-guard against the real wasm core | - -## What's Covered - -- [x] Core unit tests -- [x] Property-based tests (backlink invariant, serde roundtrip) -- [x] Cross-language contract tests (Rust ⇄ wasm/JS golden fixture) -- [x] UI update-function tests against the wasm core -- [x] CI/CD test automation (`rust-ci.yml`, `ui-ci.yml`) -- [x] Executable proof ledger with explicit claim boundaries - (`docs/verification/proof-baseline-0.md`) - -## Still Missing (for CRG C/B) - -- [ ] Browser E2E in CI (Astral smoke flows: create/edit/link/reload-restore, - keyboard-only session) — verified manually, not yet a CI job -- [ ] Real fuzz target (`fuzz_load_notebook`: arbitrary bytes → - `serde_json::from_str::` must never panic) replacing - `tests/fuzz/placeholder.txt` -- [ ] 10k-note performance benchmark (search <100 ms target) -- [ ] Accessibility gate (axe-core in E2E) -- [x] Shared Rust/WASM λδ success-vector corpus -- [ ] Extend the shared λδ corpus to errors, budgets, and formula contexts -- [ ] Canonical serialization, if byte-identical output becomes a requirement - -## Run Tests - -```bash -bun run test # cargo test (core) + bun test (UI/contract) -bun run test:rust # Rust only -bun run test:ui # UI/contract only (needs build:res + build:wasm first) -``` diff --git a/TOPOLOGY.md b/TOPOLOGY.adoc similarity index 85% rename from TOPOLOGY.md rename to TOPOLOGY.adoc index f2c2a7c..6ad73fa 100644 --- a/TOPOLOGY.md +++ b/TOPOLOGY.adoc @@ -1,19 +1,16 @@ - - - +== Nexia-List — Project Topology -# Nexia-List — Project Topology +=== System Architecture -## System Architecture +Nexia-List is *web-first*: the browser is the primary target, running +the real Rust core compiled to WebAssembly. The core carries the *λδ +(LambdaDelta)* substrate — a homoiconic Clojure-flavoured Lisp built +in-tree, not vendored — which is the single largest subsystem in the +core. The desktop shell is optional and depends on an *external* sibling +checkout of `+hyperpolymath/gossamer+`; it is intentionally not built in +this repo’s CI. -Nexia-List is **web-first**: the browser is the primary target, running the real -Rust core compiled to WebAssembly. The core carries the **λδ (LambdaDelta)** -substrate — a homoiconic Clojure-flavoured Lisp built in-tree, not vendored — -which is the single largest subsystem in the core. The desktop shell is optional -and depends on an **external** sibling checkout of `hyperpolymath/gossamer`; it -is intentionally not built in this repo's CI. - -``` +.... ┌─────────────────────────────────────────┐ │ USER INTERFACE │ │ (Note list / Editor / Canvas) │ @@ -69,16 +66,16 @@ is intentionally not built in this repo's CI. │ rust-ci.yml ui-ci.yml │ │ .machine_readable/ 0-AI-MANIFEST.a2ml │ └─────────────────────────────────────────┘ -``` +.... -Future (not yet implemented, kept out of the diagram deliberately): petgraph -graph engine, tantivy full-text search, Nickel schemas, mobile targets, and the -FL×DT intelligence modules (`index.rs`, `edge.rs`, `reason.rs`, `layout.rs`, -`trigger.rs`) described in [docs/design/](docs/design/). +Future (not yet implemented, kept out of the diagram deliberately): +petgraph graph engine, tantivy full-text search, Nickel schemas, mobile +targets, and the FL×DT intelligence modules (`+index.rs+`, `+edge.rs+`, +`+reason.rs+`, `+layout.rs+`, `+trigger.rs+`) described in docs/design/. -## Completion Dashboard +=== Completion Dashboard -``` +.... COMPONENT STATUS NOTES ───────────────────────────────── ────────────────── ───────────────────────────────── PRODUCT @@ -143,34 +140,36 @@ OVERALL: ███████░░░ ~65% Core has MOVED off the WASM bridge and onto the UI surface: continuing λδ doors (L2→L3) -``` +.... -## Key Dependencies +=== Key Dependencies -``` +.... Rust Core (incl. λδ) ──► WASM bundle ──► ReScript UI ──► Web bundle (web/dist/) │ │ ▼ ▼ JSON storage ──► IndexedDB / file download (optional) Gossamer shell -``` +.... -The wasm-bindgen **CLI must match the wasm-bindgen crate version in -`Cargo.lock` exactly** (`just doctor` checks this); a mismatch fails the browser -build with a confusing schema error. +The wasm-bindgen *CLI must match the wasm-bindgen crate version in +`+Cargo.lock+` exactly* (`+just doctor+` checks this); a mismatch fails +the browser build with a confusing schema error. -## Update Protocol +=== Update Protocol This file is maintained by both humans and AI agents. When updating: -1. **After completing a component**: Change its bar and percentage -2. **After adding a component**: Add a new row in the appropriate section -3. **After architectural changes**: Update the ASCII diagram -4. **Date**: Update the `Last updated` comment at the top of this file +[arabic] +. *After completing a component*: Change its bar and percentage +. *After adding a component*: Add a new row in the appropriate section +. *After architectural changes*: Update the ASCII diagram +. *Date*: Update the `+Last updated+` comment at the top of this file -Progress bars use: `█` (filled) and `░` (empty), 10 characters wide. -Percentages: 0%, 10%, 20%, ... 100% (in 10% increments). +Progress bars use: `+█+` (filled) and `+░+` (empty), 10 characters wide. +Percentages: 0%, 10%, 20%, … 100% (in 10% increments). -**Percentages are judgement, LOC and test counts are not.** When you revise a -row, re-derive the countable facts rather than copying them forward — this file -understated the core for two weeks because the λδ merges (#35, #36, #43) landed -without a dashboard update, and downstream docs inherited the stale figures. +*Percentages are judgement, LOC and test counts are not.* When you +revise a row, re-derive the countable facts rather than copying them +forward — this file understated the core for two weeks because the λδ +merges (#35, #36, #43) landed without a dashboard update, and downstream +docs inherited the stale figures. diff --git a/desktop/README.adoc b/desktop/README.adoc new file mode 100644 index 0000000..3045133 --- /dev/null +++ b/desktop/README.adoc @@ -0,0 +1,41 @@ +== Nexia-List Desktop Shell (optional, external dependency) + +This crate is the Gossamer webview shell for Nexia-List. *It does not +build from this repository alone* and that is intentional: + +* `+Cargo.toml+` depends on +`+gossamer-rs = { path = "../../gossamer/bindings/rust" }+`, i.e. a +checkout of +https://github.com/hyperpolymath/gossamer[hyperpolymath/gossamer] as a +*sibling directory* of this repo. +* The root workspace `+Cargo.toml+` lists `+desktop+` under `+exclude+`, +so `+cargo build+`/`+cargo test+` at the repo root never touch it, and +CI (`+rust-ci.yml+`, `+ui-ci.yml+`) deliberately skips it. + +=== Building (with the sibling checkout) + +.... +parent/ +├── nexia-list/ (this repo) +└── gossamer/ (git clone https://github.com/hyperpolymath/gossamer) +.... + +Then: `+cd desktop && cargo build+`. + +=== How the desktop shell relates to the web app + +The web app is the primary target: the ReScript UI calls the Rust core +compiled to WebAssembly (see `+docs/adr/0001-wasm-core-web-first.md+`). +The UI reaches the engine only through the store seam +(`+ui/src/store/WasmStore.res+`), whose operations deliberately mirror +the command handlers registered in `+src/main.rs+` here. + +To resume desktop work without UI changes, implement a `+GossamerStore+` +with the same signatures over `+gossamer-bridge.js+` (`+invoke()+`), and +select it at startup when the bridge is present. Known gaps to backfill +in `+src/main.rs+` at that point: + +* commands not yet registered: `+unlink_notes+`, `+move_note+`, +`+resize_note+`, `+set_attribute+` +* the `+.lock().unwrap()+` calls on the shared state should use +`+unwrap_or_else(std::sync::PoisonError::into_inner)+` diff --git a/desktop/README.md b/desktop/README.md deleted file mode 100644 index 3558f4f..0000000 --- a/desktop/README.md +++ /dev/null @@ -1,40 +0,0 @@ - -# Nexia-List Desktop Shell (optional, external dependency) - -This crate is the Gossamer webview shell for Nexia-List. **It does not build from -this repository alone** and that is intentional: - -- `Cargo.toml` depends on `gossamer-rs = { path = "../../gossamer/bindings/rust" }`, - i.e. a checkout of [hyperpolymath/gossamer](https://github.com/hyperpolymath/gossamer) - as a **sibling directory** of this repo. -- The root workspace `Cargo.toml` lists `desktop` under `exclude`, so - `cargo build`/`cargo test` at the repo root never touch it, and CI - (`rust-ci.yml`, `ui-ci.yml`) deliberately skips it. - -## Building (with the sibling checkout) - -``` -parent/ -├── nexia-list/ (this repo) -└── gossamer/ (git clone https://github.com/hyperpolymath/gossamer) -``` - -Then: `cd desktop && cargo build`. - -## How the desktop shell relates to the web app - -The web app is the primary target: the ReScript UI calls the Rust core -compiled to WebAssembly (see `docs/adr/0001-wasm-core-web-first.md`). The -UI reaches the engine only through the store seam -(`ui/src/store/WasmStore.res`), whose operations deliberately mirror the -command handlers registered in `src/main.rs` here. - -To resume desktop work without UI changes, implement a `GossamerStore` with -the same signatures over `gossamer-bridge.js` (`invoke()`), and select it at -startup when the bridge is present. Known gaps to backfill in `src/main.rs` -at that point: - -- commands not yet registered: `unlink_notes`, `move_note`, `resize_note`, - `set_attribute` -- the `.lock().unwrap()` calls on the shared state should use - `unwrap_or_else(std::sync::PoisonError::into_inner)` diff --git a/docs/adr/0001-wasm-core-web-first.adoc b/docs/adr/0001-wasm-core-web-first.adoc new file mode 100644 index 0000000..b6408ed --- /dev/null +++ b/docs/adr/0001-wasm-core-web-first.adoc @@ -0,0 +1,45 @@ +== ADR-0001: Compile the Rust core to WASM; the browser is the primary target + +* *Status:* Accepted +* *Date:* 2026-07-02 + +=== Context + +* The desktop shell depends on `+hyperpolymath/gossamer+` as a path +dependency (`+../../gossamer/bindings/rust+`) — an external sibling +checkout that is not available in this repo or its CI. The desktop layer +is therefore not buildable here. +* The three layers (Rust core, ReScript UI, shell) were disconnected: +the UI did not actually call the core, and nothing exercised the seam +between them. +* Maintaining separate "`desktop core`" and "`web core`" paths invites +type drift between engine and interface. + +=== Decision + +* Compile `+nexia-core+` to WebAssembly with wasm-bindgen +(`+bun run build:wasm+`). +* The *browser is the primary target*: the bundled app in `+web/dist/+` +runs the real Rust core client-side, with persistence via IndexedDB and +JSON file download/upload. +* The UI delegates all note/notebook semantics to the core through a +store seam; the ReScript layer holds view state only and never forks the +data model. +* Desktop becomes an *optional thin shell* added later: Gossamer wraps +the same web bundle. It stays out of this repo’s CI and requires the +external sibling checkout. + +=== Consequences + +* One engine, one data model — no type drift between platforms; every +target ships the same tested core. +* The product is buildable and testable entirely within this repo (Bun + +Rust stable + `+wasm32-unknown-unknown+` target); CI can cover the real +product. +* Browser persistence limits apply (IndexedDB quotas; explicit file +download/upload instead of transparent filesystem access) until a shell +provides native file I/O. +* WASM boundary costs: data crossing the bridge is serialized, so the +API surface must stay coarse-grained. +* The desktop experience is deferred; anything desktop-only (native +menus, file watching) waits for the optional shell. diff --git a/docs/adr/0001-wasm-core-web-first.md b/docs/adr/0001-wasm-core-web-first.md deleted file mode 100644 index 6709e22..0000000 --- a/docs/adr/0001-wasm-core-web-first.md +++ /dev/null @@ -1,45 +0,0 @@ - -# ADR-0001: Compile the Rust core to WASM; the browser is the primary target - -- **Status:** Accepted -- **Date:** 2026-07-02 - -## Context - -- The desktop shell depends on `hyperpolymath/gossamer` as a path dependency - (`../../gossamer/bindings/rust`) — an external sibling checkout that is not - available in this repo or its CI. The desktop layer is therefore not - buildable here. -- The three layers (Rust core, ReScript UI, shell) were disconnected: the UI - did not actually call the core, and nothing exercised the seam between them. -- Maintaining separate "desktop core" and "web core" paths invites type drift - between engine and interface. - -## Decision - -- Compile `nexia-core` to WebAssembly with wasm-bindgen (`bun run - build:wasm`). -- The **browser is the primary target**: the bundled app in `web/dist/` runs - the real Rust core client-side, with persistence via IndexedDB and JSON - file download/upload. -- The UI delegates all note/notebook semantics to the core through a store - seam; the ReScript layer holds view state only and never forks the data - model. -- Desktop becomes an **optional thin shell** added later: Gossamer wraps the - same web bundle. It stays out of this repo's CI and requires the external - sibling checkout. - -## Consequences - -- One engine, one data model — no type drift between platforms; every target - ships the same tested core. -- The product is buildable and testable entirely within this repo (Bun + - Rust stable + `wasm32-unknown-unknown` target); CI can cover the real - product. -- Browser persistence limits apply (IndexedDB quotas; explicit file - download/upload instead of transparent filesystem access) until a shell - provides native file I/O. -- WASM boundary costs: data crossing the bridge is serialized, so the API - surface must stay coarse-grained. -- The desktop experience is deferred; anything desktop-only (native menus, - file watching) waits for the optional shell. diff --git a/docs/adr/0002-deno-only-interpretation.adoc b/docs/adr/0002-deno-only-interpretation.adoc new file mode 100644 index 0000000..143c9de --- /dev/null +++ b/docs/adr/0002-deno-only-interpretation.adoc @@ -0,0 +1,45 @@ +== ADR-0002: Interpretation of the "`Deno only, no npm/bun/yarn/pnpm`" invariant + +____ +*Status: Superseded on 2026-08-01 by +link:0004-bun-only-toolchain.md[ADR-0004].* This file remains as the +historical rationale for the former toolchain. +____ + +* *Status:* Accepted +* *Date:* 2026-07-02 + +=== Context + +`+.machine_readable/MUST.contractile+` requires: "`no npm/bun/yarn/pnpm +dependencies — Deno only`". The UI needs packages that are published to +the npm registry (`+rescript+`, `+@rescript/core+`, `+@rescript/react+`, +`+react+`, `+esbuild+`), and Deno 2 natively resolves such packages via +`+npm:+` specifiers. A literal reading ("`nothing from the npm +registry`") would make the ReScript toolchain unusable; the invariant +needed a precise interpretation. + +=== Decision + +* *Deno is the only JS package manager and runtime.* Dependencies are +declared in `+deno.json+` (import map with `+npm:+` specifiers), +installed with `+deno install+` (`+deno task setup+`), and locked in +`+deno.lock+`. +* *The npm registry as a package _source_ is allowed.* `+npm:+` +specifiers resolved by Deno do not violate the invariant. +* *The npm CLI (and bun/yarn/pnpm) is not used* — no `+package.json+` +workflows, no `+package-lock.json+`, no `+npx+`. + +=== Consequences + +* The previous `+setup:ui+` task, which shelled out to the npm CLI, +violated this interpretation and was removed; `+deno task setup+` +replaces it. +* Reproducibility comes from `+deno.lock+`; there is exactly one JS +lockfile and one toolchain to install in CI. +* Registry-published tools (ReScript compiler, esbuild) run via +`+deno run -A npm:+` rather than through node_modules `+.bin+` +scripts. +* Future tooling must follow the same rule: if a JS dependency is +needed, add it to `+deno.json+` — never introduce `+package.json+` or +another package manager. diff --git a/docs/adr/0002-deno-only-interpretation.md b/docs/adr/0002-deno-only-interpretation.md deleted file mode 100644 index 8635a19..0000000 --- a/docs/adr/0002-deno-only-interpretation.md +++ /dev/null @@ -1,39 +0,0 @@ - -# ADR-0002: Interpretation of the "Deno only, no npm/bun/yarn/pnpm" invariant - -> **Status: Superseded on 2026-08-01 by [ADR-0004](0004-bun-only-toolchain.md).** -> This file remains as the historical rationale for the former toolchain. - -- **Status:** Accepted -- **Date:** 2026-07-02 - -## Context - -`.machine_readable/MUST.contractile` requires: "no npm/bun/yarn/pnpm -dependencies — Deno only". The UI needs packages that are published to the -npm registry (`rescript`, `@rescript/core`, `@rescript/react`, `react`, -`esbuild`), and Deno 2 natively resolves such packages via `npm:` specifiers. -A literal reading ("nothing from the npm registry") would make the ReScript -toolchain unusable; the invariant needed a precise interpretation. - -## Decision - -- **Deno is the only JS package manager and runtime.** Dependencies are - declared in `deno.json` (import map with `npm:` specifiers), installed with - `deno install` (`deno task setup`), and locked in `deno.lock`. -- **The npm registry as a package *source* is allowed.** `npm:` specifiers - resolved by Deno do not violate the invariant. -- **The npm CLI (and bun/yarn/pnpm) is not used** — no `package.json` - workflows, no `package-lock.json`, no `npx`. - -## Consequences - -- The previous `setup:ui` task, which shelled out to the npm CLI, violated - this interpretation and was removed; `deno task setup` replaces it. -- Reproducibility comes from `deno.lock`; there is exactly one JS lockfile - and one toolchain to install in CI. -- Registry-published tools (ReScript compiler, esbuild) run via - `deno run -A npm:` rather than through node_modules `.bin` scripts. -- Future tooling must follow the same rule: if a JS dependency is needed, add - it to `deno.json` — never introduce `package.json` or another package - manager. diff --git a/docs/adr/0003-lambdadelta-lisp-substrate.adoc b/docs/adr/0003-lambdadelta-lisp-substrate.adoc new file mode 100644 index 0000000..c231a93 --- /dev/null +++ b/docs/adr/0003-lambdadelta-lisp-substrate.adoc @@ -0,0 +1,294 @@ +== ADR-0003: LambdaDelta — a Lisp-power substrate for Nexia-List + +* *Status:* Accepted (fundamentals confirmed — see +link:../design/lambdadelta-spec.md[λδ spec v0.1] §9) +* *Date:* 2026-07-03 + +____ +*Update (2026-07-03):* the language fundamentals are confirmed — +Clojure- flavoured syntax, note-as-immutable-map, *hygienic macros*, +*multimethods/ protocols* for extensibility, sets/`+match+`/tagged +literals, and the *kernel/host seam* (a notebook-agnostic language +kernel with host bindings registered via a capability-carrying +interface) that makes an SDK, embedding, and the plugin ecosystem cheap. +The plugin system + minter/provisioner/ configurator/harness dev wizard +are tracked in +https://github.com/hyperpolymath/nexia-list/issues/33[issue #33]. Detail +in the spec. +____ + +____ +_A note is a letter we send to our future self._ — Mark Bernstein, _The +Tinderbox Way_ +____ + +This line is the North Star. Everything below exists to *enlarge* that +correspondence — never to fence it in. Read the framing in the Context +section before the architecture: the power is the servant, the letter is +the point. + +=== Context + +Nexia-List is a Tinderbox-like spatial hypertext tool: notes on a +canvas, first-class bidirectional links, attributes, and agents +(persistent queries). Tinderbox’s real ceiling is its _expression +language_ — agent queries, rules, OnAdd actions, and export templates +are written in a small, fixed, non-extensible mini-language. Users hit +its edges and cannot go further. + +Nexia-List’s notebook is already tree/graph-shaped data (JSON), and the +Rust core is already the single source of truth compiled to WASM (see +ADR-0001). That makes it a natural host for a *homoiconic Lisp* — one +where the notebook _is_ data the language can read, and the language’s +own code is data too. Code and notebook become the same fabric. + +The user’s intent, stated plainly: + +* Give power users *the full power of Lisp* — not a toy DSL wearing +Lisp’s name. Homoiconicity, closures, recursion, higher-order functions, +*macros*, a REPL. +* But *not* make Nexia-List "`a Lisp app.`" It remains a Tinderbox-like +product, usable for a lifetime *without ever seeing a parenthesis*. +* Treat this power *principally as liberating, enabling, and augmenting* +— the animating purpose (the North Star) is something power _serves and +enlarges_, not a constraint the product must shrink to fit. + +=== Decision + +Introduce *LambdaDelta (λδ)* — a small, homoiconic Lisp implemented *in +the Rust core and compiled to WASM*, exposing the notebook as +first-class data. λ (functions) + δ (change/transformation): the +computational soul that lets a notebook be programmed _by its owner_, +safely, everywhere the tool runs. + +Two commitments make it safe and humane: + +[arabic] +. *Progressive power (the UX spine).* Parentheses never appear until a +user opens a door marked "`power.`" Five optional levels, each layered +over the one below, none blocking it: +* *L0 — Tinderbox-basic (no code, ever):* create/drag/link notes; the +existing simple search + agent DSL. The default, forever. +* *L1 — Formulas (spreadsheet-gentle):* a computed attribute or agent +predicate can _optionally_ be a λδ expression in a friendly "`fx`" field +with autocomplete. The simple DSL still works and quietly compiles to +λδ. +* *L2 — Actions & macros:* note/agent/adornment actions as λδ; +user-defined functions and `+defmacro+`, stored _in the notebook_. +* *L3 — REPL/console:* evaluate against the live graph; define and save +functions; inspect. +* *L4 — Beyond:* user-defined commands, views, and exporters; +*computational notes* (a note whose content evaluates and renders +results inline); shareable λδ packages. +. *The letter comes first — as an enabler, not a rule.* Durable, +human-readable notes are what _make_ the augmentation possible: because +the letter is legible and lasting, computation has something real to +amplify. Durability is the launchpad, not the leash. The measure of any +feature is a generative question — _does this help the letter reach a +richer future self?_ — not a veto. + +==== Architecture (one engine, everywhere) + +* *Reader → value model → evaluator* in `+core/src/lambdadelta/+`. +Lexical scope, closures, tail calls; special forms `+quote+`, `+if+`, +`+let+`, `+fn+`/`+lambda+`, `+define+`, `+quasiquote+`/`+unquote+`, +`+defmacro+`. +* *Notebook as first-class values.* Builtins expose the whole core +surface as a Lisp library: `+(notes)+`, `+(note id)+`, `+(title n)+`, +`+(content n)+`, `+(links n)+`, `+(backlinks n)+`, `+(attr n k)+`, +`+(set-attr! n k v)+`, `+(create-note! …)+`, `+(link! a b)+`, +`+(search q)+`, `+(run-agent id)+`. Pure readers vs. `+!+`-suffixed +mutators. +* *Sandbox.* Deterministic; no I/O or network; a step/time/recursion +*budget* so a user macro can never hang the tab (heavy jobs move to a +Web Worker later). +* *Persistence = homoiconicity in action.* Functions, macros, formulas, +and agent-programs are stored _as data in the notebook_, so they +save/load/sync with everything else. A notebook carries its own +behaviour. +* *Build, don’t embed.* A focused interpreter (rather than a general +Rust Lisp crate) because the whole point is deep notebook-as-data +interop and a strict browser sandbox — a general embed fights both. No +new npm deps (Bun-only MUST); implemented in Rust/WASM (no new +TS/Python/Go); MPL-2.0. + +==== Subsumes, does not replace + +The merged features become friendly surfaces over the one engine: the +current agents DSL is L0 sugar that compiles to λδ; agent _actions_ +become arbitrary λδ programs; computed attributes, prototype +inheritance, and (future) adornment rules are λδ expressions. + +=== Tinderbox capability coverage + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Tinderbox capability |Under Nexia-List + λδ |Status +|Attributes on notes |Present (`+attributes+` map); λδ reads/writes +|Matched (typing to add) + +|Agents (persistent queries) |Present; queries become λδ predicates +|Replaced & exceeded + +|Agent actions |Arbitrary λδ programs (not just Collect/SetAttribute) +|Exceeded + +|Rules & Edicts |λδ re-evaluated on change / on schedule |Matched + +|OnAdd / smart containers |λδ action on entry to container/adornment +|Matched + +|Adornments (map regions) |Planned; with λδ actions = programmable +regions |Exceeded (when landed) + +|Action-code language |λδ _is_ this — a real language with macros +|Vastly exceeded + +|Stamps |A λδ function applied to the selection |Matched + +|Prototypes / inheritance |`+note.prototype+` exists; resolution + +computed inherited attrs |Exceeded + +|Export templates |λδ functions `+notebook → string+`; quasiquote +templating |Exceeded + +|Links |Bidirectional present; *typed* links |Partial (typing to add) + +|Map / spatial view |Present (drag/pan/zoom) |Matched + +|Outline / Timeline / Attribute-browser / Treemap |Not yet (have list + +graph) |Still lacking + +|Typed attributes (number/date/bool/color/set…) |Values are untyped JSON +today |Still lacking + +|Attribute-driven visuals (colour/size/badge) |Not yet; λδ can compute +style from a formula |Partial → exceeded + +|Rich-text note content |Plain text today |Still lacking + +|Dates / calendar / events |`+chrono+` in core; no calendar/timeline UI +|Partial + +|AppleScript automation (macOS-only) |Replaced by λδ — cross-platform, +in-app, sandboxed |Replaced & exceeded + +|Import / Export |OPML + Markdown-vault present; λδ-driven custom +formats |Matched → exceeded +|=== + +=== What will still be lacking (honest gap list) + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Gap |Why it matters |Effort +|Typed attribute system (schemas, per-type editors, validation) |Date +math, colours, sets; richer λδ values |M + +|Extra views: outline, timeline, attribute-browser |Parity for +non-spatial thinkers |M–L + +|Typed links / link types |Semantic graphs, filtered link queries |S–M + +|Rich-text / markdown rendering in notes |A real writing tool, not plain +text |M + +|Attribute/formula-driven visual styling |"`See relationships at a +glance`" payoff |M + +|Computed-attribute dependency tracking + memoization |Efficient +recompute at 10k notes |M + +|Off-main-thread (Web Worker) evaluation |Keep the UI fluid under heavy +programs |M + +|λδ std-library + package/sharing system |Turns power into an ecosystem +|L (ongoing) + +|λδ debugging / tracing / error UX |Power users must see what went wrong +|M + +|Desktop shell |Blocked on the external `+gossamer+` sibling |external + +|Sync / collaboration / mobile |Future / non-goals |L +|=== + +=== What λδ liberates that Tinderbox cannot match + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Superpower |What it enables |Why Tinderbox can’t match it +|Homoiconic notebook-as-data |Metaprogram your knowledge base; generate +notes/agents/structure |Its action code can’t treat the document as +first-class, macro-manipulable data + +|A real language (recursion, HOF, closures, macros) |User-defined DSLs & +workflows (GTD, Zettelkasten, research pipelines) as installable +packages |Its expression language is bounded and not extensible + +|Computational / literate notes |A note that evaluates and renders +results inline |It has export templates, not live in-canvas computation + +|Agents as programs, not just queries |Continuous graph maintenance: +indices, invariants, restructuring, generation |Its agents are query + +limited action + +|Programs travel with the notebook |Shareable "`smart notebooks`" that +carry their own behaviour; reproducible |Documents carry actions, not a +reusable code library + +|One portable, sandboxed automation language |Same automation on +web/desktop/mobile — deterministic and safe |AppleScript is macOS-only +and out-of-process + +|A moldable tool (Emacs-like) that stays simple by default |Users extend +commands, views, exporters — without touching L0 |Not user-extensible at +that depth + +|Verifiable substrate |Unit-test notebook programs; property-test +invariants |No comparable testable automation model + +|Open · local-first · cross-platform · free |No lock-in; your data and +programs are yours |macOS-only, closed, paid +|=== + +=== Phased rollout (each phase shippable; L0 UX untouched) + +* *L0 — Substrate:* reader, value model, evaluator, core special forms, +notebook builtins, and the safety budget in `+core/src/lambdadelta/+`; +wasm-exposed `+eval+`. Headless and fully tested (reader round-trip, +evaluator determinism, sandbox limits, property tests). No UX change — +the ideal first PR because it carries no UI risk. +* *L1 — Formulas:* "`fx`" fields for computed attributes and agent +predicates; the simple DSL compiles to λδ. +* *L2 — Actions & macros:* note/agent/adornment actions; `+defmacro+`; a +small std-library; user functions stored in the notebook. +* *L3 — REPL/console.* +* *L4 — Computational notes + packages.* + +=== Consequences + +* *Positive:* every existing feature gains a principled, unbounded +expression layer; the notebook becomes programmable by its owner; +automation is portable, sandboxed, and testable; the tool becomes +moldable while staying Tinderbox-simple by default. +* *Costs / risks (with mitigations):* +** _Browser hang from user code_ → step/time/recursion budget; +cooperative cancellation; Web Worker for heavy jobs later. +** _Scaring basic users / UX creep_ → progressive disclosure; +parentheses only appear behind an fx/console door; the simple DSL stays +the default surface; excellent error messages. +** _Notes↔values interop drift_ → one canonical bridge in core, guarded +by contract tests (as with the existing golden-fixture pattern). +** _Scope explosion_ → L0 is self-contained and verifiable headless; +ship incrementally. +** _Stored code longevity_ → definitions are data in the notebook; +eval-safety and versioning on load. + +=== Relation to other decisions + +* Builds on *ADR-0001* (Rust core → WASM, single source of truth). +* Honours *ADR-0004* (Bun-only): the interpreter is homegrown Rust, no +new JS package-manager surface. +* Respects the `+INTENT.contractile+` North Star and architectural +invariants. diff --git a/docs/adr/0003-lambdadelta-lisp-substrate.md b/docs/adr/0003-lambdadelta-lisp-substrate.md deleted file mode 100644 index 7e42fd9..0000000 --- a/docs/adr/0003-lambdadelta-lisp-substrate.md +++ /dev/null @@ -1,201 +0,0 @@ - -# ADR-0003: LambdaDelta — a Lisp-power substrate for Nexia-List - -- **Status:** Accepted (fundamentals confirmed — see - [λδ spec v0.1](../design/lambdadelta-spec.md) §9) -- **Date:** 2026-07-03 - -> **Update (2026-07-03):** the language fundamentals are confirmed — Clojure- -> flavoured syntax, note-as-immutable-map, **hygienic macros**, **multimethods/ -> protocols** for extensibility, sets/`match`/tagged literals, and the -> **kernel/host seam** (a notebook-agnostic language kernel with host bindings -> registered via a capability-carrying interface) that makes an SDK, embedding, -> and the plugin ecosystem cheap. The plugin system + minter/provisioner/ -> configurator/harness dev wizard are tracked in -> [issue #33](https://github.com/hyperpolymath/nexia-list/issues/33). Detail in -> the spec. - -> *A note is a letter we send to our future self.* -> — Mark Bernstein, *The Tinderbox Way* - -This line is the North Star. Everything below exists to **enlarge** that -correspondence — never to fence it in. Read the framing in the Context section -before the architecture: the power is the servant, the letter is the point. - -## Context - -Nexia-List is a Tinderbox-like spatial hypertext tool: notes on a canvas, -first-class bidirectional links, attributes, and agents (persistent queries). -Tinderbox's real ceiling is its *expression language* — agent queries, rules, -OnAdd actions, and export templates are written in a small, fixed, -non-extensible mini-language. Users hit its edges and cannot go further. - -Nexia-List's notebook is already tree/graph-shaped data (JSON), and the Rust -core is already the single source of truth compiled to WASM (see ADR-0001). -That makes it a natural host for a **homoiconic Lisp** — one where the notebook -*is* data the language can read, and the language's own code is data too. Code -and notebook become the same fabric. - -The user's intent, stated plainly: - -- Give power users **the full power of Lisp** — not a toy DSL wearing Lisp's - name. Homoiconicity, closures, recursion, higher-order functions, **macros**, - a REPL. -- But **not** make Nexia-List "a Lisp app." It remains a Tinderbox-like product, - usable for a lifetime **without ever seeing a parenthesis**. -- Treat this power **principally as liberating, enabling, and augmenting** — - the animating purpose (the North Star) is something power *serves and - enlarges*, not a constraint the product must shrink to fit. - -## Decision - -Introduce **LambdaDelta (λδ)** — a small, homoiconic Lisp implemented **in the -Rust core and compiled to WASM**, exposing the notebook as first-class data. -λ (functions) + δ (change/transformation): the computational soul that lets a -notebook be programmed *by its owner*, safely, everywhere the tool runs. - -Two commitments make it safe and humane: - -1. **Progressive power (the UX spine).** Parentheses never appear until a user - opens a door marked "power." Five optional levels, each layered over the one - below, none blocking it: - - **L0 — Tinderbox-basic (no code, ever):** create/drag/link notes; the - existing simple search + agent DSL. The default, forever. - - **L1 — Formulas (spreadsheet-gentle):** a computed attribute or agent - predicate can *optionally* be a λδ expression in a friendly "fx" field with - autocomplete. The simple DSL still works and quietly compiles to λδ. - - **L2 — Actions & macros:** note/agent/adornment actions as λδ; user-defined - functions and `defmacro`, stored *in the notebook*. - - **L3 — REPL/console:** evaluate against the live graph; define and save - functions; inspect. - - **L4 — Beyond:** user-defined commands, views, and exporters; - **computational notes** (a note whose content evaluates and renders results - inline); shareable λδ packages. - -2. **The letter comes first — as an enabler, not a rule.** Durable, - human-readable notes are what *make* the augmentation possible: because the - letter is legible and lasting, computation has something real to amplify. - Durability is the launchpad, not the leash. The measure of any feature is a - generative question — *does this help the letter reach a richer future - self?* — not a veto. - -### Architecture (one engine, everywhere) - -- **Reader → value model → evaluator** in `core/src/lambdadelta/`. Lexical - scope, closures, tail calls; special forms `quote`, `if`, `let`, `fn`/`lambda`, - `define`, `quasiquote`/`unquote`, `defmacro`. -- **Notebook as first-class values.** Builtins expose the whole core surface as - a Lisp library: `(notes)`, `(note id)`, `(title n)`, `(content n)`, - `(links n)`, `(backlinks n)`, `(attr n k)`, `(set-attr! n k v)`, - `(create-note! …)`, `(link! a b)`, `(search q)`, `(run-agent id)`. Pure - readers vs. `!`-suffixed mutators. -- **Sandbox.** Deterministic; no I/O or network; a step/time/recursion **budget** - so a user macro can never hang the tab (heavy jobs move to a Web Worker later). -- **Persistence = homoiconicity in action.** Functions, macros, formulas, and - agent-programs are stored *as data in the notebook*, so they save/load/sync - with everything else. A notebook carries its own behaviour. -- **Build, don't embed.** A focused interpreter (rather than a general Rust - Lisp crate) because the whole point is deep notebook-as-data interop and a - strict browser sandbox — a general embed fights both. No new npm deps - (Bun-only MUST); implemented in Rust/WASM (no new TS/Python/Go); MPL-2.0. - -### Subsumes, does not replace - -The merged features become friendly surfaces over the one engine: the current -agents DSL is L0 sugar that compiles to λδ; agent *actions* become arbitrary λδ -programs; computed attributes, prototype inheritance, and (future) adornment -rules are λδ expressions. - -## Tinderbox capability coverage - -| Tinderbox capability | Under Nexia-List + λδ | Status | -|---|---|---| -| Attributes on notes | Present (`attributes` map); λδ reads/writes | Matched (typing to add) | -| Agents (persistent queries) | Present; queries become λδ predicates | Replaced & exceeded | -| Agent actions | Arbitrary λδ programs (not just Collect/SetAttribute) | Exceeded | -| Rules & Edicts | λδ re-evaluated on change / on schedule | Matched | -| OnAdd / smart containers | λδ action on entry to container/adornment | Matched | -| Adornments (map regions) | Planned; with λδ actions = programmable regions | Exceeded (when landed) | -| Action-code language | λδ *is* this — a real language with macros | Vastly exceeded | -| Stamps | A λδ function applied to the selection | Matched | -| Prototypes / inheritance | `note.prototype` exists; resolution + computed inherited attrs | Exceeded | -| Export templates | λδ functions `notebook → string`; quasiquote templating | Exceeded | -| Links | Bidirectional present; **typed** links | Partial (typing to add) | -| Map / spatial view | Present (drag/pan/zoom) | Matched | -| Outline / Timeline / Attribute-browser / Treemap | Not yet (have list + graph) | Still lacking | -| Typed attributes (number/date/bool/color/set…) | Values are untyped JSON today | Still lacking | -| Attribute-driven visuals (colour/size/badge) | Not yet; λδ can compute style from a formula | Partial → exceeded | -| Rich-text note content | Plain text today | Still lacking | -| Dates / calendar / events | `chrono` in core; no calendar/timeline UI | Partial | -| AppleScript automation (macOS-only) | Replaced by λδ — cross-platform, in-app, sandboxed | Replaced & exceeded | -| Import / Export | OPML + Markdown-vault present; λδ-driven custom formats | Matched → exceeded | - -## What will still be lacking (honest gap list) - -| Gap | Why it matters | Effort | -|---|---|---| -| Typed attribute system (schemas, per-type editors, validation) | Date math, colours, sets; richer λδ values | M | -| Extra views: outline, timeline, attribute-browser | Parity for non-spatial thinkers | M–L | -| Typed links / link types | Semantic graphs, filtered link queries | S–M | -| Rich-text / markdown rendering in notes | A real writing tool, not plain text | M | -| Attribute/formula-driven visual styling | "See relationships at a glance" payoff | M | -| Computed-attribute dependency tracking + memoization | Efficient recompute at 10k notes | M | -| Off-main-thread (Web Worker) evaluation | Keep the UI fluid under heavy programs | M | -| λδ std-library + package/sharing system | Turns power into an ecosystem | L (ongoing) | -| λδ debugging / tracing / error UX | Power users must see what went wrong | M | -| Desktop shell | Blocked on the external `gossamer` sibling | external | -| Sync / collaboration / mobile | Future / non-goals | L | - -## What λδ liberates that Tinderbox cannot match - -| Superpower | What it enables | Why Tinderbox can't match it | -|---|---|---| -| Homoiconic notebook-as-data | Metaprogram your knowledge base; generate notes/agents/structure | Its action code can't treat the document as first-class, macro-manipulable data | -| A real language (recursion, HOF, closures, macros) | User-defined DSLs & workflows (GTD, Zettelkasten, research pipelines) as installable packages | Its expression language is bounded and not extensible | -| Computational / literate notes | A note that evaluates and renders results inline | It has export templates, not live in-canvas computation | -| Agents as programs, not just queries | Continuous graph maintenance: indices, invariants, restructuring, generation | Its agents are query + limited action | -| Programs travel with the notebook | Shareable "smart notebooks" that carry their own behaviour; reproducible | Documents carry actions, not a reusable code library | -| One portable, sandboxed automation language | Same automation on web/desktop/mobile — deterministic and safe | AppleScript is macOS-only and out-of-process | -| A moldable tool (Emacs-like) that stays simple by default | Users extend commands, views, exporters — without touching L0 | Not user-extensible at that depth | -| Verifiable substrate | Unit-test notebook programs; property-test invariants | No comparable testable automation model | -| Open · local-first · cross-platform · free | No lock-in; your data and programs are yours | macOS-only, closed, paid | - -## Phased rollout (each phase shippable; L0 UX untouched) - -- **L0 — Substrate:** reader, value model, evaluator, core special forms, - notebook builtins, and the safety budget in `core/src/lambdadelta/`; - wasm-exposed `eval`. Headless and fully tested (reader round-trip, evaluator - determinism, sandbox limits, property tests). No UX change — the ideal first - PR because it carries no UI risk. -- **L1 — Formulas:** "fx" fields for computed attributes and agent predicates; - the simple DSL compiles to λδ. -- **L2 — Actions & macros:** note/agent/adornment actions; `defmacro`; a small - std-library; user functions stored in the notebook. -- **L3 — REPL/console.** -- **L4 — Computational notes + packages.** - -## Consequences - -- **Positive:** every existing feature gains a principled, unbounded expression - layer; the notebook becomes programmable by its owner; automation is - portable, sandboxed, and testable; the tool becomes moldable while staying - Tinderbox-simple by default. -- **Costs / risks (with mitigations):** - - *Browser hang from user code* → step/time/recursion budget; cooperative - cancellation; Web Worker for heavy jobs later. - - *Scaring basic users / UX creep* → progressive disclosure; parentheses only - appear behind an fx/console door; the simple DSL stays the default surface; - excellent error messages. - - *Notes↔values interop drift* → one canonical bridge in core, guarded by - contract tests (as with the existing golden-fixture pattern). - - *Scope explosion* → L0 is self-contained and verifiable headless; ship - incrementally. - - *Stored code longevity* → definitions are data in the notebook; eval-safety - and versioning on load. - -## Relation to other decisions - -- Builds on **ADR-0001** (Rust core → WASM, single source of truth). -- Honours **ADR-0004** (Bun-only): the interpreter is homegrown Rust, no new - JS package-manager surface. -- Respects the `INTENT.contractile` North Star and architectural invariants. diff --git a/docs/adr/0004-bun-only-toolchain.adoc b/docs/adr/0004-bun-only-toolchain.adoc new file mode 100644 index 0000000..65b0b27 --- /dev/null +++ b/docs/adr/0004-bun-only-toolchain.adoc @@ -0,0 +1,37 @@ +== ADR-0004: Bun-only JavaScript toolchain + +* Status: Accepted +* Date: 2026-08-01 +* Supersedes: link:0002-deno-only-interpretation.md[ADR-0002] + +=== Context + +The Hyperpolymath estate standardized on Bun. Nexia-List previously used +Deno for dependency resolution, tasks, tests, scripts, and esbuild +orchestration. Keeping a repository-local exception would duplicate +toolchains and leave its machine-readable invariants at odds with estate +policy. + +=== Decision + +* Bun is the sole JavaScript runtime and package manager. +* Dependencies and tasks live in `+package.json+`; exact resolution +lives in `+bun.lock+`. +* `+bun install --frozen-lockfile+` is the reproducible install and CI +path. +* UI contracts use `+bun:test+`. +* Production and development bundles use Bun’s native bundler and +server. +* Biome provides JavaScript formatting and linting. +* npm, Deno, Yarn, and pnpm CLIs and lockfiles are not used. + +Rust, Cargo, wasm-bindgen, ReScript, and the browser architecture are +unchanged. + +=== Consequences + +The repository has one estate-standard JavaScript toolchain, one +JavaScript lockfile, and no compatibility layer retaining Deno APIs. +Historical documents may describe the former Deno arrangement, but +current operational documentation and machine-readable policy point +here. diff --git a/docs/adr/0004-bun-only-toolchain.md b/docs/adr/0004-bun-only-toolchain.md deleted file mode 100644 index d04186f..0000000 --- a/docs/adr/0004-bun-only-toolchain.md +++ /dev/null @@ -1,33 +0,0 @@ - -# ADR-0004: Bun-only JavaScript toolchain - -- Status: Accepted -- Date: 2026-08-01 -- Supersedes: [ADR-0002](0002-deno-only-interpretation.md) - -## Context - -The Hyperpolymath estate standardized on Bun. Nexia-List previously used Deno -for dependency resolution, tasks, tests, scripts, and esbuild orchestration. -Keeping a repository-local exception would duplicate toolchains and leave its -machine-readable invariants at odds with estate policy. - -## Decision - -- Bun is the sole JavaScript runtime and package manager. -- Dependencies and tasks live in `package.json`; exact resolution lives in - `bun.lock`. -- `bun install --frozen-lockfile` is the reproducible install and CI path. -- UI contracts use `bun:test`. -- Production and development bundles use Bun's native bundler and server. -- Biome provides JavaScript formatting and linting. -- npm, Deno, Yarn, and pnpm CLIs and lockfiles are not used. - -Rust, Cargo, wasm-bindgen, ReScript, and the browser architecture are unchanged. - -## Consequences - -The repository has one estate-standard JavaScript toolchain, one JavaScript -lockfile, and no compatibility layer retaining Deno APIs. Historical documents -may describe the former Deno arrangement, but current operational documentation -and machine-readable policy point here. diff --git a/docs/decisions/contractile-regeneration-2026-08-01.adoc b/docs/decisions/contractile-regeneration-2026-08-01.adoc new file mode 100644 index 0000000..7085fc5 --- /dev/null +++ b/docs/decisions/contractile-regeneration-2026-08-01.adoc @@ -0,0 +1,20 @@ +== Contractile regeneration decision — 2026-08-01 + +Issue #49 identified drift between `+contractiles/must/Mustfile.a2ml+` +and the generated `+contractile.just+`, while the `+contractile+` +generator was unavailable in the repository toolchain. + +Decision: + +* The canonical Mustfile now contains the missing SPDX rule alongside +the existing critical hardcoded-path rule. +* `+contractile.just+` is manually aligned with those canonical rules so +local checks are truthful today: neither rule is omitted, and the SPDX +walk excludes dependencies, build output, and generated ReScript +JavaScript. +* Generator-based regeneration is explicitly deferred until +`+contractile+` is packaged in the estate toolchain. At that point, run +`+contractile gen-just --dir contractiles+` and require a zero diff from +the checked-in file before removing this deferral. + +This is a bounded tooling deferral, not a waiver of either invariant. diff --git a/docs/decisions/contractile-regeneration-2026-08-01.md b/docs/decisions/contractile-regeneration-2026-08-01.md deleted file mode 100644 index c8ac2f7..0000000 --- a/docs/decisions/contractile-regeneration-2026-08-01.md +++ /dev/null @@ -1,20 +0,0 @@ - -# Contractile regeneration decision — 2026-08-01 - -Issue #49 identified drift between `contractiles/must/Mustfile.a2ml` and the -generated `contractile.just`, while the `contractile` generator was unavailable -in the repository toolchain. - -Decision: - -- The canonical Mustfile now contains the missing SPDX rule alongside the - existing critical hardcoded-path rule. -- `contractile.just` is manually aligned with those canonical rules so local - checks are truthful today: neither rule is omitted, and the SPDX walk excludes - dependencies, build output, and generated ReScript JavaScript. -- Generator-based regeneration is explicitly deferred until `contractile` is - packaged in the estate toolchain. At that point, run - `contractile gen-just --dir contractiles` and require a zero diff from the - checked-in file before removing this deferral. - -This is a bounded tooling deferral, not a waiver of either invariant. diff --git a/docs/design/flyinglogic-devonthink-integration.adoc b/docs/design/flyinglogic-devonthink-integration.adoc new file mode 100644 index 0000000..b2facdd --- /dev/null +++ b/docs/design/flyinglogic-devonthink-integration.adoc @@ -0,0 +1,1198 @@ +== Flying Logic × DEVONthink → nexia-list: The Definitive Integration Design + +_Status: proposed · Supersedes: three exploratory designs +(crown-jewels-first, product-minimalist, substrate-maximalist) and their +adversarial review · Audience: nexia-list core contributors · Companion +to: ADR-0003 (λδ), the λδ spec v0.1, and the ROADMAP._ + +____ +A note is a letter we send to our future self. This document is about +giving that letter two new senses: the ability to *recall its own +forgotten neighbours*, and the ability to *reason over its own claims* — +both entirely on-device, both invisible until asked for. +____ + +''''' + +=== 1. TL;DR and the Unifying Thesis + +==== 1.1 TL;DR + +We steal the *intelligence and primitives* of Flying Logic and +DEVONthink, not their feature sprawl. Two crown jewels justify genuinely +new code in the Rust core; everything else is either already free in +nexia’s model, or ships later as `+.ld+` domain packages and Agent-DSL +sugar. + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|From |Crown jewel |Becomes |New native code +|*DEVONthink* |_See Also_ — local associative recall |One concordance / +inverted index → cosine kNN, BM25 search, Rocchio classify, SimHash +dedup, auto-tag |`+core/src/index.rs+` + +|*Flying Logic* |_Live confidence propagation_ — a spreadsheet for +reasoning |One pure single-pass DAG sweep over a *separate, additive* +typed-edge channel; junctors are ordinary `+:type+`-tagged notes; +operators are a native table a domain pack extends +|`+core/src/edge.rs+`, `+core/src/reason.rs+`, later +`+core/src/layout.rs+` +|=== + +Both are *derived, rebuildable indices that live beside the notes and +are never trusted from disk* — the exact discipline nexia already proved +with `+Notebook::backlinks+`. Both surface as *no-parenthesis L0 panels +that mount only when non-empty*, and both are reachable as *pure λδ +builtins* for power users. Neither is a monolithic subsystem: each is a +Rust module + a handful of host builtins + a derived view. + +The recommended *first PR* is headless: `+core/src/index.rs+` alone, +property-tested against the existing golden fixtures, with zero UI and +zero on-disk format change (§8). + +==== 1.2 The Unifying Thesis + +Both external tools collapse onto *one spine*: + +____ +*A typed, attribute-rich graph + a local intelligence engine + λδ +programs — all local-first, all derived-and-rebuildable, all invisible +by default.* +____ + +The move that unifies them is the move nexia already made for +`+backlinks+`: + +[arabic] +. *A derived index lives beside the notes* and is rebuilt on load, never +authoritative on disk (the concordance for DEVONthink; the confidence +map and the layered layout for Flying Logic). +. *Semantics are data, dispatched on `+:type+`/`+:op+`* through λδ +multimethods (`+defmulti … :type+`), so operators, entity classes and +whole methodologies ship as `+.ld+` packages without recompiling the +core. +. *The note model is untouched.* `+Note.links+` stays the untyped +associative fabric. A parallel, empty-by-default `+edges+` channel +carries implication. Existing notebooks round-trip byte-for-byte. +. *We keep the intelligence, not the interpreter tax.* A native operator +table is the fast default for the propagation hot path; the λδ +`+combine+` multimethod is the extension path for exotic domain +operators only (never called per-node in the inner loop). +. *Progressive disclosure holds the line.* Typed reasoning is opt-in at +L1+, never imposed. Flying Logic’s mandatory entity-class discipline is +the explicit anti-pattern. + +The result: the letter still reaches a richer future self — now able to +reason over its own claims and recall its own forgotten neighbours, +entirely on-device. + +''''' + +=== 2. What we are NOT importing (honest non-goals) + +These exclusions are load-bearing, not incidental. "`Do fewer things +well`" is a design constraint we enforce, and several of these were +flagged by the review as active hazards to avoid. + +==== 2.1 Categorically out of charter (never build) + +[width="100%",cols="50%,50%",options="header",] +|=== +|Excluded |Reason +|*OCR* (image/PDF → text) |Requires Tesseract-class engines and binary +pipelines. Content is plain text; Markdown rendering is the only planned +rich step. + +|*RSS / feeds, web clipper / Sorter, email import* |Each needs network +fetch, OS integration, or proprietary binary parsing — breaks +no-network-by-default. Manual Markdown paste is the only sympathetic +path. + +|*Multi-device / cloud sync, Server/Sharing edition* |Explicit ROADMAP +non-goals (no cloud-only storage, no real-time collaboration). +Persistence stays IndexedDB + file import/export; users sync exported +JSON/vault themselves. + +|*Rich binary document handling* (PDF annotation, RTFD, web archives) +and format-conversion actions |Different product category, heavy WASM +burden. The in-scope sliver is Markdown + the existing `+exchange.rs+` +vault. + +|*PDF / PNG / MS-Project diagram export* |Heavyweight rendering plus +PM-suite interop, against local-first / no-lock-in. Markdown + OPML +already cover interop; at most a client-side SVG snapshot of the layout +view. + +|*Mandatory generative AI* (chat, DALL·E, image description), *macOS +side-effects* (Speak / Bounce / SendMail), *AppleScript/JXA host* |The +λδ sandbox is deliberately no-I/O. λδ (homoiconic, budgeted, in-core) is +the scripting escape hatch, not an OS bridge. Local models +(Ollama/llama.cpp) are opt-in host capabilities only; no data leaves the +device by default. +|=== + +==== 2.2 In-scope in spirit, but deliberately not built as designed + +[width="100%",cols="50%,50%",options="header",] +|=== +|Excluded form |What we do instead +|*Promoting `+Note.links: Vec+` → `+Vec+`* (typed links in +the load-bearing field) |Verified to touch ~6 sites +(`+notebook.rs:75/105/260+` backlink loops, wikilink derivation, the +`+linksto:+` agent term, `+NoteView.links+`, `+note_to_value+`) and to +merge L0 associations with causal edges into one vector. We add a +*separate `+edges: Vec+` channel* instead — same capability, +near-zero blast radius, byte-identical round-trip. + +|*Per-node λδ multimethod dispatch inside the propagation sweep* |Each +`+i.apply+` decrements the 1M-step `+Budget+` and readers hold +`+nb.borrow()+`, so a `+combine+` method touching any mutator panics via +`+RefCell+` double-borrow — and it is materially slower at 10k nodes on +every spinner drag. We keep a *native operator table* for the hot path; +λδ `+combine+` is reserved for exotic/domain ops (§4.4). + +|*The six TOC Thinking-Process templates* (CRT, Evaporating Cloud, FRT, +PRT, Transition, S&T) as engine features |They are curated domains + +starter graphs. Shipping them in core imposes one methodology and +violates untyped-by-default. Ship as optional `+.ld+` domain packs + +starter notebooks (data). + +|*Full probability/arithmetic operator sprawl* (Product ×, Sum-Prob ⊕, +Proportion ∷, float-flow typing) on day one |Fuzzy min/max/complement + +edge weights cover ~80% of reasoning. The rest is trivially added later +as `+combine+` methods; there is no cost to waiting. + +|*A full Dempster–Shafer belief engine* |General DS combination is +O(2^\|frame\|) per node with power-set mass functions and conflict +renormalisation — non-interactive at 10k. A *DS-lite belief/plausibility +interval* is offered as an _optional domain_ only (§5.4). + +|*Exotic confidence math as the default* |The default scalar is +fuzzy-boolean in [0,1] with 0.5 = Indeterminate. Probability and DS-lite +are per-domain opt-ins, never mixed inside one graph (§5.4). + +|*Persisting the concordance or propagation results to disk as +authoritative state* |Bloats the human-readable JSON and risks drift. +Both are rebuilt on load, exactly like `+backlinks+`. Derived state is +never trusted from disk. + +|*Animated incremental layout, presentation/step-through mode, +typed-attribute schemas as a prerequisite* |Pure polish or decoupled +workstreams. The static layered view + spinners already deliver the +capability; `+:confidence+`/`+:op+`/`+:weight+` ride the existing +untyped JSON attrs fine. +|=== + +''''' + +=== 3. Data-model deltas + +The whole delta is: *one new struct + one new field on `+Note+`*, *three +derived indices + a triggers collection on `+Notebook+`*, and *a few new +Agent-DSL predicates*. Everything else rides attributes and `+:type+`, +which the host already surfaces. + +==== 3.1 The `+Edge+` struct — a separate, additive channel + +`+Note.links+` (untyped association: wikilinks, backlinks, the L0 graph) +*stays exactly as it is* and does *not* participate in confidence flow. +Implication lives on a new, empty-by-default field so that a notebook +with no reasoning graph serialises identically to today. + +[source,rust] +---- +// core/src/edge.rs (new) +use crate::note::NoteId; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; + +/// A typed, weighted, directed implication edge — where Flying Logic's logic lives. +/// `kind` is a bare string so domains add edge classes as DATA, never as a core enum. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Edge { + pub to: NoteId, + + /// Edge class / dispatch tag: "implies", "supports", "inhibits", "feeds", + /// "contains", … Default "link" ⇒ a typed edge with no ceremony. + #[serde(default = "Edge::plain_kind", skip_serializing_if = "Edge::is_plain")] + pub kind: String, + + /// Influence weight in [-1, 1]. +1 passes through, 0 → Indeterminate (0.5), + /// -1 negates (Flying Logic's weighting transform). Default +1. + #[serde(default = "Edge::unit", skip_serializing_if = "Edge::is_unit")] + pub weight: f64, + + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + pub attrs: HashMap, +} + +impl Edge { + fn plain_kind() -> String { "link".into() } + fn is_plain(k: &str) -> bool { k == "link" } + fn unit() -> f64 { 1.0 } + fn is_unit(w: &f64) -> bool { (*w - 1.0).abs() < f64::EPSILON } +} +---- + +[source,rust] +---- +// core/src/note.rs — ONE new field on `Note`, empty for every existing note + /// Typed/weighted implication edges (the reasoning graph). Distinct from + /// `links` (untyped association). Empty ⇒ the note is invisible to the + /// reasoning engine and serialises to nothing new. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub edges: Vec, +---- + +*Why this beats the untagged `+Link+` enum.* An untagged serde shim on +the load-bearing `+links+` field is the highest-risk round-trip against +the golden fixtures (untagged ambiguity, poor error messages). An +empty-by-default `+Vec+` guarded by +`+skip_serializing_if = "Vec::is_empty"+` is provably byte-identical for +every existing notebook: no new key appears until a note actually has an +edge. `+Note::new+` initialises `+edges: Vec::new()+`. + +*Back-compat proof obligation (Phase gate).* The golden-fixture suite +(`+golden.rs+`) must show that every existing fixture serialises to the +same bytes after this field is added. This is a mechanical, checkable +gate — not a hope. + +==== 3.2 Junct / operator / entity / group notes — zero new node primitive + +The λδ host already derives `+:type+` from `+attributes["type"]+`, and +`+note_matches+` already supports `+attr:type=junct+`. So junctors, +entity classes and groups are *ordinary notes distinguished by +attributes* — no schema work. + +[source,jsonc] +---- +// An AND-junct as it sits in the notebook JSON — nothing new in the format +{ + "id": "…", "title": "AND", + "attributes": { "type": "junct", "op": "and" }, + "edges": [ { "to": "", "kind": "feeds" } ] +} +---- + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Concept |Encoding |Surfaced as +|Entity / claim node |`+attributes.type = "claim"+` (or a domain class) +|`+:type+` + +|Combining operator |`+attributes.op = "and" \| "or" \| "not" \| …+` +|`+:op+` + +|Asserted driver confidence |`+attributes.confidence = 0.8+` (f64 ∈ +[0,1]) |`+:confidence+`; editor spinner + +|Entity/edge styling |`+attributes.type+` / `+Edge.kind+` |`+render+` / +`+edge-style+` multimethod + +|Group / collapsible subgraph |`+attributes.type = "group"+`, members +via `+:contains+` edges |deferred (§8, L4) +|=== + +Computed confidences on *driven* nodes are *never persisted* — they +would masquerade as asserted drivers and bloat the JSON. Propagation is +derived, exactly like `+backlinks+`. + +==== 3.3 The concordance / similarity index — the one shared substrate + +Rebuildable and incremental, modelled on `+backlinks+`, but marked +`+#[serde(skip)]+` and *always rebuilt on load* — the strongest possible +"`don’t destabilise L0`" guarantee: zero on-disk format change. + +[source,rust] +---- +// core/src/index.rs (new) — powers See-Also, Classify, dedup, and BM25 search +use crate::note::NoteId; +use std::collections::HashMap; + +type TermId = u32; +type DocId = u32; // DENSE note index — the decisive WASM-memory move (§7) + +#[derive(Clone, Copy)] +struct Posting { doc: DocId, tf: u32 } + +#[derive(Default)] +pub struct SimilarityIndex { + interner: HashMap, TermId>, + vocab: Vec>, + + // note ↔ dense id — postings store an 8-byte (DocId, tf), NEVER a 16-byte Uuid + doc_of: HashMap, + note_of: Vec, + + postings: Vec>, // TermId → postings, sorted by doc + df: Vec, // document frequency per term + + forward: Vec>, // DocId → (term, tf) vector, for reindex + norms + doc_len: Vec, // token count per doc (BM25 |d|) + doc_norm: Vec, // cached L2 norm of the tf-idf vector + + simhash: Vec, // near-dup fingerprint, FIXED seed + content_hash: HashMap<[u8; 32], Vec>, // exact-dup buckets (blake3) + + n_docs: u32, + total_len: u64, // for avgdl +} +---- + +*The dense-`+DocId+` rule is mandatory, not an optimisation.* Postings +must store `+(u32, u32)+`, never the 16-byte `+Uuid+`. This is what +keeps the index near ~35–50 MB at 10k notes (§7) — comfortable in a +browser tab. + +Tokenisation is deterministic (Unicode word segmentation → lowercase → +stop-list → optional Porter stem), with a *fixed seed* for SimHash and +no clock/RNG — honouring the λδ sandbox and IndexedDB determinism. +Crates are Rust-only via cargo (`+unicode-segmentation+`, +`+rust-stemmers+`, `+blake3+`), so the Bun-only-for-JS MUST is +untouched; BM25/cosine/NB are hand-rolled. A hand-rolled ASCII tokenizer ++ static stop-list is the zero-dep fallback if WASM size ever matters. + +==== 3.4 `+Notebook+` deltas + +[source,rust] +---- +// core/src/notebook.rs +pub struct Notebook { + notes: HashMap, + + #[serde(default)] + backlinks: HashMap>, // unchanged (plain links) + + /// Reverse index for TYPED edges, rebuilt exactly like `backlinks`. + #[serde(default)] + in_edges: HashMap>, // (source note, edge index) + + /// The concordance. DERIVED — never persisted, rebuilt on load. + #[serde(skip)] + index: crate::index::SimilarityIndex, + + #[serde(default)] + agents: Vec, + + /// Smart-Rule triggers (event → predicate → λδ action). Empty by default. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + triggers: Vec, + + pub name: String, + pub created_at: DateTime, + pub modified_at: DateTime, +} +---- + +`+rebuild_backlinks+` becomes `+rebuild_indices+` (keeping the old name +as a thin caller): it rebuilds `+backlinks+`, `+in_edges+`, *and* +`+index+` from the notes. `+WasmNotebook::from_json+` already calls the +rebuild path, so all three are populated on every load. + +==== 3.5 Incremental maintenance — hook the write paths, including the title path + +`+set_content+` is the primary write path and already re-derives +wikilinks per edit. Reindexing hangs off it: + +[source,rust] +---- +pub fn set_content(&mut self, id: &NoteId, content: impl Into) -> Vec { + let content = content.into(); + // … existing wikilink logic … + self.index.reindex(*id, &self.title_of(id), &content); // NEW + // … +} +---- + +`+reindex+` diffs the note’s previous forward vector against the new +token bag — decrement `+df+` and remove postings for dropped terms, add +postings for new terms, recompute this doc’s `+doc_len+`, `+doc_norm+`, +and `+simhash+`. Cost is *O(tokens in the edited note)*, not the corpus. + +*Resolving the title-path desync (open tension).* Titles dominate +short-note similarity, yet `+update_title+` / `+bi_set_title+` bypass +`+set_content+`. Both title write paths *must* call +`+index.reindex(id, new_title, current_content)+` too. This is a +required part of the first PR’s contract, not a follow-up. + +*Resolving global-idf / norm drift (open tension).* Every edit changes +`+df+`/`+N+`, which technically shifts the global idf and thus every +_other_ note’s L2 norm. We do not chase this per-edit. Instead: - +`+doc_norm+` is cached per note and recomputed only for the edited note +on `+reindex+`. - See-Also and BM25 compute idf against the *current* +`+df+`/`+N+` at query time, so ranking is always consistent with the +live corpus even though cached norms lag slightly. The lag affects only +the _magnitude_ of the cosine denominator for unedited notes, never +their term content, and `+#[serde(skip)]+` + rebuild-on-load bounds any +accumulated drift to a single session. - A cheap periodic/opportunistic +full `+rebuild_indices+` (e.g. on notebook open, already free) resets +norms exactly. We accept bounded intra-session staleness as the honest, +documented trade — it is imperceptible in top-k ordering and it keeps +edits O(tokens). + +==== 3.6 Agent-DSL extensions (DEVONthink Smart Groups already == Agents) + +`+Agent { query }+` *is* a Smart Group, 1:1, already shipping — and +replicants are free (§6). We only enrich the parser: + +[source,rust] +---- +// core/src/agent.rs — extend `enum Term` +enum Term { + Text(String), Title(String), Attr(String, String), LinksTo(NoteId), Never, + Type(String), // type:junct — surfaces the FL entity class + Similar(NoteId), // similar: — DEVONthink See-Also as a saved query + Near(NoteId), // near: — cosine ≥ τ (near-duplicate) + Conf(Ordering, f64), // conf:>0.7 — filter by propagated confidence + Edge(String), // edge:supports — has an outgoing typed edge of kind +} +---- + +`+similar:+`/`+near:+`/`+conf:+` need the index or the propagation +result, which the pure `+note_matches+` (which has no notebook context) +cannot supply. *Resolution:* add a notebook-aware sibling +`+note_matches_in(note, query, &Notebook)+` used by `+run_query+`; the +pure `+note_matches+` stays for index-free terms. The whitespace-ANDed +L0 surface is unchanged; these are progressive-disclosure power +operators. + +==== 3.7 Triggers (Smart Rules) — a third `+Notebook+` collection, like `+agents+` + +[source,rust] +---- +// core/src/trigger.rs (new) +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Trigger { + pub id: Uuid, + pub event: TriggerEvent, + pub condition: String, // an Agent-DSL predicate (reuses note_matches) + pub action: String, // a λδ Action-context program + #[serde(default = "yes")] pub enabled: bool, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum TriggerEvent { + OnCreate, OnSetContent, OnSetAttr, OnLink, OnTag, OnRunAgent, + Scheduled { every: String }, // "hourly" | "daily" | "workdays" — host-driven tick +} +---- + +The *local-first subset* of DEVONthink’s event taxonomy. Networked +events (OnDownload/OnScan/OnOCR/OnSync) are dropped. The host fires a +matching trigger’s `+action+` in Action context under a fresh +`+Budget+`. Triggers travel with the notebook, like agents. + +''''' + +=== 4. The λδ layer + +The kernel already implements `+defmulti+`/`+defmethod+`, +`+register_builtin+`, a `+PRELUDE+` eval’d on `+Interp::new()+`, and +`+note_to_value+` deriving `+:type+`. We extend the bridge, add +builtins, and register a small operator prelude. + +==== 4.1 Bridge additions (`+note_to_value+`) + +Keep the bridge the single canonical translator; add keys additively +(`+:links+` unchanged): + +[source,rust] +---- +// lambdadelta_host.rs :: note_to_value — additive pairs +(Value::kw("edges"), edges_value(¬e.edges)), // vector of {:to :kind :weight :attrs} +(Value::kw("confidence"), num_or_nil(note.attributes.get("confidence"))), +(Value::kw("op"), note.attributes.get("op").map(json_to_value).unwrap_or(Value::Nil)), +---- + +==== 4.2 New builtins + +*Pure readers* (legal in Formula / Agent-predicate / Agent-query +contexts — deterministic, budgeted, no mutation): + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Builtin |Returns |Backed by +|`+(see-also n)+` / `+(see-also n k)+` |top-k similar note maps +|`+index.see_also+` + +|`+(similar? a b)+` |cosine ∈ [0,1] |`+index.cosine+` + +|`+(classify n)+` |ranked `+[class score]+` |Rocchio centroid + +|`+(duplicates n)+` |exact + near duplicates of `+n+` |blake3 + SimHash + +|`+(search-ranked q)+` |notes ranked by BM25 |`+index.bm25+` + +|`+(confidence n)+` |asserted-or-computed truth of `+n+` in [0,1] +|`+reason::propagate+` (memoised per eval) + +|`+(propagate)+` |map `+{#uuid → float}+` over driven notes +|`+reason::propagate+` + +|`+(drivers)+` / `+(driven)+` |notes with no / some in-edges |reasoning +graph + +|`+(edges n)+` / `+(in-edges n)+` |typed edges of / into `+n+` +|`+Note.edges+` / `+in_edges+` +|=== + +*`+!+`-mutators* (Action context only — budgeted, undoable): + +[width="100%",cols="50%,50%",options="header",] +|=== +|Builtin |Effect +|`+(set-confidence! id v)+` |assert a driver’s `+:confidence+` + +|`+(link-typed! from to kind weight)+` |push an `+Edge+` + +|`+(junct! op & operand-ids)+` → junct id |create +`+{:type :junct :op op}+`, wire operands → junct + +|`+(add-tag! id t)+` / `+(remove-tag! id t)+` |maintain +`+attributes.tags+` (a Vec) + +|`+(auto-classify! id thresh margin)+` |apply top class *only if* +`+top ≥ thresh ∧ top − runner-up ≥ margin+` +|=== + +Readers register via `+reader(…)+`, mutators via `+mutator(…)+`, sharing +the existing `+Rc>+` — line-for-line with the current +host bindings. + +==== 4.3 Evaluation contexts + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|Context |Bound |May mutate? |New usage +|Formula (pure, `+self+`) |`+self+` |no |`+(:confidence (propagate))+`, +`+(see-also self 5)+`, `+(classify self)+` + +|Agent-predicate (`+self+`, truthy) |`+self+` |no +|`+similar:+`/`+conf:+`/`+type:+` compile here + +|Agent-query (→ notes) |— |no +|`+(filter #(> (first-score (see-also % 1)) 0.6) (notes))+` + +|Action (`+self+`, mutating) |`+self+` |yes |`+(propagate!)+`, +`+(auto-classify! …)+`, `+(link-typed! …)+` + +|*Trigger-action* _(new)_ |`+self+`, *`+event+`* |yes |Smart-Rule +bodies; `+self+` = triggering note +|=== + +The existing L0 agent DSL still compiles to an Agent-predicate λδ +expression. `+see-also+`/`+classify+`/`+propagate+` are pure and legal +in fx fields (L1); `+auto-classify!+` and trigger bodies are +Action-context, so a probabilistic model can only touch data through the +same audited, undoable seam as any mutator. + +==== 4.4 Multimethod dispatch — pluggable semantics, with a precise native/λδ seam + +Three multimethods carry the transferred intelligence and make domains +pluggable as `+.ld+` packages: + +[source,clojure] +---- +;; operator algebra — the FL crown jewel's SEMANTICS, extensible without core edits +(defmulti combine (fn [op _inputs] op)) +(defmethod combine :and [_ xs] (reduce min 1.0 xs)) ; weakest link +(defmethod combine :or [_ xs] (reduce max 0.0 xs)) +(defmethod combine :not [_ xs] (- 1.0 (first xs))) +(defmethod combine :default [_ xs] (reduce min 1.0 xs)) ; entity default = AND-ish + +;; rendering — one card per entity/edge class +(defmulti render :type) +(defmethod render :junct [n] {:shape :diamond :label (:op n)}) +(defmethod render :claim [n] {:shape :rounded-box :spinner (:confidence n)}) + +;; edge styling / domain validation +(defmulti edge-style (fn [e] (:kind e))) +(defmulti validate :type) +---- + +*The seam contract (resolving the open tension explicitly).* The +propagation sweep in `+reason.rs+` runs a *native `+Op+` table* in Rust +for the known operators `+{:and :or :not}+` (and, once shipped, the +probability pack’s ops). It calls the λδ `+combine+` multimethod *only* +when a node’s `+:op+` is not in the native table. The contract: + +[arabic] +. *Native is the source of truth for shipped ops.* `+combine+`’s λδ +methods for `+:and/:or/:not+` exist for REPL inspection and must be +_definitionally identical_ to the native table (min / max / 1−x). A +conformance test asserts native and λδ agree on a sample grid, so +crossing the seam never silently changes semantics. +. *λδ `+combine+` is never called per-node for a known op.* This is what +avoids the verified `+Budget+` drain and `+RefCell+` re-borrow panic, +and keeps 10k-node spinner drags interactive. +. *A domain pack adds an op by adding a `+combine+` defmethod.* Unknown +ops resolve through λδ and are accepted as slower; if a domain wants its +op on the fast path, it registers a native op id via the host capability +(a documented, reviewed extension point), not by patching the evaluator. + +This makes the native table a *floor, not a ceiling*: fast by default, +infinitely extensible through data. + +==== 4.5 Domains as `+.ld+` packages + +A _domain_ is a `+.ld+` package that registers a set of +`+:type+`/`+:op+` tags plus their `+combine+` / `+render+` / +`+validate+` / `+edge-style+` methods and a suggested attribute +vocabulary — aligning 1:1 with the kernel/host + plugin-package design +(issue #33). Flying Logic’s "`domain = swappable class+operator pack`" +and DEVONthink’s Classify-by-corpus both become data, not core enums. +Shipped examples (§8, L2/L4): `+flying-logic.ld+` (junct/claim/group +types + operator algebra + render cards), `+librarian.ld+` +(confidence-gated auto-classify trigger + kNN auto-tag), and later the +TOC methodology packs and a probability operator pack. + +''''' + +=== 5. Flying Logic mapping + +==== 5.1 Capability → nexia mechanism + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Flying Logic capability |nexia mechanism |Cost +|Entity = statement node |Existing `+Note+`; class = `+attributes.type+` +→ `+:type+` |free + +|Directed causal/logical edge |New +`+Note.edges: Vec { kind, weight, attrs }+` |S (struct) + +|Junctor (AND/OR bundler) |A note `+{:type :junct :op …}+`; operands → +junct → target via typed edges |free (data) + +|Necessary vs sufficient |`+:op :and+` = min, `+:op :or+` = max, +selected in `+combine+` |free + +|Weighted / negated edge |`+Edge.weight ∈ [-1,1]+` + the weighting +transform (w=0→0.5, w<0→negation) |free + +|Confidence scalar |`+attributes.confidence: f64+`; 0.5 = Indeterminate +midpoint |free + +|*Live confidence propagation* |*`+reason::propagate+` — single-pass DAG +sweep; `+(confidence n)+` / `+(propagate)+`* |*M (crown jewel)* + +|Advanced operators (XOR/⊕/×/∷) |native `+Op+` table (hot path) + +`+defmethod combine :op+` for domain ops |S core, ∞ via packs + +|Domains (class + operator packs) |`+.ld+` packages registering +`+:type+`/`+:op+` methods + attr vocab |pack (data) + +|Entity classes & styling |`+:type+` + `+defmethod render :type+`; +prototype supplies class defaults |S + +|TOC thinking-process templates |*out-of-core* domain packs + starter +notebooks (§2) |pack + +|Automatic layered DAG layout |new `+core/src/layout.rs+` (Sugiyama) → +`+ReasoningView+` |M + +|Animated incremental layout |*deferred* — ReScript interpolation only +|— + +|Groups / collapse |`+{:type :group}+` + `+:contains+` edges; +layout/propagate as one node |M (later) + +|Back-edge cycle handling |DFS marks & excludes back edges; drives a +"`you made a loop`" cue |free (in M) + +|Confidence spinners (live) |`+SetConfidence+` msg → `+set-confidence!+` +→ re-`+propagate+` → re-render |S + +|Fuzzy vs float flow typing |`+:flow+` tag on computed value; *deferred* +|— + +|Operators inspector |settings surface writing `+:op+` (per-note + +notebook default) |S + +|PDF/PNG/MS-Project export |*out* (§2); at most a client-side SVG +snapshot |— +|=== + +==== 5.2 The reasoning-graph view (layered DAG layout) + +`+core/src/layout.rs+` runs a compact Sugiyama pipeline over the +typed-edge graph and returns *positions that are computed, never stored* +— `+Note.position+` stays owned by the manual spatial canvas, so +auto-layout and free-canvas coexist: + +[arabic] +. Cycle removal — reuse `+reason.rs+`’s back-edge set (temporarily +reverse those edges). +. Layer assignment — longest-path layering; insert dummy nodes on edges +spanning >1 layer. +. Crossing minimisation — median/barycenter, a few up/down sweeps. +. x-coordinate assignment — Brandes–Köpf for straight, balanced edges; y +from layer index; honour an orientation flag. + +Exposed as `+reasoning_layout(orient) -> [{id, x, y, layer}]+`. It +replaces the `+GraphView+` placeholder and reuses `+GraphLayout.res+`’s +`+nodePos+` seam — the ReScript view only draws; the layout comes from +Rust. Animated transitions are deferred polish. + +==== 5.3 Confidence propagation — the chosen default semantics + +[source,rust] +---- +// core/src/reason.rs +pub type Conf = f32; // [0,1]; f32 halves map memory + +/// Flying Logic's documented weighting transform, applied per in-edge +/// BEFORE the operator: w=+1 pass-through, w=0 → 0.5, w=-1 → negation. +#[inline] +fn weight_edge(v: Conf, w: f32) -> Conf { + let s = 2.0 * v - 1.0; // [0,1] → [-1,1] + (s * w + 1.0) * 0.5 // → [0,1] +} + +#[derive(Clone, Copy)] +pub enum Op { And, Or, Not, /* pack: */ Xor, Product, SumProb, Proportion } + +/// Combine already-weighted inputs. Empty input set ⇒ 0.5 (Indeterminate). +fn combine(op: Op, xw: &[(Conf, f32)]) -> Conf { + if xw.is_empty() { return 0.5; } + match op { + Op::And => xw.iter().map(|p| p.0).fold(f32::INFINITY, f32::min), // weakest link + Op::Or => xw.iter().map(|p| p.0).fold(f32::NEG_INFINITY, f32::max), + Op::Not => 1.0 - xw[0].0, + Op::Product => xw.iter().map(|p| p.0).product(), + Op::SumProb => xw.iter().fold(0.0, |a, &(b, _)| a + b - a * b), // 0.5⊕0.5=0.75 + Op::Proportion => { // zero-weight abstains + let (num, den) = xw.iter().fold((0.0, 0.0), |(n, d), &(x, w)| { + let a = w.abs(); (n + a * x, d + a) + }); + if den == 0.0 { 0.5 } else { num / den } + } + Op::Xor => fuzzy_xor(xw), // max_i min(x_i, 1 - max_{j≠i} x_j) + } +} + +/// ONE deterministic pass. (1) DFS marks back edges (edges to a node on the +/// stack) and drops them. (2) Kahn topo over the DAG. (3) Sweep: drivers keep +/// their asserted value; driven nodes combine weighted inputs. O(V+E), no fixpoint. +pub fn propagate(nb: &Notebook) -> HashMap { + let g = ReasonGraph::build(nb); // typed edges only, nodes/edges SORTED (§9) + let back = g.back_edges(); // DFS over a sorted adjacency + let order = g.kahn_topo(&back); + let mut conf = HashMap::with_capacity(order.len()); + for id in order { + let ins = g.in_edges(id, &back); + let c = if ins.is_empty() { + asserted(nb, id).unwrap_or(0.5) // DRIVER + } else { + let xw: Vec<(Conf, f32)> = ins.iter() + .map(|e| (weight_edge(conf[&e.from], e.weight as f32), e.weight as f32)) + .collect(); + combine(op_of(nb, id), &xw) // DRIVEN + }; + conf.insert(id, c); + } + conf +} +---- + +*Why fuzzy min/max weakest-link is the default:* + +* *Monotone, order-independent, single-pass.* It evaluates in one +topological sweep, O(V+E), with no fixpoint iteration — the only family +that stays interactive under spinner-drag at 10k nodes. +* *0.5 = Indeterminate is a better "`unknown`" than `+null+`.* A +principled, bounded, neutral midpoint that the whole engine runs on. +* *Negation is a number.* The edge-weight transform gives graded support +(w<1), abstention (w=0 → 0.5), and logical negation (w=−1) with no extra +node type. +* *Back edges are drawable but excluded.* Feedback loops render (dashed, +with a "`you made a loop`" cue) yet never break evaluation, because the +evaluated graph is a DAG. + +*Made domain-overridable (resolving the semantics tension):* +pluggability is layered so two domains can never disagree _inside one +graph_: - The *combine function* is chosen per node by `+:op+` (native +table + λδ extension, §4.4). - The *scalar semantics* (fuzzy vs +probability vs DS-lite interval) is chosen per *domain package*, and a +notebook’s reasoning graph declares its domain once. A note tagged with +a different domain’s scalar type is a validation error surfaced by +`+defmulti validate :type+` — the engine refuses to silently mix a fuzzy +0.7 with a DS mass function. Fuzzy is the default when no domain is +declared. + +==== 5.4 Incremental recompute + +For spinner drags, cache `+order+` and each node’s downstream-reachable +cone; on a driver change, re-sweep only the affected suffix. A full +sweep at ~10k nodes / ~30k edges is sub-millisecond in WASM; a +single-driver drag touches only its downstream cone. + +''''' + +=== 6. DEVONthink mapping + +==== 6.1 Capability → nexia mechanism + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|DEVONthink capability |nexia mechanism |Cost +|*See Also* |*`+index.see_also+` (concordance + TF-IDF + cosine kNN); +`+(see-also n k)+`; L0 "`Related notes`" panel* |*M (crown jewel)* + +|Classify |`+index.classify+` (Rocchio centroid; NB as a pack); +`+(classify n)+`; "`Suggested tags`" chips |M + +|Auto-Classify |`+auto-classify!+` — apply top class only if +`+top ≥ τ ∧ top−runner ≥ margin+`; Action-context, undoable, opt-in |S +(on Classify) + +|*Concordance / inverted index* |*`+core/src/index.rs::SimilarityIndex+` +— the one shared substrate, rebuilt like `+backlinks+`* |*M +(foundational)* + +|Smart Groups (dynamic saved search) |*already shipping* = +`+Agent { query }+`; enrich DSL (`+similar:+`, `+near:+`, `+conf:+`, +ranked) |free → S + +|Smart Rules — triggers |`+Notebook.triggers+`; host fires +`+TriggerEvent+` → predicate → λδ action; local-first subset |M + +|Smart Rules — actions |λδ Action builtins (`+set-attr!+`, +`+link-typed!+`, `+add-tag!+`, `+auto-classify!+`, `+create-note!+`) |S +(mostly exist) + +|Replicants |*free* — one `+Uuid+`; membership = which agents/tags +match; edits shared because one object |free + +|Duplicate detection |`+index.duplicates+` — blake3 exact + SimHash/LSH +near; `+(duplicates n)+` + panel |S (on index) + +|Tags + hierarchical groups |tags = `+attributes.tags+` (multi-valued) +via `+tag:+`/`+attr:+`; hierarchy = agents + prototype |S + +|Full-text search + operators/proximity |BM25 over the index; +boolean/phrase/proximity via positional postings; L0 stays +whitespace-ANDed |M + +|OCR / rich binary formats / RSS / web-clip / email / sync |*out* (§2) +|— + +|Optional generative AI |opt-in λδ host *capability* (Ollama/llama.cpp); +never default, no data leaves device |later, opt-in +|=== + +==== 6.2 The local intelligence engine + +One index, five features — the concrete pipeline: + +[source,rust] +---- +impl SimilarityIndex { + /// tf-idf weight = (1 + ln tf) · ln(N / df), idf against CURRENT df/N. + #[inline] fn w(&self, tf: u32, df: u32) -> f32 { + (1.0 + (tf as f32).ln()) * ((self.n_docs as f32) / (df as f32)).ln() + } + + /// See Also: top-k by cosine, self excluded. Walk only the query note's + /// top ~40 terms; accumulate partial dot-products over their postings. + pub fn see_also(&self, id: NoteId, k: usize) -> Vec<(NoteId, f32)> { + let d = self.doc_of[&id]; + let mut score: HashMap = HashMap::new(); + for &(t, tf) in top_terms(&self.forward[d as usize], 40) { + if (self.df[t as usize] as usize) > self.n_docs as usize / 2 { continue; } // freq-stop + let wq = self.w(tf, self.df[t as usize]); + for p in &self.postings[t as usize] { + if p.doc == d { continue; } + *score.entry(p.doc).or_default() += wq * self.w(p.tf, self.df[t as usize]); + } + } + let mut out: Vec<_> = score.into_iter() + .map(|(doc, dot)| (self.note_of[doc as usize], + dot / (self.doc_norm[d as usize] * self.doc_norm[doc as usize]))) + .collect(); + out.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0))); // deterministic tie-break (§9) + out.truncate(k); + out + } + + /// BM25 (k1≈1.2, b≈0.75) — replaces naive substring search, stands in for tantivy. + pub fn bm25(&self, query: &str, k1: f32, b: f32) -> Vec<(NoteId, f32)> { /* … */ } + + /// Rocchio nearest-centroid classify: cosine of note to each class centroid. + /// Cheaper and incrementally updatable vs multinomial NB (offered as a pack). + pub fn classify(&self, id: NoteId, centroids: &[(ClassId, SparseVec)]) -> Vec<(ClassId, f32)> { /* … */ } + + /// Duplicates: exact via blake3 buckets; near via SimHash Hamming ≤ 3, + /// candidates gathered by LSH bands (no O(n²) scan). + pub fn duplicates(&self, id: NoteId) -> Duplicates { /* … */ } +} +---- + +* *See-Also*: TF-IDF sparse vectors, cosine, top-k walking only the +query note’s ~40 heaviest terms with a frequency stop-list — +single-digit ms at 10k. +* *Classify / Auto-Classify*: each Agent (or tag value) is a class; +Rocchio centroid is the default (incrementally updatable), multinomial +naive-Bayes ships as a pack. Auto-Classify is the *confidence-gated +automation pattern*: apply only when +`+top ≥ threshold ∧ top − runner-up ≥ margin+`, else defer to the human +— routed through `+auto-classify!+` so it is undoable, budgeted, opt-in, +never silent. +* *Duplicates*: blake3 exact-hash buckets + 64-bit SimHash (fixed seed) +with LSH banding for near-dups; offer merge or "`these are the same +note`". +* *Auto-tag*: kNN label propagation over See-Also neighbours (weighted +vote above a threshold) — reuses the similarity engine, no separate +model. +* *Ranked search*: BM25 over the same index with positional postings for +phrase/proximity. The L0 whitespace-ANDed syntax is the default surface; +boolean/phrase/`+NEAR/n+` are progressive-disclosure power operators. + +==== 6.3 Smart Rules, Smart Groups, replicants + +* *Smart Groups = Agents.* Already shipping, 1:1. The only work is +enriching the DSL as the index lands (§3.6). +* *Smart Rules = triggered λδ actions.* `+TriggerEvent+` → +Agent-predicate condition → λδ Action body — DEVONthink’s +event→condition→action grammar, native to the homoiconic core. +* *Replicants are free.* A `+Note+` has one `+Uuid+`; "`membership`" is +which Agents match and which tags it carries. A single note already +appears in every matching Agent view and every tag facet, with edits +inherently shared because there is one object. Nothing is ever copied. + +==== 6.4 Positioning note (resolving the tantivy tension) + +The ROADMAP commits Phase 3/5 to *tantivy*. We consciously deviate: +tantivy gives BM25 but *not* See-Also cosine, Rocchio classify, or +SimHash dedup. Running tantivy _and_ a separate vector index means two +indices and more memory. We therefore *hand-roll one concordance for all +five features*, accepting the maintenance burden of our own +BM25/phrase/proximity implementation as the price of the shared +substrate — a hand-rolled index is comfortably sufficient at 10k notes. +This retires the "`tantivy planned but not built`" seam. We also +*reframe ROADMAP Phase 8 "`Intelligence`"*: See-Also/Classify are the +_deterministic local heart_ of that phase; any LLM is strictly optional +and opt-in, never the default — a positioning this document asks the +ROADMAP to adopt. + +''''' + +=== 7. Algorithms and performance + +==== 7.1 Concrete choices + +[width="100%",cols="50%,50%",options="header",] +|=== +|Concern |Choice +|Tokenisation |Unicode word segmentation → lowercase → stop-list → +optional Porter stem; deterministic, fixed data + +|Term weighting |TF-IDF `+(1+ln tf)·ln(N/df)+`; L2-normalised sparse +vectors + +|Similarity |cosine (dot of normalised vectors) + +|Search ranking |BM25, k1≈1.2, b≈0.75, positional postings for +phrase/proximity + +|Classify |Rocchio nearest-centroid (default); multinomial NB (pack) + +|Dedup |blake3 exact + 64-bit SimHash (fixed seed) + LSH bands + +|Propagation |edge-weight transform + native operator table; DFS +back-edge exclusion + Kahn topo; single pass + +|Layout |Sugiyama (longest-path layering → barycenter → Brandes–Köpf), +positions derived +|=== + +==== 7.2 Complexity and memory at 10k notes (~300 tokens/note, ~200 unique) + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Operation |Cost |Notes +|Full index build |O(Σ tokens) ≈ 3M |tens of ms; on load / import only + +|Incremental reindex (one edit) |O(tokens in note) |in +`+set_content+`/title path; imperceptible + +|See-Also query |O(top-terms × avg df) |40-term cap + freq-stop → +single-digit ms + +|BM25 search |O(Σ df of query terms) |ms + +|Classify (Rocchio) |O(terms(d) × #classes) |ms + +|All-pairs dedup |~O(N) via LSH bands |not O(N²) + +|Full propagation sweep |O(V+E) ≈ 40k ops |sub-ms in WASM + +|Spinner-drag recompute |O(downstream cone) |interactive + +|*Index memory* |*~35–50 MB* |inverted ≈16 MB + forward ≈16 MB + simhash +80 KB + norms 40 KB. *Requires dense-`+DocId+` interning* (postings +store 8-byte `+(u32,u32)+`, never 16-byte `+Uuid+`). Comfortable in a +browser tab. +|=== + +==== 7.3 Incremental update and the off-main-thread escape hatch + +All hot operations are incremental: reindex is per-note; propagation +re-sweeps only the downstream cone; layout runs on the reasoning +subgraph, not the whole notebook. The full build (load, import, or the +opportunistic norm-resetting rebuild) is the only O(corpus) operation, +and it is tens of ms. + +*Web Worker escape hatch.* Everything is pure, deterministic, +allocation-only Rust — no threads/network/clock/RNG (SimHash uses a +fixed seed) — so it slots into the wasm-bindgen core and honours the λδ +budget. If a very large notebook ever makes the initial build or a full +rebuild perceptible, the WASM core can be instantiated in a *Web +Worker*: the TEA loop posts the notebook JSON in, receives the built +indices out, and the main thread never blocks. This is a deployment +option, not an architectural change — the core code is identical either +way. + +''''' + +=== 8. Phased roadmap + +Aligned to the λδ L0–L4 progressive-disclosure levels and the numbered +ROADMAP phases. Each phase is independently shippable; L0 UX (no +parentheses, ever) is untouched throughout; new state is either +additive-and-skippable or `+#[serde(skip)]+`-derived. + +==== The recommended FIRST PR — headless `+SimilarityIndex+` + +*Smallest, headless, highest-value.* Build `+core/src/index.rs+` + +tokenizer + the incremental hook in `+set_content+` *and both title +write paths* + `+rebuild_indices+`. No WASM entry points, no UI. Unit- +and property-tested against the existing `+golden.rs+` / +`+invariants.rs+` fixtures, including the byte-identical round-trip +proof (the index is `+#[serde(skip)]+`, so this is automatic) and a +determinism test (sorted iteration, fixed-seed SimHash). This de-risks +the entire DEVONthink side, fills the "`tantivy planned but not built`" +seam, and lands zero user-visible change. It mirrors the L0-substrate +discipline ADR-0003 used for λδ. + +==== Phase map + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|Level / ROADMAP |Ships |New state |User-visible +|*First PR* (ROADMAP Phase 3) |headless `+index.rs+`, tokenizer, +incremental hooks, `+rebuild_indices+` |`+#[serde(skip)]+` index |none + +|*L0* (ROADMAP Phase 3 + differentiator) |See-Also / Suggested-tags / +Duplicates panels (mount only when non-empty); BM25 ranked search +replaces substring; `+Edge+` + `+reason.rs+` + confidence spinner in the +editor; `+ReasoningView+` (`+layout.rs+`) with driver/driven shading and +dashed back-edges |`+edges+` field (empty by default) |Related panel; +ranked search; spinners; reasoning canvas — all no-code + +|*L1* (ROADMAP Phase 4) |pure builtins in fx fields: +`+(see-also self k)+`, `+(classify self)+`, +`+(:confidence (propagate))+`; Agent DSL gains +`+similar:+`/`+near:+`/`+conf:+`/`+type:+`/`+edge:+` |— |computed +confidence & similarity in formulas + +|*L2* (ROADMAP Phase 4/9) |`+(propagate!)+`, `+(link-typed!)+`, +`+(junct!)+`, `+(auto-classify!)+`, `+deftrigger+`; +`+combine+`/`+render+`/`+validate+`/`+edge-style+` multimethods; +`+Notebook.triggers+`; ship `+flying-logic.ld+` + `+librarian.ld+` +|`+triggers+` collection (empty by default) |Smart Rules; domain packs + +|*L3* |REPL exploration of `+(propagate)+`, `+(see-also …)+`, +`+(duplicates …)+`, `+(back-edges)+` against the live graph under budget +|— |power console + +|*L4* (ROADMAP Phase 7 Views + Phase 9 Ecosystem) |computational notes +rendering See-Also / propagation inline; layout as a `+defview+`; TOC / +probability / DS-lite domain packs via the plugin system (#33); opt-in +local-LLM host capability |— |shareable packages +|=== + +*ROADMAP crosswalk.* Phase 3 Search → the concordance + BM25 (retires +tantivy). Phase 4 Agents → Smart Groups (exist) + Smart Rules +(triggers). Phase 7 Views → layered layout replacing the GraphView +placeholder. Phase 8 Intelligence → See-Also/Classify are the +deterministic local heart; LLM strictly optional. Phase 9 Ecosystem → +both tools as `+.ld+` domain packages. + +*Net new native surface:* one struct + one `+Note+` field, plus small +modules `+index.rs+`, `+edge.rs+`, `+reason.rs+`, `+trigger.rs+`, and +later `+layout.rs+`. Everything a Flying Logic or DEVONthink user +recognises as _intelligence_ — operators, domains, methodologies, +classification policy, organisation rules, reasoning views — ships as λδ +multimethods, `+.ld+` packages, and Agents on top. + +''''' + +=== 9. Risks and open questions + +*Resolved in this design (documented so the resolution is auditable):* + +* *Determinism under the λδ sandbox.* DFS back-edge selection and any +HashMap iteration in propagation/index must be order-stable, or golden +tests flake and the no-random contract is violated. _Resolution:_ the +`+ReasonGraph+` sorts nodes and adjacency lists before DFS/Kahn; +See-Also/search sort with an id tie-break (`+then(a.0.cmp(&b.0))+`); +SimHash uses a fixed seed; tokenisation is pure. A determinism property +test guards all four. +* *Cycle policy on a graph that had no acyclic invariant.* `+links+` +never required acyclicity and `+link_notes+` only rejects self-links. +_Resolution:_ the _new_ `+edges+` channel defines its own policy — +DFS-detected back edges are excluded from flow (single pass, no +fixpoint), drawn dashed, and surfaced via a "`you made a loop`" cue. +`+links+` is unaffected. +* *Native/λδ operator seam.* A conformance test asserts the native +`+Op+` table and the λδ `+combine+` methods agree for shipped ops, so +crossing the seam never changes semantics; unknown ops go through λδ and +are accepted as slower (§4.4). +* *Back-compat migration.* Empty-`+Vec+` + `+skip_serializing_if+` +gives a provable byte-identical round-trip on every golden fixture — the +safest of the three considered approaches, and far safer than an +untagged `+Link+` enum on the load-bearing field. +* *Index memory at 10k.* Mandated dense-`+DocId+` interning holds it to +~35–50 MB (§7.2). +* *Incremental consistency.* Title paths are hooked; idf/norm drift is +bounded to a session by `+#[serde(skip)]+` rebuild-on-load, with idf +computed against current `+df+`/`+N+` at query time (§3.5). + +*Genuinely open:* + +[arabic] +. *When to trigger the opportunistic full rebuild.* On-open is free; do +we also rebuild after N edits, or only when a norm-drift heuristic +exceeds a threshold? Needs measurement on real notebooks. +. *"`Promote this association to a typed edge`" UX.* The clean split +means a wikilink can never also be causal; a user who wants both must +create a typed edge. The affordance must avoid silent duplication (the +same relation appearing once as a link, once as an edge). Design needed. +. *Classify class-source ambiguity.* Agents _and_ tag values can both be +"`classes`". Which is the default training signal, and how do we present +two overlapping class systems without confusing the user? +. *Stemming/stop-list quality vs determinism across locales.* Porter +stemming and English stop-lists markedly improve similarity but are +language-specific; the multilingual story (and whether stemming is a +per-notebook toggle) is unspecified. +. *DS-lite domain semantics.* Offering a belief/plausibility interval as +an optional domain (§5.4) needs its own operator table and validation +rules; the interaction with the native fast path is sketched but not +specified. +. *Web Worker boundary.* If we adopt the off-main-thread build (§7.3), +the serialization cost of shuttling a large notebook to the worker must +be measured against the main-thread block it avoids. +. *Trigger execution ordering and loops.* When one trigger’s action +fires an event another trigger listens for, we need a documented +depth/reentrancy limit (the `+Budget+` bounds a single action, not a +cascade). + +''''' + +_Steal the intelligence and the primitives, not the feature sprawl. Keep +the note model untouched, the semantics as data, the hot path native, +the disclosure progressive, and everything on-device. The letter still +reaches a richer future self — now able to reason over its own claims +and recall its own forgotten neighbours._ diff --git a/docs/design/flyinglogic-devonthink-integration.md b/docs/design/flyinglogic-devonthink-integration.md deleted file mode 100644 index 44a4783..0000000 --- a/docs/design/flyinglogic-devonthink-integration.md +++ /dev/null @@ -1,666 +0,0 @@ - -# Flying Logic × DEVONthink → nexia-list: The Definitive Integration Design - -*Status: proposed · Supersedes: three exploratory designs (crown-jewels-first, product-minimalist, substrate-maximalist) and their adversarial review · Audience: nexia-list core contributors · Companion to: ADR-0003 (λδ), the λδ spec v0.1, and the ROADMAP.* - -> A note is a letter we send to our future self. This document is about giving that letter two new senses: the ability to **recall its own forgotten neighbours**, and the ability to **reason over its own claims** — both entirely on-device, both invisible until asked for. - ---- - -## 1. TL;DR and the Unifying Thesis - -### 1.1 TL;DR - -We steal the **intelligence and primitives** of Flying Logic and DEVONthink, not their feature sprawl. Two crown jewels justify genuinely new code in the Rust core; everything else is either already free in nexia's model, or ships later as `.ld` domain packages and Agent-DSL sugar. - -| From | Crown jewel | Becomes | New native code | -|---|---|---|---| -| **DEVONthink** | *See Also* — local associative recall | One concordance / inverted index → cosine kNN, BM25 search, Rocchio classify, SimHash dedup, auto-tag | `core/src/index.rs` | -| **Flying Logic** | *Live confidence propagation* — a spreadsheet for reasoning | One pure single-pass DAG sweep over a **separate, additive** typed-edge channel; junctors are ordinary `:type`-tagged notes; operators are a native table a domain pack extends | `core/src/edge.rs`, `core/src/reason.rs`, later `core/src/layout.rs` | - -Both are **derived, rebuildable indices that live beside the notes and are never trusted from disk** — the exact discipline nexia already proved with `Notebook::backlinks`. Both surface as **no-parenthesis L0 panels that mount only when non-empty**, and both are reachable as **pure λδ builtins** for power users. Neither is a monolithic subsystem: each is a Rust module + a handful of host builtins + a derived view. - -The recommended **first PR** is headless: `core/src/index.rs` alone, property-tested against the existing golden fixtures, with zero UI and zero on-disk format change (§8). - -### 1.2 The Unifying Thesis - -Both external tools collapse onto **one spine**: - -> **A typed, attribute-rich graph + a local intelligence engine + λδ programs — all local-first, all derived-and-rebuildable, all invisible by default.** - -The move that unifies them is the move nexia already made for `backlinks`: - -1. **A derived index lives beside the notes** and is rebuilt on load, never authoritative on disk (the concordance for DEVONthink; the confidence map and the layered layout for Flying Logic). -2. **Semantics are data, dispatched on `:type`/`:op`** through λδ multimethods (`defmulti … :type`), so operators, entity classes and whole methodologies ship as `.ld` packages without recompiling the core. -3. **The note model is untouched.** `Note.links` stays the untyped associative fabric. A parallel, empty-by-default `edges` channel carries implication. Existing notebooks round-trip byte-for-byte. -4. **We keep the intelligence, not the interpreter tax.** A native operator table is the fast default for the propagation hot path; the λδ `combine` multimethod is the extension path for exotic domain operators only (never called per-node in the inner loop). -5. **Progressive disclosure holds the line.** Typed reasoning is opt-in at L1+, never imposed. Flying Logic's mandatory entity-class discipline is the explicit anti-pattern. - -The result: the letter still reaches a richer future self — now able to reason over its own claims and recall its own forgotten neighbours, entirely on-device. - ---- - -## 2. What we are NOT importing (honest non-goals) - -These exclusions are load-bearing, not incidental. "Do fewer things well" is a design constraint we enforce, and several of these were flagged by the review as active hazards to avoid. - -### 2.1 Categorically out of charter (never build) - -| Excluded | Reason | -|---|---| -| **OCR** (image/PDF → text) | Requires Tesseract-class engines and binary pipelines. Content is plain text; Markdown rendering is the only planned rich step. | -| **RSS / feeds, web clipper / Sorter, email import** | Each needs network fetch, OS integration, or proprietary binary parsing — breaks no-network-by-default. Manual Markdown paste is the only sympathetic path. | -| **Multi-device / cloud sync, Server/Sharing edition** | Explicit ROADMAP non-goals (no cloud-only storage, no real-time collaboration). Persistence stays IndexedDB + file import/export; users sync exported JSON/vault themselves. | -| **Rich binary document handling** (PDF annotation, RTFD, web archives) and format-conversion actions | Different product category, heavy WASM burden. The in-scope sliver is Markdown + the existing `exchange.rs` vault. | -| **PDF / PNG / MS-Project diagram export** | Heavyweight rendering plus PM-suite interop, against local-first / no-lock-in. Markdown + OPML already cover interop; at most a client-side SVG snapshot of the layout view. | -| **Mandatory generative AI** (chat, DALL·E, image description), **macOS side-effects** (Speak / Bounce / SendMail), **AppleScript/JXA host** | The λδ sandbox is deliberately no-I/O. λδ (homoiconic, budgeted, in-core) is the scripting escape hatch, not an OS bridge. Local models (Ollama/llama.cpp) are opt-in host capabilities only; no data leaves the device by default. | - -### 2.2 In-scope in spirit, but deliberately not built as designed - -| Excluded form | What we do instead | -|---|---| -| **Promoting `Note.links: Vec` → `Vec`** (typed links in the load-bearing field) | Verified to touch ~6 sites (`notebook.rs:75/105/260` backlink loops, wikilink derivation, the `linksto:` agent term, `NoteView.links`, `note_to_value`) and to merge L0 associations with causal edges into one vector. We add a **separate `edges: Vec` channel** instead — same capability, near-zero blast radius, byte-identical round-trip. | -| **Per-node λδ multimethod dispatch inside the propagation sweep** | Each `i.apply` decrements the 1M-step `Budget` and readers hold `nb.borrow()`, so a `combine` method touching any mutator panics via `RefCell` double-borrow — and it is materially slower at 10k nodes on every spinner drag. We keep a **native operator table** for the hot path; λδ `combine` is reserved for exotic/domain ops (§4.4). | -| **The six TOC Thinking-Process templates** (CRT, Evaporating Cloud, FRT, PRT, Transition, S&T) as engine features | They are curated domains + starter graphs. Shipping them in core imposes one methodology and violates untyped-by-default. Ship as optional `.ld` domain packs + starter notebooks (data). | -| **Full probability/arithmetic operator sprawl** (Product ×, Sum-Prob ⊕, Proportion ∷, float-flow typing) on day one | Fuzzy min/max/complement + edge weights cover ~80% of reasoning. The rest is trivially added later as `combine` methods; there is no cost to waiting. | -| **A full Dempster–Shafer belief engine** | General DS combination is O(2^\|frame\|) per node with power-set mass functions and conflict renormalisation — non-interactive at 10k. A **DS-lite belief/plausibility interval** is offered as an *optional domain* only (§5.4). | -| **Exotic confidence math as the default** | The default scalar is fuzzy-boolean in [0,1] with 0.5 = Indeterminate. Probability and DS-lite are per-domain opt-ins, never mixed inside one graph (§5.4). | -| **Persisting the concordance or propagation results to disk as authoritative state** | Bloats the human-readable JSON and risks drift. Both are rebuilt on load, exactly like `backlinks`. Derived state is never trusted from disk. | -| **Animated incremental layout, presentation/step-through mode, typed-attribute schemas as a prerequisite** | Pure polish or decoupled workstreams. The static layered view + spinners already deliver the capability; `:confidence`/`:op`/`:weight` ride the existing untyped JSON attrs fine. | - ---- - -## 3. Data-model deltas - -The whole delta is: **one new struct + one new field on `Note`**, **three derived indices + a triggers collection on `Notebook`**, and **a few new Agent-DSL predicates**. Everything else rides attributes and `:type`, which the host already surfaces. - -### 3.1 The `Edge` struct — a separate, additive channel - -`Note.links` (untyped association: wikilinks, backlinks, the L0 graph) **stays exactly as it is** and does **not** participate in confidence flow. Implication lives on a new, empty-by-default field so that a notebook with no reasoning graph serialises identically to today. - -```rust -// core/src/edge.rs (new) -use crate::note::NoteId; -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; - -/// A typed, weighted, directed implication edge — where Flying Logic's logic lives. -/// `kind` is a bare string so domains add edge classes as DATA, never as a core enum. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] -pub struct Edge { - pub to: NoteId, - - /// Edge class / dispatch tag: "implies", "supports", "inhibits", "feeds", - /// "contains", … Default "link" ⇒ a typed edge with no ceremony. - #[serde(default = "Edge::plain_kind", skip_serializing_if = "Edge::is_plain")] - pub kind: String, - - /// Influence weight in [-1, 1]. +1 passes through, 0 → Indeterminate (0.5), - /// -1 negates (Flying Logic's weighting transform). Default +1. - #[serde(default = "Edge::unit", skip_serializing_if = "Edge::is_unit")] - pub weight: f64, - - #[serde(default, skip_serializing_if = "HashMap::is_empty")] - pub attrs: HashMap, -} - -impl Edge { - fn plain_kind() -> String { "link".into() } - fn is_plain(k: &str) -> bool { k == "link" } - fn unit() -> f64 { 1.0 } - fn is_unit(w: &f64) -> bool { (*w - 1.0).abs() < f64::EPSILON } -} -``` - -```rust -// core/src/note.rs — ONE new field on `Note`, empty for every existing note - /// Typed/weighted implication edges (the reasoning graph). Distinct from - /// `links` (untyped association). Empty ⇒ the note is invisible to the - /// reasoning engine and serialises to nothing new. - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub edges: Vec, -``` - -**Why this beats the untagged `Link` enum.** An untagged serde shim on the load-bearing `links` field is the highest-risk round-trip against the golden fixtures (untagged ambiguity, poor error messages). An empty-by-default `Vec` guarded by `skip_serializing_if = "Vec::is_empty"` is provably byte-identical for every existing notebook: no new key appears until a note actually has an edge. `Note::new` initialises `edges: Vec::new()`. - -**Back-compat proof obligation (Phase gate).** The golden-fixture suite (`golden.rs`) must show that every existing fixture serialises to the same bytes after this field is added. This is a mechanical, checkable gate — not a hope. - -### 3.2 Junct / operator / entity / group notes — zero new node primitive - -The λδ host already derives `:type` from `attributes["type"]`, and `note_matches` already supports `attr:type=junct`. So junctors, entity classes and groups are **ordinary notes distinguished by attributes** — no schema work. - -```jsonc -// An AND-junct as it sits in the notebook JSON — nothing new in the format -{ - "id": "…", "title": "AND", - "attributes": { "type": "junct", "op": "and" }, - "edges": [ { "to": "", "kind": "feeds" } ] -} -``` - -| Concept | Encoding | Surfaced as | -|---|---|---| -| Entity / claim node | `attributes.type = "claim"` (or a domain class) | `:type` | -| Combining operator | `attributes.op = "and" \| "or" \| "not" \| …` | `:op` | -| Asserted driver confidence | `attributes.confidence = 0.8` (f64 ∈ [0,1]) | `:confidence`; editor spinner | -| Entity/edge styling | `attributes.type` / `Edge.kind` | `render` / `edge-style` multimethod | -| Group / collapsible subgraph | `attributes.type = "group"`, members via `:contains` edges | deferred (§8, L4) | - -Computed confidences on **driven** nodes are **never persisted** — they would masquerade as asserted drivers and bloat the JSON. Propagation is derived, exactly like `backlinks`. - -### 3.3 The concordance / similarity index — the one shared substrate - -Rebuildable and incremental, modelled on `backlinks`, but marked `#[serde(skip)]` and **always rebuilt on load** — the strongest possible "don't destabilise L0" guarantee: zero on-disk format change. - -```rust -// core/src/index.rs (new) — powers See-Also, Classify, dedup, and BM25 search -use crate::note::NoteId; -use std::collections::HashMap; - -type TermId = u32; -type DocId = u32; // DENSE note index — the decisive WASM-memory move (§7) - -#[derive(Clone, Copy)] -struct Posting { doc: DocId, tf: u32 } - -#[derive(Default)] -pub struct SimilarityIndex { - interner: HashMap, TermId>, - vocab: Vec>, - - // note ↔ dense id — postings store an 8-byte (DocId, tf), NEVER a 16-byte Uuid - doc_of: HashMap, - note_of: Vec, - - postings: Vec>, // TermId → postings, sorted by doc - df: Vec, // document frequency per term - - forward: Vec>, // DocId → (term, tf) vector, for reindex + norms - doc_len: Vec, // token count per doc (BM25 |d|) - doc_norm: Vec, // cached L2 norm of the tf-idf vector - - simhash: Vec, // near-dup fingerprint, FIXED seed - content_hash: HashMap<[u8; 32], Vec>, // exact-dup buckets (blake3) - - n_docs: u32, - total_len: u64, // for avgdl -} -``` - -**The dense-`DocId` rule is mandatory, not an optimisation.** Postings must store `(u32, u32)`, never the 16-byte `Uuid`. This is what keeps the index near ~35–50 MB at 10k notes (§7) — comfortable in a browser tab. - -Tokenisation is deterministic (Unicode word segmentation → lowercase → stop-list → optional Porter stem), with a **fixed seed** for SimHash and no clock/RNG — honouring the λδ sandbox and IndexedDB determinism. Crates are Rust-only via cargo (`unicode-segmentation`, `rust-stemmers`, `blake3`), so the Bun-only-for-JS MUST is untouched; BM25/cosine/NB are hand-rolled. A hand-rolled ASCII tokenizer + static stop-list is the zero-dep fallback if WASM size ever matters. - -### 3.4 `Notebook` deltas - -```rust -// core/src/notebook.rs -pub struct Notebook { - notes: HashMap, - - #[serde(default)] - backlinks: HashMap>, // unchanged (plain links) - - /// Reverse index for TYPED edges, rebuilt exactly like `backlinks`. - #[serde(default)] - in_edges: HashMap>, // (source note, edge index) - - /// The concordance. DERIVED — never persisted, rebuilt on load. - #[serde(skip)] - index: crate::index::SimilarityIndex, - - #[serde(default)] - agents: Vec, - - /// Smart-Rule triggers (event → predicate → λδ action). Empty by default. - #[serde(default, skip_serializing_if = "Vec::is_empty")] - triggers: Vec, - - pub name: String, - pub created_at: DateTime, - pub modified_at: DateTime, -} -``` - -`rebuild_backlinks` becomes `rebuild_indices` (keeping the old name as a thin caller): it rebuilds `backlinks`, `in_edges`, **and** `index` from the notes. `WasmNotebook::from_json` already calls the rebuild path, so all three are populated on every load. - -### 3.5 Incremental maintenance — hook the write paths, including the title path - -`set_content` is the primary write path and already re-derives wikilinks per edit. Reindexing hangs off it: - -```rust -pub fn set_content(&mut self, id: &NoteId, content: impl Into) -> Vec { - let content = content.into(); - // … existing wikilink logic … - self.index.reindex(*id, &self.title_of(id), &content); // NEW - // … -} -``` - -`reindex` diffs the note's previous forward vector against the new token bag — decrement `df` and remove postings for dropped terms, add postings for new terms, recompute this doc's `doc_len`, `doc_norm`, and `simhash`. Cost is **O(tokens in the edited note)**, not the corpus. - -**Resolving the title-path desync (open tension).** Titles dominate short-note similarity, yet `update_title` / `bi_set_title` bypass `set_content`. Both title write paths **must** call `index.reindex(id, new_title, current_content)` too. This is a required part of the first PR's contract, not a follow-up. - -**Resolving global-idf / norm drift (open tension).** Every edit changes `df`/`N`, which technically shifts the global idf and thus every *other* note's L2 norm. We do not chase this per-edit. Instead: -- `doc_norm` is cached per note and recomputed only for the edited note on `reindex`. -- See-Also and BM25 compute idf against the **current** `df`/`N` at query time, so ranking is always consistent with the live corpus even though cached norms lag slightly. The lag affects only the *magnitude* of the cosine denominator for unedited notes, never their term content, and `#[serde(skip)]` + rebuild-on-load bounds any accumulated drift to a single session. -- A cheap periodic/opportunistic full `rebuild_indices` (e.g. on notebook open, already free) resets norms exactly. We accept bounded intra-session staleness as the honest, documented trade — it is imperceptible in top-k ordering and it keeps edits O(tokens). - -### 3.6 Agent-DSL extensions (DEVONthink Smart Groups already == Agents) - -`Agent { query }` **is** a Smart Group, 1:1, already shipping — and replicants are free (§6). We only enrich the parser: - -```rust -// core/src/agent.rs — extend `enum Term` -enum Term { - Text(String), Title(String), Attr(String, String), LinksTo(NoteId), Never, - Type(String), // type:junct — surfaces the FL entity class - Similar(NoteId), // similar: — DEVONthink See-Also as a saved query - Near(NoteId), // near: — cosine ≥ τ (near-duplicate) - Conf(Ordering, f64), // conf:>0.7 — filter by propagated confidence - Edge(String), // edge:supports — has an outgoing typed edge of kind -} -``` - -`similar:`/`near:`/`conf:` need the index or the propagation result, which the pure `note_matches` (which has no notebook context) cannot supply. **Resolution:** add a notebook-aware sibling `note_matches_in(note, query, &Notebook)` used by `run_query`; the pure `note_matches` stays for index-free terms. The whitespace-ANDed L0 surface is unchanged; these are progressive-disclosure power operators. - -### 3.7 Triggers (Smart Rules) — a third `Notebook` collection, like `agents` - -```rust -// core/src/trigger.rs (new) -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct Trigger { - pub id: Uuid, - pub event: TriggerEvent, - pub condition: String, // an Agent-DSL predicate (reuses note_matches) - pub action: String, // a λδ Action-context program - #[serde(default = "yes")] pub enabled: bool, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(rename_all = "kebab-case")] -pub enum TriggerEvent { - OnCreate, OnSetContent, OnSetAttr, OnLink, OnTag, OnRunAgent, - Scheduled { every: String }, // "hourly" | "daily" | "workdays" — host-driven tick -} -``` - -The **local-first subset** of DEVONthink's event taxonomy. Networked events (OnDownload/OnScan/OnOCR/OnSync) are dropped. The host fires a matching trigger's `action` in Action context under a fresh `Budget`. Triggers travel with the notebook, like agents. - ---- - -## 4. The λδ layer - -The kernel already implements `defmulti`/`defmethod`, `register_builtin`, a `PRELUDE` eval'd on `Interp::new()`, and `note_to_value` deriving `:type`. We extend the bridge, add builtins, and register a small operator prelude. - -### 4.1 Bridge additions (`note_to_value`) - -Keep the bridge the single canonical translator; add keys additively (`:links` unchanged): - -```rust -// lambdadelta_host.rs :: note_to_value — additive pairs -(Value::kw("edges"), edges_value(¬e.edges)), // vector of {:to :kind :weight :attrs} -(Value::kw("confidence"), num_or_nil(note.attributes.get("confidence"))), -(Value::kw("op"), note.attributes.get("op").map(json_to_value).unwrap_or(Value::Nil)), -``` - -### 4.2 New builtins - -**Pure readers** (legal in Formula / Agent-predicate / Agent-query contexts — deterministic, budgeted, no mutation): - -| Builtin | Returns | Backed by | -|---|---|---| -| `(see-also n)` / `(see-also n k)` | top-k similar note maps | `index.see_also` | -| `(similar? a b)` | cosine ∈ [0,1] | `index.cosine` | -| `(classify n)` | ranked `[class score]` | Rocchio centroid | -| `(duplicates n)` | exact + near duplicates of `n` | blake3 + SimHash | -| `(search-ranked q)` | notes ranked by BM25 | `index.bm25` | -| `(confidence n)` | asserted-or-computed truth of `n` in [0,1] | `reason::propagate` (memoised per eval) | -| `(propagate)` | map `{#uuid → float}` over driven notes | `reason::propagate` | -| `(drivers)` / `(driven)` | notes with no / some in-edges | reasoning graph | -| `(edges n)` / `(in-edges n)` | typed edges of / into `n` | `Note.edges` / `in_edges` | - -**`!`-mutators** (Action context only — budgeted, undoable): - -| Builtin | Effect | -|---|---| -| `(set-confidence! id v)` | assert a driver's `:confidence` | -| `(link-typed! from to kind weight)` | push an `Edge` | -| `(junct! op & operand-ids)` → junct id | create `{:type :junct :op op}`, wire operands → junct | -| `(add-tag! id t)` / `(remove-tag! id t)` | maintain `attributes.tags` (a Vec) | -| `(auto-classify! id thresh margin)` | apply top class **only if** `top ≥ thresh ∧ top − runner-up ≥ margin` | - -Readers register via `reader(…)`, mutators via `mutator(…)`, sharing the existing `Rc>` — line-for-line with the current host bindings. - -### 4.3 Evaluation contexts - -| Context | Bound | May mutate? | New usage | -|---|---|---|---| -| Formula (pure, `self`) | `self` | no | `(:confidence (propagate))`, `(see-also self 5)`, `(classify self)` | -| Agent-predicate (`self`, truthy) | `self` | no | `similar:`/`conf:`/`type:` compile here | -| Agent-query (→ notes) | — | no | `(filter #(> (first-score (see-also % 1)) 0.6) (notes))` | -| Action (`self`, mutating) | `self` | yes | `(propagate!)`, `(auto-classify! …)`, `(link-typed! …)` | -| **Trigger-action** *(new)* | `self`, **`event`** | yes | Smart-Rule bodies; `self` = triggering note | - -The existing L0 agent DSL still compiles to an Agent-predicate λδ expression. `see-also`/`classify`/`propagate` are pure and legal in fx fields (L1); `auto-classify!` and trigger bodies are Action-context, so a probabilistic model can only touch data through the same audited, undoable seam as any mutator. - -### 4.4 Multimethod dispatch — pluggable semantics, with a precise native/λδ seam - -Three multimethods carry the transferred intelligence and make domains pluggable as `.ld` packages: - -```clojure -;; operator algebra — the FL crown jewel's SEMANTICS, extensible without core edits -(defmulti combine (fn [op _inputs] op)) -(defmethod combine :and [_ xs] (reduce min 1.0 xs)) ; weakest link -(defmethod combine :or [_ xs] (reduce max 0.0 xs)) -(defmethod combine :not [_ xs] (- 1.0 (first xs))) -(defmethod combine :default [_ xs] (reduce min 1.0 xs)) ; entity default = AND-ish - -;; rendering — one card per entity/edge class -(defmulti render :type) -(defmethod render :junct [n] {:shape :diamond :label (:op n)}) -(defmethod render :claim [n] {:shape :rounded-box :spinner (:confidence n)}) - -;; edge styling / domain validation -(defmulti edge-style (fn [e] (:kind e))) -(defmulti validate :type) -``` - -**The seam contract (resolving the open tension explicitly).** The propagation sweep in `reason.rs` runs a **native `Op` table** in Rust for the known operators `{:and :or :not}` (and, once shipped, the probability pack's ops). It calls the λδ `combine` multimethod **only** when a node's `:op` is not in the native table. The contract: - -1. **Native is the source of truth for shipped ops.** `combine`'s λδ methods for `:and/:or/:not` exist for REPL inspection and must be *definitionally identical* to the native table (min / max / 1−x). A conformance test asserts native and λδ agree on a sample grid, so crossing the seam never silently changes semantics. -2. **λδ `combine` is never called per-node for a known op.** This is what avoids the verified `Budget` drain and `RefCell` re-borrow panic, and keeps 10k-node spinner drags interactive. -3. **A domain pack adds an op by adding a `combine` defmethod.** Unknown ops resolve through λδ and are accepted as slower; if a domain wants its op on the fast path, it registers a native op id via the host capability (a documented, reviewed extension point), not by patching the evaluator. - -This makes the native table a **floor, not a ceiling**: fast by default, infinitely extensible through data. - -### 4.5 Domains as `.ld` packages - -A *domain* is a `.ld` package that registers a set of `:type`/`:op` tags plus their `combine` / `render` / `validate` / `edge-style` methods and a suggested attribute vocabulary — aligning 1:1 with the kernel/host + plugin-package design (issue #33). Flying Logic's "domain = swappable class+operator pack" and DEVONthink's Classify-by-corpus both become data, not core enums. Shipped examples (§8, L2/L4): `flying-logic.ld` (junct/claim/group types + operator algebra + render cards), `librarian.ld` (confidence-gated auto-classify trigger + kNN auto-tag), and later the TOC methodology packs and a probability operator pack. - ---- - -## 5. Flying Logic mapping - -### 5.1 Capability → nexia mechanism - -| Flying Logic capability | nexia mechanism | Cost | -|---|---|---| -| Entity = statement node | Existing `Note`; class = `attributes.type` → `:type` | free | -| Directed causal/logical edge | New `Note.edges: Vec { kind, weight, attrs }` | S (struct) | -| Junctor (AND/OR bundler) | A note `{:type :junct :op …}`; operands → junct → target via typed edges | free (data) | -| Necessary vs sufficient | `:op :and` = min, `:op :or` = max, selected in `combine` | free | -| Weighted / negated edge | `Edge.weight ∈ [-1,1]` + the weighting transform (w=0→0.5, w<0→negation) | free | -| Confidence scalar | `attributes.confidence: f64`; 0.5 = Indeterminate midpoint | free | -| **Live confidence propagation** | **`reason::propagate` — single-pass DAG sweep; `(confidence n)` / `(propagate)`** | **M (crown jewel)** | -| Advanced operators (XOR/⊕/×/∷) | native `Op` table (hot path) + `defmethod combine :op` for domain ops | S core, ∞ via packs | -| Domains (class + operator packs) | `.ld` packages registering `:type`/`:op` methods + attr vocab | pack (data) | -| Entity classes & styling | `:type` + `defmethod render :type`; prototype supplies class defaults | S | -| TOC thinking-process templates | **out-of-core** domain packs + starter notebooks (§2) | pack | -| Automatic layered DAG layout | new `core/src/layout.rs` (Sugiyama) → `ReasoningView` | M | -| Animated incremental layout | **deferred** — ReScript interpolation only | — | -| Groups / collapse | `{:type :group}` + `:contains` edges; layout/propagate as one node | M (later) | -| Back-edge cycle handling | DFS marks & excludes back edges; drives a "you made a loop" cue | free (in M) | -| Confidence spinners (live) | `SetConfidence` msg → `set-confidence!` → re-`propagate` → re-render | S | -| Fuzzy vs float flow typing | `:flow` tag on computed value; **deferred** | — | -| Operators inspector | settings surface writing `:op` (per-note + notebook default) | S | -| PDF/PNG/MS-Project export | **out** (§2); at most a client-side SVG snapshot | — | - -### 5.2 The reasoning-graph view (layered DAG layout) - -`core/src/layout.rs` runs a compact Sugiyama pipeline over the typed-edge graph and returns **positions that are computed, never stored** — `Note.position` stays owned by the manual spatial canvas, so auto-layout and free-canvas coexist: - -1. Cycle removal — reuse `reason.rs`'s back-edge set (temporarily reverse those edges). -2. Layer assignment — longest-path layering; insert dummy nodes on edges spanning >1 layer. -3. Crossing minimisation — median/barycenter, a few up/down sweeps. -4. x-coordinate assignment — Brandes–Köpf for straight, balanced edges; y from layer index; honour an orientation flag. - -Exposed as `reasoning_layout(orient) -> [{id, x, y, layer}]`. It replaces the `GraphView` placeholder and reuses `GraphLayout.res`'s `nodePos` seam — the ReScript view only draws; the layout comes from Rust. Animated transitions are deferred polish. - -### 5.3 Confidence propagation — the chosen default semantics - -```rust -// core/src/reason.rs -pub type Conf = f32; // [0,1]; f32 halves map memory - -/// Flying Logic's documented weighting transform, applied per in-edge -/// BEFORE the operator: w=+1 pass-through, w=0 → 0.5, w=-1 → negation. -#[inline] -fn weight_edge(v: Conf, w: f32) -> Conf { - let s = 2.0 * v - 1.0; // [0,1] → [-1,1] - (s * w + 1.0) * 0.5 // → [0,1] -} - -#[derive(Clone, Copy)] -pub enum Op { And, Or, Not, /* pack: */ Xor, Product, SumProb, Proportion } - -/// Combine already-weighted inputs. Empty input set ⇒ 0.5 (Indeterminate). -fn combine(op: Op, xw: &[(Conf, f32)]) -> Conf { - if xw.is_empty() { return 0.5; } - match op { - Op::And => xw.iter().map(|p| p.0).fold(f32::INFINITY, f32::min), // weakest link - Op::Or => xw.iter().map(|p| p.0).fold(f32::NEG_INFINITY, f32::max), - Op::Not => 1.0 - xw[0].0, - Op::Product => xw.iter().map(|p| p.0).product(), - Op::SumProb => xw.iter().fold(0.0, |a, &(b, _)| a + b - a * b), // 0.5⊕0.5=0.75 - Op::Proportion => { // zero-weight abstains - let (num, den) = xw.iter().fold((0.0, 0.0), |(n, d), &(x, w)| { - let a = w.abs(); (n + a * x, d + a) - }); - if den == 0.0 { 0.5 } else { num / den } - } - Op::Xor => fuzzy_xor(xw), // max_i min(x_i, 1 - max_{j≠i} x_j) - } -} - -/// ONE deterministic pass. (1) DFS marks back edges (edges to a node on the -/// stack) and drops them. (2) Kahn topo over the DAG. (3) Sweep: drivers keep -/// their asserted value; driven nodes combine weighted inputs. O(V+E), no fixpoint. -pub fn propagate(nb: &Notebook) -> HashMap { - let g = ReasonGraph::build(nb); // typed edges only, nodes/edges SORTED (§9) - let back = g.back_edges(); // DFS over a sorted adjacency - let order = g.kahn_topo(&back); - let mut conf = HashMap::with_capacity(order.len()); - for id in order { - let ins = g.in_edges(id, &back); - let c = if ins.is_empty() { - asserted(nb, id).unwrap_or(0.5) // DRIVER - } else { - let xw: Vec<(Conf, f32)> = ins.iter() - .map(|e| (weight_edge(conf[&e.from], e.weight as f32), e.weight as f32)) - .collect(); - combine(op_of(nb, id), &xw) // DRIVEN - }; - conf.insert(id, c); - } - conf -} -``` - -**Why fuzzy min/max weakest-link is the default:** - -- **Monotone, order-independent, single-pass.** It evaluates in one topological sweep, O(V+E), with no fixpoint iteration — the only family that stays interactive under spinner-drag at 10k nodes. -- **0.5 = Indeterminate is a better "unknown" than `null`.** A principled, bounded, neutral midpoint that the whole engine runs on. -- **Negation is a number.** The edge-weight transform gives graded support (w<1), abstention (w=0 → 0.5), and logical negation (w=−1) with no extra node type. -- **Back edges are drawable but excluded.** Feedback loops render (dashed, with a "you made a loop" cue) yet never break evaluation, because the evaluated graph is a DAG. - -**Made domain-overridable (resolving the semantics tension):** pluggability is layered so two domains can never disagree *inside one graph*: -- The **combine function** is chosen per node by `:op` (native table + λδ extension, §4.4). -- The **scalar semantics** (fuzzy vs probability vs DS-lite interval) is chosen per **domain package**, and a notebook's reasoning graph declares its domain once. A note tagged with a different domain's scalar type is a validation error surfaced by `defmulti validate :type` — the engine refuses to silently mix a fuzzy 0.7 with a DS mass function. Fuzzy is the default when no domain is declared. - -### 5.4 Incremental recompute - -For spinner drags, cache `order` and each node's downstream-reachable cone; on a driver change, re-sweep only the affected suffix. A full sweep at ~10k nodes / ~30k edges is sub-millisecond in WASM; a single-driver drag touches only its downstream cone. - ---- - -## 6. DEVONthink mapping - -### 6.1 Capability → nexia mechanism - -| DEVONthink capability | nexia mechanism | Cost | -|---|---|---| -| **See Also** | **`index.see_also` (concordance + TF-IDF + cosine kNN); `(see-also n k)`; L0 "Related notes" panel** | **M (crown jewel)** | -| Classify | `index.classify` (Rocchio centroid; NB as a pack); `(classify n)`; "Suggested tags" chips | M | -| Auto-Classify | `auto-classify!` — apply top class only if `top ≥ τ ∧ top−runner ≥ margin`; Action-context, undoable, opt-in | S (on Classify) | -| **Concordance / inverted index** | **`core/src/index.rs::SimilarityIndex` — the one shared substrate, rebuilt like `backlinks`** | **M (foundational)** | -| Smart Groups (dynamic saved search) | **already shipping** = `Agent { query }`; enrich DSL (`similar:`, `near:`, `conf:`, ranked) | free → S | -| Smart Rules — triggers | `Notebook.triggers`; host fires `TriggerEvent` → predicate → λδ action; local-first subset | M | -| Smart Rules — actions | λδ Action builtins (`set-attr!`, `link-typed!`, `add-tag!`, `auto-classify!`, `create-note!`) | S (mostly exist) | -| Replicants | **free** — one `Uuid`; membership = which agents/tags match; edits shared because one object | free | -| Duplicate detection | `index.duplicates` — blake3 exact + SimHash/LSH near; `(duplicates n)` + panel | S (on index) | -| Tags + hierarchical groups | tags = `attributes.tags` (multi-valued) via `tag:`/`attr:`; hierarchy = agents + prototype | S | -| Full-text search + operators/proximity | BM25 over the index; boolean/phrase/proximity via positional postings; L0 stays whitespace-ANDed | M | -| OCR / rich binary formats / RSS / web-clip / email / sync | **out** (§2) | — | -| Optional generative AI | opt-in λδ host **capability** (Ollama/llama.cpp); never default, no data leaves device | later, opt-in | - -### 6.2 The local intelligence engine - -One index, five features — the concrete pipeline: - -```rust -impl SimilarityIndex { - /// tf-idf weight = (1 + ln tf) · ln(N / df), idf against CURRENT df/N. - #[inline] fn w(&self, tf: u32, df: u32) -> f32 { - (1.0 + (tf as f32).ln()) * ((self.n_docs as f32) / (df as f32)).ln() - } - - /// See Also: top-k by cosine, self excluded. Walk only the query note's - /// top ~40 terms; accumulate partial dot-products over their postings. - pub fn see_also(&self, id: NoteId, k: usize) -> Vec<(NoteId, f32)> { - let d = self.doc_of[&id]; - let mut score: HashMap = HashMap::new(); - for &(t, tf) in top_terms(&self.forward[d as usize], 40) { - if (self.df[t as usize] as usize) > self.n_docs as usize / 2 { continue; } // freq-stop - let wq = self.w(tf, self.df[t as usize]); - for p in &self.postings[t as usize] { - if p.doc == d { continue; } - *score.entry(p.doc).or_default() += wq * self.w(p.tf, self.df[t as usize]); - } - } - let mut out: Vec<_> = score.into_iter() - .map(|(doc, dot)| (self.note_of[doc as usize], - dot / (self.doc_norm[d as usize] * self.doc_norm[doc as usize]))) - .collect(); - out.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0))); // deterministic tie-break (§9) - out.truncate(k); - out - } - - /// BM25 (k1≈1.2, b≈0.75) — replaces naive substring search, stands in for tantivy. - pub fn bm25(&self, query: &str, k1: f32, b: f32) -> Vec<(NoteId, f32)> { /* … */ } - - /// Rocchio nearest-centroid classify: cosine of note to each class centroid. - /// Cheaper and incrementally updatable vs multinomial NB (offered as a pack). - pub fn classify(&self, id: NoteId, centroids: &[(ClassId, SparseVec)]) -> Vec<(ClassId, f32)> { /* … */ } - - /// Duplicates: exact via blake3 buckets; near via SimHash Hamming ≤ 3, - /// candidates gathered by LSH bands (no O(n²) scan). - pub fn duplicates(&self, id: NoteId) -> Duplicates { /* … */ } -} -``` - -- **See-Also**: TF-IDF sparse vectors, cosine, top-k walking only the query note's ~40 heaviest terms with a frequency stop-list — single-digit ms at 10k. -- **Classify / Auto-Classify**: each Agent (or tag value) is a class; Rocchio centroid is the default (incrementally updatable), multinomial naive-Bayes ships as a pack. Auto-Classify is the **confidence-gated automation pattern**: apply only when `top ≥ threshold ∧ top − runner-up ≥ margin`, else defer to the human — routed through `auto-classify!` so it is undoable, budgeted, opt-in, never silent. -- **Duplicates**: blake3 exact-hash buckets + 64-bit SimHash (fixed seed) with LSH banding for near-dups; offer merge or "these are the same note". -- **Auto-tag**: kNN label propagation over See-Also neighbours (weighted vote above a threshold) — reuses the similarity engine, no separate model. -- **Ranked search**: BM25 over the same index with positional postings for phrase/proximity. The L0 whitespace-ANDed syntax is the default surface; boolean/phrase/`NEAR/n` are progressive-disclosure power operators. - -### 6.3 Smart Rules, Smart Groups, replicants - -- **Smart Groups = Agents.** Already shipping, 1:1. The only work is enriching the DSL as the index lands (§3.6). -- **Smart Rules = triggered λδ actions.** `TriggerEvent` → Agent-predicate condition → λδ Action body — DEVONthink's event→condition→action grammar, native to the homoiconic core. -- **Replicants are free.** A `Note` has one `Uuid`; "membership" is which Agents match and which tags it carries. A single note already appears in every matching Agent view and every tag facet, with edits inherently shared because there is one object. Nothing is ever copied. - -### 6.4 Positioning note (resolving the tantivy tension) - -The ROADMAP commits Phase 3/5 to **tantivy**. We consciously deviate: tantivy gives BM25 but **not** See-Also cosine, Rocchio classify, or SimHash dedup. Running tantivy *and* a separate vector index means two indices and more memory. We therefore **hand-roll one concordance for all five features**, accepting the maintenance burden of our own BM25/phrase/proximity implementation as the price of the shared substrate — a hand-rolled index is comfortably sufficient at 10k notes. This retires the "tantivy planned but not built" seam. We also **reframe ROADMAP Phase 8 "Intelligence"**: See-Also/Classify are the *deterministic local heart* of that phase; any LLM is strictly optional and opt-in, never the default — a positioning this document asks the ROADMAP to adopt. - ---- - -## 7. Algorithms and performance - -### 7.1 Concrete choices - -| Concern | Choice | -|---|---| -| Tokenisation | Unicode word segmentation → lowercase → stop-list → optional Porter stem; deterministic, fixed data | -| Term weighting | TF-IDF `(1+ln tf)·ln(N/df)`; L2-normalised sparse vectors | -| Similarity | cosine (dot of normalised vectors) | -| Search ranking | BM25, k1≈1.2, b≈0.75, positional postings for phrase/proximity | -| Classify | Rocchio nearest-centroid (default); multinomial NB (pack) | -| Dedup | blake3 exact + 64-bit SimHash (fixed seed) + LSH bands | -| Propagation | edge-weight transform + native operator table; DFS back-edge exclusion + Kahn topo; single pass | -| Layout | Sugiyama (longest-path layering → barycenter → Brandes–Köpf), positions derived | - -### 7.2 Complexity and memory at 10k notes (~300 tokens/note, ~200 unique) - -| Operation | Cost | Notes | -|---|---|---| -| Full index build | O(Σ tokens) ≈ 3M | tens of ms; on load / import only | -| Incremental reindex (one edit) | O(tokens in note) | in `set_content`/title path; imperceptible | -| See-Also query | O(top-terms × avg df) | 40-term cap + freq-stop → single-digit ms | -| BM25 search | O(Σ df of query terms) | ms | -| Classify (Rocchio) | O(terms(d) × #classes) | ms | -| All-pairs dedup | ~O(N) via LSH bands | not O(N²) | -| Full propagation sweep | O(V+E) ≈ 40k ops | sub-ms in WASM | -| Spinner-drag recompute | O(downstream cone) | interactive | -| **Index memory** | **~35–50 MB** | inverted ≈16 MB + forward ≈16 MB + simhash 80 KB + norms 40 KB. **Requires dense-`DocId` interning** (postings store 8-byte `(u32,u32)`, never 16-byte `Uuid`). Comfortable in a browser tab. | - -### 7.3 Incremental update and the off-main-thread escape hatch - -All hot operations are incremental: reindex is per-note; propagation re-sweeps only the downstream cone; layout runs on the reasoning subgraph, not the whole notebook. The full build (load, import, or the opportunistic norm-resetting rebuild) is the only O(corpus) operation, and it is tens of ms. - -**Web Worker escape hatch.** Everything is pure, deterministic, allocation-only Rust — no threads/network/clock/RNG (SimHash uses a fixed seed) — so it slots into the wasm-bindgen core and honours the λδ budget. If a very large notebook ever makes the initial build or a full rebuild perceptible, the WASM core can be instantiated in a **Web Worker**: the TEA loop posts the notebook JSON in, receives the built indices out, and the main thread never blocks. This is a deployment option, not an architectural change — the core code is identical either way. - ---- - -## 8. Phased roadmap - -Aligned to the λδ L0–L4 progressive-disclosure levels and the numbered ROADMAP phases. Each phase is independently shippable; L0 UX (no parentheses, ever) is untouched throughout; new state is either additive-and-skippable or `#[serde(skip)]`-derived. - -### The recommended FIRST PR — headless `SimilarityIndex` - -**Smallest, headless, highest-value.** Build `core/src/index.rs` + tokenizer + the incremental hook in `set_content` **and both title write paths** + `rebuild_indices`. No WASM entry points, no UI. Unit- and property-tested against the existing `golden.rs` / `invariants.rs` fixtures, including the byte-identical round-trip proof (the index is `#[serde(skip)]`, so this is automatic) and a determinism test (sorted iteration, fixed-seed SimHash). This de-risks the entire DEVONthink side, fills the "tantivy planned but not built" seam, and lands zero user-visible change. It mirrors the L0-substrate discipline ADR-0003 used for λδ. - -### Phase map - -| Level / ROADMAP | Ships | New state | User-visible | -|---|---|---|---| -| **First PR** (ROADMAP Phase 3) | headless `index.rs`, tokenizer, incremental hooks, `rebuild_indices` | `#[serde(skip)]` index | none | -| **L0** (ROADMAP Phase 3 + differentiator) | See-Also / Suggested-tags / Duplicates panels (mount only when non-empty); BM25 ranked search replaces substring; `Edge` + `reason.rs` + confidence spinner in the editor; `ReasoningView` (`layout.rs`) with driver/driven shading and dashed back-edges | `edges` field (empty by default) | Related panel; ranked search; spinners; reasoning canvas — all no-code | -| **L1** (ROADMAP Phase 4) | pure builtins in fx fields: `(see-also self k)`, `(classify self)`, `(:confidence (propagate))`; Agent DSL gains `similar:`/`near:`/`conf:`/`type:`/`edge:` | — | computed confidence & similarity in formulas | -| **L2** (ROADMAP Phase 4/9) | `(propagate!)`, `(link-typed!)`, `(junct!)`, `(auto-classify!)`, `deftrigger`; `combine`/`render`/`validate`/`edge-style` multimethods; `Notebook.triggers`; ship `flying-logic.ld` + `librarian.ld` | `triggers` collection (empty by default) | Smart Rules; domain packs | -| **L3** | REPL exploration of `(propagate)`, `(see-also …)`, `(duplicates …)`, `(back-edges)` against the live graph under budget | — | power console | -| **L4** (ROADMAP Phase 7 Views + Phase 9 Ecosystem) | computational notes rendering See-Also / propagation inline; layout as a `defview`; TOC / probability / DS-lite domain packs via the plugin system (#33); opt-in local-LLM host capability | — | shareable packages | - -**ROADMAP crosswalk.** Phase 3 Search → the concordance + BM25 (retires tantivy). Phase 4 Agents → Smart Groups (exist) + Smart Rules (triggers). Phase 7 Views → layered layout replacing the GraphView placeholder. Phase 8 Intelligence → See-Also/Classify are the deterministic local heart; LLM strictly optional. Phase 9 Ecosystem → both tools as `.ld` domain packages. - -**Net new native surface:** one struct + one `Note` field, plus small modules `index.rs`, `edge.rs`, `reason.rs`, `trigger.rs`, and later `layout.rs`. Everything a Flying Logic or DEVONthink user recognises as *intelligence* — operators, domains, methodologies, classification policy, organisation rules, reasoning views — ships as λδ multimethods, `.ld` packages, and Agents on top. - ---- - -## 9. Risks and open questions - -**Resolved in this design (documented so the resolution is auditable):** - -- **Determinism under the λδ sandbox.** DFS back-edge selection and any HashMap iteration in propagation/index must be order-stable, or golden tests flake and the no-random contract is violated. *Resolution:* the `ReasonGraph` sorts nodes and adjacency lists before DFS/Kahn; See-Also/search sort with an id tie-break (`then(a.0.cmp(&b.0))`); SimHash uses a fixed seed; tokenisation is pure. A determinism property test guards all four. -- **Cycle policy on a graph that had no acyclic invariant.** `links` never required acyclicity and `link_notes` only rejects self-links. *Resolution:* the *new* `edges` channel defines its own policy — DFS-detected back edges are excluded from flow (single pass, no fixpoint), drawn dashed, and surfaced via a "you made a loop" cue. `links` is unaffected. -- **Native/λδ operator seam.** A conformance test asserts the native `Op` table and the λδ `combine` methods agree for shipped ops, so crossing the seam never changes semantics; unknown ops go through λδ and are accepted as slower (§4.4). -- **Back-compat migration.** Empty-`Vec` + `skip_serializing_if` gives a provable byte-identical round-trip on every golden fixture — the safest of the three considered approaches, and far safer than an untagged `Link` enum on the load-bearing field. -- **Index memory at 10k.** Mandated dense-`DocId` interning holds it to ~35–50 MB (§7.2). -- **Incremental consistency.** Title paths are hooked; idf/norm drift is bounded to a session by `#[serde(skip)]` rebuild-on-load, with idf computed against current `df`/`N` at query time (§3.5). - -**Genuinely open:** - -1. **When to trigger the opportunistic full rebuild.** On-open is free; do we also rebuild after N edits, or only when a norm-drift heuristic exceeds a threshold? Needs measurement on real notebooks. -2. **"Promote this association to a typed edge" UX.** The clean split means a wikilink can never also be causal; a user who wants both must create a typed edge. The affordance must avoid silent duplication (the same relation appearing once as a link, once as an edge). Design needed. -3. **Classify class-source ambiguity.** Agents *and* tag values can both be "classes". Which is the default training signal, and how do we present two overlapping class systems without confusing the user? -4. **Stemming/stop-list quality vs determinism across locales.** Porter stemming and English stop-lists markedly improve similarity but are language-specific; the multilingual story (and whether stemming is a per-notebook toggle) is unspecified. -5. **DS-lite domain semantics.** Offering a belief/plausibility interval as an optional domain (§5.4) needs its own operator table and validation rules; the interaction with the native fast path is sketched but not specified. -6. **Web Worker boundary.** If we adopt the off-main-thread build (§7.3), the serialization cost of shuttling a large notebook to the worker must be measured against the main-thread block it avoids. -7. **Trigger execution ordering and loops.** When one trigger's action fires an event another trigger listens for, we need a documented depth/reentrancy limit (the `Budget` bounds a single action, not a cascade). - ---- - -*Steal the intelligence and the primitives, not the feature sprawl. Keep the note model untouched, the semantics as data, the hot path native, the disclosure progressive, and everything on-device. The letter still reaches a richer future self — now able to reason over its own claims and recall its own forgotten neighbours.* diff --git a/docs/design/flyinglogic-devonthink-proofs.adoc b/docs/design/flyinglogic-devonthink-proofs.adoc new file mode 100644 index 0000000..937bff0 --- /dev/null +++ b/docs/design/flyinglogic-devonthink-proofs.adoc @@ -0,0 +1,781 @@ +== Nexia-List — Hard Problems, Solved, with Proof Designs + +_The fable-tier work. Everything below is the reasoning that is +expensive to get right and cheap to get subtly wrong. It is written so +that a lower-tier implementer (Opus) can translate each module into +Rust/ReScript against a fixed spec and discharge a checkable +*proof-obligation ledger* (§8) — without re-deriving any mathematics._ + +Companion to: the FL×DT integration design and the mind-management plan. +Assumes their decisions (additive `+edges+` channel; +derived-index-beside-the-notes discipline; native operator table + λδ +`+combine+` for exotic ops; `+.ld+` domains). + +____ +*v2 — corrections after an adversarial proof-check (Opus) against the +real `+core/+`.* Four items were materially wrong in v1 and are fixed +inline (each tagged ⚠v2): *(1)* float folds must be pinned in key-sorted +order — IEEE non-associativity means "`sort the outputs`" is _not_ +enough for determinism (§1.4, §2.4, §6); *(2)* byte-identical serde is +false until the pre-existing `+notes+`/`+backlinks+`/`+attributes+` +`+HashMap+`s serialize in canonical key order (§3); *(3)* the `+O(1)+` +posting `+remove+` must be _two-sided_ or reindex corrupts Invariant L +(§2.1); *(4)* the driver/driven boundary is strictly by `+G'+` in-degree +— `+combine(∅)+` is dead code (§1.4/§1.6). The mathematical core +(§1.2–1.7, §2.2–2.3 ideas, the `+in_edges+`/`+:not+` catches) survived. +The ledger (§8) gained the discriminating tests. +____ + +''''' + +=== 0. The delegation contract + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Tier |Owns |Must NOT need to do +|*fable (this document)* |the data-structure _shapes_ that make the +complexity claims true; the invariants; the theorems and their proofs; +the caught bugs; the proof-obligation ledger |— + +|*Opus (implementation)* |translate each `+§ spec+` block into +Rust/ReScript honouring the stated invariants; write the test named in +each `+PO-*+` ledger row until green; wire WASM entry points and TEA +views |re-derive the math, invent the data structure, or decide the +semantics +|=== + +*Reading order for Opus:* for each module, read the _Invariant_, then +the _Spec_, then implement, then make the _Proof Obligations_ pass. The +proofs here exist so that when a test fails, the implementer knows which +invariant was violated rather than guessing. + +Three bugs in the prior design were found while proving it; they are +called out inline as *⚠ CAUGHT*. + +''''' + +=== 1. Confidence propagation — the reasoning engine + +The claim to be justified: _one deterministic pass computes a +well-defined confidence for every node; feedback loops are drawn but +never break evaluation; a spinner drag recomputes only what can change._ + +==== 1.1 Setup and notation + +* The reasoning graph is `+G = (V, E)+` where `+V+` = notes and `+E+` = +typed edges (`+Note.edges+`), each edge `+(u→v)+` carrying weight +`+w ∈ [-1,1]+`. `+links+` do *not* appear here. +* A node is a *driver* if it has no in-edges in the (reduced) graph; +otherwise *driven*. +* `+conf : V → [0,1]+`. `+0.5+` is the distinguished value +_Indeterminate_. +* `+asserted(v) ∈ [0,1]+` is a driver’s stored `+:confidence+` (default +`+0.5+`). +* `+op(v)+` is the node’s combining operator (default `+:and+`; a junct +carries `+:op+`). + +==== 1.2 Theorem (Reduced-DAG). _Removing the canonical DFS back-edge set from `+G+` yields a DAG._ + +*Proof.* Run DFS over `+G+`. Classify each edge as tree / forward / +cross / back, where `+(u→v)+` is *back* iff `+v+` is on the recursion +stack (grey) when the edge is explored — i.e. `+v+` is a DFS-ancestor of +`+u+`. Assign finish times `+f+`. For every non-back edge `+(u→v)+`, +standard DFS parenthesis theory gives `+f(u) > f(v)+`. Let +`+E' = E \ Back+`. Order `+V+` by _descending_ `+f+`. Every edge of +`+E'+` then goes from an earlier to a later vertex, so this is a +topological order of `+(V, E')+`; a graph admitting a topological order +is acyclic. ∎ + +==== 1.3 Theorem (Determinism of the cut). _With a canonical DFS the back-edge set — hence `+E'+`, the topo order, and every `+conf(v)+` — is a pure function of the notebook._ + +*Why it is not free:* which edge of a cycle is the back edge depends on +DFS visitation order, and Rust `+HashMap+` iteration order is +unspecified (and, in `+wasm32+`, must never be relied on). + +*Proof / construction.* Build `+ReasonGraph+` as *sorted vectors*, never +a `+HashMap+` walked for order: - nodes in ascending `+NoteId+` order +(`+Uuid+` has a total order); - each adjacency list sorted by +`+(target NoteId, source edge index)+`. DFS pushes roots and neighbours +in that fixed order. Finish times, and therefore `+Back+`, are now a +deterministic function of the sorted structure, which is itself a +deterministic function of the notebook value. Every downstream quantity +is a pure function of `+Back+`. ∎ + +____ +*§ spec.* `+ReasonGraph::build(nb)+` returns +`+{ ids: Vec (sorted), idx: HashMap, adj: Vec> (each sorted) }+`. +`+back_edges()+` is an iterative DFS over `+adj+` using an explicit +`+Vec+` stack and a `+Vec+` — no recursion (WASM stack safety), +no `+HashMap+` iteration. +____ + +==== 1.4 Theorem (Well-defined, order-independent values). _On `+G' = (V, E')+` the assignment_ + +.... +conf(v) = asserted(v) if in-deg_{G'}(v) = 0 +conf(v) = combine(op(v), { (weight_edge(conf(u), w), w) : (u→v) ∈ E' }) otherwise +.... + +_has a unique solution, and one Kahn sweep computes it regardless of +which valid topological order Kahn happens to pick._ + +*Proof.* `+G'+` is a DAG (§1.2), so the definition is a well-founded +recursion: `+conf(v)+` refers only to `+conf(u)+` for strict +predecessors `+u+`, and the predecessor relation is a strict partial +order with no infinite descending chains. Well-founded recursion has a +unique total solution. Kahn processes each `+v+` only after all +predecessors are finalized, so it evaluates exactly the recursion. +Uniqueness of the _solution_ (not merely of one run) gives +order-independence: any linear extension yields the same `+conf+`, +because each `+conf(v)+` is fixed by the values of its predecessors, +which are fixed inductively from the drivers up. ∎ + +*Determinism caveat — `+combine+` must be a _canonical_ function of the +inputs.* Two distinct hazards: + +* _Multiset symmetry._ min, max, `+1−x+`, fuzzy-xor are order-exact. +* *⚠v2 CAUGHT (float non-associativity).* IEEE `+++`/`+×+` are +commutative but *not associative*, so `+prob-sum+` (`+1−∏(1−xᵢ)+`), +`+product+` (`+∏xᵢ+`) and `+proportion+` (`+Σ|wᵢ|xᵢ / Σ|wᵢ|+`) are _not_ +functions of the input multiset in `+f32+`: a different fold order +yields a ULP-different result, and a single ULP flips a `+total_cmp+` +tie in a ranked list (§2.4) or changes a serialized `+conf+` byte (§6). +*Mandatory spec line:* `+combine+` folds its inputs in *sorted +`+(source NoteId, source edge-index)+` order* — a pure function of the +notebook. (`+min/max/not/fuzzy-xor+` are order-exact and need no +pinning, but pinning all uniformly is simplest.) This is what makes +`+PO-1.4a+`/`+PO-6+` achievable; without it they fail intermittently. + +`+:not+` is unary — the remaining symmetry exception: + +____ +*⚠ CAUGHT (arity).* `+:not+` is unary. A `+:not+` node with in-degree ≠ +1 is undefined. Resolution: `+validate+` (§4.3) rejects it at edit time; +`+combine(:not, xs)+` asserts `+xs.len()==1+` and is only ever reached +for validated graphs. Document `+:not+` as strictly unary in +`+flying-logic.ld+`. +____ + +==== 1.5 Lemma (Edge-weight transform). _`+weight_edge(v, w) = ((2v−1)·w + 1)/2+` maps `+[0,1]×[-1,1] → [0,1]+` and realizes pass-through / abstain / negation._ + +*Proof.* `+2v−1 ∈ [-1,1]+`; with `+|w|≤1+`, `+(2v−1)w ∈ [-1,1]+`; +`++1 → [0,2]+`; `+/2 → [0,1]+` (closure). Substituting: `+w=1 ⇒ v+` +(identity); `+w=0 ⇒ 0.5+` (abstain — a zero-weight edge contributes +Indeterminate, i.e. _no information_); `+w=−1 ⇒ 1−v+` (negation). +`+∂/∂v = w+`, so monotone increasing for `+w>0+`, decreasing for +`+w<0+`, flat at `+w=0+`. ∎ + +==== 1.6 Lemma (Operator closure). _Every shipped operator maps `+[0,1]^k → [0,1]+`._ + +*Proof (the only non-obvious cases).* - *prob-sum* `+⊕+`: +`+a⊕b = a + b − ab = 1 − (1−a)(1−b)+`. For `+a,b∈[0,1]+`, +`+(1−a)(1−b)∈[0,1]+`, so `+a⊕b∈[0,1]+`; associativity gives the +`+k+`-ary form `+1 − ∏(1−xᵢ) ∈ [0,1]+`. ∎ - *proportion* `+∷+`: +`+Σ|wᵢ|xᵢ / Σ|wᵢ|+` is a convex combination of values in `+[0,1]+` ⇒ in +`+[0,1]+`; `+Σ|wᵢ|=0 ⇒ 0.5+` by definition. ∎ - *fuzzy-xor*: +`+maxᵢ min(xᵢ, 1 − max_{j≠i} xⱼ)+` — a max of mins of values in +`+[0,1]+` ⇒ in `+[0,1]+`. ∎ - min / max / `+1−x+` / product: closure +immediate. + +*⚠v2 — driver/driven boundary (resolving a v1 inconsistency).* +Classification is *strictly by `+G'+` (post-cut) in-degree*: a node with +`+in-deg_{G'}(v) = 0+` is a *driver* and takes its `+asserted+` value +(default `+0.5+`) — _even if it had in-edges in `+G+` that were all +back-edges_. Consequently `+combine+` is only ever invoked with ≥1 +surviving input; `+combine(∅)+` is *unreachable dead code* (if +defensively kept, return `+0.5+`). Practical upshot: a note carrying +`+:confidence = 0.8+` that is targeted only by cyclic edges reports +*0.8*, not `+0.5+` — its assertion is honoured, not silently discarded +(`+PO-1.6b+`). + +==== 1.7 Theorem (Incremental recompute — the spinner-drag correctness). _If only driver `+d+`’s `+asserted+` value changes, the set of nodes whose `+conf+` can change is exactly the descendants of `+d+` in `+G'+`. Re-sweeping `+descendants(d)+` in topo order recomputes them correctly and leaves all other nodes untouched._ + +*Proof.* `+conf(v)+` is determined (§1.4) by the asserted values of the +drivers that are ancestors of `+v+`. If `+d ∉ ancestors(v)+` then none +of `+v+`’s determining inputs changed, so by induction over the topo +order restricted to non-descendants, `+conf(v)+` is unchanged. +Conversely every `+v ∈ descendants(d)+` has `+d+` as an ancestor and may +change. Re-sweeping precisely `+descendants(d)+` (a subset closed under +"`successor`", processed in the global topo order) evaluates the same +recursion for exactly the affected nodes. ∎ + +____ +*§ spec.* Cache `+order: Vec+` and, per driver, +`+cone: Vec+` (its forward-reachable set, computed lazily by BFS +over `+adj+` and memoized until the edge set changes). +`+set_confidence!(d,x)+` → re-sweep `+cone(d) ∩ order+` in order. A full +sweep is `+O(V+E)+` (~40k ops at 10k notes/30k edges) — sub-millisecond +in WASM — so the cone optimization is a UX nicety for very large graphs, +not a correctness dependency. +____ + +==== 1.8 Semantics choice (why fuzzy min/max weakest-link is the default), stated as properties + +The default algebra is the one family that is simultaneously: *(P1)* +closed on `+[0,1]+`, *(P2)* symmetric (so §1.4 determinism holds), +*(P3)* evaluable in a single pass with no fixpoint (min/max are +idempotent and need no iteration to converge), and *(P4)* has a neutral +`+0.5+` that is a genuine "`unknown`" rather than a false `+0+`. +Probability (`+×+`, `+⊕+`) and DS-lite intervals are opt-in per-domain +(§4.3) precisely because they trade one of these away (e.g. `+×+` is not +idempotent, so repeated evidence double-counts — desirable sometimes, +wrong as a default). + +''''' + +=== 2. The similarity index — data-structure invariants and incremental correctness + +The claim to be justified: _one inverted index supports See-Also / BM25 +/ Classify / dedup; a single-note edit costs `+O(tokens in that note)+`, +not `+O(corpus)+`; results are exact and deterministic; deletion is +handled._ + +==== 2.1 The `+O(tokens)+` reindex — the doubly-linked posting structure (the real trick) + +____ +*⚠ CAUGHT (complexity).* The prior design stores postings "`sorted by +doc`" and claims `+O(tokens)+` incremental reindex. Removing one doc’s +posting from a sorted `+Vec+` is `+O(df)+` (shift), and there is no way +to _find_ it without an `+O(df)+` scan — so the honest cost of the naive +structure is `+O(Σ_t df(t))+`, which for common terms is `+O(N)+`, not +`+O(tokens)+`. The `+O(tokens)+` claim is only recoverable with a +different structure. +____ + +*The fix — unsorted postings with two-way back-links.* See-Also and BM25 +both accumulate scores into a `+HashMap+` by _iterating_ a +term’s postings; neither needs them sorted (only positional +phrase/proximity needs per-doc position lists, which live elsewhere). So +postings can be unsorted, which unlocks `+O(1)+` add/remove: + +[source,rust] +---- +struct Posting { doc: DocId, tf: u32, fwd: u32 } // fwd = index into forward[doc] +struct FwdEntry { term: TermId, tf: u32, slot: u32 } // slot = index into postings[term] + +// INVARIANT L (link): for every term t and every slot s, +// let p = postings[t][s]; forward[p.doc][p.fwd] == FwdEntry{ term: t, .., slot: s } +// and symmetrically for every forward entry. The two arrays are mutual inverses. +---- + +* *add(d,t,tf):* `+s = postings[t].len(); j = forward[d].len();+` push +`+Posting{d,tf,fwd:j}+`; push `+FwdEntry{t,tf,slot:s}+`; `+df[t]+=1+`. +`+O(1)+`. +* *remove(d, term at forward index `+j+`):* *⚠v2 — this is a _two-sided_ +swap-remove.* The arrays are mutual inverses, so _both_ must be fixed: +[arabic] +. _postings side:_ `+s = forward[d][j].slot+`; +`+swap_remove(postings[term], s)+`; if a posting `+P+` moved into slot +`+s+`, set `+forward[P.doc][P.fwd].slot = s+`. +. _forward side:_ `+swap_remove(forward[d], j)+`; if a `+FwdEntry Q+` +moved into index `+j+`, set `+postings[Q.term][Q.slot].fwd = j+`. +`+df[term] -= 1+`. Both fixups are `+O(1)+` (each guarded by a "`was +anything actually moved?`" check for the last-element case). *Omitting +step 2 corrupts Invariant L on the _reindex_ path* (a surviving doc +dropping a term); §2.2 deletion hides the omission only because it +`+clear()+`s `+forward[d]+` wholesale (`+PO-2.1+`). + +Therefore `+reindex+` = (diff old vs new token bag) then `+O(1)+` per +changed `+(doc,term)+` = *`+O(|B_old| + |B_new|)+`*, genuinely +`+O(tokens in the edited note)+`. Invariant L is the checkable contract +(PO-2.1). + +==== 2.2 DocId lifecycle under deletion (the gap the design left open) + +*Decision: tombstone during a session, compact on load.* DocIds are +allocated append-only (`+0,1,2,…+`) as notes are indexed. Deleting note +`+d+`: 1. for each `+FwdEntry+` in `+forward[d]+`: `+remove(d, term)+` +(§2.1), which maintains `+df+`; 2. +`+forward[d].clear(); note_of[d] = None; doc_of.remove(¬e_id); simhash[d] = DEAD; n_docs -= 1;+` +3. DocId `+d+` is *never reused* within the session. + +*Theorem (deletion preserves INV-IDX).* After deletion, (a) no posting +references `+d+`; (b) `+df[t]+` equals the number of live docs +containing `+t+`; (c) `+n_docs+` equals the live count; (d) every other +doc’s postings are unchanged _in content_ (swap-remove only relabels +slots, preserving Invariant L). *Proof.* (a) every term of `+d+` had its +posting removed in step 1; (b) each removal decremented exactly the +terms `+d+` contained; (c) step 2; (d) `+swap_remove+` moves a posting +but §2.1’s fixup restores Invariant L, and its `+(doc,tf)+` payload is +untouched. ∎ + +*Memory bound.* Tombstones (holes in `+note_of+`) number at most the +session’s deletions. Because the index is `+#[serde(skip)]+` and +*rebuilt on load* from the sorted live notes, DocIds are re-densified +every session ⇒ holes never accumulate across sessions. This is exactly +the `+backlinks+` discipline: _derived, never trusted from disk, rebuilt +on load._ + +==== 2.3 Retiring the idf/norm-drift tension — exact cosine at query time + +____ +*⚠ CAUGHT (the drift was avoidable).* The prior design caches +`+doc_norm+` under stale idf and then argues the resulting ranking error +is "`imperceptible.`" We can do better: compute the exact cosine and +delete the tension. +____ + +`+cos(q,d) = ⟨w_q, w_d⟩ / (‖w_q‖·‖w_d‖)+` with tf-idf weight +`+w(t) = (1+ln tf)·ln(N/df(t))+` under the *current* `+N, df+`. See-Also +already forms the numerator `+⟨w_q,w_d⟩+` by walking `+q+`'`s top +terms`' postings and accumulating into `+score[d]+`, all with current +idf. The only question is the denominator norms: + +* `+‖w_q‖+` — computed once from `+q+`’s forward vector under current +idf: `+O(|terms(q)|)+`. +* `+‖w_d‖+` for each candidate `+d+` — computed from `+d+`’s forward +vector under current idf: `+O(|terms(d)|)+`. + +*Theorem (exactness).* Computing numerator and both norms all under the +current `+(N, df)+` yields the exact current cosine; ranking candidates +by it yields the exact top-k. *Proof.* Immediate — every term of the +definition is evaluated under one consistent idf. ∎ + +*Cost — the honest bound (⚠v2).* Let `+C+` = candidate set (docs sharing +one of `+q+`’s top-40 terms after dropping terms with `+df > N/2+`). +Extra work is `+Σ_{d∈C}|terms(d)|+`. *Worst case is `+O(N)+` / +`+O(corpus)+`*: `+|C| ≤ 40·(N/2) = O(N)+`, so exact-at-query-time is an +_average-case_ win (real top terms are discriminative ⇒ `+C ≪ N+`), +*not* a worst-case guarantee. *Guaranteed path:* when +`+Σ_{t∈top40} df(t)+` exceeds a fixed budget `+B+`, fall back to cached +`+doc_norm+` so the query is always `+O(B)+`; also tighten the stop rule +(absolute posting cap, or `+df > N/10+`). The fallback error is a +per-doc scaling: `+score_cached(d) = score_exact(d)·ρ_d+`, +`+ρ_d = ‖w_d‖_cache/‖w_d‖_exact → 1+` as term `+df+` grows, +`+|ρ_d−1| = O(E/(df_min·ln(N/df)))+`, `+E+` = edits since that doc’s +last reindex (reset to 0 on load). So: *exact under budget `+B+`, +cached-norm above it* — the fallback is a guaranteed path, not merely a +profiling flag. Norm and numerator folds are `+TermId+`-ordered (§2.4). + +==== 2.4 Determinism of the intelligence pipeline + +Every order-sensitive step is pinned: - *tokenization* is pure (Unicode +segmentation → lowercase → static stop-list → optional Porter stem; all +fixed data); - *SimHash* uses a *fixed seed* (no RNG); - *See-Also / +BM25* sort by `+score+` then `+NoteId+` tie-break: +`+b.score.total_cmp(&a.score).then(a.id.cmp(&b.id))+`; - *no `+HashMap+` +is iterated* where output order matters (accumulate into a map, then +sort the entries); - *⚠v2 — float reductions fold in key-sorted order* +(the subtle one): the cosine numerator `+Σ_t w_q(t)·w_d(t)+` and both +norms `+Σ_t w(t)²+` iterate terms in *`+TermId+` order* (and the top-40 +cut breaks ties by `+TermId+`); `+combine+`’s +`+proportion+`/`+prob-sum+`/`+product+` fold in-edges in +`+(source NoteId, edge-index)+` order (§1.4). Because IEEE `+++`/`+×+` +are non-associative, *sorting the _outputs_ is not sufficient — the +_accumulation_ order must itself be a key-sorted, notebook-pure +function*, or a 1-ULP difference flips a `+total_cmp+` tie between two +near-equal docs and breaks `+PO-2.4+`/`+PO-6+`. Classify’s centroid +dot-products fold the same way (`+PO-2.6+`). + +Consequence (with §1.3): the entire index is a pure function of the live +note set, independent of insertion order — golden tests are stable and +the λδ no-RNG/no-clock contract holds. + +==== 2.5 Near-duplicate recall (SimHash + LSH), as a bound not a hope + +64-bit SimHash; near-dup iff Hamming distance `+≤ h₀+` (e.g. 3). Split +the 64 bits into `+b+` bands of `+r = 64/b+` bits; two docs are +LSH-candidates iff some band is identical. For a pair at Hamming +distance `+h+`, the probability a given band matches is +`+((64−h)/64)^r+`… (standard bit-sampling bound); with `+b=8, r=8+`, +pairs at `+h≤3+` collide in ≥1 band with probability +`+≥ 1 − (1 − (61/64)^8)^8 ≈ 0.999+`. Choose `+(b,r)+` from the target +`+h₀+` and acceptable false-negative rate; exact duplicates (Hamming 0) +are caught additionally by blake3 content-hash buckets (zero false +negatives). Determinism: fixed seed + `+NoteId+`-sorted candidate +output. + +==== 2.6 Master index invariant + per-op maintenance + +.... +INV-IDX ≜ Invariant L (§2.1) (postings ↔ forward are inverses) + ∧ ∀t. df[t] = #{ live d : t ∈ bag(d) } + ∧ n_docs = #{ live d } + ∧ ∀ live d. forward[d] = tfidf-bag(title(d) ++ content(d)) (under tokenization) + ∧ ∀ live d. simhash[d] = SimHash(bag(d)) (fixed seed) +.... + +[width="100%",cols="50%,50%",options="header",] +|=== +|Op |Maintenance +|`+create+` |allocate DocId, `+add+` each term, set `+simhash+`, +`+n_docs+=1+` + +|`+set_content+` |`+reindex(id, title(id), new_content)+` (diff) + +|`+set_title+` (*incl. `+update_title+`/`+bi_set_title+`*) +|`+reindex(id, new_title, content(id))+` — *⚠ CAUGHT earlier:* titles +dominate short-note similarity; both title write paths must call +reindex, or See-Also silently desyncs + +|`+delete+` |§2.2 + +|`+set_attr+` / link ops |no index change (attrs/links are not indexed +text) +|=== + +''''' + +=== 3. Serde back-compat — round-trip fidelity, and the two things that break byte-identity + +____ +*⚠v2 — revised after verification.* The v1 claim ("`any current notebook +round-trips byte-identically once new fields carry the right +attributes`") is *false as stated*, for a reason _independent of the new +fields_. Corrected below. +____ + +*What is actually required (and provable).* Existing notebooks must (a) +*load without loss* under the extended schema, and (b) gain *no new JSON +keys* while their new fields are empty/derived. This holds iff every new +field is `+#[serde(skip)]+` (derived) or +`+#[serde(default, skip_serializing_if="…is_empty")]+`, declared after +the existing fields. + +*Proof (of the required property).* A missing key deserializes via +`+default+`/`+skip+` (no effect on siblings); an empty/derived value is +omitted on serialize (no new key). ∎ This is sufficient for correctness +and for forward/backward loading — it is the property Opus must +guarantee. + +*Why _byte_-identity needs one more fix (⚠v2 — the bigger bug).* The +_existing_ struct already serializes three `+HashMap+`s in *iteration +order*: - `+notebook.rs:26+` `+notes: HashMap+` - +`+notebook.rs:29+` `+backlinks: HashMap>+` - +`+note.rs:70+` `+attributes: HashMap+` + +`+serde_json+` emits maps in iteration order, and +`+std::HashMap+`/`+HashSet+` iterate in a per-instance seed-dependent +order, so `+serialize → deserialize → serialize+` *permutes keys* for +any map with ≥2 entries (deserialize builds a fresh map with a fresh +seed). Byte-identity was therefore *never actually held* — the current +round-trip test (`+notebook.rs:394+`) checks only semantic fields, never +bytes. Adding correctly-attributed new fields is _necessary but not +sufficient_. + +*Fix (only if byte-level golden gates are wanted).* Serialize the three +maps in *canonical key order* — least-invasive is +`+#[serde(serialize_with = "sorted_map")]+` (keeps `+HashMap+` in +memory, sorts on the way out); alternatively switch to +`+BTreeMap+`/`+BTreeSet+` (larger blast radius — touches `+note.rs+`, +`+agent.rs::attribute_equals+`, `+exchange.rs+`) — and *regenerate the +golden fixtures* with the canonical serializer. + +*Theorem (corrected).* _Under canonical map ordering, any current +notebook round-trips byte-identically once new fields carry the +attributes below._ Proof exactly as above, now that key order is a pure +function of the key set. ∎ + +If byte-level gates are _not_ wanted, relax `+PO-3+` to *semantic* +round-trip equality (the property that truly matters) and skip the +canonical-serialization change. *Recommendation:* adopt +`+serialize_with+` canonical ordering — cheap, makes golden diffs +meaningful, and removes a latent cross-platform flake. + +*The `+in_edges+` sub-bug (real, independent of the above).* The prior +delta declared `+in_edges+` with `+#[serde(default)]+` and *no* +`+skip_serializing_if+`; an empty map still serializes as +`+"in_edges":{}+`, injecting a key. `+backlinks+` (`+notebook.rs:29+`, +`+#[serde(default)]+`, *persisted*) is living proof of exactly this +behaviour. Fix — `+in_edges+` is derived, so `+#[serde(skip)]+` it and +rebuild. *Wording correction:* this _deviates_ from the `+backlinks+` +discipline (which _is_ persisted), rather than "`following it`" — we are +choosing not to persist any derived reverse index. `+#[serde(skip)]+` +also requires `+SimilarityIndex: Default+`. + +[source,rust] +---- +#[serde(skip)] index: SimilarityIndex, // derived; needs Default +#[serde(skip)] in_edges: HashMap>, // ← was the bug +#[serde(default, skip_serializing_if="Vec::is_empty")] triggers: Vec, +// on Note: +#[serde(default, skip_serializing_if="Vec::is_empty")] edges: Vec, +---- + +*⚠v2 — load-path rebuild (caught by verification).* `+#[serde(skip)]+` +fields deserialize to `+Default+` (empty). `+rebuild_backlinks+` +(`+notebook.rs:257+`) is *not* called by serde on deserialize — today +only `+from_markdown_vault+` (`+exchange.rs:159+`) calls it. So the +documented load entry point (`+WasmNotebook::from_json+`) *must* call +`+rebuild_indices+` after `+serde_json::from_str+`, or +`+index+`/`+in_edges+`/`+backlinks+` are silently empty after a plain +load (`+PO-3b+`). + +''''' + +=== 4. The native / λδ seam — memory safety and semantic conformance + +==== 4.1 Lemma (No double-borrow during propagation). _Evaluating an exotic-op `+combine+` method inside the sweep cannot panic on `+RefCell+`._ + +*The hazard, precisely.* Host readers do `+nb.borrow()+` (shared); host +mutators do `+nb.borrow_mut()+`. A `+borrow_mut()+` while any shared +borrow is live is a `+BorrowMutError+` *panic*. The sweep holds a shared +borrow (or an owned snapshot); if a λδ `+combine+` method could call a +`+!+` mutator, it would `+borrow_mut()+` and panic. + +*Proof it cannot.* Evaluation contexts form a capability lattice: +`+Reader = {Read}+`, `+Action = {Read, Mutate}+`. Every builtin is +registered with a required capability; all `+!+` builtins require +`+Mutate+`. The dispatcher evaluates `+combine+` in *Reader* context, +whose capability set lacks `+Mutate+`, so any `+!+` builtin call +resolves to a _capability-denied error value_ (not a panic, not a +borrow) *before* any `+borrow_mut()+`. Therefore no `+borrow_mut()+` +occurs during the sweep; only nested shared borrows occur, which +`+RefCell+` permits. ∎ + +*Belt-and-suspenders (recommended).* `+propagate+` operates on the +`+ReasonGraph+` *snapshot* (plain `+Vec+`s), not on `+nb+` directly. +Then exotic `+combine+` methods that merely _read_ the snapshot don’t +even nest a `+RefCell+` borrow. Both mechanisms are cheap; ship both. + +*⚠v2 — three interpreter invariants the proof depends on (made explicit +after verification).* The capability argument is airtight _iff_: *(I1) +capability monotonicity* — a nested `+eval+` (via higher-order builtins +`+map+`/`+reduce+`/`+do+`/`+let+`) inherits the caller’s capability set +and can never _escalate_ to Action; *(I2) totality* — every `+!+` +builtin is registered requiring `+Mutate+` (no unlabelled mutator); +*(I3) no native bypass* — no native fn reachable from `+combine+` calls +`+nb.borrow_mut()+` directly, outside the capability gate. Without I1, a +`+combine+` calling `+(map f xs)+` re-enters eval, and a `+!+` inside +`+f+` under an escalated context would `+borrow_mut+` while the sweep +holds `+borrow()+` → panic. These are interpreter invariants, discharged +by `+PO-4.1b+`. The snapshot removes the hazard only for methods holding +*no* ambient `+Rc>+` handle. + +==== 4.2 Conformance (native table ≡ λδ methods for shipped ops) + +Native `+Op::{And,Or,Not,…}+` and the λδ `+(defmethod combine :and …)+` +etc. both implement one written spec `+S(op)+`. This is discharged as a +*proof obligation*, not a closed proof: a property test over a +deterministic grid `+Gᵏ = ({0, 1/16, …, 1}‥ arity≤4)+` asserts +`+|native(op,x) − eval_λδ(op,x)| ≤ 1e-6+` for every shipped `+op+`. +Passing certifies the seam is invisible: a node routed native vs λδ +yields the same number. Unknown ops route to λδ and are accepted as +slower (§4 of the design). *The dispatch is a total partition* +`+op ↦ Native(f) | Lambda+`, native consulted first, so the two never +both fire for one node (determinism of routing). + +==== 4.3 Domain scalar-isolation invariant (two domains can’t disagree inside one graph) + +.... +INV-DOM ≜ notebook declares ≤1 reasoning :domain D (absent ⇒ fuzzy default) + ∧ ∀ reasoning node n. op(n) ∈ ops(D) ∧ type(n) ∈ types(D) +.... + +Maintained by `+(defmulti validate :type)+` on every +`+set_attr+`/`+link_typed+`/junct creation: an `+:op+` or `+:type+` +outside `+ops(D)/types(D)+` is rejected with a diagnostic. +*Consequence:* `+propagate+` only ever sees operators from a single +algebra with a single scalar semantics (fuzzy `+[0,1]+` vs DS-lite +interval), so a fuzzy `+0.7+` can never be combined with a DS mass +function. Mixed graphs are unrepresentable, not merely discouraged. + +''''' + +=== 5. Trigger cascade termination + +*Default: firing-suppression.* A thread-local guard `+IN_TRIGGER+` is +set while a trigger action runs; host mutators check it and *do not +enqueue* new trigger events while set. *Theorem:* each host-originated +event fires each matching trigger at most once, and with a finite +trigger set the reaction terminates in one pass. *Proof.* No event is +generated during action execution ⇒ the event queue for a tick is fixed +at the tick’s start ⇒ bounded by `+|events| × |triggers|+`. ∎ + +*Optional bounded cascade* (for users who _want_ reactions to react): a +worklist with a `+visited: HashSet<(TriggerId, NoteId)>+` dedup and a +hard depth ceiling `+D+`. Terminates because the state space +`+(TriggerId × NoteId)+` is finite and never revisited, and the +`+Budget+` still bounds each individual action. + +''''' + +=== 6. Master determinism theorem (the keystone) + +*Theorem.* _For a fixed notebook value `+NB+` (fixed ids, text, edges, +attrs), each of `+build_index(NB)+`, `+propagate(NB)+`, +`+reasoning_layout(NB)+`, `+see_also/bm25/classify/duplicates+` is a +pure, deterministic function of `+NB+`, independent of `+HashMap+` +iteration order and of any prior in-memory state._ + +*Proof.* By §1.3 (sorted `+ReasonGraph+`), §2.4 (sorted tie-breaks, +fixed SimHash seed, pure tokenization, no order-bearing `+HashMap+` +iteration), and the observation that `+layout+` consumes the same sorted +`+ReasonGraph+` and the same `+Back+` set as `+propagate+` (so the +dashed-drawn edges are exactly the flow-excluded edges — a shared-source +consistency property). Each stage is a composition of sorted iteration +and pure functions, hence invariant under insertion order; composition +of deterministic functions is deterministic. ∎ + +*Corollaries.* (i) Golden/property tests are stable across runs and +platforms. (ii) The λδ sandbox’s no-RNG/no-clock contract is honored _by +construction_ for all reasoning/intelligence builtins (note _creation_ +uses `+Uuid::new_v4+`/`+Utc::now+`, but that is host-side note +authoring, outside evaluation). (iii) A Web-Worker build produces +bytewise-identical indices to a main-thread build — so §7’s +off-main-thread option in the design is safe to adopt without behavioral +change. + +''''' + +=== 7. Complexity ledger (the claims, now earned) + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Operation |Cost |Earned by +|full index build / rebuild-on-load |`+O(Σ tokens)+` |one pass + +|*incremental reindex (one edit)* |*`+O(tokens in note)+`* |§2.1 +doubly-linked postings — _not_ the sorted-Vec structure + +|See-Also query (exact) |`+O(Σ_{t∈top40} df(t) + Σ_{d∈C}|terms(d)|)+` +|§2.3 + +|BM25 search |`+O(Σ df of query terms)+` |inverted index + +|Classify (Rocchio) |`+O(|terms(d)|·#classes)+` |centroid dot products + +|dedup |`+~O(N)+` via LSH bands |§2.5 + +|full propagation sweep |`+O(V+E)+` |§1.4 single pass + +|spinner-drag recompute |`+O(|cone(d)|)+` |§1.7 + +|index memory @10k |~35–50 MB |dense `+DocId+` (8-byte postings), never +`+Uuid+` +|=== + +''''' + +=== 8. Proof-obligation ledger — the checkable gates Opus makes green + +Each row is a theorem/invariant above and the concrete Rust test that +discharges it. When a row is red, the named invariant is the place to +look. + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|PO |Discharges |Test (property unless noted) +|*PO-1.2* |Reduced-DAG (§1.2) |after `+back_edges+` removal, +`+kahn_topo+` consumes all nodes on random digraphs + +|*PO-1.3* |Determinism of cut (§1.3) |shuffle adjacency _insertion_ +order ⇒ identical `+back_edges+` and `+order+` + +|*PO-1.4* |Order-independence (§1.4) |many random valid topo +linearizations ⇒ identical `+conf+` map + +|*PO-1.4a* |*float-fold determinism (§1.4)* |permute *in-edge insertion +order* for a `+proportion+`/`+prob-sum+`/`+product+` node of in-degree +≥3 ⇒ *bit-identical* `+conf+` (fails unless the fold is +`+(source NoteId, edge-idx)+`-sorted) + +|*PO-1.4b* |`+:not+` arity (§1.4 ⚠) |a `+:not+` node with in-deg≠1 fails +`+validate+`; never reaches `+combine+` + +|*PO-1.5* |weight transform (§1.5) |unit `+w∈{-1,0,1}+`; monotonicity +sign matches `+w+` + +|*PO-1.6* |operator closure (§1.6) |random inputs ⇒ every op output ∈ +`+[0,1]+` + +|*PO-1.6b* |*all-back-edge node (§1.6)* |a node with in-edges but +`+in-deg_{G'}=0+` and `+:confidence=0.8+` reports *0.8* (driver rule), +not 0.5 + +|*PO-1.7* |incremental sweep (§1.7) |full re-sweep `+==+` cone re-sweep +after a random `+set_confidence!+` + +|*PO-2.1* |Invariant L (§2.1) |random ops *incl. a term-drop on a +surviving doc*; `+postings ↔ forward+` mutual inverses; scratch-rebuilt +`+==+` incremental, compared as *canonicalized (sorted) sets* + +|*PO-2.2* |deletion (§2.2) |after random deletes, +`+df+`,`+n_docs+`,postings `+==+` fresh rebuild + +|*PO-2.3* |exact cosine (§2.3) |`+see_also+` scores `+==+` brute-force +dense cosine within `+1e-5+`; folds in `+TermId+` order + +|*PO-2.4* |pipeline determinism (§2.4) |shuffle *note _and_ term +insertion order* ⇒ identical `+see_also/bm25/duplicates+` *scores and +order* + +|*PO-2.5* |dedup recall (§2.5) |blake3-exact dups: *all* returned; +SimHash near-dups: recall *≥ threshold* over many seeded trials (not +"`all`") + +|*PO-2.6* |*classify determinism (§7)* |permute term/centroid order ⇒ +bit-identical Rocchio scores (same fold hazard as PO-1.4a) + +|*PO-3* |serde round-trip (§3) |*semantic* round-trip equality on every +fixture (always); *byte*-identity only after canonical map serialization ++ regenerated goldens + +|*PO-3b* |*load-path rebuild (§3)* |`+WasmNotebook::from_json+` calls +`+rebuild_indices+`; after a plain `+from_str+`, +`+index+`/`+in_edges+`/`+backlinks+` are correct, not empty + +|*PO-4.1* |no double-borrow (§4.1) |a `+combine+` calling a mutator +*directly* returns capability-denied, no panic under a live borrow + +|*PO-4.1b* |*nested-eval capability (§4.1)* |a `+!+` mutator reached via +a higher-order builtin (`+map+`/`+do+`/`+let+`) under Reader context is +still denied (capability monotonicity, I1) + +|*PO-4.2* |native≡λδ conformance (§4.2) |grid test over shipped ops +within `+1e-6+` + +|*PO-4.3* |domain isolation (§4.3) |a foreign `+:op+`/`+:type+` is +rejected; a mixed-scalar graph is unconstructable + +|*PO-5* |trigger termination (§5) |self-retriggering action terminates; +bounded cascade respects depth `+D+` + +|*PO-5b* |*trigger order (§5)* |firing order is `+triggers+`-`+Vec+` +order ⇒ final state deterministic for order-sensitive actions + +|*PO-6* |master determinism (§6) |end-to-end: shuffle *note + term + +in-edge* insertion ⇒ identical index+conf+layout *bytes* (requires +PO-1.4a, PO-2.4, PO-3 canonical) +|=== + +''''' + +=== 9. Handoff note to Opus + +Implement in this order, each PR gated on its ledger rows: + +[arabic] +. *`+index.rs+` (headless).* Structure per §2.1 (doubly-linked postings +— do *not* use sorted `+Vec+`s), lifecycle §2.2, exact query §2.3, +determinism §2.4. Gate: PO-2.__, PO-3, PO-6(index part). No WASM, no UI. +__This is the whole DEVONthink side de-risked before a pixel ships.* +. *`+edge.rs+` + serde.* Additive `+edges+`; `+in_edges+` and `+index+` +`+#[serde(skip)]+` (§3 ⚠). Gate: PO-3. +. *`+reason.rs+`.* `+ReasonGraph+` (sorted, iterative DFS), +`+back_edges+`, `+kahn_topo+`, `+weight_edge+`, native `+combine+`, +`+propagate+`, cone cache. Gate: PO-1.*. +. *λδ seam.* Reader-context `+combine+` dispatch (§4.1), conformance +grid (§4.2), `+validate+` (§4.3). Gate: PO-4.*. +. *`+trigger.rs+`.* Suppression guard (§5). Gate: PO-5. +. *`+layout.rs+` + `+ReasoningView+`, See-Also/Duplicates panels.* +Consume `+back_edges+` from `+reason.rs+` for dashed edges (§6 +consistency). UI only after 1–5 are green. + +*Blocking prerequisites (fix in the spec before step 1 — the v2 +corrections):* 1. *Pin every float fold* in key-sorted order — +`+combine+` in-edges by `+(source NoteId, edge-idx)+`; cosine numerator ++ both norms + classify centroids by `+TermId+` +(`+PO-1.4a+`/`+PO-2.4+`/`+PO-2.6+`). Without this, `+PO-6+` is +unachievable. 2. *Canonical map serialization* (`+serialize_with+` +sorted, or `+BTreeMap+`) + regenerated goldens — _or_ consciously relax +`+PO-3+` to semantic equality (`+PO-3+`). 3. *Two-sided posting +`+remove+`* so reindex preserves Invariant L (`+PO-2.1+`). 4. *Load path +calls `+rebuild_indices+`* so derived state isn’t empty after a plain +deserialize (`+PO-3b+`). + +These four are the difference between "`compiles and passes`" and +"`flakes on day one.`" Everything else is a caveat the ledger absorbs. + +The math is settled above. Opus’s job is faithful translation + turning +the ledger green — not rediscovery. diff --git a/docs/design/flyinglogic-devonthink-proofs.md b/docs/design/flyinglogic-devonthink-proofs.md deleted file mode 100644 index b757e73..0000000 --- a/docs/design/flyinglogic-devonthink-proofs.md +++ /dev/null @@ -1,328 +0,0 @@ - -# Nexia-List — Hard Problems, Solved, with Proof Designs - -*The fable-tier work. Everything below is the reasoning that is expensive to get right and cheap to get subtly wrong. It is written so that a lower-tier implementer (Opus) can translate each module into Rust/ReScript against a fixed spec and discharge a checkable **proof-obligation ledger** (§8) — without re-deriving any mathematics.* - -Companion to: the FL×DT integration design and the mind-management plan. Assumes their decisions (additive `edges` channel; derived-index-beside-the-notes discipline; native operator table + λδ `combine` for exotic ops; `.ld` domains). - -> **v2 — corrections after an adversarial proof-check (Opus) against the real `core/`.** Four items were materially wrong in v1 and are fixed inline (each tagged ⚠v2): **(1)** float folds must be pinned in key-sorted order — IEEE non-associativity means "sort the outputs" is *not* enough for determinism (§1.4, §2.4, §6); **(2)** byte-identical serde is false until the pre-existing `notes`/`backlinks`/`attributes` `HashMap`s serialize in canonical key order (§3); **(3)** the `O(1)` posting `remove` must be *two-sided* or reindex corrupts Invariant L (§2.1); **(4)** the driver/driven boundary is strictly by `G'` in-degree — `combine(∅)` is dead code (§1.4/§1.6). The mathematical core (§1.2–1.7, §2.2–2.3 ideas, the `in_edges`/`:not` catches) survived. The ledger (§8) gained the discriminating tests. - ---- - -## 0. The delegation contract - -| Tier | Owns | Must NOT need to do | -|---|---|---| -| **fable (this document)** | the data-structure *shapes* that make the complexity claims true; the invariants; the theorems and their proofs; the caught bugs; the proof-obligation ledger | — | -| **Opus (implementation)** | translate each `§ spec` block into Rust/ReScript honouring the stated invariants; write the test named in each `PO-*` ledger row until green; wire WASM entry points and TEA views | re-derive the math, invent the data structure, or decide the semantics | - -**Reading order for Opus:** for each module, read the *Invariant*, then the *Spec*, then implement, then make the *Proof Obligations* pass. The proofs here exist so that when a test fails, the implementer knows which invariant was violated rather than guessing. - -Three bugs in the prior design were found while proving it; they are called out inline as **⚠ CAUGHT**. - ---- - -## 1. Confidence propagation — the reasoning engine - -The claim to be justified: *one deterministic pass computes a well-defined confidence for every node; feedback loops are drawn but never break evaluation; a spinner drag recomputes only what can change.* - -### 1.1 Setup and notation - -- The reasoning graph is `G = (V, E)` where `V` = notes and `E` = typed edges (`Note.edges`), each edge `(u→v)` carrying weight `w ∈ [-1,1]`. `links` do **not** appear here. -- A node is a **driver** if it has no in-edges in the (reduced) graph; otherwise **driven**. -- `conf : V → [0,1]`. `0.5` is the distinguished value *Indeterminate*. -- `asserted(v) ∈ [0,1]` is a driver's stored `:confidence` (default `0.5`). -- `op(v)` is the node's combining operator (default `:and`; a junct carries `:op`). - -### 1.2 Theorem (Reduced-DAG). *Removing the canonical DFS back-edge set from `G` yields a DAG.* - -**Proof.** Run DFS over `G`. Classify each edge as tree / forward / cross / back, where `(u→v)` is **back** iff `v` is on the recursion stack (grey) when the edge is explored — i.e. `v` is a DFS-ancestor of `u`. Assign finish times `f`. For every non-back edge `(u→v)`, standard DFS parenthesis theory gives `f(u) > f(v)`. Let `E' = E \ Back`. Order `V` by *descending* `f`. Every edge of `E'` then goes from an earlier to a later vertex, so this is a topological order of `(V, E')`; a graph admitting a topological order is acyclic. ∎ - -### 1.3 Theorem (Determinism of the cut). *With a canonical DFS the back-edge set — hence `E'`, the topo order, and every `conf(v)` — is a pure function of the notebook.* - -**Why it is not free:** which edge of a cycle is the back edge depends on DFS visitation order, and Rust `HashMap` iteration order is unspecified (and, in `wasm32`, must never be relied on). - -**Proof / construction.** Build `ReasonGraph` as **sorted vectors**, never a `HashMap` walked for order: -- nodes in ascending `NoteId` order (`Uuid` has a total order); -- each adjacency list sorted by `(target NoteId, source edge index)`. -DFS pushes roots and neighbours in that fixed order. Finish times, and therefore `Back`, are now a deterministic function of the sorted structure, which is itself a deterministic function of the notebook value. Every downstream quantity is a pure function of `Back`. ∎ - -> **§ spec.** `ReasonGraph::build(nb)` returns `{ ids: Vec (sorted), idx: HashMap, adj: Vec> (each sorted) }`. `back_edges()` is an iterative DFS over `adj` using an explicit `Vec` stack and a `Vec` — no recursion (WASM stack safety), no `HashMap` iteration. - -### 1.4 Theorem (Well-defined, order-independent values). *On `G' = (V, E')` the assignment* -``` -conf(v) = asserted(v) if in-deg_{G'}(v) = 0 -conf(v) = combine(op(v), { (weight_edge(conf(u), w), w) : (u→v) ∈ E' }) otherwise -``` -*has a unique solution, and one Kahn sweep computes it regardless of which valid topological order Kahn happens to pick.* - -**Proof.** `G'` is a DAG (§1.2), so the definition is a well-founded recursion: `conf(v)` refers only to `conf(u)` for strict predecessors `u`, and the predecessor relation is a strict partial order with no infinite descending chains. Well-founded recursion has a unique total solution. Kahn processes each `v` only after all predecessors are finalized, so it evaluates exactly the recursion. Uniqueness of the *solution* (not merely of one run) gives order-independence: any linear extension yields the same `conf`, because each `conf(v)` is fixed by the values of its predecessors, which are fixed inductively from the drivers up. ∎ - -**Determinism caveat — `combine` must be a *canonical* function of the inputs.** Two distinct hazards: - -- *Multiset symmetry.* min, max, `1−x`, fuzzy-xor are order-exact. -- **⚠v2 CAUGHT (float non-associativity).** IEEE `+`/`×` are commutative but **not associative**, so `prob-sum` (`1−∏(1−xᵢ)`), `product` (`∏xᵢ`) and `proportion` (`Σ|wᵢ|xᵢ / Σ|wᵢ|`) are *not* functions of the input multiset in `f32`: a different fold order yields a ULP-different result, and a single ULP flips a `total_cmp` tie in a ranked list (§2.4) or changes a serialized `conf` byte (§6). **Mandatory spec line:** `combine` folds its inputs in **sorted `(source NoteId, source edge-index)` order** — a pure function of the notebook. (`min/max/not/fuzzy-xor` are order-exact and need no pinning, but pinning all uniformly is simplest.) This is what makes `PO-1.4a`/`PO-6` achievable; without it they fail intermittently. - -`:not` is unary — the remaining symmetry exception: - -> **⚠ CAUGHT (arity).** `:not` is unary. A `:not` node with in-degree ≠ 1 is undefined. Resolution: `validate` (§4.3) rejects it at edit time; `combine(:not, xs)` asserts `xs.len()==1` and is only ever reached for validated graphs. Document `:not` as strictly unary in `flying-logic.ld`. - -### 1.5 Lemma (Edge-weight transform). *`weight_edge(v, w) = ((2v−1)·w + 1)/2` maps `[0,1]×[-1,1] → [0,1]` and realizes pass-through / abstain / negation.* - -**Proof.** `2v−1 ∈ [-1,1]`; with `|w|≤1`, `(2v−1)w ∈ [-1,1]`; `+1 → [0,2]`; `/2 → [0,1]` (closure). Substituting: `w=1 ⇒ v` (identity); `w=0 ⇒ 0.5` (abstain — a zero-weight edge contributes Indeterminate, i.e. *no information*); `w=−1 ⇒ 1−v` (negation). `∂/∂v = w`, so monotone increasing for `w>0`, decreasing for `w<0`, flat at `w=0`. ∎ - -### 1.6 Lemma (Operator closure). *Every shipped operator maps `[0,1]^k → [0,1]`.* - -**Proof (the only non-obvious cases).** -- **prob-sum** `⊕`: `a⊕b = a + b − ab = 1 − (1−a)(1−b)`. For `a,b∈[0,1]`, `(1−a)(1−b)∈[0,1]`, so `a⊕b∈[0,1]`; associativity gives the `k`-ary form `1 − ∏(1−xᵢ) ∈ [0,1]`. ∎ -- **proportion** `∷`: `Σ|wᵢ|xᵢ / Σ|wᵢ|` is a convex combination of values in `[0,1]` ⇒ in `[0,1]`; `Σ|wᵢ|=0 ⇒ 0.5` by definition. ∎ -- **fuzzy-xor**: `maxᵢ min(xᵢ, 1 − max_{j≠i} xⱼ)` — a max of mins of values in `[0,1]` ⇒ in `[0,1]`. ∎ -- min / max / `1−x` / product: closure immediate. - -**⚠v2 — driver/driven boundary (resolving a v1 inconsistency).** Classification is **strictly by `G'` (post-cut) in-degree**: a node with `in-deg_{G'}(v) = 0` is a **driver** and takes its `asserted` value (default `0.5`) — *even if it had in-edges in `G` that were all back-edges*. Consequently `combine` is only ever invoked with ≥1 surviving input; `combine(∅)` is **unreachable dead code** (if defensively kept, return `0.5`). Practical upshot: a note carrying `:confidence = 0.8` that is targeted only by cyclic edges reports **0.8**, not `0.5` — its assertion is honoured, not silently discarded (`PO-1.6b`). - -### 1.7 Theorem (Incremental recompute — the spinner-drag correctness). *If only driver `d`'s `asserted` value changes, the set of nodes whose `conf` can change is exactly the descendants of `d` in `G'`. Re-sweeping `descendants(d)` in topo order recomputes them correctly and leaves all other nodes untouched.* - -**Proof.** `conf(v)` is determined (§1.4) by the asserted values of the drivers that are ancestors of `v`. If `d ∉ ancestors(v)` then none of `v`'s determining inputs changed, so by induction over the topo order restricted to non-descendants, `conf(v)` is unchanged. Conversely every `v ∈ descendants(d)` has `d` as an ancestor and may change. Re-sweeping precisely `descendants(d)` (a subset closed under "successor", processed in the global topo order) evaluates the same recursion for exactly the affected nodes. ∎ - -> **§ spec.** Cache `order: Vec` and, per driver, `cone: Vec` (its forward-reachable set, computed lazily by BFS over `adj` and memoized until the edge set changes). `set_confidence!(d,x)` → re-sweep `cone(d) ∩ order` in order. A full sweep is `O(V+E)` (~40k ops at 10k notes/30k edges) — sub-millisecond in WASM — so the cone optimization is a UX nicety for very large graphs, not a correctness dependency. - -### 1.8 Semantics choice (why fuzzy min/max weakest-link is the default), stated as properties - -The default algebra is the one family that is simultaneously: **(P1)** closed on `[0,1]`, **(P2)** symmetric (so §1.4 determinism holds), **(P3)** evaluable in a single pass with no fixpoint (min/max are idempotent and need no iteration to converge), and **(P4)** has a neutral `0.5` that is a genuine "unknown" rather than a false `0`. Probability (`×`, `⊕`) and DS-lite intervals are opt-in per-domain (§4.3) precisely because they trade one of these away (e.g. `×` is not idempotent, so repeated evidence double-counts — desirable sometimes, wrong as a default). - ---- - -## 2. The similarity index — data-structure invariants and incremental correctness - -The claim to be justified: *one inverted index supports See-Also / BM25 / Classify / dedup; a single-note edit costs `O(tokens in that note)`, not `O(corpus)`; results are exact and deterministic; deletion is handled.* - -### 2.1 The `O(tokens)` reindex — the doubly-linked posting structure (the real trick) - -> **⚠ CAUGHT (complexity).** The prior design stores postings "sorted by doc" and claims `O(tokens)` incremental reindex. Removing one doc's posting from a sorted `Vec` is `O(df)` (shift), and there is no way to *find* it without an `O(df)` scan — so the honest cost of the naive structure is `O(Σ_t df(t))`, which for common terms is `O(N)`, not `O(tokens)`. The `O(tokens)` claim is only recoverable with a different structure. - -**The fix — unsorted postings with two-way back-links.** See-Also and BM25 both accumulate scores into a `HashMap` by *iterating* a term's postings; neither needs them sorted (only positional phrase/proximity needs per-doc position lists, which live elsewhere). So postings can be unsorted, which unlocks `O(1)` add/remove: - -```rust -struct Posting { doc: DocId, tf: u32, fwd: u32 } // fwd = index into forward[doc] -struct FwdEntry { term: TermId, tf: u32, slot: u32 } // slot = index into postings[term] - -// INVARIANT L (link): for every term t and every slot s, -// let p = postings[t][s]; forward[p.doc][p.fwd] == FwdEntry{ term: t, .., slot: s } -// and symmetrically for every forward entry. The two arrays are mutual inverses. -``` - -- **add(d,t,tf):** `s = postings[t].len(); j = forward[d].len();` push `Posting{d,tf,fwd:j}`; push `FwdEntry{t,tf,slot:s}`; `df[t]+=1`. `O(1)`. -- **remove(d, term at forward index `j`):** **⚠v2 — this is a *two-sided* swap-remove.** The arrays are mutual inverses, so *both* must be fixed: - 1. *postings side:* `s = forward[d][j].slot`; `swap_remove(postings[term], s)`; if a posting `P` moved into slot `s`, set `forward[P.doc][P.fwd].slot = s`. - 2. *forward side:* `swap_remove(forward[d], j)`; if a `FwdEntry Q` moved into index `j`, set `postings[Q.term][Q.slot].fwd = j`. - `df[term] -= 1`. Both fixups are `O(1)` (each guarded by a "was anything actually moved?" check for the last-element case). **Omitting step 2 corrupts Invariant L on the *reindex* path** (a surviving doc dropping a term); §2.2 deletion hides the omission only because it `clear()`s `forward[d]` wholesale (`PO-2.1`). - -Therefore `reindex` = (diff old vs new token bag) then `O(1)` per changed `(doc,term)` = **`O(|B_old| + |B_new|)`**, genuinely `O(tokens in the edited note)`. Invariant L is the checkable contract (PO-2.1). - -### 2.2 DocId lifecycle under deletion (the gap the design left open) - -**Decision: tombstone during a session, compact on load.** DocIds are allocated append-only (`0,1,2,…`) as notes are indexed. Deleting note `d`: -1. for each `FwdEntry` in `forward[d]`: `remove(d, term)` (§2.1), which maintains `df`; -2. `forward[d].clear(); note_of[d] = None; doc_of.remove(¬e_id); simhash[d] = DEAD; n_docs -= 1;` -3. DocId `d` is **never reused** within the session. - -**Theorem (deletion preserves INV-IDX).** After deletion, (a) no posting references `d`; (b) `df[t]` equals the number of live docs containing `t`; (c) `n_docs` equals the live count; (d) every other doc's postings are unchanged *in content* (swap-remove only relabels slots, preserving Invariant L). **Proof.** (a) every term of `d` had its posting removed in step 1; (b) each removal decremented exactly the terms `d` contained; (c) step 2; (d) `swap_remove` moves a posting but §2.1's fixup restores Invariant L, and its `(doc,tf)` payload is untouched. ∎ - -**Memory bound.** Tombstones (holes in `note_of`) number at most the session's deletions. Because the index is `#[serde(skip)]` and **rebuilt on load** from the sorted live notes, DocIds are re-densified every session ⇒ holes never accumulate across sessions. This is exactly the `backlinks` discipline: *derived, never trusted from disk, rebuilt on load.* - -### 2.3 Retiring the idf/norm-drift tension — exact cosine at query time - -> **⚠ CAUGHT (the drift was avoidable).** The prior design caches `doc_norm` under stale idf and then argues the resulting ranking error is "imperceptible." We can do better: compute the exact cosine and delete the tension. - -`cos(q,d) = ⟨w_q, w_d⟩ / (‖w_q‖·‖w_d‖)` with tf-idf weight `w(t) = (1+ln tf)·ln(N/df(t))` under the **current** `N, df`. See-Also already forms the numerator `⟨w_q,w_d⟩` by walking `q`'s top terms' postings and accumulating into `score[d]`, all with current idf. The only question is the denominator norms: - -- `‖w_q‖` — computed once from `q`'s forward vector under current idf: `O(|terms(q)|)`. -- `‖w_d‖` for each candidate `d` — computed from `d`'s forward vector under current idf: `O(|terms(d)|)`. - -**Theorem (exactness).** Computing numerator and both norms all under the current `(N, df)` yields the exact current cosine; ranking candidates by it yields the exact top-k. **Proof.** Immediate — every term of the definition is evaluated under one consistent idf. ∎ - -**Cost — the honest bound (⚠v2).** Let `C` = candidate set (docs sharing one of `q`'s top-40 terms after dropping terms with `df > N/2`). Extra work is `Σ_{d∈C}|terms(d)|`. **Worst case is `O(N)` / `O(corpus)`**: `|C| ≤ 40·(N/2) = O(N)`, so exact-at-query-time is an *average-case* win (real top terms are discriminative ⇒ `C ≪ N`), **not** a worst-case guarantee. **Guaranteed path:** when `Σ_{t∈top40} df(t)` exceeds a fixed budget `B`, fall back to cached `doc_norm` so the query is always `O(B)`; also tighten the stop rule (absolute posting cap, or `df > N/10`). The fallback error is a per-doc scaling: `score_cached(d) = score_exact(d)·ρ_d`, `ρ_d = ‖w_d‖_cache/‖w_d‖_exact → 1` as term `df` grows, `|ρ_d−1| = O(E/(df_min·ln(N/df)))`, `E` = edits since that doc's last reindex (reset to 0 on load). So: **exact under budget `B`, cached-norm above it** — the fallback is a guaranteed path, not merely a profiling flag. Norm and numerator folds are `TermId`-ordered (§2.4). - -### 2.4 Determinism of the intelligence pipeline - -Every order-sensitive step is pinned: -- **tokenization** is pure (Unicode segmentation → lowercase → static stop-list → optional Porter stem; all fixed data); -- **SimHash** uses a **fixed seed** (no RNG); -- **See-Also / BM25** sort by `score` then `NoteId` tie-break: `b.score.total_cmp(&a.score).then(a.id.cmp(&b.id))`; -- **no `HashMap` is iterated** where output order matters (accumulate into a map, then sort the entries); -- **⚠v2 — float reductions fold in key-sorted order** (the subtle one): the cosine numerator `Σ_t w_q(t)·w_d(t)` and both norms `Σ_t w(t)²` iterate terms in **`TermId` order** (and the top-40 cut breaks ties by `TermId`); `combine`'s `proportion`/`prob-sum`/`product` fold in-edges in `(source NoteId, edge-index)` order (§1.4). Because IEEE `+`/`×` are non-associative, **sorting the *outputs* is not sufficient — the *accumulation* order must itself be a key-sorted, notebook-pure function**, or a 1-ULP difference flips a `total_cmp` tie between two near-equal docs and breaks `PO-2.4`/`PO-6`. Classify's centroid dot-products fold the same way (`PO-2.6`). - -Consequence (with §1.3): the entire index is a pure function of the live note set, independent of insertion order — golden tests are stable and the λδ no-RNG/no-clock contract holds. - -### 2.5 Near-duplicate recall (SimHash + LSH), as a bound not a hope - -64-bit SimHash; near-dup iff Hamming distance `≤ h₀` (e.g. 3). Split the 64 bits into `b` bands of `r = 64/b` bits; two docs are LSH-candidates iff some band is identical. For a pair at Hamming distance `h`, the probability a given band matches is `((64−h)/64)^r`… (standard bit-sampling bound); with `b=8, r=8`, pairs at `h≤3` collide in ≥1 band with probability `≥ 1 − (1 − (61/64)^8)^8 ≈ 0.999`. Choose `(b,r)` from the target `h₀` and acceptable false-negative rate; exact duplicates (Hamming 0) are caught additionally by blake3 content-hash buckets (zero false negatives). Determinism: fixed seed + `NoteId`-sorted candidate output. - -### 2.6 Master index invariant + per-op maintenance - -``` -INV-IDX ≜ Invariant L (§2.1) (postings ↔ forward are inverses) - ∧ ∀t. df[t] = #{ live d : t ∈ bag(d) } - ∧ n_docs = #{ live d } - ∧ ∀ live d. forward[d] = tfidf-bag(title(d) ++ content(d)) (under tokenization) - ∧ ∀ live d. simhash[d] = SimHash(bag(d)) (fixed seed) -``` -| Op | Maintenance | -|---|---| -| `create` | allocate DocId, `add` each term, set `simhash`, `n_docs+=1` | -| `set_content` | `reindex(id, title(id), new_content)` (diff) | -| `set_title` (**incl. `update_title`/`bi_set_title`**) | `reindex(id, new_title, content(id))` — **⚠ CAUGHT earlier:** titles dominate short-note similarity; both title write paths must call reindex, or See-Also silently desyncs | -| `delete` | §2.2 | -| `set_attr` / link ops | no index change (attrs/links are not indexed text) | - ---- - -## 3. Serde back-compat — round-trip fidelity, and the two things that break byte-identity - -> **⚠v2 — revised after verification.** The v1 claim ("any current notebook round-trips byte-identically once new fields carry the right attributes") is **false as stated**, for a reason *independent of the new fields*. Corrected below. - -**What is actually required (and provable).** Existing notebooks must (a) **load without loss** under the extended schema, and (b) gain **no new JSON keys** while their new fields are empty/derived. This holds iff every new field is `#[serde(skip)]` (derived) or `#[serde(default, skip_serializing_if="…is_empty")]`, declared after the existing fields. - -**Proof (of the required property).** A missing key deserializes via `default`/`skip` (no effect on siblings); an empty/derived value is omitted on serialize (no new key). ∎ This is sufficient for correctness and for forward/backward loading — it is the property Opus must guarantee. - -**Why *byte*-identity needs one more fix (⚠v2 — the bigger bug).** The *existing* struct already serializes three `HashMap`s in **iteration order**: -- `notebook.rs:26` `notes: HashMap` -- `notebook.rs:29` `backlinks: HashMap>` -- `note.rs:70` `attributes: HashMap` - -`serde_json` emits maps in iteration order, and `std::HashMap`/`HashSet` iterate in a per-instance seed-dependent order, so `serialize → deserialize → serialize` **permutes keys** for any map with ≥2 entries (deserialize builds a fresh map with a fresh seed). Byte-identity was therefore **never actually held** — the current round-trip test (`notebook.rs:394`) checks only semantic fields, never bytes. Adding correctly-attributed new fields is *necessary but not sufficient*. - -**Fix (only if byte-level golden gates are wanted).** Serialize the three maps in **canonical key order** — least-invasive is `#[serde(serialize_with = "sorted_map")]` (keeps `HashMap` in memory, sorts on the way out); alternatively switch to `BTreeMap`/`BTreeSet` (larger blast radius — touches `note.rs`, `agent.rs::attribute_equals`, `exchange.rs`) — and **regenerate the golden fixtures** with the canonical serializer. - -**Theorem (corrected).** *Under canonical map ordering, any current notebook round-trips byte-identically once new fields carry the attributes below.* Proof exactly as above, now that key order is a pure function of the key set. ∎ - -If byte-level gates are *not* wanted, relax `PO-3` to **semantic** round-trip equality (the property that truly matters) and skip the canonical-serialization change. **Recommendation:** adopt `serialize_with` canonical ordering — cheap, makes golden diffs meaningful, and removes a latent cross-platform flake. - -**The `in_edges` sub-bug (real, independent of the above).** The prior delta declared `in_edges` with `#[serde(default)]` and **no** `skip_serializing_if`; an empty map still serializes as `"in_edges":{}`, injecting a key. `backlinks` (`notebook.rs:29`, `#[serde(default)]`, **persisted**) is living proof of exactly this behaviour. Fix — `in_edges` is derived, so `#[serde(skip)]` it and rebuild. **Wording correction:** this *deviates* from the `backlinks` discipline (which *is* persisted), rather than "following it" — we are choosing not to persist any derived reverse index. `#[serde(skip)]` also requires `SimilarityIndex: Default`. - -```rust -#[serde(skip)] index: SimilarityIndex, // derived; needs Default -#[serde(skip)] in_edges: HashMap>, // ← was the bug -#[serde(default, skip_serializing_if="Vec::is_empty")] triggers: Vec, -// on Note: -#[serde(default, skip_serializing_if="Vec::is_empty")] edges: Vec, -``` - -**⚠v2 — load-path rebuild (caught by verification).** `#[serde(skip)]` fields deserialize to `Default` (empty). `rebuild_backlinks` (`notebook.rs:257`) is **not** called by serde on deserialize — today only `from_markdown_vault` (`exchange.rs:159`) calls it. So the documented load entry point (`WasmNotebook::from_json`) **must** call `rebuild_indices` after `serde_json::from_str`, or `index`/`in_edges`/`backlinks` are silently empty after a plain load (`PO-3b`). - ---- - -## 4. The native / λδ seam — memory safety and semantic conformance - -### 4.1 Lemma (No double-borrow during propagation). *Evaluating an exotic-op `combine` method inside the sweep cannot panic on `RefCell`.* - -**The hazard, precisely.** Host readers do `nb.borrow()` (shared); host mutators do `nb.borrow_mut()`. A `borrow_mut()` while any shared borrow is live is a `BorrowMutError` **panic**. The sweep holds a shared borrow (or an owned snapshot); if a λδ `combine` method could call a `!` mutator, it would `borrow_mut()` and panic. - -**Proof it cannot.** Evaluation contexts form a capability lattice: `Reader = {Read}`, `Action = {Read, Mutate}`. Every builtin is registered with a required capability; all `!` builtins require `Mutate`. The dispatcher evaluates `combine` in **Reader** context, whose capability set lacks `Mutate`, so any `!` builtin call resolves to a *capability-denied error value* (not a panic, not a borrow) **before** any `borrow_mut()`. Therefore no `borrow_mut()` occurs during the sweep; only nested shared borrows occur, which `RefCell` permits. ∎ - -**Belt-and-suspenders (recommended).** `propagate` operates on the `ReasonGraph` **snapshot** (plain `Vec`s), not on `nb` directly. Then exotic `combine` methods that merely *read* the snapshot don't even nest a `RefCell` borrow. Both mechanisms are cheap; ship both. - -**⚠v2 — three interpreter invariants the proof depends on (made explicit after verification).** The capability argument is airtight *iff*: **(I1) capability monotonicity** — a nested `eval` (via higher-order builtins `map`/`reduce`/`do`/`let`) inherits the caller's capability set and can never *escalate* to Action; **(I2) totality** — every `!` builtin is registered requiring `Mutate` (no unlabelled mutator); **(I3) no native bypass** — no native fn reachable from `combine` calls `nb.borrow_mut()` directly, outside the capability gate. Without I1, a `combine` calling `(map f xs)` re-enters eval, and a `!` inside `f` under an escalated context would `borrow_mut` while the sweep holds `borrow()` → panic. These are interpreter invariants, discharged by `PO-4.1b`. The snapshot removes the hazard only for methods holding **no** ambient `Rc>` handle. - -### 4.2 Conformance (native table ≡ λδ methods for shipped ops) - -Native `Op::{And,Or,Not,…}` and the λδ `(defmethod combine :and …)` etc. both implement one written spec `S(op)`. This is discharged as a **proof obligation**, not a closed proof: a property test over a deterministic grid `Gᵏ = ({0, 1/16, …, 1}‥ arity≤4)` asserts `|native(op,x) − eval_λδ(op,x)| ≤ 1e-6` for every shipped `op`. Passing certifies the seam is invisible: a node routed native vs λδ yields the same number. Unknown ops route to λδ and are accepted as slower (§4 of the design). **The dispatch is a total partition** `op ↦ Native(f) | Lambda`, native consulted first, so the two never both fire for one node (determinism of routing). - -### 4.3 Domain scalar-isolation invariant (two domains can't disagree inside one graph) - -``` -INV-DOM ≜ notebook declares ≤1 reasoning :domain D (absent ⇒ fuzzy default) - ∧ ∀ reasoning node n. op(n) ∈ ops(D) ∧ type(n) ∈ types(D) -``` -Maintained by `(defmulti validate :type)` on every `set_attr`/`link_typed`/junct creation: an `:op` or `:type` outside `ops(D)/types(D)` is rejected with a diagnostic. **Consequence:** `propagate` only ever sees operators from a single algebra with a single scalar semantics (fuzzy `[0,1]` vs DS-lite interval), so a fuzzy `0.7` can never be combined with a DS mass function. Mixed graphs are unrepresentable, not merely discouraged. - ---- - -## 5. Trigger cascade termination - -**Default: firing-suppression.** A thread-local guard `IN_TRIGGER` is set while a trigger action runs; host mutators check it and **do not enqueue** new trigger events while set. **Theorem:** each host-originated event fires each matching trigger at most once, and with a finite trigger set the reaction terminates in one pass. **Proof.** No event is generated during action execution ⇒ the event queue for a tick is fixed at the tick's start ⇒ bounded by `|events| × |triggers|`. ∎ - -**Optional bounded cascade** (for users who *want* reactions to react): a worklist with a `visited: HashSet<(TriggerId, NoteId)>` dedup and a hard depth ceiling `D`. Terminates because the state space `(TriggerId × NoteId)` is finite and never revisited, and the `Budget` still bounds each individual action. - ---- - -## 6. Master determinism theorem (the keystone) - -**Theorem.** *For a fixed notebook value `NB` (fixed ids, text, edges, attrs), each of `build_index(NB)`, `propagate(NB)`, `reasoning_layout(NB)`, `see_also/bm25/classify/duplicates` is a pure, deterministic function of `NB`, independent of `HashMap` iteration order and of any prior in-memory state.* - -**Proof.** By §1.3 (sorted `ReasonGraph`), §2.4 (sorted tie-breaks, fixed SimHash seed, pure tokenization, no order-bearing `HashMap` iteration), and the observation that `layout` consumes the same sorted `ReasonGraph` and the same `Back` set as `propagate` (so the dashed-drawn edges are exactly the flow-excluded edges — a shared-source consistency property). Each stage is a composition of sorted iteration and pure functions, hence invariant under insertion order; composition of deterministic functions is deterministic. ∎ - -**Corollaries.** (i) Golden/property tests are stable across runs and platforms. (ii) The λδ sandbox's no-RNG/no-clock contract is honored *by construction* for all reasoning/intelligence builtins (note *creation* uses `Uuid::new_v4`/`Utc::now`, but that is host-side note authoring, outside evaluation). (iii) A Web-Worker build produces bytewise-identical indices to a main-thread build — so §7's off-main-thread option in the design is safe to adopt without behavioral change. - ---- - -## 7. Complexity ledger (the claims, now earned) - -| Operation | Cost | Earned by | -|---|---|---| -| full index build / rebuild-on-load | `O(Σ tokens)` | one pass | -| **incremental reindex (one edit)** | **`O(tokens in note)`** | §2.1 doubly-linked postings — *not* the sorted-Vec structure | -| See-Also query (exact) | `O(Σ_{t∈top40} df(t) + Σ_{d∈C}|terms(d)|)` | §2.3 | -| BM25 search | `O(Σ df of query terms)` | inverted index | -| Classify (Rocchio) | `O(|terms(d)|·#classes)` | centroid dot products | -| dedup | `~O(N)` via LSH bands | §2.5 | -| full propagation sweep | `O(V+E)` | §1.4 single pass | -| spinner-drag recompute | `O(|cone(d)|)` | §1.7 | -| index memory @10k | ~35–50 MB | dense `DocId` (8-byte postings), never `Uuid` | - ---- - -## 8. Proof-obligation ledger — the checkable gates Opus makes green - -Each row is a theorem/invariant above and the concrete Rust test that discharges it. When a row is red, the named invariant is the place to look. - -| PO | Discharges | Test (property unless noted) | -|---|---|---| -| **PO-1.2** | Reduced-DAG (§1.2) | after `back_edges` removal, `kahn_topo` consumes all nodes on random digraphs | -| **PO-1.3** | Determinism of cut (§1.3) | shuffle adjacency *insertion* order ⇒ identical `back_edges` and `order` | -| **PO-1.4** | Order-independence (§1.4) | many random valid topo linearizations ⇒ identical `conf` map | -| **PO-1.4a** | **float-fold determinism (§1.4)** | permute **in-edge insertion order** for a `proportion`/`prob-sum`/`product` node of in-degree ≥3 ⇒ **bit-identical** `conf` (fails unless the fold is `(source NoteId, edge-idx)`-sorted) | -| **PO-1.4b** | `:not` arity (§1.4 ⚠) | a `:not` node with in-deg≠1 fails `validate`; never reaches `combine` | -| **PO-1.5** | weight transform (§1.5) | unit `w∈{-1,0,1}`; monotonicity sign matches `w` | -| **PO-1.6** | operator closure (§1.6) | random inputs ⇒ every op output ∈ `[0,1]` | -| **PO-1.6b** | **all-back-edge node (§1.6)** | a node with in-edges but `in-deg_{G'}=0` and `:confidence=0.8` reports **0.8** (driver rule), not 0.5 | -| **PO-1.7** | incremental sweep (§1.7) | full re-sweep `==` cone re-sweep after a random `set_confidence!` | -| **PO-2.1** | Invariant L (§2.1) | random ops **incl. a term-drop on a surviving doc**; `postings ↔ forward` mutual inverses; scratch-rebuilt `==` incremental, compared as **canonicalized (sorted) sets** | -| **PO-2.2** | deletion (§2.2) | after random deletes, `df`,`n_docs`,postings `==` fresh rebuild | -| **PO-2.3** | exact cosine (§2.3) | `see_also` scores `==` brute-force dense cosine within `1e-5`; folds in `TermId` order | -| **PO-2.4** | pipeline determinism (§2.4) | shuffle **note *and* term insertion order** ⇒ identical `see_also/bm25/duplicates` **scores and order** | -| **PO-2.5** | dedup recall (§2.5) | blake3-exact dups: **all** returned; SimHash near-dups: recall **≥ threshold** over many seeded trials (not "all") | -| **PO-2.6** | **classify determinism (§7)** | permute term/centroid order ⇒ bit-identical Rocchio scores (same fold hazard as PO-1.4a) | -| **PO-3** | serde round-trip (§3) | **semantic** round-trip equality on every fixture (always); **byte**-identity only after canonical map serialization + regenerated goldens | -| **PO-3b** | **load-path rebuild (§3)** | `WasmNotebook::from_json` calls `rebuild_indices`; after a plain `from_str`, `index`/`in_edges`/`backlinks` are correct, not empty | -| **PO-4.1** | no double-borrow (§4.1) | a `combine` calling a mutator **directly** returns capability-denied, no panic under a live borrow | -| **PO-4.1b** | **nested-eval capability (§4.1)** | a `!` mutator reached via a higher-order builtin (`map`/`do`/`let`) under Reader context is still denied (capability monotonicity, I1) | -| **PO-4.2** | native≡λδ conformance (§4.2) | grid test over shipped ops within `1e-6` | -| **PO-4.3** | domain isolation (§4.3) | a foreign `:op`/`:type` is rejected; a mixed-scalar graph is unconstructable | -| **PO-5** | trigger termination (§5) | self-retriggering action terminates; bounded cascade respects depth `D` | -| **PO-5b** | **trigger order (§5)** | firing order is `triggers`-`Vec` order ⇒ final state deterministic for order-sensitive actions | -| **PO-6** | master determinism (§6) | end-to-end: shuffle **note + term + in-edge** insertion ⇒ identical index+conf+layout **bytes** (requires PO-1.4a, PO-2.4, PO-3 canonical) | - ---- - -## 9. Handoff note to Opus - -Implement in this order, each PR gated on its ledger rows: - -1. **`index.rs` (headless).** Structure per §2.1 (doubly-linked postings — do **not** use sorted `Vec`s), lifecycle §2.2, exact query §2.3, determinism §2.4. Gate: PO-2.*, PO-3, PO-6(index part). No WASM, no UI. *This is the whole DEVONthink side de-risked before a pixel ships.* -2. **`edge.rs` + serde.** Additive `edges`; `in_edges` and `index` `#[serde(skip)]` (§3 ⚠). Gate: PO-3. -3. **`reason.rs`.** `ReasonGraph` (sorted, iterative DFS), `back_edges`, `kahn_topo`, `weight_edge`, native `combine`, `propagate`, cone cache. Gate: PO-1.*. -4. **λδ seam.** Reader-context `combine` dispatch (§4.1), conformance grid (§4.2), `validate` (§4.3). Gate: PO-4.*. -5. **`trigger.rs`.** Suppression guard (§5). Gate: PO-5. -6. **`layout.rs` + `ReasoningView`, See-Also/Duplicates panels.** Consume `back_edges` from `reason.rs` for dashed edges (§6 consistency). UI only after 1–5 are green. - -**Blocking prerequisites (fix in the spec before step 1 — the v2 corrections):** -1. **Pin every float fold** in key-sorted order — `combine` in-edges by `(source NoteId, edge-idx)`; cosine numerator + both norms + classify centroids by `TermId` (`PO-1.4a`/`PO-2.4`/`PO-2.6`). Without this, `PO-6` is unachievable. -2. **Canonical map serialization** (`serialize_with` sorted, or `BTreeMap`) + regenerated goldens — *or* consciously relax `PO-3` to semantic equality (`PO-3`). -3. **Two-sided posting `remove`** so reindex preserves Invariant L (`PO-2.1`). -4. **Load path calls `rebuild_indices`** so derived state isn't empty after a plain deserialize (`PO-3b`). - -These four are the difference between "compiles and passes" and "flakes on day one." Everything else is a caveat the ledger absorbs. - -The math is settled above. Opus's job is faithful translation + turning the ledger green — not rediscovery. diff --git a/docs/design/lambdadelta-spec.adoc b/docs/design/lambdadelta-spec.adoc new file mode 100644 index 0000000..314f853 --- /dev/null +++ b/docs/design/lambdadelta-spec.adoc @@ -0,0 +1,394 @@ +== LambdaDelta (λδ) — Language specification, v0.1 + +____ +_A note is a letter we send to our future self._ — _The Tinderbox Way_ + +λδ exists to enlarge that correspondence. This document pins the +fundamentals we agreed before writing the interpreter (see +link:../adr/0003-lambdadelta-lisp-substrate.md[ADR-0003]): *surface +syntax*, the *note-as-value model*, the *builtin vocabulary*, and the +*kernel/host seam* that makes an SDK/plugin ecosystem cheap. +____ + +*Status:* v0.1 — the load-bearing decisions are *confirmed* and the L0 +kernel/host implementation has landed (see §9). Package format, +capability model, and the dev wizard are tracked separately in +*https://github.com/hyperpolymath/nexia-list/issues/33[issue #33]*. The +current assurance boundary is recorded in +*link:../verification/proof-baseline-0.md[Proof Baseline 0]*. + +''''' + +=== 0. Stance + +λδ is a _successor-flavoured_ Lisp, not a museum piece: homoiconic and +macro-capable like Scheme, with *hygienic macros* and modern, readable +data literals (vectors, maps, sets, keywords) in the spirit of Clojure — +because the first people to meet λδ are Tinderbox users writing a +one-line formula, not Lisp hackers. Fewer quotes, less ceremony, same +power underneath. + +Everything is an expression that returns a value. Effects on the +notebook are explicit, named with a trailing `+!+`, and only permitted +in contexts that allow them. + +''''' + +=== 1. Surface syntax _(Clojure-flavoured — confirmed)_ + +==== Literals + +.... +nil ; absence / empty +true false ; booleans +42 -3 1.5 1e9 ; numbers (i64 or f64; see §2) +"hello\n" ; strings (JSON-style escapes) +:status :due-date ; keywords (self-evaluating, interned; map keys) +title ->md note? ; symbols (kebab-case; ? = predicate, ! = mutator) +#uuid "1111…" ; tagged literal — a note id +#inst "2026-01-01T…" ; tagged literal — an instant +.... + +==== Collections + +.... +(f a b) ; a call: apply f to a, b — LIST, also code +[1 2 3] ; a vector (indexed data) +{:k 1 :j 2} ; a map (keyword→value) +#{:a :b} ; a set (Tinderbox attributes are often set-valued) +.... + +==== Reader sugar + +.... +'x => (quote x) +`x => (quasiquote x) +~x => (unquote x) +~@xs => (unquote-splicing xs) +#(… % …) => (fn [%] (… % …)) ; anonymous-fn shorthand +#tag v => a tagged literal (extensible; #uuid, #inst built in) +;; a comment to end of line +.... + +That is the whole surface. Code is data: `+(+ 1 2)+` is a three-element +list whose head is the symbol `+++`. Macros manipulate exactly these +forms. + +''''' + +=== 2. Value model _(note = immutable map — confirmed)_ + +[width="100%",cols="50%,50%",options="header",] +|=== +|λδ value |Notes +|`+nil+` |absence; the only "`empty`" + +|`+bool+` |`+true+` / `+false+` + +|`+number+` |integer (i64) or float (f64); `+/+` and decimals produce +floats + +|`+string+` |UTF-8 + +|`+symbol+` |identifiers (evaluated: looked up in scope) + +|`+keyword+` |`+:like-this+`; self-evaluating; canonical map key + +|`+list+` |linked sequence; the form of code + +|`+vector+` |indexed sequence; the form of data + +|`+set+` |`+#{…}+`; unordered, unique members + +|`+map+` |keyword→value (string keys also accepted on read) + +|`+function+` |closure (builtin or user `+fn+`) +|=== + +==== Truthiness _(only `+nil+` and `+false+` are falsy — confirmed)_ + +Everything else — including `+0+` and `+""+` — is truthy. + +==== A note _is a map_ + +Reading a note yields an *immutable snapshot map* — a note is plain +data, the most Lisp-y choice and the one that makes the notebook +homoiconic: + +[source,clojure] +---- +{:id #uuid "1111…" + :title "Meeting notes" + :content "…discussed the roadmap and [[Beta]]…" + :attrs {:status "todo" :priority 2 :tags #{:work :q3}} + :links [#uuid "2222…"] ; outgoing note ids + :backlinks [#uuid "3333…"] ; incoming note ids (read-only) + :position [120.0 80.0] ; or nil if unplaced + :size [200.0 150.0] ; or nil + :prototype nil ; or a note id + :type nil ; user-facing dispatch tag (see §3, multimethods) + :created-at #inst "2026-01-01T00:00:00Z" + :modified-at #inst "2026-01-02T00:00:00Z"} +---- + +* In a *formula* or *action*, the symbol `+self+` is bound to this map. +`+(attr self :status)+` → `+"todo"+`. +* *Reading is pure*: a note map is a value, not a live handle; it never +changes under you. +* *Mutation is explicit and id-based*: +`+(set-attr! (:id self) :status "done")+`. Mutators take an id (or a +note map, from which the id is read) and return the updated note map (or +a delta), so effects are visible and testable. + +==== Attribute ↔ JSON mapping (total, lossless) + +string→string · number→number · bool→bool · `+null+`→`+nil+` · +array→**vector** · object→**map** (string keys read as keywords). _(Sets +serialize as a tagged array `+#{…}+` so JSON stays canonical; +typed-attribute work later may make set membership explicit.)_ + +''''' + +=== 3. Special forms & extensibility + +The irreducible core the evaluator knows directly (everything else is a +function or a macro): + +[source,clojure] +---- +(quote x) ; unevaluated x ' sugar +(if test then else?) ; else defaults to nil +(do e1 … en) ; sequence; value is en +(let [x 1 y (+ x 1)] body…) ; sequential bindings; supports destructuring +(fn [a b] body…) ; lambda / closure (also (fn name […] …)) +(def name value) ; define in the notebook environment +(defmacro name [args] body…) ; HYGIENIC by default (confirmed) +(quasiquote t) / (unquote e) / (unquote-splicing e) ; ` ~ ~@ +---- + +*Hygienic macros (confirmed).* Macro-introduced bindings never capture, +and references never leak, without the author asking — +`+syntax-rules+`/`+syntax-case+` -grade hygiene, `+gensym+` available +for deliberate cases. Non-expert authors get safe macros for free; an +explicit unhygienic escape hatch may come later if justified. + +*Multimethods / protocols (confirmed) — the extensibility lever.* Open +dispatch on _any_ function of the arguments (not class-based OO), so +plugins and prototypes can extend behaviour *by note `+:type+`*: + +[source,clojure] +---- +(defmulti render :type) ; dispatch on a note's :type +(defmethod render "task" [n] (task-card n)) +(defmethod render :default [n] (plain-card n)) +---- + +This is what makes Nexia-List _moldable_ (Emacs-like) while staying +simple by default: a plugin adds `+defmethod+`s for its note types +without touching anyone else’s code. + +Provided as (hygienic) macros over the above — still "`core`" to users: +`+cond+`, `+when+`, `+and+`, `+or+`, `+->+` / `+->>+` (threading), +`+if-let+`, `+case+`, `+match+` (pattern-match/destructure notes, attrs, +results). + +''''' + +=== 4. Initial builtin vocabulary + +A deliberately small, growable standard library. `+?+` = predicate, +`+!+` = mutator. + +*Arithmetic / compare / logic* `++ - * / mod+` · `+= not= < > <= >=+` · +`+not min max abs floor ceil round+` + +*Predicates* +`+nil? true? false? number? string? symbol? keyword? list? vector? set? map? fn? note? empty?+` + +*Sequences* (lists and vectors) +`+list vector count first rest last nth get take drop reverse sort sort-by range conj cons concat map filter remove reduce some every? distinct into flatten+` + +*Sets* `+set union intersection difference subset? contains? conj disj+` + +*Strings* +`+str join split lines words trim lower upper starts-with? ends-with? includes? replace subs format+` + +*Maps* `+get assoc dissoc update keys vals contains? merge select-keys+` + +*Notebook — readers (pure)* + +[source,clojure] +---- +(notes) (note id) +(title n) (content n) (attrs n) (links n) (backlinks n) (position n) +(attr n key) ; one attribute value, or nil +(search q) ; vector of note maps matching a text query +(agents) (run-agent id) +(resolve-title s) ; note id whose title = s (case-insensitive), or nil +---- + +*Notebook — mutators (only in action contexts)* + +[source,clojure] +---- +(create-note! title) (create-note! title x y) +(set-title! id s) (set-content! id s) +(set-attr! id key v) (remove-attr! id key) +(move-note! id x y) (resize-note! id w h) +(link! from to) (unlink! from to) +(delete-note! id) +---- + +*Reflection (homoiconicity)* `+eval read quote gensym macroexpand+` + +''''' + +=== 5. Evaluation contexts + +Same language, different capabilities and bound variables — this is how +one engine serves every surface without leaking parentheses into L0: + +[width="100%",cols="20%,20%,20%,20%,20%",options="header",] +|=== +|Context |Bound vars |May mutate? |Must return |Example surface +|*Expression* |— |no |any value |the REPL + +|*Formula* (computed attribute) |`+self+` |no (pure) |attribute value +|`+(count (words (content self)))+` + +|*Agent predicate* |`+self+` |no |truthy +|`+(= (attr self :status) "todo")+` + +|*Agent query* |— |no |vector of notes/ids +|`+(filter #(> (attr % :priority) 3) (notes))+` + +|*Action* (on-create / agent-action / stamp / adornment) |`+self+` +|*yes* |ignored |`+(set-attr! (:id self) :seen true)+` +|=== + +The existing L0 agent DSL (`+attr:status=todo+`) compiles to an *Agent +predicate* λδ expression — one engine, two surfaces. + +''''' + +=== 6. Sandbox contract + +* *Deterministic & pure-by-default:* no clock, no randomness, no +network, no file/DOM access. The only implemented effects are the `+!+` +notebook mutators. Formula evaluation omits them by construction; +capability enforcement for broader action/plugin contexts is deferred +(§7). +* *Bounded:* every evaluation runs under a reduction-step count and a +recursion-depth limit. Exceeding either aborts cleanly with an error +value. A wall-clock ceiling is not implemented; native builtins +therefore remain inside the trusted termination boundary. (Heavy jobs → +Web Worker later.) +* *Errors are values / diagnostics*, never panics: unbound symbol, arity +mismatch, type error, and budget exhaustion are structured errors the UI +can show against the offending form. Capability-denied errors arrive +with the deferred capability model. + +''''' + +=== 7. Architecture: the kernel / host seam _(confirmed — the SDK enabler)_ + +The single discipline that makes an SDK, embedding, and the plugin +ecosystem (#33) cheap rather than a later rewrite. Build λδ in two +layers from day one: + +* *Kernel* (`+core/src/lambdadelta/+`) — reader · value model · +evaluator · hygienic macros · multimethods · budget. *Knows nothing +about notes.* A self-contained language that could become its own crate. +* *Host bindings* — the notebook builtins (`+notes+`, `+set-attr!+`, +`+render+`, …) are _registered into_ the kernel through a host-function +interface. The current formula entry point registers only readers; +native registrations do *not yet carry capability metadata*. Nexia-List +is the first host. + +Consequences that fall out for free: - *Embedders* depend on the kernel ++ `+register_builtin+`; λδ becomes an ecosystem beyond Nexia-List. - +*Plugin authors (planned)* ship a package (`+.ld+` source + a manifest +of entry points and _requested capabilities_); the +provisioner/configurator (#33) will grant and enforce them, and the +harness will run them against a fixture notebook under the budget. - +*Sugar is free*: almost all developer sweetness (`+defcommand+`, +`+defview+`, `+deftemplate+`, threading, `+match+`, destructuring) is +hygienic macros + reader tags _around_ the kernel — the kernel stays +tiny and stable. + +The capability model and package/manifest format are specified in *#33*; +this section fixes only that the seam exists and where it sits. +Capability non-escalation remains an engineering-blocked proof +obligation until authority is represented and checked at dispatch. + +''''' + +=== 8. Where λδ code lives (persistence) & worked examples + +User definitions are *data in the notebook*, so they save/load/sync with +everything and a notebook carries its own behaviour. Formulas and agent +queries/actions live on their owning note/agent; shared functions, +macros, and `+defmethod+`s live in a notebook-level `+:lambdadelta+` map +(name → source). A textual *`+.ld+`* form is the interchange unit for +sharing packages between notebooks (the in-notebook data form is the +source of truth). + +[source,clojure] +---- +;; L1 — computed attribute "wordcount" (pure; self = this note) +(count (words (content self))) + +;; L1 — agent predicate: due and not done +(and (= (attr self :status) "todo") (attr self :due-date)) + +;; L2 — on-create action: stamp a new note +(do (set-attr! (:id self) :status "inbox") + (set-attr! (:id self) :created-day (subs (str (:created-at self)) 0 10))) + +;; L2 — a hygienic macro: query shorthand (tagged :work) +(defmacro tagged [t] + `(filter #(contains? (or (attr % :tags) #{}) ~t) (notes))) + +;; L2 — extend rendering for a note type (plugin-style) +(defmethod render "meeting" [n] (meeting-card n)) + +;; L3 — resurfacing: last 5 notes I touched that link to "Project X" +(->> (notes) + (filter #(includes? (links %) (resolve-title "Project X"))) + (sort-by :modified-at) reverse (take 5)) + +;; L4 — a computational note: content evaluates, result renders inline. +---- + +''''' + +=== 9. Decisions (v0.1 — confirmed) + +[arabic] +. *Flavour* — Clojure-flavoured (`+[] {} #{} :kw+`, `+~+` unquote, +`+true/false/nil+`). +. *Note representation* — immutable snapshot map; mutation via id-based +`+!+`. +. *Map keys* — keywords canonical; string keys accepted on read. +. *Data sequences* — vectors `+[]+` for data, lists `+()+` for code. +. *Truthiness* — only `+nil+` and `+false+` are falsy. +. *Macros* — *hygienic by default*. +. *Extensibility* — *multimethods + protocols* (dispatch on any fn; not +class-OO). +. *Sugar* — sets `+#{}+`, threading `+-> ->>+`, `+match+`, tagged +literals `+#uuid+`/`+#inst+`, `+#(…)+`. +. *Architecture* — *kernel/host seam*: notebook-agnostic kernel + +registered host bindings (§7). +. *Interchange* — `+.ld+` text form for package sharing. +. *Name/typography* — *LambdaDelta / λδ*; ASCII trigraph `+ld+`; module +`+core/src/lambdadelta/+`. + +*Deferred (tracked in +https://github.com/hyperpolymath/nexia-list/issues/33[#33]):* +package/manifest format, the capability model, and the +minter/provisioner/ configurator/harness dev wizard. + +*Deliberately skipped for now* (overkill for a note tool): delimited +continuations, STM/`+core.async+` concurrency, transducers, full gradual +typing. `+spec+`-style attribute schemas are deferred to the +typed-attributes work. diff --git a/docs/design/lambdadelta-spec.md b/docs/design/lambdadelta-spec.md deleted file mode 100644 index 66b2f54..0000000 --- a/docs/design/lambdadelta-spec.md +++ /dev/null @@ -1,341 +0,0 @@ - -# LambdaDelta (λδ) — Language specification, v0.1 - -> *A note is a letter we send to our future self.* — *The Tinderbox Way* -> -> λδ exists to enlarge that correspondence. This document pins the fundamentals -> we agreed before writing the interpreter (see -> [ADR-0003](../adr/0003-lambdadelta-lisp-substrate.md)): **surface syntax**, -> the **note-as-value model**, the **builtin vocabulary**, and the **kernel/host -> seam** that makes an SDK/plugin ecosystem cheap. - -**Status:** v0.1 — the load-bearing decisions are **confirmed** and the L0 -kernel/host implementation has landed (see §9). Package format, capability -model, and the dev wizard are tracked separately in -**[issue #33](https://github.com/hyperpolymath/nexia-list/issues/33)**. The -current assurance boundary is recorded in -**[Proof Baseline 0](../verification/proof-baseline-0.md)**. - ---- - -## 0. Stance - -λδ is a *successor-flavoured* Lisp, not a museum piece: homoiconic and -macro-capable like Scheme, with **hygienic macros** and modern, readable data -literals (vectors, maps, sets, keywords) in the spirit of Clojure — because the -first people to meet λδ are Tinderbox users writing a one-line formula, not Lisp -hackers. Fewer quotes, less ceremony, same power underneath. - -Everything is an expression that returns a value. Effects on the notebook are -explicit, named with a trailing `!`, and only permitted in contexts that allow -them. - ---- - -## 1. Surface syntax *(Clojure-flavoured — confirmed)* - -### Literals -``` -nil ; absence / empty -true false ; booleans -42 -3 1.5 1e9 ; numbers (i64 or f64; see §2) -"hello\n" ; strings (JSON-style escapes) -:status :due-date ; keywords (self-evaluating, interned; map keys) -title ->md note? ; symbols (kebab-case; ? = predicate, ! = mutator) -#uuid "1111…" ; tagged literal — a note id -#inst "2026-01-01T…" ; tagged literal — an instant -``` - -### Collections -``` -(f a b) ; a call: apply f to a, b — LIST, also code -[1 2 3] ; a vector (indexed data) -{:k 1 :j 2} ; a map (keyword→value) -#{:a :b} ; a set (Tinderbox attributes are often set-valued) -``` - -### Reader sugar -``` -'x => (quote x) -`x => (quasiquote x) -~x => (unquote x) -~@xs => (unquote-splicing xs) -#(… % …) => (fn [%] (… % …)) ; anonymous-fn shorthand -#tag v => a tagged literal (extensible; #uuid, #inst built in) -;; a comment to end of line -``` - -That is the whole surface. Code is data: `(+ 1 2)` is a three-element list whose -head is the symbol `+`. Macros manipulate exactly these forms. - ---- - -## 2. Value model *(note = immutable map — confirmed)* - -| λδ value | Notes | -|---|---| -| `nil` | absence; the only "empty" | -| `bool` | `true` / `false` | -| `number` | integer (i64) or float (f64); `/` and decimals produce floats | -| `string` | UTF-8 | -| `symbol` | identifiers (evaluated: looked up in scope) | -| `keyword` | `:like-this`; self-evaluating; canonical map key | -| `list` | linked sequence; the form of code | -| `vector` | indexed sequence; the form of data | -| `set` | `#{…}`; unordered, unique members | -| `map` | keyword→value (string keys also accepted on read) | -| `function` | closure (builtin or user `fn`) | - -### Truthiness *(only `nil` and `false` are falsy — confirmed)* -Everything else — including `0` and `""` — is truthy. - -### A note *is a map* -Reading a note yields an **immutable snapshot map** — a note is plain data, the -most Lisp-y choice and the one that makes the notebook homoiconic: - -```clojure -{:id #uuid "1111…" - :title "Meeting notes" - :content "…discussed the roadmap and [[Beta]]…" - :attrs {:status "todo" :priority 2 :tags #{:work :q3}} - :links [#uuid "2222…"] ; outgoing note ids - :backlinks [#uuid "3333…"] ; incoming note ids (read-only) - :position [120.0 80.0] ; or nil if unplaced - :size [200.0 150.0] ; or nil - :prototype nil ; or a note id - :type nil ; user-facing dispatch tag (see §3, multimethods) - :created-at #inst "2026-01-01T00:00:00Z" - :modified-at #inst "2026-01-02T00:00:00Z"} -``` - -- In a **formula** or **action**, the symbol `self` is bound to this map. - `(attr self :status)` → `"todo"`. -- **Reading is pure**: a note map is a value, not a live handle; it never - changes under you. -- **Mutation is explicit and id-based**: `(set-attr! (:id self) :status "done")`. - Mutators take an id (or a note map, from which the id is read) and return the - updated note map (or a delta), so effects are visible and testable. - -### Attribute ↔ JSON mapping (total, lossless) -string→string · number→number · bool→bool · `null`→`nil` · array→**vector** · -object→**map** (string keys read as keywords). *(Sets serialize as a tagged -array `#{…}` so JSON stays canonical; typed-attribute work later may make set -membership explicit.)* - ---- - -## 3. Special forms & extensibility - -The irreducible core the evaluator knows directly (everything else is a function -or a macro): - -```clojure -(quote x) ; unevaluated x ' sugar -(if test then else?) ; else defaults to nil -(do e1 … en) ; sequence; value is en -(let [x 1 y (+ x 1)] body…) ; sequential bindings; supports destructuring -(fn [a b] body…) ; lambda / closure (also (fn name […] …)) -(def name value) ; define in the notebook environment -(defmacro name [args] body…) ; HYGIENIC by default (confirmed) -(quasiquote t) / (unquote e) / (unquote-splicing e) ; ` ~ ~@ -``` - -**Hygienic macros (confirmed).** Macro-introduced bindings never capture, and -references never leak, without the author asking — `syntax-rules`/`syntax-case` --grade hygiene, `gensym` available for deliberate cases. Non-expert authors get -safe macros for free; an explicit unhygienic escape hatch may come later if -justified. - -**Multimethods / protocols (confirmed) — the extensibility lever.** Open -dispatch on *any* function of the arguments (not class-based OO), so plugins and -prototypes can extend behaviour **by note `:type`**: - -```clojure -(defmulti render :type) ; dispatch on a note's :type -(defmethod render "task" [n] (task-card n)) -(defmethod render :default [n] (plain-card n)) -``` - -This is what makes Nexia-List *moldable* (Emacs-like) while staying simple by -default: a plugin adds `defmethod`s for its note types without touching anyone -else's code. - -Provided as (hygienic) macros over the above — still "core" to users: -`cond`, `when`, `and`, `or`, `->` / `->>` (threading), `if-let`, `case`, -`match` (pattern-match/destructure notes, attrs, results). - ---- - -## 4. Initial builtin vocabulary - -A deliberately small, growable standard library. `?` = predicate, `!` = mutator. - -**Arithmetic / compare / logic** -`+ - * / mod` · `= not= < > <= >=` · `not min max abs floor ceil round` - -**Predicates** -`nil? true? false? number? string? symbol? keyword? list? vector? set? map? fn? note? empty?` - -**Sequences** (lists and vectors) -`list vector count first rest last nth get take drop reverse sort sort-by range -conj cons concat map filter remove reduce some every? distinct into flatten` - -**Sets** -`set union intersection difference subset? contains? conj disj` - -**Strings** -`str join split lines words trim lower upper starts-with? ends-with? includes? -replace subs format` - -**Maps** -`get assoc dissoc update keys vals contains? merge select-keys` - -**Notebook — readers (pure)** -```clojure -(notes) (note id) -(title n) (content n) (attrs n) (links n) (backlinks n) (position n) -(attr n key) ; one attribute value, or nil -(search q) ; vector of note maps matching a text query -(agents) (run-agent id) -(resolve-title s) ; note id whose title = s (case-insensitive), or nil -``` - -**Notebook — mutators (only in action contexts)** -```clojure -(create-note! title) (create-note! title x y) -(set-title! id s) (set-content! id s) -(set-attr! id key v) (remove-attr! id key) -(move-note! id x y) (resize-note! id w h) -(link! from to) (unlink! from to) -(delete-note! id) -``` - -**Reflection (homoiconicity)** -`eval read quote gensym macroexpand` - ---- - -## 5. Evaluation contexts - -Same language, different capabilities and bound variables — this is how one -engine serves every surface without leaking parentheses into L0: - -| Context | Bound vars | May mutate? | Must return | Example surface | -|---|---|---|---|---| -| **Expression** | — | no | any value | the REPL | -| **Formula** (computed attribute) | `self` | no (pure) | attribute value | `(count (words (content self)))` | -| **Agent predicate** | `self` | no | truthy | `(= (attr self :status) "todo")` | -| **Agent query** | — | no | vector of notes/ids | `(filter #(> (attr % :priority) 3) (notes))` | -| **Action** (on-create / agent-action / stamp / adornment) | `self` | **yes** | ignored | `(set-attr! (:id self) :seen true)` | - -The existing L0 agent DSL (`attr:status=todo`) compiles to an **Agent predicate** -λδ expression — one engine, two surfaces. - ---- - -## 6. Sandbox contract - -- **Deterministic & pure-by-default:** no clock, no randomness, no network, no - file/DOM access. The only implemented effects are the `!` notebook mutators. - Formula evaluation omits them by construction; capability enforcement for - broader action/plugin contexts is deferred (§7). -- **Bounded:** every evaluation runs under a reduction-step count and a - recursion-depth limit. Exceeding either aborts cleanly with an error value. - A wall-clock ceiling is not implemented; native builtins therefore remain - inside the trusted termination boundary. (Heavy jobs → Web Worker later.) -- **Errors are values / diagnostics**, never panics: unbound symbol, arity - mismatch, type error, and budget exhaustion are structured errors the UI can - show against the offending form. Capability-denied errors arrive with the - deferred capability model. - ---- - -## 7. Architecture: the kernel / host seam *(confirmed — the SDK enabler)* - -The single discipline that makes an SDK, embedding, and the plugin ecosystem -(#33) cheap rather than a later rewrite. Build λδ in two layers from day one: - -- **Kernel** (`core/src/lambdadelta/`) — reader · value model · evaluator · - hygienic macros · multimethods · budget. **Knows nothing about notes.** A - self-contained language that could become its own crate. -- **Host bindings** — the notebook builtins (`notes`, `set-attr!`, `render`, …) - are *registered into* the kernel through a host-function interface. The - current formula entry point registers only readers; native registrations do - **not yet carry capability metadata**. Nexia-List is the first host. - -Consequences that fall out for free: -- **Embedders** depend on the kernel + `register_builtin`; λδ becomes an - ecosystem beyond Nexia-List. -- **Plugin authors (planned)** ship a package (`.ld` source + a manifest of - entry points and *requested capabilities*); the provisioner/configurator - (#33) will grant and enforce them, and the harness will run them against a - fixture notebook under the budget. -- **Sugar is free**: almost all developer sweetness (`defcommand`, `defview`, - `deftemplate`, threading, `match`, destructuring) is hygienic macros + reader - tags *around* the kernel — the kernel stays tiny and stable. - -The capability model and package/manifest format are specified in **#33**; this -section fixes only that the seam exists and where it sits. Capability -non-escalation remains an engineering-blocked proof obligation until authority -is represented and checked at dispatch. - ---- - -## 8. Where λδ code lives (persistence) & worked examples - -User definitions are **data in the notebook**, so they save/load/sync with -everything and a notebook carries its own behaviour. Formulas and agent -queries/actions live on their owning note/agent; shared functions, macros, and -`defmethod`s live in a notebook-level `:lambdadelta` map (name → source). -A textual **`.ld`** form is the interchange unit for sharing packages between -notebooks (the in-notebook data form is the source of truth). - -```clojure -;; L1 — computed attribute "wordcount" (pure; self = this note) -(count (words (content self))) - -;; L1 — agent predicate: due and not done -(and (= (attr self :status) "todo") (attr self :due-date)) - -;; L2 — on-create action: stamp a new note -(do (set-attr! (:id self) :status "inbox") - (set-attr! (:id self) :created-day (subs (str (:created-at self)) 0 10))) - -;; L2 — a hygienic macro: query shorthand (tagged :work) -(defmacro tagged [t] - `(filter #(contains? (or (attr % :tags) #{}) ~t) (notes))) - -;; L2 — extend rendering for a note type (plugin-style) -(defmethod render "meeting" [n] (meeting-card n)) - -;; L3 — resurfacing: last 5 notes I touched that link to "Project X" -(->> (notes) - (filter #(includes? (links %) (resolve-title "Project X"))) - (sort-by :modified-at) reverse (take 5)) - -;; L4 — a computational note: content evaluates, result renders inline. -``` - ---- - -## 9. Decisions (v0.1 — confirmed) - -1. **Flavour** — Clojure-flavoured (`[] {} #{} :kw`, `~` unquote, `true/false/nil`). -2. **Note representation** — immutable snapshot map; mutation via id-based `!`. -3. **Map keys** — keywords canonical; string keys accepted on read. -4. **Data sequences** — vectors `[]` for data, lists `()` for code. -5. **Truthiness** — only `nil` and `false` are falsy. -6. **Macros** — **hygienic by default**. -7. **Extensibility** — **multimethods + protocols** (dispatch on any fn; not class-OO). -8. **Sugar** — sets `#{}`, threading `-> ->>`, `match`, tagged literals `#uuid`/`#inst`, `#(…)`. -9. **Architecture** — **kernel/host seam**: notebook-agnostic kernel + registered host bindings (§7). -10. **Interchange** — `.ld` text form for package sharing. -11. **Name/typography** — **LambdaDelta / λδ**; ASCII trigraph `ld`; module `core/src/lambdadelta/`. - -**Deferred (tracked in [#33](https://github.com/hyperpolymath/nexia-list/issues/33)):** -package/manifest format, the capability model, and the minter/provisioner/ -configurator/harness dev wizard. - -**Deliberately skipped for now** (overkill for a note tool): delimited -continuations, STM/`core.async` concurrency, transducers, full gradual typing. -`spec`-style attribute schemas are deferred to the typed-attributes work. diff --git a/docs/design/mind-management-plan.adoc b/docs/design/mind-management-plan.adoc new file mode 100644 index 0000000..01a7948 --- /dev/null +++ b/docs/design/mind-management-plan.adoc @@ -0,0 +1,691 @@ +== The Management of Mind — Nexia-List’s Definitive Plan + +_A note is a letter we send to our future self._ + +Status: canonical design synthesis · Supersedes the standalone +product-vision, UI/UX-overhaul, and FL×DT-integration notes by unifying +them. The FL×DT integration decisions are *settled* and enter here as +the "`intelligence + reasoning`" track, not as open questions. + +''''' + +=== 1. TL;DR + Product Thesis + +*Nexia-List is an instrument for the real management of mind* — +capturing, structuring, linking, *reasoning over*, recalling, and +*resurfacing* decades of thought — that runs entirely on your device, +shows nothing you didn’t ask for, and works the same on every platform. +It is not a place to _store_ notes. It is a place where your notes *keep +working on your behalf* for as long as you have a mind to manage. + +Three commitments define it: + +* *Management of mind is the goal.* Every feature is judged by one +question: _does this help the letter reach a richer future self?_ The +category is crowded with capture-and-store tools that abandon the reader +at exactly the point where a mind is actually managed — *reason* and +*resurface*. That neglected second half of the loop is the product. +* *λδ (LambdaDelta) is the killer underpinning — the enabler, never the +goal.* The notebook is live homoiconic Lisp data; behavior is +multimethods dispatched on `+:type+`/`+:op+`; extensions are `+.ld+` +*packages* that are data, not code-you-must-trust. λδ is why the tool is +moldable, lifelong, and user-owned — why it can be reshaped into a form +its author never imagined and still be running in twenty years. It is a +*door, and the door is closed by default.* Most people should use Nexia +for life and never see a parenthesis. +* *Local-first, invisible-by-default, cross-platform.* The Rust core +compiles to a single WASM bundle that _is_ the engine, running +client-side. No server, no account, no cloud, no sync, no lock-in — +nothing leaves the device. Progressive disclosure L0→L4 keeps the +surface quiet: L0 is a complete, dignified tool usable for a lifetime +with zero code. One engine appears everywhere — web-first PWA as the +primary product, an optional thin webview shell (desktop _and_ mobile) +hosting the identical bundle. + +The unclaimed intersection Nexia occupies: *local-first + homoiconic + +native reasoning + on-device intelligence, with no server and no +lock-in.* No other tool holds all four. That intersection is the +product; everything else is execution. + +''''' + +=== 2. The Mind-Management Loop + +Note apps optimize _capture-and-store_. Nexia optimizes the actual cycle +a thinking person runs over a lifetime: + +.... + CAPTURE ──► STRUCTURE ──► LINK ──► REASON ──► RECALL ──► RESURFACE ──► PUBLISH + ▲ │ + └────────────────────────── the future self ────────────────────────┘ +.... + +The letter is written (*capture*), given shape (*structure*), connected +to prior letters (*link*), *argued with* (*reason*), retrieved when +needed (*recall*), pushed back to you unbidden at the moment it matters +(*resurface*), and — when it must leave the head to reach others — +handed out in a portable form (*publish*). Then the future self, now +informed, writes the next letter. The loop closes on a person, not a +database. + +Today’s tools are lopsided: excellent at capture, competent at link, and +effectively *abandoning the reader* at reason and resurface. Nexia’s +thesis is that the second half of the loop is where a mind is managed, +and it is exactly the half everyone leaves broken. + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|Stage |What the user does |The capability that serves it |Level +|*Capture* |Get the thought down instantly, no ceremony |Quick-capture +into an Inbox (`+{:inbox true}+`, no canvas dump); `+[[ ]]+` +autocomplete; plain textarea source of truth |L0 + +|*Structure* |Give shape — titles, tags, attributes, prototypes, spatial +arrangement |Inspector (attributes/tags/prototype/confidence); canvas; +outline; markdown render |L0/L1 + +|*Link* |Connect to prior letters |`+[[wikilinks]]+` (untyped +association) + backlinks — the pattern the whole spine generalizes |L0 + +|*Reason* |Ask _"`why do I believe this?`"_ and see the argument |Typed, +weighted `+edges+` channel + `+reason.rs+` fuzzy-boolean `+propagate()+` ++ `+layout.rs+` Sugiyama → *ReasoningView* |L2 + +|*Recall* |Retrieve deliberately |BM25 ranked search; palette go-to; +enrich-DSL Agents (`+similar:+`/`+near:+`/`+conf:+`/`+type:+`/`+edge:+`) +|L0/L2 + +|*Resurface* |Be found by the letter you forgot |`+index.rs+` See-Also +(TF-IDF cosine kNN); Duplicates (blake3+SimHash); kNN auto-tag — +ambient, volunteered |L1 + +|*Publish* |Hand the letter out portably |Markdown / OPML export +(already shipped); JSON import/export; `+.ld+` packages as shareable +data |L0 +|=== + +Two crown jewels — *an established build track, not open questions* — +complete the broken half: + +* *DT-style local recall* closes *recall + resurface.* One on-device +intelligence engine (`+core/src/index.rs+`) builds a concordance and +inverted index in memory, `+#[serde(skip)]+`, rebuilt on load, never +trusted from disk. From it: See-Also, BM25, Rocchio/naive-Bayes +Classify, blake3+SimHash dedupe, kNN auto-tag. The archive stops being a +graveyard you must excavate and becomes a correspondent that writes +back. +* *FL-style reasoning* closes *reason.* A separate additive `+edges+` +channel carries typed, weighted relationships; `+core/src/reason.rs+` +runs a pure single-pass DAG `+propagate()+` with a native fuzzy-boolean +operator table (0.5 = Indeterminate). Junctors and entities are ordinary +notes tagged `+{:type :junct :op and|or|not}+`. This is what lets your +notebook _argue with you_. + +Both jewels collapse onto *one pattern nexia already proved* with +`+Notebook::backlinks+`: a *derived, rebuildable index beside the notes, +never trusted from disk*, semantics dispatched on `+:type+`/`+:op+` via +λδ multimethods, surfaced as no-parenthesis L0/L1 panels. The note model +is untouched. Recall and reasoning are not new formats you can be locked +into — they are _computed views_ over letters you already own. Delete +the index; it rebuilds. That is the difference between a feature and a +liability. + +*How λδ completes the loop without owning it:* the native engines are +the *floor, not the ceiling.* See-Also, BM25, classify, propagate all +run in Rust at native speed, no Lisp required. λδ enters only at the +seams — the `+combine+` multimethod for exotic/domain reasoning +operators, Agents as saved composable queries, and `+.ld+` packages +carrying whole methodologies as data. The loop runs at full speed for +the L0 user and is infinitely re-shapable for the one who opens a door. + +''''' + +=== 3. Positioning + +The center of this Venn diagram is empty. Each incumbent is world-class +at one move and structurally incapable of another. + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|Tool |Crown jewel |Structural ceiling |What Nexia does differently +|*Tinderbox* |Spatial thinking, agents, prototypes, attribute-driven +emergence |macOS-only, closed, single-vendor mortality; a formula +language bolted on at the edge |The North Star + agent/prototype spirit +— but *cross-platform, open, homoiconic to the core* + +|*Obsidian* |Local Markdown files, plugin ecosystem, backlinks |Plugins +bolt onto a passive editor; no native reasoning or intelligence; the +graph is decorative |*Reasoning and recall are native*, not a plugin +lottery; the graph _computes_ (See-Also, propagate) + +|*Roam / Logseq* |Outliner + bidirectional block links; daily-notes +capture |The graph is _associative only_ — it links but never *argues*; +recall is manual; hosted/lock-in pressure |Typed, weighted *inference* +on top of association; ambient recall; JSON you can walk away with + +|*Tana* |Supertags, structured queries, AI-native |*Cloud-required, +account-required, subscription* — the archive is a tenant of a company +|Same structure/query power, *on your device, no account, runs after the +company is gone* + +|*DEVONthink* |The best local recall in the business (See-Also / +classify) |macOS-only, closed; document-manager, not a thinking surface; +no reasoning layer |*Its exact algorithms in portable Rust/WASM*, +married to a thinking canvas + a reasoning engine it never had + +|*Flying Logic* |Rigorous fuzzy-logic reasoning over typed graphs |A +standalone diagrammer disconnected from your _notes_ — reason in one +silo, think in another |*Reasoning happens over your actual notebook* — +junctors are notes, evidence is notes, one corpus +|=== + +Tinderbox has the philosophy but not the platforms; Obsidian the files +but not the mind; Tana the structure but not the ownership; DEVONthink +the recall but not the thinking; Flying Logic the reasoning but not the +notes. *Nexia is the only tool that puts DEVONthink’s recall and Flying +Logic’s reasoning _inside_ a Tinderbox-spirited, Obsidian-portable, +no-cloud notebook — and makes the whole thing homoiconic so it can never +ossify.* + +''''' + +=== 4. Architecture Principle — the λδ-centric spine + +*Everything expressible through λδ + a typed graph + attributes + +derived indices.* The note model is the primary, human-readable, +user-owned data. Everything intelligent is a *derived index computed +beside it* and never trusted from disk. Behavior is *λδ multimethods on +`+:type+`/`+:op+`*. Native Rust hot paths exist where perf demands (the +floor); λδ extension is available everywhere (no ceiling); `+.ld+` +packages are the moldability surface where methodologies live as data. + +Five rules keep it honest: + +[arabic] +. *Derived, never trusted; rebuildable, never load-bearing.* Backlinks, +See-Also, BM25, reasoning layout — all `+#[serde(skip)]+`, rebuilt on +load. Corrupt the index, lose nothing. The letters are primary; the +intelligence is a lens you can drop and re-grind. +. *The note model is untouched by advanced features.* The single new +field is `+edges: Vec+` — empty by default, +`+skip_serializing_if+` empty, byte-identical round-trip. `+links+` +stays untyped association. `+position+` stays user-authored; +`+layout.rs+`/`+reason.rs+` positions are derived and *never written +back* to `+Note.position+`. +. *Native is a floor, not a ceiling.* Common operations run in fast +Rust; λδ `+combine+` multimethods handle only the exotic and +domain-specific. Users never pay a Lisp tax for the 95% case and never +hit a wall in the 5%. +. *Methodologies are data, not engine.* FL operator packs, DT +methodologies, the six TOC thinking-process trees ship as `+.ld+` +*packages*. The native surface stays tiny: `+index.rs+`, `+edge.rs+`, +`+reason.rs+`, `+trigger.rs+`, later `+layout.rs+` — one struct, one +note field. +. *Invisible by default.* Parentheses never appear until a user opens a +power door. Derived panels mount only when their set is non-empty. + +[source,mermaid] +---- +flowchart TB + subgraph OWN["OWNED DATA — human-readable JSON, IndexedDB, file import/export"] + N["Notes (:type, attributes, prototype, links, size, position)"] + E["edges: Vec<Edge{to,kind,weight,attrs}> — empty by default, additive"] + P[".ld packages — methodologies, operator packs, multimethods AS DATA"] + end + subgraph DERIVED["DERIVED INDICES — #serde(skip), rebuilt on load, never on disk"] + BL[backlinks] + IDX["index.rs — concordance / inverted index → See-Also, BM25, Classify, Dedupe, Auto-tag"] + RSN["reason.rs propagate() — native fuzzy-boolean DAG"] + LAY["layout.rs — Sugiyama layered positions"] + end + subgraph DISPATCH["λδ SUBSTRATE — multimethods on :type/:op, sandboxed budget, kernel/host seam"] + MM["multimethods + combine (exotic ops only)"] + AG["Agents (enrich DSL) · triggers (Smart Rules)"] + end + N --> BL & IDX & RSN + E --> RSN --> LAY + P --> MM + MM -. "extends, never blocks" .-> IDX & RSN + N <-->|":type / :op dispatch"| DISPATCH + DERIVED -->|"no-paren L0/L1 panels"| UI["WASM ← ReScript TEA UI"] + DISPATCH --> UI + OWN -->|"single WASM source of truth"| UI +---- + +Native new surface is small: `+index.rs+`, `+edge.rs+`, `+reason.rs+`, +`+trigger.rs+`, later `+layout.rs+`, plus a `+render_markdown+` export. +Everything users recognize as "`intelligence`" is λδ multimethods + +`+.ld+` packages + Agents on top. + +''''' + +=== 5. Three Interlocking Tracks + +The plan advances on three tracks that share the same spine. They +interlock: *A* provides the moldability surface everything extends +through, *B* is the settled FL×DT intelligence engine, *C* is the +surface that makes both usable without a parenthesis. + +==== Track A — λδ Substrate & Moldability + +*What ships:* the homoiconic base stays the enabler everything is open +to — multimethods on `+:type+`/`+:op+`, sandboxed budget, kernel/host +seam, `+.ld+` package loader, and the `+combine+` multimethod as the +single extension point for exotic reasoning operators. Domains (FL +class+operator packs, DT methodologies, the six TOC trees) ship as +`+.ld+` *data packages*, never core enums. + +*Deltas:* - *Rust:* `+.ld+` package loader + signing verification (L4); +`+combine+` multimethod registration hook in `+reason.rs+` invoked *only +outside the propagation hot loop* (see §8 do-not-build). Host builtins +already present. - *λδ:* methodology packages authored as data; +multimethod definitions for domain `+:op+`s; Agent definitions in the +enrich DSL. - *ReScript:* `+Editor.LambdaCell+` renders fenced +`+\'\'\'λδ+` blocks — *source collapsed, result inline* — through the +already-exported (currently unbound) `+evalLambdadelta+` +(budgeted/sandboxed), memoized by source hash. `+powerLevel >= 3+` gates +authoring. + +==== Track B — Local Intelligence + Typed Reasoning (the settled FL×DT integration) + +*What ships (summarized; not relitigated):* - *DEVONthink crown jewel = +one local intelligence engine,* `+core/src/index.rs+`: +`+#[serde(skip)]+`, rebuilt on load, DENSE `+DocId+` interning (~35–50 +MB @ 10k notes). Concordance/inverted index → *See-Also* (TF-IDF cosine +kNN), *BM25* ranked search, *Rocchio Classify* (+ naive-Bayes pack), +*blake3+SimHash* duplicate detection, *kNN auto-tag*. Smart Groups == +existing *Agents* (enrich DSL +`+similar:+`/`+near:+`/`+conf:+`/`+type:+`/`+edge:+`). Smart Rules == +*`+Notebook.triggers+`* (event→predicate→λδ Action). Replicants are free +(one `+Uuid+`). - *Flying Logic crown jewel = a separate additive +`+edges: Vec+` channel* on `+Note+` (empty +by default, byte-identical round-trip; `+links+` stays untyped). +Junctors/entities/groups are ordinary notes tagged +`+{:type :junct :op …}+`. `+core/src/reason.rs+` = pure single-pass DAG +`+propagate()+` (DFS back-edge exclusion + Kahn topo + *native +fuzzy-boolean operator table*; 0.5 = Indeterminate; edge-weight +transforms for support/abstain/negate). `+core/src/layout.rs+` = +Sugiyama layered layout → a *ReasoningView* replacing the GraphView +placeholder; positions *derived, never stored* on `+Note.position+`. λδ +`+combine+` multimethod only for exotic/domain ops. + +*Data-model deltas:* exactly *one struct* (`+Edge+`) and *one `+Note+` +field* (`+edges+`). No promotion of `+links → Vec+`. No persisted +derived indices. New native files: `+index.rs+`, `+edge.rs+`, +`+reason.rs+`, `+trigger.rs+`, later `+layout.rs+`. + +*FIRST PR (settled):* headless `+index.rs+` + tokenizer + incremental +reindex hooks (on `+set_content+` *and both title write paths*) + +`+rebuild_indices+`; property-tested against golden fixtures; *zero UI, +zero on-disk change.* + +*Deltas:* - *Rust:* the five files above; `+wasm.rs+` exports +`+see_also+`, `+duplicates+`, `+classify+`, `+add_edge+`, +`+remove_edge+`, `+reason+`, `+render_markdown+`. - *ReScript:* +`+WasmStore+` binds them; `+Update.refreshContext+` derives +See-Also/Duplicates/tag-suggest on selection change (only when the +selected id actually changes, never per keystroke); `+refreshReasoning+` +runs on entering `+PReasoning+` and on edge/confidence edits, debounced +like autosave. + +==== Track C — UI/UX Overhaul & Cross-Platform + +*What ships:* the hand-rolled TEA grows from a single-pane +textarea+canvas into a multi-pane workspace with a real editor, command +palette, inspector, context panels, the ReasoningView, +outline/timeline/browser, quick-capture/inbox, theming, and an +installable PWA — all gated by the two disclosure mechanisms in §6, +never an ad-hoc `+if+`. + +*Deltas:* the full `+Types+`/`+Model+`/`+Msg+`/`+Update+`/`+View+` map +is §6–7. Cross-platform: the WASM bundle is the single source of truth; +web-first PWA is primary (manifest + already-registered service worker + +IndexedDB/OPFS); the optional Gossamer webview shell (external sibling +checkout, outside this repo’s CI) hosts the identical bundle for desktop +*and* mobile with zero port and no type drift; Pointer Events unify +mouse/pen/touch; file-over-app JSON/Markdown/OPML is the only sync +boundary that will ever exist. + +''''' + +=== 6. UI/UX Overhaul Specifics + +==== The disclosure ladder — two gates, never an ad-hoc `+if+` + +[width="100%",cols="25%,25%,25%,25%",options="header",] +|=== +|L |Name |What appears |Gate +|*L0* |The Letter |plain notes, `+[[wikilinks]]+`, search, list, +sidebar, canvas, quick-capture. Usable for life, zero parens. |always on + +|*L1* |Living notes |See-Also / Duplicates / Backlinks panels; Inspector +(attrs/tags/prototype/confidence); inline λδ *results* (never source); +rendered markdown |*data-gated:* mounts iff its derived set is non-empty + +|*L2* |Composer |command palette, typed edges + drag-to-link, +ReasoningView, Agents in enrich DSL, outline/timeline/browser +|*door-gated:* `+powerLevel >= 2+` + +|*L3* |Programmer |λδ code cells, `+.ld+` packages, Smart Rules +(`+Notebook.triggers+`), multimethod authoring |`+powerLevel >= 3+` + +|*L4* |Kernel/host |kernel seam, budget tuning, host builtin +registration, package signing |`+powerLevel >= 4+` +|=== + +[source,rescript] +---- +// Model.res — one scalar governs door-gating; persisted like theme. +powerLevel: int, // 0..4, default 0. A door, never a wall. + +// View.res — the two gates as reusable combinators. +let whenPower = (model, level, node) => model.powerLevel >= level ? node : React.null +let whenNonEmpty = (xs, render) => Array.length(xs) == 0 ? React.null : render(xs) +---- + +____ +*Reviewer rule (tested):* an L1 panel rendering an empty header is a +bug; an L2 button visible at `+powerLevel == 0+` is a bug. +____ + +==== Consolidated TEA deltas (the spine everything hangs on) + +`+Types.res+` — all additive; `+note+` gains exactly *one* field +(`+edges+`, matching the decided FL×DT channel): + +[source,rescript] +---- +type edgeKind = Supports | Opposes | Requires | Causes | Contradicts | Custom(string) +type edge = { to_: noteId, kind: edgeKind, weight: float, attrs: Js.Dict.t } +// note gains: edges: array, (links stays untyped association) +type confidence = float // 0.0..1.0 ; 0.5 = Indeterminate +type suggestion = { id: noteId, score: float } // See-Also / duplicates / classify rows +type mdNode = // produced by Rust core (pulldown-cmark → typed AST) + | MdText(string) | MdStrong(array) | MdEmph(array) | MdCode(string) + | MdLink(string, array) | MdWikiLink(noteId, string) | MdWikiDangling(string) + | MdHeading(int, array) | MdPara(array) | MdList(bool, array>) + | MdTask(bool, array) | MdQuote(array) | MdCodeBlock(string, string) // lang="λδ" ⇒ cell +type facet = FacetAll | FacetTag(string) | FacetAttr(string, string) +type paneContent = PList | PCanvas | PReasoning | POutline | PTimeline | PBrowser(facet) | PInbox +type pane = { id: string, content: paneContent, flex: float } +type peek = { anchor: noteId, target: noteId, x: float, y: float } +type paletteMode = PCommand | PGoto | PSlash | PLink +type editorMode = ESource | ERendered | ESplit +type theme = TSystem | TLight | TDark +type canvasTool = TSelect | TEdge(edgeKind) +type reasonNode = { id: noteId, x: float, y: float, layer: int, derived: confidence, isDriver: bool, isDriven: bool } +type reasonEdge = { from: noteId, to_: noteId, kind: edgeKind, weight: float, back: bool } +type reasonResult = { nodes: array, edges: array } +---- + +`+Model.res+` additions (grouped): +`+panes / focusedPane / inspectorOpen / peek+`; +`+palette / autocomplete+`; `+editorMode+`; +`+contextFor / seeAlso / duplicates / tagSuggest+`; `+reasoning+`; +`+outlineExpanded / timelineField / canvasTool / marquee / edgeDraft+`; +`+theme / installPrompt / narrow / powerLevel+`. + +`+Msg.res+` additions (named, grouped): editor (`+SetEditorMode+`, +`+OpenAutocomplete+`, `+AcceptAutocomplete+`, `+InsertAtCaret+`, +`+RunLambdaCell+`, `+LambdaCellResult+`); palette +(`+OpenPalette+`/`+UpdatePalette+`/`+MovePalette+`/`+RunPaletteItem+`/`+ClosePalette+`); +panes/inspector/peek +(`+SplitPane+`/`+FocusPane+`/`+SetPaneContent+`/`+ToggleInspector+`/`+OpenPeek+`/`+ClosePeek+`/`+SetAttribute+`/`+SetPrototype+`/`+SetConfidence+`/`+AddTag+`/`+RemoveTag+`); +context (`+RefreshContext+`/`+MergeDuplicate+`/`+AcceptTagSuggestion+`); +reasoning +(`+RefreshReasoning+`/`+AddEdge+`/`+RemoveEdge+`/`+SetEdgeWeight+`); +views +(`+ToggleOutlineNode+`/`+SetTimelineField+`/`+SetCanvasTool+`/`+StartMarquee+`…/`+StartEdgeDraw+`…/`+RecenterCanvas+`); +inbox (`+QuickCapture+`/`+FileFromInbox+`); env +(`+SetTheme+`/`+CapturedInstallPrompt+`/`+PromptInstall+`/`+SetNarrow+`/`+SetPowerLevel+`). + +The *one `+Update.res+` change* everything plugs into — extend the +existing "`re-run active agent after each step`" wrapper to also refresh +derived context on selection change (cheap: fires only when the selected +id changes; the DT index is incrementally maintained core-side): + +[source,rescript] +---- +let rec update = (model, msg) => model->step(msg)->refreshAgent->refreshContext(model) +and refreshContext = (prev, next) => { + let sel = m => switch m.selection { | SingleNote(id) => Some(id) | _ => None } + switch sel(next) { + | Some(id) when sel(prev) != Some(id) => + { ...next, contextFor: Some(id), + seeAlso: WasmStore.seeAlso(id), duplicates: WasmStore.duplicates(id), tagSuggest: WasmStore.classify(id) } + | _ => next } } +---- + +`+refreshReasoning+` is deliberately *not* in the hot path — it runs on +entering `+PReasoning+` and on +`+AddEdge+`/`+RemoveEdge+`/`+SetConfidence+`, debounced like +`+Persist.scheduleAutosave+`. The existing `+SetViewMode(mode)+` becomes +sugar over `+SetPaneContent(model.focusedPane, …)+` so the current +Toolbar keeps working verbatim. + +`+WasmStore.res+` seam additions (each maps to a `+wasm.rs+` method): +the complete existing surface—including `+set_attribute+`, +`+run_query+`, `+backlinks+`, and all λδ entry points—is now bound; add +the new FL×DT exports `+render_markdown+`, `+see_also+`, `+duplicates+`, +`+classify+`, `+add_edge+`, `+remove_edge+`, `+reason+`, +`+set_prototype+`. All of +`+reason+`/`+see_also+`/`+duplicates+`/`+classify+` are +`+#[serde(skip)]+`-backed and rebuilt on load; `+reasonResult+` +positions come from `+layout.rs+`, never `+Note.position+`. + +==== Surface-by-surface + +* *Editor* (`+Editor.res+`, replaces the `+NoteEditor+` textarea). +Controlled `+