diff --git a/ABI-FFI-README.md b/ABI-FFI-README.adoc similarity index 75% rename from ABI-FFI-README.md rename to ABI-FFI-README.adoc index d496803..3227315 100644 --- a/ABI-FFI-README.md +++ b/ABI-FFI-README.adoc @@ -1,17 +1,20 @@ -# EvidenceGraph ABI/FFI Documentation +== EvidenceGraph ABI/FFI Documentation -## Overview +=== Overview -This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: +This library follows the *Hyperpolymath RSR Standard* for ABI and FFI +design: -- **ABI (Application Binary Interface)** defined in **Idris2** with formal proofs -- **FFI (Foreign Function Interface)** implemented in **Zig** for C compatibility -- **Generated C headers** bridge Idris2 ABI to Zig FFI -- **Any language** can call through standard C ABI +* *ABI (Application Binary Interface)* defined in *Idris2* with formal +proofs +* *FFI (Foreign Function Interface)* implemented in *Zig* for C +compatibility +* *Generated C headers* bridge Idris2 ABI to Zig FFI +* *Any language* can call through standard C ABI -## Architecture +=== Architecture -``` +.... ┌─────────────────────────────────────────────┐ │ ABI Definitions (Idris2) │ │ src/abi/ │ @@ -43,11 +46,11 @@ This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: │ Any Language via C ABI │ │ - Rust, AffineScript, Julia, Python, etc. │ └─────────────────────────────────────────────┘ -``` +.... -## Directory Structure +=== Directory Structure -``` +.... evidence_graph/ ├── src/ │ ├── abi/ # ABI definitions (Idris2) @@ -75,15 +78,17 @@ evidence_graph/ ├── rust/ ├── affinescript/ └── julia/ -``` +.... -## Why Idris2 for ABI? +=== Why Idris2 for ABI? -### 1. **Formal Verification** +==== 1. *Formal Verification* -Idris2's dependent types allow proving properties about the ABI at compile-time: +Idris2’s dependent types allow proving properties about the ABI at +compile-time: -```idris +[source,idris] +---- -- Prove struct size is correct public export exampleStructSize : HasSize ExampleStruct 16 @@ -95,13 +100,14 @@ fieldAligned : Divides 8 (offsetOf ExampleStruct.field) -- Prove ABI is platform-compatible public export abiCompatible : Compatible (ABI 1) (ABI 2) -``` +---- -### 2. **Type Safety** +==== 2. *Type Safety* Encode invariants that C/Zig cannot express: -```idris +[source,idris] +---- -- Non-null pointer guaranteed at type level data Handle : Type where MkHandle : (ptr : Bits64) -> {auto 0 nonNull : So (ptr /= 0)} -> Handle @@ -109,13 +115,14 @@ data Handle : Type where -- Array with length proof data Buffer : (n : Nat) -> Type where MkBuffer : Vect n Byte -> Buffer n -``` +---- -### 3. **Platform Abstraction** +==== 3. *Platform Abstraction* Platform-specific types with compile-time selection: -```idris +[source,idris] +---- CInt : Platform -> Type CInt Linux = Bits32 CInt Windows = Bits32 @@ -123,13 +130,14 @@ CInt Windows = Bits32 CSize : Platform -> Type CSize Linux = Bits64 CSize Windows = Bits64 -``` +---- -### 4. **Safe Evolution** +==== 4. *Safe Evolution* Prove that new ABI versions are backward-compatible: -```idris +[source,idris] +---- -- Compiler enforces compatibility abiUpgrade : ABI 1 -> ABI 2 abiUpgrade old = MkABI2 { @@ -138,71 +146,78 @@ abiUpgrade old = MkABI2 { -- Can add new fields new_features = defaults } -``` +---- -## Why Zig for FFI? +=== Why Zig for FFI? -### 1. **C ABI Compatibility** +==== 1. *C ABI Compatibility* Zig exports C-compatible functions naturally: -```zig +[source,zig] +---- export fn library_function(param: i32) i32 { return param * 2; } -``` +---- -### 2. **Memory Safety** +==== 2. *Memory Safety* Compile-time safety without runtime overhead: -```zig +[source,zig] +---- // Null check enforced at compile time const handle = init() orelse return error.InitFailed; defer free(handle); -``` +---- -### 3. **Cross-Compilation** +==== 3. *Cross-Compilation* Built-in cross-compilation to any platform: -```bash +[source,bash] +---- zig build -Dtarget=x86_64-linux zig build -Dtarget=aarch64-macos zig build -Dtarget=x86_64-windows -``` +---- -### 4. **Zero Dependencies** +==== 4. *Zero Dependencies* No runtime, no libc required (unless explicitly needed): -```zig +[source,zig] +---- // Minimal binary size pub const lib = @import("std"); // Only includes what you use -``` +---- -## Building +=== Building -### Build FFI Library +==== Build FFI Library -```bash +[source,bash] +---- cd ffi/zig zig build # Build debug zig build -Doptimize=ReleaseFast # Build optimized zig build test # Run tests -``` +---- -### Generate C Header from Idris2 ABI +==== Generate C Header from Idris2 ABI -```bash +[source,bash] +---- cd src/abi idris2 --cg c-header Types.idr -o ../../generated/abi/evidence_graph.h -``` +---- -### Cross-Compile +==== Cross-Compile -```bash +[source,bash] +---- cd ffi/zig # Linux x86_64 @@ -213,13 +228,14 @@ zig build -Dtarget=aarch64-macos # Windows x86_64 zig build -Dtarget=x86_64-windows -``` +---- -## Usage +=== Usage -### From C +==== From C -```c +[source,c] +---- #include "evidence_graph.h" int main() { @@ -235,16 +251,19 @@ int main() { evidence_graph_free(handle); return 0; } -``` +---- Compile with: -```bash + +[source,bash] +---- gcc -o example example.c -levidence_graph -L./zig-out/lib -``` +---- -### From Idris2 +==== From Idris2 -```idris +[source,idris] +---- import EvidenceGraph.ABI.Foreign main : IO () @@ -257,11 +276,12 @@ main = do free handle putStrLn "Success" -``` +---- -### From Rust +==== From Rust -```rust +[source,rust] +---- #[link(name = "evidence_graph")] extern "C" { fn evidence_graph_init() -> *mut std::ffi::c_void; @@ -280,11 +300,12 @@ fn main() { evidence_graph_free(handle); } } -``` +---- -### From Julia +==== From Julia -```julia +[source,julia] +---- const libevidence_graph = "libevidence_graph" function init() @@ -310,27 +331,30 @@ try finally cleanup(handle) end -``` +---- -## Testing +=== Testing -### Unit Tests (Zig) +==== Unit Tests (Zig) -```bash +[source,bash] +---- cd ffi/zig zig build test -``` +---- -### Integration Tests +==== Integration Tests -```bash +[source,bash] +---- cd ffi/zig zig build test-integration -``` +---- -### ABI Verification (Idris2) +==== ABI Verification (Idris2) -```idris +[source,idris] +---- -- Compile-time verification %runElab verifyABI @@ -340,44 +364,44 @@ main = do verifyLayoutsCorrect verifyAlignmentsCorrect putStrLn "ABI verification passed" -``` +---- -## Contributing +=== Contributing When modifying the ABI/FFI: -1. **Update ABI first** (`src/abi/*.idr`) - - Modify type definitions - - Update proofs - - Ensure backward compatibility - -2. **Generate C header** - ```bash - idris2 --cg c-header src/abi/Types.idr -o generated/abi/evidence_graph.h - ``` - -3. **Update FFI implementation** (`ffi/zig/src/main.zig`) - - Implement new functions - - Match ABI types exactly - -4. **Add tests** - - Unit tests in Zig - - Integration tests - - ABI verification tests - -5. **Update documentation** - - Function signatures - - Usage examples - - Migration guide (if breaking changes) - -## License +[arabic] +. *Update ABI first* (`+src/abi/*.idr+`) +* Modify type definitions +* Update proofs +* Ensure backward compatibility +. *Generate C header* ++ +[source,bash] +---- +idris2 --cg c-header src/abi/Types.idr -o generated/abi/evidence_graph.h +---- +. *Update FFI implementation* (`+ffi/zig/src/main.zig+`) +* Implement new functions +* Match ABI types exactly +. *Add tests* +* Unit tests in Zig +* Integration tests +* ABI verification tests +. *Update documentation* +* Function signatures +* Usage examples +* Migration guide (if breaking changes) + +=== License MPL-2.0 -## See Also +=== See Also -- [Idris2 Documentation](https://idris2.readthedocs.io) -- [Zig Documentation](https://ziglang.org/documentation/master/) -- [Rhodium Standard Repositories](https://github.com/hyperpolymath/rhodium-standard-repositories) -- [FFI Migration Guide](../ffi-migration-guide.md) -- [ABI Migration Guide](../abi-migration-guide.md) +* https://idris2.readthedocs.io[Idris2 Documentation] +* https://ziglang.org/documentation/master/[Zig Documentation] +* https://github.com/hyperpolymath/rhodium-standard-repositories[Rhodium +Standard Repositories] +* link:../ffi-migration-guide.md[FFI Migration Guide] +* link:../abi-migration-guide.md[ABI Migration Guide] diff --git a/ARCHITECTURE.md b/ARCHITECTURE.adoc similarity index 60% rename from ARCHITECTURE.md rename to ARCHITECTURE.adoc index 46949e5..04c157f 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.adoc @@ -1,31 +1,40 @@ -# Evidence Graph Architecture +== Evidence Graph Architecture -## Vision +=== Vision -Infrastructure for pragmatic epistemology in investigative journalism. Combines: -- **i-docs navigation principles**: Navigation over narration, reader agency -- **PROMPT framework**: 6-dimensional epistemological scoring -- **Boundary objects theory**: Multiple perspectives on same evidence +Infrastructure for pragmatic epistemology in investigative journalism. +Combines: - *i-docs navigation principles*: Navigation over narration, +reader agency - *PROMPT framework*: 6-dimensional epistemological +scoring - *Boundary objects theory*: Multiple perspectives on same +evidence -## Philosophical Foundation +=== Philosophical Foundation -### The Core Argument +==== The Core Argument -We didn't fall from Truth to Post-Truth; we evolved to complex epistemology without building infrastructure. This system IS that infrastructure. +We didn’t fall from Truth to Post-Truth; we evolved to complex +epistemology without building infrastructure. This system IS that +infrastructure. -### Design Principles +==== Design Principles -1. **Coordination without consensus**: Different audiences navigate same evidence differently -2. **Measurable epistemology**: PROMPT scores make evidence quality explicit -3. **Progressive enhancement**: Works without JavaScript, enhanced with it -4. **Open by default**: Open source, EU data sovereignty, NUJ network adoption +[arabic] +. *Coordination without consensus*: Different audiences navigate same +evidence differently +. *Measurable epistemology*: PROMPT scores make evidence quality +explicit +. *Progressive enhancement*: Works without JavaScript, enhanced with it +. *Open by default*: Open source, EU data sovereignty, NUJ network +adoption -## Data Model +=== Data Model -### Core Entities +==== Core Entities -#### Claims -```elixir +===== Claims + +[source,elixir] +---- %Claim{ id: UUID, text: String, @@ -45,10 +54,12 @@ We didn't fall from Truth to Post-Truth; we evolved to complex epistemology with updated_at: DateTime, metadata: Map } -``` +---- + +===== Evidence -#### Evidence -```elixir +[source,elixir] +---- %Evidence{ id: UUID, title: String, @@ -65,10 +76,12 @@ We didn't fall from Truth to Post-Truth; we evolved to complex epistemology with created_at: DateTime, metadata: Map } -``` +---- -#### Relationships (Edges) -```elixir +===== Relationships (Edges) + +[source,elixir] +---- %Relationship{ _from: "claims/:uuid", _to: "evidence/:uuid", @@ -80,10 +93,12 @@ We didn't fall from Truth to Post-Truth; we evolved to complex epistemology with created_at: DateTime, metadata: Map } -``` +---- + +===== Navigation Paths -#### Navigation Paths -```elixir +[source,elixir] +---- %NavigationPath{ id: UUID, investigation_id: UUID, @@ -100,10 +115,12 @@ We didn't fall from Truth to Post-Truth; we evolved to complex epistemology with }], created_at: DateTime } -``` +---- -### Investigations (Container) -```elixir +==== Investigations (Container) + +[source,elixir] +---- %Investigation{ id: UUID, title: String, @@ -116,28 +133,25 @@ We didn't fall from Truth to Post-Truth; we evolved to complex epistemology with created_at: DateTime, updated_at: DateTime } -``` +---- + +=== Database Architecture -## Database Architecture +==== Phase 1-2: ArangoDB (Multi-Model) -### Phase 1-2: ArangoDB (Multi-Model) +*Why ArangoDB:* - Native graph + document database - Strong Elixir +integration via `+arangox+` - AQL (query language) supports complex +traversals - Production-ready, €45/month managed hosting - JSON storage +preserves Schema.org/Dublin Core -**Why ArangoDB:** -- Native graph + document database -- Strong Elixir integration via `arangox` -- AQL (query language) supports complex traversals -- Production-ready, €45/month managed hosting -- JSON storage preserves Schema.org/Dublin Core +*Collections:* - `+investigations+` (document) - `+claims+` (document) - +`+evidence+` (document) - `+navigation_paths+` (document) - +`+relationships+` (edge collection) -**Collections:** -- `investigations` (document) -- `claims` (document) -- `evidence` (document) -- `navigation_paths` (document) -- `relationships` (edge collection) +*Indexes:* -**Indexes:** -```javascript +[source,javascript] +---- // Full-text search on claims/evidence db.claims.ensureIndex({ type: "fulltext", fields: ["text"] }) db.evidence.ensureIndex({ type: "fulltext", fields: ["title", "metadata"] }) @@ -152,22 +166,22 @@ db.evidence.ensureIndex({ type: "hash", fields: ["investigation_id"] }) // Zotero sync db.evidence.ensureIndex({ type: "hash", fields: ["zotero_key"] }) -``` +---- -### Phase 3+: Optional Virtuoso (RDF/SPARQL) +==== Phase 3+: Optional Virtuoso (RDF/SPARQL) -Add only if semantic web integration becomes critical: -- Academic repository linking -- Cross-investigation semantic queries -- Linked Open Data publishing +Add only if semantic web integration becomes critical: - Academic +repository linking - Cross-investigation semantic queries - Linked Open +Data publishing -**Migration path:** JSON-LD stored from day 1 enables easy export to RDF. +*Migration path:* JSON-LD stored from day 1 enables easy export to RDF. -## API Design +=== API Design -### GraphQL Schema (Absinthe) +==== GraphQL Schema (Absinthe) -```graphql +[source,graphql] +---- type Investigation { id: ID! title: String! @@ -334,49 +348,55 @@ input CreateRelationshipInput { confidence: Float! reasoning: String } -``` +---- -### REST Endpoints (Phoenix) +==== REST Endpoints (Phoenix) For Zotero integration and simple operations: -``` +.... POST /api/v1/evidence/import # Zotero → Evidence GET /api/v1/evidence/:id/export # Evidence → Zotero JSON POST /api/v1/investigations/:id/export # Export investigation GET /api/v1/health # System health -``` +.... + +=== Technology Stack -## Technology Stack +==== Backend -### Backend -- **Elixir 1.16+** - Functional, concurrent, fault-tolerant -- **Phoenix 1.7+** - Web framework -- **Absinthe 1.7+** - GraphQL implementation -- **ArangoDB 3.11+** - Multi-model database -- **arangox** - Elixir ArangoDB driver +* *Elixir 1.16+* - Functional, concurrent, fault-tolerant +* *Phoenix 1.7+* - Web framework +* *Absinthe 1.7+* - GraphQL implementation +* *ArangoDB 3.11+* - Multi-model database +* *arangox* - Elixir ArangoDB driver -### Frontend -- **Phoenix LiveView** - Server-rendered, real-time UI (progressive enhancement) -- **D3.js** - Graph visualization -- **Alpine.js** (minimal) - Progressive JavaScript enhancement -- **Tailwind CSS** - Styling +==== Frontend -### Integration -- **Julia 1.10+** - Statistical analysis, PROMPT scoring algorithms -- **IPFS** - Evidence provenance and archival -- **Zotero API** - Two-way sync +* *Phoenix LiveView* - Server-rendered, real-time UI (progressive +enhancement) +* *D3.js* - Graph visualization +* *Alpine.js* (minimal) - Progressive JavaScript enhancement +* *Tailwind CSS* - Styling -### Infrastructure -- **Podman/Docker** - Containerization -- **Hetzner Cloud** - EU hosting (data sovereignty) -- **GitHub Actions** - CI/CD +==== Integration -## Key Algorithms +* *Julia 1.10+* - Statistical analysis, PROMPT scoring algorithms +* *IPFS* - Evidence provenance and archival +* *Zotero API* - Two-way sync -### PROMPT Overall Score Calculation +==== Infrastructure -```elixir +* *Podman/Docker* - Containerization +* *Hetzner Cloud* - EU hosting (data sovereignty) +* *GitHub Actions* - CI/CD + +=== Key Algorithms + +==== PROMPT Overall Score Calculation + +[source,elixir] +---- defmodule EvidenceGraph.Scoring do @weights %{ provenance: 0.20, @@ -393,13 +413,14 @@ defmodule EvidenceGraph.Scoring do end) end end -``` +---- -### Relationship Weight Propagation +==== Relationship Weight Propagation -For evidence chains: `claim₁ → evidence₁ → claim₂ → evidence₂` +For evidence chains: `+claim₁ → evidence₁ → claim₂ → evidence₂+` -```elixir +[source,elixir] +---- defmodule EvidenceGraph.Traversal do def propagated_weight(path) do path @@ -411,13 +432,14 @@ defmodule EvidenceGraph.Traversal do relationship.weight * relationship.confidence end end -``` +---- -### Navigation Path Scoring +==== Navigation Path Scoring Different audiences prioritize different PROMPT dimensions: -```elixir +[source,elixir] +---- @audience_weights %{ researcher: %{methodology: 0.35, replicability: 0.30, transparency: 0.20}, policymaker: %{provenance: 0.30, publication: 0.25, objective: 0.25}, @@ -426,23 +448,27 @@ Different audiences prioritize different PROMPT dimensions: affected_person: %{objective: 0.35, provenance: 0.30, transparency: 0.20}, journalist: %{provenance: 0.25, transparency: 0.25, replicability: 0.20} } -``` +---- + +=== Security & Privacy -## Security & Privacy +==== Authentication -### Authentication -- JWT tokens for API access -- Role-based access: `admin`, `journalist`, `reviewer`, `reader` -- Optional DIDs (Decentralized Identifiers) for blockchain integration +* JWT tokens for API access +* Role-based access: `+admin+`, `+journalist+`, `+reviewer+`, `+reader+` +* Optional DIDs (Decentralized Identifiers) for blockchain integration -### Data Protection -- EU GDPR compliant -- Interview subjects can be anonymized -- Evidence can be marked as `sensitive` (restricted access) -- Audit logs for all mutations +==== Data Protection -### IPFS Integration -```elixir +* EU GDPR compliant +* Interview subjects can be anonymized +* Evidence can be marked as `+sensitive+` (restricted access) +* Audit logs for all mutations + +==== IPFS Integration + +[source,elixir] +---- defmodule EvidenceGraph.IPFS do def store_evidence(file_path) do {:ok, hash} = Kubo.add(file_path) @@ -455,17 +481,18 @@ defmodule EvidenceGraph.IPFS do current_hash == evidence.ipfs_hash end end -``` +---- -## Performance Considerations +=== Performance Considerations -### Query Optimization +==== Query Optimization -**Problem:** Deep evidence chains (10+ hops) can be slow +*Problem:* Deep evidence chains (10+ hops) can be slow -**Solution:** Pre-computed materialized paths +*Solution:* Pre-computed materialized paths -```javascript +[source,javascript] +---- // ArangoDB: Store paths up to depth 3 FOR claim IN claims LET paths = ( @@ -473,48 +500,54 @@ FOR claim IN claims RETURN p ) UPDATE claim WITH { materialized_paths: paths } IN claims -``` +---- + +==== Caching Strategy -### Caching Strategy +* *GraphQL:* DataLoader for N+1 query prevention +* *LiveView:* ETS cache for PROMPT score calculations +* *ArangoDB:* Query result cache (built-in) -- **GraphQL:** DataLoader for N+1 query prevention -- **LiveView:** ETS cache for PROMPT score calculations -- **ArangoDB:** Query result cache (built-in) +==== Benchmarks (Target) -### Benchmarks (Target) +* Single investigation load: < 500ms +* Evidence chain (depth 5): < 1s +* Full-text search: < 200ms +* GraphQL mutation: < 100ms -- Single investigation load: < 500ms -- Evidence chain (depth 5): < 1s -- Full-text search: < 200ms -- GraphQL mutation: < 100ms +=== Testing Strategy -## Testing Strategy +==== Unit Tests (ExUnit) -### Unit Tests (ExUnit) -- Data model validations -- PROMPT score calculations -- Relationship weight propagation +* Data model validations +* PROMPT score calculations +* Relationship weight propagation -### Integration Tests -- GraphQL query/mutation flows -- Zotero import/export -- ArangoDB CRUD operations +==== Integration Tests -### E2E Tests (Wallaby) -- Complete investigation workflow -- Navigation path creation -- Graph visualization interaction +* GraphQL query/mutation flows +* Zotero import/export +* ArangoDB CRUD operations -### User Testing (Phase 1 Goal) -- 25 participants from NUJ network -- One complete investigation (7 claims, 30 evidence items) -- 3 navigation paths tested -- Qualitative feedback on PROMPT UI +==== E2E Tests (Wallaby) -## Migration & Deployment +* Complete investigation workflow +* Navigation path creation +* Graph visualization interaction -### Development -```bash +==== User Testing (Phase 1 Goal) + +* 25 participants from NUJ network +* One complete investigation (7 claims, 30 evidence items) +* 3 navigation paths tested +* Qualitative feedback on PROMPT UI + +=== Migration & Deployment + +==== Development + +[source,bash] +---- # Local ArangoDB podman run -p 8529:8529 -e ARANGO_ROOT_PASSWORD=dev arangodb/arangodb:3.11 @@ -523,93 +556,103 @@ mix phx.new evidence_graph --database postgres # For user auth only mix deps.get mix ecto.setup iex -S mix phx.server -``` +---- -### Production (Hetzner) -- ArangoDB Cloud: €45/month managed instance -- Phoenix: Debian 12, systemd service -- Nginx reverse proxy, Let's Encrypt SSL -- Automated backups via ArangoDB Cloud +==== Production (Hetzner) -### Data Migration Path +* ArangoDB Cloud: €45/month managed instance +* Phoenix: Debian 12, systemd service +* Nginx reverse proxy, Let’s Encrypt SSL +* Automated backups via ArangoDB Cloud -**Phase 1 → Phase 2:** Schema evolution within ArangoDB -**Phase 2 → Phase 3:** If adding Virtuoso: +==== Data Migration Path -```elixir +*Phase 1 → Phase 2:* Schema evolution within ArangoDB *Phase 2 → Phase +3:* If adding Virtuoso: + +[source,elixir] +---- # Export to JSON-LD evidence |> Jason.encode!() |> JSONLD.expand() # Import to Virtuoso Virtuoso.import_turtle(jsonld_to_turtle(data)) -``` +---- + +=== Open Questions -## Open Questions +[arabic] +. *PROMPT scoring UI*: Sliders vs. dropdowns vs. questionnaire? +. *Real-time collaboration*: Conflict resolution for concurrent edits? +. *Evidence versioning*: Full history or snapshots? +. *Mobile experience*: Responsive web or native app (future)? -1. **PROMPT scoring UI**: Sliders vs. dropdowns vs. questionnaire? -2. **Real-time collaboration**: Conflict resolution for concurrent edits? -3. **Evidence versioning**: Full history or snapshots? -4. **Mobile experience**: Responsive web or native app (future)? +=== References -## References +* ArangoDB Multi-Model: https://www.arangodb.com/docs/stable/ +* Absinthe GraphQL: https://hexdocs.pm/absinthe/ +* IPFS Docs: https://docs.ipfs.tech/ +* Dublin Core: https://www.dublincore.org/specifications/dublin-core/ +* Schema.org: https://schema.org/ -- ArangoDB Multi-Model: https://www.arangodb.com/docs/stable/ -- Absinthe GraphQL: https://hexdocs.pm/absinthe/ -- IPFS Docs: https://docs.ipfs.tech/ -- Dublin Core: https://www.dublincore.org/specifications/dublin-core/ -- Schema.org: https://schema.org/ +''''' ---- +*Last Updated:* 2026-03-13 *Status:* Phase 1 complete (v1.0.0). Phase 2 +in progress: migrating to Lithoglyph as primary evidence store. -**Last Updated:** 2026-03-13 -**Status:** Phase 1 complete (v1.0.0). Phase 2 in progress: migrating to Lithoglyph as primary evidence store. +=== Lithoglyph Migration (Phase 2→3) -## Lithoglyph Migration (Phase 2→3) +==== Current Architecture (Phase 1) -### Current Architecture (Phase 1) -``` +.... Browser → nginx → Phoenix (LiveView + REST + GraphQL) ↓ ↓ PostgreSQL ArangoDB 3.11+ (user auth) (evidence, claims, entities, relationships, navigation) -``` +.... + +==== Target Architecture (Phase 2, in progress) -### Target Architecture (Phase 2, in progress) -``` +.... Browser → nginx → Phoenix (LiveView + REST + GraphQL) ↓ ↓ ↓ PostgreSQL Lithoglyph ArangoDB (user auth) (evidence, (relationships claims, edge collection entities) only) -``` +.... -### Final Architecture (Phase 3) -``` +==== Final Architecture (Phase 3) + +.... Browser → nginx → Phoenix (LiveView + REST + GraphQL) ↓ ↓ PostgreSQL Lithoglyph (user auth) (all domain data: evidence, claims, entities, relationships) -``` - -### Why Lithoglyph - -ArangoDB was the right choice for Phase 1 (quick to prototype, multi-model). -Lithoglyph is the right choice long-term because: - -1. **Provenance is mandatory** — every mutation is a story event with actor + rationale -2. **PROMPT scores as first-class types** — `BoundedNat 0 100` verified at compile time -3. **GQL-DT dependent types** — type-safe queries with proof obligations -4. **Audit-grade WAL** — every write is journaled, reversible -5. **No data duplication** — Docudactyl → Lithoglyph → bofig queries directly (no import step) - -### Migration Strategy - -Each collection migrates independently: -1. **Evidence** (first) — highest value, most data, Lithoglyph already stores it -2. **Entities** — NER-resolved entities with aliases and merge history -3. **Claims** — with PROMPT scores as GQL-DT types -4. **Relationships** (last) — requires Factor GQL graph traversal support +.... + +==== Why Lithoglyph + +ArangoDB was the right choice for Phase 1 (quick to prototype, +multi-model). Lithoglyph is the right choice long-term because: + +[arabic] +. *Provenance is mandatory* — every mutation is a story event with actor ++ rationale +. *PROMPT scores as first-class types* — `+BoundedNat 0 100+` verified +at compile time +. *GQL-DT dependent types* — type-safe queries with proof obligations +. *Audit-grade WAL* — every write is journaled, reversible +. *No data duplication* — Docudactyl → Lithoglyph → bofig queries +directly (no import step) + +==== Migration Strategy + +Each collection migrates independently: 1. *Evidence* (first) — highest +value, most data, Lithoglyph already stores it 2. *Entities* — +NER-resolved entities with aliases and merge history 3. *Claims* — with +PROMPT scores as GQL-DT types 4. *Relationships* (last) — requires +Factor GQL graph traversal support diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc new file mode 100644 index 0000000..2ea12b9 --- /dev/null +++ b/CHANGELOG.adoc @@ -0,0 +1,316 @@ +== Changelog + +All notable changes to the Evidence Graph project will be documented in +this file. + +The format is based on https://keepachangelog.com/en/1.0.0/[Keep a +Changelog], and this project adheres to +https://semver.org/spec/v2.0.0.html[Semantic Versioning]. + +=== [Unreleased] + +==== Planned (Phase 2) + +* Zotero browser extension (one-click import) +* Multi-investigation dashboard with cross-referencing +* Real-time collaborative editing via Phoenix PubSub +* Advanced D3.js visualisations (timeline, heatmap, Sankey) +* Role-based access control +* IPFS provenance integration +* Hetzner Cloud deployment + +=== [1.0.0] - 2026-02-21 - "`Phase 1 PoC`" (Evidence Graph v1) + +==== Added + +===== LiveView Frontend + +* *5 LiveView pages*: Dashboard (investigation list), Investigation +detail, Graph visualisation, PROMPT scoring, Navigation paths +* *Core components*: layouts, navigation, responsive design +* *D3.js hooks*: Force-directed graph + radar chart visualisations + +===== User Authentication + +* *phx.gen.auth*: Registration, login, settings, magic link confirmation +* *bcrypt*: Password hashing via bcrypt_elixir +* *Swoosh*: Email delivery for magic links +* *Session-based auth*: CSRF protection on all state-changing operations + +===== Zotero REST API + +* `+POST /api/evidence/import+` - Import single evidence from Zotero +* `+POST /api/evidence/batch-import+` - Batch import multiple items +* `+GET /api/evidence/:id/export+` - Export evidence to Zotero format +* `+GET /api/investigations/:id/sync-status+` - Check Zotero sync status + +===== Production Deployment + +* *Containerfile*: Multi-stage OCI build (Elixir 1.18/OTP 27 + Debian +bookworm) +* *podman-compose.yml*: ArangoDB + PostgreSQL + app with health checks +* *nginx config*: Reverse proxy with WebSocket support +* *systemd service*: Production process management +* *runtime.exs*: Production configuration from environment variables +* *Health endpoint*: `+GET /api/health+` for container orchestration + +===== Data & Seeds + +* Expanded seed data: 7 claims, 30 evidence items, 38 relationships, 6 +navigation paths +* Audience-weighted navigation for 6 types (journalist, researcher, +policymaker, skeptic, affected person, activist) + +===== NUJ Testing Protocols + +* Task script for user testing sessions +* Consent form for participants +* Feedback form with SUS scale +* Decision matrix for Month 3 go/no-go +* 5 AsciiDoc protocol documents in docs/testing/ + +===== Compliance & Trust + +* *A2ML v2.1 Trustfile*: Full cyberwar-ready trustfile +(bofig.trustfile.a2ml) +* *Contractiles*: Updated Mustfile, Dustfile, Intentfile, Trustfile +* *RSR compliance*: All 12 required files present +* *19 GitHub Actions workflows*: Security, quality, mirroring, +enforcement +* *TOPOLOGY.md*: Architecture diagram + completion dashboard + +==== Changed + +* Bumped Phoenix from 1.7.10 to 1.8.3 +* Bumped Phoenix LiveView to 1.1.19 +* Replaced docker-compose.yml with podman-compose.yml +* Replaced Justfile (legacy Docker references) with comprehensive +Podman-based recipes +* Expanded Mustfile from 3 to 6 mandatory checks +* Updated SECURITY.md to reflect implemented authentication +* Updated ROADMAP.adoc with actual 18-month plan + +==== Removed + +* `+docker-compose.yml+` (replaced by podman-compose.yml) +* `+Podmanfile.md+` (superseded by Containerfile) +* `+affinescript.json+` (AffineScript not used in Elixir project) +* `+CHANGELOG.adoc+` (duplicate of CHANGELOG.md) +* `+CONTRIBUTING.md+` (kept CONTRIBUTING.adoc) +* `+MAINTAINERS.adoc+` (kept MAINTAINERS.md) +* `+LICENSE.txt+` (kept LICENSE) +* `+contractiles/trust/Trustfile.hs+` (replaced by A2ML Trustfile) + +==== Security + +* User authentication implemented (phx.gen.auth) +* All browser routes require authentication +* CSRF protection on all forms +* Parameterized AQL queries (unchanged) +* Ecto changesets for all input validation +* Health endpoint unauthenticated (for container orchestration) +* `+mix deps.audit+` integrated into CI + +=== [0.1.0] - 2025-11-22 - "`Foundation`" (Phase 1 Month 1) + +==== Added + +===== Core Infrastructure + +* *Elixir/Phoenix application* initialized with Phoenix 1.7.10 +* *ArangoDB integration* via `+arangox+` for multi-model database +(document + graph) +* *GraphQL API* with Absinthe 1.7 (15 queries, 11 mutations) +* *Docker Compose* setup for local development (ArangoDB + PostgreSQL) +* *Podman* alternative configuration documented + +===== Data Models + +* *Claims* schema with Ecto validation and ArangoDB integration +* *Evidence* schema with Zotero metadata (Dublin Core, Schema.org) +* *Relationships* (graph edges) with weighted support/contradiction +* *Navigation Paths* for audience-based exploration (6 audience types) +* *PROMPT Scores* framework (6-dimensional epistemological scoring) + +===== GraphQL API + +* Queries: `+claim+`, `+claims+`, `+searchClaims+`, `+evidence+`, +`+evidenceByZoteroKey+`, `+evidenceList+`, `+searchEvidence+`, +`+evidenceChain+`, `+navigationPath+`, `+navigationPaths+` +* Mutations: `+createClaim+`, `+updateClaim+`, `+deleteClaim+`, +`+createEvidence+`, `+updateEvidence+`, `+importFromZotero+`, +`+createRelationship+`, `+updateRelationship+`, `+deleteRelationship+`, +`+createNavigationPath+`, `+autoGeneratePath+` +* Custom types: `+PromptScores+`, `+Claim+`, `+Evidence+`, +`+Relationship+`, `+NavigationPath+` +* Union types: `+graph_node+` (Claim | Evidence) + +===== Algorithms + +* *Evidence chain traversal* (multi-hop graph traversal, depth-limited) +* *Shortest path* algorithm (ArangoDB native SHORTEST_PATH) +* *Propagated weight calculation* (multiplicative decay along paths) +* *Contradiction detection* (claims with conflicting evidence) +* *Audience-weighted PROMPT scoring* (different priorities per user +type) +* *Auto-generate navigation paths* (ML-free heuristic based on PROMPT +scores) + +===== Test Data + +* *UK Inflation 2023 investigation* seed dataset: +** 7 claims (primary, supporting, counter) with confidence levels +** 10 evidence items (official statistics, academic, think tanks, +interviews) +** 10 relationships (supports, contradicts, contextualizes) +** 3 navigation paths (researcher, policymaker, affected person) + +===== Documentation + +* *README.md* with quick start guide and GraphQL examples +* *ARCHITECTURE.md* (~4,000 words): data model, database design, API +specs, algorithms +* *ROADMAP.md* (~3,500 words): 18-month plan, 3 phases, decision points, +success metrics +* *CLAUDE.md* (~2,500 words): AI assistant context, philosophical core, +dev patterns +* *docs/database-evaluation.md*: ArangoDB vs SurrealDB vs Virtuoso +comparison +* *docs/zotero-integration.md*: Two-way sync design, extension code +templates + +===== Visualization + +* *D3.js graph visualization* (force-directed layout): +** Color-coded by PROMPT scores (red→yellow→green gradient) +** Interactive: drag nodes, zoom, pan, tooltips +** Relationship types: supports (green), contradicts (red), +contextualizes (gray) +** PROMPT score badges on nodes + +===== RSR Compliance (Rhodium Standard Repository) + +* *LICENSE.txt*: Dual license (MIT + Palimpsest v0.8) for emotional +safety +* *SECURITY.md*: Comprehensive security policy, vulnerability +disclosure, GDPR compliance +* *CONTRIBUTING.md*: TPCF Perimeter 3 model, 90-day reversibility, +contribution guidelines +* *CODE_OF_CONDUCT.md*: CCCP (Community-Centric Code of Practice), +emotional safety +* *MAINTAINERS.md*: Governance model, emotional temperature metrics, +decision framework +* *CHANGELOG.md*: This file (SemVer + Keep a Changelog format) + +===== Configuration + +* *mix.exs* with all dependencies (Phoenix, Absinthe, Arangox, Oban) +* **config/*.exs** for dev/test/prod/runtime environments +* *.env.example* with documented environment variables +* *.gitignore* comprehensive Elixir/Phoenix rules + +==== Changed + +* N/A (initial release) + +==== Deprecated + +* N/A + +==== Removed + +* N/A + +==== Fixed + +* N/A + +==== Security + +* *Parameterized AQL queries* to prevent injection attacks +* *Ecto changesets* for input validation +* *GraphQL schema validation* at API boundary +* *No unsafe Elixir patterns* (no `+String.to_existing_atom/1+` on user +input) +* *GDPR compliance* design: anonymizable interview subjects, soft +deletes + +=== Project Metadata + +*Repository*: https://github.com/Hyperpolymath/bofig *Contributors*: +@Hyperpolymath, Claude (AI assistant) *License*: MPL-2.0 *Status*: Phase +1 PoC v1.0.0 complete + +=== Version Numbering + +We use *Semantic Versioning* (SemVer): + +* *MAJOR*: Breaking API changes, architectural rewrites +* *MINOR*: New features, backwards-compatible +* *PATCH*: Bug fixes, documentation, security patches + +*Phase Mapping:* - 1.0.x = Phase 1 PoC (Months 1-6) - 2.0.x = Phase 2 +Platform (Months 7-12) - 3.0.0 = Phase 3 Production (Month 18) + +=== Contribution Credits + +==== 0.1.0 Contributors + +* *@Hyperpolymath*: Project concept, architecture, PhD research +integration +* *Claude (Anthropic)*: Code implementation, documentation, RSR +compliance + +_All contributors are listed in .well-known/humans.txt_ + +=== Upgrade Guide + +==== From: Nothing → 0.1.0 + +*Initial Installation:* + +[arabic] +. Clone repository +. Install dependencies: `+mix deps.get+` +. Start databases: `+docker-compose up -d+` +. Setup ArangoDB: `+iex -S mix+` → +`+EvidenceGraph.ArangoDB.setup_database()+` +. Load seed data: `+mix run priv/repo/seeds.exs+` +. Start server: `+mix phx.server+` + +See README.md for detailed instructions. + +==== Breaking Changes + +* N/A (initial release) + +=== Deprecation Warnings + +* None currently + +=== Roadmap Preview + +*Next Release (1.1.0) - Phase 1 Month 3-4:* - Zotero browser extension +(one-click import) - Two-way sync between Zotero library and Evidence +Graph - Month 3 decision point: Continue or pivot based on NUJ testing + +*Future (2.0.0) - Phase 2:* - Multi-investigation dashboard - Real-time +collaborative editing - IPFS provenance integration - Hetzner Cloud +deployment + +See ROADMAP.adoc for full 18-month plan. + +=== Contact & Support + +* *Issues*: https://github.com/Hyperpolymath/bofig/issues +* *Discussions*: https://github.com/Hyperpolymath/bofig/discussions +* *Security*: security@evidencegraph.org (see SECURITY.md) +* *Governance*: See MAINTAINERS.md + +''''' + +*Changelog Maintenance*: This file is updated with every release. For +unreleased changes, see Git commit history. + +_Format: https://keepachangelog.com/[Keep a Changelog]_ _Versioning: +https://semver.org/[Semantic Versioning]_ _Last Updated: 2026-02-21_ diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 4d8adb8..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,261 +0,0 @@ -# Changelog - -All notable changes to the Evidence Graph project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), -and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -### Planned (Phase 2) -- Zotero browser extension (one-click import) -- Multi-investigation dashboard with cross-referencing -- Real-time collaborative editing via Phoenix PubSub -- Advanced D3.js visualisations (timeline, heatmap, Sankey) -- Role-based access control -- IPFS provenance integration -- Hetzner Cloud deployment - -## [1.0.0] - 2026-02-21 - "Phase 1 PoC" (Evidence Graph v1) - -### Added - -#### LiveView Frontend -- **5 LiveView pages**: Dashboard (investigation list), Investigation detail, Graph visualisation, PROMPT scoring, Navigation paths -- **Core components**: layouts, navigation, responsive design -- **D3.js hooks**: Force-directed graph + radar chart visualisations - -#### User Authentication -- **phx.gen.auth**: Registration, login, settings, magic link confirmation -- **bcrypt**: Password hashing via bcrypt_elixir -- **Swoosh**: Email delivery for magic links -- **Session-based auth**: CSRF protection on all state-changing operations - -#### Zotero REST API -- `POST /api/evidence/import` - Import single evidence from Zotero -- `POST /api/evidence/batch-import` - Batch import multiple items -- `GET /api/evidence/:id/export` - Export evidence to Zotero format -- `GET /api/investigations/:id/sync-status` - Check Zotero sync status - -#### Production Deployment -- **Containerfile**: Multi-stage OCI build (Elixir 1.18/OTP 27 + Debian bookworm) -- **podman-compose.yml**: ArangoDB + PostgreSQL + app with health checks -- **nginx config**: Reverse proxy with WebSocket support -- **systemd service**: Production process management -- **runtime.exs**: Production configuration from environment variables -- **Health endpoint**: `GET /api/health` for container orchestration - -#### Data & Seeds -- Expanded seed data: 7 claims, 30 evidence items, 38 relationships, 6 navigation paths -- Audience-weighted navigation for 6 types (journalist, researcher, policymaker, skeptic, affected person, activist) - -#### NUJ Testing Protocols -- Task script for user testing sessions -- Consent form for participants -- Feedback form with SUS scale -- Decision matrix for Month 3 go/no-go -- 5 AsciiDoc protocol documents in docs/testing/ - -#### Compliance & Trust -- **A2ML v2.1 Trustfile**: Full cyberwar-ready trustfile (bofig.trustfile.a2ml) -- **Contractiles**: Updated Mustfile, Dustfile, Intentfile, Trustfile -- **RSR compliance**: All 12 required files present -- **19 GitHub Actions workflows**: Security, quality, mirroring, enforcement -- **TOPOLOGY.md**: Architecture diagram + completion dashboard - -### Changed -- Bumped Phoenix from 1.7.10 to 1.8.3 -- Bumped Phoenix LiveView to 1.1.19 -- Replaced docker-compose.yml with podman-compose.yml -- Replaced Justfile (legacy Docker references) with comprehensive Podman-based recipes -- Expanded Mustfile from 3 to 6 mandatory checks -- Updated SECURITY.md to reflect implemented authentication -- Updated ROADMAP.adoc with actual 18-month plan - -### Removed -- `docker-compose.yml` (replaced by podman-compose.yml) -- `Podmanfile.md` (superseded by Containerfile) -- `affinescript.json` (AffineScript not used in Elixir project) -- `CHANGELOG.adoc` (duplicate of CHANGELOG.md) -- `CONTRIBUTING.md` (kept CONTRIBUTING.adoc) -- `MAINTAINERS.adoc` (kept MAINTAINERS.md) -- `LICENSE.txt` (kept LICENSE) -- `contractiles/trust/Trustfile.hs` (replaced by A2ML Trustfile) - -### Security -- User authentication implemented (phx.gen.auth) -- All browser routes require authentication -- CSRF protection on all forms -- Parameterized AQL queries (unchanged) -- Ecto changesets for all input validation -- Health endpoint unauthenticated (for container orchestration) -- `mix deps.audit` integrated into CI - -## [0.1.0] - 2025-11-22 - "Foundation" (Phase 1 Month 1) - -### Added - -#### Core Infrastructure -- **Elixir/Phoenix application** initialized with Phoenix 1.7.10 -- **ArangoDB integration** via `arangox` for multi-model database (document + graph) -- **GraphQL API** with Absinthe 1.7 (15 queries, 11 mutations) -- **Docker Compose** setup for local development (ArangoDB + PostgreSQL) -- **Podman** alternative configuration documented - -#### Data Models -- **Claims** schema with Ecto validation and ArangoDB integration -- **Evidence** schema with Zotero metadata (Dublin Core, Schema.org) -- **Relationships** (graph edges) with weighted support/contradiction -- **Navigation Paths** for audience-based exploration (6 audience types) -- **PROMPT Scores** framework (6-dimensional epistemological scoring) - -#### GraphQL API -- Queries: `claim`, `claims`, `searchClaims`, `evidence`, `evidenceByZoteroKey`, `evidenceList`, `searchEvidence`, `evidenceChain`, `navigationPath`, `navigationPaths` -- Mutations: `createClaim`, `updateClaim`, `deleteClaim`, `createEvidence`, `updateEvidence`, `importFromZotero`, `createRelationship`, `updateRelationship`, `deleteRelationship`, `createNavigationPath`, `autoGeneratePath` -- Custom types: `PromptScores`, `Claim`, `Evidence`, `Relationship`, `NavigationPath` -- Union types: `graph_node` (Claim | Evidence) - -#### Algorithms -- **Evidence chain traversal** (multi-hop graph traversal, depth-limited) -- **Shortest path** algorithm (ArangoDB native SHORTEST_PATH) -- **Propagated weight calculation** (multiplicative decay along paths) -- **Contradiction detection** (claims with conflicting evidence) -- **Audience-weighted PROMPT scoring** (different priorities per user type) -- **Auto-generate navigation paths** (ML-free heuristic based on PROMPT scores) - -#### Test Data -- **UK Inflation 2023 investigation** seed dataset: - - 7 claims (primary, supporting, counter) with confidence levels - - 10 evidence items (official statistics, academic, think tanks, interviews) - - 10 relationships (supports, contradicts, contextualizes) - - 3 navigation paths (researcher, policymaker, affected person) - -#### Documentation -- **README.md** with quick start guide and GraphQL examples -- **ARCHITECTURE.md** (~4,000 words): data model, database design, API specs, algorithms -- **ROADMAP.md** (~3,500 words): 18-month plan, 3 phases, decision points, success metrics -- **CLAUDE.md** (~2,500 words): AI assistant context, philosophical core, dev patterns -- **docs/database-evaluation.md**: ArangoDB vs SurrealDB vs Virtuoso comparison -- **docs/zotero-integration.md**: Two-way sync design, extension code templates - -#### Visualization -- **D3.js graph visualization** (force-directed layout): - - Color-coded by PROMPT scores (red→yellow→green gradient) - - Interactive: drag nodes, zoom, pan, tooltips - - Relationship types: supports (green), contradicts (red), contextualizes (gray) - - PROMPT score badges on nodes - -#### RSR Compliance (Rhodium Standard Repository) -- **LICENSE.txt**: Dual license (MIT + Palimpsest v0.8) for emotional safety -- **SECURITY.md**: Comprehensive security policy, vulnerability disclosure, GDPR compliance -- **CONTRIBUTING.md**: TPCF Perimeter 3 model, 90-day reversibility, contribution guidelines -- **CODE_OF_CONDUCT.md**: CCCP (Community-Centric Code of Practice), emotional safety -- **MAINTAINERS.md**: Governance model, emotional temperature metrics, decision framework -- **CHANGELOG.md**: This file (SemVer + Keep a Changelog format) - -#### Configuration -- **mix.exs** with all dependencies (Phoenix, Absinthe, Arangox, Oban) -- **config/*.exs** for dev/test/prod/runtime environments -- **.env.example** with documented environment variables -- **.gitignore** comprehensive Elixir/Phoenix rules - -### Changed -- N/A (initial release) - -### Deprecated -- N/A - -### Removed -- N/A - -### Fixed -- N/A - -### Security -- **Parameterized AQL queries** to prevent injection attacks -- **Ecto changesets** for input validation -- **GraphQL schema validation** at API boundary -- **No unsafe Elixir patterns** (no `String.to_existing_atom/1` on user input) -- **GDPR compliance** design: anonymizable interview subjects, soft deletes - -## Project Metadata - -**Repository**: https://github.com/Hyperpolymath/bofig -**Contributors**: @Hyperpolymath, Claude (AI assistant) -**License**: MPL-2.0 -**Status**: Phase 1 PoC v1.0.0 complete - -## Version Numbering - -We use **Semantic Versioning** (SemVer): - -- **MAJOR**: Breaking API changes, architectural rewrites -- **MINOR**: New features, backwards-compatible -- **PATCH**: Bug fixes, documentation, security patches - -**Phase Mapping:** -- 1.0.x = Phase 1 PoC (Months 1-6) -- 2.0.x = Phase 2 Platform (Months 7-12) -- 3.0.0 = Phase 3 Production (Month 18) - -## Contribution Credits - -### 0.1.0 Contributors - -- **@Hyperpolymath**: Project concept, architecture, PhD research integration -- **Claude (Anthropic)**: Code implementation, documentation, RSR compliance - -*All contributors are listed in .well-known/humans.txt* - -## Upgrade Guide - -### From: Nothing → 0.1.0 - -**Initial Installation:** - -1. Clone repository -2. Install dependencies: `mix deps.get` -3. Start databases: `docker-compose up -d` -4. Setup ArangoDB: `iex -S mix` → `EvidenceGraph.ArangoDB.setup_database()` -5. Load seed data: `mix run priv/repo/seeds.exs` -6. Start server: `mix phx.server` - -See README.md for detailed instructions. - -### Breaking Changes - -- N/A (initial release) - -## Deprecation Warnings - -- None currently - -## Roadmap Preview - -**Next Release (1.1.0) - Phase 1 Month 3-4:** -- Zotero browser extension (one-click import) -- Two-way sync between Zotero library and Evidence Graph -- Month 3 decision point: Continue or pivot based on NUJ testing - -**Future (2.0.0) - Phase 2:** -- Multi-investigation dashboard -- Real-time collaborative editing -- IPFS provenance integration -- Hetzner Cloud deployment - -See ROADMAP.adoc for full 18-month plan. - -## Contact & Support - -- **Issues**: https://github.com/Hyperpolymath/bofig/issues -- **Discussions**: https://github.com/Hyperpolymath/bofig/discussions -- **Security**: security@evidencegraph.org (see SECURITY.md) -- **Governance**: See MAINTAINERS.md - ---- - -**Changelog Maintenance**: This file is updated with every release. For unreleased changes, see Git commit history. - -*Format: [Keep a Changelog](https://keepachangelog.com/)* -*Versioning: [Semantic Versioning](https://semver.org/)* -*Last Updated: 2026-02-21* diff --git a/CODE_OF_CONDUCT.adoc b/CODE_OF_CONDUCT.adoc new file mode 100644 index 0000000..e699067 --- /dev/null +++ b/CODE_OF_CONDUCT.adoc @@ -0,0 +1,251 @@ +== Community-Centric Code of Practice (CCCP) + +=== Our Pledge + +We, as contributors and maintainers of the Evidence Graph project, +pledge to create an emotionally safe, inclusive, and welcoming +environment for all participants, regardless of: + +* Age, body size, disability (visible or invisible) +* Ethnicity, nationality, race +* Gender identity and expression, sexual orientation +* Level of experience, education +* Socioeconomic status, employment status +* Religion or lack thereof +* Political beliefs (within bounds of mutual respect) +* Neurodivers + +ity, mental health status + +*Core Principle:* We prioritize *contributor emotional well-being* as +highly as code quality. + +=== Our Standards + +==== Expected Behavior + +*Be Kind and Respectful:* - Assume good intentions - Use welcoming and +inclusive language - Respect differing viewpoints and experiences - +Accept constructive criticism gracefully - Focus on what is best for the +community + +*Be Empathetic:* - Recognize that people have different backgrounds and +constraints - Respect personal boundaries (time, energy, emotional +capacity) - Understand that volunteering is a gift, not an obligation - +Value emotional labor (documentation, mentorship, community building) + +*Be Collaborative:* - Welcome newcomers warmly - Mentor generously - +Share knowledge without gatekeeping - Celebrate others’ contributions - +Give credit where due + +*Be Professional:* - Keep discussions technical and constructive - +Separate ideas from people - Disagree without being disagreeable - Avoid +personal attacks, name-calling, or ad hominem arguments + +==== Unacceptable Behavior + +*Harassment* includes but is not limited to: - Offensive comments +related to protected characteristics (see Pledge) - Deliberate +intimidation, stalking, or following - Sustained disruption of +discussions - Unwelcome sexual attention or advances - Publishing +others’ private information (doxxing) - Trolling, insulting/derogatory +comments, personal attacks + +*Discrimination:* - Exclusion based on identity or background - +Gatekeeping or elitism - Dismissing lived experiences - Microaggressions +(even if "`unintentional`") + +*Emotional Manipulation:* - Guilt-tripping contributors - Demanding +unpaid labor - Using maintainer status to pressure contributors - +Ignoring "`no`" or pushing boundaries + +*Technical Misconduct:* - Submitting malicious code or security +vulnerabilities - Plagiarizing others’ work - Falsifying test results or +benchmarks - Sabotaging project infrastructure + +=== Enforcement + +==== Reporting + +*How to Report:* + +[arabic] +. *Confidential Email*: conduct@evidencegraph.org (monitored by P1 +maintainers) +. *Private Message*: Contact @Hyperpolymath on GitHub +. *Third-Party*: Report to GitHub +(https://github.com/contact/report-abuse) + +*What to Include:* - Your contact information (optional if you want to +remain anonymous) - Names of people involved (or "`anonymous reporter`" +if you prefer) - Description of the incident - Date, time, and context - +Any supporting evidence (screenshots, links, etc.) - Whether the +incident is ongoing + +*Confidentiality:* - Reports handled with discretion - Reporter identity +protected unless they consent to disclosure - Only essential details +shared with enforcement team + +==== Enforcement Process + +*1. Acknowledgment (24 hours)* - Reporter receives confirmation - +Initial assessment of severity + +*2. Investigation (7 days)* - Gather facts from all parties - Review +evidence (chat logs, PR comments, etc.) - Consult with neutral third +parties if needed + +*3. Decision (14 days total)* - Determine if Code of Conduct was +violated - Decide on appropriate consequences - Inform all parties of +decision + +*4. Appeal (optional, 30 days)* - Affected party may appeal decision - +Independent reviewer examines case - Final decision issued + +==== Consequences + +Depending on severity, violations may result in: + +*Level 1: Warning* - Private written warning - Request for public or +private apology - Required hiatus from project (e.g., 30 days) + +*Level 2: Temporary Ban* - Temporary suspension (30-90 days) - No +participation in project spaces (GitHub, chat, events) - May be required +to complete training or mediation + +*Level 3: Permanent Ban* - Permanent removal from all project spaces - +Contributions may be reverted (see Reversibility policy) - Public +statement issued (to protect community) + +*Level 4: Legal Action* - For severe cases (threats, doxxing, +harassment) - Law enforcement contacted - Project disassociates entirely + +==== Scope + +This Code of Conduct applies to: + +* *GitHub spaces*: Issues, PRs, Discussions, code comments +* *Communication channels*: Email, chat, video calls (if established) +* *Events*: Conferences, meetups, online gatherings +* *Public representation*: Talks, blog posts, social media (when +representing project) + +Violations outside project spaces may be considered if they affect +project safety. + +=== Emotional Safety Framework + +==== Reversibility Protections + +Per *Palimpsest License v0.8*: + +* Contributors may withdraw work within 90 days +* No questions asked, no guilt-tripping +* Contributions reverted or replaced promptly +* See CONTRIBUTING.md for full process + +==== Emotional Temperature Monitoring + +We track and publish metrics to ensure community health: + +[cols=",,,",options="header",] +|=== +|Metric |Current |Target |Status +|Contributor Churn (6-month) |TBD |<20% |📊 Measuring +|Review Turnaround |TBD |<48h |📊 Measuring +|Issue Resolution Time |TBD |<14 days |📊 Measuring +|Governance Transparency |100% |100% |✅ On Track +|=== + +See MAINTAINERS.md for live dashboard. + +==== Conflict Resolution + +*Before escalating to enforcement:* + +[arabic] +. *Direct Communication*: Try to resolve privately (if safe) +. *Mediation*: Request a neutral third party (P2 contributor) +. *Cooling-Off Period*: Take a break, revisit later +. *Documentation*: Write out your perspective (helps clarify) + +*When to escalate immediately:* - Threats, doxxing, or harassment - +Discrimination or hate speech - Repeated boundary violations - Safety +concerns + +=== Maintainer Responsibilities + +==== Leading by Example + +Maintainers must: - Model respectful behavior - Respond to reports +promptly and fairly - Enforce Code of Conduct consistently - Protect +reporter confidentiality - Avoid conflicts of interest (recuse if +needed) + +==== Transparency + +* Enforcement decisions documented in MAINTAINERS.md (anonymized) +* Community informed of pattern trends (not individual cases) +* Annual Code of Conduct review with community input + +==== Self-Care + +Maintainers are also protected: - Can take breaks without guilt - May +delegate enforcement to trusted parties - Have same reversibility rights +as contributors + +=== Community Responsibilities + +*Everyone* can contribute to a healthy community: + +* *Welcome newcomers*: Respond to "`beginner`" questions kindly +* *Assume best intentions*: Miscommunication is common, malice is rare +* *Call in, not out*: Address issues privately first when possible +* *Support each other*: Check in on contributors, celebrate wins +* *Respect maintainer boundaries*: They’re volunteers too + +=== Attribution & Inspiration + +This Code of Conduct is inspired by: + +* https://www.contributor-covenant.org/version/2/1/code_of_conduct/[Contributor +Covenant v2.1] +* https://www.rust-lang.org/policies/code-of-conduct[Rust Code of +Conduct] +* https://www.djangoproject.com/conduct/[Django Code of Conduct] +* https://palimpsest.license/v0.8/[Palimpsest License v0.8] +* CCCP (Community-Centric Code of Practice) principles + +Modified to emphasize *emotional safety* and *reversibility* as core +values. + +=== Contact + +* *Code of Conduct Team*: conduct@evidencegraph.org +* *Lead Maintainer*: @Hyperpolymath (GitHub) +* *Anonymous Reporting*: (third-party service TBD for Phase 2) + +=== Amendments + +This Code of Conduct may be updated to: - Clarify ambiguities - Address +new scenarios - Incorporate community feedback + +*Process:* 1. Proposed changes published in GitHub Discussion 2. +Community comment period (30 days) 3. Maintainer decision with reasoning +4. Changelog entry in MAINTAINERS.md + +*Last Amended:* 2025-11-22 (initial version) + +=== License + +This Code of Conduct is licensed under *CC BY 4.0* (Creative Commons +Attribution). You may adapt it for your own projects with attribution to +Evidence Graph. + +''''' + +*Remember:* This community is built on trust, empathy, and shared +commitment to investigative journalism. Let’s make it a place where +everyone can contribute their best work without fear or pressure. + +_Last Updated: 2025-11-22_ _Version: 1.0 (CCCP + Palimpsest v0.8)_ diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index 7425db5..0000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,272 +0,0 @@ -# Community-Centric Code of Practice (CCCP) - -## Our Pledge - -We, as contributors and maintainers of the Evidence Graph project, pledge to create an emotionally safe, inclusive, and welcoming environment for all participants, regardless of: - -- Age, body size, disability (visible or invisible) -- Ethnicity, nationality, race -- Gender identity and expression, sexual orientation -- Level of experience, education -- Socioeconomic status, employment status -- Religion or lack thereof -- Political beliefs (within bounds of mutual respect) -- Neurodivers - -ity, mental health status - -**Core Principle:** We prioritize **contributor emotional well-being** as highly as code quality. - -## Our Standards - -### Expected Behavior - -**Be Kind and Respectful:** -- Assume good intentions -- Use welcoming and inclusive language -- Respect differing viewpoints and experiences -- Accept constructive criticism gracefully -- Focus on what is best for the community - -**Be Empathetic:** -- Recognize that people have different backgrounds and constraints -- Respect personal boundaries (time, energy, emotional capacity) -- Understand that volunteering is a gift, not an obligation -- Value emotional labor (documentation, mentorship, community building) - -**Be Collaborative:** -- Welcome newcomers warmly -- Mentor generously -- Share knowledge without gatekeeping -- Celebrate others' contributions -- Give credit where due - -**Be Professional:** -- Keep discussions technical and constructive -- Separate ideas from people -- Disagree without being disagreeable -- Avoid personal attacks, name-calling, or ad hominem arguments - -### Unacceptable Behavior - -**Harassment** includes but is not limited to: -- Offensive comments related to protected characteristics (see Pledge) -- Deliberate intimidation, stalking, or following -- Sustained disruption of discussions -- Unwelcome sexual attention or advances -- Publishing others' private information (doxxing) -- Trolling, insulting/derogatory comments, personal attacks - -**Discrimination:** -- Exclusion based on identity or background -- Gatekeeping or elitism -- Dismissing lived experiences -- Microaggressions (even if "unintentional") - -**Emotional Manipulation:** -- Guilt-tripping contributors -- Demanding unpaid labor -- Using maintainer status to pressure contributors -- Ignoring "no" or pushing boundaries - -**Technical Misconduct:** -- Submitting malicious code or security vulnerabilities -- Plagiarizing others' work -- Falsifying test results or benchmarks -- Sabotaging project infrastructure - -## Enforcement - -### Reporting - -**How to Report:** - -1. **Confidential Email**: conduct@evidencegraph.org (monitored by P1 maintainers) -2. **Private Message**: Contact @Hyperpolymath on GitHub -3. **Third-Party**: Report to GitHub (https://github.com/contact/report-abuse) - -**What to Include:** -- Your contact information (optional if you want to remain anonymous) -- Names of people involved (or "anonymous reporter" if you prefer) -- Description of the incident -- Date, time, and context -- Any supporting evidence (screenshots, links, etc.) -- Whether the incident is ongoing - -**Confidentiality:** -- Reports handled with discretion -- Reporter identity protected unless they consent to disclosure -- Only essential details shared with enforcement team - -### Enforcement Process - -**1. Acknowledgment (24 hours)** -- Reporter receives confirmation -- Initial assessment of severity - -**2. Investigation (7 days)** -- Gather facts from all parties -- Review evidence (chat logs, PR comments, etc.) -- Consult with neutral third parties if needed - -**3. Decision (14 days total)** -- Determine if Code of Conduct was violated -- Decide on appropriate consequences -- Inform all parties of decision - -**4. Appeal (optional, 30 days)** -- Affected party may appeal decision -- Independent reviewer examines case -- Final decision issued - -### Consequences - -Depending on severity, violations may result in: - -**Level 1: Warning** -- Private written warning -- Request for public or private apology -- Required hiatus from project (e.g., 30 days) - -**Level 2: Temporary Ban** -- Temporary suspension (30-90 days) -- No participation in project spaces (GitHub, chat, events) -- May be required to complete training or mediation - -**Level 3: Permanent Ban** -- Permanent removal from all project spaces -- Contributions may be reverted (see Reversibility policy) -- Public statement issued (to protect community) - -**Level 4: Legal Action** -- For severe cases (threats, doxxing, harassment) -- Law enforcement contacted -- Project disassociates entirely - -### Scope - -This Code of Conduct applies to: - -- **GitHub spaces**: Issues, PRs, Discussions, code comments -- **Communication channels**: Email, chat, video calls (if established) -- **Events**: Conferences, meetups, online gatherings -- **Public representation**: Talks, blog posts, social media (when representing project) - -Violations outside project spaces may be considered if they affect project safety. - -## Emotional Safety Framework - -### Reversibility Protections - -Per **Palimpsest License v0.8**: - -- Contributors may withdraw work within 90 days -- No questions asked, no guilt-tripping -- Contributions reverted or replaced promptly -- See CONTRIBUTING.md for full process - -### Emotional Temperature Monitoring - -We track and publish metrics to ensure community health: - -| Metric | Current | Target | Status | -|--------|---------|--------|--------| -| Contributor Churn (6-month) | TBD | <20% | 📊 Measuring | -| Review Turnaround | TBD | <48h | 📊 Measuring | -| Issue Resolution Time | TBD | <14 days | 📊 Measuring | -| Governance Transparency | 100% | 100% | ✅ On Track | - -See MAINTAINERS.md for live dashboard. - -### Conflict Resolution - -**Before escalating to enforcement:** - -1. **Direct Communication**: Try to resolve privately (if safe) -2. **Mediation**: Request a neutral third party (P2 contributor) -3. **Cooling-Off Period**: Take a break, revisit later -4. **Documentation**: Write out your perspective (helps clarify) - -**When to escalate immediately:** -- Threats, doxxing, or harassment -- Discrimination or hate speech -- Repeated boundary violations -- Safety concerns - -## Maintainer Responsibilities - -### Leading by Example - -Maintainers must: -- Model respectful behavior -- Respond to reports promptly and fairly -- Enforce Code of Conduct consistently -- Protect reporter confidentiality -- Avoid conflicts of interest (recuse if needed) - -### Transparency - -- Enforcement decisions documented in MAINTAINERS.md (anonymized) -- Community informed of pattern trends (not individual cases) -- Annual Code of Conduct review with community input - -### Self-Care - -Maintainers are also protected: -- Can take breaks without guilt -- May delegate enforcement to trusted parties -- Have same reversibility rights as contributors - -## Community Responsibilities - -**Everyone** can contribute to a healthy community: - -- **Welcome newcomers**: Respond to "beginner" questions kindly -- **Assume best intentions**: Miscommunication is common, malice is rare -- **Call in, not out**: Address issues privately first when possible -- **Support each other**: Check in on contributors, celebrate wins -- **Respect maintainer boundaries**: They're volunteers too - -## Attribution & Inspiration - -This Code of Conduct is inspired by: - -- [Contributor Covenant v2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/) -- [Rust Code of Conduct](https://www.rust-lang.org/policies/code-of-conduct) -- [Django Code of Conduct](https://www.djangoproject.com/conduct/) -- [Palimpsest License v0.8](https://palimpsest.license/v0.8/) -- CCCP (Community-Centric Code of Practice) principles - -Modified to emphasize **emotional safety** and **reversibility** as core values. - -## Contact - -- **Code of Conduct Team**: conduct@evidencegraph.org -- **Lead Maintainer**: @Hyperpolymath (GitHub) -- **Anonymous Reporting**: (third-party service TBD for Phase 2) - -## Amendments - -This Code of Conduct may be updated to: -- Clarify ambiguities -- Address new scenarios -- Incorporate community feedback - -**Process:** -1. Proposed changes published in GitHub Discussion -2. Community comment period (30 days) -3. Maintainer decision with reasoning -4. Changelog entry in MAINTAINERS.md - -**Last Amended:** 2025-11-22 (initial version) - -## License - -This Code of Conduct is licensed under **CC BY 4.0** (Creative Commons Attribution). You may adapt it for your own projects with attribution to Evidence Graph. - ---- - -**Remember:** This community is built on trust, empathy, and shared commitment to investigative journalism. Let's make it a place where everyone can contribute their best work without fear or pressure. - -*Last Updated: 2025-11-22* -*Version: 1.0 (CCCP + Palimpsest v0.8)* diff --git a/CONTRIBUTING.adoc b/CONTRIBUTING.adoc index eb045d6..dd089ae 100644 --- a/CONTRIBUTING.adoc +++ b/CONTRIBUTING.adoc @@ -1,20 +1,71 @@ -// SPDX-License-Identifier: CC-BY-SA-4.0 -= Contributing Guide +== Contributing -== Getting Started +Thank you for your interest in contributing! We follow a "`Dual-Track`" +architecture where human-readable documentation lives in the root and +machine-readable policies live in `+.machine_readable/+`. -1. Fork the repository -2. Create a feature branch from `main` -3. Sign off commits (`git commit -s`) -4. Submit a pull request +=== How to Contribute -== Commit Guidelines +We welcome contributions in many forms: -* Conventional commits: `type(scope): description` -* Sign all commits (DCO required) -* Atomic, focused commits +* *Code:* Improving the core stack or extensions +* *Documentation:* Enhancing docs or AI manifests +* *Testing:* Adding property-based tests or formal proofs +* *Bug reports:* Filing clear, reproducible issues -== License +=== Getting Started -Contributions licensed under project license. +[arabic] +. *Read the AI Manifest:* Start with `+0-AI-MANIFEST.a2ml+` (if present) +to understand the repository structure. +. *Environment:* Use `+guix develop+` or `+direnv allow+` to set up your +tools. +. *Task Runner:* Use `+just+` to see available commands +(`+just --list+`). +=== Development Workflow + +==== Branch Naming + +.... +docs/short-description # Documentation +test/what-added # Test additions +feat/short-description # New features +fix/issue-number-description # Bug fixes +refactor/what-changed # Code improvements +security/what-fixed # Security fixes +.... + +==== Commit Messages + +We follow https://www.conventionalcommits.org/[Conventional Commits]: + +.... +(): + +[optional body] + +[optional footer] +.... + +Types: `+feat+`, `+fix+`, `+docs+`, `+test+`, `+refactor+`, `+ci+`, +`+chore+`, `+security+` + +=== Reporting Bugs + +Before reporting: 1. Search existing issues 2. Check if it’s already +fixed in `+main+` + +When reporting, include: - Clear, descriptive title - Environment +details (OS, versions, toolchain) - Steps to reproduce - Expected vs +actual behaviour + +=== Code of Conduct + +All contributors are expected to adhere to our +link:CODE_OF_CONDUCT.md[Code of Conduct]. + +=== License + +By contributing, you agree that your contributions will be licensed +under the same license as the project (see LICENSE). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 90e87dc..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,66 +0,0 @@ - -# Contributing - -Thank you for your interest in contributing! We follow a "Dual-Track" architecture where human-readable documentation lives in the root and machine-readable policies live in `.machine_readable/`. - -## How to Contribute - -We welcome contributions in many forms: - -- **Code:** Improving the core stack or extensions -- **Documentation:** Enhancing docs or AI manifests -- **Testing:** Adding property-based tests or formal proofs -- **Bug reports:** Filing clear, reproducible issues - -## Getting Started - -1. **Read the AI Manifest:** Start with `0-AI-MANIFEST.a2ml` (if present) to understand the repository structure. -2. **Environment:** Use `guix develop` or `direnv allow` to set up your tools. -3. **Task Runner:** Use `just` to see available commands (`just --list`). - -## Development Workflow - -### Branch Naming - -``` -docs/short-description # Documentation -test/what-added # Test additions -feat/short-description # New features -fix/issue-number-description # Bug fixes -refactor/what-changed # Code improvements -security/what-fixed # Security fixes -``` - -### Commit Messages - -We follow [Conventional Commits](https://www.conventionalcommits.org/): - -``` -(): - -[optional body] - -[optional footer] -``` - -Types: `feat`, `fix`, `docs`, `test`, `refactor`, `ci`, `chore`, `security` - -## Reporting Bugs - -Before reporting: -1. Search existing issues -2. Check if it's already fixed in `main` - -When reporting, include: -- Clear, descriptive title -- Environment details (OS, versions, toolchain) -- Steps to reproduce -- Expected vs actual behaviour - -## Code of Conduct - -All contributors are expected to adhere to our [Code of Conduct](CODE_OF_CONDUCT.md). - -## License - -By contributing, you agree that your contributions will be licensed under the same license as the project (see [LICENSE](LICENSE)). diff --git a/GOVERNANCE.adoc b/GOVERNANCE.adoc new file mode 100644 index 0000000..9b836fb --- /dev/null +++ b/GOVERNANCE.adoc @@ -0,0 +1,60 @@ +== Governance + +=== Overview + +This project is governed by the following principles and structures to +ensure transparent, inclusive, and effective decision-making. + +=== Roles and Responsibilities + +==== Maintainers + +Maintainers are responsible for: - Reviewing and merging pull requests - +Managing releases and versioning - Ensuring code quality and standards - +Triaging issues and bug reports - Community engagement and support + +==== Contributors + +Contributors are expected to: - Follow the code of conduct - Submit +well-documented pull requests - Write tests for new functionality - +Maintain existing tests - Update documentation as needed + +=== Decision Making + +==== Minor Changes + +* Can be made by any maintainer +* Include bug fixes, documentation updates, dependency updates + +==== Major Changes + +* Require discussion in issues or pull requests +* Include new features, architectural changes, API changes +* Need approval from at least 2 maintainers + +==== Breaking Changes + +* Require RFC (Request for Comments) process +* Need approval from majority of maintainers +* Must include migration guide + +=== Code of Conduct + +All participants are expected to follow our Code of Conduct. Violations +can be reported to the maintainers. + +=== Communication + +* *Issues*: For bug reports and feature requests +* *Discussions*: For questions and general discussion +* *Pull Requests*: For code contributions + +=== Licensing + +All contributions are made under the terms of the repository’s LICENSE +file. By submitting a pull request, you agree to license your +contributions accordingly. + +''''' + +_Last updated: 2026-07-18_ diff --git a/GOVERNANCE.md b/GOVERNANCE.md deleted file mode 100644 index e27364c..0000000 --- a/GOVERNANCE.md +++ /dev/null @@ -1,60 +0,0 @@ -# Governance - -## Overview - -This project is governed by the following principles and structures to ensure transparent, inclusive, and effective decision-making. - -## Roles and Responsibilities - -### Maintainers - -Maintainers are responsible for: -- Reviewing and merging pull requests -- Managing releases and versioning -- Ensuring code quality and standards -- Triaging issues and bug reports -- Community engagement and support - -### Contributors - -Contributors are expected to: -- Follow the code of conduct -- Submit well-documented pull requests -- Write tests for new functionality -- Maintain existing tests -- Update documentation as needed - -## Decision Making - -### Minor Changes -- Can be made by any maintainer -- Include bug fixes, documentation updates, dependency updates - -### Major Changes -- Require discussion in issues or pull requests -- Include new features, architectural changes, API changes -- Need approval from at least 2 maintainers - -### Breaking Changes -- Require RFC (Request for Comments) process -- Need approval from majority of maintainers -- Must include migration guide - -## Code of Conduct - -All participants are expected to follow our Code of Conduct. Violations can be reported to the maintainers. - -## Communication - -- **Issues**: For bug reports and feature requests -- **Discussions**: For questions and general discussion -- **Pull Requests**: For code contributions - -## Licensing - -All contributions are made under the terms of the repository's LICENSE file. -By submitting a pull request, you agree to license your contributions accordingly. - ---- - -*Last updated: 2026-07-18* diff --git a/MAINTAINERS.adoc b/MAINTAINERS.adoc new file mode 100644 index 0000000..6ce758a --- /dev/null +++ b/MAINTAINERS.adoc @@ -0,0 +1,353 @@ +== Project Maintainers & Governance + +=== Current Maintainers + +==== Perimeter 1 (Core Team) + +[width="100%",cols="19%,22%,17%,22%,20%",options="header",] +|=== +|Name |GitHub |Role |Joined |Areas +|@Hyperpolymath |https://github.com/Hyperpolymath[@Hyperpolymath] +|Project Lead |2025-11-22 |Architecture, PhD Research, NUJ Network +|=== + +==== Perimeter 2 (Contributors) + +_No P2 contributors yet. Earn P2 status after 3+ merged PRs (see +CONTRIBUTING.md)_ + +==== Perimeter 3 (Community) + +*Everyone else!* Open contribution welcome. See CONTRIBUTING.md for how +to participate. + +=== Governance Model + +==== Decision-Making Framework + +*Decision Tiers:* + +[width="100%",cols="16%,17%,46%,21%",options="header",] +|=== +|Tier |Scope |Decision Maker(s) |Process +|*1. Critical* |License changes, CoC amendments, infrastructure access +|P1 consensus + community input (30-day RFC) |RFC → Discussion → Vote + +|*2. Major* |Architecture changes, breaking API changes, dependency +changes |P1 decision + P2 consultation |Design doc → Review → Approval + +|*3. Minor* |New features, refactors, documentation |P2 approval (2 +reviewers) |PR → Review → Merge + +|*4. Trivial* |Bug fixes, typos, style |Single P2 reviewer |PR → Quick +review → Merge +|=== + +==== RFC (Request for Comments) Process + +For Tier 1 decisions: + +[arabic] +. *Draft RFC*: Maintainer writes proposal in `+docs/rfcs/NNNN-title.md+` +. *Publication*: RFC announced in GitHub Discussion +. *Comment Period*: Minimum 30 days for community feedback +. *Revision*: Author addresses feedback, updates RFC +. *Vote*: P1 maintainers vote (simple majority, or consensus for +critical changes) +. *Implementation*: Accepted RFCs move to ROADMAP.md +. *Rejection*: Rejected RFCs remain as historical record with reasoning + +*Example RFCs:* - RFC 0001: Switching from ArangoDB to +PostgreSQL+AgensGraph - RFC 0002: Adding user authentication (Phase 2) - +RFC 0003: IPFS integration design - RFC 0004: Multi-tenancy support + +==== Consensus vs. Voting + +*Prefer consensus:* - Discuss until all parties are satisfied (or "`can +live with`" decision) - Explicitly ask for objections - Document +dissenting opinions + +*Use voting when:* - Consensus cannot be reached after good-faith effort +- Time-sensitive decisions (security patches) - Clear binary choice +(yes/no) + +*Voting Rules:* - P1 maintainers: 1 vote each - Simple majority (>50%) +for most decisions - Supermajority (≥67%) for critical decisions +(license, CoC, governance changes) - Tie-breaker: Project Lead +(@Hyperpolymath initially) + +=== Responsibilities + +==== Perimeter 1 (Core Team) + +*Must:* - Respond to security issues within 24 hours - Review PRs within +48 hours (or delegate) - Publish monthly progress update - Maintain +emotional temperature metrics (see below) - Uphold Code of Conduct +fairly - Make decisions transparently (document in GitHub) + +*May:* - Merge own PRs for trivial changes (docs, typos) - Delegate +reviews to P2 contributors - Take breaks (no guilt - announce hiatus in +advance if >2 weeks) + +*Must Not:* - Make unilateral decisions on Tier 1 matters - Push +directly to `+main+` without PR (except emergencies) - Abuse position +for personal gain - Violate contributor trust or confidentiality + +==== Perimeter 2 (Contributors) + +*Must:* - Review assigned PRs within 72 hours - Mentor newcomers (answer +questions, guide contributions) - Participate in monthly contributor +calls (when established) - Follow Code of Conduct + +*May:* - Merge PRs after review by 1 other P2/P1 maintainer - Triage +issues (labeling, requesting more info) - Close stale PRs/issues (after +30 days inactivity) + +*Benefits:* - Listed in MAINTAINERS.md and .well-known/humans.txt - +Invitation to private contributor chat (Phase 2) - Early access to +roadmap discussions - Potential co-authorship on academic papers (for +research contributors) + +==== Perimeter 3 (Community) + +*Expectations:* - Follow Code of Conduct - Respect maintainer time +(they’re volunteers) - Search existing issues before opening duplicates +- Provide reproduction steps for bugs + +*No Obligations:* - No pressure to respond quickly (or at all) - Can +withdraw contributions within 90 days - Can step away anytime without +explanation + +=== Emotional Temperature Dashboard + +Per *Palimpsest License v0.8*, we publish metrics on contributor +well-being: + +==== Current Metrics (Phase 1) + +*Data Collection Period*: 2025-11-22 onwards (initial baseline) + +[width="100%",cols="20%,31%,17%,17%,15%",options="header",] +|=== +|Metric |Current Value |Target |Status |Trend +|*Contributor Churn* (% leaving within 6 months) |0% (N=1, new project) +|<20% |✅ Baseline |- + +|*Review Turnaround* (hours to first review) |N/A (no PRs yet) |<48h |📊 +TBD |- + +|*PR Merge Time* (days from open to merge) |N/A |<7 days |📊 TBD |- + +|*Issue Response Time* (hours to first response) |N/A |<24h |📊 TBD |- + +|*Governance Transparency* (% decisions documented) |100% |100% |✅ On +Track |▲ Improving + +|*Code of Conduct Violations* (count per quarter) |0 |0 |✅ Excellent |- + +|*Reversibility Requests* (contributions withdrawn) |0 |N/A |ℹ️ Info +Only |- + +|*Maintainer Burnout Risk* (subjective self-report, 1-10) |3 (low) |<5 +|✅ Healthy |- +|=== + +*Interpretation:* - ✅ Green: Meeting or exceeding target - ⚠️ Yellow: +Approaching threshold, needs attention - 🔴 Red: Below target, immediate +action required - 📊 TBD: Insufficient data yet - ℹ️ Info Only: +Informational metric, no target + +==== How Metrics Are Calculated + +*Contributor Churn:* - Tracked via GitHub API (contributor activity) - +Counts contributors with ≥1 merged PR who have no activity in last 6 +months - Excludes one-time contributors (by design) + +*Review/Merge Times:* - Automated via GitHub Actions (webhook on PR +events) - Median time calculated weekly - Outliers (>30 days) +investigated for patterns + +*Governance Transparency:* - Manual audit quarterly - Checks: Are all +decisions in GitHub Issues/Discussions? Are RFCs published? - Aiming for +100% (no backroom decisions) + +*Maintainer Burnout:* - Self-reported monthly by P1/P2 maintainers - +Scale: 1 (energized) to 10 (burned out) - Triggers discussion if any +maintainer reports >7 + +==== Intervention Strategies + +If metrics decline: + +[arabic] +. *Slow Review Times?* +* Recruit more P2 reviewers +* Reduce PR scope requirements +* Pause new feature PRs temporarily +. *High Churn?* +* Exit interviews with departing contributors +* Address common pain points (docs, onboarding) +* Review Code of Conduct enforcement +. *Maintainer Burnout?* +* Rotate responsibilities +* Recruit co-maintainers +* Reduce project scope (pause Phase 2 items) + +=== Communication Channels + +==== Public + +* *GitHub Issues*: Bug reports, feature requests +* *GitHub Discussions*: General questions, ideas, RFCs +* *Documentation*: README, ARCHITECTURE, ROADMAP + +==== Semi-Public (P2+) + +_Phase 2:_ - *Matrix/Discord*: Real-time chat for contributors - +*Monthly Calls*: Video sync (recorded, transcripts published) + +==== Private (P1 only) + +* *Email*: security@evidencegraph.org (security issues) +* *Email*: conduct@evidencegraph.org (Code of Conduct reports) +* *Signal/Matrix*: Encrypted chat for sensitive governance + +=== Conflict Resolution + +==== Between Contributors + +[arabic] +. *Direct Dialogue*: Encouraged to resolve privately (if safe) +. *Mediation*: P2 contributor acts as neutral facilitator +. *Escalation*: P1 maintainer makes decision if needed + +==== With Maintainers + +[arabic] +. *Appeal*: Contact different P1 maintainer or governance email +. *Third-Party*: Suggest external mediator (community ombudsperson, +future) +. *Fork*: If irreconcilable, forking is a legitimate option (we’ll still +credit your work!) + +==== Maintainer Accountability + +* P1 maintainers are also bound by Code of Conduct +* Violations handled by other P1 maintainers or external party +* Serious violations may result in maintainer removal (community vote) + +=== Succession Planning + +==== Adding Maintainers + +*Perimeter 2 Promotion:* - After 3+ merged PRs (substantive, not +trivial) - Nomination by P1 maintainer - Consensus among existing P2 +contributors + +*Perimeter 1 Promotion:* - After 6+ months as P2 contributor - +Demonstrated technical skill, community leadership, CoC adherence - +Unanimous P1 consensus - Explicit acceptance by nominee (no pressure) + +==== Removing Maintainers + +*Voluntary:* - Announce intention to step down - Transfer +responsibilities (reviews, projects) - Moved to "`Emeritus`" status +(listed as alumni) + +*Involuntary:* - Only for Code of Conduct violations or prolonged +inactivity (>1 year) - P1 consensus decision - Graceful transition +period (30 days) + +==== Emeritus Maintainers + +_Future section for retired maintainers who shaped the project._ + +=== Project Lead Succession + +*Current Lead*: @Hyperpolymath (until PhD completion, estimated Month +18) + +*Succession Plan (Phase 3):* 1. Identify 2-3 potential successors from +P2 contributors 2. Gradual responsibility transfer (Month 15-17) 3. +Community input period (30 days) 4. New lead appointed via P1 consensus +5. Previous lead remains as P1 advisor (unless stepping away entirely) + +=== Legal & Financial Governance + +==== Ownership + +* *Code*: Dual-licensed MIT + Palimpsest v0.8, contributors retain +copyright +* *Trademark*: "`Evidence Graph`" name (if registered) owned by project, +not individual +* *Domain*: evidencegraph.org (if acquired) held in trust for project + +==== Funding (Phase 2+) + +*If grants/donations received:* - Transparent accounting (quarterly +reports) - Funds used for: Hosting, conferences, contributor stipends +(if budget allows) - Major expenditures (>€1,000) require P1 consensus - +Financial records public (or summary if privacy needed) + +*Potential Sponsors:* - Mozilla MOSS, Knight Foundation, EU Horizon - +NUJ (National Union of Journalists) partnership - Academic institution +grants + +*No Ads, No Data Selling:* - This project will never monetize user data +- No advertising or tracking beyond essential analytics - Privacy-first, +investigative journalism-friendly + +=== Acknowledgments + +==== Recognition + +Contributors recognized in multiple places: + +[arabic] +. *.well-known/humans.txt*: Public roll of honor +. *CHANGELOG.md*: Specific contributions noted +. *Academic Papers*: Co-authorship for research contributors (with +consent) +. *Conference Talks*: Credit to implementers of featured work + +==== Awards & Grants + +If the project wins awards or receives grants, credit shared: + +* Maintainers acknowledged publicly +* Financial awards distributed to active contributors (if applicable) +* Trophy/plaque displayed in README + +=== Governance History + +==== Major Decisions + +[width="100%",cols="19%,29%,26%,26%",options="header",] +|=== +|Date |Decision |Process |Outcome +|2025-11-22 |Project initiated, TPCF Perimeter 3 chosen |Initial design +|Approved by @Hyperpolymath + +|TBD |(Future decisions logged here) | | +|=== + +==== RFC Archive + +_RFCs will be stored in `+docs/rfcs/+` with status +(Accepted/Rejected/Superseded)_ + +=== Contact + +* *Governance Questions*: governance@evidencegraph.org (or open GitHub +Discussion) +* *Lead Maintainer*: @Hyperpolymath +* *Code of Conduct*: conduct@evidencegraph.org + +''''' + +*Last Updated*: 2025-11-22 *Governance Model*: TPCF Perimeter 3 +(Community Sandbox) + Palimpsest v0.8 *Emotional Temperature*: ✅ +Healthy (Phase 1 baseline) + +_This governance model is a living document. Suggest improvements via +GitHub Discussion or PR._ diff --git a/MAINTAINERS.md b/MAINTAINERS.md deleted file mode 100644 index 57995f6..0000000 --- a/MAINTAINERS.md +++ /dev/null @@ -1,342 +0,0 @@ -# Project Maintainers & Governance - -## Current Maintainers - -### Perimeter 1 (Core Team) - -| Name | GitHub | Role | Joined | Areas | -|------|--------|------|--------|-------| -| @Hyperpolymath | [@Hyperpolymath](https://github.com/Hyperpolymath) | Project Lead | 2025-11-22 | Architecture, PhD Research, NUJ Network | - -### Perimeter 2 (Contributors) - -*No P2 contributors yet. Earn P2 status after 3+ merged PRs (see CONTRIBUTING.md)* - -### Perimeter 3 (Community) - -**Everyone else!** Open contribution welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to participate. - -## Governance Model - -### Decision-Making Framework - -**Decision Tiers:** - -| Tier | Scope | Decision Maker(s) | Process | -|------|-------|-------------------|---------| -| **1. Critical** | License changes, CoC amendments, infrastructure access | P1 consensus + community input (30-day RFC) | RFC → Discussion → Vote | -| **2. Major** | Architecture changes, breaking API changes, dependency changes | P1 decision + P2 consultation | Design doc → Review → Approval | -| **3. Minor** | New features, refactors, documentation | P2 approval (2 reviewers) | PR → Review → Merge | -| **4. Trivial** | Bug fixes, typos, style | Single P2 reviewer | PR → Quick review → Merge | - -### RFC (Request for Comments) Process - -For Tier 1 decisions: - -1. **Draft RFC**: Maintainer writes proposal in `docs/rfcs/NNNN-title.md` -2. **Publication**: RFC announced in GitHub Discussion -3. **Comment Period**: Minimum 30 days for community feedback -4. **Revision**: Author addresses feedback, updates RFC -5. **Vote**: P1 maintainers vote (simple majority, or consensus for critical changes) -6. **Implementation**: Accepted RFCs move to ROADMAP.md -7. **Rejection**: Rejected RFCs remain as historical record with reasoning - -**Example RFCs:** -- RFC 0001: Switching from ArangoDB to PostgreSQL+AgensGraph -- RFC 0002: Adding user authentication (Phase 2) -- RFC 0003: IPFS integration design -- RFC 0004: Multi-tenancy support - -### Consensus vs. Voting - -**Prefer consensus:** -- Discuss until all parties are satisfied (or "can live with" decision) -- Explicitly ask for objections -- Document dissenting opinions - -**Use voting when:** -- Consensus cannot be reached after good-faith effort -- Time-sensitive decisions (security patches) -- Clear binary choice (yes/no) - -**Voting Rules:** -- P1 maintainers: 1 vote each -- Simple majority (>50%) for most decisions -- Supermajority (≥67%) for critical decisions (license, CoC, governance changes) -- Tie-breaker: Project Lead (@Hyperpolymath initially) - -## Responsibilities - -### Perimeter 1 (Core Team) - -**Must:** -- Respond to security issues within 24 hours -- Review PRs within 48 hours (or delegate) -- Publish monthly progress update -- Maintain emotional temperature metrics (see below) -- Uphold Code of Conduct fairly -- Make decisions transparently (document in GitHub) - -**May:** -- Merge own PRs for trivial changes (docs, typos) -- Delegate reviews to P2 contributors -- Take breaks (no guilt - announce hiatus in advance if >2 weeks) - -**Must Not:** -- Make unilateral decisions on Tier 1 matters -- Push directly to `main` without PR (except emergencies) -- Abuse position for personal gain -- Violate contributor trust or confidentiality - -### Perimeter 2 (Contributors) - -**Must:** -- Review assigned PRs within 72 hours -- Mentor newcomers (answer questions, guide contributions) -- Participate in monthly contributor calls (when established) -- Follow Code of Conduct - -**May:** -- Merge PRs after review by 1 other P2/P1 maintainer -- Triage issues (labeling, requesting more info) -- Close stale PRs/issues (after 30 days inactivity) - -**Benefits:** -- Listed in MAINTAINERS.md and .well-known/humans.txt -- Invitation to private contributor chat (Phase 2) -- Early access to roadmap discussions -- Potential co-authorship on academic papers (for research contributors) - -### Perimeter 3 (Community) - -**Expectations:** -- Follow Code of Conduct -- Respect maintainer time (they're volunteers) -- Search existing issues before opening duplicates -- Provide reproduction steps for bugs - -**No Obligations:** -- No pressure to respond quickly (or at all) -- Can withdraw contributions within 90 days -- Can step away anytime without explanation - -## Emotional Temperature Dashboard - -Per **Palimpsest License v0.8**, we publish metrics on contributor well-being: - -### Current Metrics (Phase 1) - -**Data Collection Period**: 2025-11-22 onwards (initial baseline) - -| Metric | Current Value | Target | Status | Trend | -|--------|--------------|--------|--------|-------| -| **Contributor Churn** (% leaving within 6 months) | 0% (N=1, new project) | <20% | ✅ Baseline | - | -| **Review Turnaround** (hours to first review) | N/A (no PRs yet) | <48h | 📊 TBD | - | -| **PR Merge Time** (days from open to merge) | N/A | <7 days | 📊 TBD | - | -| **Issue Response Time** (hours to first response) | N/A | <24h | 📊 TBD | - | -| **Governance Transparency** (% decisions documented) | 100% | 100% | ✅ On Track | ▲ Improving | -| **Code of Conduct Violations** (count per quarter) | 0 | 0 | ✅ Excellent | - | -| **Reversibility Requests** (contributions withdrawn) | 0 | N/A | ℹ️ Info Only | - | -| **Maintainer Burnout Risk** (subjective self-report, 1-10) | 3 (low) | <5 | ✅ Healthy | - | - -**Interpretation:** -- ✅ Green: Meeting or exceeding target -- ⚠️ Yellow: Approaching threshold, needs attention -- 🔴 Red: Below target, immediate action required -- 📊 TBD: Insufficient data yet -- ℹ️ Info Only: Informational metric, no target - -### How Metrics Are Calculated - -**Contributor Churn:** -- Tracked via GitHub API (contributor activity) -- Counts contributors with ≥1 merged PR who have no activity in last 6 months -- Excludes one-time contributors (by design) - -**Review/Merge Times:** -- Automated via GitHub Actions (webhook on PR events) -- Median time calculated weekly -- Outliers (>30 days) investigated for patterns - -**Governance Transparency:** -- Manual audit quarterly -- Checks: Are all decisions in GitHub Issues/Discussions? Are RFCs published? -- Aiming for 100% (no backroom decisions) - -**Maintainer Burnout:** -- Self-reported monthly by P1/P2 maintainers -- Scale: 1 (energized) to 10 (burned out) -- Triggers discussion if any maintainer reports >7 - -### Intervention Strategies - -If metrics decline: - -1. **Slow Review Times?** - - Recruit more P2 reviewers - - Reduce PR scope requirements - - Pause new feature PRs temporarily - -2. **High Churn?** - - Exit interviews with departing contributors - - Address common pain points (docs, onboarding) - - Review Code of Conduct enforcement - -3. **Maintainer Burnout?** - - Rotate responsibilities - - Recruit co-maintainers - - Reduce project scope (pause Phase 2 items) - -## Communication Channels - -### Public - -- **GitHub Issues**: Bug reports, feature requests -- **GitHub Discussions**: General questions, ideas, RFCs -- **Documentation**: README, ARCHITECTURE, ROADMAP - -### Semi-Public (P2+) - -*Phase 2:* -- **Matrix/Discord**: Real-time chat for contributors -- **Monthly Calls**: Video sync (recorded, transcripts published) - -### Private (P1 only) - -- **Email**: security@evidencegraph.org (security issues) -- **Email**: conduct@evidencegraph.org (Code of Conduct reports) -- **Signal/Matrix**: Encrypted chat for sensitive governance - -## Conflict Resolution - -### Between Contributors - -1. **Direct Dialogue**: Encouraged to resolve privately (if safe) -2. **Mediation**: P2 contributor acts as neutral facilitator -3. **Escalation**: P1 maintainer makes decision if needed - -### With Maintainers - -1. **Appeal**: Contact different P1 maintainer or governance email -2. **Third-Party**: Suggest external mediator (community ombudsperson, future) -3. **Fork**: If irreconcilable, forking is a legitimate option (we'll still credit your work!) - -### Maintainer Accountability - -- P1 maintainers are also bound by Code of Conduct -- Violations handled by other P1 maintainers or external party -- Serious violations may result in maintainer removal (community vote) - -## Succession Planning - -### Adding Maintainers - -**Perimeter 2 Promotion:** -- After 3+ merged PRs (substantive, not trivial) -- Nomination by P1 maintainer -- Consensus among existing P2 contributors - -**Perimeter 1 Promotion:** -- After 6+ months as P2 contributor -- Demonstrated technical skill, community leadership, CoC adherence -- Unanimous P1 consensus -- Explicit acceptance by nominee (no pressure) - -### Removing Maintainers - -**Voluntary:** -- Announce intention to step down -- Transfer responsibilities (reviews, projects) -- Moved to "Emeritus" status (listed as alumni) - -**Involuntary:** -- Only for Code of Conduct violations or prolonged inactivity (>1 year) -- P1 consensus decision -- Graceful transition period (30 days) - -### Emeritus Maintainers - -*Future section for retired maintainers who shaped the project.* - -## Project Lead Succession - -**Current Lead**: @Hyperpolymath (until PhD completion, estimated Month 18) - -**Succession Plan (Phase 3):** -1. Identify 2-3 potential successors from P2 contributors -2. Gradual responsibility transfer (Month 15-17) -3. Community input period (30 days) -4. New lead appointed via P1 consensus -5. Previous lead remains as P1 advisor (unless stepping away entirely) - -## Legal & Financial Governance - -### Ownership - -- **Code**: Dual-licensed MIT + Palimpsest v0.8, contributors retain copyright -- **Trademark**: "Evidence Graph" name (if registered) owned by project, not individual -- **Domain**: evidencegraph.org (if acquired) held in trust for project - -### Funding (Phase 2+) - -**If grants/donations received:** -- Transparent accounting (quarterly reports) -- Funds used for: Hosting, conferences, contributor stipends (if budget allows) -- Major expenditures (>€1,000) require P1 consensus -- Financial records public (or summary if privacy needed) - -**Potential Sponsors:** -- Mozilla MOSS, Knight Foundation, EU Horizon -- NUJ (National Union of Journalists) partnership -- Academic institution grants - -**No Ads, No Data Selling:** -- This project will never monetize user data -- No advertising or tracking beyond essential analytics -- Privacy-first, investigative journalism-friendly - -## Acknowledgments - -### Recognition - -Contributors recognized in multiple places: - -1. **.well-known/humans.txt**: Public roll of honor -2. **CHANGELOG.md**: Specific contributions noted -3. **Academic Papers**: Co-authorship for research contributors (with consent) -4. **Conference Talks**: Credit to implementers of featured work - -### Awards & Grants - -If the project wins awards or receives grants, credit shared: - -- Maintainers acknowledged publicly -- Financial awards distributed to active contributors (if applicable) -- Trophy/plaque displayed in README - -## Governance History - -### Major Decisions - -| Date | Decision | Process | Outcome | -|------|----------|---------|---------| -| 2025-11-22 | Project initiated, TPCF Perimeter 3 chosen | Initial design | Approved by @Hyperpolymath | -| TBD | (Future decisions logged here) | | | - -### RFC Archive - -*RFCs will be stored in `docs/rfcs/` with status (Accepted/Rejected/Superseded)* - -## Contact - -- **Governance Questions**: governance@evidencegraph.org (or open GitHub Discussion) -- **Lead Maintainer**: @Hyperpolymath -- **Code of Conduct**: conduct@evidencegraph.org - ---- - -**Last Updated**: 2025-11-22 -**Governance Model**: TPCF Perimeter 3 (Community Sandbox) + Palimpsest v0.8 -**Emotional Temperature**: ✅ Healthy (Phase 1 baseline) - -*This governance model is a living document. Suggest improvements via GitHub Discussion or PR.* diff --git a/README.adoc b/README.adoc index 5b2a1c3..f5e6280 100644 --- a/README.adoc +++ b/README.adoc @@ -1,131 +1,156 @@ -= Binary-Origami Figurator -image:https://img.shields.io/badge/License-MPL--2.0-blue.svg[License: PMPL-1.0,link="https://github.com/hyperpolymath/palimpsest-license"] -// SPDX-License-Identifier: CC-BY-SA-4.0 -// SPDX-FileCopyrightText: 2025 hyperpolymath - - -> Infrastructure for pragmatic epistemology. Combining - -- i-docs navigation, -- PROMPT epistemological scoring, and -- boundary objects theory. - -An evidence graph for investigative journalism. - -*Status:* Phase 1 (PoC) - v1.0.0 Release -*Version:* 1.0.0 - -== New Here? Start with the Wiki +https://github.com/sponsors/hyperpolymath[image:https://img.shields.io/badge/Sponsor-%E2%9D%A4-pink?logo=github[Sponsor]] -[TIP] -==== -*Confused by the terminology?* The link:wiki/index.adoc[Binary-Origami Wiki] explains everything in metaphors, diagrams, and plain language. -==== += Binary-Origami Figurator +image:https://img.shields.io/badge/License-MPL–2.0-blue.svg[License: +PMPL-1.0,link="`https://github.com/hyperpolymath/palimpsest-license`"] +// SPDX-License-Identifier: CC-BY-SA-4.0 // SPDX-FileCopyrightText: 2025 +hyperpolymath -[quote] ____ -This isn't just a database. It's *infrastructure for folding and unfolding evidence*—so everyone can see the shape that fits their needs. +Infrastructure for pragmatic epistemology. Combining ____ -=== Why "Binary-Origami"? - -* *Binary*: Evidence is stored as clear, connected data (supports/contradicts, 0-100 scores) -* *Origami*: The same evidence can be "folded" into different forms for different audiences -* *Figuration*: The rules for folding/unfolding are transparent and reversible +* i-docs navigation, +* PROMPT epistemological scoring, and +* boundary objects theory. -[cols="1,2,1"] -|=== -| Concept | Description | Learn More +An evidence graph for investigative journalism. -| i-docs Navigation -| "Choose Your Own Adventure" for evidence -| link:wiki/navigation.adoc[Wiki] +_Status:_ Phase 1 (PoC) - v1.0.0 Release _Version:_ 1.0.0 -| PROMPT Scoring -| "Nutrition labels" for trustworthiness -| link:wiki/prompt.adoc[Wiki] +== New Here? Start with the Wiki -| Boundary Objects -| "Shared maps" with multiple routes -| link:wiki/boundary-objects.adoc[Wiki] +== [TIP] + +== _Confused by the terminology?_ The link:wiki/index.adoc[Binary-Origami Wiki] explains everything in metaphors, diagrams, and plain language. + +[quote] ____ This isn’t just a database. It’s _infrastructure for +folding and unfolding evidence_—so everyone can see the shape that fits +their needs. ____ + +=== Why "`Binary-Origami`"? + +* _Binary_: Evidence is stored as clear, connected data +(supports/contradicts, 0-100 scores) +* _Origami_: The same evidence can be "`folded`" into different forms +for different audiences +* _Figuration_: The rules for folding/unfolding are transparent and +reversible + +[cols="`1,2,1`"] |=== | Concept | Description | Learn More + +[verse] +-- +i-docs Navigation +"`Choose Your Own Adventure`" for evidence +link:wiki/navigation.adoc[Wiki] +-- + +[verse] +-- +PROMPT Scoring +"`Nutrition labels`" for trustworthiness +link:wiki/prompt.adoc[Wiki] +-- + +[verse] +-- +Boundary Objects +"`Shared maps`" with multiple routes +link:wiki/boundary-objects.adoc[Wiki] +-- + +[verse] +-- +Evidence Graphs +The "`skeleton`" beneath the origami +link:wiki/graphs.adoc[Wiki] +-- -| Evidence Graphs -| The "skeleton" beneath the origami -| link:wiki/graphs.adoc[Wiki] |=== === For the Impatient -1. Try the <> to see it in action -2. Skim the link:wiki/binary-origami.adoc[Binary-Origami Metaphor] page -3. Dive into the link:wiki/faqs.adoc[FAQ] if something's unclear +[arabic] +. Try the <> to see it in action +. Skim the link:wiki/binary-origami.adoc[Binary-Origami Metaphor] page +. Dive into the link:wiki/faqs.adoc[FAQ] if something’s unclear == Vision -We didn't fall from Truth to Post-Truth; we evolved to complex epistemology without building infrastructure. *This system IS that infrastructure.* +We didn’t fall from Truth to Post-Truth; we evolved to complex +epistemology without building infrastructure. _This system IS that +infrastructure._ === Core Concepts -1. *i-docs Navigation*: Navigation over narration, reader agency -2. *PROMPT Framework*: 6-dimensional epistemological scoring (Provenance, Replicability, Objective, Methodology, Publication, Transparency) -3. *Boundary Objects*: Multiple audience perspectives on same evidence -4. Evidence Graph for Investigative Journalism amd Related Disciplines +[arabic] +. _i-docs Navigation_: Navigation over narration, reader agency +. _PROMPT Framework_: 6-dimensional epistemological scoring (Provenance, +Replicability, Objective, Methodology, Publication, Transparency) +. _Boundary Objects_: Multiple audience perspectives on same evidence +. Evidence Graph for Investigative Journalism amd Related Disciplines -[[quick-start]] -== Quick Start +[[quick-start]] == Quick Start === Prerequisites -- *Elixir 1.18+* & *Erlang/OTP 27+* -- *Phoenix 1.8+* -- *Podman* & *podman-compose* -- *just* (task runner): https://github.com/casey/just +* _Elixir 1.18+_ & _Erlang/OTP 27+_ +* _Phoenix 1.8+_ +* _Podman_ & _podman-compose_ +* _just_ (task runner): https://github.com/casey/just === 1. Clone Repository -```bash +[source,bash] +---- git clone https://github.com/Hyperpolymath/bofig.git cd bofig -``` +---- === 2. Start Databases -```bash +[source,bash] +---- podman-compose up -d -``` +---- Verify ArangoDB is running: http://localhost:8529 (root/dev) === 3. Full Setup (deps, databases, seeds) -```bash +[source,bash] +---- just setup -``` +---- Or manually: -```bash + +[source,bash] +---- mix deps.get mix ecto.create mix run -e "EvidenceGraph.ArangoDB.setup_database()" mix run priv/repo/seeds.exs -``` +---- === 4. Start Phoenix Server -```bash +[source,bash] +---- just dev -``` +---- -Visit: -- *Application*: http://localhost:4000 (register/login required) -- *GraphQL Playground*: http://localhost:4000/api/graphiql (dev only) -- *Health Check*: http://localhost:4000/api/health +Visit: - _Application_: http://localhost:4000 (register/login required) +- _GraphQL Playground_: http://localhost:4000/api/graphiql (dev only) - +_Health Check_: http://localhost:4000/api/health == GraphQL API Examples === Query: Get All Claims -```graphql +[source,graphql] +---- query { claims(investigationId: "uk_inflation_2023") { id @@ -151,11 +176,12 @@ query { } } } -``` +---- === Query: Evidence Chain (Graph Traversal) -```graphql +[source,graphql] +---- query { evidenceChain(claimId: "claim_1", maxDepth: 3) { rootClaim { @@ -179,11 +205,12 @@ query { maxDepth } } -``` +---- === Mutation: Create Claim -```graphql +[source,graphql] +---- mutation { createClaim(input: { investigationId: "uk_inflation_2023" @@ -206,11 +233,12 @@ mutation { } } } -``` +---- === Mutation: Import from Zotero -```graphql +[source,graphql] +---- mutation { importFromZotero( investigationId: "uk_inflation_2023" @@ -228,11 +256,12 @@ mutation { zoteroKey } } -``` +---- === Query: Navigation Paths -```graphql +[source,graphql] +---- query { navigationPaths( investigationId: "uk_inflation_2023" @@ -249,11 +278,12 @@ query { } } } -``` +---- === Mutation: Auto-Generate Navigation Path -```graphql +[source,graphql] +---- mutation { autoGeneratePath( investigationId: "uk_inflation_2023" @@ -267,11 +297,11 @@ mutation { } } } -``` +---- == Project Structure -``` +.... bofig/ ├── lib/ │ ├── evidence_graph/ # Core business logic @@ -303,44 +333,49 @@ bofig/ ├── CLAUDE.md # AI assistant context ├── Containerfile # OCI container build └── podman-compose.yml # Container orchestration -``` +.... == UK Inflation 2023 Test Dataset The seed data includes a complete investigation: -- *7 Claims* (primary, supporting, counter) -- *10 Evidence items* (expand to 30) - - Official statistics: ONS CPI, Ofgem, BoE - - Academic: Peer-reviewed studies - - Think tanks: Resolution Foundation, IFS - - Interviews: Expert opinions -- *10 Relationships* (supports/contradicts/contextualizes) -- *3 Navigation Paths*: - 1. *Researcher*: Evidence-first, methodology priority - 2. *Policymaker*: Authoritative sources, recommendations - 3. *Affected Person*: Personal impact, clarity +* _7 Claims_ (primary, supporting, counter) +* _10 Evidence items_ (expand to 30) +** Official statistics: ONS CPI, Ofgem, BoE +** Academic: Peer-reviewed studies +** Think tanks: Resolution Foundation, IFS +** Interviews: Expert opinions +* _10 Relationships_ (supports/contradicts/contextualizes) +* _3 Navigation Paths_: +[arabic] +. _Researcher_: Evidence-first, methodology priority +. _Policymaker_: Authoritative sources, recommendations +. _Affected Person_: Personal impact, clarity === PROMPT Score Examples -| Evidence | Prov | Repl | Obj | Meth | Pub | Trans | Overall | -|----------|------|------|-----|------|-----|-------|---------| -| ONS CPI Data | 100 | 100 | 95 | 95 | 100 | 95 | *97.5* | -| Academic Study | 85 | 80 | 75 | 85 | 90 | 75 | *81.8* | -| Think Tank Report | 75 | 70 | 65 | 75 | 80 | 70 | *72.3* | -| Expert Interview | 85 | 45 | 60 | 50 | 40 | 75 | *59.0* | +[cols=",,,,,,,",options="header",] +|=== +|Evidence |Prov |Repl |Obj |Meth |Pub |Trans |Overall +|ONS CPI Data |100 |100 |95 |95 |100 |95 |_97.5_ +|Academic Study |85 |80 |75 |85 |90 |75 |_81.8_ +|Think Tank Report |75 |70 |65 |75 |80 |70 |_72.3_ +|Expert Interview |85 |45 |60 |50 |40 |75 |_59.0_ +|=== == Development === Run Tests -```bash +[source,bash] +---- mix test -``` +---- === Interactive Shell -```bash +[source,bash] +---- iex -S mix phx.server = Query ArangoDB directly @@ -354,11 +389,12 @@ iex> EvidenceGraph.Claims.get_claim("claim_1") = Evidence chain traversal iex> EvidenceGraph.Relationships.evidence_chain("claim_1", 3) -``` +---- === Code Quality -```bash +[source,bash] +---- = Format code mix format @@ -370,32 +406,37 @@ mix credo = Type checking mix dialyzer -``` +---- == Deployment (Phase 2) -- *Hosting*: Hetzner Cloud (EU data sovereignty) -- *ArangoDB*: ArangoDB Oasis (€45/month) -- *Phoenix*: Systemd service, Nginx reverse proxy -- *CI/CD*: GitHub Actions +* _Hosting_: Hetzner Cloud (EU data sovereignty) +* _ArangoDB_: ArangoDB Oasis (€45/month) +* _Phoenix_: Systemd service, Nginx reverse proxy +* _CI/CD_: GitHub Actions == Documentation === Conceptual (Start Here!) -- link:wiki/index.adoc[Binary-Origami Wiki] - Explains the metaphor and concepts in plain language -- link:wiki/binary-origami.adoc[The Metaphor] - Why "Binary-Origami Figuration"? -- link:wiki/folding-101.adoc[Folding 101] - Hands-on tutorial -- link:wiki/faqs.adoc[FAQ] - Common questions answered -- link:wiki/glossary.adoc[Glossary] - Term definitions +* link:wiki/index.adoc[Binary-Origami Wiki] - Explains the metaphor and +concepts in plain language +* link:wiki/binary-origami.adoc[The Metaphor] - Why "`Binary-Origami +Figuration`"? +* link:wiki/folding-101.adoc[Folding 101] - Hands-on tutorial +* link:wiki/faqs.adoc[FAQ] - Common questions answered +* link:wiki/glossary.adoc[Glossary] - Term definitions === Technical -- link:ARCHITECTURE.md[ARCHITECTURE.md] - Data model, database design, API specs -- link:ROADMAP.md[ROADMAP.md] - 18-month implementation plan -- link:docs/database-evaluation.md[docs/database-evaluation.md] - ArangoDB comparison -- link:docs/zotero-integration.md[docs/zotero-integration.md] - Two-way sync design -- link:CLAUDE.md[CLAUDE.md] - AI assistant context +* link:ARCHITECTURE.md[ARCHITECTURE.md] - Data model, database design, +API specs +* link:ROADMAP.md[ROADMAP.md] - 18-month implementation plan +* link:docs/database-evaluation.md[docs/database-evaluation.md] - +ArangoDB comparison +* link:docs/zotero-integration.md[docs/zotero-integration.md] - Two-way +sync design +* link:CLAUDE.md[CLAUDE.md] - AI assistant context == Key Features @@ -405,13 +446,16 @@ mix dialyzer * GraphQL API with Absinthe (15 queries, 11 mutations) * PROMPT epistemological scoring (6 dimensions, audience weighting) * Claims, Evidence, Relationships, Navigation Paths data models -* Graph traversal algorithms (evidence chains, shortest path, contradiction detection) +* Graph traversal algorithms (evidence chains, shortest path, +contradiction detection) * Zotero REST API (import, export, batch-import, sync-status) -* Phoenix 1.8 LiveView frontend (5 pages: Dashboard, Investigation, Graph, PROMPT, Navigation) +* Phoenix 1.8 LiveView frontend (5 pages: Dashboard, Investigation, +Graph, PROMPT, Navigation) * User authentication (phx.gen.auth with bcrypt, magic links) * D3.js force-directed graph + radar chart visualisations * Audience-weighted navigation paths (6 types) -* UK Inflation 2023 test dataset (7 claims, 30 evidence, 38 relationships) +* UK Inflation 2023 test dataset (7 claims, 30 evidence, 38 +relationships) * Production deployment (Containerfile, nginx, systemd) * NUJ user testing protocols * A2ML v2.1 Cyberwar-Ready Trustfile @@ -428,25 +472,30 @@ mix dialyzer == Philosophy -This isn't just a database. It's infrastructure for *coordinating without consensus*. +This isn’t just a database. It’s infrastructure for _coordinating +without consensus_. -Every design choice asks: -1. Does this support multiple audience perspectives? -2. Does this make epistemology measurable? -3. Does this enable navigation over narration? +Every design choice asks: 1. Does this support multiple audience +perspectives? 2. Does this make epistemology measurable? 3. Does this +enable navigation over narration? == Contributing -Open source from day 1. See [ROADMAP.md](ROADMAP.md) for planned features. +Open source from day 1. See ROADMAP.md for planned features. -*Month 3 = Decision Point*: User testing with 25 NUJ journalists determines go/no-go. +_Month 3 = Decision Point_: User testing with 25 NUJ journalists +determines go/no-go. == Related Projects -* https://github.com/hyperpolymath/lithoglyph[LithoglyphDB] - The narrative-first, reversible, audit-grade database -* https://github.com/hyperpolymath/gpnl[GPNL] - Dependently-typed Glyph Projection Language (compile-time proofs) -* https://github.com/hyperpolymath/lithoglyph-studio[Lithoglyph Studio] - Zero-friction GUI for non-technical users -* https://github.com/hyperpolymath/zotero-formdb[Zotero-Lithoglyph] - Reference manager with PROMPT scores +* https://github.com/hyperpolymath/lithoglyph[LithoglyphDB] - The +narrative-first, reversible, audit-grade database +* https://github.com/hyperpolymath/gpnl[GPNL] - Dependently-typed Glyph +Projection Language (compile-time proofs) +* https://github.com/hyperpolymath/lithoglyph-studio[Lithoglyph Studio] +- Zero-friction GUI for non-technical users +* https://github.com/hyperpolymath/zotero-formdb[Zotero-Lithoglyph] - +Reference manager with PROMPT scores == License @@ -454,19 +503,20 @@ link:LICENSE[MPL-2.0] (Palimpsest License) == Contact -- *Repository*: https://github.com/Hyperpolymath/bofig -- *Issues*: https://github.com/Hyperpolymath/bofig/issues -- *User Testing*: NUJ network (Month 3, 6, 12) - ---- +* _Repository_: https://github.com/Hyperpolymath/bofig +* _Issues_: https://github.com/Hyperpolymath/bofig/issues +* _User Testing_: NUJ network (Month 3, 6, 12) -*Built with:* Elixir, Phoenix, ArangoDB, Absinthe, LiveView, D3.js +''''' -*Inspired by:* i-docs (PMPL-1.0 Open Doc Lab), Boundary Objects (Star & Griesemer), Pragmatic Epistemology +_Built with:_ Elixir, Phoenix, ArangoDB, Absinthe, LiveView, D3.js -*Last Updated:* 2026-02-21 +_Inspired by:_ i-docs (PMPL-1.0 Open Doc Lab), Boundary Objects (Star & +Griesemer), Pragmatic Epistemology +_Last Updated:_ 2026-02-21 == Architecture -See link:TOPOLOGY.md[TOPOLOGY.md] for a visual architecture map and completion dashboard. +See link:TOPOLOGY.md[TOPOLOGY.md] for a visual architecture map and +completion dashboard. diff --git a/README.md b/README.md deleted file mode 100644 index 751cd95..0000000 --- a/README.md +++ /dev/null @@ -1,474 +0,0 @@ -[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-pink?logo=github)](https://github.com/sponsors/hyperpolymath) - -= Binary-Origami Figurator -image:https://img.shields.io/badge/License-MPL--2.0-blue.svg[License: PMPL-1.0,link="https://github.com/hyperpolymath/palimpsest-license"] -// SPDX-License-Identifier: CC-BY-SA-4.0 -// SPDX-FileCopyrightText: 2025 hyperpolymath - - -> Infrastructure for pragmatic epistemology. Combining - -- i-docs navigation, -- PROMPT epistemological scoring, and -- boundary objects theory. - -An evidence graph for investigative journalism. - -*Status:* Phase 1 (PoC) - v1.0.0 Release -*Version:* 1.0.0 - -== New Here? Start with the Wiki - -[TIP] -==== -*Confused by the terminology?* The link:wiki/index.adoc[Binary-Origami Wiki] explains everything in metaphors, diagrams, and plain language. -==== - -[quote] -____ -This isn't just a database. It's *infrastructure for folding and unfolding evidence*—so everyone can see the shape that fits their needs. -____ - -=== Why "Binary-Origami"? - -* *Binary*: Evidence is stored as clear, connected data (supports/contradicts, 0-100 scores) -* *Origami*: The same evidence can be "folded" into different forms for different audiences -* *Figuration*: The rules for folding/unfolding are transparent and reversible - -[cols="1,2,1"] -|=== -| Concept | Description | Learn More - -| i-docs Navigation -| "Choose Your Own Adventure" for evidence -| link:wiki/navigation.adoc[Wiki] - -| PROMPT Scoring -| "Nutrition labels" for trustworthiness -| link:wiki/prompt.adoc[Wiki] - -| Boundary Objects -| "Shared maps" with multiple routes -| link:wiki/boundary-objects.adoc[Wiki] - -| Evidence Graphs -| The "skeleton" beneath the origami -| link:wiki/graphs.adoc[Wiki] -|=== - -=== For the Impatient - -1. Try the <> to see it in action -2. Skim the link:wiki/binary-origami.adoc[Binary-Origami Metaphor] page -3. Dive into the link:wiki/faqs.adoc[FAQ] if something's unclear - -== Vision - -We didn't fall from Truth to Post-Truth; we evolved to complex epistemology without building infrastructure. *This system IS that infrastructure.* - -=== Core Concepts - -1. *i-docs Navigation*: Navigation over narration, reader agency -2. *PROMPT Framework*: 6-dimensional epistemological scoring (Provenance, Replicability, Objective, Methodology, Publication, Transparency) -3. *Boundary Objects*: Multiple audience perspectives on same evidence -4. Evidence Graph for Investigative Journalism amd Related Disciplines - -[[quick-start]] -== Quick Start - -=== Prerequisites - -- *Elixir 1.18+* & *Erlang/OTP 27+* -- *Phoenix 1.8+* -- *Podman* & *podman-compose* -- *just* (task runner): https://github.com/casey/just - -=== 1. Clone Repository - -```bash -git clone https://github.com/Hyperpolymath/bofig.git -cd bofig -``` - -=== 2. Start Databases - -```bash -podman-compose up -d -``` - -Verify ArangoDB is running: http://localhost:8529 (root/dev) - -=== 3. Full Setup (deps, databases, seeds) - -```bash -just setup -``` - -Or manually: -```bash -mix deps.get -mix ecto.create -mix run -e "EvidenceGraph.ArangoDB.setup_database()" -mix run priv/repo/seeds.exs -``` - -=== 4. Start Phoenix Server - -```bash -just dev -``` - -Visit: -- *Application*: http://localhost:4000 (register/login required) -- *GraphQL Playground*: http://localhost:4000/api/graphiql (dev only) -- *Health Check*: http://localhost:4000/api/health - -== GraphQL API Examples - -=== Query: Get All Claims - -```graphql -query { - claims(investigationId: "uk_inflation_2023") { - id - text - claimType - confidenceLevel - promptScores { - provenance - replicability - objective - methodology - publication - transparency - overall - } - supportingEvidence { - evidence { - title - evidenceType - } - weight - confidence - } - } -} -``` - -=== Query: Evidence Chain (Graph Traversal) - -```graphql -query { - evidenceChain(claimId: "claim_1", maxDepth: 3) { - rootClaim { - text - } - nodes { - ... on Claim { - id - text - } - ... on Evidence { - id - title - } - } - edges { - relationshipType - weight - confidence - } - maxDepth - } -} -``` - -=== Mutation: Create Claim - -```graphql -mutation { - createClaim(input: { - investigationId: "uk_inflation_2023" - text: "Inflation disproportionately affected renters" - claimType: SUPPORTING - confidenceLevel: 0.85 - promptScores: { - provenance: 70 - replicability: 65 - objective: 75 - methodology: 70 - publication: 65 - transparency: 70 - } - }) { - id - text - promptScores { - overall - } - } -} -``` - -=== Mutation: Import from Zotero - -```graphql -mutation { - importFromZotero( - investigationId: "uk_inflation_2023" - zoteroJson: { - key: "ABC123" - itemType: "journalArticle" - title: "New Economic Study" - url: "https://doi.org/10.1111/example" - creators: [{name: "Smith, J."}] - tags: [{tag: "economics"}] - } - ) { - id - title - zoteroKey - } -} -``` - -=== Query: Navigation Paths - -```graphql -query { - navigationPaths( - investigationId: "uk_inflation_2023" - audienceType: RESEARCHER - ) { - id - name - description - pathNodes { - entityId - entityType - order - context - } - } -} -``` - -=== Mutation: Auto-Generate Navigation Path - -```graphql -mutation { - autoGeneratePath( - investigationId: "uk_inflation_2023" - audienceType: SKEPTIC - ) { - id - name - pathNodes { - entityId - order - } - } -} -``` - -== Project Structure - -``` -bofig/ -├── lib/ -│ ├── evidence_graph/ # Core business logic -│ │ ├── claims/ # Claims context -│ │ │ └── claim.ex -│ │ ├── evidence/ # Evidence context -│ │ │ └── evidence.ex -│ │ ├── relationships/ # Graph edges -│ │ │ └── relationship.ex -│ │ ├── navigation/ # Audience paths -│ │ │ └── path.ex -│ │ ├── arango.ex # ArangoDB client -│ │ ├── prompt_scores.ex # PROMPT scoring -│ │ └── application.ex # OTP supervisor -│ └── evidence_graph_web/ # Phoenix web layer -│ ├── schema/ # GraphQL schema -│ │ ├── types/ # Type definitions -│ │ └── schema.ex # Root schema -│ ├── endpoint.ex -│ └── router.ex -├── priv/repo/ -│ └── seeds.exs # UK Inflation 2023 test data -├── config/ # Environment configs -├── docs/ # Architecture docs -│ ├── database-evaluation.md -│ └── zotero-integration.md -├── ARCHITECTURE.md # Data model, API design -├── ROADMAP.md # 18-month plan -├── CLAUDE.md # AI assistant context -├── Containerfile # OCI container build -└── podman-compose.yml # Container orchestration -``` - -== UK Inflation 2023 Test Dataset - -The seed data includes a complete investigation: - -- *7 Claims* (primary, supporting, counter) -- *10 Evidence items* (expand to 30) - - Official statistics: ONS CPI, Ofgem, BoE - - Academic: Peer-reviewed studies - - Think tanks: Resolution Foundation, IFS - - Interviews: Expert opinions -- *10 Relationships* (supports/contradicts/contextualizes) -- *3 Navigation Paths*: - 1. *Researcher*: Evidence-first, methodology priority - 2. *Policymaker*: Authoritative sources, recommendations - 3. *Affected Person*: Personal impact, clarity - -=== PROMPT Score Examples - -| Evidence | Prov | Repl | Obj | Meth | Pub | Trans | Overall | -|----------|------|------|-----|------|-----|-------|---------| -| ONS CPI Data | 100 | 100 | 95 | 95 | 100 | 95 | *97.5* | -| Academic Study | 85 | 80 | 75 | 85 | 90 | 75 | *81.8* | -| Think Tank Report | 75 | 70 | 65 | 75 | 80 | 70 | *72.3* | -| Expert Interview | 85 | 45 | 60 | 50 | 40 | 75 | *59.0* | - -== Development - -=== Run Tests - -```bash -mix test -``` - -=== Interactive Shell - -```bash -iex -S mix phx.server - -= Query ArangoDB directly - -iex> EvidenceGraph.ArangoDB.query("FOR c IN claims RETURN c") - -= Get a claim - -iex> EvidenceGraph.Claims.get_claim("claim_1") - -= Evidence chain traversal - -iex> EvidenceGraph.Relationships.evidence_chain("claim_1", 3) -``` - -=== Code Quality - -```bash -= Format code - -mix format - -= Static analysis - -mix credo - -= Type checking - -mix dialyzer -``` - -== Deployment (Phase 2) - -- *Hosting*: Hetzner Cloud (EU data sovereignty) -- *ArangoDB*: ArangoDB Oasis (€45/month) -- *Phoenix*: Systemd service, Nginx reverse proxy -- *CI/CD*: GitHub Actions - -== Documentation - -=== Conceptual (Start Here!) - -- link:wiki/index.adoc[Binary-Origami Wiki] - Explains the metaphor and concepts in plain language -- link:wiki/binary-origami.adoc[The Metaphor] - Why "Binary-Origami Figuration"? -- link:wiki/folding-101.adoc[Folding 101] - Hands-on tutorial -- link:wiki/faqs.adoc[FAQ] - Common questions answered -- link:wiki/glossary.adoc[Glossary] - Term definitions - -=== Technical - -- link:ARCHITECTURE.md[ARCHITECTURE.md] - Data model, database design, API specs -- link:ROADMAP.md[ROADMAP.md] - 18-month implementation plan -- link:docs/database-evaluation.md[docs/database-evaluation.md] - ArangoDB comparison -- link:docs/zotero-integration.md[docs/zotero-integration.md] - Two-way sync design -- link:CLAUDE.md[CLAUDE.md] - AI assistant context - -== Key Features - -=== Implemented (v1.0.0) - -* Multi-model ArangoDB integration (document + graph) -* GraphQL API with Absinthe (15 queries, 11 mutations) -* PROMPT epistemological scoring (6 dimensions, audience weighting) -* Claims, Evidence, Relationships, Navigation Paths data models -* Graph traversal algorithms (evidence chains, shortest path, contradiction detection) -* Zotero REST API (import, export, batch-import, sync-status) -* Phoenix 1.8 LiveView frontend (5 pages: Dashboard, Investigation, Graph, PROMPT, Navigation) -* User authentication (phx.gen.auth with bcrypt, magic links) -* D3.js force-directed graph + radar chart visualisations -* Audience-weighted navigation paths (6 types) -* UK Inflation 2023 test dataset (7 claims, 30 evidence, 38 relationships) -* Production deployment (Containerfile, nginx, systemd) -* NUJ user testing protocols -* A2ML v2.1 Cyberwar-Ready Trustfile -* 257 tests, 0 failures, full RSR compliance - -=== Coming Next (Phase 2) - -* Zotero browser extension (one-click import) -* Multi-investigation dashboard -* Real-time collaborative editing -* Advanced visualisations (timeline, heatmap, Sankey) -* IPFS provenance integration -* Hetzner Cloud deployment - -== Philosophy - -This isn't just a database. It's infrastructure for *coordinating without consensus*. - -Every design choice asks: -1. Does this support multiple audience perspectives? -2. Does this make epistemology measurable? -3. Does this enable navigation over narration? - -== Contributing - -Open source from day 1. See [ROADMAP.md](ROADMAP.md) for planned features. - -*Month 3 = Decision Point*: User testing with 25 NUJ journalists determines go/no-go. - -== Related Projects - -* https://github.com/hyperpolymath/lithoglyph[LithoglyphDB] - The narrative-first, reversible, audit-grade database -* https://github.com/hyperpolymath/gpnl[GPNL] - Dependently-typed Glyph Projection Language (compile-time proofs) -* https://github.com/hyperpolymath/lithoglyph-studio[Lithoglyph Studio] - Zero-friction GUI for non-technical users -* https://github.com/hyperpolymath/zotero-formdb[Zotero-Lithoglyph] - Reference manager with PROMPT scores - -== License - -link:LICENSE[MPL-2.0] (Palimpsest License) - -== Contact - -- *Repository*: https://github.com/Hyperpolymath/bofig -- *Issues*: https://github.com/Hyperpolymath/bofig/issues -- *User Testing*: NUJ network (Month 3, 6, 12) - ---- - -*Built with:* Elixir, Phoenix, ArangoDB, Absinthe, LiveView, D3.js - -*Inspired by:* i-docs (PMPL-1.0 Open Doc Lab), Boundary Objects (Star & Griesemer), Pragmatic Epistemology - -*Last Updated:* 2026-02-21 - - -== Architecture - -See link:TOPOLOGY.md[TOPOLOGY.md] for a visual architecture map and completion dashboard. diff --git a/REQUIRES_INITIALISATION.adoc b/REQUIRES_INITIALISATION.adoc new file mode 100644 index 0000000..8c93317 --- /dev/null +++ b/REQUIRES_INITIALISATION.adoc @@ -0,0 +1,112 @@ +== REQUIRES INITIALISATION + +*This repository is not finished being set up.* 8 substitution token(s) +across 2 file(s) still have no value. + +=== Why this is not already done + +This repo was created from `+hyperpolymath/rsr-template-repo+`. The mint +(`+just repo-init+`) fills every token that has a single mechanical +answer — owner, repo, author, dates, licence, branch — and it has done +so here. + +The tokens below are the ones it _deliberately cannot_ answer. They need +a decision or a fact that exists only in your head: what this project is +for, what command builds it, which port the service listens on, whether +a PGP key is held at all. The template’s own token vocabulary says as +much — you cannot sensibly answer "`required invariants`" in a +thirty-second bootstrap. + +They were left *visibly unfilled on purpose*. The alternatives were both +worse: inventing plausible values would put confident falsehoods into a +security policy and an architecture document, and silently deleting the +sections would hide the fact that a decision is owed. A visible gap is +honest; a fabricated answer is not. + +=== Do not delete this file until every item below is resolved + +This file is the only marker that the work is outstanding. Deleting it +early does not finish the setup, it just conceals it — and the next +person or agent to arrive will reasonably assume the repo is complete. + +* *If you are a person:* delete this file yourself once the last item is +done. +* *If you are an agent:* resolve what you legitimately can, leave the +rest, and delete this file only when no token below remains anywhere in +the tree. Do not delete it to make a gate go green. + +Re-running the estate top-up tool will remove this file automatically +once nothing is outstanding, so the safest way to finish is to fix the +tokens and let the check confirm it. + +=== What is needed, and where it goes + +==== `+{{CONSUMER1}}+` + +A downstream repo that consumes this one. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{CONSUMER2}}+` + +A second downstream consumer. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{DEP1}}+` + +First named dependency, in .machine_readable/INTENT.contractile. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{DEP2}}+` + +Second named dependency, in .machine_readable/INTENT.contractile. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{MONOREPO_OR_STANDALONE}}+` + +Literally '`monorepo`' or '`standalone`'. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{ONE_PARAGRAPH_ANTI_PURPOSE}}+` + +A paragraph on what this deliberately is NOT for. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{ONE_PARAGRAPH_PURPOSE}}+` + +A paragraph on what this is for. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{PROJECT_UNIQUE_STRENGTH}}+` + +What this does that its alternatives do not. + +Appears in: + +* `+.machine_readable/agent_instructions/methodology.a2ml+` + +''''' + +Generated by the estate top-up pass. Rationale and the governing rulings +are in `+hyperpolymath/standards+`; the token vocabulary is +`+.machine_readable/ai/PLACEHOLDERS.adoc+` in `+rsr-template-repo+`. diff --git a/REQUIRES_INITIALISATION.md b/REQUIRES_INITIALISATION.md deleted file mode 100644 index b5a1dbf..0000000 --- a/REQUIRES_INITIALISATION.md +++ /dev/null @@ -1,110 +0,0 @@ - - -# REQUIRES INITIALISATION - -**This repository is not finished being set up.** 8 substitution token(s) across 2 file(s) still have no value. - -## Why this is not already done - -This repo was created from `hyperpolymath/rsr-template-repo`. The mint -(`just repo-init`) fills every token that has a single mechanical answer — -owner, repo, author, dates, licence, branch — and it has done so here. - -The tokens below are the ones it *deliberately cannot* answer. They need a -decision or a fact that exists only in your head: what this project is for, -what command builds it, which port the service listens on, whether a PGP key -is held at all. The template's own token vocabulary says as much — you cannot -sensibly answer "required invariants" in a thirty-second bootstrap. - -They were left **visibly unfilled on purpose**. The alternatives were both -worse: inventing plausible values would put confident falsehoods into a -security policy and an architecture document, and silently deleting the -sections would hide the fact that a decision is owed. A visible gap is -honest; a fabricated answer is not. - -## Do not delete this file until every item below is resolved - -This file is the only marker that the work is outstanding. Deleting it early -does not finish the setup, it just conceals it — and the next person or agent -to arrive will reasonably assume the repo is complete. - -- **If you are a person:** delete this file yourself once the last item is done. -- **If you are an agent:** resolve what you legitimately can, leave the rest, - and delete this file only when no token below remains anywhere in the tree. - Do not delete it to make a gate go green. - -Re-running the estate top-up tool will remove this file automatically once -nothing is outstanding, so the safest way to finish is to fix the tokens and -let the check confirm it. - -## What is needed, and where it goes - -### `{{CONSUMER1}}` - -A downstream repo that consumes this one. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{CONSUMER2}}` - -A second downstream consumer. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{DEP1}}` - -First named dependency, in .machine_readable/INTENT.contractile. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{DEP2}}` - -Second named dependency, in .machine_readable/INTENT.contractile. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{MONOREPO_OR_STANDALONE}}` - -Literally 'monorepo' or 'standalone'. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{ONE_PARAGRAPH_ANTI_PURPOSE}}` - -A paragraph on what this deliberately is NOT for. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{ONE_PARAGRAPH_PURPOSE}}` - -A paragraph on what this is for. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{PROJECT_UNIQUE_STRENGTH}}` - -What this does that its alternatives do not. - -Appears in: - -- `.machine_readable/agent_instructions/methodology.a2ml` - ---- - -Generated by the estate top-up pass. Rationale and the governing rulings are -in `hyperpolymath/standards`; the token vocabulary is -`.machine_readable/ai/PLACEHOLDERS.adoc` in `rsr-template-repo`. diff --git a/RSR_COMPLIANCE.adoc b/RSR_COMPLIANCE.adoc new file mode 100644 index 0000000..042b65a --- /dev/null +++ b/RSR_COMPLIANCE.adoc @@ -0,0 +1,501 @@ +== RSR Framework Compliance Report + +*Project:* Evidence Graph for Investigative Journalism *RSR Version:* +Adapted for Elixir/Phoenix (from Rust-based Rhodium minimal example) +*Compliance Level:* 🥈 *SILVER* (11/12 criteria) *Last Assessed:* +2025-11-22 + +''''' + +=== Executive Summary + +The Evidence Graph project achieves *RSR Silver Level* compliance, +implementing 11 of 12 core Rhodium Standard Repository framework +criteria. As an Elixir/Phoenix project, certain criteria (e.g., +Rust-specific memory safety) are adapted to ecosystem-appropriate +equivalents. + +*Strengths:* - ✅ Complete documentation suite (LICENSE, SECURITY, +CONTRIBUTING, CoC, MAINTAINERS, CHANGELOG) - ✅ .well-known/ directory +fully implemented (RFC 9116 security.txt, ai.txt, humans.txt) - ✅ TPCF +Perimeter 3 (Community Sandbox) governance model - ✅ Palimpsest License +v0.8 for emotional safety and reversibility - ✅ Task automation +(Justfile with 30+ recipes) - ✅ CI/CD pipeline (.gitlab-ci.yml with RSR +compliance checks) + +*Areas for Improvement:* - ⚠️ Test coverage (seeds.exs only, need +comprehensive ExUnit test suite - Month 2 goal) - ⚠️ Offline-first +consideration (currently requires network for ArangoDB/GraphQL - +architectural trade-off) + +''''' + +=== RSR Framework Criteria Checklist + +==== 1. Type Safety ✅ + +*Requirement:* Compile-time type guarantees to prevent runtime errors. + +*Implementation:* - *Elixir Typespecs:* All public functions have +`+@spec+` declarations - *Ecto Schemas:* Database entities defined with +strict types - *GraphQL Schema:* Absinthe provides strong typing at API +boundary - *Pattern Matching:* Elixir’s pattern matching enforces type +safety at runtime + +*Examples:* + +[source,elixir] +---- +@spec create_claim(map()) :: {:ok, Claim.t()} | {:error, Ecto.Changeset.t()} +def create_claim(attrs) do + %Claim{} + |> Claim.changeset(attrs) + |> insert_to_arango() +end +---- + +*Compliance Level:* ✅ *Full* (Elixir ecosystem equivalent of Rust +compile-time checks) + +''''' + +==== 2. Memory Safety ✅ + +*Requirement:* No memory leaks, buffer overflows, or use-after-free +errors. + +*Implementation:* - *BEAM VM Garbage Collection:* Automatic memory +management - *No Manual Allocation:* Elixir abstracts memory management +entirely - *Process Isolation:* Each BEAM process has isolated heap - +*No Unsafe Blocks:* Elixir has no `+unsafe+` keyword (unlike Rust) + +*Trade-off:* - Rust: Zero-cost abstractions, compile-time ownership - +Elixir: GC overhead, but guaranteed memory safety via BEAM VM + +*Compliance Level:* ✅ *Full* (Different approach, same safety +guarantees) + +''''' + +==== 3. Offline-First ⚠️ + +*Requirement:* Works without network connectivity (air-gapped +environments). + +*Current Status:* - ❌ Requires ArangoDB (can be local, but needs +running database) - ❌ GraphQL API assumes client-server architecture - +❌ No LocalStorage/IndexedDB fallback for browser (Phase 1 has no +browser UI yet) + +*Future Improvements (Phase 2-3):* - ✅ CRDTs for offline-first state +(see ROADMAP.md Month 13-14) - ✅ Service Workers for PWA offline mode +(see ARCHITECTURE.md) - ✅ Optional IPFS for peer-to-peer data sync + +*Rationale for Non-Compliance:* - Investigative journalism use case +assumes network availability for collaboration - Multi-user graph +database inherently requires coordination - Acceptable trade-off for +Phase 1 PoC + +*Compliance Level:* ⚠️ *Partial* (Architectural decision, not technical +limitation) + +''''' + +==== 4. Documentation - README ✅ + +*Requirement:* Comprehensive README with quick start, examples, and +project overview. + +*Implementation:* - ✅ Quick start guide (5-step setup) - ✅ GraphQL API +examples (8 queries/mutations) - ✅ Project structure diagram - ✅ +Development workflow (test, lint, deploy) - ✅ UK Inflation 2023 dataset +description - ✅ Contact & support information + +*Length:* ~500 lines, ~3,500 words + +*Compliance Level:* ✅ *Exceeds* (Detailed, beginner-friendly) + +''''' + +==== 5. Documentation - LICENSE ✅ + +*Requirement:* Clear license terms, dual-license support. + +*Implementation:* - ✅ *Dual License:* MIT + Palimpsest v0.8 - ✅ MIT: +Standard open source, permissive - ✅ Palimpsest v0.8: Emotional safety +provisions (reversibility, no coerced labor, emotional temperature +disclosure) - ✅ Contributor License Agreement (implicit via +CONTRIBUTING.md) - ✅ Third-party license acknowledgments + +*Unique Features:* - 90-day contribution withdrawal window - Emotional +temperature metrics requirement - TPCF perimeter framework integration + +*Compliance Level:* ✅ *Exceeds* (Dual license provides more protections +than typical OSS) + +''''' + +==== 6. Documentation - SECURITY.md ✅ + +*Requirement:* Vulnerability disclosure policy, supported versions, +contact info. + +*Implementation:* - ✅ Supported versions table (0.1.x = active) - ✅ +Threat model (10+ security dimensions) - ✅ Reporting procedures (email, +GitHub Security Advisory) - ✅ Response timeline (24h acknowledgment, +30-90 day fix) - ✅ Severity levels (Critical/High/Medium/Low) - ✅ +Known limitations documented (no auth in Phase 1) - ✅ GDPR compliance +section (EU data sovereignty) - ✅ Secure coding practices (OWASP Top +10) - ✅ Incident response plan + +*Compliance Level:* ✅ *Exceeds* (Comprehensive, GDPR-aware) + +''''' + +==== 7. Documentation - CONTRIBUTING.md ✅ + +*Requirement:* Contribution guidelines, code standards, review process. + +*Implementation:* - ✅ *TPCF Perimeter 3 Model:* Graduated trust +(P1/P2/P3 access levels) - ✅ Setup instructions (7-step guide) - ✅ +Branching strategy (GitHub Flow) - ✅ Commit message format +(Conventional Commits) - ✅ Code standards (Elixir Style Guide) - ✅ +Testing requirements (80% coverage target) - ✅ Review process (48h +first review, 7 day merge) - ✅ *Reversibility & Emotional Safety* +(90-day withdrawal, no coercion) + +*Unique Features:* - Explicit "`no coerced labor`" policy - Emotional +temperature monitoring - Advancement path (P3 → P2 → P1) + +*Compliance Level:* ✅ *Exceeds* (Emotional safety-first approach) + +''''' + +==== 8. Documentation - CODE_OF_CONDUCT.md ✅ + +*Requirement:* Community standards, enforcement procedures. + +*Implementation:* - ✅ Based on *CCCP (Community-Centric Code of +Practice)* - ✅ Expected behavior (kind, empathetic, collaborative) - ✅ +Unacceptable behavior (harassment, discrimination, manipulation) - ✅ +Reporting process (3 channels: email, GitHub PM, third-party) - ✅ +Enforcement (4 levels: warning, temp ban, perm ban, legal action) - ✅ +Emotional safety framework (reversibility, temperature metrics) - ✅ +Conflict resolution (mediation before escalation) + +*Unique Features:* - Prioritizes contributor emotional well-being - +Integrates with Palimpsest License reversibility - Transparent +enforcement (decisions documented) + +*Compliance Level:* ✅ *Exceeds* (Beyond Contributor Covenant, adds +emotional safety) + +''''' + +==== 9. Documentation - MAINTAINERS.md ✅ + +*Requirement:* Governance model, decision-making process, maintainer +list. + +*Implementation:* - ✅ Current maintainers (P1/P2/P3 lists) - ✅ +Decision-making framework (4 tiers: Critical/Major/Minor/Trivial) - ✅ +RFC process (for Tier 1 decisions) - ✅ *Emotional Temperature +Dashboard* (8 metrics tracked) - ✅ Communication channels (public, +semi-public, private) - ✅ Conflict resolution procedures - ✅ +Succession planning (P2→P1 promotion path) - ✅ Legal & financial +governance (transparent accounting) + +*Unique Features:* - Real-time emotional temperature metrics (churn, +burnout risk, review times) - Tri-Perimeter governance integrated - +Project Lead succession plan (Month 18) + +*Compliance Level:* ✅ *Exceeds* (Transparent, data-driven governance) + +''''' + +==== 10. Documentation - CHANGELOG.md ✅ + +*Requirement:* Version history following Keep a Changelog format, +SemVer. + +*Implementation:* - ✅ https://keepachangelog.com/[Keep a Changelog] +format - ✅ Semantic Versioning (0.1.0 = Phase 1 PoC) - ✅ Sections: +Added/Changed/Deprecated/Removed/Fixed/Security - ✅ Contributor credits +(per version) - ✅ Upgrade guide (from 0.0.0 → 0.1.0) - ✅ Roadmap +preview (next 3 releases) + +*Compliance Level:* ✅ *Full* (Standard-compliant) + +''''' + +==== 11. .well-known/ Directory ✅ + +*Requirement:* RFC 9116 security.txt, ai.txt, humans.txt for +discoverability. + +*Implementation:* + +===== security.txt (RFC 9116) + +* ✅ Contact field (email + GitHub Security Advisory) +* ✅ Expires field (2026-11-22) +* ✅ Preferred-Languages (en) +* ✅ Canonical URL +* ✅ Policy link (SECURITY.md) +* ✅ Acknowledgments link (humans.txt) + +===== ai.txt (AI Training Policy) + +* ✅ Training permissions (conditional, with attribution) +* ✅ Attribution requirements (cite project + contributors) +* ✅ Specific AI agent policies (OpenAI, Anthropic, Google, Meta, etc.) +* ✅ Data protection (GDPR compliance, no PII training) +* ✅ Ethical use guidelines (discourage misinformation generation) +* ✅ BibTeX citation format for academic use + +===== humans.txt (Attribution) + +* ✅ Team members (@Hyperpolymath, Claude AI) +* ✅ Inspirations & acknowledgments +* ✅ Project info (name, repo, status, purpose) +* ✅ Technical details (backend, frontend, database, API) +* ✅ Governance model (TPCF Perimeter 3) +* ✅ Contributor credits (P1/P2/P3 lists) +* ✅ Philosophy & values + +*Compliance Level:* ✅ *Exceeds* (Comprehensive, RFC-compliant, +AI-aware) + +''''' + +==== 12. Build System - Justfile ✅ + +*Requirement:* Task automation with clear recipes (just, make, etc.). + +*Implementation:* - ✅ *30+ recipes* covering: - Setup & installation +(`+install+`, `+setup+`, `+db-reset+`) - Development (`+dev+`, +`+dev-iex+`, `+db-start/stop/restart+`) - Code quality (`+check+`, +`+format+`, `+lint+`, `+dialyzer+`, `+security-scan+`) - Testing +(`+test+`, `+test-coverage+`, `+test-watch+`) - GraphQL (`+graphiql+`, +`+graphql-example+`) - Database (`+arango-ui+`, `+arango-query+`, +`+export-investigation+`) - Documentation (`+docs+`, `+docs-open+`, +`+docs-check+`) - Security (`+secret+`, `+security-txt-check+`, +`+security-docs-check+`) - *RSR Compliance* (`+rsr-check+` with scoring) +- Build & release (`+build+`, `+build-assets+`, `+clean+`) - Utility +(`+stats+`, `+health+`, `+version+`, `+help+`, `+quick-start+`) + +*Example RSR Check:* + +[source,bash] +---- +just rsr-check +# 🔍 Checking RSR Framework compliance... +# ✅ LICENSE.txt (dual MIT + Palimpsest v0.8) +# ✅ SECURITY.md +# ... +# 📊 RSR Compliance Score: 11/12 +# 🥈 RSR Silver: 10-11/12 +---- + +*Compliance Level:* ✅ *Exceeds* (Extensive automation, +self-verification) + +''''' + +==== 13. CI/CD Configuration ✅ + +*Requirement:* Automated testing, linting, and deployment pipeline. + +*Implementation:* - ✅ *.gitlab-ci.yml* with 5 stages: setup, lint, +test, security, deploy - ✅ *Setup:* Dependency caching, asset +compilation - ✅ *Lint:* Format check (mix format –check-formatted), +Credo linting - ✅ *Test:* ExUnit tests with coverage reporting +(Cobertura) - ✅ *Security:* Dependency audit (mix deps.audit), OWASP +check (future) - ✅ *RSR Compliance Check:* Automated 12-point checklist +in CI - ✅ *Deploy:* Staging & production (manual, Phase 2+) - ✅ +*Scheduled:* Weekly security scans + +*CI Artifacts:* - JUnit XML test reports - Cobertura coverage reports - +Security audit results + +*Compliance Level:* ✅ *Full* (Automated RSR verification in CI) + +''''' + +==== 14. Test Suite ⚠️ + +*Requirement:* 80%+ test coverage, unit + integration tests. + +*Current Status:* - ⚠️ *Seeds only:* `+priv/repo/seeds.exs+` provides +data fixtures - ❌ *No ExUnit tests yet* (planned for Month 2) - ❌ *No +coverage measurement* (need excoveralls) - ❌ *No integration tests* +(GraphQL, ArangoDB) + +*Planned (Phase 1 Month 2):* - ✅ Unit tests for all contexts (Claims, +Evidence, Relationships, Navigation) - ✅ GraphQL integration tests (all +queries/mutations) - ✅ Property-based tests (StreamData for PROMPT +scoring) - ✅ Coverage: Target 80% (ExCoveralls) + +*Compliance Level:* ⚠️ *In Progress* (Architecture complete, tests +pending) + +''''' + +==== 15. TPCF Perimeter Documentation ✅ + +*Requirement:* Document Tri-Perimeter Contribution Framework level. + +*Implementation:* - ✅ *Current Perimeter: P3 (Community Sandbox)* - ✅ +Documented in CONTRIBUTING.md (Perimeter access table) - ✅ Documented +in MAINTAINERS.md (Governance model) - ✅ Documented in humans.txt +(Contributor lists) - ✅ Advancement criteria (3+ merged PRs for P3→P2) +- ✅ Responsibilities per perimeter (Must/May/Must Not) + +*Perimeter Definitions:* - *P1 (Core):* @Hyperpolymath - Full access, +deploy keys, RFC decisions - *P2 (Contributors):* (None yet) - Merge +rights, mentor role, contributor calls - *P3 (Community):* Everyone else +- Fork, PR, discuss + +*Compliance Level:* ✅ *Full* (Transparent, graduated trust model) + +''''' + +=== RSR Compliance Score + +==== Summary Table + +[width="100%",cols="44%,30%,26%",options="header",] +|=== +|Criterion |Status |Notes +|1. Type Safety |✅ Full |Elixir typespecs, Ecto schemas, GraphQL + +|2. Memory Safety |✅ Full |BEAM VM guarantees (GC, no manual alloc) + +|3. Offline-First |⚠️ Partial |Architectural trade-off, CRDTs planned +Phase 3 + +|4. README |✅ Exceeds |3,500 words, comprehensive + +|5. LICENSE |✅ Exceeds |Dual MIT + Palimpsest v0.8 + +|6. SECURITY.md |✅ Exceeds |10+ security dimensions, GDPR + +|7. CONTRIBUTING.md |✅ Exceeds |TPCF P3, emotional safety + +|8. CODE_OF_CONDUCT.md |✅ Exceeds |CCCP, reversibility + +|9. MAINTAINERS.md |✅ Exceeds |Emotional temperature metrics + +|10. CHANGELOG.md |✅ Full |SemVer, Keep a Changelog + +|11. .well-known/ |✅ Exceeds |security.txt (RFC 9116), ai.txt, +humans.txt + +|12. Justfile |✅ Exceeds |30+ recipes, RSR self-check + +|13. CI/CD |✅ Full |GitLab CI with RSR compliance stage + +|14. Test Suite |⚠️ In Progress |Seeds only, ExUnit planned Month 2 + +|15. TPCF Docs |✅ Full |P3 perimeter documented +|=== + +*Score:* *11 Full + 2 Partial* = *11/12* for Silver (test suite pending) + +==== Compliance Level + +* 🥇 *Gold:* 12/12 + 80% test coverage + production deployment +* 🥈 *Silver:* 10-11/12 (Current) +* 🎯 *Bronze:* 8-9/12 + +*Current Level:* 🥈 *SILVER* + +*Path to Gold:* 1. Implement comprehensive test suite (Month 2) 2. +Achieve 80%+ coverage 3. Production deployment (Month 11) 4. CRDTs for +offline-first (Month 13-14, Phase 3) + +''''' + +=== Elixir/Phoenix RSR Adaptations + +==== Differences from Rust Rhodium Minimal + +[width="100%",cols="19%,25%,39%,17%",options="header",] +|=== +|Criterion |Rust (Rhodium) |Elixir (Evidence Graph) |Rationale +|*Type Safety* |Compile-time ownership |Typespecs + pattern matching +|Both prevent runtime errors, different approaches + +|*Memory Safety* |Zero unsafe blocks |BEAM VM GC |Rust: manual, Elixir: +automatic (both safe) + +|*Offline-First* |No network calls |Requires database |Trade-off: +Collaboration > offline (journalism context) + +|*Dependencies* |Zero (100 LOC) |15+ (Phoenix, Absinthe, Arangox) |Rust: +minimal, Elixir: ecosystem leverage + +|*Build System* |cargo |mix + Justfile |cargo ≈ mix, Justfile adds task +automation +|=== + +==== Justification for Deviations + +*Offline-First (⚠️ Partial):* - *Rhodium Minimal:* 100 lines, no +dependencies, works air-gapped - *Evidence Graph:* Multi-user graph +database, real-time collaboration - *Decision:* Acceptable trade-off for +investigative journalism use case - *Future:* CRDTs (Phase 3) enable +offline-first eventually + +*Dependencies (Accepted):* - *Rhodium Minimal:* Zero dependencies, +self-contained - *Evidence Graph:* Phoenix framework, ArangoDB, GraphQL +- *Decision:* Leverage mature ecosystem over reinventing wheels - +*Security:* All dependencies audited (`+mix deps.audit+` in CI) + +''''' + +=== Recommendations + +==== Immediate (Month 2) + +[arabic] +. *Test Suite:* Write ExUnit tests for all contexts +. *Coverage:* Add excoveralls, target 80% +. *Documentation:* Add inline @doc and @spec to all functions + +==== Short-Term (Month 3-6) + +[arabic, start=4] +. *User Testing:* 25 NUJ journalists (Month 3 decision point) +. *Security Audit:* External penetration test before Phase 2 +. *Dependency Updates:* Automated Dependabot integration + +==== Long-Term (Phase 2-3) + +[arabic, start=7] +. *CRDTs:* Offline-first state for full RSR Gold compliance +. *Production Deployment:* Hetzner Cloud, ArangoDB Oasis +. *RSR Gold Verification:* Re-assess after Phase 3 + +''''' + +=== Conclusion + +*Evidence Graph achieves RSR Silver Level compliance*, demonstrating +commitment to: - Emotional safety (Palimpsest License, CCCP Code of +Conduct) - Security (RFC 9116 security.txt, 10+ dimensions) - +Transparency (TPCF governance, public metrics) - Quality (Justfile +automation, CI/CD) + +*Next Milestone:* RSR Gold (requires 80% test coverage + offline-first +CRDTs) + +*Assessment Date:* 2025-11-22 *Assessor:* Claude (AI Co-Creator) + +@Hyperpolymath (Project Lead) *Next Review:* 2025-12-22 (Monthly during +Phase 1) + +''''' + +_This compliance report is maintained in accordance with the Rhodium +Standard Repository framework, adapted for Elixir/Phoenix ecosystem +conventions._ diff --git a/RSR_COMPLIANCE.md b/RSR_COMPLIANCE.md deleted file mode 100644 index 5eb5b5b..0000000 --- a/RSR_COMPLIANCE.md +++ /dev/null @@ -1,474 +0,0 @@ -# RSR Framework Compliance Report - -**Project:** Evidence Graph for Investigative Journalism -**RSR Version:** Adapted for Elixir/Phoenix (from Rust-based Rhodium minimal example) -**Compliance Level:** 🥈 **SILVER** (11/12 criteria) -**Last Assessed:** 2025-11-22 - ---- - -## Executive Summary - -The Evidence Graph project achieves **RSR Silver Level** compliance, implementing 11 of 12 core Rhodium Standard Repository framework criteria. As an Elixir/Phoenix project, certain criteria (e.g., Rust-specific memory safety) are adapted to ecosystem-appropriate equivalents. - -**Strengths:** -- ✅ Complete documentation suite (LICENSE, SECURITY, CONTRIBUTING, CoC, MAINTAINERS, CHANGELOG) -- ✅ .well-known/ directory fully implemented (RFC 9116 security.txt, ai.txt, humans.txt) -- ✅ TPCF Perimeter 3 (Community Sandbox) governance model -- ✅ Palimpsest License v0.8 for emotional safety and reversibility -- ✅ Task automation (Justfile with 30+ recipes) -- ✅ CI/CD pipeline (.gitlab-ci.yml with RSR compliance checks) - -**Areas for Improvement:** -- ⚠️ Test coverage (seeds.exs only, need comprehensive ExUnit test suite - Month 2 goal) -- ⚠️ Offline-first consideration (currently requires network for ArangoDB/GraphQL - architectural trade-off) - ---- - -## RSR Framework Criteria Checklist - -### 1. Type Safety ✅ - -**Requirement:** Compile-time type guarantees to prevent runtime errors. - -**Implementation:** -- **Elixir Typespecs:** All public functions have `@spec` declarations -- **Ecto Schemas:** Database entities defined with strict types -- **GraphQL Schema:** Absinthe provides strong typing at API boundary -- **Pattern Matching:** Elixir's pattern matching enforces type safety at runtime - -**Examples:** -```elixir -@spec create_claim(map()) :: {:ok, Claim.t()} | {:error, Ecto.Changeset.t()} -def create_claim(attrs) do - %Claim{} - |> Claim.changeset(attrs) - |> insert_to_arango() -end -``` - -**Compliance Level:** ✅ **Full** (Elixir ecosystem equivalent of Rust compile-time checks) - ---- - -### 2. Memory Safety ✅ - -**Requirement:** No memory leaks, buffer overflows, or use-after-free errors. - -**Implementation:** -- **BEAM VM Garbage Collection:** Automatic memory management -- **No Manual Allocation:** Elixir abstracts memory management entirely -- **Process Isolation:** Each BEAM process has isolated heap -- **No Unsafe Blocks:** Elixir has no `unsafe` keyword (unlike Rust) - -**Trade-off:** -- Rust: Zero-cost abstractions, compile-time ownership -- Elixir: GC overhead, but guaranteed memory safety via BEAM VM - -**Compliance Level:** ✅ **Full** (Different approach, same safety guarantees) - ---- - -### 3. Offline-First ⚠️ - -**Requirement:** Works without network connectivity (air-gapped environments). - -**Current Status:** -- ❌ Requires ArangoDB (can be local, but needs running database) -- ❌ GraphQL API assumes client-server architecture -- ❌ No LocalStorage/IndexedDB fallback for browser (Phase 1 has no browser UI yet) - -**Future Improvements (Phase 2-3):** -- ✅ CRDTs for offline-first state (see ROADMAP.md Month 13-14) -- ✅ Service Workers for PWA offline mode (see ARCHITECTURE.md) -- ✅ Optional IPFS for peer-to-peer data sync - -**Rationale for Non-Compliance:** -- Investigative journalism use case assumes network availability for collaboration -- Multi-user graph database inherently requires coordination -- Acceptable trade-off for Phase 1 PoC - -**Compliance Level:** ⚠️ **Partial** (Architectural decision, not technical limitation) - ---- - -### 4. Documentation - README ✅ - -**Requirement:** Comprehensive README with quick start, examples, and project overview. - -**Implementation:** -- ✅ Quick start guide (5-step setup) -- ✅ GraphQL API examples (8 queries/mutations) -- ✅ Project structure diagram -- ✅ Development workflow (test, lint, deploy) -- ✅ UK Inflation 2023 dataset description -- ✅ Contact & support information - -**Length:** ~500 lines, ~3,500 words - -**Compliance Level:** ✅ **Exceeds** (Detailed, beginner-friendly) - ---- - -### 5. Documentation - LICENSE ✅ - -**Requirement:** Clear license terms, dual-license support. - -**Implementation:** -- ✅ **Dual License:** MIT + Palimpsest v0.8 -- ✅ MIT: Standard open source, permissive -- ✅ Palimpsest v0.8: Emotional safety provisions (reversibility, no coerced labor, emotional temperature disclosure) -- ✅ Contributor License Agreement (implicit via CONTRIBUTING.md) -- ✅ Third-party license acknowledgments - -**Unique Features:** -- 90-day contribution withdrawal window -- Emotional temperature metrics requirement -- TPCF perimeter framework integration - -**Compliance Level:** ✅ **Exceeds** (Dual license provides more protections than typical OSS) - ---- - -### 6. Documentation - SECURITY.md ✅ - -**Requirement:** Vulnerability disclosure policy, supported versions, contact info. - -**Implementation:** -- ✅ Supported versions table (0.1.x = active) -- ✅ Threat model (10+ security dimensions) -- ✅ Reporting procedures (email, GitHub Security Advisory) -- ✅ Response timeline (24h acknowledgment, 30-90 day fix) -- ✅ Severity levels (Critical/High/Medium/Low) -- ✅ Known limitations documented (no auth in Phase 1) -- ✅ GDPR compliance section (EU data sovereignty) -- ✅ Secure coding practices (OWASP Top 10) -- ✅ Incident response plan - -**Compliance Level:** ✅ **Exceeds** (Comprehensive, GDPR-aware) - ---- - -### 7. Documentation - CONTRIBUTING.md ✅ - -**Requirement:** Contribution guidelines, code standards, review process. - -**Implementation:** -- ✅ **TPCF Perimeter 3 Model:** Graduated trust (P1/P2/P3 access levels) -- ✅ Setup instructions (7-step guide) -- ✅ Branching strategy (GitHub Flow) -- ✅ Commit message format (Conventional Commits) -- ✅ Code standards (Elixir Style Guide) -- ✅ Testing requirements (80% coverage target) -- ✅ Review process (48h first review, 7 day merge) -- ✅ **Reversibility & Emotional Safety** (90-day withdrawal, no coercion) - -**Unique Features:** -- Explicit "no coerced labor" policy -- Emotional temperature monitoring -- Advancement path (P3 → P2 → P1) - -**Compliance Level:** ✅ **Exceeds** (Emotional safety-first approach) - ---- - -### 8. Documentation - CODE_OF_CONDUCT.md ✅ - -**Requirement:** Community standards, enforcement procedures. - -**Implementation:** -- ✅ Based on **CCCP (Community-Centric Code of Practice)** -- ✅ Expected behavior (kind, empathetic, collaborative) -- ✅ Unacceptable behavior (harassment, discrimination, manipulation) -- ✅ Reporting process (3 channels: email, GitHub PM, third-party) -- ✅ Enforcement (4 levels: warning, temp ban, perm ban, legal action) -- ✅ Emotional safety framework (reversibility, temperature metrics) -- ✅ Conflict resolution (mediation before escalation) - -**Unique Features:** -- Prioritizes contributor emotional well-being -- Integrates with Palimpsest License reversibility -- Transparent enforcement (decisions documented) - -**Compliance Level:** ✅ **Exceeds** (Beyond Contributor Covenant, adds emotional safety) - ---- - -### 9. Documentation - MAINTAINERS.md ✅ - -**Requirement:** Governance model, decision-making process, maintainer list. - -**Implementation:** -- ✅ Current maintainers (P1/P2/P3 lists) -- ✅ Decision-making framework (4 tiers: Critical/Major/Minor/Trivial) -- ✅ RFC process (for Tier 1 decisions) -- ✅ **Emotional Temperature Dashboard** (8 metrics tracked) -- ✅ Communication channels (public, semi-public, private) -- ✅ Conflict resolution procedures -- ✅ Succession planning (P2→P1 promotion path) -- ✅ Legal & financial governance (transparent accounting) - -**Unique Features:** -- Real-time emotional temperature metrics (churn, burnout risk, review times) -- Tri-Perimeter governance integrated -- Project Lead succession plan (Month 18) - -**Compliance Level:** ✅ **Exceeds** (Transparent, data-driven governance) - ---- - -### 10. Documentation - CHANGELOG.md ✅ - -**Requirement:** Version history following Keep a Changelog format, SemVer. - -**Implementation:** -- ✅ [Keep a Changelog](https://keepachangelog.com/) format -- ✅ Semantic Versioning (0.1.0 = Phase 1 PoC) -- ✅ Sections: Added/Changed/Deprecated/Removed/Fixed/Security -- ✅ Contributor credits (per version) -- ✅ Upgrade guide (from 0.0.0 → 0.1.0) -- ✅ Roadmap preview (next 3 releases) - -**Compliance Level:** ✅ **Full** (Standard-compliant) - ---- - -### 11. .well-known/ Directory ✅ - -**Requirement:** RFC 9116 security.txt, ai.txt, humans.txt for discoverability. - -**Implementation:** - -#### security.txt (RFC 9116) -- ✅ Contact field (email + GitHub Security Advisory) -- ✅ Expires field (2026-11-22) -- ✅ Preferred-Languages (en) -- ✅ Canonical URL -- ✅ Policy link (SECURITY.md) -- ✅ Acknowledgments link (humans.txt) - -#### ai.txt (AI Training Policy) -- ✅ Training permissions (conditional, with attribution) -- ✅ Attribution requirements (cite project + contributors) -- ✅ Specific AI agent policies (OpenAI, Anthropic, Google, Meta, etc.) -- ✅ Data protection (GDPR compliance, no PII training) -- ✅ Ethical use guidelines (discourage misinformation generation) -- ✅ BibTeX citation format for academic use - -#### humans.txt (Attribution) -- ✅ Team members (@Hyperpolymath, Claude AI) -- ✅ Inspirations & acknowledgments -- ✅ Project info (name, repo, status, purpose) -- ✅ Technical details (backend, frontend, database, API) -- ✅ Governance model (TPCF Perimeter 3) -- ✅ Contributor credits (P1/P2/P3 lists) -- ✅ Philosophy & values - -**Compliance Level:** ✅ **Exceeds** (Comprehensive, RFC-compliant, AI-aware) - ---- - -### 12. Build System - Justfile ✅ - -**Requirement:** Task automation with clear recipes (just, make, etc.). - -**Implementation:** -- ✅ **30+ recipes** covering: - - Setup & installation (`install`, `setup`, `db-reset`) - - Development (`dev`, `dev-iex`, `db-start/stop/restart`) - - Code quality (`check`, `format`, `lint`, `dialyzer`, `security-scan`) - - Testing (`test`, `test-coverage`, `test-watch`) - - GraphQL (`graphiql`, `graphql-example`) - - Database (`arango-ui`, `arango-query`, `export-investigation`) - - Documentation (`docs`, `docs-open`, `docs-check`) - - Security (`secret`, `security-txt-check`, `security-docs-check`) - - **RSR Compliance** (`rsr-check` with scoring) - - Build & release (`build`, `build-assets`, `clean`) - - Utility (`stats`, `health`, `version`, `help`, `quick-start`) - -**Example RSR Check:** -```bash -just rsr-check -# 🔍 Checking RSR Framework compliance... -# ✅ LICENSE.txt (dual MIT + Palimpsest v0.8) -# ✅ SECURITY.md -# ... -# 📊 RSR Compliance Score: 11/12 -# 🥈 RSR Silver: 10-11/12 -``` - -**Compliance Level:** ✅ **Exceeds** (Extensive automation, self-verification) - ---- - -### 13. CI/CD Configuration ✅ - -**Requirement:** Automated testing, linting, and deployment pipeline. - -**Implementation:** -- ✅ **.gitlab-ci.yml** with 5 stages: setup, lint, test, security, deploy -- ✅ **Setup:** Dependency caching, asset compilation -- ✅ **Lint:** Format check (mix format --check-formatted), Credo linting -- ✅ **Test:** ExUnit tests with coverage reporting (Cobertura) -- ✅ **Security:** Dependency audit (mix deps.audit), OWASP check (future) -- ✅ **RSR Compliance Check:** Automated 12-point checklist in CI -- ✅ **Deploy:** Staging & production (manual, Phase 2+) -- ✅ **Scheduled:** Weekly security scans - -**CI Artifacts:** -- JUnit XML test reports -- Cobertura coverage reports -- Security audit results - -**Compliance Level:** ✅ **Full** (Automated RSR verification in CI) - ---- - -### 14. Test Suite ⚠️ - -**Requirement:** 80%+ test coverage, unit + integration tests. - -**Current Status:** -- ⚠️ **Seeds only:** `priv/repo/seeds.exs` provides data fixtures -- ❌ **No ExUnit tests yet** (planned for Month 2) -- ❌ **No coverage measurement** (need excoveralls) -- ❌ **No integration tests** (GraphQL, ArangoDB) - -**Planned (Phase 1 Month 2):** -- ✅ Unit tests for all contexts (Claims, Evidence, Relationships, Navigation) -- ✅ GraphQL integration tests (all queries/mutations) -- ✅ Property-based tests (StreamData for PROMPT scoring) -- ✅ Coverage: Target 80% (ExCoveralls) - -**Compliance Level:** ⚠️ **In Progress** (Architecture complete, tests pending) - ---- - -### 15. TPCF Perimeter Documentation ✅ - -**Requirement:** Document Tri-Perimeter Contribution Framework level. - -**Implementation:** -- ✅ **Current Perimeter: P3 (Community Sandbox)** -- ✅ Documented in CONTRIBUTING.md (Perimeter access table) -- ✅ Documented in MAINTAINERS.md (Governance model) -- ✅ Documented in humans.txt (Contributor lists) -- ✅ Advancement criteria (3+ merged PRs for P3→P2) -- ✅ Responsibilities per perimeter (Must/May/Must Not) - -**Perimeter Definitions:** -- **P1 (Core):** @Hyperpolymath - Full access, deploy keys, RFC decisions -- **P2 (Contributors):** (None yet) - Merge rights, mentor role, contributor calls -- **P3 (Community):** Everyone else - Fork, PR, discuss - -**Compliance Level:** ✅ **Full** (Transparent, graduated trust model) - ---- - -## RSR Compliance Score - -### Summary Table - -| Criterion | Status | Notes | -|-----------|--------|-------| -| 1. Type Safety | ✅ Full | Elixir typespecs, Ecto schemas, GraphQL | -| 2. Memory Safety | ✅ Full | BEAM VM guarantees (GC, no manual alloc) | -| 3. Offline-First | ⚠️ Partial | Architectural trade-off, CRDTs planned Phase 3 | -| 4. README | ✅ Exceeds | 3,500 words, comprehensive | -| 5. LICENSE | ✅ Exceeds | Dual MIT + Palimpsest v0.8 | -| 6. SECURITY.md | ✅ Exceeds | 10+ security dimensions, GDPR | -| 7. CONTRIBUTING.md | ✅ Exceeds | TPCF P3, emotional safety | -| 8. CODE_OF_CONDUCT.md | ✅ Exceeds | CCCP, reversibility | -| 9. MAINTAINERS.md | ✅ Exceeds | Emotional temperature metrics | -| 10. CHANGELOG.md | ✅ Full | SemVer, Keep a Changelog | -| 11. .well-known/ | ✅ Exceeds | security.txt (RFC 9116), ai.txt, humans.txt | -| 12. Justfile | ✅ Exceeds | 30+ recipes, RSR self-check | -| 13. CI/CD | ✅ Full | GitLab CI with RSR compliance stage | -| 14. Test Suite | ⚠️ In Progress | Seeds only, ExUnit planned Month 2 | -| 15. TPCF Docs | ✅ Full | P3 perimeter documented | - -**Score:** **11 Full + 2 Partial** = **11/12** for Silver (test suite pending) - -### Compliance Level - -- 🥇 **Gold:** 12/12 + 80% test coverage + production deployment -- 🥈 **Silver:** 10-11/12 (Current) -- 🎯 **Bronze:** 8-9/12 - -**Current Level:** 🥈 **SILVER** - -**Path to Gold:** -1. Implement comprehensive test suite (Month 2) -2. Achieve 80%+ coverage -3. Production deployment (Month 11) -4. CRDTs for offline-first (Month 13-14, Phase 3) - ---- - -## Elixir/Phoenix RSR Adaptations - -### Differences from Rust Rhodium Minimal - -| Criterion | Rust (Rhodium) | Elixir (Evidence Graph) | Rationale | -|-----------|----------------|-------------------------|-----------| -| **Type Safety** | Compile-time ownership | Typespecs + pattern matching | Both prevent runtime errors, different approaches | -| **Memory Safety** | Zero unsafe blocks | BEAM VM GC | Rust: manual, Elixir: automatic (both safe) | -| **Offline-First** | No network calls | Requires database | Trade-off: Collaboration > offline (journalism context) | -| **Dependencies** | Zero (100 LOC) | 15+ (Phoenix, Absinthe, Arangox) | Rust: minimal, Elixir: ecosystem leverage | -| **Build System** | cargo | mix + Justfile | cargo ≈ mix, Justfile adds task automation | - -### Justification for Deviations - -**Offline-First (⚠️ Partial):** -- **Rhodium Minimal:** 100 lines, no dependencies, works air-gapped -- **Evidence Graph:** Multi-user graph database, real-time collaboration -- **Decision:** Acceptable trade-off for investigative journalism use case -- **Future:** CRDTs (Phase 3) enable offline-first eventually - -**Dependencies (Accepted):** -- **Rhodium Minimal:** Zero dependencies, self-contained -- **Evidence Graph:** Phoenix framework, ArangoDB, GraphQL -- **Decision:** Leverage mature ecosystem over reinventing wheels -- **Security:** All dependencies audited (`mix deps.audit` in CI) - ---- - -## Recommendations - -### Immediate (Month 2) - -1. **Test Suite:** Write ExUnit tests for all contexts -2. **Coverage:** Add excoveralls, target 80% -3. **Documentation:** Add inline @doc and @spec to all functions - -### Short-Term (Month 3-6) - -4. **User Testing:** 25 NUJ journalists (Month 3 decision point) -5. **Security Audit:** External penetration test before Phase 2 -6. **Dependency Updates:** Automated Dependabot integration - -### Long-Term (Phase 2-3) - -7. **CRDTs:** Offline-first state for full RSR Gold compliance -8. **Production Deployment:** Hetzner Cloud, ArangoDB Oasis -9. **RSR Gold Verification:** Re-assess after Phase 3 - ---- - -## Conclusion - -**Evidence Graph achieves RSR Silver Level compliance**, demonstrating commitment to: -- Emotional safety (Palimpsest License, CCCP Code of Conduct) -- Security (RFC 9116 security.txt, 10+ dimensions) -- Transparency (TPCF governance, public metrics) -- Quality (Justfile automation, CI/CD) - -**Next Milestone:** RSR Gold (requires 80% test coverage + offline-first CRDTs) - -**Assessment Date:** 2025-11-22 -**Assessor:** Claude (AI Co-Creator) + @Hyperpolymath (Project Lead) -**Next Review:** 2025-12-22 (Monthly during Phase 1) - ---- - -*This compliance report is maintained in accordance with the Rhodium Standard Repository framework, adapted for Elixir/Phoenix ecosystem conventions.* diff --git a/SECURITY.adoc b/SECURITY.adoc new file mode 100644 index 0000000..7997dbb --- /dev/null +++ b/SECURITY.adoc @@ -0,0 +1,336 @@ +== Security Policy + +=== Supported Versions + +Currently supported versions for security updates: + +[cols=",,",options="header",] +|=== +|Version |Support Status |EOL Date +|1.0.x (Phase 1 PoC) |Active Development |TBD (Month 6) +|0.1.x |Superseded by 1.0.0 |2026-02-21 +|=== + +=== Security Model + +==== Threat Model + +*In Scope:* - GraphQL API injection attacks - ArangoDB query injection +(AQL) - Authentication/authorization bypasses (Phase 2+) - XSS in web +interface - CSRF in state-changing operations - Sensitive data exposure +(interview subjects, confidential evidence) - Dependency vulnerabilities + +*Out of Scope (Current Phase):* - DDoS attacks (no production deployment +yet) - Physical security of infrastructure - Social engineering of users +- Client-side attacks on user browsers (beyond standard web security) + +==== Security Architecture + +*10+ Dimensions of Security (RSR Framework):* + +[arabic] +. *Type Safety*: Elixir compile-time checks, Ecto schemas, GraphQL type +system +. *Memory Safety*: BEAM VM memory isolation, no manual memory management +. *Input Validation*: Ecto changesets, GraphQL schema validation, AQL +parameterization +. *Authentication*: phx.gen.auth with BCrypt password hashing, magic +link login +. *Authorization*: Session-based auth with CSRF protection (RBAC planned +Phase 2) +. *Data Protection*: EU GDPR compliance, anonymized interview subjects +. *Transport Security*: HTTPS/TLS in production (Phase 2) +. *Audit Logging*: All mutations logged with user attribution +. *Dependency Scanning*: Automated via CI/CD (see Justfile +`+security-scan+`) +. *IPFS Provenance*: (Phase 2) Tamper-evident evidence storage with hash +verification + +==== EU GDPR Compliance + +This project handles sensitive investigative journalism data: + +* *Anonymization*: Interview subjects can be anonymized in database +* *Right to Erasure*: Evidence marked as `+deleted+` (soft delete) with +audit trail +* *Data Minimization*: Only essential metadata collected +* *Encryption*: Database-level encryption in production (ArangoDB +Enterprise) +* *Data Sovereignty*: EU hosting (Hetzner Cloud) + +=== Reporting a Vulnerability + +==== Where to Report + +*DO NOT* open public GitHub issues for security vulnerabilities. + +*Preferred Methods (in order):* + +[arabic] +. *Email*: security@evidencegraph.org (PGP key below) +. *GitHub Security Advisory*: +https://github.com/Hyperpolymath/bofig/security/advisories/new +. *GitLab Confidential Issue*: (if hosted on GitLab) + +==== What to Include + +Please provide: + +[arabic] +. *Description*: Detailed explanation of the vulnerability +. *Impact*: What an attacker could do (CVSS score if calculated) +. *Reproduction*: Step-by-step instructions to reproduce +. *Environment*: Version, OS, configuration +. *Proposed Fix*: If you have suggestions (optional) +. *Disclosure Timeline*: When you plan to publish (if at all) + +==== Response Timeline + +We commit to: + +* *24 hours*: Initial acknowledgment of report +* *7 days*: Preliminary assessment (severity, affected versions) +* *30 days*: Fix deployed or detailed remediation plan +* *90 days*: Public disclosure (coordinated with reporter) + +==== Severity Levels + +[width="100%",cols="31%,42%,27%",options="header",] +|=== +|Severity |Response Time |Example +|*Critical* |24 hours |Remote code execution, authentication bypass +|*High* |7 days |SQL/AQL injection, XSS, privilege escalation +|*Medium* |30 days |CSRF, information disclosure +|*Low* |90 days |Minor information leaks, rate limiting issues +|=== + +==== Bounty Program + +*Current Status*: No formal bug bounty (Phase 1 PoC) + +*Phase 2+*: Considering partnership with: - HackerOne / Bugcrowd - +EU-specific platforms (YesWeHack) + +*Acknowledgments*: Security researchers will be credited in: - +CHANGELOG.md - .well-known/humans.txt - Security Hall of Fame (if +program established) + +=== Known Security Limitations (v1.0.0) + +==== Current Gaps + +[arabic] +. *No Rate Limiting*: API can be abused +* *Mitigation*: Reverse proxy with rate limits (Nginx config provided) +* *Fix*: Phase 2 (Phoenix rate limiting plug) +. *No RBAC*: All authenticated users have equal access +* *Mitigation*: Authentication required for all routes +* *Fix*: Phase 2 (role-based access control) +. *Security Headers*: Basic headers via Phoenix, full CSP pending +* *Mitigation*: Nginx adds HSTS, X-Frame-Options in production +* *Fix*: Phase 2 (comprehensive CSP policy) + +==== Resolved Since v0.1.0 + +[arabic] +. *Authentication*: Implemented via phx.gen.auth (bcrypt, sessions, +CSRF) +. *Dependency Scanning*: Automated via `+mix deps.audit+` in CI and +Justfile +. *Input Validation*: Ecto changesets on all user input, parameterized +AQL queries + +==== Secure Coding Practices + +*We follow:* - OWASP Top 10 prevention guidelines - Elixir Security +Working Group recommendations - Phoenix Security Guide - ArangoDB +Security Best Practices + +*We avoid:* - `+String.to_existing_atom/1+` on user input (atom +exhaustion) - `+Code.eval_string/1+` on untrusted input - Storing +plaintext secrets (use environment variables) - SQL injection (we use +ArangoDB with parameterized queries) + +=== Security Testing + +==== Automated Scanning + +[source,bash] +---- +# Dependency vulnerability scan +mix deps.audit + +# Static analysis +mix credo --strict + +# Code quality +mix dialyzer + +# Security-focused linting +just security-scan # (see Justfile) +---- + +==== Manual Testing + +*GraphQL API Fuzzing:* + +[source,bash] +---- +# Test injection attacks +echo '{ claims(investigationId: "inv\"; DROP TABLE claims;--") { id } }' | \ + http POST :4000/api/graphql query=@- +---- + +*AQL Injection Testing:* + +[source,elixir] +---- +# Should be safe due to parameterized queries +EvidenceGraph.Claims.search_claims("'; DROP COLLECTION claims;--") +---- + +==== Penetration Testing + +*Phase 2+ Plan:* - External penetration test before public launch - Red +team exercise on production infrastructure - OWASP ZAP automated +scanning in CI/CD + +=== Security Updates + +==== Notification Channels + +Subscribe to security announcements: + +[arabic] +. *GitHub Watch*: Enable "`Custom`" notifications → "`Security alerts`" +. *RSS Feed*: +https://github.com/Hyperpolymath/bofig/security/advisories.atom +. *Mailing List*: security-announce@evidencegraph.org (low-traffic) + +==== Changelog + +All security fixes documented in CHANGELOG.md with: - CVE ID (if +assigned) - Severity level - Affected versions - Credit to reporter + +=== Dependency Security + +==== Current Dependencies (Phase 1) + +*Critical Dependencies:* - Phoenix 1.7.x (web framework) - Absinthe +1.7.x (GraphQL) - Arangox 0.5.x (database driver) - Ecto 3.11.x +(schemas/validation) + +*Audit Process:* 1. Weekly `+mix deps.audit+` check 2. Automated GitHub +Dependabot alerts 3. Manual review of security advisories 4. Test suite +run before all updates + +==== Update Policy + +* *Critical vulnerabilities*: Patch within 24 hours +* *High severity*: Update within 7 days +* *Medium/Low*: Include in next minor release + +=== Infrastructure Security (Phase 2) + +==== Production Deployment + +*Hetzner Cloud (EU):* - Firewall: Ports 80/443 only, no SSH from public +internet - SSH: Key-based auth only, fail2ban active - Nginx: Reverse +proxy with rate limiting, ModSecurity WAF - SSL: Let’s Encrypt with +HSTS, TLS 1.3 only + +*ArangoDB Oasis:* - Managed service, automatic security patches - +Network isolation (VPC peering) - Encryption at rest + in transit - +Daily backups with 30-day retention + +==== Secrets Management + +*Never commit:* - `+SECRET_KEY_BASE+` - `+ARANGO_PASSWORD+` - API keys, +tokens, credentials + +*Use:* - Environment variables (`+.env+` ignored by Git) - Phoenix +releases config (`+config/runtime.exs+`) - Hetzner Cloud metadata +service (production) - 1Password/Vault for team secrets (Phase 2) + +=== Incident Response + +==== Process + +[arabic] +. *Detection*: Automated alerts, user reports, security scans +. *Containment*: Isolate affected systems, revoke credentials +. *Eradication*: Apply patches, remove malicious code +. *Recovery*: Restore from backups, verify integrity +. *Lessons Learned*: Post-mortem, update security measures + +==== Communication + +* *Internal*: Maintainers via encrypted channel (Signal/Matrix) +* *Users*: Security advisory published within 24 hours of fix +* *Public*: CVE assignment for critical issues, blog post with details + +==== Data Breach Protocol + +If sensitive investigative data is compromised: + +[arabic] +. Notify affected journalists within 24 hours +. Report to data protection authority (GDPR Article 33) +. Publish incident report with timeline +. Offer mitigation (password reset, evidence re-upload) +. Forensic analysis to prevent recurrence + +=== Compliance & Audits + +==== Standards + +* *EU GDPR*: Article 25 (Privacy by Design), Article 32 (Security) +* *ISO 27001*: Information security management (future certification) +* *OWASP ASVS*: Application Security Verification Standard Level 2 +* *CWE Top 25*: Common Weakness Enumeration prevention + +==== Audit Trail + +All security-relevant events logged: - Authentication attempts - +Authorization failures - Data access (GDPR Article 30) - Configuration +changes - Security updates applied + +Logs retained for 90 days (GDPR minimum), 1 year for security incidents. + +=== Contact Information + +==== Security Team + +* *Lead*: @Hyperpolymath (GitHub) +* *Email*: security@evidencegraph.org +* *PGP Key*: [To be published] +* *Response Hours*: Mon-Fri 9am-5pm UTC (best-effort, volunteer project) + +==== Escalation + +For urgent issues (active exploitation): - *Phone*: [To be published for +Phase 2] - *Matrix*: [To be set up] + +=== Acknowledgments + +We thank the following security researchers: + +_(None yet - Hall of Fame will be added as reports are received)_ + +=== Resources + +* *OWASP Top 10*: https://owasp.org/www-project-top-ten/ +* *Elixir Security*: +https://elixir-lang.org/blog/2021/10/13/security-working-group/ +* *Phoenix Security Guide*: https://hexdocs.pm/phoenix/security.html +* *ArangoDB Security*: +https://www.arangodb.com/docs/stable/security.html + +''''' + +*Last Updated*: 2026-02-21 *Policy Version*: 1.1 (v1.0.0 Release) *Next +Review*: 2026-03-21 (monthly during Phase 1) + +_This security policy is maintained in accordance with RSR (Rhodium +Standard Repository) framework requirements and RFC 9116 +(security.txt)._ diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index dbd238d..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,335 +0,0 @@ -# Security Policy - -## Supported Versions - -Currently supported versions for security updates: - -| Version | Support Status | EOL Date | -|---------|---------------|----------| -| 1.0.x (Phase 1 PoC) | Active Development | TBD (Month 6) | -| 0.1.x | Superseded by 1.0.0 | 2026-02-21 | - -## Security Model - -### Threat Model - -**In Scope:** -- GraphQL API injection attacks -- ArangoDB query injection (AQL) -- Authentication/authorization bypasses (Phase 2+) -- XSS in web interface -- CSRF in state-changing operations -- Sensitive data exposure (interview subjects, confidential evidence) -- Dependency vulnerabilities - -**Out of Scope (Current Phase):** -- DDoS attacks (no production deployment yet) -- Physical security of infrastructure -- Social engineering of users -- Client-side attacks on user browsers (beyond standard web security) - -### Security Architecture - -**10+ Dimensions of Security (RSR Framework):** - -1. **Type Safety**: Elixir compile-time checks, Ecto schemas, GraphQL type system -2. **Memory Safety**: BEAM VM memory isolation, no manual memory management -3. **Input Validation**: Ecto changesets, GraphQL schema validation, AQL parameterization -4. **Authentication**: phx.gen.auth with BCrypt password hashing, magic link login -5. **Authorization**: Session-based auth with CSRF protection (RBAC planned Phase 2) -6. **Data Protection**: EU GDPR compliance, anonymized interview subjects -7. **Transport Security**: HTTPS/TLS in production (Phase 2) -8. **Audit Logging**: All mutations logged with user attribution -9. **Dependency Scanning**: Automated via CI/CD (see Justfile `security-scan`) -10. **IPFS Provenance**: (Phase 2) Tamper-evident evidence storage with hash verification - -### EU GDPR Compliance - -This project handles sensitive investigative journalism data: - -- **Anonymization**: Interview subjects can be anonymized in database -- **Right to Erasure**: Evidence marked as `deleted` (soft delete) with audit trail -- **Data Minimization**: Only essential metadata collected -- **Encryption**: Database-level encryption in production (ArangoDB Enterprise) -- **Data Sovereignty**: EU hosting (Hetzner Cloud) - -## Reporting a Vulnerability - -### Where to Report - -**DO NOT** open public GitHub issues for security vulnerabilities. - -**Preferred Methods (in order):** - -1. **Email**: security@evidencegraph.org (PGP key below) -2. **GitHub Security Advisory**: https://github.com/Hyperpolymath/bofig/security/advisories/new -3. **GitLab Confidential Issue**: (if hosted on GitLab) - -### What to Include - -Please provide: - -1. **Description**: Detailed explanation of the vulnerability -2. **Impact**: What an attacker could do (CVSS score if calculated) -3. **Reproduction**: Step-by-step instructions to reproduce -4. **Environment**: Version, OS, configuration -5. **Proposed Fix**: If you have suggestions (optional) -6. **Disclosure Timeline**: When you plan to publish (if at all) - -### Response Timeline - -We commit to: - -- **24 hours**: Initial acknowledgment of report -- **7 days**: Preliminary assessment (severity, affected versions) -- **30 days**: Fix deployed or detailed remediation plan -- **90 days**: Public disclosure (coordinated with reporter) - -### Severity Levels - -| Severity | Response Time | Example | -|----------|--------------|---------| -| **Critical** | 24 hours | Remote code execution, authentication bypass | -| **High** | 7 days | SQL/AQL injection, XSS, privilege escalation | -| **Medium** | 30 days | CSRF, information disclosure | -| **Low** | 90 days | Minor information leaks, rate limiting issues | - -### Bounty Program - -**Current Status**: No formal bug bounty (Phase 1 PoC) - -**Phase 2+**: Considering partnership with: -- HackerOne / Bugcrowd -- EU-specific platforms (YesWeHack) - -**Acknowledgments**: Security researchers will be credited in: -- CHANGELOG.md -- .well-known/humans.txt -- Security Hall of Fame (if program established) - -## Known Security Limitations (v1.0.0) - -### Current Gaps - -1. **No Rate Limiting**: API can be abused - - **Mitigation**: Reverse proxy with rate limits (Nginx config provided) - - **Fix**: Phase 2 (Phoenix rate limiting plug) - -2. **No RBAC**: All authenticated users have equal access - - **Mitigation**: Authentication required for all routes - - **Fix**: Phase 2 (role-based access control) - -3. **Security Headers**: Basic headers via Phoenix, full CSP pending - - **Mitigation**: Nginx adds HSTS, X-Frame-Options in production - - **Fix**: Phase 2 (comprehensive CSP policy) - -### Resolved Since v0.1.0 - -1. **Authentication**: Implemented via phx.gen.auth (bcrypt, sessions, CSRF) -2. **Dependency Scanning**: Automated via `mix deps.audit` in CI and Justfile -3. **Input Validation**: Ecto changesets on all user input, parameterized AQL queries - -### Secure Coding Practices - -**We follow:** -- OWASP Top 10 prevention guidelines -- Elixir Security Working Group recommendations -- Phoenix Security Guide -- ArangoDB Security Best Practices - -**We avoid:** -- `String.to_existing_atom/1` on user input (atom exhaustion) -- `Code.eval_string/1` on untrusted input -- Storing plaintext secrets (use environment variables) -- SQL injection (we use ArangoDB with parameterized queries) - -## Security Testing - -### Automated Scanning - -```bash -# Dependency vulnerability scan -mix deps.audit - -# Static analysis -mix credo --strict - -# Code quality -mix dialyzer - -# Security-focused linting -just security-scan # (see Justfile) -``` - -### Manual Testing - -**GraphQL API Fuzzing:** -```bash -# Test injection attacks -echo '{ claims(investigationId: "inv\"; DROP TABLE claims;--") { id } }' | \ - http POST :4000/api/graphql query=@- -``` - -**AQL Injection Testing:** -```elixir -# Should be safe due to parameterized queries -EvidenceGraph.Claims.search_claims("'; DROP COLLECTION claims;--") -``` - -### Penetration Testing - -**Phase 2+ Plan:** -- External penetration test before public launch -- Red team exercise on production infrastructure -- OWASP ZAP automated scanning in CI/CD - -## Security Updates - -### Notification Channels - -Subscribe to security announcements: - -1. **GitHub Watch**: Enable "Custom" notifications → "Security alerts" -2. **RSS Feed**: https://github.com/Hyperpolymath/bofig/security/advisories.atom -3. **Mailing List**: security-announce@evidencegraph.org (low-traffic) - -### Changelog - -All security fixes documented in CHANGELOG.md with: -- CVE ID (if assigned) -- Severity level -- Affected versions -- Credit to reporter - -## Dependency Security - -### Current Dependencies (Phase 1) - -**Critical Dependencies:** -- Phoenix 1.7.x (web framework) -- Absinthe 1.7.x (GraphQL) -- Arangox 0.5.x (database driver) -- Ecto 3.11.x (schemas/validation) - -**Audit Process:** -1. Weekly `mix deps.audit` check -2. Automated GitHub Dependabot alerts -3. Manual review of security advisories -4. Test suite run before all updates - -### Update Policy - -- **Critical vulnerabilities**: Patch within 24 hours -- **High severity**: Update within 7 days -- **Medium/Low**: Include in next minor release - -## Infrastructure Security (Phase 2) - -### Production Deployment - -**Hetzner Cloud (EU):** -- Firewall: Ports 80/443 only, no SSH from public internet -- SSH: Key-based auth only, fail2ban active -- Nginx: Reverse proxy with rate limiting, ModSecurity WAF -- SSL: Let's Encrypt with HSTS, TLS 1.3 only - -**ArangoDB Oasis:** -- Managed service, automatic security patches -- Network isolation (VPC peering) -- Encryption at rest + in transit -- Daily backups with 30-day retention - -### Secrets Management - -**Never commit:** -- `SECRET_KEY_BASE` -- `ARANGO_PASSWORD` -- API keys, tokens, credentials - -**Use:** -- Environment variables (`.env` ignored by Git) -- Phoenix releases config (`config/runtime.exs`) -- Hetzner Cloud metadata service (production) -- 1Password/Vault for team secrets (Phase 2) - -## Incident Response - -### Process - -1. **Detection**: Automated alerts, user reports, security scans -2. **Containment**: Isolate affected systems, revoke credentials -3. **Eradication**: Apply patches, remove malicious code -4. **Recovery**: Restore from backups, verify integrity -5. **Lessons Learned**: Post-mortem, update security measures - -### Communication - -- **Internal**: Maintainers via encrypted channel (Signal/Matrix) -- **Users**: Security advisory published within 24 hours of fix -- **Public**: CVE assignment for critical issues, blog post with details - -### Data Breach Protocol - -If sensitive investigative data is compromised: - -1. Notify affected journalists within 24 hours -2. Report to data protection authority (GDPR Article 33) -3. Publish incident report with timeline -4. Offer mitigation (password reset, evidence re-upload) -5. Forensic analysis to prevent recurrence - -## Compliance & Audits - -### Standards - -- **EU GDPR**: Article 25 (Privacy by Design), Article 32 (Security) -- **ISO 27001**: Information security management (future certification) -- **OWASP ASVS**: Application Security Verification Standard Level 2 -- **CWE Top 25**: Common Weakness Enumeration prevention - -### Audit Trail - -All security-relevant events logged: -- Authentication attempts -- Authorization failures -- Data access (GDPR Article 30) -- Configuration changes -- Security updates applied - -Logs retained for 90 days (GDPR minimum), 1 year for security incidents. - -## Contact Information - -### Security Team - -- **Lead**: @Hyperpolymath (GitHub) -- **Email**: security@evidencegraph.org -- **PGP Key**: [To be published] -- **Response Hours**: Mon-Fri 9am-5pm UTC (best-effort, volunteer project) - -### Escalation - -For urgent issues (active exploitation): -- **Phone**: [To be published for Phase 2] -- **Matrix**: [To be set up] - -## Acknowledgments - -We thank the following security researchers: - -*(None yet - Hall of Fame will be added as reports are received)* - -## Resources - -- **OWASP Top 10**: https://owasp.org/www-project-top-ten/ -- **Elixir Security**: https://elixir-lang.org/blog/2021/10/13/security-working-group/ -- **Phoenix Security Guide**: https://hexdocs.pm/phoenix/security.html -- **ArangoDB Security**: https://www.arangodb.com/docs/stable/security.html - ---- - -**Last Updated**: 2026-02-21 -**Policy Version**: 1.1 (v1.0.0 Release) -**Next Review**: 2026-03-21 (monthly during Phase 1) - -*This security policy is maintained in accordance with RSR (Rhodium Standard Repository) framework requirements and RFC 9116 (security.txt).* diff --git a/TEST-NEEDS.adoc b/TEST-NEEDS.adoc new file mode 100644 index 0000000..f63ac81 --- /dev/null +++ b/TEST-NEEDS.adoc @@ -0,0 +1,577 @@ +== TEST-NEEDS.md — Bofig Test Coverage Report + +=== SPDX-License-Identifier: CC-BY-SA-4.0 + +*Project:* bofig (Evidence Graph Visualization Library) *Date:* +2026-04-04 *Status:* CRG Grade C Achieved + +=== CRG Grade: C — ACHIEVED 2026-04-04 + +*Maintainer:* Jonathan D.A. Jewell +6759885+hyperpolymath@users.noreply.github.com + +''''' + +=== Executive Summary + +Comprehensive test suite added for bofig AffineScript/D3.js +visualization library. All core data structures, operations, and +security requirements now have full coverage across unit, +property-based, E2E, aspect (security), and benchmark tests. + +*CRG Grade:* C (meets all Code Review Grade C requirements) + +''''' + +=== Test Coverage Summary + +==== Test Categories + +[width="100%",cols="30%,20%,22%,28%",options="header",] +|=== +|Category |Tests |Status |Coverage +|*Unit* |25 test steps |PASS |Node/Link/GraphData creation, validation, +field access + +|*Property-Based (P2P)* |15 test steps |PASS |Invariant properties, +scaling, idempotence + +|*E2E (Lifecycle)* |18 test steps |PASS +|Create→Add→Link→Serialize→Deserialize + +|*Security (Aspect)* |27 test steps |PASS |XSS, injection, overflow, +malformed data + +|*Benchmarks* |26 benchmarks |PASS |Performance baseline for node/link +operations +|=== + +*Total Test Steps:* 85 *Total Benchmarks:* 26 *Pass Rate:* 100% (110/110 +tests) + +''''' + +=== What Was Added + +==== 1. Test Infrastructure + +*File:* `+deno.json+` - Deno task definitions (test, bench, fmt, lint) - +Standard library imports - Test runner configuration + +==== 2. Unit Tests (`+tests/unit/evidence_graph_test.ts+`) + +*Coverage:* Node creation, link creation, GraphData structure, +validation, boundary conditions + +*Test Groups:* - Node Creation (9 tests) - Valid claim/evidence node +creation - Boundary values (promptScore 0-100) - Rejection of empty +fields - Rejection of invalid promptScore ranges + +* Link Creation (7 tests) +** Standard relationship types (supports, contradicts, contextualizes) +** Custom relationship types +** Validation of source/target/relationship fields +** Rejection of empty fields +* GraphData Structure (4 tests) +** Empty graph creation +** Graph with nodes +** Graph with links +** Complex multi-node, multi-link graphs +* Null/Undefined Handling (4 tests) +** Rejection of null/undefined in required fields +** NaN promptScore handling +** Empty array handling +* Field Access (2 tests) +** Node field accessibility +** Link field accessibility + +==== 3. Property-Based Tests (`+tests/property/graph_properties_test.ts+`) + +*Coverage:* Invariant properties that must hold for all valid graphs + +*Property Groups:* - Node Count Invariant (2 tests) - Graph has exactly +N nodes for N inputs - Node count maintained after link addition + +* Link Reference Integrity (2 tests) +** All link sources reference existing nodes +** All link targets reference existing nodes +* promptScore Range Invariant (2 tests) +** All promptScores in [0.0, 100.0] +** promptScore is finite number, not NaN/Infinity +* nodeType Validity (2 tests) +** nodeType always non-empty string +** Common types (claim, evidence) are valid +* Relationship Semantics (2 tests) +** Relationship types are valid strings +** Source/target validation +* Graph Completeness (2 tests) +** All node IDs unique and non-empty +** All labels non-empty +* Scaling Properties (2 tests) +** Graph scales to 1000 nodes +** Graph scales to 10,000 links +* Idempotence (1 test) +** Creating same graph twice produces identical results + +==== 4. End-to-End Tests (`+tests/e2e/graph_lifecycle_test.ts+`) + +*Coverage:* Complete graph lifecycle workflows + +*Test Groups:* - Basic Lifecycle (3 tests) - Create empty graph - Add +single node - Add multiple nodes with verification + +* Link Management (4 tests) +** Add link between nodes +** Reject invalid source/target +** Allow multiple links to same node +* Serialization & Deserialization (5 tests) +** Serialize empty graph to JSON +** Serialize graph with nodes +** Serialize graph with links +** Deserialize back to EvidenceGraph +** Data integrity through round-trip +* Complete Workflow (2 tests) +** Full investigation workflow (5 nodes, 4 links) +** Graph update and re-serialization +* Query Operations (4 tests) +** Find node by ID +** Handle non-existent nodes +** Get links for node +** Filter nodes by type + +==== 5. Security Aspect Tests (`+tests/aspect/security_test.ts+`) + +*Coverage:* XSS prevention, injection, overflow, malformed data + +*Test Groups:* - XSS Prevention (4 tests) - Script tag sanitization - +Event handler escaping - HTML special character escaping - Sanitization +in node creation + +* Input Size Limits (5 tests) +** Reject oversized IDs (>255 chars) +** Reject oversized labels (>1000 chars) +** Reject oversized nodeType (>100 chars) +** Accept maximum valid lengths +** Handle 10K nodes + 100K links +* Prototype Pollution Prevention (3 tests) +** Protect against `+__proto__+` injection +** Protect against constructor manipulation +** Treat "`prototype`" as literal string +* Malformed Data Handling (5 tests) +** Reject NaN promptScore +** Reject Infinity promptScore +** Handle null bytes safely +** Reject negative promptScore +** Reject promptScore > 100 +* Type Coercion Attacks (3 tests) +** Handle numeric string coercion +** Handle boolean-like values +** Reject non-string types +* Unicode and Special Characters (4 tests) +** Unicode in labels (日本語, العربية) +** Emoji support +** RTL text (עברית, فارسی) +** Zero-width character handling +* Injection Prevention (2 tests) +** JSON injection prevention +** Regex injection prevention + +==== 6. Benchmarks (`+tests/bench/graph_bench.ts+`) + +*Coverage:* Performance baseline for core operations + +*Benchmark Categories:* + +* Node Creation (3 benchmarks) +** 100 nodes: 5.8 µs/iter +** 1,000 nodes: 60.5 µs/iter +** 10,000 nodes: 656.1 µs/iter +* Link Traversal (3 benchmarks) +** 100 nodes + 500 links: 26.6 µs/iter +** 1,000 nodes + 5,000 links: 287 µs/iter +** 5,000 nodes + 50,000 links: 3.9 ms/iter +* Serialization (3 benchmarks) +** 100-node graph: 29.6 µs/iter +** 1,000-node graph: 275.9 µs/iter +** 10,000-node graph: 2.9 ms/iter +* Deserialization (3 benchmarks) +** 100-node graph: 91.9 µs/iter +** 1,000-node graph: 982.7 µs/iter +** 10,000-node graph: 10.4 ms/iter +* Filter Operations (2 benchmarks) +** Filter 1,000 nodes by type: 76.5 µs/iter +** Filter 10,000 nodes by type: 890 µs/iter +* Search Operations (2 benchmarks) +** Find in 1,000-node graph: 67.2 µs/iter +** Find in 10,000-node graph: 793.6 µs/iter +* Aggregation (2 benchmarks) +** Average promptScore (1,000 nodes): 76.2 µs/iter +** Average promptScore (10,000 nodes): 878 µs/iter +* Deduplication (2 benchmarks) +** Deduplicate 1,000 nodes: 233.9 µs/iter +** Deduplicate 10,000 nodes: 3.1 ms/iter +* Link Lookup (2 benchmarks) +** Find by source (5,000 links): 299.5 µs/iter +** Find by source (50,000 links): 3.5 ms/iter +* Round-trip (2 benchmarks) +** Serialize/deserialize 1,000 nodes: 932.3 µs/iter +** Serialize/deserialize 5,000 nodes: 4.8 ms/iter +* Bulk Operations (2 benchmarks) +** Create and serialize 10K nodes: 3.2 ms/iter +** Create 10K nodes + 100K links: 8.8 ms/iter + +''''' + +=== CRG Grade C Requirements Met + +==== ✓ Unit Tests + +* *Requirement:* Node creation with correct fields (id, label, nodeType, +promptScore) +* *Coverage:* 9 dedicated unit tests covering all node creation +scenarios +* *Status:* PASS + +==== ✓ Smoke Tests + +* *Requirement:* Basic graph creation and operation verification +* *Coverage:* E2E lifecycle tests (18 steps) verify complete workflows +* *Status:* PASS + +==== ✓ Build Tests + +* *Requirement:* Project compiles and runs without errors +* *Coverage:* `+deno test --allow-all tests/+` runs 25 test suites with +0 failures +* *Status:* PASS + +==== ✓ P2P (Property-Based) Tests + +* *Requirement:* For any set of N nodes, graph has exactly N nodes; +Links reference existing nodes; promptScore in [0,100]; nodeType is +valid +* *Coverage:* 8 test suites (15 test steps) validate invariant +properties +* *Status:* PASS +** Node count invariant: 2 tests +** Link reference integrity: 2 tests +** promptScore range: 2 tests +** nodeType validity: 2 tests +** Scaling properties: 2 tests +** Idempotence: 1 test + +==== ✓ E2E (End-to-End) Tests + +* *Requirement:* Graph initialization → render → update data → +re-render; Create → add nodes → add links → serialize → deserialize +* *Coverage:* 5 test groups (18 test steps) covering complete lifecycle +* *Status:* PASS +** Lifecycle: 3 tests +** Link management: 4 tests +** Serialization/deserialization: 5 tests +** Workflow completion: 2 tests +** Query operations: 4 tests + +==== ✓ Reflexive Tests + +* *Requirement:* Self-describing tests that validate graph against its +own structure +* *Coverage:* Query operations (4 tests) verify graph contents +* *Status:* PASS +** Node lookup by ID +** Link enumeration +** Type-based filtering +** Count validation + +==== ✓ Contract Tests + +* *Requirement:* Input validation, error handling, field contracts +* *Coverage:* Unit tests (25 steps) enforce all contracts +* *Status:* PASS +** Field presence and type validation +** Range validation for promptScore +** Enum validation for nodeType/relationship +** Empty field rejection + +==== ✓ Aspect Tests (Security) + +* *Requirement:* XSS prevention, injection prevention, oversized input +handling, malformed data +* *Coverage:* 7 security test groups (27 test steps) +* *Status:* PASS +** XSS injection: 4 tests +** Input size limits: 5 tests +** Prototype pollution: 3 tests +** Malformed data: 5 tests +** Type coercion: 3 tests +** Unicode/special chars: 4 tests +** Injection: 2 tests + +==== ✓ Benchmarks Baselined + +* *Requirement:* Performance benchmarks for core operations +* *Coverage:* 26 benchmarks covering all operation categories +* *Status:* PASS +** Creation: 3 benchmarks (5.8 µs - 656.1 µs) +** Traversal: 3 benchmarks (26.6 µs - 3.9 ms) +** Serialization: 3 benchmarks (29.6 µs - 2.9 ms) +** Deserialization: 3 benchmarks (91.9 µs - 10.4 ms) +** Filtering: 2 benchmarks (76.5 µs - 890 µs) +** Search: 2 benchmarks (67.2 µs - 793.6 µs) +** Aggregation: 2 benchmarks (76.2 µs - 878 µs) +** Deduplication: 2 benchmarks (233.9 µs - 3.1 ms) +** Link lookup: 2 benchmarks (299.5 µs - 3.5 ms) +** Round-trip: 2 benchmarks (932.3 µs - 4.8 ms) +** Bulk operations: 2 benchmarks (3.2 ms - 8.8 ms) + +''''' + +=== Files Added + +[width="100%",cols="29%,40%,31%",options="header",] +|=== +|File |Purpose |Lines +|`+deno.json+` |Test infrastructure config |17 +|`+tests/unit/evidence_graph_test.ts+` |Unit tests |307 +|`+tests/property/graph_properties_test.ts+` |Property-based tests |286 +|`+tests/e2e/graph_lifecycle_test.ts+` |End-to-end tests |389 +|`+tests/aspect/security_test.ts+` |Security aspect tests |408 +|`+tests/bench/graph_bench.ts+` |Performance benchmarks |223 +|`+TEST-NEEDS.md+` |This report |- +|=== + +*Total New Test Code:* 2,014 lines (including headers and documentation) + +''''' + +=== Running Tests + +==== Run All Tests + +[source,bash] +---- +deno test --allow-all tests/ +---- + +==== Run Test Categories + +[source,bash] +---- +# Unit tests only +deno test --allow-all tests/unit/ + +# Property tests only +deno test --allow-all tests/property/ + +# E2E tests only +deno test --allow-all tests/e2e/ + +# Security tests only +deno test --allow-all tests/aspect/ +---- + +==== Run Benchmarks + +[source,bash] +---- +deno bench --allow-all tests/bench/ +---- + +==== Format Tests + +[source,bash] +---- +deno fmt tests/ +---- + +==== Lint Tests + +[source,bash] +---- +deno lint tests/ +---- + +''''' + +=== Test Quality Metrics + +[cols=",,",options="header",] +|=== +|Metric |Value |Status +|*Total Tests* |85 test steps |✓ PASS +|*Pass Rate* |100% (85/85) |✓ PASS +|*Coverage Targets* |6/6 CRG C categories |✓ ALL MET +|*Benchmark Suites* |26 benchmarks |✓ PASS +|*Security Tests* |27 steps |✓ PASS +|*E2E Workflows* |5 categories |✓ PASS +|*Property Invariants* |8 categories |✓ PASS +|*Code Quality* |No unwrap(), no placeholders |✓ PASS +|=== + +''''' + +=== Key Test Features + +[arabic] +. *No External Dependencies* — Pure TypeScript tests using Deno std +library +. *No Stub/Placeholder Tests* — All tests implement real assertions +. *Comprehensive Error Handling* — Tests validate both success and +failure paths +. *Security-First* — Dedicated security aspect tests for XSS, injection, +overflow +. *Performance Baseline* — 26 benchmarks establish performance +expectations +. *Idempotent* — Tests can run in any order with consistent results +. *Isolated* — Each test is independent with no shared state +. *Well-Documented* — Test headers explain purpose and coverage +. *Deno-Native* — Uses Deno test runner, no Node.js or npm required +. *SPDX-Licensed* — All test files carry MPL-2.0 header + +''''' + +=== Assertions & Validation + +==== Unit Tests + +* `+createNode()+` validates: id, label, nodeType, promptScore range +* `+createLink()+` validates: source, target, relationship +* Field access verified on all data structures +* Boundary conditions tested (0, 100 for promptScore) + +==== Property Tests + +* Cardinality: graph.nodes.length === input count +* Referential integrity: all link endpoints exist +* Range invariants: promptScore ∈ [0, 100] +* Enum invariants: nodeType is string +* Scaling: tested to 10K nodes and 10K links +* Idempotence: f(f(x)) = f(x) + +==== E2E Tests + +* Lifecycle: empty → add nodes → add links → serialize → deserialize +* Atomicity: link addition fails if endpoints don’t exist +* Consistency: serialized data roundtrips correctly +* Query: node lookup, link enumeration, type filtering + +==== Security Tests + +* XSS: script tags escaped to `+<+` / `+>+` +* Size limits: enforced on id (255), label (1000), nodeType (100) +* Type safety: null/undefined/NaN rejected +* Injection: JSON strings don’t parse unexpectedly +* Unicode: emoji, RTL, zero-width chars handled safely + +==== Benchmarks + +* Operations timed at ns granularity (Deno.bench) +* Percentiles tracked (p75, p99, p995) +* Scaling validated: linear time for O(n) operations +* Baseline established for future regression testing + +''''' + +=== What Tests Cover (AffineScript Source Analysis) + +Tests model these AffineScript types and functions: + +[source,typescript] +---- +// From src/EvidenceGraph.res +type node = { + id: string, // ← tested: id validation, empty check + label: string, // ← tested: label validation, empty check + nodeType: string, // ← tested: string type, enum-like values + promptScore: float, // ← tested: range [0,100], NaN/Infinity rejection +} + +type link = { + source: string, // ← tested: must exist as node id + target: string, // ← tested: must exist as node id + relationship: string, // ← tested: enum values (supports, contradicts, contextualizes) +} + +type graphData = { + nodes: array, // ← tested: cardinality invariant + links: array, // ← tested: referential integrity +} + +// Functions tested indirectly +let getRelationshipColor: relationship => string // ← color mapping verified +let getNodeColor: (nodeType, promptScore) => string // ← color calculation verified +let make: (string, width?, height?) => t // ← container initialization +let initSVG: t => unit // ← SVG structure (via data validation) +let loadData: (t, investigationId) => promise // ← async workflow +let render: t => t // ← rendering verification +---- + +''''' + +=== What Tests Don’t Cover (Out of Scope) + +[arabic] +. *D3.js Integration* — Requires browser DOM/JSDOM (not available in +Deno) +. *SVG Rendering* — Requires graphical rendering (tested via structure) +. *Force Simulation* — D3 physics engine (not reimplemented) +. *GraphQL Queries* — External API (tested via data shape) +. *Fetch/HTTP* — Network layer (tested via contract) + +These are integration concerns tested in browser/E2E frameworks, not +unit tests. + +''''' + +=== Maintenance & Future Work + +==== To Maintain Test Suite + +* Run `+deno test --allow-all tests/+` before every commit +* Benchmark baselines should remain within 2x of established values +* New features should add corresponding unit + property + E2E tests +* Security aspect tests should be updated if new input validation rules +added + +==== To Extend Tests + +* Add contract tests if PROMPT scoring algorithm changes +* Add reflexive tests if graph structure becomes more complex +* Add performance tests for D3.js integration when available +* Add accessibility tests for DOM output + +==== For AffineScript Integration + +* If AffineScript compilation is available, tests can be run against +compiled JS +* Current tests model the API contract; compiled JS should pass them +unchanged +* Consider adding snapshot tests for SVG output once DOM rendering +tested + +''''' + +=== Conclusion + +*CRG Grade C: ACHIEVED* + +The bofig repository now has comprehensive test coverage meeting all +Code Review Grade C requirements: + +✓ Unit tests (node, link, GraphData validation) ✓ Smoke tests (basic +operations) ✓ Build tests (compilation and execution) ✓ P2P tests +(invariant properties, scaling, idempotence) ✓ E2E tests (complete +lifecycle workflows) ✓ Reflexive tests (graph self-validation) ✓ +Contract tests (API contracts and error handling) ✓ Aspect tests +(security: XSS, injection, overflow, malformed data) ✓ Benchmarks (26 +baselines for performance tracking) + +*85 test steps, 100% pass rate, 0 placeholders.* + +The test suite is maintainable, idempotent, and ready for continuous +integration. + +''''' + +*Generated:* 2026-04-04 *Author:* Jonathan D.A. Jewell +6759885+hyperpolymath@users.noreply.github.com *License:* MPL-2.0 diff --git a/TEST-NEEDS.md b/TEST-NEEDS.md deleted file mode 100644 index 4f6fbd5..0000000 --- a/TEST-NEEDS.md +++ /dev/null @@ -1,550 +0,0 @@ -# TEST-NEEDS.md — Bofig Test Coverage Report - -## SPDX-License-Identifier: CC-BY-SA-4.0 - -**Project:** bofig (Evidence Graph Visualization Library) -**Date:** 2026-04-04 -**Status:** CRG Grade C Achieved - -## CRG Grade: C — ACHIEVED 2026-04-04 -**Maintainer:** Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> - ---- - -## Executive Summary - -Comprehensive test suite added for bofig AffineScript/D3.js visualization library. All core data structures, operations, and security requirements now have full coverage across unit, property-based, E2E, aspect (security), and benchmark tests. - -**CRG Grade:** C (meets all Code Review Grade C requirements) - ---- - -## Test Coverage Summary - -### Test Categories - -| Category | Tests | Status | Coverage | -|----------|-------|--------|----------| -| **Unit** | 25 test steps | PASS | Node/Link/GraphData creation, validation, field access | -| **Property-Based (P2P)** | 15 test steps | PASS | Invariant properties, scaling, idempotence | -| **E2E (Lifecycle)** | 18 test steps | PASS | Create→Add→Link→Serialize→Deserialize | -| **Security (Aspect)** | 27 test steps | PASS | XSS, injection, overflow, malformed data | -| **Benchmarks** | 26 benchmarks | PASS | Performance baseline for node/link operations | - -**Total Test Steps:** 85 -**Total Benchmarks:** 26 -**Pass Rate:** 100% (110/110 tests) - ---- - -## What Was Added - -### 1. Test Infrastructure - -**File:** `deno.json` -- Deno task definitions (test, bench, fmt, lint) -- Standard library imports -- Test runner configuration - -### 2. Unit Tests (`tests/unit/evidence_graph_test.ts`) - -**Coverage:** Node creation, link creation, GraphData structure, validation, boundary conditions - -**Test Groups:** -- Node Creation (9 tests) - - Valid claim/evidence node creation - - Boundary values (promptScore 0-100) - - Rejection of empty fields - - Rejection of invalid promptScore ranges - -- Link Creation (7 tests) - - Standard relationship types (supports, contradicts, contextualizes) - - Custom relationship types - - Validation of source/target/relationship fields - - Rejection of empty fields - -- GraphData Structure (4 tests) - - Empty graph creation - - Graph with nodes - - Graph with links - - Complex multi-node, multi-link graphs - -- Null/Undefined Handling (4 tests) - - Rejection of null/undefined in required fields - - NaN promptScore handling - - Empty array handling - -- Field Access (2 tests) - - Node field accessibility - - Link field accessibility - -### 3. Property-Based Tests (`tests/property/graph_properties_test.ts`) - -**Coverage:** Invariant properties that must hold for all valid graphs - -**Property Groups:** -- Node Count Invariant (2 tests) - - Graph has exactly N nodes for N inputs - - Node count maintained after link addition - -- Link Reference Integrity (2 tests) - - All link sources reference existing nodes - - All link targets reference existing nodes - -- promptScore Range Invariant (2 tests) - - All promptScores in [0.0, 100.0] - - promptScore is finite number, not NaN/Infinity - -- nodeType Validity (2 tests) - - nodeType always non-empty string - - Common types (claim, evidence) are valid - -- Relationship Semantics (2 tests) - - Relationship types are valid strings - - Source/target validation - -- Graph Completeness (2 tests) - - All node IDs unique and non-empty - - All labels non-empty - -- Scaling Properties (2 tests) - - Graph scales to 1000 nodes - - Graph scales to 10,000 links - -- Idempotence (1 test) - - Creating same graph twice produces identical results - -### 4. End-to-End Tests (`tests/e2e/graph_lifecycle_test.ts`) - -**Coverage:** Complete graph lifecycle workflows - -**Test Groups:** -- Basic Lifecycle (3 tests) - - Create empty graph - - Add single node - - Add multiple nodes with verification - -- Link Management (4 tests) - - Add link between nodes - - Reject invalid source/target - - Allow multiple links to same node - -- Serialization & Deserialization (5 tests) - - Serialize empty graph to JSON - - Serialize graph with nodes - - Serialize graph with links - - Deserialize back to EvidenceGraph - - Data integrity through round-trip - -- Complete Workflow (2 tests) - - Full investigation workflow (5 nodes, 4 links) - - Graph update and re-serialization - -- Query Operations (4 tests) - - Find node by ID - - Handle non-existent nodes - - Get links for node - - Filter nodes by type - -### 5. Security Aspect Tests (`tests/aspect/security_test.ts`) - -**Coverage:** XSS prevention, injection, overflow, malformed data - -**Test Groups:** -- XSS Prevention (4 tests) - - Script tag sanitization - - Event handler escaping - - HTML special character escaping - - Sanitization in node creation - -- Input Size Limits (5 tests) - - Reject oversized IDs (>255 chars) - - Reject oversized labels (>1000 chars) - - Reject oversized nodeType (>100 chars) - - Accept maximum valid lengths - - Handle 10K nodes + 100K links - -- Prototype Pollution Prevention (3 tests) - - Protect against `__proto__` injection - - Protect against constructor manipulation - - Treat "prototype" as literal string - -- Malformed Data Handling (5 tests) - - Reject NaN promptScore - - Reject Infinity promptScore - - Handle null bytes safely - - Reject negative promptScore - - Reject promptScore > 100 - -- Type Coercion Attacks (3 tests) - - Handle numeric string coercion - - Handle boolean-like values - - Reject non-string types - -- Unicode and Special Characters (4 tests) - - Unicode in labels (日本語, العربية) - - Emoji support - - RTL text (עברית, فارسی) - - Zero-width character handling - -- Injection Prevention (2 tests) - - JSON injection prevention - - Regex injection prevention - -### 6. Benchmarks (`tests/bench/graph_bench.ts`) - -**Coverage:** Performance baseline for core operations - -**Benchmark Categories:** - -- Node Creation (3 benchmarks) - - 100 nodes: 5.8 µs/iter - - 1,000 nodes: 60.5 µs/iter - - 10,000 nodes: 656.1 µs/iter - -- Link Traversal (3 benchmarks) - - 100 nodes + 500 links: 26.6 µs/iter - - 1,000 nodes + 5,000 links: 287 µs/iter - - 5,000 nodes + 50,000 links: 3.9 ms/iter - -- Serialization (3 benchmarks) - - 100-node graph: 29.6 µs/iter - - 1,000-node graph: 275.9 µs/iter - - 10,000-node graph: 2.9 ms/iter - -- Deserialization (3 benchmarks) - - 100-node graph: 91.9 µs/iter - - 1,000-node graph: 982.7 µs/iter - - 10,000-node graph: 10.4 ms/iter - -- Filter Operations (2 benchmarks) - - Filter 1,000 nodes by type: 76.5 µs/iter - - Filter 10,000 nodes by type: 890 µs/iter - -- Search Operations (2 benchmarks) - - Find in 1,000-node graph: 67.2 µs/iter - - Find in 10,000-node graph: 793.6 µs/iter - -- Aggregation (2 benchmarks) - - Average promptScore (1,000 nodes): 76.2 µs/iter - - Average promptScore (10,000 nodes): 878 µs/iter - -- Deduplication (2 benchmarks) - - Deduplicate 1,000 nodes: 233.9 µs/iter - - Deduplicate 10,000 nodes: 3.1 ms/iter - -- Link Lookup (2 benchmarks) - - Find by source (5,000 links): 299.5 µs/iter - - Find by source (50,000 links): 3.5 ms/iter - -- Round-trip (2 benchmarks) - - Serialize/deserialize 1,000 nodes: 932.3 µs/iter - - Serialize/deserialize 5,000 nodes: 4.8 ms/iter - -- Bulk Operations (2 benchmarks) - - Create and serialize 10K nodes: 3.2 ms/iter - - Create 10K nodes + 100K links: 8.8 ms/iter - ---- - -## CRG Grade C Requirements Met - -### ✓ Unit Tests -- **Requirement:** Node creation with correct fields (id, label, nodeType, promptScore) -- **Coverage:** 9 dedicated unit tests covering all node creation scenarios -- **Status:** PASS - -### ✓ Smoke Tests -- **Requirement:** Basic graph creation and operation verification -- **Coverage:** E2E lifecycle tests (18 steps) verify complete workflows -- **Status:** PASS - -### ✓ Build Tests -- **Requirement:** Project compiles and runs without errors -- **Coverage:** `deno test --allow-all tests/` runs 25 test suites with 0 failures -- **Status:** PASS - -### ✓ P2P (Property-Based) Tests -- **Requirement:** For any set of N nodes, graph has exactly N nodes; Links reference existing nodes; promptScore in [0,100]; nodeType is valid -- **Coverage:** 8 test suites (15 test steps) validate invariant properties -- **Status:** PASS - - Node count invariant: 2 tests - - Link reference integrity: 2 tests - - promptScore range: 2 tests - - nodeType validity: 2 tests - - Scaling properties: 2 tests - - Idempotence: 1 test - -### ✓ E2E (End-to-End) Tests -- **Requirement:** Graph initialization → render → update data → re-render; Create → add nodes → add links → serialize → deserialize -- **Coverage:** 5 test groups (18 test steps) covering complete lifecycle -- **Status:** PASS - - Lifecycle: 3 tests - - Link management: 4 tests - - Serialization/deserialization: 5 tests - - Workflow completion: 2 tests - - Query operations: 4 tests - -### ✓ Reflexive Tests -- **Requirement:** Self-describing tests that validate graph against its own structure -- **Coverage:** Query operations (4 tests) verify graph contents -- **Status:** PASS - - Node lookup by ID - - Link enumeration - - Type-based filtering - - Count validation - -### ✓ Contract Tests -- **Requirement:** Input validation, error handling, field contracts -- **Coverage:** Unit tests (25 steps) enforce all contracts -- **Status:** PASS - - Field presence and type validation - - Range validation for promptScore - - Enum validation for nodeType/relationship - - Empty field rejection - -### ✓ Aspect Tests (Security) -- **Requirement:** XSS prevention, injection prevention, oversized input handling, malformed data -- **Coverage:** 7 security test groups (27 test steps) -- **Status:** PASS - - XSS injection: 4 tests - - Input size limits: 5 tests - - Prototype pollution: 3 tests - - Malformed data: 5 tests - - Type coercion: 3 tests - - Unicode/special chars: 4 tests - - Injection: 2 tests - -### ✓ Benchmarks Baselined -- **Requirement:** Performance benchmarks for core operations -- **Coverage:** 26 benchmarks covering all operation categories -- **Status:** PASS - - Creation: 3 benchmarks (5.8 µs - 656.1 µs) - - Traversal: 3 benchmarks (26.6 µs - 3.9 ms) - - Serialization: 3 benchmarks (29.6 µs - 2.9 ms) - - Deserialization: 3 benchmarks (91.9 µs - 10.4 ms) - - Filtering: 2 benchmarks (76.5 µs - 890 µs) - - Search: 2 benchmarks (67.2 µs - 793.6 µs) - - Aggregation: 2 benchmarks (76.2 µs - 878 µs) - - Deduplication: 2 benchmarks (233.9 µs - 3.1 ms) - - Link lookup: 2 benchmarks (299.5 µs - 3.5 ms) - - Round-trip: 2 benchmarks (932.3 µs - 4.8 ms) - - Bulk operations: 2 benchmarks (3.2 ms - 8.8 ms) - ---- - -## Files Added - -| File | Purpose | Lines | -|------|---------|-------| -| `deno.json` | Test infrastructure config | 17 | -| `tests/unit/evidence_graph_test.ts` | Unit tests | 307 | -| `tests/property/graph_properties_test.ts` | Property-based tests | 286 | -| `tests/e2e/graph_lifecycle_test.ts` | End-to-end tests | 389 | -| `tests/aspect/security_test.ts` | Security aspect tests | 408 | -| `tests/bench/graph_bench.ts` | Performance benchmarks | 223 | -| `TEST-NEEDS.md` | This report | - | - -**Total New Test Code:** 2,014 lines (including headers and documentation) - ---- - -## Running Tests - -### Run All Tests -```bash -deno test --allow-all tests/ -``` - -### Run Test Categories -```bash -# Unit tests only -deno test --allow-all tests/unit/ - -# Property tests only -deno test --allow-all tests/property/ - -# E2E tests only -deno test --allow-all tests/e2e/ - -# Security tests only -deno test --allow-all tests/aspect/ -``` - -### Run Benchmarks -```bash -deno bench --allow-all tests/bench/ -``` - -### Format Tests -```bash -deno fmt tests/ -``` - -### Lint Tests -```bash -deno lint tests/ -``` - ---- - -## Test Quality Metrics - -| Metric | Value | Status | -|--------|-------|--------| -| **Total Tests** | 85 test steps | ✓ PASS | -| **Pass Rate** | 100% (85/85) | ✓ PASS | -| **Coverage Targets** | 6/6 CRG C categories | ✓ ALL MET | -| **Benchmark Suites** | 26 benchmarks | ✓ PASS | -| **Security Tests** | 27 steps | ✓ PASS | -| **E2E Workflows** | 5 categories | ✓ PASS | -| **Property Invariants** | 8 categories | ✓ PASS | -| **Code Quality** | No unwrap(), no placeholders | ✓ PASS | - ---- - -## Key Test Features - -1. **No External Dependencies** — Pure TypeScript tests using Deno std library -2. **No Stub/Placeholder Tests** — All tests implement real assertions -3. **Comprehensive Error Handling** — Tests validate both success and failure paths -4. **Security-First** — Dedicated security aspect tests for XSS, injection, overflow -5. **Performance Baseline** — 26 benchmarks establish performance expectations -6. **Idempotent** — Tests can run in any order with consistent results -7. **Isolated** — Each test is independent with no shared state -8. **Well-Documented** — Test headers explain purpose and coverage -9. **Deno-Native** — Uses Deno test runner, no Node.js or npm required -10. **SPDX-Licensed** — All test files carry MPL-2.0 header - ---- - -## Assertions & Validation - -### Unit Tests -- `createNode()` validates: id, label, nodeType, promptScore range -- `createLink()` validates: source, target, relationship -- Field access verified on all data structures -- Boundary conditions tested (0, 100 for promptScore) - -### Property Tests -- Cardinality: graph.nodes.length === input count -- Referential integrity: all link endpoints exist -- Range invariants: promptScore ∈ [0, 100] -- Enum invariants: nodeType is string -- Scaling: tested to 10K nodes and 10K links -- Idempotence: f(f(x)) = f(x) - -### E2E Tests -- Lifecycle: empty → add nodes → add links → serialize → deserialize -- Atomicity: link addition fails if endpoints don't exist -- Consistency: serialized data roundtrips correctly -- Query: node lookup, link enumeration, type filtering - -### Security Tests -- XSS: script tags escaped to `<` / `>` -- Size limits: enforced on id (255), label (1000), nodeType (100) -- Type safety: null/undefined/NaN rejected -- Injection: JSON strings don't parse unexpectedly -- Unicode: emoji, RTL, zero-width chars handled safely - -### Benchmarks -- Operations timed at ns granularity (Deno.bench) -- Percentiles tracked (p75, p99, p995) -- Scaling validated: linear time for O(n) operations -- Baseline established for future regression testing - ---- - -## What Tests Cover (AffineScript Source Analysis) - -Tests model these AffineScript types and functions: - -```typescript -// From src/EvidenceGraph.res -type node = { - id: string, // ← tested: id validation, empty check - label: string, // ← tested: label validation, empty check - nodeType: string, // ← tested: string type, enum-like values - promptScore: float, // ← tested: range [0,100], NaN/Infinity rejection -} - -type link = { - source: string, // ← tested: must exist as node id - target: string, // ← tested: must exist as node id - relationship: string, // ← tested: enum values (supports, contradicts, contextualizes) -} - -type graphData = { - nodes: array, // ← tested: cardinality invariant - links: array, // ← tested: referential integrity -} - -// Functions tested indirectly -let getRelationshipColor: relationship => string // ← color mapping verified -let getNodeColor: (nodeType, promptScore) => string // ← color calculation verified -let make: (string, width?, height?) => t // ← container initialization -let initSVG: t => unit // ← SVG structure (via data validation) -let loadData: (t, investigationId) => promise // ← async workflow -let render: t => t // ← rendering verification -``` - ---- - -## What Tests Don't Cover (Out of Scope) - -1. **D3.js Integration** — Requires browser DOM/JSDOM (not available in Deno) -2. **SVG Rendering** — Requires graphical rendering (tested via structure) -3. **Force Simulation** — D3 physics engine (not reimplemented) -4. **GraphQL Queries** — External API (tested via data shape) -5. **Fetch/HTTP** — Network layer (tested via contract) - -These are integration concerns tested in browser/E2E frameworks, not unit tests. - ---- - -## Maintenance & Future Work - -### To Maintain Test Suite -- Run `deno test --allow-all tests/` before every commit -- Benchmark baselines should remain within 2x of established values -- New features should add corresponding unit + property + E2E tests -- Security aspect tests should be updated if new input validation rules added - -### To Extend Tests -- Add contract tests if PROMPT scoring algorithm changes -- Add reflexive tests if graph structure becomes more complex -- Add performance tests for D3.js integration when available -- Add accessibility tests for DOM output - -### For AffineScript Integration -- If AffineScript compilation is available, tests can be run against compiled JS -- Current tests model the API contract; compiled JS should pass them unchanged -- Consider adding snapshot tests for SVG output once DOM rendering tested - ---- - -## Conclusion - -**CRG Grade C: ACHIEVED** - -The bofig repository now has comprehensive test coverage meeting all Code Review Grade C requirements: - -✓ Unit tests (node, link, GraphData validation) -✓ Smoke tests (basic operations) -✓ Build tests (compilation and execution) -✓ P2P tests (invariant properties, scaling, idempotence) -✓ E2E tests (complete lifecycle workflows) -✓ Reflexive tests (graph self-validation) -✓ Contract tests (API contracts and error handling) -✓ Aspect tests (security: XSS, injection, overflow, malformed data) -✓ Benchmarks (26 baselines for performance tracking) - -**85 test steps, 100% pass rate, 0 placeholders.** - -The test suite is maintainable, idempotent, and ready for continuous integration. - ---- - -**Generated:** 2026-04-04 -**Author:** Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com> -**License:** MPL-2.0 diff --git a/TOPOLOGY.md b/TOPOLOGY.adoc similarity index 70% rename from TOPOLOGY.md rename to TOPOLOGY.adoc index 114dae6..219219c 100644 --- a/TOPOLOGY.md +++ b/TOPOLOGY.adoc @@ -1,11 +1,12 @@ -# SPDX-License-Identifier: CC-BY-SA-4.0 -# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) +== SPDX-License-Identifier: CC-BY-SA-4.0 -# TOPOLOGY.md - Evidence Graph (bofig) +== Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) j.d.a.jewell@open.ac.uk -## System Architecture +== TOPOLOGY.md - Evidence Graph (bofig) -``` +=== System Architecture + +.... +-------------------+ | Web Browser | | (D3.js + LiveView)| @@ -56,25 +57,38 @@ | (Tesla HTTP) | | Oban background | +------------------+ -``` +.... + +=== Component Overview + +[width="100%",cols="36%,35%,29%",options="header",] +|=== +|Component |Technology |Purpose +|Web UI |Phoenix LiveView + D3.js |Real-time investigation dashboard, +graph viz, PROMPT radar + +|GraphQL API |Absinthe 1.7 |Typed query/mutation interface for claims, +evidence, navigation + +|REST API |Phoenix Controllers |Zotero import/export, sync status + +|Auth |phx.gen.auth + bcrypt |Session-based user authentication -## Component Overview +|Graph DB |ArangoDB 3.11 (Arangox) |Claims, evidence, relationships, +navigation paths -| Component | Technology | Purpose | -|-----------|-----------|---------| -| Web UI | Phoenix LiveView + D3.js | Real-time investigation dashboard, graph viz, PROMPT radar | -| GraphQL API | Absinthe 1.7 | Typed query/mutation interface for claims, evidence, navigation | -| REST API | Phoenix Controllers | Zotero import/export, sync status | -| Auth | phx.gen.auth + bcrypt | Session-based user authentication | -| Graph DB | ArangoDB 3.11 (Arangox) | Claims, evidence, relationships, navigation paths | -| Relational DB | PostgreSQL (Ecto) | User accounts, tokens, Oban jobs | -| Background jobs | Oban 2.17 | Zotero incremental sync polling | -| Email | Swoosh | Magic link auth, password reset | -| HTTP client | Tesla + Mint | Zotero API communication | +|Relational DB |PostgreSQL (Ecto) |User accounts, tokens, Oban jobs -## Completion Dashboard +|Background jobs |Oban 2.17 |Zotero incremental sync polling -``` +|Email |Swoosh |Magic link auth, password reset + +|HTTP client |Tesla + Mint |Zotero API communication +|=== + +=== Completion Dashboard + +.... Phoenix Backend Core contexts ██████████ 100% Claims, Evidence, Relationships, Navigation, PromptScores GraphQL schema ██████████ 100% Queries + mutations for all entities @@ -120,36 +134,51 @@ RSR Compliance .well-known/ ██████████ 100% security.txt, ai.txt, humans.txt AI manifest ██████████ 100% 0-AI-MANIFEST.a2ml .machine_readable ██████████ 100% STATE.scm, META.scm, ECOSYSTEM.scm -``` - -## Key Dependencies - -| Dependency | Version | Purpose | -|-----------|---------|---------| -| Elixir | ~> 1.16 | Language runtime | -| Phoenix | ~> 1.8.3 | Web framework | -| Phoenix LiveView | ~> 1.1.19 | Real-time UI | -| Absinthe | ~> 1.7 | GraphQL | -| Arangox | ~> 0.7.0 | ArangoDB driver | -| Ecto SQL | ~> 3.11 | PostgreSQL (auth only) | -| Oban | ~> 2.17 | Background jobs | -| Tesla | ~> 1.8 | HTTP client | -| bcrypt_elixir | ~> 3.0 | Password hashing | -| Swoosh | ~> 1.4 | Email delivery | -| D3.js | v7 | Graph + radar visualisation | -| Tailwind CSS | ~> 0.2 | Styling | - -## File Structure Summary - -| Directory | Files | Description | -|-----------|-------|-------------| -| `lib/evidence_graph/` | 28 .ex | Core business logic (contexts, schemas, workers) | -| `lib/evidence_graph_web/` | 25 .ex | Web layer (router, controllers, LiveView, GraphQL) | -| `lib/evidence_graph_web/` | 7 .heex | HTML templates | -| `test/` | 17 .exs | ExUnit tests (257 tests) | -| `assets/js/` | 7 .js | LiveView client, D3.js hooks | -| `assets/css/` | 1 .css | Tailwind entry point | -| `config/` | 4 .exs | Environment configs | -| `docs/` | 8 files | Architecture, testing, integration docs | -| `deploy/` | 3 files | Nginx, systemd, env template | -| `priv/static/.well-known/` | 3 files | security.txt, ai.txt, humans.txt | +.... + +=== Key Dependencies + +[cols=",,",options="header",] +|=== +|Dependency |Version |Purpose +|Elixir |~> 1.16 |Language runtime +|Phoenix |~> 1.8.3 |Web framework +|Phoenix LiveView |~> 1.1.19 |Real-time UI +|Absinthe |~> 1.7 |GraphQL +|Arangox |~> 0.7.0 |ArangoDB driver +|Ecto SQL |~> 3.11 |PostgreSQL (auth only) +|Oban |~> 2.17 |Background jobs +|Tesla |~> 1.8 |HTTP client +|bcrypt_elixir |~> 3.0 |Password hashing +|Swoosh |~> 1.4 |Email delivery +|D3.js |v7 |Graph + radar visualisation +|Tailwind CSS |~> 0.2 |Styling +|=== + +=== File Structure Summary + +[width="100%",cols="37%,22%,41%",options="header",] +|=== +|Directory |Files |Description +|`+lib/evidence_graph/+` |28 .ex |Core business logic (contexts, +schemas, workers) + +|`+lib/evidence_graph_web/+` |25 .ex |Web layer (router, controllers, +LiveView, GraphQL) + +|`+lib/evidence_graph_web/+` |7 .heex |HTML templates + +|`+test/+` |17 .exs |ExUnit tests (257 tests) + +|`+assets/js/+` |7 .js |LiveView client, D3.js hooks + +|`+assets/css/+` |1 .css |Tailwind entry point + +|`+config/+` |4 .exs |Environment configs + +|`+docs/+` |8 files |Architecture, testing, integration docs + +|`+deploy/+` |3 files |Nginx, systemd, env template + +|`+priv/static/.well-known/+` |3 files |security.txt, ai.txt, humans.txt +|=== diff --git a/docs/EPSTEIN-FILES-WORK-PATHWAY.adoc b/docs/EPSTEIN-FILES-WORK-PATHWAY.adoc new file mode 100644 index 0000000..a117616 --- /dev/null +++ b/docs/EPSTEIN-FILES-WORK-PATHWAY.adoc @@ -0,0 +1,496 @@ +== Epstein Files — Complete Work Pathway + +== Tests & Benchmarks at Every Stage + +== + +== SPDX-License-Identifier: CC-BY-SA-4.0 + +== Author: Jonathan D.A. Jewell + +== Created: 2026-03-11 + +== + +== This document maps every implementation step needed to process the + +== Epstein files (3M+ files, 218GB, 12 data sets, 23K entities) through + +== the Docudactyl → Lithoglyph → Bofig pipeline, with test criteria + +== and performance benchmarks for each milestone. + +''''' + +=== Dataset Characteristics + +[width="100%",cols="62%,38%",options="header",] +|=== +|Attribute |Value +|Total files |~3,200,000 + +|Total size |~218 GB + +|Data sets |12 (flight logs, court filings, depositions, financial +records, photos, communications, …) + +|Named entities (est.) |23,000+ unique persons, orgs, locations + +|Primary format |Scanned PDF (96 DPI, many poor quality) + +|Secondary formats |TIFF, JPEG, DOCX, XLS, email (EML/PST) + +|Redaction style |Overlay-only (text stream often intact) + +|Financial transactions (est.) |16,000+ + +|Languages |English (primary), French, some Spanish + +|Time span |1990s–2024 +|=== + +''''' + +=== Phase 1: Docudactyl Extraction Pipeline (Weeks 1–6) + +==== Step 1.1: Core OCR + Text Extraction (DONE — existing stages 0-8) + +Already implemented in `+stages.zig+`: - Language detection, +readability, keywords, citations - OCR confidence, perceptual hash, TOC +extraction - Multi-language OCR, subtitle extraction + +*Tests:* - [x] Unit test: each stage function with known input produces +expected Cap’n Proto output - [x] Integration test: 10-document +mini-corpus end-to-end - [ ] Benchmark: single-node throughput for +scanned PDFs (target: 2 docs/sec on 8-core) + +==== Step 1.2: Redaction Detection (DONE — bit 20, `+stageRedactionDetect+`) + +Scans Poppler annotations for Type 12 (POPPLER_ANNOT_REDACT), checks if +text is extractable under overlay-only redactions. + +*Tests:* - [ ] T-RED-1: Synthetic PDF with 5 /Redact annotations → +count=5, status="`redacted`" - [ ] T-RED-2: PDF with black fill +rectangles but no /Redact annots → status="`clean`" (future: heuristic +upgrade) - [ ] T-RED-3: PDF with overlay redaction + recoverable text → +recoverable_count > 0 - [ ] T-RED-4: Non-PDF input (JPEG) → +status="`not_applicable`" - [ ] T-RED-5: Corrupt/unreadable PDF → +status="`error`" + +*Benchmarks:* - [ ] B-RED-1: 1000 PDFs (mixed redacted/clean) — target: +<500ms per document - [ ] B-RED-2: Memory usage during annotation scan — +target: <50MB peak per document + +==== Step 1.3: Financial Entity Extraction (DONE — bit 21, `+stageFinancialExtract+`) + +Pattern-based detection of currency symbols ($, £, €), ISO codes (USD, +GBP, EUR, CHF, JPY, CAD), account-like digit sequences (8-20 digits). + +*Tests:* - [ ] T-FIN-1: Text "`$1,234.56 paid to account 12345678`" → +amounts=1, accounts=1 - [ ] T-FIN-2: Text "`USD 50,000 transferred`" → +amounts=1 - [ ] T-FIN-3: Text "`£2.3 million to HSBC account +1234-5678-9012`" → amounts=1, accounts=1 - [ ] T-FIN-4: Phone numbers +should NOT match as accounts (7-digit filter) - [ ] T-FIN-5: Empty text +→ status="`none_found`", amounts=0, accounts=0 - [ ] T-FIN-6: Mixed +currencies in single document → correct total count + +*Benchmarks:* - [ ] B-FIN-1: 10MB text document scan — target: <200ms - +[ ] B-FIN-2: Accuracy on annotated Epstein financial records sample (50 +docs) — target: >80% recall + +==== Step 1.4: Legal NER (DONE — bit 22, `+stageLegalNer+`) + +Pattern-based detection of case citations ("`v.`"), docket numbers +("`No.`", "`Case`"), statute references ("`U.S.C.`", "`§`"). + +*Tests:* - [ ] T-LEG-1: "`Doe v. Epstein`" → case_citations=1 - [ ] +T-LEG-2: "`No. 08-cv-1234`" → docket_refs=1 - [ ] T-LEG-3: "`18 U.S.C. § +1591`" → statute_refs=1 (both U.S.C. and § counted) - [ ] T-LEG-4: +"`vs.`" variant → case_citations=1 - [ ] T-LEG-5: Real Epstein court +filing excerpt → realistic counts - [ ] T-LEG-6: Non-legal document +(flight log) → all counts = 0 + +*Benchmarks:* - [ ] B-LEG-1: 5MB legal document — target: <150ms - [ ] +B-LEG-2: Precision on annotated legal corpus (100 docs) — target: >75% + +==== Step 1.5: Speaker Identification (bit 23, ML dispatch) + +ML-based speaker diarization via ONNX Runtime. Dispatches to stage_id=5 +(speaker_id.onnx model). + +*Tests:* - [ ] T-SPK-1: With ML handle + model → status="`ok`", +speaker_count > 0 - [ ] T-SPK-2: Without ML handle → +status="`not_available`" - [ ] T-SPK-3: Non-audio input → graceful +fallback - [ ] T-SPK-4: Deposition audio with 2 speakers → +speaker_count=2 + +*Benchmarks:* - [ ] B-SPK-1: 30-minute deposition audio — target: <60s +inference - [ ] B-SPK-2: Memory usage during diarization — target: <2GB + +==== Step 1.6: STAGE_INVESTIGATIVE Preset Validation + +The `+STAGE_INVESTIGATIVE+` preset combines all investigative stages. + +*Tests:* - [ ] T-INV-1: STAGE_INVESTIGATIVE includes bits 20-23 - [ ] +T-INV-2: STAGE_ALL includes all 24 stages - [ ] T-INV-3: runStages with +STAGE_INVESTIGATIVE on a legal PDF → all 4 stages produce output - [ ] +T-INV-4: runStages with STAGE_INVESTIGATIVE on audio file → speaker ID +runs, redaction skipped + +==== Step 1.7: Multi-Locale HPC Cluster Test (D1) + +Chapel-based parallel processing on GASNet/IBV transport. + +*Tests:* - [ ] T-HPC-1: 4-node cluster processes 100 documents without +error - [ ] T-HPC-2: Load balancing: no single node processes >40% of +total - [ ] T-HPC-3: Node failure recovery: cluster continues if 1 of 4 +nodes drops - [ ] T-HPC-4: Identical results on 1-node vs 4-node runs +(determinism) + +*Benchmarks:* - [ ] B-HPC-1: 10,000 scanned PDFs on 4-node cluster — +target: <30 minutes - [ ] B-HPC-2: 100,000 PDFs on 16-node cluster — +target: <2 hours - [ ] B-HPC-3: Linear scaling factor — target: >0.7x +per added node - [ ] B-HPC-4: Full Epstein corpus (3.2M files) on 256 +nodes — target: <4 hours + +''''' + +=== Phase 2: Lithoglyph Ingest & Storage (Weeks 4–10) + +==== Step 2.1: Zig 0.15.2 HTTP API Migration (L1) + +83 call sites need updating for the new Zig HTTP API. + +*Tests:* - [ ] T-ZIG-1: All 83 call sites compile with zig 0.15.2 - [ ] +T-ZIG-2: HTTP server starts and responds to GET /health - [ ] T-ZIG-3: +GQL INSERT via HTTP returns 200 + created record ID - [ ] T-ZIG-4: +Concurrent 100-request stress test — no crashes + +*Benchmarks:* - [ ] B-ZIG-1: GQL INSERT latency — target: <5ms p99 - [ ] +B-ZIG-2: Batch INSERT (1000 records) — target: <500ms total + +==== Step 2.2: Evidence Collection Schema (L3) + +Collections: `+bofig_evidence+`, `+bofig_claims+`, +`+bofig_relationships+` + +*Tests:* - [ ] T-EVD-1: CREATE bofig_evidence collection succeeds - [ ] +T-EVD-2: INSERT evidence record with all PROMPT dimensions - [ ] +T-EVD-3: QUERY evidence by SHA-256 hash (dedup lookup) - [ ] T-EVD-4: +QUERY evidence by entity name (cross-reference) - [ ] T-EVD-5: All +mutations have actor + rationale (Lithoglyph invariant) + +==== Step 2.3: Financial Transaction Collection (L4) + +Schema: source, destination, amount, currency, date, instrument, +intermediary + +*Tests:* - [ ] T-FTX-1: INSERT transaction record with full fields - [ ] +T-FTX-2: QUERY transaction chain (A→B→C) via GQL path traversal - [ ] +T-FTX-3: Aggregate: total flow between two entities - [ ] T-FTX-4: +Temporal: transactions within date range - [ ] T-FTX-5: Anomaly: detect +round-number patterns (e.g., exactly $10,000) + +*Benchmarks:* - [ ] B-FTX-1: 16,000 transaction inserts — target: <10 +seconds - [ ] B-FTX-2: Transaction chain query (depth 5) — target: +<100ms + +==== Step 2.4: Entity Collection + Co-Reference Resolution (L5) + +Alias tracking: "`Jeffrey Epstein`" = "`J. Epstein`" = "`Epstein, +Jeffrey`" + +*Tests:* - [ ] T-ENT-1: CREATE entity with primary name - [ ] T-ENT-2: +ADD alias to existing entity - [ ] T-ENT-3: MERGE two entities (logged +in journal with rationale) - [ ] T-ENT-4: REVERSE merge (undo +co-reference error) - [ ] T-ENT-5: QUERY all documents mentioning entity +(across aliases) - [ ] T-ENT-6: No orphaned aliases after merge/unmerge +cycle + +*Benchmarks:* - [ ] B-ENT-1: 23,000 entity inserts with alias resolution +— target: <60 seconds - [ ] B-ENT-2: Entity lookup by any alias — +target: <10ms + +==== Step 2.5: Docudactyl → Lithoglyph Ingest Bridge (D2 + L6) + +Cap’n Proto → GQL INSERT with auto-PROMPT scoring. + +*Tests:* - [ ] T-BRG-1: Single Cap’n Proto StageResults → Lithoglyph +evidence record - [ ] T-BRG-2: PROMPT auto-scoring from extraction +metadata: - OCR confidence 90+ → Provenance score 0.8+ - Multiple +corroborating documents → Replicability score increases - Court filing +(official source) → Publication score 0.9+ - [ ] T-BRG-3: SHA-256 dedup: +duplicate document skipped with log - [ ] T-BRG-4: Batch import 1000 +records — all arrive with provenance - [ ] T-BRG-5: +Actor="`docudactyl-pipeline`", Rationale includes run ID + +*Benchmarks:* - [ ] B-BRG-1: 10,000 records batch import — target: <30 +seconds - [ ] B-BRG-2: 3.2M records full import — target: <6 hours + +''''' + +=== Phase 3: Bofig Evidence Graph (Weeks 8–14) + +==== Step 3.1: Entity Resolution Module (B1) + +NER output → co-reference → unified entity graph in ArangoDB. + +*Tests:* - [ ] T-ER-1: "`J. Epstein`" and "`Jeffrey Epstein`" merge to +one vertex - [ ] T-ER-2: "`Ghislaine Maxwell`" and "`G. Maxwell`" merge +- [ ] T-ER-3: "`Bill Clinton`" and "`William J. Clinton`" merge - [ ] +T-ER-4: "`John Smith`" NOT auto-merged with different "`John Smith`" +(ambiguity threshold) - [ ] T-ER-5: Merge decision logged in Lithoglyph +journal - [ ] T-ER-6: Undo merge restores two separate entities + +*Benchmarks:* - [ ] B-ER-1: 23,000 entities resolved in — target: <5 +minutes - [ ] B-ER-2: Co-reference accuracy on annotated subset — +target: >85% + +==== Step 3.2: Financial Transaction Graph + GraphQL (B2) + +New GraphQL queries for transaction analysis. + +*Tests:* - [ ] T-FGQL-1: `+transactionChain(entityId, depth: 3)+` +returns connected flows - [ ] T-FGQL-2: +`+totalFlow(from, to, dateRange)+` returns aggregate - [ ] T-FGQL-3: +`+anomalies(entityId)+` flags round numbers, structuring patterns - [ ] +T-FGQL-4: Sankey diagram data format correct for D3.js - [ ] T-FGQL-5: +Empty result for entity with no transactions + +*Benchmarks:* - [ ] B-FGQL-1: Transaction chain depth 5 with 16K +transactions — target: <200ms - [ ] B-FGQL-2: Full Sankey data for top +20 entities — target: <1s + +==== Step 3.3: Timeline D3 Visualization (B3) + +Event reconstruction from extracted dates across documents. + +*Tests:* - [ ] T-TL-1: Timeline renders with 100+ events - [ ] T-TL-2: +Zoom to month/week/day granularity - [ ] T-TL-3: Click event → source +document link - [ ] T-TL-4: Filter by entity (e.g., show only events +involving "`Epstein`") - [ ] T-TL-5: Temporal credibility overlay (L7 +data) + +*Benchmarks:* - [ ] B-TL-1: Render 10,000 events — target: <2s initial +load - [ ] B-TL-2: Filter/zoom response — target: <200ms + +==== Step 3.4: Witness Testimony Module (B4) + +Speaker ID + claim extraction + corroboration scoring. + +*Tests:* - [ ] T-WIT-1: Deposition text → extracted claims with speaker +attribution - [ ] T-WIT-2: Corroboration: claim in deposition A matches +claim in deposition B → score increases - [ ] T-WIT-3: Contradiction: +claim A contradicts claim B → flagged with both sources - [ ] T-WIT-4: +Impeachment detection: witness statement contradicts own prior statement +- [ ] T-WIT-5: PROMPT scores for testimony weighted by speaker +credibility + +==== Step 3.5: Contradiction Dashboard (B6) + +Automated surfacing of conflicting accounts. + +*Tests:* - [ ] T-CTR-1: Two documents with contradictory claims → +contradiction record created - [ ] T-CTR-2: UI displays both sources +with PROMPT scores - [ ] T-CTR-3: Resolution: user can mark +contradiction as "`resolved`" with rationale - [ ] T-CTR-4: Filter +contradictions by entity, date, topic - [ ] T-CTR-5: Priority ranking by +evidence quality (higher PROMPT = more serious contradiction) + +==== Step 3.6: Batch Evidence Import (B5) + +GenServer consuming Docudactyl output. + +*Tests:* - [ ] T-BEI-1: GenServer starts and connects to Lithoglyph - [ +] T-BEI-2: Process 100 records → all visible in Bofig UI - [ ] T-BEI-3: +Duplicate handling: same SHA-256 → skip with log - [ ] T-BEI-4: Error +recovery: malformed record → skip, continue, report - [ ] T-BEI-5: +Progress reporting: "`Imported 5000/10000 records`" + +*Benchmarks:* - [ ] B-BEI-1: 10,000 records import → visible in UI — +target: <2 minutes - [ ] B-BEI-2: Memory usage during import — target: +<500MB + +''''' + +=== Phase 4: Investigation Features (Weeks 12–18) + +==== Step 4.1: Redaction Audit Trail (B11 + D6) + +Track what was redacted, by whom, whether text was recovered. + +*Tests:* - [ ] T-RAT-1: Docudactyl flags redacted document → Bofig shows +redaction badge - [ ] T-RAT-2: Recovered text (overlay-only) → available +but marked as "`recovered from redaction`" - [ ] T-RAT-3: Timeline: +document first released redacted (2020), unredacted version released +(2023) - [ ] T-RAT-4: Audit: who accessed recovered text, when, for what +purpose - [ ] T-RAT-5: Integration with Lithoglyph journal — reversible +access grants + +==== Step 4.2: RBAC + Sensitivity (B10) + +Graduated access for sensitive evidence. + +*Tests:* - [ ] T-RBAC-1: Sealed material visible only to authorized +users - [ ] T-RBAC-2: Source protection: informant identity hidden from +non-editors - [ ] T-RBAC-3: Export restrictions: sensitive evidence not +included in public reports - [ ] T-RBAC-4: Audit log: all access to +sensitive material recorded - [ ] T-RBAC-5: Role hierarchy: admin > +editor > viewer > public + +==== Step 4.3: Full-Text Search + Faceted Filters (B9) + +ArangoDB fulltext indexes with faceted navigation. + +*Tests:* - [ ] T-FTS-1: Search "`Lolita Express`" → relevant flight log +documents - [ ] T-FTS-2: Facet by document type (court filing, +deposition, financial record) - [ ] T-FTS-3: Facet by date range - [ ] +T-FTS-4: Facet by entity involvement - [ ] T-FTS-5: Highlight search +terms in results + +*Benchmarks:* - [ ] B-FTS-1: Full-text search across 3.2M documents — +target: <500ms - [ ] B-FTS-2: Faceted filter application — target: +<200ms + +==== Step 4.4: Temporal Credibility Model (L7) + +Source reputation evolving over time. + +*Tests:* - [ ] T-TCR-1: New source starts at neutral credibility - [ ] +T-TCR-2: Source’s claim independently verified → credibility increases - +[ ] T-TCR-3: Source caught in contradiction → credibility decreases - [ +] T-TCR-4: Source retraction → credibility impact + retraction logged - +[ ] T-TCR-5: Time-travel: "`What was this source’s credibility on +2023-01-15?`" - [ ] T-TCR-6: Credibility affects PROMPT scores of all +evidence from that source + +''''' + +=== Phase 5: Scale & Production (Weeks 16–22) + +==== Step 5.1: Full Corpus Run + +Process all 3.2M Epstein files through the complete pipeline. + +*Tests:* - [ ] T-FCR-1: All files processed without pipeline crash - [ ] +T-FCR-2: <0.1% error rate (files that failed extraction) - [ ] T-FCR-3: +Dedup: identify exact duplicates across data sets - [ ] T-FCR-4: +Near-dedup: identify visually similar images (perceptual hash) - [ ] +T-FCR-5: Entity graph fully connected (no isolated entity clusters that +should be linked) + +*Benchmarks:* - [ ] B-FCR-1: Full extraction (256 nodes) — target: <4 +hours - [ ] B-FCR-2: Full Lithoglyph import — target: <8 hours - [ ] +B-FCR-3: Full entity resolution — target: <2 hours - [ ] B-FCR-4: Total +pipeline cold start to searchable — target: <24 hours - [ ] B-FCR-5: +Storage: Lithoglyph database size — target: <100GB for metadata (excl. +raw files) + +==== Step 5.2: Cross-Investigation Linking (L8) + +Shared evidence across investigations (Epstein ↔ Maxwell ↔ related +cases). + +*Tests:* - [ ] T-XIL-1: Evidence in investigation A also relevant to +investigation B → linked - [ ] T-XIL-2: Entity appearing in both +investigations → surfaced automatically - [ ] T-XIL-3: New investigation +inherits relevant evidence from existing investigations - [ ] T-XIL-4: +Access controls per investigation (B10) + +==== Step 5.3: IPFS Provenance (B12) + +Tamper-proof evidence archival. + +*Tests:* - [ ] T-IPFS-1: Evidence pinned to IPFS with CID stored in +Lithoglyph - [ ] T-IPFS-2: Retrieve evidence by CID → matches SHA-256 in +Lithoglyph - [ ] T-IPFS-3: Evidence tampering detected (hash mismatch) - +[ ] T-IPFS-4: Offline operation: evidence accessible from local cache +when IPFS unavailable + +==== Step 5.4: Audience-Specific Navigation (All 6 Audience Types) + +Each audience type sees the same evidence graph with different +weighting. + +*Tests:* - [ ] T-AUD-1: Journalist view: balanced credibility, source +protection, story threads - [ ] T-AUD-2: Researcher view: methodology +scores prominent, reproducibility highlighted - [ ] T-AUD-3: Policymaker +view: authoritative sources first, regulatory implications - [ ] +T-AUD-4: Affected person view: personal impact, clear language, trigger +warnings - [ ] T-AUD-5: Skeptic view: transparency scores, verification +steps, weakest links highlighted - [ ] T-AUD-6: Activist view: evidence +quality, actionable findings, campaign-relevant - [ ] T-AUD-7: Same +evidence, different PROMPT dimension weights per audience → different +ordering + +''''' + +=== Summary: Work Completion Tracker + +[cols=",,,,,",options="header",] +|=== +|# |Step |Repo |Status |Tests |Benchmarks +|1.1 |Core OCR + Text |Docudactyl |DONE |Partial |0/1 +|1.2 |Redaction Detection |Docudactyl |DONE (code) |0/5 |0/2 +|1.3 |Financial Extraction |Docudactyl |DONE (code) |0/6 |0/2 +|1.4 |Legal NER |Docudactyl |DONE (code) |0/6 |0/2 +|1.5 |Speaker ID |Docudactyl |DONE (dispatch) |0/4 |0/2 +|1.6 |Investigative Preset |Docudactyl |DONE |0/4 |— +|1.7 |HPC Cluster Test |Docudactyl |TODO |0/4 |0/4 +|2.1 |Zig API Migration |Lithoglyph |*DONE* |0/4 |0/2 +|2.2 |Evidence Schema |Lithoglyph |*DONE* |0/5 |— +|2.3 |Financial Txn Collection |Lithoglyph |TODO |0/5 |0/2 +|2.4 |Entity + Co-Ref |Lithoglyph |TODO |0/6 |0/2 +|2.5 |Ingest Bridge |Both |TODO |0/5 |0/2 +|3.1 |Entity Resolution |Bofig |TODO |0/6 |0/2 +|3.2 |Financial GraphQL |Bofig |TODO |0/5 |0/2 +|3.3 |Timeline Viz |Bofig |TODO |0/5 |0/2 +|3.4 |Witness Testimony |Bofig |TODO |0/5 |— +|3.5 |Contradiction Dashboard |Bofig |TODO |0/5 |— +|3.6 |Batch Import |Bofig |TODO |0/5 |0/2 +|4.1 |Redaction Audit |Bofig |TODO |0/5 |— +|4.2 |RBAC |Bofig |TODO |0/5 |— +|4.3 |Full-Text Search |Bofig |TODO |0/5 |0/2 +|4.4 |Temporal Credibility |Lithoglyph |TODO |0/6 |— +|5.1 |Full Corpus Run |All |TODO |0/5 |0/5 +|5.2 |Cross-Investigation |Lithoglyph |TODO |0/4 |— +|5.3 |IPFS Provenance |Bofig |TODO |0/4 |— +|5.4 |Audience Navigation |Bofig |TODO |0/7 |— +|=== + +*Totals: 130 tests, 35 benchmarks across 26 steps* *Current: 4 steps +code-complete, 0 tests written, 0 benchmarks run* + +''''' + +=== Critical Path + +.... +D1 (HPC cluster) ──────────┐ +L1 (Zig migration) ────┐ │ +L2 (Rename) ────────────┤ │ + ▼ ▼ + L3 (Schema) + │ + L6 (Bridge) ←── D2 (Adapter) + │ + B5 (Import) + │ + ┌─────────┼─────────┐ + ▼ ▼ ▼ + B1 (Entity) B2 (Fin) B3 (Timeline) + │ │ │ + └─────────┼─────────┘ + ▼ + B4 (Testimony) + B6 (Contradictions) + │ + B10 (RBAC) + B9 (Search) + │ + 5.1 (Full Run) + │ + 5.4 (Audiences) +.... + +*Longest path: D1 → L3 → L6 → B5 → B1 → B4 → B10 → 5.1 → 5.4 = 9 +sequential dependencies* *Estimated calendar: ~22 weeks with parallelism +across repos* diff --git a/docs/EPSTEIN-FILES-WORK-PATHWAY.md b/docs/EPSTEIN-FILES-WORK-PATHWAY.md deleted file mode 100644 index a4f3b35..0000000 --- a/docs/EPSTEIN-FILES-WORK-PATHWAY.md +++ /dev/null @@ -1,477 +0,0 @@ -# Epstein Files — Complete Work Pathway -# Tests & Benchmarks at Every Stage -# -# SPDX-License-Identifier: CC-BY-SA-4.0 -# Author: Jonathan D.A. Jewell -# Created: 2026-03-11 -# -# This document maps every implementation step needed to process the -# Epstein files (3M+ files, 218GB, 12 data sets, 23K entities) through -# the Docudactyl → Lithoglyph → Bofig pipeline, with test criteria -# and performance benchmarks for each milestone. - ---- - -## Dataset Characteristics - -| Attribute | Value | -|-----------|-------| -| Total files | ~3,200,000 | -| Total size | ~218 GB | -| Data sets | 12 (flight logs, court filings, depositions, financial records, photos, communications, ...) | -| Named entities (est.) | 23,000+ unique persons, orgs, locations | -| Primary format | Scanned PDF (96 DPI, many poor quality) | -| Secondary formats | TIFF, JPEG, DOCX, XLS, email (EML/PST) | -| Redaction style | Overlay-only (text stream often intact) | -| Financial transactions (est.) | 16,000+ | -| Languages | English (primary), French, some Spanish | -| Time span | 1990s–2024 | - ---- - -## Phase 1: Docudactyl Extraction Pipeline (Weeks 1–6) - -### Step 1.1: Core OCR + Text Extraction (DONE — existing stages 0-8) - -Already implemented in `stages.zig`: -- Language detection, readability, keywords, citations -- OCR confidence, perceptual hash, TOC extraction -- Multi-language OCR, subtitle extraction - -**Tests:** -- [x] Unit test: each stage function with known input produces expected Cap'n Proto output -- [x] Integration test: 10-document mini-corpus end-to-end -- [ ] Benchmark: single-node throughput for scanned PDFs (target: 2 docs/sec on 8-core) - -### Step 1.2: Redaction Detection (DONE — bit 20, `stageRedactionDetect`) - -Scans Poppler annotations for Type 12 (POPPLER_ANNOT_REDACT), checks if -text is extractable under overlay-only redactions. - -**Tests:** -- [ ] T-RED-1: Synthetic PDF with 5 /Redact annotations → count=5, status="redacted" -- [ ] T-RED-2: PDF with black fill rectangles but no /Redact annots → status="clean" (future: heuristic upgrade) -- [ ] T-RED-3: PDF with overlay redaction + recoverable text → recoverable_count > 0 -- [ ] T-RED-4: Non-PDF input (JPEG) → status="not_applicable" -- [ ] T-RED-5: Corrupt/unreadable PDF → status="error" - -**Benchmarks:** -- [ ] B-RED-1: 1000 PDFs (mixed redacted/clean) — target: <500ms per document -- [ ] B-RED-2: Memory usage during annotation scan — target: <50MB peak per document - -### Step 1.3: Financial Entity Extraction (DONE — bit 21, `stageFinancialExtract`) - -Pattern-based detection of currency symbols ($, £, €), ISO codes (USD, GBP, EUR, CHF, JPY, CAD), -account-like digit sequences (8-20 digits). - -**Tests:** -- [ ] T-FIN-1: Text "$1,234.56 paid to account 12345678" → amounts=1, accounts=1 -- [ ] T-FIN-2: Text "USD 50,000 transferred" → amounts=1 -- [ ] T-FIN-3: Text "£2.3 million to HSBC account 1234-5678-9012" → amounts=1, accounts=1 -- [ ] T-FIN-4: Phone numbers should NOT match as accounts (7-digit filter) -- [ ] T-FIN-5: Empty text → status="none_found", amounts=0, accounts=0 -- [ ] T-FIN-6: Mixed currencies in single document → correct total count - -**Benchmarks:** -- [ ] B-FIN-1: 10MB text document scan — target: <200ms -- [ ] B-FIN-2: Accuracy on annotated Epstein financial records sample (50 docs) — target: >80% recall - -### Step 1.4: Legal NER (DONE — bit 22, `stageLegalNer`) - -Pattern-based detection of case citations ("v."), docket numbers ("No.", "Case"), -statute references ("U.S.C.", "§"). - -**Tests:** -- [ ] T-LEG-1: "Doe v. Epstein" → case_citations=1 -- [ ] T-LEG-2: "No. 08-cv-1234" → docket_refs=1 -- [ ] T-LEG-3: "18 U.S.C. § 1591" → statute_refs=1 (both U.S.C. and § counted) -- [ ] T-LEG-4: "vs." variant → case_citations=1 -- [ ] T-LEG-5: Real Epstein court filing excerpt → realistic counts -- [ ] T-LEG-6: Non-legal document (flight log) → all counts = 0 - -**Benchmarks:** -- [ ] B-LEG-1: 5MB legal document — target: <150ms -- [ ] B-LEG-2: Precision on annotated legal corpus (100 docs) — target: >75% - -### Step 1.5: Speaker Identification (bit 23, ML dispatch) - -ML-based speaker diarization via ONNX Runtime. Dispatches to stage_id=5 -(speaker_id.onnx model). - -**Tests:** -- [ ] T-SPK-1: With ML handle + model → status="ok", speaker_count > 0 -- [ ] T-SPK-2: Without ML handle → status="not_available" -- [ ] T-SPK-3: Non-audio input → graceful fallback -- [ ] T-SPK-4: Deposition audio with 2 speakers → speaker_count=2 - -**Benchmarks:** -- [ ] B-SPK-1: 30-minute deposition audio — target: <60s inference -- [ ] B-SPK-2: Memory usage during diarization — target: <2GB - -### Step 1.6: STAGE_INVESTIGATIVE Preset Validation - -The `STAGE_INVESTIGATIVE` preset combines all investigative stages. - -**Tests:** -- [ ] T-INV-1: STAGE_INVESTIGATIVE includes bits 20-23 -- [ ] T-INV-2: STAGE_ALL includes all 24 stages -- [ ] T-INV-3: runStages with STAGE_INVESTIGATIVE on a legal PDF → all 4 stages produce output -- [ ] T-INV-4: runStages with STAGE_INVESTIGATIVE on audio file → speaker ID runs, redaction skipped - -### Step 1.7: Multi-Locale HPC Cluster Test (D1) - -Chapel-based parallel processing on GASNet/IBV transport. - -**Tests:** -- [ ] T-HPC-1: 4-node cluster processes 100 documents without error -- [ ] T-HPC-2: Load balancing: no single node processes >40% of total -- [ ] T-HPC-3: Node failure recovery: cluster continues if 1 of 4 nodes drops -- [ ] T-HPC-4: Identical results on 1-node vs 4-node runs (determinism) - -**Benchmarks:** -- [ ] B-HPC-1: 10,000 scanned PDFs on 4-node cluster — target: <30 minutes -- [ ] B-HPC-2: 100,000 PDFs on 16-node cluster — target: <2 hours -- [ ] B-HPC-3: Linear scaling factor — target: >0.7x per added node -- [ ] B-HPC-4: Full Epstein corpus (3.2M files) on 256 nodes — target: <4 hours - ---- - -## Phase 2: Lithoglyph Ingest & Storage (Weeks 4–10) - -### Step 2.1: Zig 0.15.2 HTTP API Migration (L1) - -83 call sites need updating for the new Zig HTTP API. - -**Tests:** -- [ ] T-ZIG-1: All 83 call sites compile with zig 0.15.2 -- [ ] T-ZIG-2: HTTP server starts and responds to GET /health -- [ ] T-ZIG-3: GQL INSERT via HTTP returns 200 + created record ID -- [ ] T-ZIG-4: Concurrent 100-request stress test — no crashes - -**Benchmarks:** -- [ ] B-ZIG-1: GQL INSERT latency — target: <5ms p99 -- [ ] B-ZIG-2: Batch INSERT (1000 records) — target: <500ms total - -### Step 2.2: Evidence Collection Schema (L3) - -Collections: `bofig_evidence`, `bofig_claims`, `bofig_relationships` - -**Tests:** -- [ ] T-EVD-1: CREATE bofig_evidence collection succeeds -- [ ] T-EVD-2: INSERT evidence record with all PROMPT dimensions -- [ ] T-EVD-3: QUERY evidence by SHA-256 hash (dedup lookup) -- [ ] T-EVD-4: QUERY evidence by entity name (cross-reference) -- [ ] T-EVD-5: All mutations have actor + rationale (Lithoglyph invariant) - -### Step 2.3: Financial Transaction Collection (L4) - -Schema: source, destination, amount, currency, date, instrument, intermediary - -**Tests:** -- [ ] T-FTX-1: INSERT transaction record with full fields -- [ ] T-FTX-2: QUERY transaction chain (A→B→C) via GQL path traversal -- [ ] T-FTX-3: Aggregate: total flow between two entities -- [ ] T-FTX-4: Temporal: transactions within date range -- [ ] T-FTX-5: Anomaly: detect round-number patterns (e.g., exactly $10,000) - -**Benchmarks:** -- [ ] B-FTX-1: 16,000 transaction inserts — target: <10 seconds -- [ ] B-FTX-2: Transaction chain query (depth 5) — target: <100ms - -### Step 2.4: Entity Collection + Co-Reference Resolution (L5) - -Alias tracking: "Jeffrey Epstein" = "J. Epstein" = "Epstein, Jeffrey" - -**Tests:** -- [ ] T-ENT-1: CREATE entity with primary name -- [ ] T-ENT-2: ADD alias to existing entity -- [ ] T-ENT-3: MERGE two entities (logged in journal with rationale) -- [ ] T-ENT-4: REVERSE merge (undo co-reference error) -- [ ] T-ENT-5: QUERY all documents mentioning entity (across aliases) -- [ ] T-ENT-6: No orphaned aliases after merge/unmerge cycle - -**Benchmarks:** -- [ ] B-ENT-1: 23,000 entity inserts with alias resolution — target: <60 seconds -- [ ] B-ENT-2: Entity lookup by any alias — target: <10ms - -### Step 2.5: Docudactyl → Lithoglyph Ingest Bridge (D2 + L6) - -Cap'n Proto → GQL INSERT with auto-PROMPT scoring. - -**Tests:** -- [ ] T-BRG-1: Single Cap'n Proto StageResults → Lithoglyph evidence record -- [ ] T-BRG-2: PROMPT auto-scoring from extraction metadata: - - OCR confidence 90+ → Provenance score 0.8+ - - Multiple corroborating documents → Replicability score increases - - Court filing (official source) → Publication score 0.9+ -- [ ] T-BRG-3: SHA-256 dedup: duplicate document skipped with log -- [ ] T-BRG-4: Batch import 1000 records — all arrive with provenance -- [ ] T-BRG-5: Actor="docudactyl-pipeline", Rationale includes run ID - -**Benchmarks:** -- [ ] B-BRG-1: 10,000 records batch import — target: <30 seconds -- [ ] B-BRG-2: 3.2M records full import — target: <6 hours - ---- - -## Phase 3: Bofig Evidence Graph (Weeks 8–14) - -### Step 3.1: Entity Resolution Module (B1) - -NER output → co-reference → unified entity graph in ArangoDB. - -**Tests:** -- [ ] T-ER-1: "J. Epstein" and "Jeffrey Epstein" merge to one vertex -- [ ] T-ER-2: "Ghislaine Maxwell" and "G. Maxwell" merge -- [ ] T-ER-3: "Bill Clinton" and "William J. Clinton" merge -- [ ] T-ER-4: "John Smith" NOT auto-merged with different "John Smith" (ambiguity threshold) -- [ ] T-ER-5: Merge decision logged in Lithoglyph journal -- [ ] T-ER-6: Undo merge restores two separate entities - -**Benchmarks:** -- [ ] B-ER-1: 23,000 entities resolved in — target: <5 minutes -- [ ] B-ER-2: Co-reference accuracy on annotated subset — target: >85% - -### Step 3.2: Financial Transaction Graph + GraphQL (B2) - -New GraphQL queries for transaction analysis. - -**Tests:** -- [ ] T-FGQL-1: `transactionChain(entityId, depth: 3)` returns connected flows -- [ ] T-FGQL-2: `totalFlow(from, to, dateRange)` returns aggregate -- [ ] T-FGQL-3: `anomalies(entityId)` flags round numbers, structuring patterns -- [ ] T-FGQL-4: Sankey diagram data format correct for D3.js -- [ ] T-FGQL-5: Empty result for entity with no transactions - -**Benchmarks:** -- [ ] B-FGQL-1: Transaction chain depth 5 with 16K transactions — target: <200ms -- [ ] B-FGQL-2: Full Sankey data for top 20 entities — target: <1s - -### Step 3.3: Timeline D3 Visualization (B3) - -Event reconstruction from extracted dates across documents. - -**Tests:** -- [ ] T-TL-1: Timeline renders with 100+ events -- [ ] T-TL-2: Zoom to month/week/day granularity -- [ ] T-TL-3: Click event → source document link -- [ ] T-TL-4: Filter by entity (e.g., show only events involving "Epstein") -- [ ] T-TL-5: Temporal credibility overlay (L7 data) - -**Benchmarks:** -- [ ] B-TL-1: Render 10,000 events — target: <2s initial load -- [ ] B-TL-2: Filter/zoom response — target: <200ms - -### Step 3.4: Witness Testimony Module (B4) - -Speaker ID + claim extraction + corroboration scoring. - -**Tests:** -- [ ] T-WIT-1: Deposition text → extracted claims with speaker attribution -- [ ] T-WIT-2: Corroboration: claim in deposition A matches claim in deposition B → score increases -- [ ] T-WIT-3: Contradiction: claim A contradicts claim B → flagged with both sources -- [ ] T-WIT-4: Impeachment detection: witness statement contradicts own prior statement -- [ ] T-WIT-5: PROMPT scores for testimony weighted by speaker credibility - -### Step 3.5: Contradiction Dashboard (B6) - -Automated surfacing of conflicting accounts. - -**Tests:** -- [ ] T-CTR-1: Two documents with contradictory claims → contradiction record created -- [ ] T-CTR-2: UI displays both sources with PROMPT scores -- [ ] T-CTR-3: Resolution: user can mark contradiction as "resolved" with rationale -- [ ] T-CTR-4: Filter contradictions by entity, date, topic -- [ ] T-CTR-5: Priority ranking by evidence quality (higher PROMPT = more serious contradiction) - -### Step 3.6: Batch Evidence Import (B5) - -GenServer consuming Docudactyl output. - -**Tests:** -- [ ] T-BEI-1: GenServer starts and connects to Lithoglyph -- [ ] T-BEI-2: Process 100 records → all visible in Bofig UI -- [ ] T-BEI-3: Duplicate handling: same SHA-256 → skip with log -- [ ] T-BEI-4: Error recovery: malformed record → skip, continue, report -- [ ] T-BEI-5: Progress reporting: "Imported 5000/10000 records" - -**Benchmarks:** -- [ ] B-BEI-1: 10,000 records import → visible in UI — target: <2 minutes -- [ ] B-BEI-2: Memory usage during import — target: <500MB - ---- - -## Phase 4: Investigation Features (Weeks 12–18) - -### Step 4.1: Redaction Audit Trail (B11 + D6) - -Track what was redacted, by whom, whether text was recovered. - -**Tests:** -- [ ] T-RAT-1: Docudactyl flags redacted document → Bofig shows redaction badge -- [ ] T-RAT-2: Recovered text (overlay-only) → available but marked as "recovered from redaction" -- [ ] T-RAT-3: Timeline: document first released redacted (2020), unredacted version released (2023) -- [ ] T-RAT-4: Audit: who accessed recovered text, when, for what purpose -- [ ] T-RAT-5: Integration with Lithoglyph journal — reversible access grants - -### Step 4.2: RBAC + Sensitivity (B10) - -Graduated access for sensitive evidence. - -**Tests:** -- [ ] T-RBAC-1: Sealed material visible only to authorized users -- [ ] T-RBAC-2: Source protection: informant identity hidden from non-editors -- [ ] T-RBAC-3: Export restrictions: sensitive evidence not included in public reports -- [ ] T-RBAC-4: Audit log: all access to sensitive material recorded -- [ ] T-RBAC-5: Role hierarchy: admin > editor > viewer > public - -### Step 4.3: Full-Text Search + Faceted Filters (B9) - -ArangoDB fulltext indexes with faceted navigation. - -**Tests:** -- [ ] T-FTS-1: Search "Lolita Express" → relevant flight log documents -- [ ] T-FTS-2: Facet by document type (court filing, deposition, financial record) -- [ ] T-FTS-3: Facet by date range -- [ ] T-FTS-4: Facet by entity involvement -- [ ] T-FTS-5: Highlight search terms in results - -**Benchmarks:** -- [ ] B-FTS-1: Full-text search across 3.2M documents — target: <500ms -- [ ] B-FTS-2: Faceted filter application — target: <200ms - -### Step 4.4: Temporal Credibility Model (L7) - -Source reputation evolving over time. - -**Tests:** -- [ ] T-TCR-1: New source starts at neutral credibility -- [ ] T-TCR-2: Source's claim independently verified → credibility increases -- [ ] T-TCR-3: Source caught in contradiction → credibility decreases -- [ ] T-TCR-4: Source retraction → credibility impact + retraction logged -- [ ] T-TCR-5: Time-travel: "What was this source's credibility on 2023-01-15?" -- [ ] T-TCR-6: Credibility affects PROMPT scores of all evidence from that source - ---- - -## Phase 5: Scale & Production (Weeks 16–22) - -### Step 5.1: Full Corpus Run - -Process all 3.2M Epstein files through the complete pipeline. - -**Tests:** -- [ ] T-FCR-1: All files processed without pipeline crash -- [ ] T-FCR-2: <0.1% error rate (files that failed extraction) -- [ ] T-FCR-3: Dedup: identify exact duplicates across data sets -- [ ] T-FCR-4: Near-dedup: identify visually similar images (perceptual hash) -- [ ] T-FCR-5: Entity graph fully connected (no isolated entity clusters that should be linked) - -**Benchmarks:** -- [ ] B-FCR-1: Full extraction (256 nodes) — target: <4 hours -- [ ] B-FCR-2: Full Lithoglyph import — target: <8 hours -- [ ] B-FCR-3: Full entity resolution — target: <2 hours -- [ ] B-FCR-4: Total pipeline cold start to searchable — target: <24 hours -- [ ] B-FCR-5: Storage: Lithoglyph database size — target: <100GB for metadata (excl. raw files) - -### Step 5.2: Cross-Investigation Linking (L8) - -Shared evidence across investigations (Epstein ↔ Maxwell ↔ related cases). - -**Tests:** -- [ ] T-XIL-1: Evidence in investigation A also relevant to investigation B → linked -- [ ] T-XIL-2: Entity appearing in both investigations → surfaced automatically -- [ ] T-XIL-3: New investigation inherits relevant evidence from existing investigations -- [ ] T-XIL-4: Access controls per investigation (B10) - -### Step 5.3: IPFS Provenance (B12) - -Tamper-proof evidence archival. - -**Tests:** -- [ ] T-IPFS-1: Evidence pinned to IPFS with CID stored in Lithoglyph -- [ ] T-IPFS-2: Retrieve evidence by CID → matches SHA-256 in Lithoglyph -- [ ] T-IPFS-3: Evidence tampering detected (hash mismatch) -- [ ] T-IPFS-4: Offline operation: evidence accessible from local cache when IPFS unavailable - -### Step 5.4: Audience-Specific Navigation (All 6 Audience Types) - -Each audience type sees the same evidence graph with different weighting. - -**Tests:** -- [ ] T-AUD-1: Journalist view: balanced credibility, source protection, story threads -- [ ] T-AUD-2: Researcher view: methodology scores prominent, reproducibility highlighted -- [ ] T-AUD-3: Policymaker view: authoritative sources first, regulatory implications -- [ ] T-AUD-4: Affected person view: personal impact, clear language, trigger warnings -- [ ] T-AUD-5: Skeptic view: transparency scores, verification steps, weakest links highlighted -- [ ] T-AUD-6: Activist view: evidence quality, actionable findings, campaign-relevant -- [ ] T-AUD-7: Same evidence, different PROMPT dimension weights per audience → different ordering - ---- - -## Summary: Work Completion Tracker - -| # | Step | Repo | Status | Tests | Benchmarks | -|---|------|------|--------|-------|------------| -| 1.1 | Core OCR + Text | Docudactyl | DONE | Partial | 0/1 | -| 1.2 | Redaction Detection | Docudactyl | DONE (code) | 0/5 | 0/2 | -| 1.3 | Financial Extraction | Docudactyl | DONE (code) | 0/6 | 0/2 | -| 1.4 | Legal NER | Docudactyl | DONE (code) | 0/6 | 0/2 | -| 1.5 | Speaker ID | Docudactyl | DONE (dispatch) | 0/4 | 0/2 | -| 1.6 | Investigative Preset | Docudactyl | DONE | 0/4 | — | -| 1.7 | HPC Cluster Test | Docudactyl | TODO | 0/4 | 0/4 | -| 2.1 | Zig API Migration | Lithoglyph | **DONE** | 0/4 | 0/2 | -| 2.2 | Evidence Schema | Lithoglyph | **DONE** | 0/5 | — | -| 2.3 | Financial Txn Collection | Lithoglyph | TODO | 0/5 | 0/2 | -| 2.4 | Entity + Co-Ref | Lithoglyph | TODO | 0/6 | 0/2 | -| 2.5 | Ingest Bridge | Both | TODO | 0/5 | 0/2 | -| 3.1 | Entity Resolution | Bofig | TODO | 0/6 | 0/2 | -| 3.2 | Financial GraphQL | Bofig | TODO | 0/5 | 0/2 | -| 3.3 | Timeline Viz | Bofig | TODO | 0/5 | 0/2 | -| 3.4 | Witness Testimony | Bofig | TODO | 0/5 | — | -| 3.5 | Contradiction Dashboard | Bofig | TODO | 0/5 | — | -| 3.6 | Batch Import | Bofig | TODO | 0/5 | 0/2 | -| 4.1 | Redaction Audit | Bofig | TODO | 0/5 | — | -| 4.2 | RBAC | Bofig | TODO | 0/5 | — | -| 4.3 | Full-Text Search | Bofig | TODO | 0/5 | 0/2 | -| 4.4 | Temporal Credibility | Lithoglyph | TODO | 0/6 | — | -| 5.1 | Full Corpus Run | All | TODO | 0/5 | 0/5 | -| 5.2 | Cross-Investigation | Lithoglyph | TODO | 0/4 | — | -| 5.3 | IPFS Provenance | Bofig | TODO | 0/4 | — | -| 5.4 | Audience Navigation | Bofig | TODO | 0/7 | — | - -**Totals: 130 tests, 35 benchmarks across 26 steps** -**Current: 4 steps code-complete, 0 tests written, 0 benchmarks run** - ---- - -## Critical Path - -``` -D1 (HPC cluster) ──────────┐ -L1 (Zig migration) ────┐ │ -L2 (Rename) ────────────┤ │ - ▼ ▼ - L3 (Schema) - │ - L6 (Bridge) ←── D2 (Adapter) - │ - B5 (Import) - │ - ┌─────────┼─────────┐ - ▼ ▼ ▼ - B1 (Entity) B2 (Fin) B3 (Timeline) - │ │ │ - └─────────┼─────────┘ - ▼ - B4 (Testimony) + B6 (Contradictions) - │ - B10 (RBAC) + B9 (Search) - │ - 5.1 (Full Run) - │ - 5.4 (Audiences) -``` - -**Longest path: D1 → L3 → L6 → B5 → B1 → B4 → B10 → 5.1 → 5.4 = 9 sequential dependencies** -**Estimated calendar: ~22 weeks with parallelism across repos** diff --git a/docs/INTEGRATION-PLAN.adoc b/docs/INTEGRATION-PLAN.adoc new file mode 100644 index 0000000..5cbfe9f --- /dev/null +++ b/docs/INTEGRATION-PLAN.adoc @@ -0,0 +1,345 @@ +== Investigative Evidence Pipeline — Integration Plan + +== Bofig + Docudactyl + Lithoglyph + +== + +== SPDX-License-Identifier: CC-BY-SA-4.0 + +== Author: Jonathan D.A. Jewell + +== Created: 2026-03-11 + +=== Vision + +A complete pipeline from raw document dumps (court filings, flight logs, +financial records, testimony transcripts) through HPC extraction, into +an audit-grade database, rendered as a navigable evidence graph with +audience-specific views. + +.... +Raw Documents (200K+ files) + → Docudactyl (HPC extraction: OCR, NER, metadata, classification) + → Lithoglyph (audit-grade storage: provenance, reversibility, PROMPT) + → Bofig (evidence graph: claims, relationships, navigation) + → 6 audience views (journalist, researcher, policymaker, ...) +.... + +''''' + +=== Repo Assignments + +==== Repo 1: Docudactyl (bofig/docudactyl/) + +*Role in pipeline:* Ingestion layer. Takes raw documents and produces +structured extraction results. + +*Status:* v0.4.0, 97% complete. Awaiting multi-locale cluster testing. + +[width="100%",cols="11%,17%,29%,23%,20%",options="header",] +|=== +|# |Task |Priority |Effort |Notes +|D1 |Multi-locale HPC cluster test (GASNet/IBV, 4+ nodes) |Critical +|Medium |Only v0.4.1 blocker + +|D2 |Cap’n Proto → Lithoglyph output adapter |High |Medium |New output +stage that emits GQL-compatible evidence records + +|D3 |Legal document NER model |High |Medium |Docket numbers, case names, +judge names, legal citations + +|D4 |Financial record extraction stage |High |Medium |Transaction +amounts, dates, account identifiers, counterparties + +|D5 |Speaker identification stage (testimony/depositions) |Medium |Large +|Who said what — maps to witness testimony in bofig + +|D6 |Redaction detection stage |Medium |Small |Flag redacted regions, +track unredaction over time + +|D7 |British Library pilot (170M items) |Low |Large |v1.0.0 milestone +|=== + +*Key output contract:* Each processed document produces: - Extracted +text (OCR’d if needed) + confidence score - NER entities (people, orgs, +locations, dates, amounts) - SHA-256 + perceptual hash (dedup) - +Metadata (Dublin Core + format-specific) - Auto-PROMPT scores derived +from extraction quality - Language, keywords, citations + +''''' + +==== Repo 2: Lithoglyph (nextgen-databases/lithoglyph/) + +*Role in pipeline:* Provenance layer. Stores all evidence with full +audit trail, reversibility, and PROMPT scoring. + +*Status:* 75% complete. L1/L2/L3 complete. formdb-http/ renamed to +lith-http/. + +[width="100%",cols="11%,17%,29%,23%,20%",options="header",] +|=== +|# |Task |Priority |Effort |Notes +|L1 |Zig 0.15.2 HTTP API migration (83 call sites) |Critical |Medium +|*COMPLETE* — Reader/Writer pattern applied + +|L2 |FormBD → Lithoglyph rename (Google trademark) |Critical |Small +|*COMPLETE* — formdb-http/ → lith-http/ + +|L3 |Evidence collection schema for bofig |High |Small |*COMPLETE* — 5 +collections in glyphbase/examples/bofig-evidence.json + 5 FQL test +vectors + +|L4 |Financial transaction collection |High |Small |source, destination, +amount, date, instrument, intermediary + +|L5 |Entity collection + co-reference resolution |High |Medium +|Person/org/location entities with alias tracking + +|L6 |Ingest bridge: Docudactyl Cap’n Proto → GQL INSERT |High |Medium +|Batch import with auto-PROMPT scoring from extraction metadata + +|L7 |Temporal credibility model |Medium |Medium |Source reputation +updates over time (retractions, discrediting) + +|L8 |Cross-investigation linking |Medium |Small |Shared evidence +collections, automatic surfacing + +|L9 |ControlPlane clustering (Elixir/OTP) |Low |Large |Multi-node +Lithoglyph for scale +|=== + +*Key guarantees Lithoglyph provides:* - Every mutation has actor + +rationale (accountability) - Reversible operations (retractions with +explanation) - Time-travel queries ("`what did we know on date X?`") - +PROMPT scores as first-class citizens - Constraints-as-ethics (invalid +evidence relationships rejected with explanation) - Dependent-type +proofs (GQL-DT) for score bounds + +''''' + +==== Repo 3: Bofig (bofig/) + +*Role in pipeline:* Application layer. Evidence graph, PROMPT scoring, +audience-specific navigation, collaborative investigation. + +*Status:* Phase 1 complete (v1.0.0). 257 tests, 0 failures. + +[width="100%",cols="11%,17%,29%,23%,20%",options="header",] +|=== +|# |Task |Priority |Effort |Notes +|B1 |Entity resolution module |Critical |Medium |NER output → +co-reference → unified entity graph + +|B2 |Financial transaction graph + GraphQL schema |High |Small |New +queries: transaction chains, flow analysis, anomaly detection + +|B3 |Timeline D3 visualization |High |Small |Phase 2 planned — event +reconstruction from extracted dates + +|B4 |Witness testimony module |High |Large |Speaker ID, claim +extraction, corroboration scoring, impeachment detection + +|B5 |Batch evidence import (Docudactyl output) |High |Medium |GenServer +consuming structured extraction results + +|B6 |Contradiction dashboard |High |Small |Surface conflicting accounts +across documents automatically + +|B7 |Multi-investigation dashboard |Medium |Medium |Phase 2 — +cross-referencing, shared evidence + +|B8 |Real-time collaborative editing (PubSub) |Medium |Medium |Phase 2 — +multiple journalists mapping evidence simultaneously + +|B9 |Full-text search + faceted filters |Medium |Small |Phase 2 — +ArangoDB fulltext indexes + +|B10 |RBAC + sensitivity layer |Medium |Medium |Phase 2 — protect +sources, sealed material, graduated access + +|B11 |Redaction audit trail |Medium |Small |Track what was +redacted/unredacted, by whom, integrated with Lithoglyph journal + +|B12 |IPFS provenance integration |Low |Medium |Phase 2 — tamper-proof +evidence archival + +|B13 |Lithoglyph provenance layer |Low |Large |Phase 3 — +supplement/replace ArangoDB with Lithoglyph for audit-grade storage + +|B14 |Public API + SDK |Low |Medium |Phase 3 + +|B15 |LEAN4 formal proofs for crypto protocols |Low |Large |Phase 3 +|=== + +''''' + +=== Integration Points (How They Fit Together) + +==== Integration 1: Docudactyl → Lithoglyph (D2 + L6) + +.... +Docudactyl Cap'n Proto output + → Adapter (D2) serializes to GQL INSERT statements + → Lithoglyph ingest bridge (L6) batch-imports with: + - Auto-PROMPT scoring from extraction confidence + - SHA-256 dedup against existing evidence + - Actor="docudactyl-pipeline", Rationale="Batch extraction run {id}" + - Provenance: source file path, extraction timestamp, OCR confidence +.... + +==== Integration 2: Lithoglyph → Bofig (L3 + B5) + +.... +Lithoglyph evidence/entity/transaction collections + → Bofig GenServer (B5) queries Lithoglyph via GQL + → Maps to ArangoDB graph (Phase 2) or direct Lithoglyph queries (Phase 3) + → PROMPT scores flow from Lithoglyph → bofig UI + → Provenance metadata available on hover/click in UI +.... + +==== Integration 3: Entity Resolution Loop (D3/D4/D5 → L5 → B1) + +.... +Docudactyl NER extracts raw entities + → Lithoglyph stores with alias tracking (L5) + → Bofig entity resolution (B1) merges: + "J. Epstein" + "Jeffrey Epstein" + "Epstein, Jeffrey" → one node + → Merge decision logged in Lithoglyph journal with rationale + → Reversible if co-reference was incorrect +.... + +==== Integration 4: Financial Flow Analysis (D4 → L4 → B2) + +.... +Docudactyl extracts transactions from bank records (D4) + → Lithoglyph financial_transactions collection (L4) + → Bofig GraphQL: transactionChain(entityId, depth) (B2) + → D3 Sankey diagram: money flow visualization + → Anomaly flagging: unusual patterns surface automatically +.... + +==== Integration 5: Temporal Reconstruction (D3 → L7 → B3) + +.... +Docudactyl extracts dates from all documents + → Lithoglyph stores with temporal metadata + → Bofig timeline view (B3): + - What happened when + - What was known when + - Source credibility at each point in time (L7) +.... + +''''' + +=== Execution Phases + +==== Phase A: Foundation (Weeks 1-4) + +*Goal:* End-to-end pipeline working with test data + +* D1: Multi-locale cluster test (docudactyl v0.4.1 release) +* L1: Zig API migration (unblocks Lithoglyph REST) +* L2: Lithoglyph rename +* L3: Evidence collection schema +* B5: Batch evidence import + +*Milestone:* 100 test documents flow from Docudactyl → Lithoglyph → +Bofig + +==== Phase B: Entity & Financial (Weeks 5-8) + +*Goal:* Cross-document intelligence + +* D2: Cap’n Proto → Lithoglyph adapter +* L5: Entity collection + co-reference +* L6: Ingest bridge +* B1: Entity resolution module +* L4: Financial transaction collection +* B2: Financial transaction graph + GraphQL +* B3: Timeline visualization + +*Milestone:* Entity graph links people/orgs across 1000+ documents + +==== Phase C: Investigation Features (Weeks 9-14) + +*Goal:* Production-ready for real investigations + +* D3: Legal document NER +* D4: Financial record extraction +* D5: Speaker identification +* B4: Witness testimony module +* B6: Contradiction dashboard +* B7: Multi-investigation dashboard +* B9: Full-text search +* B10: RBAC + sensitivity + +*Milestone:* Ready for NUJ journalist user testing with real +investigation data + +==== Phase D: Scale & Trust (Weeks 15-20) + +*Goal:* Audit-grade, formally verified + +* L7: Temporal credibility +* L8: Cross-investigation linking +* B8: Real-time collaboration +* B11: Redaction audit trail +* B12: IPFS provenance +* B13: Lithoglyph provenance layer (replace ArangoDB) +* D6: Redaction detection +* B14: Public API + SDK + +*Milestone:* Production deployment on Hetzner, NUJ partnership + +''''' + +=== Epstein Files — Worked Example + +To process the Epstein files (~200K documents) through this pipeline: + +[arabic] +. *Ingest:* Docudactyl processes all PDFs/images across HPC cluster +* OCR on scanned court filings (GPU-accelerated) +* NER extracts: people (witnesses, lawyers, judges), dates, locations, +organizations +* Financial extraction: amounts, accounts, transaction dates +* ~3.7 hours cold run on 256 nodes +. *Store:* Lithoglyph receives structured output +* Each document becomes an evidence record with PROMPT scores +* Entity co-reference resolution: "`Jane Doe 3`" linked across 47 +documents +* Financial transactions form their own collection +* Every import logged with provenance (source file, extraction +confidence) +. *Graph:* Bofig builds the evidence graph +* Claims extracted from depositions and filings +* Relationships weighted by evidence quality (PROMPT) +* Contradictions automatically surfaced +* Financial flows visualized as Sankey diagrams +. *Navigate:* Six audience-specific paths +* Journalist: balanced credibility view, source protection +* Researcher: methodology-focused, reproducibility scores +* Policymaker: authoritative sources first, regulatory implications +* Affected person: personal impact stories, clear language +* Skeptic: transparency-focused, verification steps +* Activist: evidence quality, actionable findings +. *Audit:* Every step is reversible and traceable +* "`Why do we believe claim X?`" → full evidence chain with PROMPT +scores +* "`When did we learn fact Y?`" → time-travel query to discovery date +* "`Who added evidence Z?`" → actor + rationale from Lithoglyph journal +* Evidence discredited → reversible operation with explanation + +''''' + +=== Also Noted: Axiom.jl Outstanding Work + +File: +`+developer-ecosystem/julia-ecosystem/packages/Axiom.jl/TODO-URGENT-COPROCESSOR-CONSOLIDATION.md+` + +*Status: OUTSTANDING (not stale)* - Created 2026-02-27, marked "`NOT +STARTED`" - Proposes splitting `+abstract.jl+` (3,059 lines, 195 +definitions) into 7 files - The file has actually grown since the TODO +was written - Not blocking anything but increasingly needed as the file +becomes unwieldy - Recommend scheduling this as a separate refactoring +session diff --git a/docs/INTEGRATION-PLAN.md b/docs/INTEGRATION-PLAN.md deleted file mode 100644 index 6f3a6ee..0000000 --- a/docs/INTEGRATION-PLAN.md +++ /dev/null @@ -1,271 +0,0 @@ -# Investigative Evidence Pipeline — Integration Plan -# Bofig + Docudactyl + Lithoglyph -# -# SPDX-License-Identifier: CC-BY-SA-4.0 -# Author: Jonathan D.A. Jewell -# Created: 2026-03-11 - -## Vision - -A complete pipeline from raw document dumps (court filings, flight logs, -financial records, testimony transcripts) through HPC extraction, into an -audit-grade database, rendered as a navigable evidence graph with -audience-specific views. - -``` -Raw Documents (200K+ files) - → Docudactyl (HPC extraction: OCR, NER, metadata, classification) - → Lithoglyph (audit-grade storage: provenance, reversibility, PROMPT) - → Bofig (evidence graph: claims, relationships, navigation) - → 6 audience views (journalist, researcher, policymaker, ...) -``` - ---- - -## Repo Assignments - -### Repo 1: Docudactyl (bofig/docudactyl/) - -**Role in pipeline:** Ingestion layer. Takes raw documents and produces -structured extraction results. - -**Status:** v0.4.0, 97% complete. Awaiting multi-locale cluster testing. - -| # | Task | Priority | Effort | Notes | -|---|------|----------|--------|-------| -| D1 | Multi-locale HPC cluster test (GASNet/IBV, 4+ nodes) | Critical | Medium | Only v0.4.1 blocker | -| D2 | Cap'n Proto → Lithoglyph output adapter | High | Medium | New output stage that emits GQL-compatible evidence records | -| D3 | Legal document NER model | High | Medium | Docket numbers, case names, judge names, legal citations | -| D4 | Financial record extraction stage | High | Medium | Transaction amounts, dates, account identifiers, counterparties | -| D5 | Speaker identification stage (testimony/depositions) | Medium | Large | Who said what — maps to witness testimony in bofig | -| D6 | Redaction detection stage | Medium | Small | Flag redacted regions, track unredaction over time | -| D7 | British Library pilot (170M items) | Low | Large | v1.0.0 milestone | - -**Key output contract:** Each processed document produces: -- Extracted text (OCR'd if needed) + confidence score -- NER entities (people, orgs, locations, dates, amounts) -- SHA-256 + perceptual hash (dedup) -- Metadata (Dublin Core + format-specific) -- Auto-PROMPT scores derived from extraction quality -- Language, keywords, citations - ---- - -### Repo 2: Lithoglyph (nextgen-databases/lithoglyph/) - -**Role in pipeline:** Provenance layer. Stores all evidence with full audit -trail, reversibility, and PROMPT scoring. - -**Status:** 75% complete. L1/L2/L3 complete. formdb-http/ renamed to lith-http/. - -| # | Task | Priority | Effort | Notes | -|---|------|----------|--------|-------| -| L1 | Zig 0.15.2 HTTP API migration (83 call sites) | Critical | Medium | **COMPLETE** — Reader/Writer pattern applied | -| L2 | FormBD → Lithoglyph rename (Google trademark) | Critical | Small | **COMPLETE** — formdb-http/ → lith-http/ | -| L3 | Evidence collection schema for bofig | High | Small | **COMPLETE** — 5 collections in glyphbase/examples/bofig-evidence.json + 5 FQL test vectors | -| L4 | Financial transaction collection | High | Small | source, destination, amount, date, instrument, intermediary | -| L5 | Entity collection + co-reference resolution | High | Medium | Person/org/location entities with alias tracking | -| L6 | Ingest bridge: Docudactyl Cap'n Proto → GQL INSERT | High | Medium | Batch import with auto-PROMPT scoring from extraction metadata | -| L7 | Temporal credibility model | Medium | Medium | Source reputation updates over time (retractions, discrediting) | -| L8 | Cross-investigation linking | Medium | Small | Shared evidence collections, automatic surfacing | -| L9 | ControlPlane clustering (Elixir/OTP) | Low | Large | Multi-node Lithoglyph for scale | - -**Key guarantees Lithoglyph provides:** -- Every mutation has actor + rationale (accountability) -- Reversible operations (retractions with explanation) -- Time-travel queries ("what did we know on date X?") -- PROMPT scores as first-class citizens -- Constraints-as-ethics (invalid evidence relationships rejected with explanation) -- Dependent-type proofs (GQL-DT) for score bounds - ---- - -### Repo 3: Bofig (bofig/) - -**Role in pipeline:** Application layer. Evidence graph, PROMPT scoring, -audience-specific navigation, collaborative investigation. - -**Status:** Phase 1 complete (v1.0.0). 257 tests, 0 failures. - -| # | Task | Priority | Effort | Notes | -|---|------|----------|--------|-------| -| B1 | Entity resolution module | Critical | Medium | NER output → co-reference → unified entity graph | -| B2 | Financial transaction graph + GraphQL schema | High | Small | New queries: transaction chains, flow analysis, anomaly detection | -| B3 | Timeline D3 visualization | High | Small | Phase 2 planned — event reconstruction from extracted dates | -| B4 | Witness testimony module | High | Large | Speaker ID, claim extraction, corroboration scoring, impeachment detection | -| B5 | Batch evidence import (Docudactyl output) | High | Medium | GenServer consuming structured extraction results | -| B6 | Contradiction dashboard | High | Small | Surface conflicting accounts across documents automatically | -| B7 | Multi-investigation dashboard | Medium | Medium | Phase 2 — cross-referencing, shared evidence | -| B8 | Real-time collaborative editing (PubSub) | Medium | Medium | Phase 2 — multiple journalists mapping evidence simultaneously | -| B9 | Full-text search + faceted filters | Medium | Small | Phase 2 — ArangoDB fulltext indexes | -| B10 | RBAC + sensitivity layer | Medium | Medium | Phase 2 — protect sources, sealed material, graduated access | -| B11 | Redaction audit trail | Medium | Small | Track what was redacted/unredacted, by whom, integrated with Lithoglyph journal | -| B12 | IPFS provenance integration | Low | Medium | Phase 2 — tamper-proof evidence archival | -| B13 | Lithoglyph provenance layer | Low | Large | Phase 3 — supplement/replace ArangoDB with Lithoglyph for audit-grade storage | -| B14 | Public API + SDK | Low | Medium | Phase 3 | -| B15 | LEAN4 formal proofs for crypto protocols | Low | Large | Phase 3 | - ---- - -## Integration Points (How They Fit Together) - -### Integration 1: Docudactyl → Lithoglyph (D2 + L6) - -``` -Docudactyl Cap'n Proto output - → Adapter (D2) serializes to GQL INSERT statements - → Lithoglyph ingest bridge (L6) batch-imports with: - - Auto-PROMPT scoring from extraction confidence - - SHA-256 dedup against existing evidence - - Actor="docudactyl-pipeline", Rationale="Batch extraction run {id}" - - Provenance: source file path, extraction timestamp, OCR confidence -``` - -### Integration 2: Lithoglyph → Bofig (L3 + B5) - -``` -Lithoglyph evidence/entity/transaction collections - → Bofig GenServer (B5) queries Lithoglyph via GQL - → Maps to ArangoDB graph (Phase 2) or direct Lithoglyph queries (Phase 3) - → PROMPT scores flow from Lithoglyph → bofig UI - → Provenance metadata available on hover/click in UI -``` - -### Integration 3: Entity Resolution Loop (D3/D4/D5 → L5 → B1) - -``` -Docudactyl NER extracts raw entities - → Lithoglyph stores with alias tracking (L5) - → Bofig entity resolution (B1) merges: - "J. Epstein" + "Jeffrey Epstein" + "Epstein, Jeffrey" → one node - → Merge decision logged in Lithoglyph journal with rationale - → Reversible if co-reference was incorrect -``` - -### Integration 4: Financial Flow Analysis (D4 → L4 → B2) - -``` -Docudactyl extracts transactions from bank records (D4) - → Lithoglyph financial_transactions collection (L4) - → Bofig GraphQL: transactionChain(entityId, depth) (B2) - → D3 Sankey diagram: money flow visualization - → Anomaly flagging: unusual patterns surface automatically -``` - -### Integration 5: Temporal Reconstruction (D3 → L7 → B3) - -``` -Docudactyl extracts dates from all documents - → Lithoglyph stores with temporal metadata - → Bofig timeline view (B3): - - What happened when - - What was known when - - Source credibility at each point in time (L7) -``` - ---- - -## Execution Phases - -### Phase A: Foundation (Weeks 1-4) -**Goal:** End-to-end pipeline working with test data - -- D1: Multi-locale cluster test (docudactyl v0.4.1 release) -- L1: Zig API migration (unblocks Lithoglyph REST) -- L2: Lithoglyph rename -- L3: Evidence collection schema -- B5: Batch evidence import - -**Milestone:** 100 test documents flow from Docudactyl → Lithoglyph → Bofig - -### Phase B: Entity & Financial (Weeks 5-8) -**Goal:** Cross-document intelligence - -- D2: Cap'n Proto → Lithoglyph adapter -- L5: Entity collection + co-reference -- L6: Ingest bridge -- B1: Entity resolution module -- L4: Financial transaction collection -- B2: Financial transaction graph + GraphQL -- B3: Timeline visualization - -**Milestone:** Entity graph links people/orgs across 1000+ documents - -### Phase C: Investigation Features (Weeks 9-14) -**Goal:** Production-ready for real investigations - -- D3: Legal document NER -- D4: Financial record extraction -- D5: Speaker identification -- B4: Witness testimony module -- B6: Contradiction dashboard -- B7: Multi-investigation dashboard -- B9: Full-text search -- B10: RBAC + sensitivity - -**Milestone:** Ready for NUJ journalist user testing with real investigation data - -### Phase D: Scale & Trust (Weeks 15-20) -**Goal:** Audit-grade, formally verified - -- L7: Temporal credibility -- L8: Cross-investigation linking -- B8: Real-time collaboration -- B11: Redaction audit trail -- B12: IPFS provenance -- B13: Lithoglyph provenance layer (replace ArangoDB) -- D6: Redaction detection -- B14: Public API + SDK - -**Milestone:** Production deployment on Hetzner, NUJ partnership - ---- - -## Epstein Files — Worked Example - -To process the Epstein files (~200K documents) through this pipeline: - -1. **Ingest:** Docudactyl processes all PDFs/images across HPC cluster - - OCR on scanned court filings (GPU-accelerated) - - NER extracts: people (witnesses, lawyers, judges), dates, locations, organizations - - Financial extraction: amounts, accounts, transaction dates - - ~3.7 hours cold run on 256 nodes - -2. **Store:** Lithoglyph receives structured output - - Each document becomes an evidence record with PROMPT scores - - Entity co-reference resolution: "Jane Doe 3" linked across 47 documents - - Financial transactions form their own collection - - Every import logged with provenance (source file, extraction confidence) - -3. **Graph:** Bofig builds the evidence graph - - Claims extracted from depositions and filings - - Relationships weighted by evidence quality (PROMPT) - - Contradictions automatically surfaced - - Financial flows visualized as Sankey diagrams - -4. **Navigate:** Six audience-specific paths - - Journalist: balanced credibility view, source protection - - Researcher: methodology-focused, reproducibility scores - - Policymaker: authoritative sources first, regulatory implications - - Affected person: personal impact stories, clear language - - Skeptic: transparency-focused, verification steps - - Activist: evidence quality, actionable findings - -5. **Audit:** Every step is reversible and traceable - - "Why do we believe claim X?" → full evidence chain with PROMPT scores - - "When did we learn fact Y?" → time-travel query to discovery date - - "Who added evidence Z?" → actor + rationale from Lithoglyph journal - - Evidence discredited → reversible operation with explanation - ---- - -## Also Noted: Axiom.jl Outstanding Work - -File: `developer-ecosystem/julia-ecosystem/packages/Axiom.jl/TODO-URGENT-COPROCESSOR-CONSOLIDATION.md` - -**Status: OUTSTANDING (not stale)** -- Created 2026-02-27, marked "NOT STARTED" -- Proposes splitting `abstract.jl` (3,059 lines, 195 definitions) into 7 files -- The file has actually grown since the TODO was written -- Not blocking anything but increasingly needed as the file becomes unwieldy -- Recommend scheduling this as a separate refactoring session diff --git a/docs/database-evaluation.md b/docs/database-evaluation.adoc similarity index 67% rename from docs/database-evaluation.md rename to docs/database-evaluation.adoc index dcbdf2a..c54105e 100644 --- a/docs/database-evaluation.md +++ b/docs/database-evaluation.adoc @@ -1,51 +1,63 @@ -# Database Evaluation: ArangoDB vs SurrealDB vs Virtuoso +== Database Evaluation: ArangoDB vs SurrealDB vs Virtuoso -## Executive Summary +=== Executive Summary -**Winner: ArangoDB** for Phase 1-2, with optional Virtuoso addition in Phase 3+ +*Winner: ArangoDB* for Phase 1-2, with optional Virtuoso addition in +Phase 3+ -| Criteria | ArangoDB | SurrealDB | Virtuoso | -|----------|----------|-----------|----------| -| **Multi-model** | ✅ Native | ✅ Native | ❌ RDF-only | -| **Graph queries** | ✅ AQL | ✅ SurrealQL | ✅ SPARQL | -| **Elixir support** | ✅ arangox | ⚠️ HTTP only | ⚠️ SPARQL HTTP | -| **Production ready** | ✅ Yes | ⚠️ New (2022) | ✅ Yes | -| **Managed hosting** | ✅ €45/month | ❌ Self-host | ✅ €€€ | -| **JSON storage** | ✅ Native | ✅ Native | ❌ RDF triples | -| **Semantic web** | ⚠️ Via export | ⚠️ Via export | ✅ Native | -| **Learning curve** | Medium | Medium | Steep (RDF) | +[cols=",,,",options="header",] +|=== +|Criteria |ArangoDB |SurrealDB |Virtuoso +|*Multi-model* |✅ Native |✅ Native |❌ RDF-only +|*Graph queries* |✅ AQL |✅ SurrealQL |✅ SPARQL +|*Elixir support* |✅ arangox |⚠️ HTTP only |⚠️ SPARQL HTTP +|*Production ready* |✅ Yes |⚠️ New (2022) |✅ Yes +|*Managed hosting* |✅ €45/month |❌ Self-host |✅ €€€ +|*JSON storage* |✅ Native |✅ Native |❌ RDF triples +|*Semantic web* |⚠️ Via export |⚠️ Via export |✅ Native +|*Learning curve* |Medium |Medium |Steep (RDF) +|=== ---- +''''' -## ArangoDB +=== ArangoDB -### Strengths -1. **Multi-model done right**: Document + Graph + Key-Value in one database -2. **AQL is intuitive**: SQL-like syntax for graph traversals -3. **Strong Elixir integration**: `arangox` library mature and maintained -4. **Production-proven**: Used by companies like Cisco, Barclays -5. **Managed hosting**: ArangoDB Oasis (€45/month includes backups, monitoring) -6. **JSON-first**: Schema-less documents, perfect for Evidence metadata +==== Strengths -### Weaknesses -1. **Cost**: Not free at scale (but €45/month is reasonable) -2. **No native RDF**: Must export to JSON-LD for semantic web -3. **Smaller community**: Than PostgreSQL/Neo4j +[arabic] +. *Multi-model done right*: Document + Graph + Key-Value in one database +. *AQL is intuitive*: SQL-like syntax for graph traversals +. *Strong Elixir integration*: `+arangox+` library mature and maintained +. *Production-proven*: Used by companies like Cisco, Barclays +. *Managed hosting*: ArangoDB Oasis (€45/month includes backups, +monitoring) +. *JSON-first*: Schema-less documents, perfect for Evidence metadata -### Sample Queries +==== Weaknesses -#### Create Collections -```javascript +[arabic] +. *Cost*: Not free at scale (but €45/month is reasonable) +. *No native RDF*: Must export to JSON-LD for semantic web +. *Smaller community*: Than PostgreSQL/Neo4j + +==== Sample Queries + +===== Create Collections + +[source,javascript] +---- // Setup db._createDocumentCollection("investigations"); db._createDocumentCollection("claims"); db._createDocumentCollection("evidence"); db._createDocumentCollection("navigation_paths"); db._createEdgeCollection("relationships"); -``` +---- -#### Insert Data -```javascript +===== Insert Data + +[source,javascript] +---- // Insert investigation db.investigations.insert({ _key: "uk_inflation_2023", @@ -164,10 +176,12 @@ db.relationships.insert({ reasoning: "Energy prices provide context for overall inflation rate", created_by: "sarah.johnson@example.com" }); -``` +---- + +===== Query: Find All Evidence Supporting a Claim -#### Query: Find All Evidence Supporting a Claim -```javascript +[source,javascript] +---- FOR claim IN claims FILTER claim._key == "claim_1" FOR v, e, p IN 1..1 OUTBOUND claim relationships @@ -188,10 +202,12 @@ FOR claim IN claims v.prompt_scores.transparency * 0.15 ) } -``` +---- + +*Output:* -**Output:** -```json +[source,json] +---- { "claim": "UK inflation reached 40-year high of 11.1% in October 2022", "evidence": "Consumer Price Index Data - October 2022", @@ -200,10 +216,12 @@ FOR claim IN claims "reasoning": "ONS CPI data directly confirms the 11.1% figure", "prompt_overall": 97.5 } -``` +---- -#### Query: Evidence Chain (Multi-Hop Traversal) -```javascript +===== Query: Evidence Chain (Multi-Hop Traversal) + +[source,javascript] +---- // Find all nodes within 3 hops of a claim FOR claim IN claims FILTER claim._key == "claim_1" @@ -215,10 +233,12 @@ FOR claim IN claims depth: LENGTH(p.edges), path_weight: PRODUCT(p.edges[*].weight) } -``` +---- + +*Output:* -**Output:** -```json +[source,json] +---- [ { "node_type": "evidence", @@ -242,10 +262,12 @@ FOR claim IN claims "path_weight": 0.56 } ] -``` +---- + +===== Query: Find Contradictions -#### Query: Find Contradictions -```javascript +[source,javascript] +---- FOR claim IN claims FILTER claim.investigation_id == "uk_inflation_2023" LET supporting = ( @@ -266,10 +288,12 @@ FOR claim IN claims net_support: SUM(supporting[*].weight) - SUM(contradicting[*].weight), contradictions: contradicting[*].evidence.title } -``` +---- -#### Query: Audience-Weighted PROMPT Scores -```javascript +===== Query: Audience-Weighted PROMPT Scores + +[source,javascript] +---- // Researcher perspective (prioritizes methodology, replicability) LET researcher_weights = { provenance: 0.15, @@ -296,10 +320,12 @@ FOR evidence IN evidence researcher_score: researcher_score, scores: evidence.prompt_scores } -``` +---- + +===== Query: Full-Text Search -#### Query: Full-Text Search -```javascript +[source,javascript] +---- // Requires fulltext index: db.evidence.ensureIndex({ type: "fulltext", fields: ["title", "tags"] }); @@ -309,11 +335,12 @@ FOR doc IN FULLTEXT(evidence, "title,tags", "inflation,energy") tags: doc.tags, score: BM25(doc) // Built-in relevance scoring } -``` +---- -### Elixir Integration (arangox) +==== Elixir Integration (arangox) -```elixir +[source,elixir] +---- # mix.exs {:arangox, "~> 0.5.2"} @@ -376,28 +403,34 @@ defmodule EvidenceGraph.ArangoDB do ) end end -``` +---- ---- +''''' -## SurrealDB +=== SurrealDB -### Strengths -1. **Modern architecture**: Built with Rust, async-first -2. **Multi-model**: Documents + Graph + Realtime -3. **GraphQL-like syntax**: Familiar to web developers -4. **Embedded mode**: Can run in-process (great for testing) -5. **Permissions system**: Fine-grained access control built-in +==== Strengths -### Weaknesses -1. **New/immature**: Only released in 2022, smaller ecosystem -2. **No managed hosting**: Must self-host (more ops burden) -3. **Elixir support lacking**: No native driver, must use HTTP API -4. **Limited production deployments**: Unproven at scale -5. **Smaller community**: Fewer Stack Overflow answers +[arabic] +. *Modern architecture*: Built with Rust, async-first +. *Multi-model*: Documents + Graph + Realtime +. *GraphQL-like syntax*: Familiar to web developers +. *Embedded mode*: Can run in-process (great for testing) +. *Permissions system*: Fine-grained access control built-in -### Sample Query (SurrealQL) -```sql +==== Weaknesses + +[arabic] +. *New/immature*: Only released in 2022, smaller ecosystem +. *No managed hosting*: Must self-host (more ops burden) +. *Elixir support lacking*: No native driver, must use HTTP API +. *Limited production deployments*: Unproven at scale +. *Smaller community*: Fewer Stack Overflow answers + +==== Sample Query (SurrealQL) + +[source,sql] +---- -- Create claim with relationship CREATE claims:claim_1 SET text = "UK inflation reached 11.1%", @@ -414,31 +447,41 @@ RELATE claims:claim_1->supports->evidence:ons_cpi_data SET -- Traverse graph SELECT ->supports->evidence.* FROM claims:claim_1; -``` +---- + +==== Verdict + +*Wait and see.* Exciting tech, but too risky for Phase 1. Revisit in +2026 if community grows. -### Verdict -**Wait and see.** Exciting tech, but too risky for Phase 1. Revisit in 2026 if community grows. +''''' ---- +=== Virtuoso -## Virtuoso +==== Strengths -### Strengths -1. **RDF native**: Best-in-class SPARQL performance -2. **Semantic web integration**: Direct compatibility with academic repositories -3. **Mature**: 20+ years of development -4. **Federated queries**: Can query across multiple SPARQL endpoints -5. **Linked Open Data**: Natural fit for cross-investigation linking +[arabic] +. *RDF native*: Best-in-class SPARQL performance +. *Semantic web integration*: Direct compatibility with academic +repositories +. *Mature*: 20+ years of development +. *Federated queries*: Can query across multiple SPARQL endpoints +. *Linked Open Data*: Natural fit for cross-investigation linking -### Weaknesses -1. **RDF-only**: Documents must be converted to triples (awkward for JSON) -2. **Steep learning curve**: SPARQL + ontologies require semantic web knowledge -3. **Overkill for Phase 1**: Don't need semantic web until Phase 3 -4. **Schema rigidity**: RDF schemas less flexible than JSON documents -5. **Elixir support**: HTTP SPARQL only, no native driver +==== Weaknesses -### Sample Query (SPARQL) -```sparql +[arabic] +. *RDF-only*: Documents must be converted to triples (awkward for JSON) +. *Steep learning curve*: SPARQL + ontologies require semantic web +knowledge +. *Overkill for Phase 1*: Don’t need semantic web until Phase 3 +. *Schema rigidity*: RDF schemas less flexible than JSON documents +. *Elixir support*: HTTP SPARQL only, no native driver + +==== Sample Query (SPARQL) + +[source,sparql] +---- PREFIX eg: PREFIX dc: @@ -452,18 +495,20 @@ WHERE { ?evidence dc:title ?evidenceTitle . } ORDER BY DESC(?weight) -``` +---- + +==== Verdict -### Verdict -**Phase 3 addition.** Store JSON-LD from day 1 in ArangoDB. Export to Virtuoso only if semantic web features become critical. +*Phase 3 addition.* Store JSON-LD from day 1 in ArangoDB. Export to +Virtuoso only if semantic web features become critical. ---- +''''' -## Hybrid Approach: The Winner +=== Hybrid Approach: The Winner -### Architecture B (Progressive Enhancement) +==== Architecture B (Progressive Enhancement) -``` +.... Phase 1-2: ArangoDB only ├── Store documents as JSON (flexible schema) ├── Store JSON-LD context for future RDF export @@ -475,10 +520,12 @@ Phase 3+: Add Virtuoso (optional) ├── Add SPARQL endpoint for academic integration ├── Keep ArangoDB as primary database └── Virtuoso as read-only semantic web layer -``` +.... -### Data Format (JSON-LD Ready) -```json +==== Data Format (JSON-LD Ready) + +[source,json] +---- { "@context": "https://schema.org", "@type": "Claim", @@ -499,21 +546,19 @@ Phase 3+: Add Virtuoso (optional) "replicability": 100 } } -``` +---- -**Benefits:** -- Start simple (ArangoDB) -- Preserve migration path (JSON-LD) -- Add semantic web when needed -- No vendor lock-in +*Benefits:* - Start simple (ArangoDB) - Preserve migration path +(JSON-LD) - Add semantic web when needed - No vendor lock-in ---- +''''' -## Local Testing Guide +=== Local Testing Guide -### Setup ArangoDB (Podman) +==== Setup ArangoDB (Podman) -```bash +[source,bash] +---- # Pull image podman pull arangodb/arangodb:3.11 @@ -528,11 +573,12 @@ podman run -d \ open http://localhost:8529 # Username: root # Password: dev -``` +---- -### Load Test Data +==== Load Test Data -```bash +[source,bash] +---- # Create database curl -X POST http://localhost:8529/_db/_system/_api/database \ -u root:dev \ @@ -541,11 +587,12 @@ curl -X POST http://localhost:8529/_db/_system/_api/database \ # Create collections (see queries above) # Use ArangoDB web UI > Collections > New Collection -``` +---- -### Benchmark Queries +==== Benchmark Queries -```javascript +[source,javascript] +---- // In ArangoDB web UI > Queries tab // Test 1: Simple claim lookup (should be <10ms) @@ -568,42 +615,45 @@ FOR evidence IN evidence COLLECT investigation = evidence.investigation_id AGGREGATE avg_provenance = AVG(evidence.prompt_scores.provenance) RETURN {investigation, avg_provenance} -``` +---- + +==== Expected Performance (Local) -### Expected Performance (Local) -- Simple lookup: ~5ms -- 3-hop traversal (100 nodes): ~50ms -- Full-text search: ~100ms -- Aggregations: ~30ms +* Simple lookup: ~5ms +* 3-hop traversal (100 nodes): ~50ms +* Full-text search: ~100ms +* Aggregations: ~30ms ---- +''''' -## Decision Matrix +=== Decision Matrix -| Use Case | ArangoDB | SurrealDB | Virtuoso | -|----------|----------|-----------|----------| -| Flexible JSON storage | ✅ Perfect | ✅ Good | ❌ Poor (triples) | -| Graph traversals | ✅ AQL excellent | ✅ Good | ✅ SPARQL excellent | -| Elixir integration | ✅ arangox mature | ⚠️ HTTP only | ⚠️ SPARQL HTTP | -| Production hosting | ✅ Managed €45 | ❌ Self-host | ✅ Managed €€€ | -| Learning curve | ✅ Medium | ✅ Medium | ❌ Steep | -| Academic integration | ⚠️ Export needed | ⚠️ Export needed | ✅ Native RDF | -| Cost | ✅ €540/year | ✅ Free (self-host) | ❌ €€€ | -| Risk | ✅ Low | ⚠️ High (new) | ✅ Low (mature) | +[width="100%",cols="26%,24%,26%,24%",options="header",] +|=== +|Use Case |ArangoDB |SurrealDB |Virtuoso +|Flexible JSON storage |✅ Perfect |✅ Good |❌ Poor (triples) +|Graph traversals |✅ AQL excellent |✅ Good |✅ SPARQL excellent +|Elixir integration |✅ arangox mature |⚠️ HTTP only |⚠️ SPARQL HTTP +|Production hosting |✅ Managed €45 |❌ Self-host |✅ Managed €€€ +|Learning curve |✅ Medium |✅ Medium |❌ Steep +|Academic integration |⚠️ Export needed |⚠️ Export needed |✅ Native RDF +|Cost |✅ €540/year |✅ Free (self-host) |❌ €€€ +|Risk |✅ Low |⚠️ High (new) |✅ Low (mature) +|=== -**Final Decision:** ArangoDB for Phase 1-2, evaluate Virtuoso in Phase 3. +*Final Decision:* ArangoDB for Phase 1-2, evaluate Virtuoso in Phase 3. ---- +''''' -## References +=== References -- ArangoDB Docs: https://www.arangodb.com/docs/stable/ -- SurrealDB Docs: https://surrealdb.com/docs -- Virtuoso Docs: http://virtuoso.openlinksw.com/ -- JSON-LD: https://json-ld.org/ -- arangox GitHub: https://github.com/ArangoDB-Community/arangox +* ArangoDB Docs: https://www.arangodb.com/docs/stable/ +* SurrealDB Docs: https://surrealdb.com/docs +* Virtuoso Docs: http://virtuoso.openlinksw.com/ +* JSON-LD: https://json-ld.org/ +* arangox GitHub: https://github.com/ArangoDB-Community/arangox ---- +''''' -**Last Updated:** 2025-11-22 -**Next Review:** Month 3 (benchmark decision point) +*Last Updated:* 2025-11-22 *Next Review:* Month 3 (benchmark decision +point) diff --git a/docs/tech-debt-2026-05-26.adoc b/docs/tech-debt-2026-05-26.adoc new file mode 100644 index 0000000..adf7c01 --- /dev/null +++ b/docs/tech-debt-2026-05-26.adoc @@ -0,0 +1,66 @@ +== Tech-Debt Audit — bofig — 2026-05-26 + +*Source:* estate-wide automated scan 2026-05-26. *Companion:* +https://github.com/hyperpolymath/standards/tree/main/docs/audits[`+hyperpolymath/standards+` +2026-05-26-estate-*-debt audits]. *Combined severity:* `+2026-05-22+`. + +This file records the _raw findings_ — it does not by itself fix the +debt. Each section ends with a '`Recommended next move`' line; closing +the debt is follow-up work. + +=== 1. Proof debt + +Scanner counted the following markers in proof-bearing files of this +repo: + +.... +files= 6 | Coq-Axm/Adm= 0 | Lean-srry/ax= 0 | Agda-pst= 0 | Idr-blv= 0 | Idr-prtl= 0 | Fstr-asm= 0 | TODO= 0 | Unsafe= 0 +.... + +*Total markers:* 0. *Severity:* `+>00+`. + +*Recommended next move:* none — no proof-debt markers detected. + +=== 2. Licence debt + +[cols=",",options="header",] +|=== +|Field |Value +|LICENSE file |`+LICENSE+` +|SPDX header |`+MPL-2.0+` +|Manifest licence |`+NONE+` +|Body classifier |`+Palimp-MPL-2.0+` +|Severity |`+ok+` +|=== + +*Recommended next move:* none for licence. + +=== 3. Documentation debt + +[cols=",",options="header",] +|=== +|Field |Value +|README lines |472 +|`+docs/+` files |10 +|`+docs/+` LoC |4175 +|CHANGELOG.md |Y +|CONTRIBUTING.md |Y +|CODE_OF_CONDUCT.md |Y +|SECURITY.md |Y +|Severity |`+readme=472 docs=10/4175+` +|=== + +=== Cross-references + +* Estate proof-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md+` +* Estate licence-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md+` +* Estate documentation-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md+` + +''''' + +🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). +This file is informational — closing the debt is follow-up work owned by +the maintainer. diff --git a/docs/tech-debt-2026-05-26.md b/docs/tech-debt-2026-05-26.md deleted file mode 100644 index 63321c1..0000000 --- a/docs/tech-debt-2026-05-26.md +++ /dev/null @@ -1,60 +0,0 @@ - - -# Tech-Debt Audit — bofig — 2026-05-26 - -**Source:** estate-wide automated scan 2026-05-26. -**Companion:** [`hyperpolymath/standards` 2026-05-26-estate-*-debt audits](https://github.com/hyperpolymath/standards/tree/main/docs/audits). -**Combined severity:** `2026-05-22`. - -This file records the *raw findings* — it does not by itself fix the debt. Each section ends with a 'Recommended next move' line; closing the debt is follow-up work. - -## 1. Proof debt - -Scanner counted the following markers in proof-bearing files of this repo: - -``` -files= 6 | Coq-Axm/Adm= 0 | Lean-srry/ax= 0 | Agda-pst= 0 | Idr-blv= 0 | Idr-prtl= 0 | Fstr-asm= 0 | TODO= 0 | Unsafe= 0 -``` - -**Total markers:** 0. **Severity:** `>00`. - -**Recommended next move:** none — no proof-debt markers detected. - -## 2. Licence debt - -| Field | Value | -|---|---| -| LICENSE file | `LICENSE` | -| SPDX header | `MPL-2.0` | -| Manifest licence | `NONE` | -| Body classifier | `Palimp-MPL-2.0` | -| Severity | `ok` | - -**Recommended next move:** none for licence. - -## 3. Documentation debt - -| Field | Value | -|---|---| -| README lines | 472 | -| `docs/` files | 10 | -| `docs/` LoC | 4175 | -| CHANGELOG.md | Y | -| CONTRIBUTING.md | Y | -| CODE_OF_CONDUCT.md | Y | -| SECURITY.md | Y | -| Severity | `readme=472 docs=10/4175` | - - -## Cross-references - -- Estate proof-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md` -- Estate licence-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md` -- Estate documentation-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md` - ---- - -🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). This file is informational — closing the debt is follow-up work owned by the maintainer. diff --git a/docs/zotero-integration.md b/docs/zotero-integration.adoc similarity index 92% rename from docs/zotero-integration.md rename to docs/zotero-integration.adoc index e066679..70c7120 100644 --- a/docs/zotero-integration.md +++ b/docs/zotero-integration.adoc @@ -1,15 +1,15 @@ -# Zotero Integration Design +== Zotero Integration Design -## Overview +=== Overview -Two-way sync between Zotero and Evidence Graph: -- **Export:** Zotero items → Evidence Graph (via modified extension) -- **Import:** Evidence Graph → Zotero (via API + manual) -- **Metadata:** Preserve Dublin Core, Schema.org, custom fields +Two-way sync between Zotero and Evidence Graph: - *Export:* Zotero items +→ Evidence Graph (via modified extension) - *Import:* Evidence Graph → +Zotero (via API + manual) - *Metadata:* Preserve Dublin Core, +Schema.org, custom fields -## Architecture +=== Architecture -``` +.... ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ Zotero Client │ │ Evidence Graph │ │ ArangoDB │ │ │ │ Phoenix API │ │ │ @@ -28,16 +28,18 @@ Two-way sync between Zotero and Evidence Graph: │ │ js plugin │ │ import to Zotero) │ └───────────┘ │ └─────────────────┘ -``` +.... -## Zotero → Evidence Graph Export +=== Zotero → Evidence Graph Export -### Modified Extension: `lib/exporter.js` +==== Modified Extension: `+lib/exporter.js+` -The existing bofig repository contains an old Zotero→Voyant export plugin. We'll extend it to POST to Evidence Graph API. +The existing bofig repository contains an old Zotero→Voyant export +plugin. We’ll extend it to POST to Evidence Graph API. -#### Original Plugin Structure (2017) -``` +===== Original Plugin Structure (2017) + +.... lib/ ├── exporter.js # Main export logic ├── translators/ @@ -45,10 +47,11 @@ lib/ └── chrome/ └── content/ └── overlay.xul # Firefox UI overlay -``` +.... + +===== New Structure (2025+) -#### New Structure (2025+) -``` +.... lib/ ├── exporter.js # Extended with Evidence Graph POST ├── translators/ @@ -59,11 +62,12 @@ lib/ └── content/ ├── overlay.xul # Updated for modern Zotero └── prompt.xhtml # NEW: PROMPT scoring dialog -``` +.... -### Code: `lib/translators/evidencegraph.js` +==== Code: `+lib/translators/evidencegraph.js+` -```javascript +[source,javascript] +---- /** * Evidence Graph Translator * Exports Zotero items to Evidence Graph API @@ -318,13 +322,14 @@ const EvidenceGraphTranslator = { if (typeof module !== 'undefined') { module.exports = EvidenceGraphTranslator; } -``` +---- -### Code: `lib/chrome/content/prompt.xhtml` +==== Code: `+lib/chrome/content/prompt.xhtml+` PROMPT scoring dialog shown before export: -```xml +[source,xml] +---- @@ -410,13 +415,14 @@ PROMPT scoring dialog shown before export: -``` +---- -### Code: `lib/config.js` +==== Code: `+lib/config.js+` User configuration for API endpoint: -```javascript +[source,javascript] +---- const EvidenceGraphConfig = { getApiEndpoint() { return Zotero.Prefs.get('extensions.evidencegraph.apiEndpoint') || @@ -447,13 +453,14 @@ const EvidenceGraphConfig = { return Zotero.Prefs.get('extensions.evidencegraph.promptForScores') !== false; } }; -``` +---- -## Evidence Graph → Zotero Import +=== Evidence Graph → Zotero Import -### API Endpoint: `/api/v1/evidence/:id/export` +==== API Endpoint: `+/api/v1/evidence/:id/export+` -```elixir +[source,elixir] +---- # lib/evidence_graph_web/controllers/evidence_controller.ex defmodule EvidenceGraphWeb.EvidenceController do @@ -514,24 +521,27 @@ defmodule EvidenceGraphWeb.EvidenceController do end end end -``` +---- -### Manual Import Workflow +==== Manual Import Workflow -1. User clicks "Export to Zotero" in Evidence Graph UI -2. Browser downloads `evidence_123.json` -3. User opens Zotero → File → Import → Select JSON file -4. Zotero creates new item with metadata + PROMPT scores in "Extra" field +[arabic] +. User clicks "`Export to Zotero`" in Evidence Graph UI +. Browser downloads `+evidence_123.json+` +. User opens Zotero → File → Import → Select JSON file +. Zotero creates new item with metadata + PROMPT scores in "`Extra`" +field -## Sync Strategy +=== Sync Strategy -### Conflict Resolution +==== Conflict Resolution -**Scenario:** User updates item in both Zotero AND Evidence Graph +*Scenario:* User updates item in both Zotero AND Evidence Graph -**Solution:** Last-write-wins with version tracking +*Solution:* Last-write-wins with version tracking -```elixir +[source,elixir] +---- defmodule EvidenceGraph.Sync do def sync_from_zotero(evidence_id, zotero_item) do evidence = get_evidence!(evidence_id) @@ -551,13 +561,14 @@ defmodule EvidenceGraph.Sync do end end end -``` +---- -### Webhook Support (Future) +==== Webhook Support (Future) -Zotero doesn't support webhooks, but we can poll: +Zotero doesn’t support webhooks, but we can poll: -```elixir +[source,elixir] +---- # Run every 15 minutes via Oban job defmodule EvidenceGraph.Workers.ZoteroSync do use Oban.Worker, queue: :sync @@ -581,13 +592,14 @@ defmodule EvidenceGraph.Workers.ZoteroSync do :ok end end -``` +---- -## Testing Plan +=== Testing Plan -### Unit Tests +==== Unit Tests -```elixir +[source,elixir] +---- # test/evidence_graph/zotero_test.exs defmodule EvidenceGraph.ZoteroTest do @@ -633,11 +645,12 @@ defmodule EvidenceGraph.ZoteroTest do end end end -``` +---- -### Integration Tests +==== Integration Tests -```elixir +[source,elixir] +---- # test/evidence_graph_web/controllers/evidence_controller_test.exs defmodule EvidenceGraphWeb.EvidenceControllerTest do @@ -670,38 +683,42 @@ defmodule EvidenceGraphWeb.EvidenceControllerTest do assert get_resp_header(conn, "content-disposition") != [] end end -``` +---- + +=== User Documentation -## User Documentation +==== Setup Guide -### Setup Guide +*1. Install Zotero Extension* -**1. Install Zotero Extension** -```bash +[source,bash] +---- # Download from GitHub releases curl -LO https://github.com/Hyperpolymath/bofig/releases/latest/bofig-evidencegraph.xpi # In Zotero: Tools → Add-ons → Gear Icon → Install Add-on From File -``` +---- -**2. Configure API Endpoint** -``` +*2. Configure API Endpoint* + +.... Zotero → Edit → Preferences → Evidence Graph - API Endpoint: https://api.evidencegraph.org - API Key: [paste your key from Evidence Graph settings] - Default Investigation: [select from dropdown] -``` +.... + +*3. Export Items* -**3. Export Items** -``` +.... 1. Select items in Zotero library 2. Right-click → Export to Evidence Graph 3. (Optional) Adjust PROMPT scores in dialog 4. Click "Export" 5. Items appear in Evidence Graph investigation -``` +.... ---- +''''' -**Last Updated:** 2025-11-22 -**Status:** Design complete, implementation pending +*Last Updated:* 2025-11-22 *Status:* Design complete, implementation +pending diff --git a/llm-warmup-dev.adoc b/llm-warmup-dev.adoc new file mode 100644 index 0000000..a3c82db --- /dev/null +++ b/llm-warmup-dev.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — bofig (Developer) + +=== What is bofig? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-dev.md b/llm-warmup-dev.md deleted file mode 100644 index ae47b48..0000000 --- a/llm-warmup-dev.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — bofig (Developer) - -## What is bofig? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.adoc b/llm-warmup-user.adoc new file mode 100644 index 0000000..c276489 --- /dev/null +++ b/llm-warmup-user.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — bofig (User) + +=== What is bofig? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.md b/llm-warmup-user.md deleted file mode 100644 index 5d71772..0000000 --- a/llm-warmup-user.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — bofig (User) - -## What is bofig? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture