Unified Modular Runtime: Consolidate workers into continuum-core + Rust ORM #265
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Architecture: - StorageAdapter trait for database-agnostic operations - SqliteAdapter implementation using worker thread (rusqlite not Send+Sync) - DataModule ServiceModule with adapter cache keyed by dbPath - ALL commands require dbPath parameter - NO defaults, NO env vars, NO fallbacks - TypeScript owns path configuration, passes to Rust per-request Files: - orm/types.rs: DataRecord, StorageResult, CollectionSchema (ts-rs generated) - orm/query.rs: QueryOperator, StorageQuery, QueryBuilder - orm/adapter.rs: StorageAdapter trait, AdapterConfig, naming utils - orm/sqlite.rs: Full SQLite implementation with worker thread pattern - modules/data.rs: data/* command handlers with adapter cache Tests: 29 ORM + 2 DataModule = 31 tests passing BREAKING: TypeScript must now pass dbPath in all data/* commands
- RustVectorSearchClient: dbPath now required parameter (no defaultDbPath) - SqliteStorageAdapter: requires filename OR basePath+databaseName - RustStorageAdapter: stores dbPath as class property, passes to managers - VectorSearchAdapterBase: dbPath required in constructor - SqliteVectorSearchManager: dbPath required TypeScript adapters verified working. Rust ORM module exists but is NOT wired into main data flow yet - that's a separate task.
…fety Phase 0 of Rust ORM migration - creates unified entry point for all data ops: - ORM.ts: Unified data access layer wrapping DataDaemon - ORMConfig.ts: Feature flags for per-collection backend routing - ORMLogger.ts: Operation logging and metrics - Migrated 26 command files from DataDaemon.* to ORM.* - Migrated 6 daemon files from DataDaemon.* to ORM.* Rust fixes: - Added unsafe to 11 FFI functions that dereference raw pointers - Fixed generator to handle duplicate type exports across subdirs - Created connection_manager.rs for pool-per-database design - Created vector.rs for VectorSearchAdapter trait No fallbacks policy: backend selection is deterministic, no silent bypasses.
AIs were failing with "Unknown adapter type: default/ai/adapter" because the schema didn't list valid options. Now description includes: - WARNING that most commands use default DB automatically - Suggests data/list or data/read instead - Lists valid adapters: sqlite, json, vector, graph, rust - Example usage
- CallerDetector.ts: DataDaemon.read → ORM.read - PersonaUser.ts: DataDaemon.query/update/store → ORM.* - PersonaAutonomousLoop.ts: DataDaemon.query/store/update/read → ORM.* - PersonaTaskExecutor.ts: DataDaemon.query/store/update → ORM.* 111 violations remaining in system/ (non-test files)
Migrated 17 additional files from DataDaemon → ORM: - Core services: EmbeddingService - RAG builders: ChatRAGBuilder, CodebaseRAGBuilder - RAG sources: ConversationHistorySource, PersonaIdentitySource, SocialMediaRAGSource - User modules: PersonaMessageEvaluator, PersonaResponseGenerator, SelfTaskGenerator, TrainingBuffer, PersonaMemory - User types: BaseUser, HumanUser, AgentUser, UserIdentityResolver - Storage: SQLiteStateBackend - Genome: TrainingDatasetBuilder Files with jtagContext keep DataDaemon import for event context. All data operations now route through ORM layer.
- Create ORMRustClient.ts: IPC client for data/* commands to Rust DataModule - Remove SqlNamingConverter from ORM layer (Rust SqliteAdapter handles naming) - Add proper TypeScript types: RustStorageResult, RustDataRecord, RustIPCResponse - Add filter format conversion: TypeScript $eq/$gt → Rust eq/gt - Update ORMConfig: All collections configured for Rust, FORCE_TYPESCRIPT=true The ORM layer is now database-agnostic. Rust SqliteAdapter handles all snake_case/camelCase conversion internally. Store/update operations still failing when Rust enabled - needs further debugging.
- Generator extracts collection names from entities → typed COLLECTIONS constant - ORM methods now typed with CollectionName, not string (compile-time safety) - Runtime validation in data commands for user input before casting - Fixed all ts-rs warnings by removing skip_serializing_if (incompatible with TS) - Removed dead code (extract_db_path), added allow for planned priority_queue - PersonaRAGContextEntity wired into collection registry - Integration tests for Rust ORM backend
- Remove limit: -1 in AICostServerCommand (undefined = no limit for Rust ORM) - Remove duplicate pricing warning from adapter (PricingManager logs it) - Clean data.rs: remove 12+ debug eprintln per query, keep error logging - Clean runtime.rs: remove 8 debug eprintln per command route - Clean sqlite.rs: remove 6 debug eprintln per create operation Net: -42 lines of debug spam that was flooding logs
- ORM CRUD methods now route directly to Rust (no useRust checks) - Removed ~50 lines of dead console.log debug statements - Removed shouldShadow import (unused) - Updated header comments to reflect Rust-first architecture - Updated RUST-ORM-ARCHITECTURE.md with Phase 5 progress
- Add Events.emit() with proper jtagContext to ORM.store/update/remove - Events now route to browser for real-time UI updates - Data commands use ORM for Rust-backed operations
…ugh Rust - All maintenance operations now use ORMRustClient -> continuum-core DataModule - Added listCollections, clear, clearAll, truncate to ORMLogger operation types - Updated architecture doc with Phase 5 progress and Phase 6 performance issues - Verified: AIs responsive (7-8s), ORM routing confirmed via server logs
- 1.7GB database (143k ai_generations, 96k memories, 68k cognition) - Queries hitting 140-225ms on indexed tables - All personas share single ORMRustClient socket - Priority: timing instrumentation, archiving, compound indexes
- Log parse/adapter/query breakdown for operations >50ms - Helps identify whether slowness is in Rust DataModule or IPC layer - ORM shows 100ms+ but Rust parse is 0ms → IPC likely bottleneck
- Migrate logger worker to LoggerModule in continuum-core
- Migrate search worker to SearchModule with full command routing
- Add TypeScript wrappers for search/{list,execute,vector,params}
- Add RustCoreIPCClient search methods for IPC routing
- Disable standalone search worker in workers-config.json
- Fix TTS panic: use synthesize_speech_async() in VoiceModule
(avoids nested tokio runtime when called from async context)
- Remove timeout from route_command_sync for streaming voice
- Various clippy fixes across Rust modules
EmbeddingModule (absorbs standalone embedding worker):
- Created modules/embedding.rs (331 lines) implementing ServiceModule
- Commands: embedding/generate, embedding/model/{load,list,info,unload}
- Uses fastembed ONNX (~5ms vs 80ms Ollama)
- Binary protocol for f32 embedding vectors
- Disabled embedding worker in workers-config.json
Clippy fixes:
- Removed unused HashMap import in memory/mod.rs
- Fixed unused loop variables in orchestrator_tests.rs
- Prefixed unused vars with _ in rag.rs, connection_manager.rs
- Added #[allow(dead_code)] for test-only code
- Removed unused imports in call_server_routing_test.rs
SCSS deprecation fix:
- Updated live-widget.scss: darken() -> color.adjust($lightness: -N%)
Build status: 495 Rust tests pass, TypeScript clean, 25 SCSS files compile
The EmbeddingModule was absorbed into continuum-core, but the TypeScript
client was still using the old newline-delimited protocol. continuum-core
uses length-prefixed binary framing:
- JSON: [4 bytes u32 BE length][JSON payload bytes]
- Binary: [4 bytes u32 BE total_length][JSON header][\0][raw binary]
Changes:
- Socket path: /tmp/jtag-embedding.sock → /tmp/continuum-core.sock
- Ping: 'ping' command → 'health-check' command
- Response format: {status, data} → {success, result}
- Frame parsing: newline-delimited → length-prefixed with processFrames()
- Binary protocol: newline separator → null byte separator
Performance verified: 7ms per embedding (Rust/fastembed speed).
- PersonaResponseGenerator enriches context with userId for identity detection - Decision commands (rank, vote, propose, create) use context.userId first - Removed redundant params: voterId, proposerId, WithCaller interfaces - Other commands (dm, live/join, live/leave, ai/sleep, canvas/stroke/add) updated - SkillProposeServerCommand passes context instead of proposerId - PersonaToolExecutor: userId injection only for workspace-scoped commands - Documentation: docs/PERSONA-COGNITION-IDENTITY-REFACTORING.md
- Add vector/search command to DataModule with in-memory caching - VectorCache stores vectors per (dbPath, collection) for instant subsequent searches - Parallel cosine similarity using rayon with 4-way SIMD-style unrolling - Update RustVectorSearchClient to use continuum-core socket - Remove handle-based API (DataModule takes dbPath directly) - Update VectorSearchAdapterBase error message for continuum-core Note: data-daemon worker remains enabled due to DataDaemonServer dependency. RustWorkerStorageAdapter.ts identified as dead code (never instantiated).
- Remove connectRustDataWorker() from DataDaemonServer (dead code) - RustWorkerStorageAdapter was never used - ORM uses ORMRustClient - Disable data-daemon in workers-config.json - Vector search and all CRUD now route through continuum-core DataModule This completes the data layer migration to the unified runtime.
… in continuum-core
…ence (7,511 lines)
- Logger.ts, ConsoleDaemonServer.ts, LoggerDaemonServer.ts: update socket from /tmp/jtag-logger-worker.sock to /tmp/continuum-core.sock - LoggerWorkerClient.ts: update command names from write-log/ping to log/write and log/ping (ServiceModule prefix pattern) - WorkerClient.ts: update SKIP_TIMING_TYPES for new command names - Remove CommsTestDaemonServer from Logger level registry (daemon removed) - Remove dead getRustDataDaemonSocket() from ServerConfig.ts - Remove COMMS_TEST from generated-command-constants.ts - Update workers/README.md with current 3-worker structure - Add deprecation notices to 6 docs referencing old architecture: - LOGGER-DAEMON-VERIFICATION.md, RUST-ORM-ARCHITECTURE.md - ADAPTER-ARCHITECTURE.md, RUST-WORKER-IPC-PROTOCOL.md - RUST-WORKER-PATH-ANALYSIS.md, RUST-WORKER-ARCHITECTURE.md - Update logger-daemon README with Phase 4a architecture note LoggerModule now part of continuum-core (Phase 4a of modular runtime).
- SqlExecutor.ts: update comment noting RustSqliteExecutor removal - TimingServerCommand.ts: add note about data-daemon absorption - TimingTypes.ts: add deprecation note about timing file - Clean stale dist/ directories (compiled artifacts from removed workers) The timing command needs adaptation for continuum-core DataModule.
- Remove shared/ipc/training/ directory (training worker removed) - Remove shared/ipc/logger/generated/ directory (unused generated types) Both directories contained dead code from workers that were absorbed into continuum-core.
- LoggerDaemon.ts: update comment to reference continuum-core.sock - generate-logger-daemon.ts: update instructions for Phase 4a architecture - logger-daemon-spec.ts: update lifecycle socket paths - test-ffi.ts: use continuum-core.sock for initialization - test-voice-loop.ts: update server start instructions All logger references now point to the unified runtime.
LoggerModule was absorbed into continuum-core in Phase 4a, but main.rs still required an external logger socket argument. This fix: - Makes logger socket argument optional (only IPC socket required) - Updates main.rs doc comments to reflect unified runtime modules - Updates shared logger_client.rs to reference continuum-core socket - Updates logger_integration.rs test to use correct socket path - Updates PERFORMANCE.md to note internal logger The server now starts correctly with just one argument: continuum-core-server /tmp/continuum-core.sock
- ORM.vectorSearch now calls Rust vector/search via ORMRustClient - VectorSearchServerCommand updated to use ORM instead of DataDaemon - Rust vector search has in-memory caching and rayon parallel cosine similarity Dead code deleted (1,316 lines): - server/JsonFileStorageAdapter.ts (1,061 lines) - no imports - server/StorageAdapterFactory.ts (158 lines) - replaced by DefaultStorageAdapterFactory - server/test-sqlite.ts (97 lines) - standalone test file, no imports
- Move ORM.ts from shared/ to server/ (proper separation) - Add dbPath parameter to all ORM and ORMRustClient methods - Add DatabaseHandleRegistry.getDbPath() for handle → path resolution - Update data commands (list, create, update, vector-search) to use ORM - Add dbPath to VectorSearchOptions for per-persona database support - Update 60+ files to import from server/ORM instead of shared/ORM All CRUD operations now route: Command → ORM → ORMRustClient → Rust DataModule
- Route ORM.generateEmbedding() → RustEmbeddingClient → continuum-core - Uses fastembed (ONNX-based) for ~5ms embeddings vs ~80ms Ollama HTTP - Add generateEmbedding, indexVector, backfillVectors to ORMOperation type - Initialize metrics for new vector operations in ORMLogger Remaining TypeScript adapter usage: - indexVector, backfillVectors (need Rust implementation) - getVectorIndexStats, getVectorSearchCapabilities - Paginated queries (cursor management)
Routes additional ORM vector operations to Rust: - indexVector: stores embedding in record via vector/index command - getVectorIndexStats: retrieves vector stats via vector/stats command - invalidateVectorCache: clears Rust vector cache via vector/invalidate-cache Adds comprehensive Rust tests: - test_vector_index_and_stats: create, index, verify stats - test_vector_search_basic: 3-record search, verify similarity - test_vector_cache_invalidation: cache population and invalidation - test_cosine_similarity: unit test for similarity function All 6 Rust data module tests pass. TypeScript compilation verified. backfillVectors remains in TypeScript pending cross-module coordination.
- Add data/query-open, data/query-next, data/query-close commands to Rust - Server-side cursor management via DashMap (lock-free reads) - Wrap responses in StorageResult format for TypeScript compatibility - Add vector/stats response wrapping for consistent API - Fix all Rust tests for new response structure - Eliminate IPC overhead per page - state is maintained Rust-side Tests pass, deployed and verified working.
- Add vector/backfill command to Rust DataModule - Add public generate_embedding() and generate_embeddings_batch() to EmbeddingModule - Cross-module call: DataModule uses EmbeddingModule for batch embeddings - Process records in configurable batch sizes for memory efficiency - Add test_backfill_vectors test (passes with all 8 data module tests) - Update ORMRustClient and ORM.ts to route through Rust All vector operations now fully in Rust.
- Add IncludeSpec interface (JoinSpec is now an alias) - Add StorageQueryWithInclude interface with includes/joins properties - StorageQueryWithJoin remains as alias for backwards compatibility - Terminology aligns with Prisma/GraphQL (include vs SQL-specific join) Full property rename (joins→includes) deferred to SqliteStorageAdapter deletion.
DatabaseHandleRegistry no longer creates TypeScript SqliteStorageAdapter instances. It now acts as a handle → path mapping service only. Changes: - Remove adapter storage (Map<DbHandle, DataStorageAdapter>) - Convert to path-only registry that maps handles to database paths - Add typed DB_HANDLES constants to prevent magic strings - Deprecate getAdapter() - returns null with warning - Fix DataDaemon import: getRegisteredEntity from EntityRegistry - Remove unused getAdapter() call from ArchiveDaemonServer Architecture: - All data I/O now routes through ORM → ORMRustClient → Rust DataModule - DatabaseHandleRegistry provides getDbPath() for handle → path resolution - Database paths still come from ServerConfig (single source of truth) - SqliteStorageAdapter still exists but only used for schema initialization
Bug: When ORM.update() emitted data:*:updated events, it was sending
only the partial update data (e.g., {id, modelConfig: {provider: 'candle'}})
instead of the complete entity. This caused the browser UI to replace
full user objects with partial stubs, showing "Unknown User".
Fix: ORM.update() now fetches the full entity after the update completes
and emits that complete data in the event. This ensures the browser
receives all fields (displayName, userType, etc.) when processing updates.
Also adds suppressEvents parameter for future use in bulk operations
where event emission would be undesirable.
Now that all data operations go through ORM → ORMRustClient → Rust DataModule: Deleted managers/ (4 files): - SqliteQueryExecutor.ts (677 lines) - SqliteSchemaManager.ts (469 lines) - SqliteWriteManager.ts (432 lines) - SqliteVectorSearchManager.ts (315 lines) Deleted SQLite support files: - SqliteStorageAdapter.ts (1,043 lines) - SqlStorageAdapterBase.ts (210 lines) - SqliteQueryBuilder.ts (238 lines) - SqliteTransactionManager.ts (69 lines) - SqliteRawExecutor.ts (71 lines) - SqlExecutor.ts (27 lines) - SqlNamingConverter.ts (34 lines) Deleted tests importing deleted files (4 files): - database-backend-agnostic.test.ts (777 lines) - database-sqlite-integration.test.ts (630 lines) - jtag-sqlite.test.ts (260 lines) - sqlite-orm.test.ts (458 lines) Updated: - DefaultStorageAdapterFactory: 'sqlite' type returns MemoryStorageAdapter (no-op) - VectorSearchAdapterBase: pass collection directly to Rust (no SqlNamingConverter)
Dead files removed: - StorageMigrationService.ts - not imported anywhere - QueryBuilder.ts (daemons/data-daemon) - replaced by Rust ORM - QueryBuilder.ts (system/data/query) - not imported anywhere - query-builder.test.ts, querybuilder-debug.test.ts Outdated docs removed: - SQL-ADAPTER-DEBUGGING-RECOVERY-PLAN.md - SQL-ADAPTER-REFACTOR-PLAN.md - VECTOR-SEARCH-ARCHITECTURE.md (referenced deleted SqliteStorageAdapter)
Contributor
There was a problem hiding this comment.
Pull request overview
Consolidates multiple Node/Rust workers into the unified continuum-core runtime and routes TypeScript data operations through a Rust-backed ORM, removing large amounts of legacy adapter/manager code and updating commands/docs accordingly.
Changes:
- Consolidated worker IPC targets to
/tmp/continuum-core.sockand updated daemon/command wiring. - Migrated data access callsites from
DataDaemontoORM, plus added ORM config + logging utilities. - Added generators/docs for collection constants + Rust ORM/search modules; removed legacy SQLite adapter/manager infrastructure.
Reviewed changes
Copilot reviewed 124 out of 349 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/debug/jtag/generator/specs/logger-daemon-spec.ts | Updates logger daemon lifecycle text to new unified socket/module. |
| src/debug/jtag/generator/generate-rust-bindings.ts | Removes data-daemon-worker bindings and improves master barrel export generation. |
| src/debug/jtag/generator/generate-logger-daemon.ts | Updates generator output hints for continuum-core socket/module. |
| src/debug/jtag/generator/generate-collection-constants.ts | New generator for type-safe collection constants derived from entity definitions. |
| src/debug/jtag/generated-command-schemas.json | Regenerates schemas and adds new search commands + updates descriptions. |
| src/debug/jtag/docs/plans/RUST-WORKER-ARCHITECTURE.md | Marks older worker architecture doc as evolved with continuum-core consolidation. |
| src/debug/jtag/docs/architecture/RUST-WORKER-PATH-ANALYSIS.md | Marks analysis as historical and updates socket/command prefix info. |
| src/debug/jtag/docs/architecture/RUST-WORKER-IPC-PROTOCOL.md | Notes workers are consolidated while protocol remains. |
| src/debug/jtag/docs/RUST-ORM-ARCHITECTURE.md | New doc describing Rust-first ORM architecture and migration. |
| src/debug/jtag/docs/PERSONA-COGNITION-IDENTITY-REFACTORING.md | New doc describing identity flow standardization via context.userId. |
| src/debug/jtag/docs/LOGGER-DAEMON-VERIFICATION.md | Marks logger verification doc as historical; updates to continuum-core. |
| src/debug/jtag/docs/ADAPTER-ARCHITECTURE.md | Updates data adapater architecture to ORM → Rust DataModule. |
| src/debug/jtag/daemons/user-daemon/server/UserDaemonServer.ts | Migrates DataDaemon CRUD operations to ORM. |
| src/debug/jtag/daemons/training-daemon/server/TrainingDaemonServer.ts | Migrates storage calls to ORM and updates comments. |
| src/debug/jtag/daemons/system-daemon/shared/SystemDaemon.ts | Migrates config persistence/query to ORM. |
| src/debug/jtag/daemons/session-daemon/server/SessionStateHelper.ts | Migrates user state read to ORM. |
| src/debug/jtag/daemons/session-daemon/server/SessionDaemonServer.ts | Migrates user queries/reads to ORM. |
| src/debug/jtag/daemons/room-membership-daemon/server/RoomMembershipDaemonServer.ts | Migrates membership reconciliation reads/updates to ORM. |
| src/debug/jtag/daemons/logger-daemon/shared/LoggerDaemon.ts | Updates lifecycle comment to continuum-core LoggerModule/socket. |
| src/debug/jtag/daemons/logger-daemon/server/LoggerDaemonServer.ts | Updates socket path to /tmp/continuum-core.sock. |
| src/debug/jtag/daemons/logger-daemon/README.md | Updates logger daemon README to unified runtime + log command prefix. |
| src/debug/jtag/daemons/data-daemon/shared/VectorSearchTypes.ts | Adds dbPath and deprecates dbHandle for vector search options. |
| src/debug/jtag/daemons/data-daemon/shared/SqlNamingConverter.ts | Deletes TS-side SQL naming converter (now handled in Rust). |
| src/debug/jtag/daemons/data-daemon/shared/ORMLogger.ts | New in-memory metrics logging for ORM operations/migration. |
| src/debug/jtag/daemons/data-daemon/shared/ORMConfig.ts | New deterministic backend selection config (typescript/rust/shadow). |
| src/debug/jtag/daemons/data-daemon/shared/DataStorageAdapter.ts | Introduces storage-agnostic “include” terminology + backwards compatible types. |
| src/debug/jtag/daemons/data-daemon/shared/DataDaemon.ts | Switches description-field lookup from SqliteStorageAdapter to EntityRegistry. |
| src/debug/jtag/daemons/data-daemon/server/test-sqlite.ts | Removes standalone sqlite3 debugging script. |
| src/debug/jtag/daemons/data-daemon/server/managers/SqliteWriteManager.ts | Removes TS-side SQLite write manager. |
| src/debug/jtag/daemons/data-daemon/server/managers/SqliteVectorSearchManager.ts | Removes TS-side SQLite vector search manager. |
| src/debug/jtag/daemons/data-daemon/server/VectorSearchAdapterBase.ts | Routes vector search to Rust without TS table-name conversion; enforces dbPath. |
| src/debug/jtag/daemons/data-daemon/server/StorageMigrationService.ts | Removes legacy storage migration service. |
| src/debug/jtag/daemons/data-daemon/server/StorageAdapterFactory.ts | Removes old adapter factory implementation. |
| src/debug/jtag/daemons/data-daemon/server/SqliteTransactionManager.ts | Removes TS-side sqlite transaction manager. |
| src/debug/jtag/daemons/data-daemon/server/SqliteRawExecutor.ts | Removes TS-side sqlite executor. |
| src/debug/jtag/daemons/data-daemon/server/SqliteQueryBuilder.ts | Removes TS-side SQL query builder. |
| src/debug/jtag/daemons/data-daemon/server/SqlStorageAdapterBase.ts | Removes SQL adapter base class. |
| src/debug/jtag/daemons/data-daemon/server/SqlExecutor.ts | Removes TS SQL executor interface. |
| src/debug/jtag/daemons/data-daemon/server/SQL-ADAPTER-DEBUGGING-RECOVERY-PLAN.md | Removes debugging plan doc for now-deleted adapter refactor. |
| src/debug/jtag/daemons/data-daemon/server/RustSqliteExecutor.ts | Removes old Rust SQL bridge (superseded by ORM → continuum-core). |
| src/debug/jtag/daemons/data-daemon/server/DefaultStorageAdapterFactory.ts | Makes sqlite config return a no-op Memory adapter since ORM owns storage. |
| src/debug/jtag/daemons/data-daemon/server/DataDaemonServer.ts | Removes Rust data-worker connection logic (dead code) and documents new flow. |
| src/debug/jtag/daemons/console-daemon/server/ConsoleDaemonServer.ts | Updates logger socket path to continuum-core. |
| src/debug/jtag/daemons/comms-test-daemon/shared/CommsTestDaemon.ts | Deletes throwaway comms test daemon. |
| src/debug/jtag/daemons/comms-test-daemon/server/CommsTestDaemonServer.ts | Deletes throwaway comms test daemon server impl. |
| src/debug/jtag/daemons/archive-daemon/server/ArchiveDaemonServer.ts | Removes unused adapter lookup; clarifies archive handle alias behavior. |
| src/debug/jtag/daemons/ai-provider-daemon/shared/adapters/BaseOpenAICompatibleAdapter.ts | Removes duplicate warning log (delegated to PricingManager). |
| src/debug/jtag/commands/workspace/task/list/server/TaskListServerCommand.ts | Migrates task query to ORM. |
| src/debug/jtag/commands/workspace/task/create/server/TaskCreateServerCommand.ts | Migrates assignee validation + task store to ORM. |
| src/debug/jtag/commands/workspace/task/complete/server/TaskCompleteServerCommand.ts | Migrates task fetch/update to ORM. |
| src/debug/jtag/commands/workspace/git/shared/resolveWorkspacePath.ts | Migrates user read to ORM. |
| src/debug/jtag/commands/skill/validate/server/SkillValidateServerCommand.ts | Migrates skill read/update to ORM. |
| src/debug/jtag/commands/skill/propose/server/SkillProposeServerCommand.ts | Migrates skill read/update to ORM and switches proposer identity to context-driven. |
| src/debug/jtag/commands/skill/list/server/SkillListServerCommand.ts | Migrates skill list query to ORM. |
| src/debug/jtag/commands/skill/generate/server/SkillGenerateServerCommand.ts | Migrates skill read/update to ORM. |
| src/debug/jtag/commands/skill/activate/server/SkillActivateServerCommand.ts | Migrates governance read + skill updates to ORM. |
| src/debug/jtag/commands/session/get-user/server/SessionGetUserServerCommand.ts | Migrates user reads to ORM (supports PersonaUser direct userId). |
| src/debug/jtag/commands/search/vector/shared/SearchVectorTypes.ts | New types for Rust SearchModule vector similarity search. |
| src/debug/jtag/commands/search/vector/server/SearchVectorServerCommand.ts | New server command routing vector similarity search to continuum-core IPC. |
| src/debug/jtag/commands/search/params/shared/SearchParamsTypes.ts | New types for querying algorithm params from Rust SearchModule. |
| src/debug/jtag/commands/search/params/server/SearchParamsServerCommand.ts | New server command routing search params to continuum-core IPC. |
| src/debug/jtag/commands/search/list/shared/SearchListTypes.ts | New types for listing search algorithms. |
| src/debug/jtag/commands/search/list/server/SearchListServerCommand.ts | New server command routing search list to continuum-core IPC. |
| src/debug/jtag/commands/search/execute/shared/SearchExecuteTypes.ts | New types for executing text search via Rust SearchModule. |
| src/debug/jtag/commands/search/execute/server/SearchExecuteServerCommand.ts | New server command routing text search to continuum-core IPC. |
| src/debug/jtag/commands/rag/load/server/RAGLoadServerCommand.ts | Migrates message/room queries to ORM. |
| src/debug/jtag/commands/development/timing/shared/TimingTypes.ts | Updates timing docs to note data-daemon consolidation into continuum-core. |
| src/debug/jtag/commands/development/timing/server/TimingServerCommand.ts | Updates docs to reflect timing file may no longer exist. |
| src/debug/jtag/commands/data/vector-search/server/VectorSearchServerCommand.ts | Simplifies vector search by routing all requests through ORM (dbHandle handled internally). |
| src/debug/jtag/commands/data/update/shared/DataUpdateTypes.ts | Adds suppressEvents flag for ORM-backed updates. |
| src/debug/jtag/commands/data/update/server/DataUpdateServerCommand.ts | Routes updates through ORM; resolves dbHandle→dbPath; adds suppressEvents. |
| src/debug/jtag/commands/data/truncate/server/DataTruncateServerCommand.ts | Validates collection at runtime and routes truncate through ORM. |
| src/debug/jtag/commands/data/shared/BaseDataTypes.ts | Adds collection usage docs for CLI ergonomics. |
| src/debug/jtag/commands/data/read/shared/DataReadTypes.ts | Adds docs/examples for read command. |
| src/debug/jtag/commands/data/read/server/DataReadServerCommand.ts | Validates collection at runtime; routes read through ORM; uses generated constants. |
| src/debug/jtag/commands/data/query-open/server/QueryOpenServerCommand.ts | Switches paginated query open to ORM. |
| src/debug/jtag/commands/data/query-next/server/QueryNextServerCommand.ts | Switches paginated query next-page to ORM. |
| src/debug/jtag/commands/data/query-close/server/QueryCloseServerCommand.ts | Switches paginated query close to ORM. |
| src/debug/jtag/commands/data/open/shared/DataOpenTypes.ts | Expands docs/warnings for advanced multi-db handle usage. |
| src/debug/jtag/commands/data/open/server/DataOpenServerCommand.ts | Improves adapter validation and error messages. |
| src/debug/jtag/commands/data/list/shared/DataListTypes.ts | Adds docs/examples for list command. |
| src/debug/jtag/commands/data/list/server/DataListServerCommand.ts | Routes list/count through ORM; improves missing-collection error. |
| src/debug/jtag/commands/data/generate-embedding/server/GenerateEmbeddingServerCommand.ts | Routes embedding generation through ORM. |
| src/debug/jtag/commands/data/delete/server/DataDeleteServerCommand.ts | Validates collection at runtime; routes delete/read through ORM. |
| src/debug/jtag/commands/data/create/server/DataCreateServerCommand.ts | Routes create through ORM; resolves dbHandle→dbPath. |
| src/debug/jtag/commands/data/clear/server/DataClearServerCommand.ts | Routes clear/listCollections through ORM. |
| src/debug/jtag/commands/data/backfill-vectors/server/BackfillVectorsServerCommand.ts | Routes backfill through ORM. |
| src/debug/jtag/commands/comms-test/shared/CommsTestTypes.ts | Deletes comms-test command types. |
| src/debug/jtag/commands/comms-test/server/CommsTestServerCommand.ts | Deletes comms-test server command. |
| src/debug/jtag/commands/comms-test/package.json | Removes comms-test command package definition. |
| src/debug/jtag/commands/collaboration/live/leave/server/LiveLeaveServerCommand.ts | Prioritizes identity via context.userId; clarifies fallback chain. |
| src/debug/jtag/commands/collaboration/live/join/server/LiveJoinServerCommand.ts | Prioritizes identity via context.userId; clarifies fallback chain. |
| src/debug/jtag/commands/collaboration/dm/server/DmServerCommand.ts | Prioritizes identity via context.userId; clarifies fallback chain. |
| src/debug/jtag/commands/collaboration/decision/vote/server/DecisionVoteServerCommand.ts | Removes injected caller params; uses context.userId and CLI fallback. |
| src/debug/jtag/commands/collaboration/decision/rank/shared/DecisionRankTypes.ts | Removes explicit voterId param in favor of context.userId. |
| src/debug/jtag/commands/collaboration/decision/rank/server/DecisionRankServerCommand.ts | Uses context.userId first; improves error + logging when missing. |
| src/debug/jtag/commands/collaboration/decision/propose/shared/DecisionProposeTypes.ts | Removes proposerId param in favor of context.userId. |
| src/debug/jtag/commands/collaboration/decision/propose/server/DecisionProposeServerCommand.ts | Uses context.userId first and explicitly sets chat senderId for notifications. |
| src/debug/jtag/commands/collaboration/decision/create/server/DecisionCreateServerCommand.ts | Uses context.userId first for caller identity; CLI fallback. |
| src/debug/jtag/commands/collaboration/chat/send/server/ChatSendServerCommand.ts | Uses context.userId first for caller identity; CLI fallback retained. |
| src/debug/jtag/commands/collaboration/chat/poll/server/ChatPollServerCommand.ts | Migrates message queries to ORM. |
| src/debug/jtag/commands/canvas/stroke/add/server/CanvasStrokeAddServerCommand.ts | Uses context.userId first to resolve creator; migrates store to ORM. |
| src/debug/jtag/commands/ai/thoughtstream/server/ThoughtStreamServerCommand.ts | Migrates reads/queries to ORM. |
| src/debug/jtag/commands/ai/status/server/AIStatusServerCommand.ts | Migrates persona query to ORM. |
| src/debug/jtag/commands/ai/sleep/server/AiSleepServerCommand.ts | Uses context.userId first for persona targeting; clarifies fallback chain. |
| src/debug/jtag/commands/ai/rag/inspect/server/RAGInspectServerCommand.ts | Migrates message read to ORM. |
| src/debug/jtag/commands/ai/generate/server/AIGenerateServerCommand.ts | Migrates persona query to ORM. |
| src/debug/jtag/commands/ai/cost/server/AICostServerCommand.ts | Removes limit: -1 pattern; relies on “no limit” semantics. |
| docs/COMMAND-SHORTHAND-DESIGN.md | Adds approved design doc for command shorthand aliases. |
| CLAUDE.md | Adds guidance to avoid manual cargo builds outside npm scripts. |
Files not reviewed (1)
- src/debug/jtag/examples/widget-ui/package-lock.json: Language not supported
Comments suppressed due to low confidence (1)
src/debug/jtag/generator/generate-rust-bindings.ts:113
parseExportedTypes()only detectsexport type ...declarations, butts-rscommonly emitsexport interface ...and may also emit enums. This will miss many exports, causing duplicate detection (and “already exported” skipping) to be unreliable; update the parser to also detectexport interface,export enum, etc., and ensure it correctly reflects the kinds of exports emitted intoshared/generated/*.
function parseExportedTypes(filePath: string): string[] {
const content = fs.readFileSync(filePath, 'utf-8');
const matches = content.matchAll(/export type\s+(\w+)/g);
return Array.from(matches, m => m[1]);
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+168
to
195
| const dirTypes: string[] = []; | ||
| for (const f of files) { | ||
| const types = parseExportedTypes(path.join(dirPath, f)); | ||
| dirTypes.push(...types); | ||
| } | ||
|
|
||
| // Check if this dir has any duplicates | ||
| const hasDuplicates = dirTypes.some(t => duplicateTypes.has(t)); | ||
|
|
||
| if (!hasDuplicates) { | ||
| // Safe to use wildcard | ||
| lines.push(`export * from './${dir}';`); | ||
| } else { | ||
| // Use explicit exports, skipping types that are duplicated | ||
| // Only export from the FIRST directory that had the type | ||
| lines.push(`// ${dir}: explicit exports (has duplicate types)`); | ||
| for (const typeName of dirTypes) { | ||
| if (duplicateTypes.has(typeName)) { | ||
| // Only export if this is the first directory | ||
| if (typeToDir.get(typeName) === dir) { | ||
| lines.push(`export type { ${typeName} } from './${dir}';`); | ||
| } | ||
| // else skip - it's exported from another dir | ||
| } else { | ||
| lines.push(`export type { ${typeName} } from './${dir}';`); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
dirTypes is collected as an array and may include duplicates (e.g., if the same type name is exported from multiple files in the same subdir, or the regex matches multiple times). That can generate duplicate export type { X } from './dir' lines in the master barrel, which can fail TypeScript builds. Consider deduping dirTypes (e.g., via a Set) and generating exports in a stable sorted order.
Comment on lines
+214
to
219
| export interface StorageQueryWithInclude extends StorageQuery { | ||
| /** Include specifications for related collections (alias for joins) */ | ||
| readonly includes?: readonly IncludeSpec[]; | ||
| /** Join specifications for related collections */ | ||
| readonly joins: readonly JoinSpec[]; | ||
| readonly joins: readonly IncludeSpec[]; | ||
| } |
There was a problem hiding this comment.
StorageQueryWithInclude introduces includes? (and the docs/examples reference queryWithInclude and includes), but still requires joins to be present. This makes the new “include” terminology awkward to adopt (callers must supply joins even if they only want includes). If includes is intended to replace joins, make joins optional (or remove it from the new interface and keep it only via the deprecated StorageQueryWithJoin alias).
Comment on lines
+30
to
+42
| await this.rustClient.connect(); | ||
| const result = await this.rustClient.searchVector( | ||
| params.queryVector, | ||
| params.corpusVectors, | ||
| params.normalize ?? true, | ||
| params.threshold ?? 0.0 | ||
| ); | ||
| this.rustClient.disconnect(); | ||
|
|
||
| return transformPayload(payload, { | ||
| algorithm: 'cosine', | ||
| ...result, | ||
| }); |
There was a problem hiding this comment.
disconnect() won’t run if connect() or searchVector() throws, which can leak IPC connections/resources and cause cascading failures under repeated calls. Wrap the connect/execute sequence in a try { ... } finally { this.rustClient.disconnect(); } (and optionally only disconnect if connect succeeded).
Suggested change
| await this.rustClient.connect(); | |
| const result = await this.rustClient.searchVector( | |
| params.queryVector, | |
| params.corpusVectors, | |
| params.normalize ?? true, | |
| params.threshold ?? 0.0 | |
| ); | |
| this.rustClient.disconnect(); | |
| return transformPayload(payload, { | |
| algorithm: 'cosine', | |
| ...result, | |
| }); | |
| let connected = false; | |
| try { | |
| await this.rustClient.connect(); | |
| connected = true; | |
| const result = await this.rustClient.searchVector( | |
| params.queryVector, | |
| params.corpusVectors, | |
| params.normalize ?? true, | |
| params.threshold ?? 0.0 | |
| ); | |
| return transformPayload(payload, { | |
| algorithm: 'cosine', | |
| ...result, | |
| }); | |
| } finally { | |
| if (connected) { | |
| this.rustClient.disconnect(); | |
| } | |
| } |
Comment on lines
+72
to
+74
| if (shouldLogThis) { | ||
| console.log(`[ORM] ${operation} ${collection}`, details ? JSON.stringify(details).slice(0, 200) : ''); | ||
| } |
There was a problem hiding this comment.
JSON.stringify(details) can throw (e.g., circular references), which would turn “logging” into an operation failure. Consider guarding stringification (try/catch) and falling back to a safe representation when serialization fails.
Comment on lines
+39
to
+44
| const entity = await ORM.store( | ||
| collection as CollectionName, | ||
| params.data as BaseEntity, | ||
| params.suppressEvents ?? false, | ||
| dbPath | ||
| ); |
There was a problem hiding this comment.
This command casts user input (params.collection: string) to CollectionName without runtime validation, while other commands in this PR add isValidCollection() checks (e.g., data/read, data/delete, data/truncate). To keep behavior consistent and error messages clearer, validate params.collection with isValidCollection() before casting (and return a structured error result when invalid).
Comment on lines
+35
to
+42
| const entity = await ORM.update( | ||
| collection as CollectionName, | ||
| params.id, | ||
| params.data as Partial<BaseEntity>, | ||
| params.incrementVersion ?? true, | ||
| dbPath, | ||
| params.suppressEvents ?? false | ||
| ); |
There was a problem hiding this comment.
Similar to data/create, data/update casts params.collection to CollectionName without validating it first. Since this is a user-facing CLI command and other commands now validate collection names, add an isValidCollection() check to return a clear “Invalid collection name” error instead of relying on a Rust-side failure.
Comment on lines
+161
to
+165
| // Add each collection as a constant | ||
| for (const info of this.collections) { | ||
| lines.push(` /** From ${info.entityName} */`); | ||
| lines.push(` ${info.constantKey}: '${info.collectionName}' as const,`); | ||
| } |
There was a problem hiding this comment.
The generator does not detect collisions (two entities producing the same constantKey, or multiple entities mapping to the same collectionName). A collision will produce duplicate object keys in the generated COLLECTIONS, leading to confusing “last one wins” behavior or lint/TS issues. Consider tracking seen constantKey/collectionName values during extraction and failing the generation with a clear error when duplicates are found.
|
|
||
| if (result.success && result.items && result.items.length > 0) { | ||
| const user = result.items[0]; | ||
| console.log('🔧 DmServerCommand.resolveCallerIdentity USING CONTEXT userId', { userId: params.context.userId }); |
There was a problem hiding this comment.
This introduces a direct console.log in a server command. Since other parts of the codebase use structured loggers (this.log.*) and since this will be noisy in production, switch this to this.log.debug(...) (or remove once verified) to keep output consistent and controllable via log levels.
Suggested change
| console.log('🔧 DmServerCommand.resolveCallerIdentity USING CONTEXT userId', { userId: params.context.userId }); | |
| this.log.debug('DmServerCommand.resolveCallerIdentity USING CONTEXT userId', { userId: params.context.userId }); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Major architectural consolidation that routes all data operations through Rust, deleting ~16K lines of redundant TypeScript code.
What Changed
Worker Consolidation (Phases 4a-4e)
logger,search,embedding,data-daemonworkers intocontinuum-coreORM Migration (Phases 7-13)
ORM.ts → ORMRustClient → Rust DataModuledbPathparameterDead Code Cleanup
SqliteStorageAdapter.ts(1,043 lines) + 6 supporting classesPerformance
The system is noticeably faster:
Stats
Test plan
npm run build:tspassesnpm run test:precommitpasses./jtag pingconfirms server + browser connected./jtag data/listconfirms ORM → Rust data operations worknpm start🤖 Generated with Claude Code