back merge main into feature branch#41
Merged
Merged
Conversation
* security: remove pre-release internal analysis from public tree Internal development artifacts (AI-assisted code analysis, threat models, vulnerability findings with exact locations) were included in the initial open-source release. These are not official audit deliverables and expose implementation details inappropriate for a public repository. The verified third-party audit (CERT-IN) is added in the next commit. * docs: add CERT-IN security audit certificate for v1.1.0 Add Terasoft Technologies CERT-IN certificate (TERA/CERT-IN/03/2026/CR/16) with VERIFICATION.md for independent checksum verification. Add security audit badges and Security section to README. Audited archive: jmdn-1.1.0.zip MD5: 508a6dc5f7061a27a0344c504aea9116
* docs: add CHANGELOG.md following Keep a Changelog format * docs: use [Unreleased] header per Keep a Changelog convention
* feat: Implement FastSync V2 with ImmuDB for block and account data management and iteration. * test: load settings before running the Merkle tree test * feat: Add new `FastSyncV2` gRPC endpoint with corresponding client and server implementations. * refactor: construct and use targetNodeInfo for fastsync requests instead of fs.NodeInfo. * fix: prevent uint64 underflow in CheckNonceAndGetLatest block scan (#22) The inner loop used `for i := currentBlock; i >= startBlock; i--` with uint64 variables. When startBlock is 0 (chain has fewer blocks than batchSize), the loop processes block 0, then decrements i to uint64 max (~1.8e19) instead of stopping. Since uint64 max >= 0 is always true, the loop spins indefinitely issuing VerifiedGet calls for non-existent blocks, flooding logs with "tbtree: key not found" errors and blocking transaction submission on fresh/low-block-count chains. Fix: restructure as a top-decrement loop `for i := currentBlock + 1; i > startBlock; { i-- }` so i never passes through zero and the underflow is impossible. Co-authored-by: Doc <saishibu@jupitermeta.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: Update checksum version to 2 and hardcode protocol version for priorRouter, and standardize NodeInfo import alias. * refactor: update sync protocol with extended transaction fields, ZK commitment serialization, and improved FastSync V2 documentation * Fix/fastsync (#23) * feat: Update checksum version to 2 and hardcode protocol version for priorRouter, and standardize NodeInfo import alias. * refactor: update sync protocol with extended transaction fields, ZK commitment serialization, and improved FastSync V2 documentation * fix: update sync version parameters to ensure correct transport selection for bisection callbacks * chore: update priorsyncVersion to 2 in fastsyncv2.go * fix: implement force update logic for existing blocks in immudb writers * feat: implement event-driven vote collection and notification system - Added a notification mechanism for the sequencer's vote collector in ListenerHandler. - Enhanced the Consensus struct to manage vote notifications and round contexts. - Updated vote result storage to be scoped by block hash, allowing for better tracking of votes per round. - Refactored vote processing to utilize event-driven architecture, improving efficiency and responsiveness. - Introduced targeted vote trigger broadcasts to committee peers, ensuring only relevant nodes participate in voting. * feat: implement incremental startup sync in FastsyncV2 to catch up on missed blocks * chore: add project planning documentation and workspace configuration for Sequencer refactoring * docs: add documentation for consensus architecture and MCP tools, and update ignore patterns * feat: implement fastsyncv2 CLI command and add nil-safety checks for database state retrieval * fix: handle key not found errors in immudb account manager by returning zero balance or creating new accounts * refactor: update PoTS to use commsVersion and resolve full peer addresses from peerstore * feat: add script to scan accounts database for duplicate nonces * feat: add detailed account listing and simplify duplicate nonce report output * fix: implement robust block marker reconciliation and increase database operation timeouts * feat: introduce FastSync V2 configuration and initialization - Added FastSync V2 settings to jmdn_default.yaml, allowing for enhanced control over sync behavior. - Updated main.go to conditionally initialize FastSync V2 based on configuration, including support for allowed peers. - Enhanced FastSyncSettings struct in config/settings/config.go to manage sync options and defaults. - Implemented default values for FastSync settings in loader.go to ensure proper initialization. * refactor(config): standardize fastsync defaults and struct order * fix: prevent HeaderSync from prematurely advancing the latest_block marker by restoring its original value after writes * refactor: simplify GetLatestBlockNumber by removing retry logic and reconciliation fallback * refactor: remove manual block reconciliation logic from immudb_adapter and fix import ordering * chore(deps): pin JMDN-FastSync to fix/auth-error-handling branch commit (fcd35ff) * chore(deps): upgrade JMDN-FastSync to e464cfd (auth propagation fix) Picks up the fix for HeaderSync workers receiving stale auth tokens after SyncConfirmation detects divergence. Fresh UUIDs are now propagated in-place to remotes and hs.ServerAuth is updated to the confirming peer's token before buildBatches runs. * refactor: add Timestamp field to Transaction struct in multiple files for improved transaction data handling * chore: update dependencies in go.mod and go.sum, including upgrades to OpenTelemetry and ClickHouse configurations * feat: implement account management features in immudb_account_manager and integrate AccountSync in fastsyncv2 - Added GetAccountByAddress and WriteAccounts methods for account retrieval and batch writing. - Introduced immudbNonceIter for efficient account iteration and caching. - Enhanced fastsyncv2 with AccountSync to handle zero-transaction accounts and ensure all tagged accounts are present before reconciliation. - Updated go.mod and go.sum for dependency versions. * chore: clean up go.mod and go.sum by removing unused dependencies and updating versions - Removed indirect dependencies for ClickHouse and Brotli. - Updated the version for the Ion package. - Added new indirect dependencies for gopsutil and wmi. - Cleaned up unnecessary replace directives in go.mod. * feat: enhance account retrieval and header synchronization in immudb - Updated GetAccountsByNonces to scan the database for accounts matching nonces, improving reliability by eliminating reliance on in-memory cache. - Added logic to retrieve the latest block header in GetBlockDetails, ensuring accurate synchronization. - Implemented header_latest_block update in WriteHeaders to maintain correct Merkle range for SyncConfirmation. * feat: enhance account fetching and local account creation in fastsyncv2 - Improved the handling of missing tagged accounts by tracking which addresses are returned from the server. - Added logic to create local accounts with zero balance for addresses not found on the server, ensuring all tagged accounts are accounted for before reconciliation. - Updated logging for better visibility into the account fetching process. * Revert "feat: enhance account fetching and local account creation in fastsyncv2" This reverts commit 99b300c. * feat: add account synchronization command and functionality in fastsyncv2 - Introduced the 'accountsync' command to sync missing accounts from a specified peer without block synchronization. - Implemented the AccountSyncOnly method in fastsyncv2 to handle account-only synchronization, ensuring efficient data retrieval when blocks are identical. - Updated CLI help text to include the new accountsync command for user guidance. * feat: add AccountSync method to CLIService for account synchronization - Introduced the AccountSync RPC method to the CLIService for syncing accounts from a peer. - Updated generated gRPC files to reflect changes in the proto definitions. - Enhanced the UnimplementedCLIServiceServer to include the AccountSync method with appropriate error handling. * feat: implement accountsync command and functionality for account synchronization - Added the 'accountsync' command to the CLI for syncing missing accounts from a specified peer without block synchronization. - Implemented the HandleAccountSync method in the CommandHandler to manage account-only synchronization. - Updated the CLI help text to include the new accountsync command for user guidance. - Enhanced the Client and GRPC_Server to support the new AccountSync functionality. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260519052600-4d00a637873f in both go.mod and go.sum files, ensuring compatibility with the latest changes. * feat(fastsync): enforce strict read-only/serve-only sync guardrails - Removed hardcoded 15m FastSync timeout and wired to 'cfg.FastSync.SyncTimeout'. - Introduced 'PullAllowed' config constraint in the CLI 'CommandHandler'. - Hardened CLI execution paths ('fastsync', 'fastsyncv2', 'accountsync', 'firstsync client') to strictly reject manual pull operations when a node is configured as a serve-only participant ('sync: false'). - Renamed internal CLI boolean flags to explicitly distinguish between pulling and serving capabilities. * refactor(fastsync): standardize config naming to clarify network directionality - Renamed 'Sync' to 'EnablePulling' and 'StartupSync' to 'PullOnStartup' in 'FastSyncSettings'. - Updated underlying YAML tags and viper bindings to use 'enable_pulling' and 'pull_on_startup'. - Modified application bootstrap and CLI handlers to accurately reflect the new flags. - Retained 'Enabled' as the master engine switch to accurately represent full initialization state. - Updated log messaging to accurately reflect 'serve-only participant' mode when pulling is disabled. * feat(scripts): add merkle_check.go for post-sync block integrity validation - Fetches all ZKBlocks from ImmuDB and computes a canonical SHA256 hash per block (BlockHash excluded — it is derived and would be circular). - Builds a single global Merkle tree over all block hashes with indexed FromBlock/ToBlock ranges on each node for O(log n) divergence tracing. - Fixed connection pool race: cancel() is now called AFTER GetLatestBlockNumber, not before, so the pooled connection is not returned prematurely. - Fixed boundary-shift hash collision: all variable-length fields (strings, byte slices, tx.Data) are now written with a 4-byte length prefix via writeVar() to prevent (e.g.) 'AB'+'CD' == 'A'+'BCD' producing identical hashes. - Added tx.ChainID and AccessList coverage to the per-transaction hash. - Output JSON includes from_block/to_block reflecting the actual covered range (skipped error blocks are excluded from the range). - Progress logging now always prints for runs of ≤100 blocks. * chore: increase HTTP server read and write timeouts to 60 seconds * feat: enhance BatchRestoreAccounts with deduplication and chunked operations - Implemented deduplication for address and DID entries in BatchRestoreAccounts to ensure unique keys and maintain the latest updates. - Added logic to chunk operations into smaller batches for execution, adhering to ImmuDB's MaxTxEntries limit, improving transaction handling and performance. - Updated logging for better error tracking during batch execution. * feat: optimize BatchRestoreAccounts with pre-fetching of existing accounts - Implemented a pre-fetch mechanism to retrieve all existing account values in a single GetAll RPC, reducing connection pool exhaustion during concurrent operations. - Refactored the logic to check for existing accounts using the pre-fetched data, improving efficiency and maintaining correctness in account updates. - Enhanced handling of account creation and updates to ensure only necessary writes occur, preserving the latest data while avoiding unnecessary operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260520134039-a630fd87e742 in both go.mod and go.sum files for compatibility with recent changes. - Adjusted the scanning order in getKeysBatch to ascending for reliable prefix filtering. * feat: introduce cursor-based pagination for account listing - Added ListAccountsPaginatedCursor function to provide a cursor-based alternative to ListAccountsPaginated, improving performance by avoiding O(N²) re-scans. - Updated account manager to utilize the new cursor-based pagination in NewAccountNonceIterator and GetAccountsByNonces methods, enhancing efficiency in account retrieval. - Adjusted related logic to support cursor management, ensuring seamless iteration through accounts without unnecessary overhead. * Fix/accountsync/performance (#35) * feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> * feat(logging): add custom headers support to OTEL config and upgrade ion to v0.4.2 * Add/redisstreams (#37) * feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. * feat: integrate Redis for asynchronous account synchronization - Added Redis support for the account sync worker, enabling WriteAccounts and BatchUpdateAccounts to enqueue operations to a Redis Stream, decoupling them from the ImmuDB commit latency. - Updated configuration to include Redis connection settings in jmdn_default.yaml and corresponding defaults. - Enhanced error handling to ensure proper initialization of the sync worker and logging for better traceability. - Refactored account management functions to utilize the new Redis streaming mechanism for improved performance and reliability. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> * Remove unused functions for transaction and access list conversion in immudb_account_manager.go to streamline code and improve maintainability. * feat(config): add Redis authentication and complete Viper env bindings This commit introduces proper support for Redis authentication and ensures all nested configuration structures perfectly map to environment variables via Viper. Changes: - Add Password field to RedisSettings struct in config.go. - Update main.go to pass cfg.Database.Redis.Password to the Redis client during initialization. - Update defaults.go and jmdn_default.yaml to explicitly bind Redis URL to 127.0.0.1:6379 instead of an empty string or wildcard. - Update loader.go's setDefaults() to explicitly register all top-level Security settings (e.g., global_rate_limit, trusted_proxies). - Introduce dynamic Viper default registration for Security.Services in loader.go, fully enabling runtime environment variable overrides for nested map fields (e.g. JMDN_SECURITY_SERVICES_EXPLORER_API_RATE_LIMIT). * fix(fastsync): map missing ChainID, AccessList, and LogsBloom fields - FastsyncV2 adapter now maps ChainID and AccessList into non-headers protobufs - immudb_data_writer and immudb_block_nonheaders properly serialize/deserialize these fields to defaultdb - immudb_headers_writer now correctly stores and serves LogsBloom This fixes the Merkle hash divergence on newly fast-synced nodes. * fix(accountsdb): preserve DIDs and metadata during sync - account_sync_worker: parseAccountsPayload now correctly pushes the DID key into the stream so BatchRestoreAccounts successfully creates the bound did: reference for new accounts. - account_immuclient: BatchRestoreAccounts now performs field-merging (DIDAddress, CreatedAt, AccountType, Metadata) when processing partial account updates from Fastsync Reconciliation. This prevents data loss for active accounts. - account_immuclient: PutNonceofAccount migrated to a monotonic nanosecond counter (Option C) to resolve silent uint64 overflow bugs. * feat(migration): separate nonce patching scripts and use O(1) nonce fetch - Created standalone Scripts/replay_blocks_and_export.go for computing TxCount - Created standalone Scripts/patch_accounts.go for updating Nonces and standardizing timestamps - Replaced CheckNonceAndGetLatest in Security.go with O(1) GetAccount verification - Fixed new account generation in DB_OPs to initialize Nonce to 0 natively - Hardened DID registration to enforce UnixNano() timestamps * fix(security): resolve same-block nonce replay vulnerabilities and decouple TxCountSent - schema: added TxCountSent to Account schema to decouple analytical stats from cryptographic Nonce - db: added UpdateAccountSenderState to handle atomic balance and state updates - core: injected execution-time strict nonce validation (tx.Nonce < didDoc.Nonce) in deductFromSender to eliminate same-block double-execution loophole - core: fixed severe bug where sequencer failed to persist Nonce updates to DB after block execution - scripts: created patch_accounts_v2.go to safely migrate accountsdb by mapping TxCount -> TxCountSent and MaxNonce+1 -> Nonce * fix(redis): remove hard startup dependency for account sync Moved Redis consumer group initialization into an asynchronous retry loop inside runWorker. The node now boots successfully even if Redis is completely offline, gracefully degrading bulk sync operations while preserving live transaction processing. Retry interval set to 30s to minimize log spam. * refactor(account_sync): streamline account sync worker initialization and enhance Redis integration This commit refactors the account synchronization process by modifying the initialization of the account sync worker. The StartAccountSyncWorker function now creates a WorkerManager and installs it as the package-level queue, allowing for lazy worker startup. Additionally, the BatchRestoreAccounts function has been updated to accept a context parameter, improving its integration with the overall context management. The logging messages have been adjusted for clarity, and new methods for managing the worker lifecycle have been introduced to enhance performance and reliability. * chore(deps): update JMDN-FastSync to v0.0.0-20260601052219-40e74741de7c and add redis/go-redis/v9 v9.19.0 dependency - Updated the JMDN-FastSync dependency version in go.mod and go.sum for compatibility with recent changes. - Added redis/go-redis/v9 v9.19.0 as a new dependency in go.mod and go.sum. * fix(accountsdb): remove dead PutNonceofAccount to prevent time.Now() overflow risks * refactor: cleanly decouple consensus account origination and fix StateID generation - Reverted Account options and restored pure 'GenerateStateID()' for locally originated accounts. - Removed hacky deterministic 'StateID' origination from 'BlockProcessing'. Consensus now strictly requires accounts to exist before transacting. - Added pre-flight hook in 'eth_getBalance' facade to natively originate and gossip missing accounts via P2P before consensus. - Updated 'Account' adapter mappings to cross-wire Fastsync 'types.Account' safely ('StateID' -> 'Nonce', 'Nonce' -> 'TxNonce'). - Secured incoming P2P DID Gossips using 'StorePropagatedAccount' to perfectly preserve network StateIDs. * feat(accounts): align struct to Fastsync by renaming StateID to Nonce and adding TxNonce * Implement bounded enqueue for account synchronization - Introduced a new helper function `enqueueRecordsChunked` to split account records into fixed-size messages, ensuring each message adheres to the `maxRecordsPerMessage` limit to prevent exceeding Redis's bulk string size. - Updated `WriteAccounts` and `BatchUpdateAccounts` methods to utilize the new chunking logic, enhancing reliability by allowing best-effort delivery across chunks. - Added a white-box test for the chunking logic to validate message boundaries and error handling. - Updated documentation to reflect changes in the enqueue process and message size constraints. - Added a replace directive in `go.mod` for local development with the JMDN-FastSync repository. * fix: Correct Ethereum TxNonce validation logic to prevent intra-block replays - Security module now exclusively uses SecurityCache for TxNonce resolution, drastically reducing DB I/O during batch validation. - Cache enforces mutation upon successful verification (UpdateTxNonce), mitigating intra-block replay attack vectors. - Renamed cache methods to strictly differentiate between Ethereum TxNonce and Fastsync Nonce (StateID). - Fixed static off-by-one error (MaxNonce vs MaxNonce + 1) in V3 migration script. * fix(pubsub): close topic race condition * chore: fix govet deprecation warnings for Go 1.25+ - Replaced deprecated ioutil.ReadFile and ioutil.WriteFile with os.ReadFile and os.WriteFile in bls-sgin.go - Replaced deprecated reflect.Ptr with reflect.Pointer in seednode.go - Applied auto-formatting via make fmt to clean up unused imports These changes resolve linter errors introduced by standard library deprecations in modern Go versions while keeping the logic functionally identical. * docs: add CHANGELOG [Unreleased] entries and fix fastsync yaml keys * docs: reframe CHANGELOG motivation — node divergence → consensus unblock * ci: trigger checks --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> Co-authored-by: Doc <saishibu@jupitermeta.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: neerajchowdary889 <neerajchowdary889@gmail.com> Co-authored-by: Neeraj Vipparla <57310710+neerajvipparla@users.noreply.github.com>
…error handling (#40) * feat: Implement FastSync V2 with ImmuDB for block and account data management and iteration. * test: load settings before running the Merkle tree test * feat: Add new `FastSyncV2` gRPC endpoint with corresponding client and server implementations. * refactor: construct and use targetNodeInfo for fastsync requests instead of fs.NodeInfo. * fix: prevent uint64 underflow in CheckNonceAndGetLatest block scan (#22) The inner loop used `for i := currentBlock; i >= startBlock; i--` with uint64 variables. When startBlock is 0 (chain has fewer blocks than batchSize), the loop processes block 0, then decrements i to uint64 max (~1.8e19) instead of stopping. Since uint64 max >= 0 is always true, the loop spins indefinitely issuing VerifiedGet calls for non-existent blocks, flooding logs with "tbtree: key not found" errors and blocking transaction submission on fresh/low-block-count chains. Fix: restructure as a top-decrement loop `for i := currentBlock + 1; i > startBlock; { i-- }` so i never passes through zero and the underflow is impossible. Co-authored-by: Doc <saishibu@jupitermeta.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * feat: Update checksum version to 2 and hardcode protocol version for priorRouter, and standardize NodeInfo import alias. * refactor: update sync protocol with extended transaction fields, ZK commitment serialization, and improved FastSync V2 documentation * Fix/fastsync (#23) * feat: Update checksum version to 2 and hardcode protocol version for priorRouter, and standardize NodeInfo import alias. * refactor: update sync protocol with extended transaction fields, ZK commitment serialization, and improved FastSync V2 documentation * fix: update sync version parameters to ensure correct transport selection for bisection callbacks * chore: update priorsyncVersion to 2 in fastsyncv2.go * fix: implement force update logic for existing blocks in immudb writers * feat: implement event-driven vote collection and notification system - Added a notification mechanism for the sequencer's vote collector in ListenerHandler. - Enhanced the Consensus struct to manage vote notifications and round contexts. - Updated vote result storage to be scoped by block hash, allowing for better tracking of votes per round. - Refactored vote processing to utilize event-driven architecture, improving efficiency and responsiveness. - Introduced targeted vote trigger broadcasts to committee peers, ensuring only relevant nodes participate in voting. * feat: implement incremental startup sync in FastsyncV2 to catch up on missed blocks * chore: add project planning documentation and workspace configuration for Sequencer refactoring * docs: add documentation for consensus architecture and MCP tools, and update ignore patterns * feat: implement fastsyncv2 CLI command and add nil-safety checks for database state retrieval * fix: handle key not found errors in immudb account manager by returning zero balance or creating new accounts * refactor: update PoTS to use commsVersion and resolve full peer addresses from peerstore * feat: add script to scan accounts database for duplicate nonces * feat: add detailed account listing and simplify duplicate nonce report output * fix: implement robust block marker reconciliation and increase database operation timeouts * feat: introduce FastSync V2 configuration and initialization - Added FastSync V2 settings to jmdn_default.yaml, allowing for enhanced control over sync behavior. - Updated main.go to conditionally initialize FastSync V2 based on configuration, including support for allowed peers. - Enhanced FastSyncSettings struct in config/settings/config.go to manage sync options and defaults. - Implemented default values for FastSync settings in loader.go to ensure proper initialization. * refactor(config): standardize fastsync defaults and struct order * fix: prevent HeaderSync from prematurely advancing the latest_block marker by restoring its original value after writes * refactor: simplify GetLatestBlockNumber by removing retry logic and reconciliation fallback * refactor: remove manual block reconciliation logic from immudb_adapter and fix import ordering * chore(deps): pin JMDN-FastSync to fix/auth-error-handling branch commit (fcd35ff) * chore(deps): upgrade JMDN-FastSync to e464cfd (auth propagation fix) Picks up the fix for HeaderSync workers receiving stale auth tokens after SyncConfirmation detects divergence. Fresh UUIDs are now propagated in-place to remotes and hs.ServerAuth is updated to the confirming peer's token before buildBatches runs. * refactor: add Timestamp field to Transaction struct in multiple files for improved transaction data handling * chore: update dependencies in go.mod and go.sum, including upgrades to OpenTelemetry and ClickHouse configurations * feat: implement account management features in immudb_account_manager and integrate AccountSync in fastsyncv2 - Added GetAccountByAddress and WriteAccounts methods for account retrieval and batch writing. - Introduced immudbNonceIter for efficient account iteration and caching. - Enhanced fastsyncv2 with AccountSync to handle zero-transaction accounts and ensure all tagged accounts are present before reconciliation. - Updated go.mod and go.sum for dependency versions. * chore: clean up go.mod and go.sum by removing unused dependencies and updating versions - Removed indirect dependencies for ClickHouse and Brotli. - Updated the version for the Ion package. - Added new indirect dependencies for gopsutil and wmi. - Cleaned up unnecessary replace directives in go.mod. * feat: enhance account retrieval and header synchronization in immudb - Updated GetAccountsByNonces to scan the database for accounts matching nonces, improving reliability by eliminating reliance on in-memory cache. - Added logic to retrieve the latest block header in GetBlockDetails, ensuring accurate synchronization. - Implemented header_latest_block update in WriteHeaders to maintain correct Merkle range for SyncConfirmation. * feat: enhance account fetching and local account creation in fastsyncv2 - Improved the handling of missing tagged accounts by tracking which addresses are returned from the server. - Added logic to create local accounts with zero balance for addresses not found on the server, ensuring all tagged accounts are accounted for before reconciliation. - Updated logging for better visibility into the account fetching process. * Revert "feat: enhance account fetching and local account creation in fastsyncv2" This reverts commit 99b300c. * feat: add account synchronization command and functionality in fastsyncv2 - Introduced the 'accountsync' command to sync missing accounts from a specified peer without block synchronization. - Implemented the AccountSyncOnly method in fastsyncv2 to handle account-only synchronization, ensuring efficient data retrieval when blocks are identical. - Updated CLI help text to include the new accountsync command for user guidance. * feat: add AccountSync method to CLIService for account synchronization - Introduced the AccountSync RPC method to the CLIService for syncing accounts from a peer. - Updated generated gRPC files to reflect changes in the proto definitions. - Enhanced the UnimplementedCLIServiceServer to include the AccountSync method with appropriate error handling. * feat: implement accountsync command and functionality for account synchronization - Added the 'accountsync' command to the CLI for syncing missing accounts from a specified peer without block synchronization. - Implemented the HandleAccountSync method in the CommandHandler to manage account-only synchronization. - Updated the CLI help text to include the new accountsync command for user guidance. - Enhanced the Client and GRPC_Server to support the new AccountSync functionality. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260519052600-4d00a637873f in both go.mod and go.sum files, ensuring compatibility with the latest changes. * feat(fastsync): enforce strict read-only/serve-only sync guardrails - Removed hardcoded 15m FastSync timeout and wired to 'cfg.FastSync.SyncTimeout'. - Introduced 'PullAllowed' config constraint in the CLI 'CommandHandler'. - Hardened CLI execution paths ('fastsync', 'fastsyncv2', 'accountsync', 'firstsync client') to strictly reject manual pull operations when a node is configured as a serve-only participant ('sync: false'). - Renamed internal CLI boolean flags to explicitly distinguish between pulling and serving capabilities. * refactor(fastsync): standardize config naming to clarify network directionality - Renamed 'Sync' to 'EnablePulling' and 'StartupSync' to 'PullOnStartup' in 'FastSyncSettings'. - Updated underlying YAML tags and viper bindings to use 'enable_pulling' and 'pull_on_startup'. - Modified application bootstrap and CLI handlers to accurately reflect the new flags. - Retained 'Enabled' as the master engine switch to accurately represent full initialization state. - Updated log messaging to accurately reflect 'serve-only participant' mode when pulling is disabled. * feat(scripts): add merkle_check.go for post-sync block integrity validation - Fetches all ZKBlocks from ImmuDB and computes a canonical SHA256 hash per block (BlockHash excluded — it is derived and would be circular). - Builds a single global Merkle tree over all block hashes with indexed FromBlock/ToBlock ranges on each node for O(log n) divergence tracing. - Fixed connection pool race: cancel() is now called AFTER GetLatestBlockNumber, not before, so the pooled connection is not returned prematurely. - Fixed boundary-shift hash collision: all variable-length fields (strings, byte slices, tx.Data) are now written with a 4-byte length prefix via writeVar() to prevent (e.g.) 'AB'+'CD' == 'A'+'BCD' producing identical hashes. - Added tx.ChainID and AccessList coverage to the per-transaction hash. - Output JSON includes from_block/to_block reflecting the actual covered range (skipped error blocks are excluded from the range). - Progress logging now always prints for runs of ≤100 blocks. * chore: increase HTTP server read and write timeouts to 60 seconds * feat: enhance BatchRestoreAccounts with deduplication and chunked operations - Implemented deduplication for address and DID entries in BatchRestoreAccounts to ensure unique keys and maintain the latest updates. - Added logic to chunk operations into smaller batches for execution, adhering to ImmuDB's MaxTxEntries limit, improving transaction handling and performance. - Updated logging for better error tracking during batch execution. * feat: optimize BatchRestoreAccounts with pre-fetching of existing accounts - Implemented a pre-fetch mechanism to retrieve all existing account values in a single GetAll RPC, reducing connection pool exhaustion during concurrent operations. - Refactored the logic to check for existing accounts using the pre-fetched data, improving efficiency and maintaining correctness in account updates. - Enhanced handling of account creation and updates to ensure only necessary writes occur, preserving the latest data while avoiding unnecessary operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260520134039-a630fd87e742 in both go.mod and go.sum files for compatibility with recent changes. - Adjusted the scanning order in getKeysBatch to ascending for reliable prefix filtering. * feat: introduce cursor-based pagination for account listing - Added ListAccountsPaginatedCursor function to provide a cursor-based alternative to ListAccountsPaginated, improving performance by avoiding O(N²) re-scans. - Updated account manager to utilize the new cursor-based pagination in NewAccountNonceIterator and GetAccountsByNonces methods, enhancing efficiency in account retrieval. - Adjusted related logic to support cursor management, ensuring seamless iteration through accounts without unnecessary overhead. * Fix/accountsync/performance (#35) * feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> * feat(logging): add custom headers support to OTEL config and upgrade ion to v0.4.2 * Add/redisstreams (#37) * feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. * feat: integrate Redis for asynchronous account synchronization - Added Redis support for the account sync worker, enabling WriteAccounts and BatchUpdateAccounts to enqueue operations to a Redis Stream, decoupling them from the ImmuDB commit latency. - Updated configuration to include Redis connection settings in jmdn_default.yaml and corresponding defaults. - Enhanced error handling to ensure proper initialization of the sync worker and logging for better traceability. - Refactored account management functions to utilize the new Redis streaming mechanism for improved performance and reliability. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> * Remove unused functions for transaction and access list conversion in immudb_account_manager.go to streamline code and improve maintainability. * feat(config): add Redis authentication and complete Viper env bindings This commit introduces proper support for Redis authentication and ensures all nested configuration structures perfectly map to environment variables via Viper. Changes: - Add Password field to RedisSettings struct in config.go. - Update main.go to pass cfg.Database.Redis.Password to the Redis client during initialization. - Update defaults.go and jmdn_default.yaml to explicitly bind Redis URL to 127.0.0.1:6379 instead of an empty string or wildcard. - Update loader.go's setDefaults() to explicitly register all top-level Security settings (e.g., global_rate_limit, trusted_proxies). - Introduce dynamic Viper default registration for Security.Services in loader.go, fully enabling runtime environment variable overrides for nested map fields (e.g. JMDN_SECURITY_SERVICES_EXPLORER_API_RATE_LIMIT). * fix(fastsync): map missing ChainID, AccessList, and LogsBloom fields - FastsyncV2 adapter now maps ChainID and AccessList into non-headers protobufs - immudb_data_writer and immudb_block_nonheaders properly serialize/deserialize these fields to defaultdb - immudb_headers_writer now correctly stores and serves LogsBloom This fixes the Merkle hash divergence on newly fast-synced nodes. * fix(accountsdb): preserve DIDs and metadata during sync - account_sync_worker: parseAccountsPayload now correctly pushes the DID key into the stream so BatchRestoreAccounts successfully creates the bound did: reference for new accounts. - account_immuclient: BatchRestoreAccounts now performs field-merging (DIDAddress, CreatedAt, AccountType, Metadata) when processing partial account updates from Fastsync Reconciliation. This prevents data loss for active accounts. - account_immuclient: PutNonceofAccount migrated to a monotonic nanosecond counter (Option C) to resolve silent uint64 overflow bugs. * feat(migration): separate nonce patching scripts and use O(1) nonce fetch - Created standalone Scripts/replay_blocks_and_export.go for computing TxCount - Created standalone Scripts/patch_accounts.go for updating Nonces and standardizing timestamps - Replaced CheckNonceAndGetLatest in Security.go with O(1) GetAccount verification - Fixed new account generation in DB_OPs to initialize Nonce to 0 natively - Hardened DID registration to enforce UnixNano() timestamps * fix(security): resolve same-block nonce replay vulnerabilities and decouple TxCountSent - schema: added TxCountSent to Account schema to decouple analytical stats from cryptographic Nonce - db: added UpdateAccountSenderState to handle atomic balance and state updates - core: injected execution-time strict nonce validation (tx.Nonce < didDoc.Nonce) in deductFromSender to eliminate same-block double-execution loophole - core: fixed severe bug where sequencer failed to persist Nonce updates to DB after block execution - scripts: created patch_accounts_v2.go to safely migrate accountsdb by mapping TxCount -> TxCountSent and MaxNonce+1 -> Nonce * fix(redis): remove hard startup dependency for account sync Moved Redis consumer group initialization into an asynchronous retry loop inside runWorker. The node now boots successfully even if Redis is completely offline, gracefully degrading bulk sync operations while preserving live transaction processing. Retry interval set to 30s to minimize log spam. * refactor(account_sync): streamline account sync worker initialization and enhance Redis integration This commit refactors the account synchronization process by modifying the initialization of the account sync worker. The StartAccountSyncWorker function now creates a WorkerManager and installs it as the package-level queue, allowing for lazy worker startup. Additionally, the BatchRestoreAccounts function has been updated to accept a context parameter, improving its integration with the overall context management. The logging messages have been adjusted for clarity, and new methods for managing the worker lifecycle have been introduced to enhance performance and reliability. * chore(deps): update JMDN-FastSync to v0.0.0-20260601052219-40e74741de7c and add redis/go-redis/v9 v9.19.0 dependency - Updated the JMDN-FastSync dependency version in go.mod and go.sum for compatibility with recent changes. - Added redis/go-redis/v9 v9.19.0 as a new dependency in go.mod and go.sum. * fix(accountsdb): remove dead PutNonceofAccount to prevent time.Now() overflow risks * refactor: cleanly decouple consensus account origination and fix StateID generation - Reverted Account options and restored pure 'GenerateStateID()' for locally originated accounts. - Removed hacky deterministic 'StateID' origination from 'BlockProcessing'. Consensus now strictly requires accounts to exist before transacting. - Added pre-flight hook in 'eth_getBalance' facade to natively originate and gossip missing accounts via P2P before consensus. - Updated 'Account' adapter mappings to cross-wire Fastsync 'types.Account' safely ('StateID' -> 'Nonce', 'Nonce' -> 'TxNonce'). - Secured incoming P2P DID Gossips using 'StorePropagatedAccount' to perfectly preserve network StateIDs. * feat(accounts): align struct to Fastsync by renaming StateID to Nonce and adding TxNonce * Implement bounded enqueue for account synchronization - Introduced a new helper function `enqueueRecordsChunked` to split account records into fixed-size messages, ensuring each message adheres to the `maxRecordsPerMessage` limit to prevent exceeding Redis's bulk string size. - Updated `WriteAccounts` and `BatchUpdateAccounts` methods to utilize the new chunking logic, enhancing reliability by allowing best-effort delivery across chunks. - Added a white-box test for the chunking logic to validate message boundaries and error handling. - Updated documentation to reflect changes in the enqueue process and message size constraints. - Added a replace directive in `go.mod` for local development with the JMDN-FastSync repository. * fix: Correct Ethereum TxNonce validation logic to prevent intra-block replays - Security module now exclusively uses SecurityCache for TxNonce resolution, drastically reducing DB I/O during batch validation. - Cache enforces mutation upon successful verification (UpdateTxNonce), mitigating intra-block replay attack vectors. - Renamed cache methods to strictly differentiate between Ethereum TxNonce and Fastsync Nonce (StateID). - Fixed static off-by-one error (MaxNonce vs MaxNonce + 1) in V3 migration script. * fix(pubsub): close topic race condition * chore: fix govet deprecation warnings for Go 1.25+ - Replaced deprecated ioutil.ReadFile and ioutil.WriteFile with os.ReadFile and os.WriteFile in bls-sgin.go - Replaced deprecated reflect.Ptr with reflect.Pointer in seednode.go - Applied auto-formatting via make fmt to clean up unused imports These changes resolve linter errors introduced by standard library deprecations in modern Go versions while keeping the logic functionally identical. * fix(consensus): remove duplicate block processing and fix error propagation - Removes duplicate ProcessBlockLocally from BroadcastBlockToEveryNodeWithExtraData - BroadcastAndProcessBlock no longer returns an error when consensus is not reached (it is a valid BFT outcome) - ProcessVoteCollection now only logs ERROR for genuine infrastructure failures - Removes ad-hoc CONSENSUSERROR prefix in favor of structured context - Fixes double error wrapping that polluted error strings * fix(consensus): structured logging, alert deduplication, and trace propagation - Replaces all unstructured log.Printf/fmt.Printf with structured telemetry using ion and NamedLogger - Contexts are now actively threaded down through consensus state machine methods to preserve OpenTelemetry trace chains - Deduplicates consensus failure alerts by removing the redundant alert in BroadcastAndProcessBlock - Fixes severity levels (Consensus failed -> WARNING, CRDT unsubscribe -> DEBUG/WARN) - Removes emojis from structured log output to ensure parser compatibility - Cleans up unused log imports in the Sequencer triggers map and statemachine files * docs: add CHANGELOG entries for fix/consensus-logging-truthlayer * ci: disable sonarqube job until service is restored --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com> Co-authored-by: Doc <saishibu@jupitermeta.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: neerajchowdary889 <neerajchowdary889@gmail.com> Co-authored-by: Neeraj Vipparla <57310710+neerajvipparla@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
Testing
Checklist
make buildpassesmake lintpassesmake fmt-checkpasses