Add/catchup#45
Open
saishibunb wants to merge 23 commits into
Open
Conversation
…ization - Added CatchUpSync method to CommandHandler for syncing from a specified block. - Updated main.go to trigger CatchUpSync on startup based on configuration settings. - Enhanced FastSyncSettings struct to include CatchUpFromBlock and CatchUpPeer options. - Updated CLI commands to support new catchup functionality and provide usage instructions. - Added relevant entries to .gitignore for new files and documentation. This change enhances the synchronization process by allowing nodes to catch up from a specific block, improving overall efficiency and reliability in block data management.
- Updated CLI commands to reflect new usage patterns for the catchup command, allowing an optional from_block parameter. - Improved error handling and user feedback for catchup synchronization, defaulting from_block to 0 for auto-detection. - Enhanced FastSyncSettings to clarify the purpose of CatchUpFromBlock and CatchUpPeer. - Refactored the HandleCatchUpSync method to support improved synchronization logic and peer address resolution. These changes streamline the catchup process, making it more user-friendly and robust for block synchronization.
- Modified the HandleCatchUpSync method to use BlockHeight instead of BlockMerge for determining the latest block number from peers, improving accuracy in synchronization. - Added error handling for cases where the remote block height is zero, enhancing robustness against outdated server responses. - Cleaned up go.sum by removing unused dependencies related to JMDN-FastSync, streamlining the dependency management. These changes enhance the reliability of the catchup process and ensure a cleaner project structure.
- Updated the JMDN-FastSync dependency to version v0.0.0-20260623122144-03caeb87c1bc in go.mod to ensure compatibility with recent changes.
- Introduced a variable to track the highest block number written during the batch process, ensuring accurate updates to the latest block. - Removed the previous guard that prevented overwriting transactions in the DataSync response, allowing for proper handling of blocks with no transactions. - Enhanced error handling for block updates, ensuring that stale data is cleared and the latest block number is updated correctly after processing. - Updated comments for clarity on the logic and implications of changes made to transaction handling and block synchronization. These changes enhance the reliability and correctness of the immudb data writing process.
- Updated the blockNeedsDataSync function call to pass the block directly instead of as a pointer, improving clarity and consistency in the synchronization logic. - This change ensures that the synchronization process accurately assesses the need for data synchronization for each block, enhancing overall reliability in the catchup mechanism.
…hronization logic - Updated the JMDN-FastSync dependency to version v0.0.0-20260623170445-89b4a3381bf4 in go.mod for compatibility with recent changes. - Added logic in HandleCatchUpSync to advance the latest_block to remoteTip, ensuring accurate updates during the catchup process. - Improved logging for error handling when updating the latest_block, enhancing visibility into synchronization issues.
….sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260623172225-ee9db84a5609 in go.mod for compatibility with recent changes. - Added new checksum entries in go.sum to reflect the updated dependency version.
…unt_manager - Increased timeout for GetTransactionsForAccount to 60 seconds to accommodate longer processing times. - Added writeAccountsDirect function to handle account writing directly to ImmuDB when Redis is unavailable, ensuring data integrity without external dependencies. - Improved logging for direct writes to provide better visibility into account processing. - Introduced a delay in HandleCatchUpSync to allow ImmuDB to settle before reconciliation, enhancing synchronization reliability. These changes improve the robustness and reliability of account management and synchronization processes.
- Added logic to collect tagged accounts from local blocks when data synchronization is skipped, ensuring that account balances are updated during reconciliation. - Implemented the collectTaggedAccountsFromBlocks function to scan local transactions and gather unique sender and receiver addresses for improved balance management. These changes improve the accuracy of account balance updates in the catchup process, enhancing overall synchronization reliability.
…cess - Introduced diagnostic counters to track absent blocks, blocks without proofs, blocks with gas but no transactions, and complete blocks during the catchup synchronization. - Enhanced logging to provide insights into the synchronization process, improving visibility into the reasons for block pass/fail outcomes. These changes improve the monitoring and debugging capabilities of the catchup synchronization logic.
- Increased timeout for context in GetTransactionsByAccount to 120 seconds to accommodate larger batch processing. - Updated batch size from 100 to 500 to reduce the number of round-trips during block scans, significantly improving performance. - Refactored block retrieval to use GetBlocksRange for batch processing, enhancing efficiency and error handling. - Improved logging for error scenarios when retrieving block batches, ensuring better visibility into issues during account transaction processing. These changes enhance the performance and reliability of transaction retrieval and account management in the system.
- Updated HandleCatchUpSync to always scan local blocks for tagged accounts, improving reconciliation accuracy when DataSync is skipped or partially completed. - Merged results from local scans with DataSync outcomes to ensure all relevant accounts are included for balance updates. - Improved logging to provide better insights into the number of accounts collected and merged during the reconciliation process. These changes enhance the reliability of account balance updates during the catchup synchronization, ensuring comprehensive account management.
- Enhanced the BatchUpdateAccounts function to handle cases where the account queue is not initialized by logging the situation and directly writing updates to ImmuDB. - Introduced a new batchUpdateAccountsDirect function to manage synchronous updates to ImmuDB, ensuring data integrity when Redis is unavailable. - Improved error logging for enqueue failures, providing better visibility into fallback scenarios during account updates. These changes enhance the robustness of account updates and ensure reliable processing in various operational conditions.
…atchup synchronization - Updated the JMDN-FastSync dependency to version v0.0.0-20260623190349-1e218bd06f7b in go.mod and go.sum for compatibility with recent changes. - Improved logging in HandleCatchUpSync to include the number of accounts processed during reconciliation and the duration of the reconciliation phase, enhancing visibility into the synchronization process.
- Updated the JMDN-FastSync dependency to version v0.0.0-20260623191406-56133298595b in go.mod and added corresponding checksum entries in go.sum to ensure compatibility with recent changes.
…ieval in range - Updated the JMDN-FastSync dependency to version v0.0.0-20260624044452-d59628408292 in go.mod and go.sum for compatibility with recent changes. - Introduced GetTransactionsByAccountInRange function to retrieve transactions for an account within a specified block range, enhancing transaction management capabilities. - Added GetTransactionsForAccountInRange method in immudb_account_manager to facilitate transaction retrieval in a specified range, improving account transaction handling.
…ions in connection pool - Updated the JMDN-FastSync dependency to version v0.0.0-20260624061943-72fc8e50b45c in go.mod and go.sum for compatibility with recent changes. - Increased the MaxConnections in the DefaultConnectionPoolConfig from 20 to 30 to enhance connection handling capabilities.
- Added new fields `TxNonce` and `TxCountSent` to the `accountUpdateWire` struct to support additional transaction metadata. - Updated the `BatchUpdateAccounts` function to include the new fields during account updates, improving data accuracy. - Refactored reconciliation logic in `HandleCatchUpSync` and `handleSyncInternal` to utilize a single-pass delta approach for account updates, optimizing performance and reducing database scans. These changes enhance the efficiency and reliability of account updates and reconciliation processes.
…ation - Introduced the EnableCatchup field in FastSyncSettings to control automatic execution of HandleCatchUpSync on startup, requiring EnablePulling to be true. - Updated main logic to default catch_up_from_block to 0 for full scans from genesis when EnableCatchup is enabled. - Set default value for EnableCatchup to false in the default configuration. These changes enhance the flexibility of the fast synchronization process by allowing users to enable or disable automatic catchup behavior.
…ock handling - Removed the fallback logic for fromBlock in HandleCatchUpSync, which previously defaulted to 1 when fromBlock was 0. This change encourages callers to explicitly set fromBlock for clarity and correctness. - Updated comments to reflect the new expectations for the fromBlock parameter, enhancing code readability. These changes streamline the catchup synchronization process and improve the clarity of parameter usage.
- Added detailed logging for header and block fetching phases in HandleCatchUpSync, including the number of headers/blocks being processed and the duration of each phase. - Introduced tagBlockCount function to calculate the total number of blocks described by a Tag, improving clarity in block counting. - Implemented watchProgress function to log ongoing progress during header and block synchronization, enhancing visibility into the catchup process. These changes improve the observability and performance tracking of the catchup synchronization process, providing better insights during execution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
Testing
Checklist
make buildpassesmake lintpassesmake fmt-checkpasses