feat: Implement Oracle and RocksDB storage services for chunk management#78
Open
Emanuel250YT wants to merge 3 commits into
Open
feat: Implement Oracle and RocksDB storage services for chunk management#78Emanuel250YT wants to merge 3 commits into
Emanuel250YT wants to merge 3 commits into
Conversation
Emanuel250YT
commented
May 17, 2026
Emanuel250YT
commented
- Add OracleService for structured metadata storage with CRUD operations.
- Introduce RocksDBService for raw binary chunk storage with key-value access.
- Create StorageService as a unified facade for interacting with both storage backends.
- Establish StorageModule to encapsulate storage-related services.
- Define AvaDBStorage smart contract for decentralized chunk storage on Avalanche.
- Implement custom error handling in AvaDBErrors for better clarity.
- Create interfaces for AvaDBStorage to define contract interactions.
- Add deployment script for AvaDBStorage and Registrar contracts.
- Configure Hardhat for custom AvaDB network and deployment settings.
- Set up TypeScript configuration for the project.
- Updated @solarity/hardhat-zkit to version 0.5.17 in package.json - Added patch for ZkitTSGenerator to fix path handling in @solarity/zktype - Created deploy script for AvaDB with deployment order for verifiers, library, registrar, and AvaDB - Implemented comprehensive tests for AvaDB including record creation, updating, deletion, and access control
- Add OracleService for structured metadata storage with CRUD operations. - Introduce RocksDBService for raw binary chunk storage with key-value access. - Create StorageService as a unified facade for interacting with both storage backends. - Establish StorageModule to encapsulate storage-related services. - Define AvaDBStorage smart contract for decentralized chunk storage on Avalanche. - Implement custom error handling in AvaDBErrors for better clarity. - Create interfaces for AvaDBStorage to define contract interactions. - Add deployment script for AvaDBStorage and Registrar contracts. - Configure Hardhat for custom AvaDB network and deployment settings. - Set up TypeScript configuration for the project.
Co-authored-by: Emanuel250YT <61527540+Emanuel250YT@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR layers a new decentralised chunk-storage subsystem onto the existing EncryptedERC repo. It adds an AvaDBStorage Solidity contract (hot/cool replication model, per-chunk access list, registrar-gated uploads), a deployment script + Hardhat network entries for an "avadb" custom L1, and a new NestJS service under avadb-node/ that ingests ChunkUploaded events, verifies hashes, persists raw bytes to RocksDB, persists metadata to Oracle, and calls back confirmReplication on-chain. README is expanded with architecture and operator docs.
Changes:
- New
AvaDBStoragecontract + interface + custom errors, plus deploy script andavadb/avalanche/fujiHardhat networks. - New NestJS
avadb-node(StorageModule wrappingOracleServiceandRocksDBService,BlockchainServiceevent listener,ReplicatorServicepipeline,QueryService/controllers, Swagger setup). - README documentation for the AvaDB storage system;
avadb-node/dist/compiled artifacts accidentally checked in.
Reviewed changes
Copilot reviewed 26 out of 74 changed files in this pull request and generated 21 comments.
Show a summary per file
| File | Description |
|---|---|
contracts/avadb/AvaDBStorage.sol |
Main new contract; hot/cool flow, access control, replicator admin. |
contracts/avadb/interfaces/IAvaDBStorage.sol |
Interface, events, ChunkMetadata struct. |
contracts/avadb/errors/AvaDBErrors.sol |
New custom error declarations. |
scripts/deploy-avadb.ts |
Deploys verifier, Registrar, and AvaDBStorage. |
hardhat.config.ts |
Adds avalanche, fuji, avadb networks and PK plumbing. |
README.md |
New AvaDB section, plus incidental blank-line edits in eERC docs. |
avadb-node/src/storage/oracle.service.ts |
Oracle DDL + CRUD; uses reserved OWNER column and broken ROWNUM pagination. |
avadb-node/src/storage/rocksdb.service.ts |
RocksDB key/value layer with iterator-based listing. |
avadb-node/src/storage/storage.service.ts |
Facade combining Oracle + RocksDB (non-atomic despite the wording). |
avadb-node/src/replicator/replicator.service.ts |
Persists locally then confirms on-chain; uses dynamic ethers import. |
avadb-node/src/blockchain/blockchain.service.ts |
Listens to contract events; constructs Wallet from possibly empty key. |
avadb-node/src/blockchain/abi/AvaDBStorage.json |
Hand-maintained ABI duplicated under dist/. |
avadb-node/src/query/dto/query.dto.ts |
Query DTO with fragile boolean coercion. |
avadb-node/tsconfig.json |
Disables most strict* flags for the new service. |
avadb-node/package.json |
Pulls in unmaintained rocksdb@5.2.1. |
avadb-node/dist/** |
Committed compiled output; should be gitignored. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| BEGIN | ||
| EXECUTE IMMEDIATE 'CREATE TABLE CHUNKS ( | ||
| CHUNK_ID VARCHAR2(66) PRIMARY KEY, | ||
| OWNER VARCHAR2(42) NOT NULL, |
Comment on lines
+276
to
+282
| const sql = ` | ||
| SELECT * FROM ( | ||
| SELECT c.*, ROWNUM rn FROM CHUNKS c | ||
| WHERE ${conditions.join(' AND ')} | ||
| ORDER BY ${safeOrderBy} ${safeOrder} | ||
| ) WHERE rn > :offset AND rn <= :limitPlusOffset | ||
| `; |
Comment on lines
+27
to
+41
| * Persist a chunk to both backends atomically. | ||
| * 1. Raw bytes → RocksDB | ||
| * 2. Metadata record → Oracle | ||
| */ | ||
| async storeChunk( | ||
| meta: ChunkRecord, | ||
| rawData: Buffer, | ||
| ): Promise<void> { | ||
| this.logger.debug(`Storing chunk ${meta.chunkId} (${rawData.length} bytes)`); | ||
|
|
||
| await Promise.all([ | ||
| this.rocksdb.putChunk(meta.chunkId, rawData), | ||
| this.oracle.upsertChunk(meta), | ||
| ]); | ||
| } |
Comment on lines
+128
to
+182
| try { | ||
| // Skip if we already have this chunk locally | ||
| const alreadyStored = await this.storage.hasChunk(chunkId); | ||
| if (alreadyStored) { | ||
| this.logger.debug(`Chunk ${chunkId} already stored — skipping`); | ||
| return; | ||
| } | ||
|
|
||
| // ── a) Decode raw bytes from event ────────────────────────────────── | ||
| const rawBytes = Buffer.from( | ||
| event.data.startsWith('0x') ? event.data.slice(2) : event.data, | ||
| 'hex', | ||
| ); | ||
|
|
||
| // ── b) Verify content hash ────────────────────────────────────────── | ||
| const { ethers } = await import('ethers'); | ||
| const computedId = ethers.keccak256(rawBytes); | ||
| if (computedId.toLowerCase() !== chunkId.toLowerCase()) { | ||
| this.logger.error( | ||
| `Hash mismatch for chunk ${chunkId}: got ${computedId}`, | ||
| ); | ||
| return; | ||
| } | ||
|
|
||
| // ── c) Compute CID (sha256 hex) ───────────────────────────────────── | ||
| const cid = '0x' + crypto.createHash('sha256').update(rawBytes).digest('hex'); | ||
|
|
||
| // ── d) Persist in RocksDB + Oracle ────────────────────────────────── | ||
| await this.storage.storeChunk( | ||
| { | ||
| chunkId, | ||
| owner: event.owner, | ||
| contentHash: chunkId, | ||
| cid, | ||
| requiredReplicas: Number(event.requiredReplicas), | ||
| confirmedReplicas: 0, | ||
| state: 0 /* Hot */, | ||
| uploadedAt: new Date(Number(event.timestamp) * 1000), | ||
| isPrivate: event.isPrivate, | ||
| blockNumber: event.blockNumber, | ||
| txHash: event.transactionHash, | ||
| }, | ||
| rawBytes, | ||
| ); | ||
|
|
||
| // ── e) Confirm replication on-chain ───────────────────────────────── | ||
| await this.blockchain.confirmReplication( | ||
| chunkId, | ||
| cid, | ||
| this.nodeEndpoint, | ||
| ); | ||
|
|
||
| this.logger.log( | ||
| `Replicated chunk ${chunkId} (${rawBytes.length} bytes) → cid=${cid}`, | ||
| ); |
Comment on lines
+143
to
+144
| const { ethers } = await import('ethers'); | ||
| const computedId = ethers.keccak256(rawBytes); |
Comment on lines
+297
to
+318
| function getChunkMetadata( | ||
| bytes32 chunkId | ||
| ) external view returns (ChunkMetadata memory meta) { | ||
| if (_chunks[chunkId].owner == address(0)) revert AvaDB_ChunkNotFound(); | ||
| return _chunks[chunkId]; | ||
| } | ||
|
|
||
| /** | ||
| * @notice Check if `user` can read a chunk | ||
| * @param chunkId Identifier of the chunk | ||
| * @param user Address to check | ||
| * @return true if the chunk is public or the user has been granted access | ||
| */ | ||
| function isChunkAccessible( | ||
| bytes32 chunkId, | ||
| address user | ||
| ) external view returns (bool) { | ||
| ChunkMetadata storage chunk = _chunks[chunkId]; | ||
| if (chunk.owner == address(0)) return false; | ||
| if (!chunk.isPrivate) return true; | ||
| return hasAccess[chunkId][user]; | ||
| } |
Comment on lines
+25
to
+29
| @ApiPropertyOptional({ description: 'Filter private/public chunks' }) | ||
| @IsOptional() | ||
| @IsBoolean() | ||
| @Type(() => Boolean) | ||
| isPrivate?: boolean; |
Comment on lines
+143
to
+234
| function uploadChunk( | ||
| bytes calldata data, | ||
| uint256 requiredReplicas, | ||
| bool isPrivate | ||
| ) external returns (bytes32 chunkId) { | ||
| if (!registrar.isUserRegistered(msg.sender)) | ||
| revert AvaDB_UserNotRegistered(); | ||
| if (requiredReplicas == 0) revert AvaDB_InvalidReplicas(); | ||
|
|
||
| chunkId = keccak256(data); | ||
|
|
||
| if (_chunks[chunkId].owner != address(0)) revert AvaDB_AlreadyUploaded(); | ||
|
|
||
| _chunks[chunkId] = ChunkMetadata({ | ||
| owner: msg.sender, | ||
| contentHash: chunkId, | ||
| cid: "", | ||
| requiredReplicas: requiredReplicas, | ||
| confirmedReplicas: 0, | ||
| state: DataState.Hot, | ||
| uploadedAt: block.timestamp, | ||
| isPrivate: isPrivate | ||
| }); | ||
|
|
||
| // Owner always has access | ||
| hasAccess[chunkId][msg.sender] = true; | ||
|
|
||
| // Emit data into event logs — cheap hot storage | ||
| emit ChunkUploaded( | ||
| chunkId, | ||
| msg.sender, | ||
| requiredReplicas, | ||
| data, | ||
| isPrivate, | ||
| block.timestamp | ||
| ); | ||
| } | ||
|
|
||
| /////////////////////////////////////////////////// | ||
| /// Replication /// | ||
| /////////////////////////////////////////////////// | ||
|
|
||
| /** | ||
| * @notice Called by a replicator node after it has persisted the chunk | ||
| * locally. Once the confirmation count reaches the cool threshold, | ||
| * the chunk state is flipped to Cool and a `ChunkCooled` event is | ||
| * emitted so all other nodes can discard the hot-cache entry. | ||
| * | ||
| * @param chunkId The identifier of the chunk (keccak256 of raw data) | ||
| * @param cid Content identifier assigned by the replicator (e.g. IPFS CID) | ||
| * @param location Network address / endpoint where this node serves the data | ||
| */ | ||
| function confirmReplication( | ||
| bytes32 chunkId, | ||
| string calldata cid, | ||
| string calldata location | ||
| ) external { | ||
| if (!registeredReplicators[msg.sender]) revert AvaDB_NotReplicator(); | ||
|
|
||
| ChunkMetadata storage chunk = _chunks[chunkId]; | ||
| if (chunk.owner == address(0)) revert AvaDB_ChunkNotFound(); | ||
| if (chunk.state == DataState.Cool) revert AvaDB_ChunkAlreadyCool(); | ||
| if (bytes(replicatorLocations[chunkId][msg.sender]).length > 0) | ||
| revert AvaDB_AlreadyReplicated(); | ||
|
|
||
| replicatorLocations[chunkId][msg.sender] = location; | ||
| chunk.confirmedReplicas++; | ||
|
|
||
| // First replicator wins the CID assignment | ||
| if (bytes(chunk.cid).length == 0) { | ||
| chunk.cid = cid; | ||
| } | ||
|
|
||
| emit ReplicationConfirmed( | ||
| chunkId, | ||
| msg.sender, | ||
| location, | ||
| cid, | ||
| chunk.confirmedReplicas, | ||
| chunk.requiredReplicas | ||
| ); | ||
|
|
||
| // Transition to Cool when ≥ threshold% confirmed | ||
| uint256 needed = (chunk.requiredReplicas * replicationThreshold) / 100; | ||
| // Ensure at least 1 replicator is required | ||
| if (needed == 0) needed = 1; | ||
|
|
||
| if (chunk.confirmedReplicas >= needed) { | ||
| chunk.state = DataState.Cool; | ||
| emit ChunkCooled(chunkId, chunk.cid, chunk.confirmedReplicas); | ||
| } | ||
| } |
Comment on lines
+11
to
+14
| # AvaDB — Encrypted ERC-20 Protocol + Decentralised Storage | ||
|
|
||
| > **AvaDB** extends the eERC privacy stack with a fully on-chain decentralised | ||
| > storage layer optimised for the AvaDB custom L1 (Chain ID `1152111412`). |
| - **Built-in Compliance**: Supports external and rotatable auditors, ensuring regulatory compliance. | ||
|
|
||
|
|
||
| - **Dual-Mode Operation**: Supports both creating new private tokens and converting existing ERC-20 tokens their private versions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.