feat(umbp): chunked DRAM MR registration for NICs with limited max_mr_size#251
Open
feat(umbp): chunked DRAM MR registration for NICs with limited max_mr_size#251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ibv_reg_mrregistration into chunks to prevent abort whenthe NIC's
max_mr_sizeis smaller than DRAMTier capacitymax_mr_sizeat runtime viaibv_device_attr; alsosupports
UMBP_MAX_MR_CHUNK_SIZEenv var andmax_mr_chunk_sizeconfigfield for manual override
max_mr_sizeHeartbeat()silently discardingtier_capacitiesupdatesChanges
IOEngine / RDMA layer:
Backend::GetMaxMemoryRegionSize()virtual method;RdmaBackendqueries
min(max_mr_size)across all devices with a 4 GB cap for PensandoIOEngineexposesGetMaxMemoryRegionSize()delegating to backendsPoolClient auto-chunking:
Init()detects chunk size, splits DRAM buffer registration intoper-chunk
export_dram_mems_entries; normalizes toSIZE_MAX(nochunking, zero overhead) when
effective_chunk >= max(buffer_size)RegisterMemory()splits into chunks;RegisteredRegiongains agroup_basefield soDeregisterMemory()can batch-remove all chunksbelonging to one original registration call
DRAMTier allocator:
SetAlignmentBoundary()to guaranteeAllocate()never returns aslot that crosses a chunk boundary
Location ID mapping:
LocalStorageManager::BuildTierLocationInfo()changed fromstatictoinstance method; maps global offset →
chunk_index:chunk_offsetwhenchunking is active
UMBPClient::MaybePublishLocal()adapted accordinglyConfig & bindings:
UMBPDistributedConfig/PoolClientConfiggainmax_mr_chunk_sizeUMBPClientconstructor readsUMBP_MAX_MR_CHUNK_SIZEenv varBug fix:
ClientRegistry::Heartbeat()had(void)tier_capacities— capacityupdates from clients were silently dropped. Now writes them into
ClientRecord.Test plan
HeartbeatUpdatesCapacities)max_mr_chunk_sizeis readable/writableBUILD_UMBP=1 BUILD_TESTS=1 pip install .builds cleanlyUMBP_MAX_MR_CHUNK_SIZE=1048576fordistributed Put/Get across chunk boundaries