Repository navigation
Lend part of a card from a running engine and get it back, without a park or a reload #189
Description
Project home: https://github.com/TacoTakumi/vram-lend
My setup
- Ubuntu 24.04, NVIDIA driver 610.43.02, CUDA 13.
- 2x RTX 3090 (24 GB), 1x RTX 5060 Ti (16 GB) and 2x RTX 4060 Ti (16 GB). 96 GB of VRAM, 128 GB RAM.
- llama-swap serves everything. Most models run on llama.cpp. GLM-5.3-Flash and DeepSeek V4.1 Flash run on Project Maya, Qwen 3.8 Flash Next runs on Strata.
- Right now Maya runs GLM-5.3-Flash and DSV41 as three llama-swap entries, on 2, 3 or 5 cards.
- ComfyUI runs under llama-swap on the 5060 Ti.
- VoiceManager runs on a 4060 Ti.
What I want to do
When I want ComfyUI or a voice model, I switch Maya from the 5 card entry to the 2 or 3 card entry. The model reloads and I lose the session cache. When I am done I switch back and it reloads again.
Maya keeps the experts in system RAM and a cache of them on the cards. It can already give part of that cache away inside its own process. The vision encoder starts only when a picture arrives, in VRAM that the model lends from its expert cache. I want the same thing between processes. Maya drops part of its cache on one card and keeps serving, only slower. ComfyUI or the voice model uses that memory. When the borrower is done, Maya takes the memory back and fills its cache again.
So the engine does not stop. It lends N MiB on one named card and gets it back later. Park is all or nothing and needs-mib is fixed at the grant, so neither covers this. It is close to the fit and scenes item on your roadmap, but here the holder keeps running.
Questions
- Would you take a lease that its holder can shrink and grow while it serves? The flow I have in mind:
- A borrower asks for N MiB on card X.
- Berth asks the holder to give back N MiB on card X.
- The holder answers when the memory is free. Berth checks the driver read, as it does for a release now.
- Berth grants the borrower a sized lease.
- When the borrower ends, Berth tells the holder it can take the memory back.
- Should this go in the park protocol? Something like this would work for Maya and Strata:
POST /berth/lend {"card": "0000:05:00.0", "mib": 8192} -> 200 when the memory is free, 409 if the engine cannot give that much
POST /berth/return {"card": "0000:05:00.0"} -> 200 when the engine holds it again
GET /berth/state -> {"state": "hot", "lent": {"0000:05:00.0": 8192}}
-
My model holds five cards and lends from one of them. Will the v0.4 lease table let one lease cover several cards, with a size for each card?
-
I will build this in my own forks of Berth, Maya and Strata. If the shape fits where Berth is going, would you take the Berth side back as a PR?
Thanks!
Activity
T-Crypt commentedon Oct 23, 2026
Yes. This fits Berth, and a box like yours is what multi-GPU needs. We'll build the Berth side.
1. A lease that shrinks and grows. Yes, with your flow. Berth asks the holder for a lend before it would drain it, so Maya keeps serving where today it stops. Two rules on our side:
cudaFreeor its allocator's release), not keep it in a pool./berth/returnafter the borrower's lease ends and the driver shows its memory gone, so the two never overlap. If a lend doesn't free the memory within the release grace (30 s), Berth falls back to today's drain.2. The protocol. Yes, beside
/berth/parkand/berth/resume, in your shape, with three changes:cardis the PCI address, in the formberth daemon --gputakes (0000:05:00.0).409says what the engine can lend right now,{"max_mib": 6144}, so Berth stops asking instead of guessing.GET /berth/stateaddslendablebesidelent: per card, the most the engine will give. Berth reads it before it asks.An engine opts in on its entry with
lend = "hook", aspark = "hook"works today.3. One lease across five cards. Not in v0.3. Today each card has its own berthd, so a process on five cards holds a lease on one card's daemon and shows as a stowaway on the other four. v0.4 brings one berthd for every card, with one lease that covers several cards and a size for each. A lend then shrinks that lease on one card and leaves the other four alone.
4. Your forks. Build the Maya side. We'll build the Berth side on a branch and link it here, so we don't write it twice. If you change the Berth side while you test, send it as a PR and we'll review it.
What would help tonight
All of this runs on v0.3.6, or with no Berth at all.
When the branch can lend, we'll post a small stub engine that serves
/berth/lendand/berth/return, so you can test the Berth side before Maya's is done.TacoTakumi commentedon Oct 23, 2026
Maya's five card entry is serving right now, so 1, 2 and 4 are below. 3 and 5 need the cards free, so they come later.
1. Cards
Card 4 drives the display.
2. Maya on five cards
Maya is one process across all five cards.
Each card has about 600 MiB free. From Maya's load log, each card holds one range of layers, and most of each card is the expert pool:
"Everything else" is dense weights, KV at 256K, activations and buffers. So a lend comes out of the expert pool and no layers move.
3. Maya's lend against the driver
Maya already lends part of its expert pool to its own vision encoder. It empties the tail of the pool, frees it with cudaFree and keeps serving. When the encoder ends it allocates the tail again. Today that lend has three limits:
So I can't do 8 GiB on any card yet. When the rig is free I'll time the existing 2.69 GB lend on card 0 against nvidia-smi. Then I'll extend it in my fork to any card, any size and while requests run, and post the 8 GiB numbers.
4. llama-swap entries
Paths cut. The three Maya entries differ only in the config file, which sets the cards:
In the matrix, ComfyUI can run with the 2 card entry. The 3 and 5 card entries evict it. DeepSeek V4.1 on Maya has the same three entries.
VoiceManager is not in llama-swap. It runs on card 3 (09:00.0), and each voice session starts two audio.cpp sidecars there. The TTS sidecar (Breeze-TTS-2 q8_0) uses 5136 MiB, and the STT sidecar (Nemotron 3.5 f16) uses about 1480 MiB. That is about 6.5 GiB per session, and it is freed when the session ends. With Maya on five cards, card 3 has about 600 MiB free, so voice can't run. That is a case I want the lend for.
5. Two berthd on the 3090s
When the cards are free.
T-Crypt commentedon Oct 23, 2026
Thank you, this is what we needed. It settles the order of work.
What your numbers decide
The model all of this follows is written up in
docs/design.md(in review now, together with the first piece built on it: parking an engine in RAM withcuda-checkpoint, #191). Your box is stage C in its table.What we build, in order
nvidia-smireports it.berth run --gpu <bus id>for a run that uses one card (ComfyUI on 05:00.0, a voice sidecar on 09:00.0)./berth/lendand/berth/returnon the protocol as agreed, asked of the share on the borrower's card.We'll post here when step 1 can run on your box, with the exact commands.
What still helps, when the rig is free
nvidia-smi, then your 8 GiB numbers from the fork.TacoTakumi commentedon Oct 24, 2026
VoiceManager
The VoiceManager server is one long-lived process. For each session it starts two child processes on 09:00.0: the TTS sidecar when the client connects, and the STT sidecar when the session first needs it. At the end of the session it stops both (SIGTERM, then SIGKILL after 5 s).
The engines don't change after startup, so the size of a session is known. One share per voice session, about 6.5 GiB on 09:00.0. For us the harness that starts a voice conversation asks for it before it starts VoiceManager and gives it back when voice ends. VoiceManager could ask per session itself later, but I don't expect every borrower to know about Berth, so the caller asking is the general case.
Item 3: Maya's existing lend, timed against nvidia-smi
Maya on five cards, engine idle between requests, one picture per request so the vision encoder starts each time. A loop read nvidia-smi for 01:00.0 about every 140 ms. Times are from when the client sent the request.
So the driver check works with Maya's lend, and it frees in about a tenth of a second. This is the lend that exists today: card 0 only, a fixed size, engine idle. The 8 GiB numbers on any card, and a lend while requests run, come from my fork.
Item 5: two berthd on the 3090s, v0.3.6, driver 610.43.02
Run as you wrote it, with Maya on all five cards. Both daemons came up and both statuses answered.
The b daemon (02:00.0) shows the same, with gpu1 leased and gpu0 inventory only. Each daemon sees Maya as a squatter with 23498 MiB on its own card, and each found llama-swap by itself. I have the JSON for both if you want it.
T-Crypt commentedon Oct 24, 2026
Thank you, that answers everything we asked.
VoiceManager. One share of about 6.5 GiB on 09:00.0. The harness asks for it before it starts voice and gives it back when voice ends. That's a plain sized lease. One detail is on us. The sidecars are children of VoiceManager, not of the harness, so today Berth would count their memory on 09:00.0 as a stowaway rather than under the harness's lease. The lease will name the processes it covers, so the sidecars count against the 6.5 GiB share.
Item 3. Maya's process drops by 2758 MiB 0.10 to 0.19 s after the request. That drop is the driver read a lend waits for, and it comes well inside the 30 s grace. The return follows rule 2 in
docs/design.md. Berth tells Maya to grow back only after the driver shows the borrower's memory gone, so Maya never regrows into memory a sidecar still holds.Item 5. Each daemon saw Maya on its own card at the right size (23498 MiB), so per-card counting by bus ID works on driver 610. The run also shows what goes wrong with one daemon per card. Both daemons found llama-swap and would each drain it on their own. Both would also admit parks into the same system RAM. One berthd per machine fixes both.
One change to the order. Before the multi-card lease, we're fixing how Berth counts system RAM for parks (#193). Today Berth charges a park its VRAM footprint, not the RAM it holds. On our 4090 box, Berth charges a parked ComfyUI 17.9 GiB, and the process holds 3.2 GiB. With five cards parking into one 128 GB pool, Berth has to measure that per process first. It's a short piece of work, and it lands before step 1.
One more reading, while Maya serves. Maya keeps its experts in system RAM, and how it holds them decides how Berth counts them:
If the experts show up in RssFile (an mmap of the model file), the kernel counts them as available, and a park beside Maya could push them out of the page cache. If they show up in RssAnon, RssShmem or VmPin, the kernel already counts them as used.
TacoTakumi commentedon Oct 24, 2026
The engine side of the lend works in my Maya fork, for both engines (GLM-5.3-Flash and DeepSeek-V4.1), on any card, while requests run.
https://github.com/TacoTakumi/project-maya/tree/vram-lend
The routes are not the /berth/lend and /berth/return we agreed. Maya serves engine-neutral names so any client can use them, and Maya does not track who borrows. Berth keeps that ledger. A lend names the GPU in a gpu field and returns a lease, and a return names the lease, not the GPU.
How it behaves:
Measured on the rig with an 8 GiB lend on 09:00.0 (a 4060 Ti) while one request streams. The times are from when the client sent the request, and nvidia-smi was read every 50 ms.
GLM on all five cards:
DeepSeek on all five cards:
Lendable right after load, GLM on five cards, in MiB: 01:00.0 15372, 02:00.0 16688, 05:00.0 9648, 09:00.0 10320, 0A:00.0 8256. Maya keeps enough on each card that prompt reading stays fast. At the full lendable on the 5060 Ti a 4K prompt still reads at 91% of its speed.
One limit matters for the 30 s grace. A lend that arrives while a long prompt is being read waits for the end of that piece of the read. I saw 9 to 14 s with a 12K token prompt. A very long prompt could push a lend past 30 s. Then the lend answers 503, and Maya gives that memory back by itself when the engine lends it. The state lists that lease until it is back.
Can Berth call these routes? The flow would be: lend on the borrower's card, wait for the driver check as now, and return the lease by its id when the borrower's lease ends.
Strata next, but not today.
T-Crypt commentedon Oct 25, 2026
That's fine. Strata can come whenever it suits you. There's no rush, since we want to get this right.
Yes, Berth can call these routes. Engine-neutral names, with Berth keeping the ledger, fit how we planned the lend. We'll build the Berth side against
/vram/lend,/vram/returnand/vram/state:retrykeeps the borrower's share counted on that card, and Berth retries until Maya takes the memory back./vram/statelists it./vram/stateis the read that tells Berth which shares are still lent.We'll keep working on our side in the meantime, starting with the system RAM count for parks (#193), and we'll post here when we need a specific test from the rig. The
/procreading from my last comment would still help whenever Maya is serving.TacoTakumi commentedon Oct 25, 2026
Here is the /proc reading from the vram-lend build on all five cards, taken 20 s into a streamed answer. The DeepSeek binary is maya-ds4, so pgrep -x maya finds only GLM.
GLM-5.3-Flash
So the GLM experts are in RssShmem, and the kernel already counts them as used.
DeepSeek-V4.1
So most DeepSeek experts are in RssFile. The kernel counts them as available, and a park beside DeepSeek could push them out of the page cache.
One thing for the count: cudaHostAlloc memory is pinned, but VmPin and VmLck read 0 for it. VmPin can't tell Berth what Maya holds. RssShmem can.
The page cache was warm before both loads (108 GiB cached), because these files were read in earlier runs. So the DeepSeek pages came from the cache and not from the disk.
Edit: the project now has a home: https://github.com/TacoTakumi/vram-lend
T-Crypt commentedon Oct 26, 2026
The Berth side of the lend is in the first v0.4 pre-release, v0.4.0-rc.1, as an experimental feature. It is a GitHub pre-release, so a normal install stays on v0.3.6. Testing a pre-release covers installing it.
It calls your
/vram/lend,/vram/returnand/vram/stateroutes as agreed. The contract is in contrib/LEND-PROTOCOL.md. This build lends on one card: eachberthdgoverns one card, so the test is a berthd on 09:00.0, Maya's run declaringlend = "hook"withlend_url, and a sized borrower (VoiceManager with--needsaround 6.5 GiB) asking for that card. Lends across Maya's five cards under one lease come in a later pre-release.What we'd like to see: the borrower granted only after the card shows the memory free, Maya still serving, and the take-back after the borrower stops. A 409 with
retryshould be asked again about once a second.berth eventsaround one lend and return shows all of it (lease.lending,lease.lent,lease.returned).TacoTakumi commentedon Oct 26, 2026
rc.1 on the rig
berth 0.4.0-rc.1, Ubuntu 24.04, driver 615.71.09. One berthd on 09:00.0 (RTX 4060 Ti, baseline 121 MiB). Maya's vram-lend build runs on all five cards under berth run --as maya, with lend = "hook" and lend_url. VoiceManager runs under berth run --needs-mib 6656 and asks for 09:00.0 while Maya streams an answer.
Two bugs
The lend itself works. Berth asked for 6656 MiB, Maya gave 6672 MiB in 0.34 s, and the next card read confirmed it. In the same tick Berth dropped the lend ("it is draining") and drained Maya. Voice got the card 34 s later, after Maya was gone. The result is the same with the borrower at foreground and at interactive.
The ask leaves out the card's baseline. Maya holds the whole card, so avail is 0 and the 121 MiB baseline drops out of the shortfall. After the lend the waiter is still 105 MiB short. lendRoom has no case for a holder that already lent to this waiter, so the waiter falls through to the drain.
With the ask fixed, the grant works, but the take-back never comes. retirable counts the holder's whole card plus the lent memory against the ceiling, and that never fits on a card with a baseline. Maya stayed lent until I restarted berthd, and the re-attach returned it.
The fix
Two small changes in internal/broker/lend.go. The ask adds back what the holders count over the ceiling. retirable caps the holder's own count at the ceiling, so only other leases can block a take-back. The new test fails on both before the change. It's one commit on next, with a changelog entry:
TacoTakumi@095ef50
With the fix, one real voice session
I did not see a 409 with retry. Berth asks for the return only after the borrower's memory is gone, so Maya never answered 409 in these runs.
T-Crypt commentedon Oct 26, 2026
Thank you for running this on the rig, and for tracing both bugs down to the line. Your reading was right. The lend landed, the waiter came up short by the baseline, and nothing in
lendRoomcovered a holder that had already lent to that waiter.I moved the fix one step back, into the number both of your call sites read.
budget()counted a whole-card holder at the card's total, and the baseline sits inside that total. The ceiling then took the baseline off the card a second time. In #211 a whole-card holder counts the ceiling instead: the total minus the reserve, the baseline and any stowaways. That one change fixes the ask, the grant and the take-back.One difference you will see on the rig: Berth now asks Maya for the waiter's need alone, 6656 MiB rather than 6777, because Maya's count no longer includes the 121 MiB baseline. Your test is the basis of the new one, and you're credited as co-author on the commit. It fails on
nextwith the drain you saw.#211 is merged into
nextand will be in v0.4.0-rc.2. When that's out, could you run the same voice session again? The sameberth eventslines would show it, withlease.lendingasking for 6656 MiB.You're right about the 409. Berth asks for the return only after the card shows the borrower's memory gone, so Maya has no reason to refuse it. The retry path is for an engine that is busy at the moment of the ask. Strata answers 409 while a request runs, and I'll cover that path in a live test with Strata on the 4090 here.
Next on our side is lending across Maya's cards under one lease: one berthd governing all five cards, so voice can borrow on 09:00.0 from the same lease that holds the 3090s.
TacoTakumi commentedon Oct 26, 2026
Thanks! I'll run the same voice session on rc.2 when it's out.
If you want to try the lend on Strata, my fork serves the same /vram/lend, /vram/return and /vram/state routes as Maya, so a Strata entry can use lend = "hook" with lend_url. It's opt in with "vram_lend": true in the server config.
https://github.com/TacoTakumi/Strata/tree/vram-lend
Niko1221/Strata#4168
T-Crypt commentedon Oct 26, 2026
v0.4.0-rc.2 is out with the baseline fix (#211). Like rc.1, it's a GitHub pre-release, so a normal install stays on v0.3.6.
We also ran a live lend here on an RTX 4090, with Strata as the holder and a 3 GiB borrower. It found one more case, fixed in #215, which will be in rc.3. Strata gave 2939 MiB when asked for 3072, and the card's own free memory covered the gap. berthd confirmed the lend from the card, then counted only the engine's figure and drained the holder. If Maya ever gives less than it's asked for, you'd see the same thing on rc.2. In your runs it gave a little more, so rc.2 should be enough for the voice session.
With #215, the 4090 run went: lend asked, confirmed 3.7 s later, borrower granted, borrower gone, and the memory back with Strata 0.38 s after the card showed it free. Strata stayed on the card throughout.
T-Crypt commentedon Oct 26, 2026
v0.4.0-rc.3 is out, also as a pre-release. Use it rather than rc.2. It has the fix for an engine that lends a little less than asked (#215), and one more lend fix from the 4090 tests (#216). berthd used to give up on a lend 30 s after the ask while
berth rungave the engine 60 s to answer. It now waits the same 60 s for the answer, and the 30 s for the card to show the memory free starts when the engine answers. On the 4090, a lend asked mid-answer landed 38.9 s after the ask and the borrower was granted with no drain.Your voice session test on rc.3 would cover all of it.
TacoTakumi commentedon Oct 27, 2026
rc.3 on the rig
Same setup as the rc.1 run: berthd on 09:00.0, Maya under berth run --as maya with lend = "hook", VoiceManager under berth run --needs-mib 6656 at foreground, asking for the card while Maya streams an answer. One real voice session, TTS then STT.
The lend, the grant and the take-back all work on rc.3. No drain.
One note on the baseline. This berthd started on an empty card, so its baseline is 15 MiB. The 121 MiB in the rc.1 run was Maya's own context on the card at the time berthd read it. The ask is 6656 MiB in both cases with #211, so it changes nothing here.
T-Crypt commentedon Oct 27, 2026
Thanks for the run. It shows each part of the lend working on rc.3:
Thanks for the baseline note too. The 121 MiB from the rc.1 run is accounted for.
Next is one berthd for every card on a machine, so your five cards run under one daemon and one
berth status. The first two parts are merged: the node that holds the cards (#218), and onenvidia-smiread for every card (#219). The part that opens every card comes next. Then a lease can name its card, and one lease can span several cards, which covers Maya across all five. It goes out as an rc, and I'll ping you here with what to run.T-Crypt commentedon Oct 27, 2026
v0.4.0-rc.4 is out as a pre-release. It runs every card under one berthd.
--gpu allis the new default, and each card is namedgpuplus its index. Each card keeps its own leases and queue, so a request on one card never drains a holder on another.berth run --gpu gpu3, orgpu = "gpu3"on an engine entry, puts that engine's lease on gpu3. A PCI address works too (--gpu 0000:09:00.0). The run setsCUDA_VISIBLE_DEVICESto the card's UUID unless your environment already sets it, so a launcher that picks its own cards keeps its setting.berth statuslists every card and adds a CARD column to the lease tables.One lease across several cards isn't in this build. Maya on all five cards will hold its lease on the card you name, and its memory on the other four shows as stowaways there. That part comes next.
What would help:
berth daemonwith no--gpu, and pasteberth status.--gpu 0000:09:00.0and VoiceManager with the same flag. Pasteberth eventsaround the lend and the return.If you install the user service again,
berth service installnow writes--gpu all. A unit you already have keeps the--gpuit was written with.TacoTakumi commentedon Oct 27, 2026
rc.4 on the rig
One berthd with no --gpu finds all five cards and names them gpu0 to gpu4. Same voice session as the rc.3 run, with Maya and VoiceManager both on the 4060 Ti at 09:00.0.
The PCI form of --gpu is refused
berth run --gpu 0000:09:00.0 is refused when berthd runs with --gpu all:
00000000:09:00.0 gives the same answer. --gpu gpu3 works, so both runs used gpu3. I think the cause is in internal/cli/daemon.go. With --gpu all, each card gets gpuSel{index: d.Index} and no address, so node.Card.Addr is empty and Node.Card never matches a PCI address. The device from gpu.List already has the BusID.
CUDA_VISIBLE_DEVICES
berth run sets gpu3's UUID on Maya's server. Maya's server then sets its engine's CUDA_VISIBLE_DEVICES from its own config, so Maya still runs on all five cards. VoiceManager names its card as cuda:3 in its config. With only the UUID visible, cuda:3 doesn't exist, so for this run I set it to cuda:0.
The lend on gpu3
berth status during the lend:
Maya's memory on the other four cards shows as squatters, as you said it would. Each one adds a squatter.appeared event and 1 to the squatters caught count on the board.
T-Crypt commentedon Oct 27, 2026
Thanks for the rc.4 run. The lend on gpu3 matches rc.3. Voice was granted 1.54 s after it asked, Maya was back at full size 32 ms after the return, and the largest token gap was 0.43 s.
The PCI form of
--gpu. Your reading of the cause is right. With--gpu all, berthd builds each card from its index and never records the card's PCI address, so a run that names0000:09:00.0matches nothing. A berthd started with--gpu 3has the same gap. I'll fix it so every card carries the address from the device list, and the fix goes out in rc.5. Until then,--gpu gpu3is the way to name that card.CUDA_VISIBLE_DEVICES. This is how CUDA treats the variable. When it lists one UUID, CUDA numbers the listed cards from 0, so the engine sees gpu3 ascuda:0. Your change tocuda:0is the right setting for a run that names one card. If VoiceManager has to keepcuda:3, setCUDA_VISIBLE_DEVICESin its environment and Berth leaves it alone. One caution. CUDA's default order is fastest card first, not PCI order, so withoutCUDA_DEVICE_ORDER=PCI_BUS_IDthe card CUDA callscuda:3may not be the one at 09:00.0. The UUID avoids that.Maya on five cards. That is the part I'm building now.
berth run --gpu gpu0,gpu1,gpu2,gpu3,gpu4, or the same list ingpu =on the entry, will hold one lease on all five cards, granted on all of them at once. The run gets all five UUIDs inCUDA_VISIBLE_DEVICES, in the order you list them, and Maya's memory stops showing up as squatters on the other cards. A drain on any of the five stops Maya.The first version of that lease can't lend. A voice request on gpu3 would drain Maya instead of borrowing from it. Lending from one card of a multi-card lease comes after that. So for your voice setup, keep Maya's lease on gpu3 as you have it now, and use the multi-card lease when you want the squatter lines gone and can live without the lend. I'll ping you here when it is in an rc.
T-Crypt commentedon Oct 27, 2026
v0.4.0-rc.5 is out as a pre-release. It has the two things from your rc.4 run.
--gpuby PCI address.berth run --gpu 0000:09:00.0and00000000:09:00.0now work when berthd runs with--gpu all(#229). Each card takes its address from the device list.One lease across several cards (#230). Give
--gpua comma list, or put the same list ingpu =on the entry:berthd grants the lease on all five cards at once, and only when each one fits. Maya's server starts with
CUDA_VISIBLE_DEVICESlisting the five UUIDs in the order you named them. Its PID is attached on every card, so its memory there counts as the lease's and not as a squatter's. A drain on any of the five stops Maya, and its exit frees all five.As I said before, this lease can't lend yet. A voice request on gpu3 drains Maya instead of borrowing from it. Keep Maya on gpu3 alone for the voice setup.
What would help:
berth status. It should show one lease ID on gpu0 through gpu4 and nobuild/mayasquatter lines.berth run --gpu gpu3and pasteberth events --jsonfrom that point. It should show Maya drained once and every card freed. The text form ofberth eventsdoesn't print the card yet, so the five grant lines look the same there.--jsonnames each card.berth run --gpu 0000:09:00.0once, to confirm the address form.T-Crypt commentedon Oct 28, 2026
v0.4.0-rc.6 adds two things for a machine with several cards:
berth eventsprints the card on each line when berthd governs more than one, so you no longer need--format jsonto see it. On one card the lines are unchanged.The three checks from my last comment still apply, and rc.6 is the build to run them on.
TacoTakumi commentedon Oct 28, 2026
rc.6 on the rig
All three checks match what you expected. One berthd with no --gpu, berth 0.4.0-rc.6.
C1: Maya under one lease on five cards
Maya starts under berth run --as maya --gpu gpu0,gpu1,gpu2,gpu3,gpu4. The lease is granted on all five cards in the same millisecond, and Maya is ready 45 s later.
There are no squatter rows. Maya's server gets the five UUIDs in CUDA_VISIBLE_DEVICES, in the order I listed them. Maya's server then sets its engine's CUDA_VISIBLE_DEVICES to 0,1,2,3,4 with CUDA_DEVICE_ORDER=PCI_BUS_ID from its own config. Those are the same five cards.
C2: a foreground run on gpu3 drains Maya
With Maya idle I ran berth run --needs-mib 1 --priority foreground --gpu gpu3 -- sleep 20.
The same events from berth events --format json
{"seq":24,"at":"2026-10-28T10:48:24.949908707-03:00","card":"gpu3","kind":"lease.requested","lease":"c2-probe-4","engine":"c2-probe","bytes":1048576,"detail":"foreground"} {"seq":25,"at":"2026-10-28T10:48:24.949925813-03:00","card":"gpu3","kind":"lease.draining","lease":"maya-3","engine":"maya","detail":"c2-probe (foreground) needs the card"} {"seq":26,"at":"2026-10-28T10:48:24.949927527-03:00","card":"gpu3","kind":"lease.waiting","lease":"c2-probe-4","engine":"c2-probe","detail":"next in line: draining maya [interactive, active]"} {"seq":27,"at":"2026-10-28T10:48:25.757092397-03:00","card":"gpu3","kind":"lease.waiting","lease":"c2-probe-4","engine":"c2-probe","detail":"next in line: waiting for maya [interactive, draining] to free memory"} {"seq":28,"at":"2026-10-28T10:48:32.984261051-03:00","card":"gpu0","kind":"lease.released","lease":"maya-3","engine":"maya","bytes":23794286592} {"seq":29,"at":"2026-10-28T10:48:32.98427174-03:00","card":"gpu1","kind":"lease.released","lease":"maya-3","engine":"maya","bytes":587202560} {"seq":30,"at":"2026-10-28T10:48:32.984277345-03:00","card":"gpu2","kind":"lease.released","lease":"maya-3","engine":"maya","bytes":499122176} {"seq":31,"at":"2026-10-28T10:48:32.984282588-03:00","card":"gpu3","kind":"lease.released","lease":"maya-3","engine":"maya","bytes":450887680} {"seq":32,"at":"2026-10-28T10:48:32.984293647-03:00","card":"gpu3","kind":"lease.waiting","lease":"c2-probe-4","engine":"c2-probe","detail":"next in line: waiting for maya [interactive, releasing] to free memory"} {"seq":33,"at":"2026-10-28T10:48:32.984298875-03:00","card":"gpu4","kind":"lease.released","lease":"maya-3","engine":"maya","bytes":450887680} {"seq":34,"at":"2026-10-28T10:48:33.745381131-03:00","card":"gpu0","kind":"lease.freed","lease":"maya-3","engine":"maya","bytes":23918018560} {"seq":35,"at":"2026-10-28T10:48:33.746462671-03:00","card":"gpu1","kind":"lease.freed","lease":"maya-3","engine":"maya","bytes":24580718592} {"seq":36,"at":"2026-10-28T10:48:33.74658267-03:00","card":"gpu4","kind":"lease.freed","lease":"maya-3","engine":"maya","bytes":15223226368} {"seq":37,"at":"2026-10-28T10:48:33.747179133-03:00","card":"gpu3","kind":"lease.freed","lease":"maya-3","engine":"maya","bytes":16022241280} {"seq":38,"at":"2026-10-28T10:48:33.747190492-03:00","card":"gpu3","kind":"lease.granted","lease":"c2-probe-4","engine":"c2-probe","detail":"waited 8.789s"} {"seq":39,"at":"2026-10-28T10:48:33.747199828-03:00","card":"gpu2","kind":"lease.freed","lease":"maya-3","engine":"maya","bytes":15929966592} {"seq":40,"at":"2026-10-28T10:48:53.778234678-03:00","card":"gpu3","kind":"lease.released","lease":"c2-probe-4","engine":"c2-probe"} {"seq":41,"at":"2026-10-28T10:48:55.67691238-03:00","card":"gpu3","kind":"lease.freed","lease":"c2-probe-4","engine":"c2-probe"}C3: --gpu by PCI address
berth run --gpu 0000:09:00.0 and --gpu 00000000:09:00.0 are both granted on gpu3, with gpu3's UUID in CUDA_VISIBLE_DEVICES.
I didn't run a lend from the five-card lease, since you said it can't lend yet. The voice setup stays on the gpu3 lease.
T-Crypt commentedon Oct 28, 2026
Thanks for running all three. Each one matches what berthd should do, so the five-card lease is good for v0.4.0.
Two notes on C2:
lease.releasedandlease.freedcount different things.releasedis what the lease still held on each card when Maya's process exited. Maya had spent 8 s shutting down by then and had already given back most of gpu1 to gpu4.freedis the most the lease held on that card, and it fires once Maya is gone from the card.Lending from a lease on several cards won't be in v0.4.0. Keep the voice setup on its own gpu3 lease until then.
rajputkunal4151-jpg commentedon Oct 29, 2026
Voice command
T-Crypt commentedon Oct 29, 2026
v0.4.0-rc.8 is the build I plan to release as v0.4.0. It has three changes that matter on a rig with five cards:
probe_error.berth notifytitles name the card (gpu1: GPU passed to a VM), and squatter and alert webhooks carry acardfield.on gpu0..gpu4.If you can run it for a day the way you run rc.6, any change in what C1 and C2 showed would hold the release. Lending from a lease on several cards is still after v0.4.0.