Run usbdevfs URBs asynchronously over IOKit - #375
Open
jotpalch wants to merge 2 commits into
Open
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
jotpalch
force-pushed
the
pr-d-usb-async
branch
from
September 8, 2026 07:08
e83b260 to
3a6b624
Compare
jserv
reviewed
Sep 8, 2026
jserv
reviewed
Sep 8, 2026
jserv
reviewed
Sep 8, 2026
jserv
reviewed
Sep 8, 2026
| u->inflight = 0; | ||
| u->nurbs = 0; | ||
| u->inflight_bytes = 0; | ||
| u->disconnected = false; |
Contributor
There was a problem hiding this comment.
disc_drained is not reset with the rest of the per-open async state, and nothing else clears it: usbdev_teardown_locked resets disconnected but leaves this one, and usbdev_init zeroes the table once at startup. A slot whose previous open drained after a disconnect therefore starts its next open with the flag already set, so that open's own disconnect skips the drain in usbdev_do_reap and answers ENODEV with URBs still pending.
Suggested change
| u->disconnected = false; | |
| u->disconnected = false; | |
| u->disc_drained = false; |
Contributor
Author
There was a problem hiding this comment.
Confirmed, and your patch is what landed, with the same pair added to usbdev_teardown_locked beside the disconnected reset it already had, since that is the other place the per-open async state is cleared. Reached through a slot reused across a close and a reopen, because the allocator takes the lowest free slot and two opens in a row land on the same one: with neither reset, the second open loses the URB only its own drain could return (2 back of 3, ep83's lost). I measured the redundancy rather than assuming it, and it cuts both ways: removing either reset alone leaves the lane green, and removing both is what fails it, so the assertion kills the bug but not each site on its own. That is stated in the commit message rather than left for a reader to discover. This one is inside CAP_REAP_AFTER_DISCONNECT, which this project's own review round had recorded as having no test at all.
jserv
reviewed
Sep 8, 2026
jotpalch
force-pushed
the
pr-d-usb-async
branch
from
September 8, 2026 16:07
3a6b624 to
04ca77b
Compare
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/internals.md">
<violation number="1" location="docs/internals.md:1043">
P1: When the drain times out, this restart can submit a queued URB while the orphaned survivor is still owned by IOKit, violating the one-in-flight-per-endpoint guarantee. Keep that endpoint blocked until the orphan callback has arrived, or otherwise prove the abort has fully released the pipe before kicking its FIFO.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| the interface's pipeRefs, and the whole pipe table -- so the two decide it | ||
| once, in `usbdev_drain_for_change`, rather than each on its own. A drain | ||
| that expires settles the slot's own counters and unlinks the survivors, and | ||
| then restarts every endpoint FIFO exactly as the successful path does: |
There was a problem hiding this comment.
P1: When the drain times out, this restart can submit a queued URB while the orphaned survivor is still owned by IOKit, violating the one-in-flight-per-endpoint guarantee. Keep that endpoint blocked until the orphan callback has arrived, or otherwise prove the abort has fully released the pipe before kicking its FIFO.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internals.md, line 1043:
<comment>When the drain times out, this restart can submit a queued URB while the orphaned survivor is still owned by IOKit, violating the one-in-flight-per-endpoint guarantee. Keep that endpoint blocked until the orphan callback has arrived, or otherwise prove the abort has fully released the pipe before kicking its FIFO.</comment>
<file context>
@@ -1038,7 +1038,15 @@ Known gaps at this stage, each printed as an XFAIL by
- once, in `usbdev_drain_for_change`, rather than each on its own.
+ once, in `usbdev_drain_for_change`, rather than each on its own. A drain
+ that expires settles the slot's own counters and unlinks the survivors, and
+ then restarts every endpoint FIFO exactly as the successful path does:
+ `draining` shut all of them, including the ones the kill did not match, and
+ an endpoint whose leader completed inside the window would otherwise be left
</file context>
jotpalch
force-pushed
the
pr-d-usb-async
branch
from
September 9, 2026 06:52
04ca77b to
ce9ab8b
Compare
There was a problem hiding this comment.
2 issues found across 9 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/syscall/usbdev.c">
<violation number="1">
P3: The added usbdev_flush_disc_peers(u) calls in usbdev_fd_cleanup and usbdev_retire_unpublished can never walk peers: usbdev_teardown_locked clears disc_peers_pending to false on every path before these run, and usbdev_flush_disc_peers returns on a false exchange without touching the table lock or calling usbdev_mark_peers_disconnected_tlocked. So the added comment's claim that "a teardown whose kill found the device gone has peers to tell" is not served by these call sites. The peer propagation these sites intend happens instead through usbdev_async_cb's own flush during the kill drain (before teardown clears the flag); if that is the intended mechanism, drop the two flush calls and the comment, or move the flag clear to after the flush if a close-time-discovered disconnect must reach peers.</violation>
</file>
<file name="tests/test-usbdev-urb-loopback.c">
<violation number="1" location="tests/test-usbdev-urb-loopback.c:1272">
P3: In t_control_charge, the CHECK(n < CHARGE_POOL) fails without stopping execution, and the next line writes pool[n]. If the fill loop ever reaches n == CHARGE_POOL (64), `mk_bulk(&pool[n], ...)` and `submit(&pool[n])` index pool out of bounds. Guard the post-loop URBs with an early return when n >= CHARGE_POOL, or stop the fill loop below the array bound.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| TEST("the allowance filled with URBs that will not complete"); | ||
| CHECK(n > 0 && n < CHARGE_POOL, "%d URBs accepted", n); | ||
|
|
||
| mk_bulk(&pool[n], EP_IN, big, 4096); |
There was a problem hiding this comment.
P3: In t_control_charge, the CHECK(n < CHARGE_POOL) fails without stopping execution, and the next line writes pool[n]. If the fill loop ever reaches n == CHARGE_POOL (64), mk_bulk(&pool[n], ...) and submit(&pool[n]) index pool out of bounds. Guard the post-loop URBs with an early return when n >= CHARGE_POOL, or stop the fill loop below the array bound.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-usbdev-urb-loopback.c, line 1272:
<comment>In t_control_charge, the CHECK(n < CHARGE_POOL) fails without stopping execution, and the next line writes pool[n]. If the fill loop ever reaches n == CHARGE_POOL (64), `mk_bulk(&pool[n], ...)` and `submit(&pool[n])` index pool out of bounds. Guard the post-loop URBs with an early return when n >= CHARGE_POOL, or stop the fill loop below the array bound.</comment>
<file context>
@@ -1152,7 +1195,175 @@ static void t_disc_drained_reset(void)
+ TEST("the allowance filled with URBs that will not complete");
+ CHECK(n > 0 && n < CHARGE_POOL, "%d URBs accepted", n);
+
+ mk_bulk(&pool[n], EP_IN, big, 4096);
+ TEST("a further 4 KiB URB no longer fits");
+ CHECK(submit(&pool[n]) == -ENOMEM, "submit rc=%d", errno);
</file context>
jserv
reviewed
Sep 9, 2026
Comment on lines
+1245
to
+1254
| ### Deviations From Linux | ||
|
|
||
| | usbfs behavior | elfuse behavior | | ||
| |---|---| | ||
| | `RESET` re-enumerates the device | clears the claimed pipes' stall state and returns 0; `USBDeviceReEnumerate` would tear down every open plugin handle | | ||
| | isochronous URBs | `EINVAL` | | ||
| | `DISCSIGNAL` delivers a signal on disconnect | signal and context stored, never delivered | | ||
| | sync `BULK` on an interrupt endpoint works (converted to an interrupt URB) | `EINVAL`; the conversion exists only on the async URB path | | ||
| | `BULK_CONTINUATION` unlinks the rest of the cascade on error | flag accepted, no cascade unlink | | ||
| | `dup` of a usbfs fd works | `EBADF` | |
jserv
reviewed
Sep 9, 2026
| opening the host's `/dev/cu.*` node instead. | ||
| - USB mass storage will never be claimable, even with capture; macOS | ||
| does not release it. | ||
| - Isochronous URBs are unimplemented; submitting one reports `EINVAL`. |
Contributor
There was a problem hiding this comment.
Is it feasible to implement isochronous URB?
One lazily-started CFRunLoop host thread (libusb's darwin model) carries CreateDeviceAsyncEventSource / CreateInterfaceAsyncEventSource completions and an IOServiceAddInterestNotification disconnect watch; the thread only ever touches usbdev-owned host memory, never guest memory (the netlink.c blocking-recv rule), so exec and guest teardown need no join. The record's owning slot is the one field the completion callback reads before it can take any lock, so it is atomic: ThreadSanitizer cannot see through IODispatchCalloutFromCFMessage and reported it on a cold-start run. SUBMITURB runs proc_do_submiturb's checks in proc_do_submiturb's order (devio.c:1644-1661): the flags mask, then USBFS_XFER_MAX, then a NULL buffer with a positive length, then the endpoint. So an absent endpoint outranks an unknown transfer type, the ISO refusal and the control arm's eight-byte minimum, and a 2 GB URB on the default control pipe is -EINVAL rather than an allocation. That is not do_proc_bulk's order, which resolves the endpoint first and which pr-c pins for the synchronous ioctl; the two kernel paths really do differ. The endpoint-recipient control check shares pr-c's kernel-shaped helper: ep0 allowed, reserved wIndex bits -EINVAL, warn-and-flip direction fallback. Guest data is bounce-buffered, and actual_length is clamped to the submitted buffer, the way every HCD bounds urb->actual_length by transfer_buffer_length rather than passing a device-supplied count through. At most one URB per endpoint is in flight at IOKit, because AbortPipe cancels a whole pipe and DISCARDURB must cancel one URB. That alone is not enough: the abort is issued with async_lock dropped, and inside that window the target can complete normally and the completion callback can start the queued follower for the abort to hit instead. The endpoint's FIFO therefore stays shut while an abort for it is running, and DISCARDURB waits, bounded at 2s, for the record to leave the pending list -- which is also what makes an immediately following REAPURBNDELAY find the URB the way usb_kill_urb leaves it. Aborted-while-discarding reaps -ENOENT, any other abort -ECONNRESET, and a start IOKit refuses is completed through that same URB-status map rather than the syscall map, whose Aborted row would write -EINTR into urb->status. The resource limits are Linux's. The 16 MB byte budget is the one the synchronous transfers already charge, not a second of its own, because usbfs_memory_usage is a single kernel-wide static (devio.c:143-178) and two counters would let this path queue a budget behind whatever a BULK ioctl holds. Each record is charged alongside its buffer so zero-length URBs are bounded too, and no URB-count cap is left: the per-fd 256-record backstop this drops had refused a 257th eight-byte URB, which is exactly the ring depth libusb's async API builds for bulk streaming, and a lane here now queues 66001 of them on one fd. ZERO_PACKET emits its trailing write with a bounded timeout, with async_lock dropped, and folds a failure into urb->status. The untimed call under the lock could stop completions for every usbdevfs fd in the process, since they all ride one event thread, and a discarded result reported a failed terminating packet as success. SHORT_NOT_OK becomes -EREMOTEIO on a short IN. REAPURB blocks via io_wait_fd_or_interrupted on the completion pipe and returns -EINTR with syscall_restart_forbid() because proc_reapurb does not restart (devio.c:2116-2117); REAPURBNDELAY is -EAGAIN/-ENODEV. What that pipe carries is a level, not one byte per completion: it holds exactly one byte while the fd has a completion to hand back or has been disconnected, and none otherwise, restored under async_lock by every path that can move either term. A byte per completion is unsound now that there is no URB-count cap, because a zero-length URB costs only sizeof(usbdev_urb_t) against the 16 MB budget and far more records fit than the pipe has room for bytes: a nonblocking write whose byte does not fit leaves its record reapable with nothing readable behind it, and poll, select and epoll then all answer 0 for an fd whose next reap returns at once. The reap takes its read under the same lock that dequeued, because a decision made outside it races a completion arriving behind it and strands the token that completion just wrote. Disconnect raises the level and nothing lowers it again, which is what a sticky unmaskable POLLERR|POLLHUP wants. A level is edged once per rise rather than once per completion, so an EPOLLET registration that reaps one URB per wake instead of draining to EAGAIN would wait here where Linux, whose usbfs wakes its queue per completion, fires again; internals.md states that, unreachable for a guest that keeps the EPOLLET contract and untested for the same reason. CAP_REAP_AFTER_DISCONNECT is one invariant: after a disconnect every URB still in flight comes back before any reap answers -ENODEV, whichever reap flavor asked first. Linux gets it for free, because usbdev_remove runs destroy_all_async -- a synchronous usb_kill_urb each -- before it wakes the reapers, so no reap can observe the disconnect until the pending list is already empty. IOKit completes nothing of its own when a device terminates, so this engine issues that kill itself at the first reap that finds the completion list empty on a disconnected fd, and its aborts land asynchronously. The window Linux does not have -- disconnected, aborts issued, URBs not back yet -- is therefore real here, so -ENODEV is decided by the pending list being empty and never by a flag recording that an earlier pass did the work. disc_drained is a one-shot for issuing the aborts and nothing more; a blocking reap drains whatever is still out whether or not that latch is already set. The flag is per slot and the table is zeroed once, at startup, so it resets with the rest of the per-open async state at both places that reset the disconnect flag -- otherwise a slot whose previous open drained starts its next open with the drain already claimed, and that open's own disconnect issues no aborts at all. All four disconnect fields -- disconnected, disc_drained, the deadline below and the peer-walk debt -- reset together in both places, rather than some of them here and the rest left to teardown. The disconnect-watch refcon packs the slot index in a field sized from USBDEV_MAX_FDS behind a static assertion, not in a hand-written four bits: 32 slots through four bits decoded slot 16+k as slot k, so half the table never saw a disconnect and the other half could be marked gone while still attached. A drain that outlives the 2s teardown deadline settles the slot's nurbs/inflight/inflight_bytes and unlinks the survivors before marking them orphaned: the late IOKit callback then frees only the record it owns and the interface pin it holds (no slot counters, no disconnect map, no readiness token), so a reused slot can never be corrupted by a URB from a previous open, and a later kill/teardown scan cannot re-find an orphan and wait another 2s. The wait is spelled with utils.h's timespec_deadline_in_ms, the tree's one form for a pthread_cond_timedwait deadline at twelve other sites, rather than a second copy of the seconds-plus-carry arithmetic; the ceiling the non- blocking arm measures is the same constant on CLOCK_MONOTONIC, and neither is written as an offset from the other. The process-wide byte budget is not one of those counters and is not settled there: it accounts live memory, and the survivor's buffer is still allocated and still owned by an in-flight IOKit transfer, so its refund travels with the free in urb_free_orphan_locked. A charge that never comes back is a transfer whose callback never arrived, which is the honest answer rather than a leak. That path also restarts every endpoint FIFO before it returns, exactly as the success path does: draining shuts all of them, the ones a per-interface kill does not match included, so a leader that completed inside the window left its follower queued with nothing in flight, and the orphans' own late callbacks take the orphaned early return and start nothing at all. The kick covers all 256 keys because the endpoints that need it are precisely the ones the kill did not touch. SETINTERFACE and SETCONFIGURATION both branch on that answer, and in one place rather than each on its own: both retire the handles the survivors still reference -- the interface's pipeRefs, and the whole pipe table -- so a drain that missed its deadline refuses the change instead of making it with transfers still outstanding at IOKit. Linux reaches neither answer, because usb_kill_urb does not give up; the 2s bound is ours, so the EBUSY is a stated deviation in internals.md. SETCONFIGURATION kills the device's URBs (usb_disable_device) including ep0's, which no interface claim covers, and RESET kills them too -- the half of usb_reset_device this layer can do -- but RESET proceeds whatever the drain answered, deliberately: a wedged transfer is the state a guest issues RESET to escape, and a stall clear leaves the pipeRefs where they are. A per-pipe stall clear that fails during RESET is logged rather than returned: the clears are the substitute, and this board answers kIOUSBTransactionTimeout on a device usb_reset_device would reset without complaint. poll/select/epoll: the fd's pipe raises host POLLIN, remapped to guest POLLOUT|POLLWRNORM (devio.c:2833-2847) in sys_ppoll and pselect6; epoll registers EVFILT_READ but reports EPOLLOUT via a per-registration flag. The writability half of that remap reads the same derived capability the ioctl and write gates read rather than testing the access mode against O_RDONLY: usbdev_poll gates EPOLLOUT|EPOLLWRNORM on FMODE_WRITE (devio.c:2840), and access mode 3 is not O_RDONLY yet carries no FMODE_WRITE either. No guest can currently observe the difference, because reaching POLLOUT needs a completion and SUBMITURB on such an fd is already -EPERM, so this is stated as a consistency fix and not as a bug with a test: an assertion here would pass with the literal test too. The host-side interest is always armed regardless of the events the guest asked for: a disconnect must wake even a read-only poll/select/epoll wait with the unmaskable POLLERR|POLLHUP (devio.c:2842-2845), and the completion pipe is the only wake source that reaches such a wait. A wake that maps to nothing guest-visible re-blocks everywhere: ppoll/pselect withdraw the woken entry's host interest (unreaped completions keep the pipe readable, so leaving it armed busy-spins at 100% CPU) and resume in bounded 200ms slices that re-check the disconnect map; epoll mutes the fired knote (EV_DISABLE, or a disabled re-add for a fired EV_ONESHOT so a wake the guest never saw cannot consume an EPOLLONESHOT arm) and re-enters the kevent slice loop instead of returning 0 before the timeout, which Linux ep_poll never does. Undoing those mutes reads the registration and applies its knote change in one step under the instance lock, the way the hangup and mute stamping loops beside it already do. With the lock dropped in between, an EPOLL_CTL_MOD on another thread could re-arm the registration -- clearing oneshot_armed and re-adding an enabled knote -- after the delete had been decided on, and the EV_DELETE then retired the knote MOD had just installed, leaving an active registration with nothing behind it and a disconnected fd, whose pipe already held a wake, silent until its wait timed out. The kevent is changelist-only, so the lock is never held across a wait. Disconnect surfaces POLLERR|POLLHUP through a lock-free per-fd map, read() then reports -ENODEV (devio.c:323-325), and every non-reap ioctl does too. The three sites that can learn the device is gone from an IOKit answer -- SUBMITURB's start, the completion callback, and a queued follower's start from that callback -- share one predicate, so they cannot disagree about which codes mean it; the follower was the one that wrote only the URB's -ENODEV and left the fd unstamped, so a poller saw no POLLERR|POLLHUP and a later ioctl passed the disconnected gate. select() counts a usbfs entry once per set it was asked about rather than once for the completion pipe the host actually watched, so a disconnected fd in both readfds and writefds answers 2, the way fs/select.c reaches EPOLLERR from both POLLIN_SET and POLLOUT_SET. The two maps are cleared before the fd number can be polled through and before the slot is released, and only while no live entry answers to that number: clearing them unconditionally after teardown erased a sibling's bit on a fd number it had already reopened. The lock order in internal.h gains the per-entry async_lock beneath the entry lock and usbdev_loop_lock as a leaf. What do_vfs_ioctl answers before it calls f_op->unlocked_ioctl is answered ahead of all of that, the way io.c already answers FIONBIO and FIOASYNC for these fds: FIOQSIZE, FIGETBSZ, FIFREEZE, FITHAW, FS_IOC_FIEMAP, FICLONE, FICLONERANGE, FIDEDUPERANGE, FS_IOC_GETFSUUID and FS_IOC_GETFSSYSFSPATH each stop in an arm of that switch and never reach usbfs, so connected() is never consulted for any of them. Without that the device question this arm gained was put for them too: measured on a terminated loopback device, all ten answered -1/ENODEV with revents 0x18 on an fd nothing had stamped, and the same on a marked one, where Linux answers -ENOTTY for FIOQSIZE out of do_vfs_ioctl's own arm and asks the device nothing. They answer -ENOTTY here, marked fd or fresh, which is what this layer answered them before the arm learned to ask. Ahead of the FMODE_WRITE gate as well, because that is where do_vfs_ioctl sits relative to the whole file operation: on a read-only fd the ten moved from -EPERM to -ENOTTY with them. FIONREAD did not, and is not in the set: do_vfs_ioctl hands that one to vfs_ioctl for anything that is not a regular file, so it does reach usbfs. Neither are the FS_IOC_*FLAGS and FS_IOC_FS*XATTR arms, which answer -ENOIOCTLCMD for an inode carrying no fileattr operations and are retried through vfs_ioctl. Both meet the gate, and should: measured, FIONREAD on a departed device is -1/ENODEV with the stamp on either fd. Two sentences narrowed to what stands behind them. The comment over the request codes this file defines and dispatches nowhere claimed they made the surface written down the whole surface; the whole surface is Linux's, and CLAIM_PORT, RELEASE_PORT, FREE_STREAMS, DROP_PRIVILEGES, CONNINFO_EX, FORBID_SUSPEND and ALLOW_SUSPEND have no code here at all. They reach that arm without one, so what the three stand for is the arm and not a count, which is all the table needs of them. And internals.md said a marked fd reports ENODEV to every other ioctl: it reports it to every other usbdevfs ioctl, and the requests answered before f_op->unlocked_ioctl are not among them. The lane drives both halves of that sentence now instead of the first alone. Document the engine in internals.md, the unprivileged scope in usage.md and the README limitations, and the hardware-test policy in testing.md. usage.md states what a stock distribution lsusb actually does here: usbutils reaches libusb through libudev, whose monitor wants SO_ATTACH_FILTER, elfuse's netlink layer answers ENOPROTOOPT, and libusb_init gives up with -99. That gap belongs to the netlink layer, not here, but the worked example had to stop claiming otherwise. GET_CAPABILITIES names what this engine honours -- ZERO_PACKET, emitted from the completion callback, and REAP_AFTER_DISCONNECT, now that the reap arm both answers ahead of the connected gate and produces the drain. BULK_CONTINUATION stays clear: the flag is accepted without its error-cascade unlink, and a guest that read the bit would rely on the cascade. Coverage: the fixture the ioctl lane runs against has no IOKit service, so it stops at SUBMITURB's argument gate, and the arithmetic behind that gate now lives in src/syscall/usbdev-urb.h with its own native host binary. tests/test-usbdev-urb-host.c makes 781 assertions over the refcon, the argument gate, the transferred-count clamp, the ZERO_PACKET predicate and the endpoint start gate; reverting each of those five to the shape it had fails it (144, 3, 3, 1 and 1 assertions, and the refcon one is a compile error while the static assertion stands). tests/test-usbdev-ioctl.c gains eleven SUBMITURB argument-order assertions, seven of which fail against the order the engine had first, the endpoint resolved ahead of the gate and no USBFS_XFER_MAX bound at all, and prints XFAILs for the gaps that stay: CLEAR_HALT and RESETEP cancel an in-flight URB because IOKit exposes no stall clear that leaves the pipe alone, a URB's signr and DISCSIGNAL are accepted and never delivered, ISO is -EINVAL once the endpoint resolves, and DISCARDURB's wait is bounded where usb_kill_urb's is not -- that XFAIL now spells out which kernel call is on the path, because the comment naming it has been read as usb_unlink_urb twice: proc_unlinkurb calls usb_kill_urb, the synchronous one, so waiting is what matches Linux and only the 2s ceiling diverges. Verified against the real ESP32-S3 at 303a:1001, one driver run three times: GET_CAPABILITIES is 0x11; a control URB reaps 18 bytes through poll+reap; a bulk IN discard reaps -ENOENT in 6700 of 6700 discards and the REAPURBNDELAY issued straight afterwards finds the URB 6700 of 6700 times, in each of the three runs; 300 URBs queue on one fd and all 300 reap back; 200 rounds of discarding one of two queued ep0 URBs produce 0 bystander -ECONNRESET; RESET returns all 3 queued URBs; a short IN reports 18, not 255, with nothing written past it, and -EREMOTEIO under SHORT_NOT_OK. ThreadSanitizer and AddressSanitizer/UBSan are clean across the same runs, including the driver whose first run reported the record-owner race. A disconnect is a fact about the device, so it is stamped on every usbfs fd open on it, the way usbdev_remove walks udev->filelist. It used to be written only where the news arrived: the terminate watch is armed from SUBMITURB alone and all four marking sites are the async engine's, so a second fd on one node -- an ordinary shape, one fd for control and another for bulk -- was told nothing. Measured with two fds on one device: the submitting one had poll revents 0x0018 and GET_CAPABILITIES -ENODEV while the other had 0x0000 and 0, and that did not change when the second fd's own synchronous CONTROL answered -ENODEV -- publishing a disconnect is not the same as originating one, and only the async engine originated. Every op now translates its IOKit status through one helper that stamps on the device-gone codes, the synchronous transfers and the setup calls included, so the fd that asks is the fd that finds out and its peers are told behind it. That covers the two places whose IOKit status is deliberately not returned: the lazy USBDeviceOpen, whose refusal is tolerated, and RESET's per-pipe stall clear, which is logged. The places that do return a status but dropped this one are below. Tolerating a refusal is not the same as discarding the news that the device left, and RESET in particular answered 0 with the fd unstamped where Linux, whose usbdev_do_ioctl gate runs before proc_resetdevice, answers -ENODEV. The predicate that decides origination is narrower than the -ENODEV row of the syscall map, deliberately and not by oversight. ioret_neg_errno folds kIOReturnNotOpen in with NoDevice and NotAttached, because a device you cannot reach is -ENODEV either way; usbdev_ioret_device_gone covers only the latter two. NotOpen is what IOKit answers for a handle nobody has opened yet, which is the ordinary state of a control request arriving before the lazy open, and that request is answered from a device still plainly attached, so stamping on it would mark a live fd gone on its first ioctl. The file header and internals.md now say which of the two sets is which: "any op that answers -ENODEV originates the disconnect" reads as though they were one set, and they are not. The walk matches the devkey the SETCONFIGURATION claim check already walks, and runs under the table lock, so a slot cannot be torn down and reused between the match and the stamp. Two of the four stamp sites run with async_lock held and the walk takes the peers' one at a time, so those record the debt in a per-slot atomic and it is paid where that lock is already gone: on every ioctl's way out through usbdev_release, and at the completion callback's exits. The file header and internals.md both described the device-wide behavior the code did not have, and now describe what it does. REAPURBNDELAY no longer runs the post-disconnect kill inline. proc_reapurbnonblock pops async_getcompleted and answers -EAGAIN or -ENODEV -- no kill and no wait -- because usbdev_remove has already run destroy_all_async by the time a reap can see the disconnect. This engine owes that kill because IOKit delivers nothing of its own, and performing the whole of it from a non-blocking ioctl meant sitting out the drain's 2s deadline holding async_lock against every SUBMITURB and DISCARDURB on the fd: measured against a wedged endpoint, REAPURBNDELAY returned -ENODEV after 2006 ms and the pass after it after 0. The aborts are what recovers the URBs and they return at once; the wait is only how a caller learns the callbacks landed, and this caller may not want that. So the kill splits in two: a blocking REAPURB aborts, waits and loops, and REAPURBNDELAY issues the same aborts and answers straight away. What it answers is -EAGAIN, not -ENODEV, until those aborts land. Making the split by having the non-blocking pass latch disc_drained gave that one flag two jobs -- issue the aborts once, and report the device drained -- and the second job was wrong the moment the first ran without waiting: every later reap, blocking included, then read the latch and answered -ENODEV with the URBs still in flight. A libusb event loop polls non-blocking before it blocks, so that was the ordinary path. Measured with a never-completing endpoint plus a terminate: REAPURBNDELAY rc=-19 urb=(nil), then REAPURB rc=-19 in 0 ms, with the URB arriving 400 ms after both -- exactly what the capability bit promises will not happen. The latch now only gates the aborts, -ENODEV waits on the pending list, and a non-blocking reap in the meantime gets proc_reapurbnonblock's other answer. Both flavors reach that end, and reach it inside one drain deadline of the aborts. Deciding -ENODEV on the pending list alone left only the blocking arm with a ceiling: its 2s wait times out and orphans the survivors, which empties the list. A REAPURBNDELAY loop had none, so on a wire that answers no abort it issued the aborts on its first pass and could only answer -EAGAIN from then on, for as long as the wire stayed silent. That loop is what libusb_handle_events_timeout(0) runs, so it is the common shape and not a corner: measured against ep81:wedge(600000) with the fd stamped, 4000 ms of REAPURBNDELAY gave 1958386 -EAGAIN and no -ENODEV, with poll holding POLLERR|POLLHUP throughout, where Linux answers connected(ps) ? -EAGAIN : -ENODEV and the application tears down. The non-blocking arm now measures the deadline the blocking one waits out -- the aborts are timestamped when issued, and past the deadline it orphans the survivors itself and answers -ENODEV -- so the contract that it may not block is kept: the -ENODEV pass measured 0 ms, with the deadline spent by the caller's own spinning. The orphaning is one routine shared by both arms rather than a second copy, because what they have to leave behind is the same slot state. Deciding it by widening the latch would have been the earlier bug again: the latch says the aborts went out, and only the clock says the wire has had its time. The interface-enumeration ops answer for the device rather than for the interface. GETDRIVER, DISCONNECT_CLAIM and USBDEVFS_IOCTL's DISCONNECT and CONNECT arms all reach the interface through one CreateInterfaceIterator whose IOReturn was discarded into IO_OBJECT_NULL -- indistinguishable from "the enumeration ran and this interface is not in it". On a device that had gone with u->dev still cached, GETDRIVER answered -ENODATA and the other three -EINVAL, each naming a missing interface on a device that was missing entirely, and none of them stamped the fd, so usbdev_ioctl's own connected() gate could not answer the ioctl behind them either. The lookup now reports that status, the callers return it, and the not-found errno stays each caller's own. Above the lookup sits a short circuit that answered none of that: an interface any usbfs fd holds reports the driver usbfs without the enumeration being opened at all, and a claim is this layer's own bookkeeping, which knows nothing about the device and stays true after it leaves. So on the same departed device GETDRIVER answered 0 with driver usbfs, both USBDEVFS_IOCTL arms answered -EBUSY, and DISCONNECT_CLAIM answered 0 -- taking a fresh claim on an interface of a device that had gone -- each of them leaving the fd unstamped, so poll stayed silent and the ioctl behind them answered from the stale slot too. The four now put the device question to the enumeration before the claim answers rather than instead of it, which is the invariant the helper carries: none of them answers anything, an interface number out of range included, until the device has been asked in that call. Out of range is part of the invariant because two of the three put their own bound above the question, so one departed device gave two answers to one probe: GETDRIVER ifnum 200 was -ENODEV with poll revents 0x18, while DISCONNECT_CLAIM ifnum 200, USBDEVFS_IOCTL ifno 200 and ifno -1 were all -EINVAL with revents 0x0. Neither proc_disconnect_claim nor proc_ioctl bounds the number at all -- their -EINVAL is usb_ifnum_to_if coming back NULL, and proc_ioctl repeats connected() for itself before that lookup -- so the bound here stands in for the lookup and cannot outrank the device the lookup would have run on. All four answer -ENODEV and stamp the fd now. CLAIMINTERFACE, RELEASEINTERFACE and SETINTERFACE keep -EINVAL for 64, where claimintf does bound the number, against the width of ps->ifclaimed rather than against anything the device says; Linux gates that bound behind connected() too, which is a recorded deviation rather than a closed one, because closing it would put a registry enumeration ahead of a check that reads no device state. Linux gets the same answer one level up, from the connected() gate usbdev_do_ioctl runs before any of them. The same drop sat in the pipe-map build: a GetPipeProperties that answers device-gone was a continue, so CLAIMINTERFACE returned 0 with a map shorter than the interface's endpoint count and the guest's next SUBMITURB answered -ENOENT, no such endpoint, where Linux answers -ENODEV. A code about one pipeRef still skips that pipeRef; a code about the device ends the build. SETCONFIGURATION puts the same question to the same enumeration -- has any interface a host driver bound -- through a second copy of the call, and that copy still discarded the status. A device that had departed therefore read as "the enumeration ran and no driver is bound", so the op carried on into the configuration change and answered whatever the wire made of it, stamping nothing, where Linux's connected() gate answers -ENODEV for it as for the other four. All five now open the enumeration through one helper, which is where the "IO_OBJECT_NULL is two answers" invariant is written, and a failed enumeration is that failure's errno rather than "nothing matched". RESET asks the device before it decides it has nothing to do. Its stall clears are the only IOKit calls it makes, and with no interface claimed it makes none of them, so on a device that had gone it ran the whole body and returned 0: measured on a node with no IOKit object behind it, rc=0, and on a fresh fd of a terminated device, rc=0 with revents 0x0. libusb_reset_device is the recovery call an application reaches for after an error, so success there is the answer that keeps it from finding out. Linux runs connected() before proc_resetdevice. The per-pipe clear still translates its own status, which now covers only the window between the gate and the loop rather than standing in for the gate, and the comment saying otherwise is narrowed to that. What the connected() gate itself reads is narrowed with it. It reads what this fd has been told, not what the device is doing: the mark arrives from the terminate watch, from a peer's walk, or from this fd's own next call into IOKit, and open(2) resolves no service, so an fd opened after the device left starts every call with nothing on it. What closes that window is the op itself putting the question to the device, and not every op does. Which ones, and what each answers instead, is a row per request in the table below; this message does not count them, for the reason that paragraph gives. The comment that claimed the gate covered everything says what it covers instead. The synchronous CONTROL path charges the transfer allowance it was outside of. do_proc_control books PAGE_SIZE + sizeof(struct urb) + sizeof(struct usb_ctrlrequest) after the recipient check and the wLength cap and refunds it at its single exit (devio.c:1187 and :1269) -- a fixed amount rather than the request's length, because the kernel bounces every control transfer through one page. This charged in do_proc_bulk's place and nowhere here, so with 16771328 bytes in flight a synchronous BULK answered -ENOMEM and a synchronous CONTROL of wLength 4096 went through and reported 4096. Confirmed on the board: with the allowance held by 45 undeliverable URBs, BULK and CONTROL now both answer -ENOMEM where CONTROL answered 18. The exits below the charge become one exit, for the reason the bulk path's did. internals.md claimed the charge happened where Linux charges it, which is now true rather than merely written. REAPURB takes one fd-table window instead of two. A pass called fd_snapshot for the readiness pipe and then usbdev_acquire, which proved the generation against a snapshot of its own, so a sibling's close and reopen in between satisfied the second and not the first: the pass ran on the new description's side-table entry while settling readiness on -- and, blocking, parking on -- the previous descriptor's pipe. It now pins the descriptor and takes the entry out of one window through host_fd_ref_open_entry, the shape sys_fstatfs uses, and hands that snapshot to the lookup, which no longer takes one of its own; the pin also replaces the dup the wait used to need. ELFUSE_USBDEV_REAP_DELAY_US widens the window as ELFUSE_FD_IDENTITY_WINDOW_US widens fstatfs's, and tests/test-usbdev-ioctl.c swaps the description inside it: -EBADF now, where the two-window form answered -EAGAIN from the wrong entry. Two gaps are recorded rather than closed, each a printed XFAIL carrying both values. The per-endpoint abort shutter is an invariant of the paths that abort deliberately and not of the slot: DISCARDURB and every wholesale kill raise it, CLEAR_HALT and RESETEP do not, because ClearPipeStallBothEnds aborts the pipe as a side effect and IOKit exposes no variant that does not, so a queued follower can be started behind a stall clear's abort. Raising it there needs the count, the release and the kick DISCARDURB spells out, so the comment claiming it slot-wide is narrowed to what holds instead. And check_ctrlrecip lets a printer's GET_DEVICE_ID through untouched -- it returns before the index &= 0xff when the request type is 0xa1, the request is 0 and usb_find_alt_setting(actconfig, index >> 8, index & 0xff) has class USB_CLASS_PRINTER -- where both control paths here read wIndex & 0xff as an interface number for every non-vendor interface recipient. On a printer that implicitly claims the alt setting's number instead of the interface's; demonstrating it needs a printer attached, so it is a deviations row. The connected() gate stops being prose and becomes a table. What every usbdevfs ioctl answers on a device that has gone was written into comments, wrongly twice: a count of four ops that do not ask IOKit, against a code path where the number was larger and the members were not the ones named. tests/usbdev-ioctl-departed.tbl records it as data instead -- one row per request, the Linux answer read once out of devio.c with the line it came from, and a note saying why -- and scripts/gen-usbdev-ioctl-departed.py reads the surface out of usbdev_ioctl's own dispatch and refuses to emit unless the two agree. An ioctl added to the layer now fails make check until somebody records what Linux answers for it. The comments that carried the enumeration point at the table and stop counting. Two answers the table exposed are wrong and are fixed here. An interface an fd already holds short-circuits inside usbdev_claim_locked before anything asks the device, which is right for the implicit claim a transfer takes -- the transfer behind it asks a moment later -- and wrong for CLAIMINTERFACE and RELEASEINTERFACE, where the claim is the whole op: an fd that took its claim before the device left and armed no disconnect watch got 0 from both, with the fd unstamped and poll silent, where Linux answers -ENODEV from usbdev_do_ioctl's gate. usbdev_claim_ioctl and usbdev_release_ioctl put the question on that one arm, leaving every implicit claim and the teardown path untouched. And DISCARDURB scanned the pending list, which is this layer's bookkeeping and stays true after the device leaves, so an unplug reached libusb_cancel_transfer's caller as LIBUSB_ERROR_NOT_FOUND rather than NO_DEVICE; the arm that finds no record now asks, and the arm that finds one still reaches AbortPipe and pays nothing. One answer the table could not expose is fixed here too, and the table is why it stayed hidden. usbdev_ioctl's default arm returned -ENOTTY before anything asked IOKit, so a cmd this layer does not serve answered -ENOTTY and stamped nothing on a departed device: measured on a terminated loopback device, HUB_PORTINFO, ALLOC_STREAMS and WAIT_FOR_RESUME were all -1/ENOTTY with revents 0x0 on the asking fd and on a peer. Linux runs connected() ahead of usbdev_do_ioctl's switch and -ENOTTY is that switch's default, so all three are -ENODEV with POLLERR|POLLHUP there. libusb reads the difference: ENOTTY on ALLOC_STREAMS is NOT_SUPPORTED and ENODEV is NO_DEVICE, so an unplug read as "this kernel has no streams" and never expired. The arm asks first and answers -ENOTTY once the enumeration has run. The join could not reach that arm, which is the same defect one level up. It was built from the case labels in usbdev_ioctl, so no row could name the arm that catches what no label matches, while the comment above the dispatch asserted the table said what every arm answers. A row may now name a USBDEVFS_ request this file defines and dispatches nowhere -- those three -- and the generator refuses to emit while a default arm exists with no row driving it. Deleting the ask from the arm fails the lane; making the arm answer -ENODEV always fails it too, because -ENOTTY on a device that is there is asserted alongside. Three divergences the table exposed are kept, and carry both values as XFAIL rows rather than a sentence: claimintf's interface-number bound and proc_do_submiturb's argument gate stay ahead of the device question, because both are decided against the request alone and tests/test-usbdev-ioctl.c pins that order against a node with no device behind it, and the two reaps decide on this fd's own disconnect stamp where reap_as decides on connected(ps). Two comments called a deviation "a row in the deviations table" when it is prose in the bullet list above that table. They name where it is.
The async URB engine landed with no in-tree lane over the wire half of it: ELFUSE_USB_FIXTURE's devices have no IOKit service behind them and stop at SUBMITURB's argument gate, and IOKit publishes no loopback device to borrow. So ELFUSE_USB_FIXTURE grows a mode that is one. The seam is the two IOKit COM vtables and nothing above them: every wire call goes through IOUSBDeviceInterface650 ** or IOUSBInterfaceInterface800 ** as (*h)->Method(h, ...), so src/syscall/usbdev-fixture.c hands back an object whose first member is a vtable of the same shape, and the URB records, the per-endpoint FIFO, usbdev_async_cb, the SHORT_NOT_OK and ZERO_PACKET predicates, the readiness and disconnect maps, REAPURB, the drain and all of poll.c's usbdev paths stay the code that runs against a board. Completions arrive from a one-shot CFRunLoopTimer on the event thread, where IODispatchCalloutFromCFMessage would have delivered them, and the aborts are asynchronous the way IOKit's are. What the fixture does is data rather than a flag: ELFUSE_USB_LOOPBACK scripts a per-endpoint sequence of outcomes, each naming the IOReturn it stands for -- ok and ok(n) kIOReturnSuccess, short(n) kIOReturnUnderrun 0xe00002e7, stall kIOUSBPipeStalled 0xe000404f, timeout kIOUSBTransactionTimeout 0xe0004051, nodev kIOReturnNoDevice 0xe00002c0, err and refuse any code verbatim (refuse from the submit call itself), plus delay(ms), terminate, zlpfail, never (accepted and not completed until something aborts the pipe, which is what every kill does) and wedge(ms) (never, plus an abort that lands ms later, default 2500, which is the only step that outlives the engine's 2s drain deadline and so the only one that reaches the orphaning path) -- and a guest rewrites the script, reads back a log of what crossed the seam and terminates the device through vendor control requests on the fixture device. That log is what puts ZERO_PACKET's trailing packet on an observable wire instead of inferring it from the predicate. Off, the seam is five if (u->fake) branches, one has-device probe and one bind call, behind a mode resolved once per process, and the flag is set for one modeled location only, so every other device takes the path it took before, the existing fixture modes included: their descriptor blobs come out byte-identical. The fixture device carries no synthetic io_service_t: u->service stays IO_OBJECT_NULL and the flag says so, which keeps every IOObjectRelease and usbdev_arm_disconnect_watch's NULL-service guard correct by construction. A vtable slot usbdev.c calls and the fixture leaves NULL would be a null call rather than a compile error, so that explicit list is checked once when the fixture stands up. test-usbdev-ioctl-loopback re-runs the whole fd-contract lane with the loopback device present and answers 130 passed, 0 failed. That is five more than ELFUSE_USB_FIXTURE=1's 125, and the five are the memory allowance: with no claimable device the allowance cannot be reached at all, so those assertions skip there. Every assertion both runs make answers the same in both. None of it ships. src/syscall/usbdev-fixture.c is a translation unit under src/ that exists only so a test can run, and this tree has no precedent for one. The env-var fault hooks that usb-sysfs.c, usbdev.c, fs-stat.c and fs.c carry are precedented, but each is a handful of lines inside a file that is there for production reasons, and the synthetic USB tree in usb-sysfs.c earns its place the same way: it lets lsusb work on a machine with no devices, so a non-testing user gets something from it. A device that echoes back what was written to it does nothing for anyone outside an assertion. So the seam has two implementations and the build picks one: src/syscall/usbdev-fixture-stub.c in every build, answering false and -ENODEV and modeling nothing, and usbdev-fixture.c in its place only when USB_LOOPBACK_FIXTURE=1 asks for it. Listing both would be a duplicate-symbol link error, which is the property that keeps a default build from quietly acquiring the model. Not one call site is conditionally compiled, so usbdev.c is one program in both builds and the fixture cannot drift into code a default build never sees; what a default build carries of it is the stub's own text, 44 bytes across seven functions against the model's 11032 (size -m on the two objects), and nm over a default build/elfuse finds the seven entry points and, of the model, nothing else. The device's descriptor half stays where it is. usb-sysfs.c gains one more entry in the canned spec table it already carries for ELFUSE_USB_FIXTURE, and the endpoint addresses move to runtime/usb-fixture.h so the blob emitted there and the pipe properties the model reports cannot drift. That is the precedented shape: a few lines inside a file the product needs anyway, and in a build with no model behind them they stand up one more service-less device, which is what every other mode's devices already are. The two loopback lanes need a binary that has the model, and make check builds one for them: mk/tests.mk re-enters make with USB_LOOPBACK_FIXTURE=1 and ELFUSE_BIN pointing at build/elfuse-loopback, which shares every object but the fixture's with the outer build and leaves build/elfuse without it. Asking through the variable rather than through a second link line here is what keeps what the lanes run the same thing a reader gets from make USB_LOOPBACK_FIXTURE=1. make lint reads ALL_SRCS rather than SRCS, so turning the model out of the build does not also turn off the checking of it, and cppcheck walks the tracked sources, so it sees both either way. tests/test-usbdev-urb-loopback.c makes 151 assertions over submit, completion, the queue, discard, poll, epoll, both reap modes, the disconnect drain, ZERO_PACKET, the fixture's own control plane, the 2s drain deadline and the two ioctls that refuse on it, a queued follower that starts after the device is gone, select's per-set count, and a watch armed after the terminate had already fired, the reap invariant CAP_REAP_AFTER_DISCONNECT states under all three reap orderings, and a synchronous op meeting a device that has gone. Four more scenarios reach what the engine only does under load or after something has gone wrong: a backlog of 66001 completions, larger than the pipe has room for bytes, built without the event thread at all -- a leader that never completes, a queue of refused followers behind it, and one DISCARDURB, so every one of them is on the completed list before the first reap; the process-wide byte budget asked, with a 9 MiB probe that fits alone and not beside 8 MiB, whether an orphaned record still holds its charge; an ep0 leader and follower carried through a drain that misses its deadline on another endpoint; and the disconnect drain run twice on one slot, across a close and a reopen. The disconnect scenario itself now leaves one completion unreaped at the unplug, which is the pass the drain decision has to survive. Fifteen deliberate breaks of the engine were each applied, rebuilt and run against it. A completion callback that leaves actual_length at zero fails 7: the OUT's count, the IN round trip, the short count, SHORT_NOT_OK's -EREMOTEIO, the over-report clamp, the interrupt sibling and the ZERO_PACKET OUT. An urb_complete_locked that always writes status 0 fails 13: SHORT_NOT_OK, -EPIPE, -ETIMEDOUT, -EPROTO, the refused follower, all three discard rows, the failed terminating packet, the drain's -ENOENT, the stall that proves a refused oversize script left the loaded one in place, and the two rows of the follower that starts into a gone device. A usbdev_ep_may_start that ignores the in-flight URB fails 20. Fourteen are the one-in-flight-per-endpoint count the fixture logs, the follower's start time, both queued-discard rows, the refused follower whose script step the leader eats when the two start together, the follower IOKit refuses with NoDevice, which never gets to be a follower at all, the six rows of the backlog that has no queue left to build out of, and both slot-reuse drains. The other six are in the reap path: the oldest-first order two queued URBs come back in, the reap that owes the disconnect kill and must answer EAGAIN rather than ENODEV, the URB the aborts recovered handed back once it lands, and all three CAP_REAP_AFTER_DISCONNECT orderings. A DISCARDURB that returns 0 without doing anything fails 5 and then runs the lane into its per-stage alarm. A reap that skips the disconnect drain fails 5 and then runs the lane into its alarm: the two URBs never handed back, both slot-reuse drains, the invariant scenario's non-blocking ordering, and then a blocking reap in that same scenario that waits for aborts nobody issued. A readiness map set unconditionally fails 2: both rows that ask what a disconnect with nothing left to reap reports get 0x11c where POLLERR|POLLHUP alone is due, because the disconnect's own wake byte keeps the pipe readable for good and the map then answers reapable over it. A ZERO_PACKET predicate wired to false fails 2, the packet the fixture never sees and the failed-packet status. SHORT_NOT_OK ignored fails 1; the transferred-count clamp removed fails 1 (actual_length 999 for a 32-byte buffer); and a refused start routed through the syscall map fails 1. That last one has to be a queued follower: a leader's refusal is a syscall return value, and kIOReturnNotOpen is -ENODEV in both maps, so only a follower reaches the one row where they differ -- kIOReturnAborted, -ECONNRESET in urb->status against the -EINTR the syscall map answers and the kernel never writes into a URB. The five engine answers the new scenarios exist for were broken the same way. A readiness pipe that writes one byte per completion instead of holding a level fails 3: poll, epoll and select each block their full 1000 ms while the REAPURBNDELAY issued straight after them returns the completion none of the three could see. An orphaned record whose charge is handed back at the drain deadline rather than with its buffer fails 1, the 9 MiB probe that must not fit while 8 MiB is still IOKit's. A timeout path that clears draining without restarting the FIFOs fails 1, the ep0 follower left queued with nothing in flight. A disc_drained that is not reset with the rest of the per-open async state fails 8: the second open on the same slot loses the URB only its own drain could return; the invariant scenario loses its non-blocking ordering and its poll-then-block one, the blocking one alone surviving; the non-blocking ceiling loses all three of its rows, the pass that owes the kill, the same pass half a second in, and the deadline rather than the wire deciding; and the disconnect scenario loses both of its, every in-flight URB handed back and the errno usb_kill_urb leaves on each. The two reset sites are deliberately redundant, so removing either alone leaves the lane green and removing both is what fails it. And a drain decided on a reap pass that also has a completion to hand back fails 7: that pass returns before the kill it just claimed can run, so both slot-reuse drains, both rows of the REAPURBNDELAY that owes the kill, the invariant scenario's non-blocking ordering and both disconnect rows go with it. The reap invariant scenario asks the one question the drain exists to answer -- whether any reap can say -ENODEV while a URB is still out -- under all three orderings a guest can produce, and the ordering that matters is the libusb one: poll non-blocking, then block. Restoring the single flag that both issued the aborts and reported the device drained fails 3, and the blocking reap's row names the measurement, ENODEV in 0 ms with 2 of 3 URBs back. Its other half asks whether a reap can go on NOT saying -ENODEV once no URB can come back, which is the same invariant read the other way: ep81:wedge(4000) is a wire that answers its abort well past the drain deadline, so what decides is the deadline, and a REAPURBNDELAY loop over it must reach -ENODEV at the deadline and not at the wire. Removing the non-blocking ceiling fails it with 940207 EAGAIN and no ENODEV in 3500 ms; orphaning as soon as the aborts are out rather than a deadline later fails it the other way, ENODEV at 532 ms with the URB still recoverable. Both halves also assert the elapsed time of the calls themselves, so a ceiling implemented by making the non-blocking arm wait fails on 1203 ms rather than passing quietly. The wedge in the neighboring scenario shrinks to 1200 ms for the same reason: its claim is that the URB comes back on its own timetable rather than being given up on, and past the deadline that is no longer what decides. The old timing still passes -- no probe falls in the gap -- but it passed by accident. Three more scenarios cover the engine's interface queries against a device that has gone. All need a wire that is gone over an fd nothing has stamped, which the lane already reaches by opening a fresh fd on the terminated device. The first walks GETDRIVER, DISCONNECT_CLAIM and both USBDEVFS_IOCTL arms over one such fd each, asserting -ENODEV and the stamp behind it; before the engine fix they answered -ENODATA and three times -EINVAL with poll revents 0x0. It asks about an interface number no fd holds, which is what reaches the enumeration. The second walks the same four over the interface the main fd holds, which is the branch that answers out of the claim: before the engine fix GETDRIVER answered 0 with driver usbfs, both USBDEVFS_IOCTL arms answered -EBUSY and DISCONNECT_CLAIM answered 0, taking a claim on a device that had gone, all four with poll revents 0x0. Which branch a number takes is the whole difference between the two, so they are written as one loop each rather than merged. The third is the pipe-map race, and the fixture grows the only step that reaches it: a terminate with wIndex bit 1 tears the claimed interface's pipes down and changes nothing else, so GetNumEndpoints still sizes the map and GetPipeProperties answers NoDevice while it is filled in. Before the fix the claim answered 0 and the SUBMITURB behind it -ENOENT. The seam carries one rule now, written where CreateInterfaceIterator is: an entry that stands for a request on the wire, or for a question put to the device's user client, answers NoDevice once the device has terminated, because that is what IOKit answers; an entry outside the rule is named there with its reason. CreateInterfaceIterator was tied first and was the only one, which left the model answering success for calls a departed device would refuse, so a regression above it could land green. The interface open, ClearPipeStallBothEnds and SetAlternateInterface are tied with it; SetConfiguration follows the same rule. Measured with the engine gate below reverted and these untied, CLAIMINTERFACE, CLEAR_HALT, RESETEP and SETINTERFACE on a departed device all answer 0; with the ties and the same reverted gate, all four answer -ENODEV and only the missing stamp gives the regression away. Outside the rule, by name: GetNumEndpoints and GetPipeProperties are served from descriptor state the interface user client already holds, and what stops GetPipeProperties is the pipes being torn down, its own flag; AbortPipe and USBDeviceAbortPipeZero have to land after the device leaves, because that is what the disconnect drain issues; the two closes have to succeed, or teardown leaks the handle it is releasing; the two async event sources are local plumbing over the mach port. USBDeviceOpen is the one entry the rule covers and the code does not, and not because IOKit answers differently: usbdev_lazy_device_open tolerates the refusal but still translates it, so a NoDevice there would stamp the fd from inside the same call that arms the terminate watch, and the late-delivery arm in usbdev_fixture_watch would have nothing left to be observed by. Which of the two should originate is a question about that arm, not about this seam, and it is not settled here. The engine's claim path is what the interface open sits behind. It asked usbdev_ensure_dev_plugin, which hands back a cached handle and puts no question to IOKit; before this commit the only arm below it was the registry one, whose usbdev_iface_service runs the enumeration anyway, so the device was asked either way. The fixture arm collapses the service lookup, the plugin and USBInterfaceOpen into one seam call and asks nothing, so the claim went through on a device that had gone, and so did every op that claims implicitly through usbdev_pipe_for_ep: measured, CLAIMINTERFACE, CLEAR_HALT, RESETEP and SETINTERFACE each answered 0 with poll revents 0x0 on a fresh fd of a terminated device, where Linux's connected() gate makes all four -ENODEV. The question moves ahead of the branch, where it covers both arms, and the invariant is written there: a claim is granted only against a device the call has just reached. Two scenarios assert that. One walks CLAIMINTERFACE, CLEAR_HALT, RESETEP, SETINTERFACE and RESET over one fresh fd each, asserting -ENODEV and the stamp. The other asserts the ordering the engine commit repaired, which needs an interface number out of range to show: on the same departed device GETDRIVER ifnum 200 was -ENODEV with revents 0x18 while DISCONNECT_CLAIM ifnum 200, USBDEVFS_IOCTL ifno 200 and ifno -1 were -EINVAL with revents 0x0. An in-range number reaches the enumeration and is answered correctly either way, which is why the two scenarios above this one pass on the unfixed engine and this one does not. The watch-after-terminate scenario is restaged for the same reason. It used to reach the arming by claiming an interface of the departed device and submitting a bulk URB with an unmapped buffer, and that claim now answers -ENODEV and stamps the fd, which would have left the assertion passing on a wake it did not test. It submits a vendor request on the default control pipe instead, whose setup packet is the last eight bytes of a mapped page and whose data begins on an unmapped one: the one URB shape that skips both the endpoint lookup and check_ctrlrecip's implicit claim, so it reaches the ep0 event source, arms the watch, and fails on -EFAULT. That -EFAULT is what keeps the assertion honest -- an fd already carrying the mark would have been answered -ENODEV by the gate before the handler ran -- so the wake has no source but the watch. The synchronous-op scenario opens two fds on the departed device, neither of which ever submits, so no terminate watch is armed on either and one synchronous CONTROL is the only possible source of the news: routing IOKit statuses back through the bare translation, so that ops report a disconnect without originating one, fails 3 -- the asking fd's poll and its later ioctl, and the peer that was never told. What that lane cannot answer stays on the board, and docs/testing.md says so: real timing, NAKs, maxpacket segmentation, DMA alignment and throughput; that IOKit really delivers completions on the runloop, and the IODispatchCalloutFromCFMessage opacity behind the URB record's atomic owner, which a timer callout fully visible to ThreadSanitizer does not reproduce; exclusive-access arbitration and kernel-driver binding, so GETDRIVER and DISCONNECT_CLAIM against a real driver; a real SET_CONFIGURATION, altsetting renumbering and port RESET; that a device actually receives the zero-length packet, as opposed to elfuse emitting it under the right predicate; and a physical unplug mid-transfer. Two of the engine's own answers are board-only for the same reason, and breaking either of them leaves the lane green: the ZERO_PACKET write's dropped async_lock and its bounded timeout only matter against an endpoint that NAKs, and the fixture answers a zero-length write immediately and ignores both timeout arguments. So is the bystander window ep_aborting shuts, because the fixture retargets an aborted transfer's timer under its own lock and can never start a follower into an abort that is still running; the 200-round board round remains the evidence for it. The XFAILs are unchanged. Verified against the real ESP32-S3 at 303a:1001, one driver run three times: GET_CAPABILITIES is 0x11; a control URB reaps 18 bytes through poll+reap; a bulk IN discard reaps -ENOENT in 6700 of 6700 discards and the REAPURBNDELAY issued straight afterwards finds the URB 6700 of 6700 times, in each of the three runs; 300 URBs queue on one fd and all 300 reap back; 200 rounds of discarding one of two queued ep0 URBs produce 0 bystander -ECONNRESET; RESET returns all 3 queued URBs; a short IN reports 18, not 255, with nothing written past it, and -EREMOTEIO under SHORT_NOT_OK. ThreadSanitizer and AddressSanitizer/UBSan are clean across the same runs, including the driver whose first run reported the record-owner race. The four board drivers were re-run against the board with the seam in the tree and against a build with it excised, the seam being off in the first: every line matches but the wall-clock measurements, which jitter from run to run rather than settling. In the pair of runs quoted here the three timeout figures read 5005 against 5084 ms, 5051 against 5080 ms and 6168 against 6112 ms, and one concurrency latency read 0 against 1 ms. The lane grows the three scenarios the engine's cross-fd and non-blocking answers need, and each of them fails against the shape the engine had. t_follower_device_gone keeps its own fd for the stamp and now also asserts the main one, which submitted nothing on its behalf: a disconnect that one fd provokes has to reach the other fd open on the node, 0x0000 and 0 without the walk and POLLERR|POLLHUP and -ENODEV with it. t_control_charge fills the byte budget with URBs that cannot complete -- only an endpoint's leader reaches IOKit, so a single never step parks the whole allowance behind it -- and asserts that a synchronous CONTROL of 4096 is refused beside a 4 KiB URB and goes through again once the URBs come back. t_reap_ndelay_never_waits times the REAPURBNDELAY that owes the post-disconnect kill against a wedged endpoint, where the elapsed time is the whole measurement: 0 ms, and 2010 ms with the engine's split reverted. wedge is what makes that measurable, the same property that gives the drain deadline a lane. Two existing scenarios move with them. Both provoke a disconnect on a second fd, and both restored the main fd afterwards on the premise that the stamp was private to the fd that noticed it. It is not, so each closes the main fd and opens a fresh one instead -- the fixture's device is still there, nothing was terminated -- which is also what leaves the terminate scenario at the end a device that has not already been marked gone. One more scenario runs after the terminate. t_setconfig_on_a_departed_device drops the node's last interface claim -- SETCONFIGURATION is refused outright while any is held, and the main fd is stamped by then so its RELEASEINTERFACE would release nothing -- and puts SETCONFIGURATION to a fresh fd on the departed device. Before the engine translated that enumeration's status it answered 0 and left the fd unstamped; it answers -ENODEV and raises POLLERR|POLLHUP now. And the device is made to leave. tests/test-usbdev-ioctl-departed.c terminates the loopback device once and then drives every usbdevfs ioctl the layer implements against it, asserting four columns per request: the ioctl(2) return, the errno behind it, the poll revents on the fd that asked, and the revents on a second fd open on the same node, which is what says the disconnect was recorded against the device and not against one caller. The rows come from tests/usbdev-ioctl-departed.tbl through the generator, so the lane covers the surface by construction and a request with no driver behind it fails to link rather than going undriven. Three runs, because one process can hold only one departed device: the first correct -ENODEV stamps every fd open on the node, so the two rows that need a claim taken before the device left -- the ones that found CLAIMINTERFACE and RELEASEINTERFACE answering 0 -- get a process each. A synchronous-only fd is what makes those reachable at all: the terminate watch is armed by the async paths, so an fd that only ever claimed and transferred is never told the device went, which is the state Linux's connected() has no equivalent of. The lane brings the event thread up on a throwaway fd and closes it before the terminate, so no watch is left to stamp anything early. The table the lane produces, run against this branch: twenty-eight rows over twenty-three requests, of which nineteen rows, covering seventeen of those requests, answer -ENODEV with both fds stamped, as Linux does. Six of the remaining nine carry both values as XFAIL rows -- GET_CAPABILITIES, GET_SPEED, CONNECTINFO and DISCSIGNAL answer from the open-time model, REAPURB waits where reap_as returns and REAPURBNDELAY answers -EAGAIN where proc_reapurbnonblock answers -ENODEV -- and three record argument gates this layer keeps ahead of the device question. An XFAIL that starts answering what Linux answers fails the lane too, so a gap cannot close unnoticed either. Every row asks on an fd that has not been told the device left, which is what makes the per-request answers worth measuring and what leaves the other half of the disconnect contract unmeasured: what an fd that HAS been told answers. internals.md states that half as a universal over the usbdevfs surface, so the fresh phase ends by driving all twenty-eight requests on a marked fd and asserting -ENODEV with POLLERR|POLLHUP on each. Letting one request past the gate leaves every row passing and fails only that sweep. It ends by driving where that universal stops, too, because a universal narrowed and not measured is the same sentence with fewer words: the ten requests do_vfs_ioctl answers before f_op->unlocked_ioctl, on the marked fd and on a fresh one, each -ENOTTY with the fd's own revents unchanged, and FIONBIO, which io.c answers ahead of this layer, still 0. Removing the engine's carve-out fails both sweeps, 0 of 10 on each fd; sparing only the default arm's device question, which is where the divergence was first read, still fails the marked fd's, 0 of 10 -- the gate ahead of that arm reaches the same requests and had to be answered first.
jotpalch
force-pushed
the
pr-d-usb-async
branch
from
September 13, 2026 10:03
ce9ab8b to
2ee3279
Compare
Contributor
Author
|
The red check here is |
jserv
reviewed
Sep 13, 2026
| # the loopback device model when USB_LOOPBACK_FIXTURE asks for it. Listing both | ||
| # would be a duplicate-symbol link error, which is the property that keeps a | ||
| # default build from quietly acquiring the model. | ||
| ifeq ($(USB_LOOPBACK_FIXTURE),1) |
Contributor
There was a problem hiding this comment.
BUILD_FLAVOR is $(strip $(CFLAGS)) (mk/common.mk) and the fixture switch changes SRCS, not CFLAGS, so flipping it wipes nothing. Run the command mk/config.mk documents, make USB_LOOPBACK_FIXTURE=1, and the model is linked into build/elfuse; a later plain make finds build/elfuse newer than every object in the rebuilt OBJS and relinks nothing, so the default binary keeps the model until someone runs make clean. The make check path is safe because elfuse-loopback overrides ELFUSE_BIN, which is what the nm evidence measured. Either fold the fixture selection into the flavor stamp so the tree wipes on a flip, or make the documented manual command target $(ELFUSE_LOOPBACK_BIN) too.
jserv
reviewed
Sep 13, 2026
| * record whether their pipe actually fired BEFORE the writeback | ||
| * below overwrites revents; the disarm pass keys off that. | ||
| */ | ||
| usb_pipe_fired[i] = !usb_disarmed[i] && (before & POLLIN) != 0; |
Contributor
There was a problem hiding this comment.
usb_pipe_fired only recognises POLLIN, but a usbfs entry's readiness pipe reports POLLHUP once teardown closes its write end, which a concurrent close() of a polled fd does while host_refs still pins the read end. POLLHUP maps to no guest event (the fd is gone, so fd_snapshot fails and the maps are clear), ret drops back to 0, the entry is not disarmed because its pipe never "fired", and the goto ppoll_retry below re-polls the same standing POLLHUP immediately: an indefinite ppoll burns a core until a signal arrives. Treat POLLHUP|POLLERR|POLLNVAL as an invisible fire alongside POLLIN so the disarm pass withdraws the interest.
jserv
reviewed
Sep 13, 2026
| * EOF/errors on the pipe only happen while the fd is being torn down -- | ||
| * report the hangup pair Linux uses for a removed device. | ||
| */ | ||
| if (kev->filter == EVFILT_READ && (reg->events & LINUX_EPOLLOUT) && |
Contributor
There was a problem hiding this comment.
The writable interest here is reg->events & LINUX_EPOLLOUT, while the poll and select half answers LINUX_POLLOUT | LINUX_POLLWRNORM and masks by the guest's events (usbdev_poll_guest_revents). A registration that asks for EPOLLWRNORM alone therefore never sees a completion here and would see it through poll, where Linux's usbdev_poll returns both bits and ep_send_events masks by what was requested. epoll_mute_usb_wakes's wantout and the stamp path read the same single-bit mask. Use EPOLLOUT|EPOLLWRNORM as the writable mask in all three.
jserv
reviewed
Sep 13, 2026
| return; | ||
| usbdev_t *u = &usbdev_fds[idx]; | ||
| pthread_mutex_lock(&usbdev_table_lock); | ||
| if (u->used && (u->generation & USBDEV_WATCH_GEN_MASK) == gen) { |
Contributor
There was a problem hiding this comment.
The peer walk applies o->dead to every slot it stamps (usbdev_mark_key_disconnected_tlocked); this test does not. A terminate notification IOKit had already dispatched can run after usbdev_fd_cleanup has torn the slot down and cleared the maps but before usbdev_unref clears used and generation, so used is still true and the packed generation still matches. It then stamps discmap for a guest fd number a sibling's open() may already have taken, leaving that fd permanently reported POLLERR|POLLHUP and answering ENODEV. Add && !u->dead.
jserv
reviewed
Sep 13, 2026
| *tail = rec; | ||
| } | ||
|
|
||
| static void urb_pending_unlink(usbdev_t *u, usbdev_urb_t *rec) |
Contributor
There was a problem hiding this comment.
urb_pending_unlink and usbdev_kick_ep_locked both walk pending_head from the head, and the list is fd-wide rather than per endpoint. With the 256-record cap gone, a ring queued on one endpoint sits in front of every other endpoint's records, so each completion on a second endpoint walks the whole first queue twice while holding async_lock on the one event thread that carries every usbdevfs fd's completions. A guest can fill the 16 MB budget with about 87000 zero-length records and drive a second endpoint against them. A per-endpoint intrusive FIFO plus an in-flight pointer makes append, unlink and restart O(1); worth doing in this change, since removing the cap is what exposes it.
jserv
reviewed
Sep 13, 2026
| u->nurbs--; | ||
| u->inflight_bytes -= r->charge; | ||
| r->orphaned = true; | ||
| for (int fi_i = 0; r->intf && fi_i < USBDEV_MAX_IFACES; fi_i++) { |
Contributor
There was a problem hiding this comment.
The orphan pin is guarded by r->intf, so an ep0 record (which carries intf == NULL) leaves this loop having incremented nothing. Nothing else records that an ep0 request is still outstanding: the record is already unlinked from pending, so a later usbdev_kill_urbs_locked(u, NULL) in usbdev_teardown_locked finds no in-flight URB, returns drained, and CFReleases dev_src and Releases u->dev while DeviceRequestAsyncTO still owns the record. Reachable through usbdev_do_reap's overdue arm, which orphans on one call and leaves the close to a later one. Give the device the same pin usbdev_iface_t.orphans gives an interface, and let usbdev_teardown_locked read it the way usbdev_release_locked reads the interface count.
jserv
reviewed
Sep 13, 2026
| * the entry lock this thread holds. | ||
| */ | ||
| if (pipe == 0) | ||
| (void) (*u->dev)->USBDeviceAbortPipeZero(u->dev); |
Contributor
There was a problem hiding this comment.
Both aborts here discard their IOReturn. usbdev_arm_disconnect_watch calls itself best effort and names "kIOReturnNoDevice detection on ops" as the fallback, and this is an op whose return says exactly that. With no interest notification armed, a DISCARDURB against a departed device aborts nothing, waits out the 2 s deadline, logs and returns 0 with the record still URB_INFLIGHT in pending and the fd unstamped; a following REAPURB then blocks forever, because the drain arms all require disc. Route both through usbdev_ioret_op (or test usbdev_ioret_device_gone and stamp) instead of casting to void.
jserv
reviewed
Sep 13, 2026
| bool overdue = !rec && disc && pending && !block && u->disc_drained && | ||
| usbdev_now_ms() >= u->disc_orphan_at_ms; | ||
| if (overdue) { | ||
| usbdev_orphan_stalled_locked(u, NULL); |
Contributor
There was a problem hiding this comment.
usbdev_orphan_stalled_locked unlinks the in-flight record and marks it orphaned without appending it to completed_head, and the late callback frees it through the rec->orphaned arm rather than completing it, so the URB pointer is never handed back and the same pass then answers -ENODEV through the disc && !pending test below. That is the one path where CAP_REAP_AFTER_DISCONNECT, which GET_CAPABILITIES raises, does not hold. The 2 s bound is a deliberate trade, so this belongs in the XFAIL list beside discard-latency rather than being left to the capability bit, or the orphan should complete with the status usb_kill_urb leaves.
jserv
reviewed
Sep 13, 2026
| surface it describes. | ||
|
|
||
| USB-layer coverage is split by what it needs. `test-uevent-socket` needs no | ||
| hardware and runs in the matrix like any other unit test, and so do the two |
Contributor
There was a problem hiding this comment.
Neither usbdevfs lane is in the matrix: grep usbdev tests/test-matrix.sh is empty on both sides of this change, and test-usbdev-urb-host is a native macOS binary, so it could not run there in any case (the matrix runs aarch64 guest binaries under elfuse and qemu). test-uevent-socket, the lane this sentence compares them to, is registered. Say they run from make check instead, or register them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: piece D of the USB layer described in #319, on top of #366 (piece C, the usbdevfs descriptor and its synchronous ioctls) and through it on #334 and #351. #366 answered
ENOTTYforSUBMITURB/REAPURB/DISCARDURBand reportedGET_CAPABILITIESas 0; this piece implements them and raises the two bits it can honour. The tty aliasing piece follows and is not here. Two commits: the engine, then the loopback fixture that tests it.Summary
SUBMITURB,REAPURB,REAPURBNDELAY,DISCARDURB,poll/select/epollon a usbdevfs fd, andCAP_REAP_AFTER_DISCONNECT's disconnect drain, overIOUSBDeviceInterface650/IOUSBInterfaceInterface800async completions delivered on one lazily-started CFRunLoop host thread -- libusb's own darwin model.GET_CAPABILITIESgoes from 0 to0x11.The engine is verified three ways: a native host binary over the arithmetic (781 assertions), a new IOKit loopback fixture that lets the whole engine run without a board (44 assertions, plus the entire #366 fd-contract lane re-run against a device that can actually complete a transfer), and the attached ESP32-S3 for everything the fixture cannot show. All numbers below are from runs made while writing this description.
The two commits
SUBMITURB,REAPURB,REAPURBNDELAY,DISCARDURB, the per-endpoint queue, the disconnect drain,poll/select/epoll, and the native host binary over the arithmetic. 16 files, 2968 insertions, 182 deletions. No fixture file exists in the tree at this commit, and it builds, passes every gate, and passesmake checkthere.The boundary is the shipped binary. The model does not go into one (below), so the second commit is a translation unit no default build links plus the wiring that keeps it that way, and the engine can be read without it.
What
One transfer per endpoint in flight, and what it costs. Linux
DISCARDURBcancels exactly one URB. IOKit'sAbortPipe(USBDeviceAbortPipeZerofor ep0) cancels every outstanding transfer on the pipe. There is no IOKit call with Linux's granularity, so the Linux semantics are kept by never giving IOKit more than one transfer per endpoint to abort: later submissions queue inside elfuse and are started from the completion callback.The cost is real and worth stating plainly. A guest gets queue depth but no overlap: an endpoint is idle for a full host round trip between one transfer completing and the next starting, so a bulk-streaming guest that submits a ring to keep the pipe busy gets its URBs served one at a time. The ring itself is honoured -- 300 URBs queued on one fd on the board all queued (first error 0) and all 300 reaped back -- and the loopback lane asserts both halves of the price: never more than one in flight per endpoint, and the follower started only after the leader completed. Throughput on an endpoint is therefore bounded by round-trip latency, not by IOKit's pipelining. Removing the queue would restore overlap and lose
DISCARDURB's one-URB granularity; that trade is the one design decision this piece is built around.One URB on the wire is not by itself sufficient. The abort is issued with
async_lockdropped, and in that window the target can complete normally and the completion callback can start the queued follower for the abort to hit instead. So the endpoint's FIFO stays shut while an abort for it runs, andDISCARDURBwaits (bounded at 2s) for the record to leave the pending list -- which is also what makes an immediately followingREAPURBNDELAYfind the URB the wayusb_kill_urbleaves it. On the board, across 63 runs, 12600 rounds of discarding one of two queued ep0 URBs produced 0 bystanderECONNRESET, and 1260 bulk-IN discards all reapedENOENT, with theREAPURBNDELAYstraight afterwards finding the URB every time.Argument validation follows
proc_do_submiturb, notdo_proc_bulk. Flags mask, thenUSBFS_XFER_MAX, then a NULL buffer with a positive length, then the endpoint (devio.c:1631-1658). So an absent endpoint outranks an unknown transfer type, the ISO refusal and the control arm's eight-byte minimum, and a 2 GB URB on the default control pipe isEINVALrather than an allocation. That is deliberately not the order #366 pins for the synchronousBULKioctl, which resolves the endpoint first; the two kernel paths genuinely differ.Resource limits are Linux's, and two of elfuse's own are gone. The 16 MB budget is the one the synchronous transfers already charge, because
usbfs_memory_usageis a single kernel-wide static (devio.c:134, 143-178) and a second counter would let this path queue a budget behind whatever aBULKioctl holds. Records are charged alongside their buffers so zero-length URBs are bounded, and there is no URB-count cap: the old per-fd 256-record backstop refused a 257th eight-byte URB, which is exactly the ring depth libusb's async API builds for bulk streaming.The disconnect path.
usbdev_removerunsdestroy_all_asyncbefore it wakes the reapers. IOKit completes nothing of its own when a device terminates, so the first reap that finds the completion list empty on a disconnected fd issues the kill itself. The disconnect-watch refcon packs the slot index in a field sized fromUSBDEV_MAX_FDSbehind a static assertion rather than a hand-written four bits: 32 slots through four bits decoded slot 16+k as slot k, which the host binary now pins -- reverting it fails 144 assertions, exactly 128 round trips plus 16 collisions.poll/select/epoll (644 lines added to
poll.c). The fd's completion pipe raises hostPOLLIN, remapped to guestPOLLOUT|POLLWRNORM(devio.c:2830-2843); epoll registersEVFILT_READand reportsEPOLLOUTthrough a per-registration flag. Host-side interest is armed regardless of what the guest asked for, because a disconnect must wake even a read-only wait with the unmaskablePOLLERR|POLLHUP(devio.c:2839-2842). A wake that maps to nothing guest-visible re-blocks: ppoll/pselect withdraw the woken entry's host interest (unreaped completions keep the pipe readable, so leaving it armed busy-spins at 100% CPU) and resume in bounded 200ms slices; epoll mutes the fired knote and re-enters the kevent slice loop rather than returning 0 before the timeout, which Linuxep_pollnever does.read()on a disconnected fd isENODEV(devio.c:320-322), andREAPURBreturnsEINTRwithsyscall_restart_forbid()becausereap_asdoes not restart (devio.c:2113-2114).The writability half of the remap reads the same derived capability the ioctl and write gates read rather than testing the access mode against
O_RDONLY, matchingusbdev_poll'sFMODE_WRITEgate (devio.c:2837) -- access mode 3 is notO_RDONLYand carries noFMODE_WRITEeither. This is offered as a consistency fix with no test, and deliberately not as a bug: no guest can observe the difference, because reachingPOLLOUTneeds a completion andSUBMITURBon such an fd is alreadyEPERM, so an assertion here would pass against the literal test too.Capability bits, and the evidence each is honoured
An earlier piece in this series was caught raising a bit it did not honour, so each bit is listed with what proves it.
CAP_ZERO_PACKET(0x01) -- honoured. The trailing zero-length write is emitted from the completion callback with a bounded timeout andasync_lockdropped, and a failure folds intourb->status. The loopback fixture logs what crossed the seam, so the lane asserts the trailing zero-length packet reached the wire, a short OUT gets no terminating packet, and a failed terminating packet lands in urb->status -- observed on a wire, not inferred from the predicate. Wiring the predicate to false fails 2 assertions; the untimed-under-lock version reported a failed terminating packet as success.CAP_REAP_AFTER_DISCONNECT(0x10) -- honoured. Loopback asserts CAP_REAP_AFTER_DISCONNECT hands back every in-flight URB, each one carries the errno usb_kill_urb leaves, and REAPURB is ENODEV once the drain is done; a reap that skips the drain fails 2. On the board,RESETreturned all 3 queued URBs (rc=0, 3 of 3).CAP_BULK_CONTINUATION(0x02) -- stays clear, on purpose. The flag is accepted but its error-cascade unlink is not implemented, and a guest that read the bit would rely on the cascade.NO_PACKET_SIZE_LIMandBULK_SCATTER_GATHERstay clear; neither is implemented.GET_CAPABILITIESreads0x11on the board in every run, and the loopback lane asserts it too.The loopback fixture: what it is, and what it cannot show
The engine needed a device that can complete a URB, and IOKit publishes no loopback one, so
ELFUSE_USB_FIXTURE=loopbackis one.The seam is exactly two IOKit COM vtables and nothing above them. Every wire call in
usbdev.cgoes throughIOUSBDeviceInterface650 **orIOUSBInterfaceInterface800 **as(*h)->Method(h, ...), sosrc/syscall/usbdev-fixture.chands back an object whose first member is a vtable of the same shape. Everything above stays the code that runs against a board: the URB records, the per-endpoint FIFO,usbdev_async_cb, theSHORT_NOT_OKandZERO_PACKETpredicates, the readiness and disconnect maps,REAPURB, the drain, and all 644 poll.c lines. Completions arrive from a one-shotCFRunLoopTimeron the event thread, whereIODispatchCalloutFromCFMessagewould have delivered them, and aborts are asynchronous the way IOKit's are.What the fixture does is data rather than a flag:
ELFUSE_USB_LOOPBACKscripts a per-endpoint sequence of outcomes, each naming the IOReturn it stands for --short(n)kIOReturnUnderrun0xe00002e7,stallkIOUSBPipeStalled0xe000404f,timeoutkIOUSBTransactionTimeout0xe0004051,nodevkIOReturnNoDevice0xe00002c0, plusok,ok(n),err,refuse,delay(ms),never,terminateandzlpfail-- and a guest rewrites the script, reads back a log of what crossed the seam, and terminates the device through vendor control requests.What it cannot show, stated so the coverage is not overread. Three of the engine's own answers are board-only, and breaking any of the three leaves the lane at 44 passed, 0 failed:
ZERO_PACKETwrite's droppedasync_lock-- the fixture answers a zero-length write immediately, so nothing NAKs and nothing stalls behind the lock;ep_abortingshuts -- the fixture retargets an aborted transfer's timer under its own lock, so it can never start a follower into an abort that is still running.The 200-round board driver, run 63 times for 12600 rounds with 0 bystander
ECONNRESET, remains the only evidence for (3). More broadly,docs/testing.mdrecords what stays on hardware: real timing, NAKs, maxpacket segmentation, DMA alignment and throughput; that IOKit really delivers completions on the runloop, and theIODispatchCalloutFromCFMessageopacity behind the URB record's atomic owner, which a timer callout fully visible to ThreadSanitizer does not reproduce; exclusive-access arbitration and kernel-driver binding; a realSET_CONFIGURATION, altsetting renumbering and portRESET; that a device actually receives the zero-length packet; and a physical unplug mid-transfer.None of it is in a default build.
src/syscall/usbdev-fixture.cis a translation unit undersrc/that exists only so a test can run, and this tree has no precedent for one. The env-var fault hooks thatusb-sysfs.c,usbdev.c,fs-stat.candfs.ccarry are precedented, but each is a handful of lines inside a file that is there for production reasons; and the synthetic USB tree inusb-sysfs.cearns its place the same way -- it letslsusbwork on a machine with no devices, so a non-testing user gets something from it. A device that echoes back what was written to it does nothing for anyone outside an assertion.So the seam has two implementations and the build picks one.
src/syscall/usbdev-fixture-stub.c-- 72 lines,usbdev_fixture_loopback()answering false and the rest inert -- is inSRCSin every build;usbdev-fixture.ctakes its place only whenUSB_LOOPBACK_FIXTURE=1asks for it (mk/config.mk, and theSRCSblock in the top-levelMakefile). Listing both would be a duplicate-symbol link error, which is the property that keeps a default build from quietly acquiring the model. Not one call site inusbdev.cis conditionally compiled, so it is one program in both builds and the fixture cannot drift into code a default build never sees.To get the fixture, ask for it:
make checkneeds no flag.mk/tests.mkre-enters make withUSB_LOOPBACK_FIXTURE=1 ELFUSE_BIN=build/elfuse-loopbackand the two loopback lanes run that binary, so what they run is the build a reader gets from the variable and cannot drift from it, while a plainmakestill leavesbuild/elfusewithout the model. OverridingELFUSE_BINrather thanBUILD_DIRshares every object but the fixture's, so a loopback build compiles exactly one extra file, and a command-line override reaches the sub-make throughMAKEFLAGS, so the sanitizer lanes get a loopback binary of their own flavor.make lintreads a newALL_SRCSrather thanSRCS, so gating the model out of the build does not gate it out of clang-tidy, and cppcheck walks the tracked sources, so it sees both either way.What a default build carries of it is measured rather than asserted. On a clean rebuild of the tip:
After a full
make check-- which linkselfuse-loopbackfor the lanes --nm build/elfusestill finds zerofx_*symbols.Off, the seam is five
if (u->fake)branches, one has-device probe and one bind call, behind a mode resolved once per process and set for one modeled location only. Every other device takes the path it took before -- the existing fixture modes' descriptor blobs come out byte-identical seam-in versus seam-out. The fixture device carries no syntheticio_service_t(u->servicestaysIO_OBJECT_NULLand the flag says so), which keeps everyIOObjectReleaseandusbdev_arm_disconnect_watch's NULL-service guard correct by construction. A vtable slotusbdev.ccalls that the fixture left NULL would be a null call rather than a compile error, so the list is checked once when the fixture stands up.To show the seam changes nothing on real hardware, the four board drivers were run against the board twice: once on the engine commit, where no fixture file exists at all, and once on the tip's default build, where the seam exists and the stub answers it. The two logs are byte-identical once the wall-clock figures are normalized. Those figures jitter run to run -- the three timeouts read 5019, 5046 and 6176 ms in the first against 5082, 5033 and 6146 ms in the second -- and timing figures in this description are quoted with the run they came from and compared as jitter, never as values to reproduce.
Deviations
Eight XFAIL rows, printed by
test-usbdev-ioctlwith both values so they live in the lane's own output rather than only here.The last three are inherited unchanged from #366. Separately,
docs/usage.mdnow states what a stock distributionlsusbactually does here: usbutils reaches libusb through libudev, whose monitor wantsSO_ATTACH_FILTER, elfuse's netlink layer answersENOPROTOOPT, andlibusb_initgives up with -99. That gap belongs to the netlink layer, not to this piece, but the worked example had to stop claiming otherwise.Evidence
Both commits were checked out on their own, each from
make clean, and each ran the whole set. The engine commit first, with no fixture anywhere in the tree:Then the tip, from a fully clean tree, with
build/elfuse-loopbacklinked inside the lane that needs it:The gate set, run on each commit separately: commit-log,
commentflow --check,check-format (clang-format 22.1.8),
.ci/check-cppcheck.sh, check-newline,check-security, check-matrix-lists, check-eintr-contract, check-lock-order,
check-atomics, check-ascii, check-svc-tails, check-skill-refs,
check-proof-targets, check-syscall-coverage and
make lint-- all exit 0 onboth.
The loopback run answers 108 where the
FIXTURE=1run answers 103. The five extra are exactly the in-flight transfer allowance -- with no claimable device the claim answersENODEVbefore any length is looked at, so those five skip there and print why. Every assertion both runs make answers the same in both.Negative controls, because a lane that only ever passes proves nothing:
SUBMITURBargument-order assertions intest-usbdev-ioctl, seven fail against the order the engine had first -- endpoint resolved ahead of the gate, noUSBFS_XFER_MAXbound at all.actual_lengthat zero: 7.urb_complete_lockedalways writing status 0: 10.usbdev_ep_may_startignoring the in-flight URB: 6.DISCARDURBreturning 0 without doing anything: 5, then the lane runs into its per-stage alarm. A reap that skips the disconnect drain: 2. A readiness map set unconditionally: 1 -- the disconnect that must reportPOLLERR|POLLHUPalone reports 0x11c, because the wake byte keeps the pipe readable for good.ZERO_PACKETpredicate wired to false: 2.SHORT_NOT_OKignored: 1. Transferred-count clamp removed: 1 (actual_length999 for a 32-byte buffer). A refused start routed through the syscall map: 1 -- that one has to be a queued follower, because a leader's refusal is a syscall return value andkIOReturnNotOpenisENODEVin both maps, so only a follower reaches the one row where they differ (kIOReturnAborted,ECONNRESETinurb->statusagainst theEINTRthe syscall map answers and the kernel never writes into a URB).make USB_LOOPBACK_FIXTURE=1, to confirm the lane still kills a broken engine now that the fixture is behind a variable:urb_complete_lockedalways writing status 0 gives 34 passed, 10 failed; the readiness map set unconditionally gives 43 passed, 1 failed, onrevents=0x11cwherePOLLERR|POLLHUPalone is wanted. Both exit 1.On the attached ESP32-S3 at
303a:1001, re-measured for this description against a clean rebuild of the tip: 63 consecutive runs of the review driver, and one run each of the other six.That last row is why the URB record's owning slot is
_Atomic: it is the one field the completion callback reads before it can take any lock, and ThreadSanitizer cannot see throughIODispatchCalloutFromCFMessage, so it reported the access on a cold-start run.Summary by cubic
Implements the async usbdevfs URB path —
SUBMITURB,REAPURB,REAPURBNDELAY,DISCARDURB, poll/select/epoll, and the disconnect drain — over IOKit async completions carried by one lazily-started CFRunLoop thread, so URBs now transfer instead of answeringENOTTY.GET_CAPABILITIESgoes from 0 to0x11(CAP_ZERO_PACKETandCAP_REAP_AFTER_DISCONNECThonoured;CAP_BULK_CONTINUATIONstays clear).DISCARDURBsemantics by never giving IOKit more than one transfer per endpoint; later submissions queue and start from the completion callback. The cost is real: an endpoint idles for a full host round trip between transfers, so there is no IOKit pipelining.SUBMITURBin the kernel's order (flags mask,USBFS_XFER_MAX, NULL buffer, then endpoint), deliberately differing from the syncBULKioctl's order.CONTROLpath now charges the allowance too; drops the old 256-URB-per-fd cap so libusb's bulk ring depth works.REAPURBNDELAYissues aborts and returns immediately instead of waiting out the drain;REAPURBtakes one fd-table window, closing a close/reopen race.GETDRIVER,DISCONNECT_CLAIM, theUSBDEVFS_IOCTLarms),RESET,CLAIMINTERFACE/RELEASEINTERFACE, the ioctl default arm, and the ten requestsdo_vfs_ioctlanswers beforeunlocked_ioctlall put the device question first, matching Linux'sconnected()gate.ZERO_PACKETemits the trailing zero-length write from the completion callback;SHORT_NOT_OKreports-EREMOTEIO.REAPURBreturnsEINTRwithout restart;read()on a disconnected fd isENODEV.Testing
ELFUSE_USB_FIXTURE=loopback) behind the two IOKit COM vtables, so the whole submit/complete/reap path runs with no board, including scripted wire outcomes viaELFUSE_USB_LOOPBACK. It is not in default builds:USB_LOOPBACK_FIXTURE=1swaps a 44-byte stub for the model.devio.c(tests/usbdev-ioctl-departed.tbl), generated into the test so a new ioctl without a row failsmake check.DISCARDURBwaits up to 2s whereusb_kill_urbdoes not,CLEAR_HALTcancels an in-flight URB,signr/DISCSIGNALare accepted but never delivered, isochronous URBs stayEINVAL, and short bulk OUTs reportEIO.Written for commit 2ee3279. Summary will update on new commits.