Windows: Freezing in Kernel with libusb-win32

[sonatique]

And does this mean that, even if a given device is "officially" using libusb0 (i.e. libusb0.sys, libusb0.dll reported by Device Manager), then in the end all functions of winusb_interface are backed by libusbK implementation and none of libusb0 code will be called?

I am having this understanding based on this code snippet from windows_winsub.c, funciton winusbx_init, line 2505:

// NB: The below for loop works because the sub_api value for WinUSB
// is a higher value than that of libusbK and libusb0
for (; sub_api < SUB_API_WINUSB; sub_api++) {
    libusbK_Set(sub_api, AbortPipe, true);
    libusbK_Set(sub_api, ControlTransfer, true);
    libusbK_Set(sub_api, FlushPipe, true);
    libusbK_Set(sub_api, Free, true);
    libusbK_Set(sub_api, GetAssociatedInterface, true);
    libusbK_Set(sub_api, Initialize, true);
    libusbK_Set(sub_api, ReadPipe, true);
    libusbK_Set(sub_api, ResetPipe, true);
    libusbK_Set(sub_api, SetCurrentAlternateSetting, true);
    libusbK_Set(sub_api, SetPipePolicy, true);
    libusbK_Set(sub_api, WritePipe, true);
//etc.

sub_api takes value from 0 to 1, i.e. SUB_API_LIBUBSK and SUB_API_LIBUSB0, meanig that function pointers corresponding to libusb0 are actually pointing to libusbK.

Thank you in advance for your help understanding this.
Best regards.

[mcuee]

libusb0.dll (libusb-win32 API) library is not used. libusbK.dll will talk to libusb0.sys directly through IOCTLs. libusbK.sys (KMDF based) supports all the IOCTLs from libusb0.sys (WDM based).
https://github.com/mcuee/libusbk/blob/master/libusbK/src/drv_api.h

Does the following Wiki FAQ entry answer your question?
https://github.com/libusb/libusb/wiki/FAQ#user-content-libusbwin32_libusbK_and_libusb_project_what_about_openusb

1. WinUSB.sys kernel driver from Microsoft, through WinUSB.dll from Microsoft (if libusbK.dll is not present), or through libusbK.dll from libusbK project if libusbK.dll is present.
   libusb-1.0.dll --> WinUSB.dll --> WinUSB.sys (if libusbK.dll is not present)
   libusb-1.0.dll --> libusbK.dll --> WinUSB.dll --> WinUSB.sys

Edit --> Wiki updated. For libusb 1.0.24 onwards, it is alway use the following scheme, regardless of the presence of libusbK.dll.
libusb-1.0.dll --> WinUSB.dll --> WinUSB.sys

2. libusbK.sys kernel driver from libusbK project, through libusbK.dll
   libusb-1.0.dll --> libusbK.dll --> libusbK.sys

3. libusb0.sys kernel driver from libusb-win32 project, through libusbK.dll from libusbK project
   libusb-1.0.dll --> libusbK.dll --> libusb0.sys

[sonatique]

Hello, thank you for your prompt response. Sorry that I missed the Wiki FAQ.
Yes I got my answers, thank you.

The reason behind it is that I have an issue with libusb_clear_halt: when called to a libusb0.sys device, it sometimes freeze completely, making the whole process unkillable. I am trying to understand where in the call it freezes exactly and started by looking at libusb sources.

But then I guess I shall have a look a libusbK so start with.

[mcuee]

Just wondering what is the version of libusb0.sys are you using? You may want to try the latest 1.2.7.3 release if you are still using the older libusb0.sys 1.2.6.0 version.

Does the problem occure if you switch to WinUSB driver or libusbK.sys driver? If you suspect libusbK.dll issue, you may want to remove libusbK.dll and use WinUSB driver to try it out.

Another thing, some devices may choke at libusb_clear_halt due to FW bugs (eg: USB data toggle) problem and may need libusb_reset to go back to the normal state.

[sonatique]

Hello @mcuee : I am using 1.2.7.3 indeed. And this behavior is happening with plain physical Windows (in case you wonder if I am on virtual machine, since I posted PR #1202 .

I still need to test with switching drivers indeed, though it is currently not an option to use them for real, beside for testing.

And yes: I am pretty sure that there is some bugs in the device FW. Right before I get a freezing clear_halt I can see failed ControlTransfer (out), that should not happen.

What I see is that as soon as I have a failed ControlTransfer, the next clear_halt will freeze for ever.

My problem is that I currently don't have access to the FW, so fixing the bug is not an option. I am writing complicated code to detect the failure condition and try a way to reset the device withtout freezing or causing other issues on the software side.

However, I feel like having a function that can freeze so hard that it prevent even Task Manager from killing its calling process, with no way of cancelling it, is not something that is desirable in general.

I don't understand why clear_halt could not return with an error if it can not do its task in a reasonable amount of time. And I didn't test yet, but I am pretty sure (based on what I observed so far) that reset_device would freeze as well.

Did I miss something in the API that could either let me specify a timeout for those functions, or let me call another function to cancel them?

[tormodvolden]

Interesting. However buggy the device is, this shouldn't freeze the program or the machine if a timeout* is given.

There are some parts in libusb0.sys, especially the error paths, that I am unsure about. Do you have the WDK so that you can build and test patched drivers? Can you reproduce this in a VM (would be nice for such tests)?

*) if not in the high-level API, there should be some timeout /somewhere/. libusb0.sys uses the timeout specified in the IOCTL, also for clear_feature.

[sonatique]

Thank you for your interest in my issue. "However buggy the device is, this shouldn't freeze the program": yes, that's my point, I am happy that we agree.

As for now I am working on finding a workaround at the application level, that's the priority.

But in parallel I will swap drivers to better understand who is responsible for this behavior, and in the end decide whether I could try to fix it or rather change driver for good. I should be able to build patched libusb0.sys versions if it happens that this code is where the freeze occurs.

For reference: my device is made of a FX3 connected to a Zynq. The issue happen when the Zynq is power-cycled while the FX3 is kept alive. It occurs with a probabily of 1/5 approximately, so there is probably some dependency of the exact timing of the events.

Also, I just noticed that I have the same issue with various devices when I hiibernate my computer, remove the devices, then wake it up: it seems that some part of the library / driver do not realize what just happened and go on with the tasks they were doing. If I call clear_halt at this point, it will freeze 100% of the times.

[sonatique]

I am wondering whether the "freezes" I experience are somehow related to what is described in #1127 . I will try with the code proposed in this PR as well.

[mcuee]

@sonatique

Any updates on this issue? Thanks.

[sonatique]

@mcuee : Regarding the "freeze": I should soon be able to reproduce it and study it more in depth. I'll keep you posted

[sonatique]

@mcuee : So I have been able to reproduce a similar issue, quite easily actually.
I made a while(1){...} loop issuing libusb_control_transfer of type Out|Vendor|Device to a previously open device. The detail of the transfer doesn't really matter but it corresponds to a command that is supported by the physical device (FX2).
In this loop I also print the return value if < 0.

I start the program, I can see that control transfer out are correctly transmitted and received, then I simply disconnect the USB cable (physically).

The behavior that occurs depends on the driver used. With libusb0.sys 1.2.7.3, the next transfer will fail with error code LIBUSB_ERROR_IO, but the next one will completely freeze the program.
Since I am launching my program from Visual Studio I can break. I see that the program is stuck in the call at line 3157 of windows_winusb.c
if (!WinUSBX[sub_api].ControlTransfer(winusb_handle, *setup ...
and I can see that it is somewhere in libusbK.dll.

From there I have an unkillable process that will stay there until I reboot the OS.

Using kernel debugger I can see that it is frozen somewhere within libusb0:

0: kd> !process 0 7 sam3u_benchmark.exe
PROCESS ffffb38cbaec2080
    SessionId: 1  Cid: 6adc    Peb: e03fc78000  ParentCid: 5924
    DirBase: 6b7b63002  ObjectTable: ffff9d02c8118cc0  HandleCount:   2.
    Image: sam3u_benchmark.exe
    VadRoot ffffb38cbaddf300 Vads 53 Clone 0 Private 324. Modified 259. Locked 0.
    DeviceMap ffff9d02a64ea090
    Token                             ffff9d02c9093060
    ElapsedTime                       01:14:28.309
    UserTime                          00:00:00.046
    KernelTime                        00:00:00.046
    QuotaPoolUsage[PagedPool]         107520
    QuotaPoolUsage[NonPagedPool]      7456
    Working Set Sizes (now,min,max)  (1532, 50, 345) (6128KB, 200KB, 1380KB)
    PeakWorkingSetSize                1885
    VirtualSize                       4183 Mb
    PeakVirtualSize                   4188 Mb
    PageFaultCount                    2017
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      535

        THREAD ffffb38cbaac1080  Cid 6adc.4274  Teb: 000000e03fc79000 Win32Thread: ffffb38cbadd08f0 WAIT: (Executive) KernelMode Non-Alertable
            ffffae0f7001b5f0  NotificationEvent
        IRP List:
            ffffb38cba1b6a20: (0006,0508) Flags: 00060030  Mdl: 00000000
        Not impersonating
        DeviceMap                 ffff9d02a64ea090
        Owning Process            ffffb38cbaec2080       Image:         sam3u_benchmark.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      402061
        Context Switch Count      7847           IdealProcessor: 2
        UserTime                  00:00:00.250
        KernelTime                00:00:00.078
        Win32 Start Address 0x00007ff6c81234f4
        Stack Init ffffae0f7001bc90 Current ffffae0f7001b1d0
        Base ffffae0f7001c000 Limit ffffae0f70016000 Call 0000000000000000
        Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           : Args to Child                                                           : Call Site
        ffffae0f`7001b210 fffff802`4fc1bca0 : 00000000`00000008 00000000`ffffffff 00000000`00000000 ffffb38c`a9187158 : nt!KiSwapContext+0x76
        ffffae0f`7001b350 fffff802`4fc1b1cf : 00000801`000005fd 00000000`00000000 ffffae0f`7001b510 00000000`00000000 : nt!KiSwapThread+0x500
        ffffae0f`7001b400 fffff802`4fc1aa73 : ffffae0f`00000000 00000000`00000000 00000000`00000300 ffffb38c`baac11c0 : nt!KiCommitThreadWait+0x14f
        ffffae0f`7001b4a0 fffff802`8e363f2a : ffffae0f`7001b5f0 00000000`00000000 ffffae0f`7001b600 00000000`00000000 : nt!KeWaitForSingleObject+0x233
        ffffae0f`7001b590 fffff802`8e36713e : 001f0003`ffff0000 fffff802`8e369b50 ffffae0f`7001b7e0 00000000`00000000 : libusb0+0x3f2a
        ffffae0f`7001b620 fffff802`8e362bc0 : ffffb38c`ba8ce1a0 00000000`00000000 ffffb38c`ba8ce1a0 ffffb38c`9bfeca00 : libusb0+0x713e
        ffffae0f`7001b720 fffff802`8e3614b9 : 00000000`00000002 ffffb38c`ba1b6a20 00000000`00000000 00000000`00000000 : libusb0+0x2bc0
        ffffae0f`7001b7d0 fffff802`4fc10665 : ffffb38c`b9034500 00000000`00000000 ffffae0f`20206f49 00000000`00000020 : libusb0+0x14b9
        ffffae0f`7001b800 fffff802`50000fec : 00000000`00000002 00000000`00000000 ffffb38c`b9034500 ffffb38c`baec2080 : nt!IofCallDriver+0x55
        ffffae0f`7001b840 fffff802`50000c41 : ffffb38c`ba1b6e98 ffffae0f`7001bb80 00000000`00040005 ffffb38c`ba1b6e98 : nt!IopSynchronousServiceTail+0x34c
        ffffae0f`7001b8e0 fffff802`4fffffb6 : 00000000`00000000 00000000`000002fc 00000000`00000000 000000e0`3feff890 : nt!IopXxxControlFile+0xc71
        ffffae0f`7001ba20 fffff802`4fe105f8 : ffffb38c`baac1080 000000e0`3feff7b8 ffffae0f`7001baa8 00000000`00000000 : nt!NtDeviceIoControlFile+0x56
        ffffae0f`7001ba90 00007ffc`d950cf34 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x28 (TrapFrame @ ffffae0f`7001bb00)
        000000e0`3feff788 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffc`d950cf34


0: kd> !processirps ffffb38cbaec2080
**** PROCESS ffffb38cbaec2080 (Image: sam3u_benchmar) ****

Checking threads for IRPs.

  Thread ffffb38cbaac1080:

    IRP ffffb38cba1b6a20 - Owned by \Driver\libusb0 for device libusb00001 (ffffb38cba8ce050)

Checking file objects for IRPs.

I tried to have a look at libusb-win32 source code, but so far I didn't understand exactly what is the issue, and I doubt I will.

Using Zadig to replace driver by either WinUSB or libusbK and trying the same experiment result in no freeze at all: after I unplugged the USB cable, libusb_control_transfer simply returns LIBUSB_ERROR_NOT_FOUND .

So, it is not exactly the issue with libusb_clear_halt, but it shows that there is a problem with libusb0.sys, I guess

I think this also shows that there is no real problem with libsub-1.0 itself.