zlp-send quirk should default to true (and perhaps be removed)

[fabled]

The USB ZLP (Zero Length Packet) with Bulk Transfers seems to have been widely misunderstood. Especially because OSes need to offer low-level driver implementation APIs that have the ability to omit sending it. But the ZLP really needs to be generated in various conditions or things will break or have delays in randomly seeming manner.

USB 2.0 specification section 5.8.3 Bulk Transfer Packet Size Constraints says the following:

A bulk transfer is complete when the endpoint does one of the following:
•Has transferred exactly the amount of data expected
•Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet
When a bulk transfer is complete, the Host Controller retires the current IRP and advances to the next IRP. If a data payload is received that is larger than expected, all pending bulk IRPs for that endpoint will be aborted/retired.

In addition to the above, the host controller offers an IRP level timeout after which the IRP times out. The application may then choose to consider the timeout as a success or an error (e.g. based on the data received).

On Bulk endpoints the transfer typically maps to a read or write operation - also typically on application level a full APDU (e.g. HTTP request) does not need to happen within a single transfer, but it can span multiple transfer. That is, you can write multiple times (do multiple transfers) and the other side then on application level determines based on content when the APDU/HTTP request is complete.

The significant thing is that if the final transfer is a multiple of the USB maximum packet size and the ZLP is not sent, the transfer is not considered complete. While some USB stacks might still handle it. The USB specification is clear that the IRP is not complete, and the application will not see it. And this is a host controller (not OS) related feature. (Think of transfer complete as "flush the buffers from host controller to OS/application".) Without ZLP when needed, the IRP will typically timeout at some point and then it is completely up to the device if it proceeds handling at as valid data, or generating a timeout error. I've seen some USB-IPP devices where this actually depends on what type of request was being made. Other side effects are that print jobs do not print immediately (if any of the IPP requests suffer for this, the timeout needs happen - typically several seconds).

The other problem is that when streaming large amounts of print data, we might end up sending huge transfer sizes. That is we send full USB packets continuously for a long time. The USB specification says that the data payload is larger than expected, all pending bulk IRPs will be aborted/retired. Now some printers handle this fine, but I've seen devices that crash (though this should really be fixed on the device to). USB-IPP does not state maximum transfer size, but the Appendix C there suggests that typically large requests are split to multiple transfers. There is not requirement on boundaries, alignment other matters. When reading IPP-USB the transfers just map to read/write.

I personally think that libusb should have inverted the flag. Most users typically want ZLPs to be automatically generated or things will break in strange ways randomly and depending on device. LIBUSB_TRANSFER_ADD_ZERO_PACKET should really be LIBUSB_TRANSFER_NO_ZERO_PACKET. And have ZLP generation on by default.

But given the following facts on not generating ZLPs:

• is against the "spirit" of USB 5.8.3 and IPP-USB
• will cause random timeouts which different devices handle differently (some work but with additional delay causing perf issues, some error out)
• may cause other compatibility issues during streaming of large print data

I strongly recommend to enable zlp-send by default, or even just removing the quirk.

Thank you for considering.

PS. Perhaps this will also explain more about the received ZLPs. Normally the OS will drain these from received data. But applications can observe them if a they issue bulk read, and the peripheral sends no data (ZLP) as a response. Typically this means the buffers are flushed from the peripheral and no data is pending. But depending on the internal implementation of a peripheral this might be just an implementation quirk. In other cases it can actually mean what it should: there is no more data going to be available. So its always safe to ignore ZLPs, and typically it should be used to indicate that the upper protocol layer should now look into what is happening.

[alexpevzner]

Actually, zlp-send quirk was introduced in 0.9.29, and previous version of ipp-usb didn't use the LIBUSB_TRANSFER_ADD_ZERO_PACKET at all.

So I've just added a new quirk just for completeness and left the previously existent behaviour by default.

Regarding data streaming, I've looked into the http implementation, currently it never writes more that 32K at once (defined by the copyBufPoolSize constant at server.go), but may be it makes sense to make this limit explicit, so newer versions of Go stdlib will not break ipp-usb behaviour here...

[alexpevzner]

If I enable zlp-send by default, it will change ipp-usb behaviour for all devices. I'm concerned about the possible regression...

[fabled]

Actually, zlp-send quirk was introduced in 0.9.29, and previous version of ipp-usb didn't use the LIBUSB_TRANSFER_ADD_ZERO_PACKET at all.

Yes, ipp-usb behavior has been buggy all the time in this regard.

So I've just added a new quirk just for completeness and left the previously existent behaviour by default.

I think the proper fix would have been to just add the flag unconditionally without a quirk.

currently it never writes more that 32K at once

But this is not what the printer sees. Since 32K is a multiple of the usb packet size, and if no ZLP is sent, all consecutive 32K bulk sends get combined to one transfer. Exactly because ZLP is not sent. This is exactly the point of a ZLP. To signal the peripheral controller that transfer is done and should be flushed from hardware to software stack. If this does not happe, the peripheral ends up seeing huge several megabyte transfers. While the device should still handle this, I've now seen devices that break when this happens. So in theory LIBUSB_TRANSFER_ADD_ZERO_PACKET is not strictly needed here even if it helps with compliance (I think no place says what is the maximum support USB transfer size, but it generally helps if its not too huge).

The more important fix this does is that if a shorter HTTP/IPP request is sent (or above mentioned large stream), and the transfer size happens to be a multiple of USB packet size, the flush ends up happening via a timeout. In this cases this is a really important performance fix.

OTOH, in some platforms LIBUSB_TRANSFER_ADD_ZERO_PACKET is really an endpoint feature (and not a transfer specific one), so it should be set consistently for all transfers.

If I enable zlp-send by default, it will change ipp-usb behaviour for all devices. I'm concerned about the possible regression...

There should be zero regressions as ZLP is USB standard specified feature which must be used, and also IPP-USB specification indicates that HTTP requests should be split to multiple write requests. In the very unlikely case of regression, it is a peripheral firmware bug.

The reason why libusb does not set it by default, is that sometimes applications may want to intentionally stream data that appears as single transfer, but is generated by the code dynamically and the length might not be known in advance. In this case the application can just generate usb bulk writes as it wants. And to signal end of transfer the application should explicitly do a usb bulk write of zero length in the cases it is needed by usb specification.

Since ipp-usb is not generating zero length usb bulk writes itself, it should set LIBUSB_TRANSFER_ADD_ZERO_PACKET. This flag does not always send ZLP, only in the cases it is needed (after a bulk transfer that is multiple of usb packet size).

Alternatively, you should never set the flag, but instead generate a ZLP manually when the full HTTP layer request is sent and the last USB bulk transfer was a multiple of USB packet size.

[fabled]

defined by the copyBufPoolSize constant at server.go

The http transfer code in Go ends up using io.CopyBuffer to do the actual work. Which does in a loop something like:

            nr, er := src.Read(buf)
           if nr > 0 {
                  nw, ew := dst.Write(buf[0:nr])

Basically, the data that gets written is at maximum 32K, but it can be shorter if the network read was shorter. This results that the issue is not visible if there is even temporary starvation of the reader. Because if this loop sends a buffer that is not a multiple of usb packet size to your code it goes directly to a usb transfer. And since the different size, usb protocol says its ends up signalling end of transfer.

So in the current code base, the usb transfer are non-deterministic. There is no maximum limit (32K omitting ZLP), but the transfer size can be any random size due to potential socket read starvation.

Given this fact that USB transfers lengths are already generated non-deterministically, the generation of ZLP in the case where it is required by the specification can only improve the situation (by removing the timeouts and making sure that no multi megabyte transfers occur).

But given this, if manual ZLP sending is implemented, it would probably make sense to rewrite the full usb transfer code to always use usb transfers that are multiple of usb packet size.

[fabled]

Perhaps it would make sense to implement ReadFrom to avoid golang buffering. We could then control the buffering better. This avoid extra copies and allow us to always send transfers multiples of USB packet size which would also improve performance.