When should a USB device send a ZLP on a bulk pipe?

Asked 25/1, 2017 at 15:54 Answered 26/1, 2017 at 11:6

I'm writing firmware for a USB 2.0 full speed device that communicates with a WinUSB host, with one Bulk Pipe in each direction. When should the device send a zero-length packet (ZLP) to terminate an IN transfer, and how does it know that it should?

Section 5.8.3 of the USB 2.0 spec says:

A bulk transfer is complete when the endpoint does one of the following:

Has transferred exactly the amount of data expected

Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet [ZLP]

I interpret this to mean that a ZLP should be sent when the transfer size is an integer multiple of the max packet size, and the "expected" size of the transfer is greater than the actual size (i.e. what is available to be sent). But how does the recipient know what's expected?

For instance, I'm using the WinUSBNet wrapper in C#. When I read from the pipe like this

int bytesRead;
buffer = new byte[128];
try
{
   bytesRead = m_PipeIN.Read(buffer);
   buffer = buffer.Take(bytesRead).ToArray();
}

the library calls WinUsb_ReadPipe() like this:

WinUsb_ReadPipe(InterfaceHandle(ifaceIndex),
   pipeID,
   pBuffer + offset,
   (uint)bytesToRead,
   out bytesRead,
   IntPtr.Zero);

Suppose the device has exactly 128 bytes to send, and max packet size is 64 bytes. How does the device determine what the host is "expecting", thus whether it should send a ZLP to terminate the transfer?

(Similar to this question, but that one is about control pipes. I'm asking about bulk pipes.)

Ursola answered 25/1, 2017 at 15:54 Comment(0)

Explanation of the spec:

Case 1

Has transferred exactly the amount of data expected

This means that if the host is expecting X amount of bytes, and you send exactly X amount of bytes, the transfer stops right there. MPS and ZLP don't play into it.

Case 2

Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet [ZLP]

This means that if the host is expecting X bytes but you want to send only Y bytes, where Y < X, the transfer is complete as soon as you do a "short" packet, a packet less the MPS. If Y bytes is a multiple of MPS, then you would have to do a ZLP.

Example 1 (no ZLP)

MPS = 512, the host expects 8192 bytes.

You want to send only 1500 bytes. The payload would go over in 3 packets like this:

Packet 0: [512 bytes]  MPS
Packet 1: [512 bytes]  MPS
Packet 2: [476 bytes]  short packet

When the host gets the short packet, it knows the transfer is complete, and won't continue asking for more packets for the transfer.

Example 2 (with ZLP)

MPS = 512, the host expects 8192 bytes.

You want to send only 2048 bytes. The payload would go over in 4 packets like this:

Packet 0: [512 bytes]  MPS
Packet 1: [512 bytes]  MPS
Packet 2: [512 bytes]  MPS
Packet 3: [512 bytes]  MPS

At this point, the host has received 4 MPS-sized packets so it doesn't know the transfer is complete. So it will continue to request packets from the device.

Packet 4: [0 bytes]  short packet (ZLP)

When the host gets the short packet, it knows the transfer is complete, and won't continue asking for more packets for the transfer.

Determining Transfer Size

You may be wondering how to determine the "expected" amount of bytes since BULK transfers do not have a length like CTRL transfers do. This is determined entirely by the higher-level protocol that specifies how to do transfers on the BULK pipes. The host and device both follow this protocol and thus they are in sync about how much data to transfer at any given time.

This protocol is typically specified by a class specification, like the mass-storage class protocol, or it could be some very simple protocol of your own design.

Polio answered 26/1, 2017 at 6:57 Comment(2)

You may be wondering how to determine the "expected" amount of bytes since BULK transfers do not have a length like CTRL transfers do.

Yes, you've hit the nail on the head. This is determined entirely by the higher-level protocol... Isn't this bad design on the part of USB-IF? In your ex 2, upon providing data for the IN transfer, the protocol/class layer either needs to [1] provide a boolean to the USB driver saying whether or not to include the ZLP, or [2] send the ZLP itself by sending a zero-length transaction when it gets the next data request. Munging layers... – Ursola 26/1, 2017 at 17:3

That's right, and it is by design. The lower levels of the stack don't know about the upper level procotols. It has no notion that the transfer is 8k but we're only sending 2k. – Polio 27/1, 2017 at 3:44

Transfers a packet with a payload size less than wMaxPacketSize or transfers a zero-length packet [ZLP]

a ZLP has to be send when the length of payload data is exactly an integer multiple of wMaxPacketSize

The USB spec defines that if the last packet of a bulk transfer has the exact size of the endpoint max packet size, the whole transfer must be terminated by a zero length urb.

If apps don't sent this in such a situation libusb times out and the initial urb is never sent resulting in a broken application.

All Kernel drivers use the URB_ZERO_PACKET to comply to the spec correctly.

source: http://libusb.org/ticket/6

in case that data length is exactly an integer multiple of wMaxPacketSize the first ending condition packetSize < wMaxPacketSize does not apply because in this case packetSize = wMaxPacketSize.

so to indicate the information that the last packet has been send you need a ZLP, else the other side would expect more data

there are several other situations when ZLPs are sent see i.e. USB in a nutshell website

Warfold answered 26/1, 2017 at 11:6 Comment(17)

i think you send the ZLP when the last packet has exactly the wMaxPacketSize - But not always when it is wMaxPacketSize, right? "USB in a nutshell" says, A bulk transfer is considered complete when it has transferred the exact amount of data requested... As I read it, it's saying that in that case, the ZLP is not needed. But how does the device determine whether the next Bulk IN Transaction belongs to the previous transfer or the next one? – Ursola 26/1, 2017 at 16:43

in usb only the host can initiate communication, even if a device has an interrupt it has to wait until the host polls it. so only the host can start a new bulk transfer. it does this by sending an IN token to the specific endpoint IN: When the host is ready to receive bulk data it issues an IN Token. (beyondlogic.org/usbnutshell/usb4.shtml#Bulk) the host can only start a new bulk transfer if it has ended the previous, either by receiving a ZLP or a packet shorter than wMaxPacketSize – Warfold 26/1, 2017 at 19:8

host can only start a new bulk transfer if it has ended the previous, either by receiving a ZLP or a packet shorter than wMaxPacketSize

- Not true, per the quote of the USB spec in my question. (First bullet.) – Ursola 26/1, 2017 at 19:25

host can only start a new bulk transfer if it has ended the previous, either by receiving a ZLP or a packet shorter than wMaxPacketSize to the same endpoint With the statement a packet shorter than wMaxPacketSize i assume that it is the last packet and the full data is transfered with it. because this is the breaking condition : the statement has transfered the complete data is equivalent to transfered a packet with less than wMaxPacketSize – Warfold 26/1, 2017 at 19:38

Incorrect. See again my quote of the USB 2.0 spec in the question. A bulk transfer is complete when the endpoint does *one* of the following... (Emphasis mine.) You're saying that the second bullet must always be true. ...I'm writing firmware, so I'm concerned with the device perspective. I'm not talking about a host interrupting its own transfer. – Ursola 26/1, 2017 at 19:42

yes, i think the second bullet has to be always true. else if data has length that is not an integer multiple of wMaxPacketSize the remaining bytes have to be stuffed by USB protocol (to get a full packet). this is not the case i think... so i think you send a ZLP if data length is an integer multiple of wMaxPacketSize – Warfold 26/1, 2017 at 23:4

If that were true, the first bullet wouldn't be there. Are you seriously claiming that the USB 2.0 specification is wrong about how USB works? ... Mind you, I am also complaining about the ambiguity the behavior introduces. But if the higher layers are aware of it, they can account for and resolve it. – Ursola 26/1, 2017 at 23:13

@cp.engr: It would be correct to say that a TCP connection becomes unwriteable if either the local endpoint sends a packet with a FIN flag, or the remote endpoint catches fire and explodes. This does not imply you need to detect when the remote has exploded and skip sending a FIN flag. – Conjure 27/1, 2017 at 0:32

@BenVoigt, becoming unwritable and successfully completing a transfer are two different things. If the remote host explodes before terminating the message he sends me, I'm not going to process it. – Ursola 27/1, 2017 at 0:43

@ralfhtp, I think I've made a decent case that your answer here is incorrect. I will add to the evidence in comments or chat to follow this comment, using citations from the libusb ticket you cited. If you can neither convince me that I'm mistaken (now seems unlikely), nor correct your answer, then I anticipate downvoting. – Ursola 1/2, 2017 at 17:29

[Note to future readers: ralf's citation was to a bug report from the open source "libusb" library. There is substantial discussion there about what the correct implementation would be. Not all statements are definitive. Now on to evidence...] – Ursola 1/2, 2017 at 17:33

1. First, consider that the change implemented (and the discussion preceding it) was to add a flag to each "submit transfer" request indicating whether or not a ZLP needs to follow if (transfer_len % dev->wMaxPacketSize == 0). – Ursola 1/2, 2017 at 17:33

2. You quoted "M.S."'s statement,

The USB spec defines that if the last packet of a bulk transfer has the exact size of the endpoint max packet size, the whole transfer must be terminated by a zero length urb.

This is incorrect. Even M.S. himself seems to know this, when he later says,

It is *common* for usb bulk transfers in general and not some random mutation of a specific usb device (like the iPhone).

[Emphasis mine.] Common, not universal. – Ursola 1/2, 2017 at 17:34

3. User dsd references USB 2.0 spec section 5.8.3 in his comment #15. Read the 3 paragraphs beginning with, Note that it says *one* of the following, not both... – Ursola 1/2, 2017 at 17:36

4. More user dsd:

If you enable this flag all the time in this fashion, you are going to break things for a lot of users. I have a device with a bulk out endpoint, max packet size 32, and each command I send on this endpoint is 32 bytes. If I send 31 bytes, I get a stall. If I send 0 bytes, I get a stall. Commands are always exactly 32 bytes, as documented in the device specifications. If you implement a patch as you describe, every time I send a 32-byte command your patch will cause an illegal ZLP to be sent after, which will cause an endpoint stall.

– Ursola 1/2, 2017 at 17:36

5. User dsd again: We seem to have reached agreement above that some protocols require ZLPs and some do not. – Ursola 1/2, 2017 at 17:37

*Some* protocols which use USB require an extra zero length data packet to signal end-of-transfer on bulk endpoints, if the last data packet is exactly wMaxPacketSize bytes long.

[Emphasis mine.] Note some protocols, not all. – Ursola 1/2, 2017 at 17:37

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++