PROBLEM: USB isochronous urb leak on EHCI driver

Tue Jan 6 02:12:37 AEDT 2015

>
> On Mon, 15 Dec 2014, Michael Tessier wrote:
>
> > Hi,
> > 
> > I am dealing with a USB EHCI driver bug. Here is the info:
> > 
> > My configuration:
> > -----------------
> > 
> > Host: Freescale i.MX512 with ARM Cortex A8 (USB 2.0 host controller) 
> > Linux kernel: 2.6.31, using EHCI USB driver
>
> As mentioned by other people, the age of that kernel makes any bug report completely irrelevant.  It's hard to count the number of non-trivial changes that have  > been made to the isochronous code in ehci-hcd since 2.6.31, but there have been quite a few.
>
> > Hub: 4-PORT USB 1.1 HUB (Texas Instruments PN: tusb2046b)
> > Devices: 4 USB 1.1 audio codecs (Texas Instruments PN: pcm2901)
> > 
> > Note: each codec is being used in R/W access, so with 4 codecs, I have
> > 4 playback and 4 capture streams.
> > 
> > My problem:
> > -----------
> > 
> > I have usb urb leaks when connecting more than 1 codec to the USB 1.1 
> > Hub.
>
> What do you mean by "urb leak"?  Normally, people use the word "leak"  
> to refer to memory that is dynamically allocated and never deallocated, but you seem to mean something else.

You are right. What I mean by leak is the following: At application level,
all my calls to "Read" or "Write" operation to the codec driver will return
with the correct amount of bytes read/written, with a "choppy" sound. Then
when looking at lower levels:

snd_pcm_oss_write (pcm_oss.c) 	-> OK
snd_pcm_lib_write (pcm_lib.c) 	-> OK
usb_submit_urb 	(urb.c)		-> FAIL with 3 codecs

The "FAIL" here indicates that the total amount of bytes transferred does
not correspond to what was expected. And indeed the sound is "choppy" when
using more than a certain amount of bandwidth. However this amount of
bandwidth is higher when connecting only 1 codec with different settings
(48khz-stereo 16-bits instead of 32 khz-mono 16-bits).So at some point it
looks like the bug is in the scheduler, only with several isochronous links.

>
> > (the result is that some of the audio data is not transferred, part of 
> > the sound is simply missing) No problem when using only 1 of the 4 
> > codecs connected to the hub; When I connect a second codec, the sound 
> > quality starts to degrade. With 3 codecs, we just cannot recognize a 
> > speach.
> > 
> > Tests and observations:
> > -----------------------
> > 
> > Since I have 3 usb ports available on the i.MX512, I tried to connect
> > 3 codecs directly on USB ports: the sound is perfect on each of the 
> > three ports.
> > 
> > I bought a consumer USB 2.0 Hub: no problem when using 3 codecs 
> > connected to that Hub, however, the audio will completly stop on all 
> > channels when connecting the 4th codec.
>
> Above you said the sound started to degrade when the second codec was connected; here you say there is no problem when using 3 of them.  
> Which is it?  Do you mean that the high-speed hub works better than the full-speed hub?
>
Yes, that's it. Using the high-speed hub will allow for more data throughput
before starting to "miss" some usb packets (and result in a choppy sound).

> > I checked the communication between the Hub (USB 1.1) and the Host 
> > controller (USB 2.0) with a scope and concluded that the communication 
> > speed is 1.5 MBytes/s has expected (so the communication is downgraded 
> > to USB 1.1, since codecs and hub are USB
> > 1.1 devices).
> > 
> > Also, I know that there is physically enough bandwidth to transfer the 
> > data for two reasons:
> > 1) I have an older CPU with a USB 1.1 host controller (using the OHCI 
> > driver), using the same hub and the same codecs: works like a champ, 
> > using less than 50% of the available bandwidth (observed with a
> > scope)
> > 2) 1 audio stream is 32khz-mono, 16 bits = 64 kB/s,
> > 4 codecs = 8 streams(R/W) x 64 kB/s = 512 kB/s (out of 1.5MB/s)
>
> The amount of bandwidth available is usually not as much of an issue as the ability of the scheduling alogorithm to divide the bandwidth among the streams.  The
> algorithm is not very smart and it often runs into a wall even when lots of physical bandwidth is still available.

That is interresting, however, I have an older kernel running an OHCI
driver which is able to handle 4 codecs. Same usb hardware (codecs and
hub), but older kernel on a different CPU, with much less power. This makes
me believe that there's a solution to make it work...

> > I noticed that my sound problem starts happening with only 2 codecs
> > (4 streams, 256 kB/s). I first thought that it was a bandwidth 
> > limitation, so I decided to connect only 1 codec using more bandwidth.
> > I configured it to 48khz-stereo (16-bits), using 384 kB/s for both 
> > read and write streams: no problem. With that configuration, the scope 
> > shows about 30% of total bandwidth usage (300us used out of 1ms 
> > periods). Then, I added a second codec (48khz-stereo-16bits): very 
> > strange, now the total bandwidth usage felt down to about 200us, which 
> > seems to keep the same, whatever the number of codec I add (I also 
> > tried 3 and 4...). So it looks like the scheduler is not able to 
> > properly allocate Isochronous time slots when more than one device is 
> > connected to the hub. However, without the hub, it works perfectly.
>
> How does your hardware connect the host controller to a full-speed device?  Is there an internal hub (Intel motherboards have used this approach)?  Is there a 
> companion USB-1.1 controller (older motherboards from Intel and other companys have used this approach)?  Does the EHCI controller have a built-in Transaction 
> Translator (some SOC systems use this approach)?

The CPU is a Freescale i.MX512, with 3 USB 2.0 Host controllers. My hub
is connected to the main CPU board with a standard USB cable, so it's easy
to swap my 4-port hub from a USB 1.1 to a USB 2.0. My codecs are always
the same: USB 1.1 Texas Instruments PN# pcm2901. I don't believe there's
a built-in Transaction Translator. How can I check that?

> > Another interresting fact is that at application level, the Read and 
> > Write operations are returning the good amount of bytes read/written.
> > This is not the case at kernel level: I noticed that function 
> > "usb_submit_urb" (from /drivers/usb/core/urb.c) will only tranfer part 
> > of the "urbs" when the sound is degraded. I tried to figure out where 
> > the leak comes from without success. Also, there are no error messages 
> > from kernel so everything appears to work well, excepted that part of 
> > the sound is missing!
> > 
> > I can't change my hardware (this is in the hand of customers), so the 
> > only possible solution for me is to correct the software.
> > 
> > I tried to change my ehci driver with the one from kernel 2.6.39.4 but 
> > did not work, same problem.
> > 
> > Question:
> > ---------
> > 
> > Before attempting to upgrade to an earlier kernel driver (this is
>
> "upgrade to an earlier kernel driver" is a contradiction in terms.  
> Moving to an earlier driver would be a _downgrade_.

Sorry, I meant to say "newer"...

> > a fairly big amount of work), I would really like to know if this 
> > problem would still be in the 3.x kernels. Has anyone seen that issue 
> > in 3.x kernels?
>
> It depends a lot on the system hardware.  Many people are using USB audio in 3.x kernels with no problem.  On the other hand, some people have reported a bug 
> (quite different from yours) so recently that the patch to fix it has not yet been merged.

I understand. However, if one could test the following with a 3.x kernel:
- CPU with USB 2.0 Host controller (using EHCI-hcd driver)
- 4-port USB 1.1 Hub
- 4x USB codecs (configured at 32khz-mono, 16-bits audio)

Then try to stream audio on each of the 4 codecs at the same time (this
includes one Read and one Write stream on each codec, so total of 4 "Read"
and 4 "Write" streams. Then listen to the output...

If sound is ok when using only 1 codec and becomes choppy when adding a
second codec, then it means that this issue is still in the 3.x kernel. This
answer will tell me if it is worth working on using a newer kernel or not.
I have to say that I'm not a linux expert, so I see the migration to a newer
kernel as a quite big amount of work...

> > I am pretty new to USB driver debugging, so any ideas of where/how to 
> > find solutions will be appreciated. Thank you very much in advance for 
> > the support. Also don't hesitate to redirect me if I'm not at the 
> > right place to ask these questions. I can also provide some code if 
> > someone need it to help.
>
> Your first step should be to use an up-to-date kernel, as recommended by other people.
>
> Alan Stern

Thank you for your prompt response.

Michael Tessier