Did Bufferbloat kill my net radio?
Low rate streams suffer when bloat dominates

About 5 years ago I stopped listening to internet radio.

If I (or someone else) was also using the Net connection, the audio dropouts per minute were so annoying that I shifted to purely downloadable media such as podcasts. I started using streamripper on the radio stations I liked, and downloading those streams, at night, when I wasn’t otherwise using the network. I would dump the files on my laptop or mp3 player, and lo and behold I had a steady supply of new music and news to listen to, that wasn’t annoying to listen to.

This shift in my behavior meant that I had to give up live and random internet radio entirely. And I did. Listening to internet radio had become intolerable.

I didn’t understand why it was happening at the time. I had tons and tons of bandwidth. I had the most advanced OS in the world. I was working on bleeding edge hardware. WTF?

I surmised that the Internet was going to hell, and moved on. Later on I observed people with Mac and Vista hardware also driven mad by their inability to websurf and listen to Internet radio. They had a choice of one or the other.

[[!img /images/housenet.png]] Recently I added a NAS/gw to my network. It is both my internet gateway and a 3.5TB file server. As it was on ²⁴⁄₇ and using only 17 watts I thought it was a good place to move my radio storage to rather than my laptop, so I could save energy.

Imagine my surprise when I tried to stream radio from that - across two wireless cards, to my laptop - AND do other stuff - only to have THAT audio start breaking up. My internal network runs at 1000Mbit/sec. My radios are 802.11n - I typically get 70Mbit/sec. Surely a little .128Mbit tcp stream should co-exist with all other streams on the network? With just one, at least?

Nope. I ran a load generating test tool (iperf) while running an audio stream via samba (cifs - it’s an old fashioned filesharing protocol, not web based, you may have heard of it), with the current Linux defaults throughout the network, and hit dropout city.

Now, maybe, I have a handle on the real problem; a theory. It could be bufferbloat! The huge amounts of buffering in my (linux and mac devices) is interfering with modern TCP’s congestion control servo mechanisms, ramping up too much while bloated buffers filled.

Did bufferbloat kill my internet radio? You would think that two tcp streams - one low volume - 128kB/sec - and another - running as fast as it could - 70Mbit - could co-exist, wouldn’t you?

They don’t. TCP/ip is supposed to be fair. It doesn’t appear to be. I knew life was unfair, but TCP is supposed to be!

So, after reducing my txqueuelens to a small number (16, down from a 1000), and my dma tx buffers as low as they can go (64, down from 256), one of my music players (aqualung) started working over my internal wireless network… under a saturating load generated by iperf - perfectly! Another (mplayer) still doesn’t. Apparently aqualung is doing deeper buffering of it’s own that’s larger than the current (much smaller) period of the bufferbloat induced jitter. How much, I don’t know.

I need to procede cautiously, remembering rfc968, and retest a few assumptions from the above, but:

My hypothesis is: 64 DMA tx buffers is STILL too much. Shrinking that’s going to require some hacking, but thankfully I have source code to all the devices in the loop and I figure it’ll only take me a couple days to see if I can bring that down to, say, 4, throughout.

There’s numerous other hypotheses I can try on top of this - various forms of traffic shaping - exploring the buffering mechanisms mplayer and aqualung use - exploring samba’s internals - looking at packet drops and retransmits - exploring what SACK does - trying out ECN - experimenting with radio speeds - in a nice, simple test environment.

All of these tests are aimed at solving a real problem:

Getting my beloved internet radio back.

IF my real problem with streaming radio has been, specifically, wireless bufferbloat all along I’ll be a really, really, really happy guy. Getting a solution worldwide may require an effort on the scale of Y2K - but at least - maybe - hopefully - having this diagnosis will lead to a cure, and the internet’s potential for even more interactive applications like voip and gaming, fulfilled.

Update: Wireless bufferbloat is proving far worse than expected. And my initial attempts at implementing all the above seemed to improve streaming over CIFS, but did not fix it entirely. From looking at the TCP traces, aqualung does a LOT of buffering, mplayer does nearly none, and so we end up with requests absurdly delayed by my IWL driver 130ms or more, further delayed by other buffers I have no control over, and streaming fails. I have fixes for the IWL and AR71xx but have not had time to test them, as I’ve been too busy helping get bufferbloat.net up and running.

If you think you are suffering from bufferbloat, here are some simpler tests than mine you can perform.

February 10, 2011
884 words