Quality of Service

I'm going to take off my MySQL DBA hat and put on my network admin hat for a moment and talk about Quality of Service (QoS). This is largely in the scope of using QoS for residential and small office setups where you largely see non-symmetric connections, often with fairly low upload speeds, at least here in the US. My own home connection is a U-Verse FTTP install that gives me 50mbps down and a measly 5mbps up.

To start, maybe it's best to play out a common scenario. You're trying to get some work done and have to SSH into a server, while uploading some larger files (say backups or a video or something); while your significant other is wanting to watch Netflix; and your friend who is sleeping on your couch this week due to an argument he had with his girlfriend wants to play a heated game of League of Legends. All this plays out and what happens? No one is happy. Your SSH session is all clunky and you keep misspelling commands; your friend is yelling at someone over Mumble about why they lost to newbs; and Netflix, if it is working, is in ugly standard definition with blocky artifacts everywhere and/or the evil rebuffing screen.

A lot of people I talk to seem to complain about their ISP for the above scenario. And sure, ISPs could offer us awesome super speed symmetric connections but, at least in the US, many ISPs come up with poor excuses while we all wait for Google Fiber to finally bring competition to broadband. As an aside, think it is coincidence ATT starts offering crazy high speed fiber in Austin, TX when Google Fiber comes to town? I don’t think so. Likewise ISPs like to throttle YouTube and Netflix these days (there are ways around that, but that’s for another article). Either way, apart from blaming the ISP, there’s actually plenty that can be done with a bit of work to fix this problem on your own by implementing QoS on your network.

QoS is a somewhat broad topic but essentially the biggest thing we are trying to solve for is avoiding saturation of the Internet connection. The idea is trying to avoid saturation and make things fair (or, if you really wanted to, totally unfair) for various services. So using the scenario above, we have quite a few types of services. Terminal SSH, Mumble, and League of Legends all depend on having a low latency connection since there is a lot of time dependent interaction. Netflix doesn’t care as much about latency, but needs a steady packet flow. Our file uploads just need to go fast enough to where it doesn’t time out but otherwise we don’t care much about speed, although we don’t want it to impact everything else. So how might we fix all that?

Effectively what is happening here is that we are sending packets out as fast as possible. When we try to send them out faster than our interface allows, they start getting queued. Without putting intelligence on how the packets are queued, that queue can get saturated with packets. That means when we send out a packet for SSH, it gets stuck at the end of that queue and has to wait for everything in front to finish. SSH doesn’t really send out that many packets, as compared to say a file upload; but we want to make sure they go out as fast as possible so we don’t experience delays. On the inbound side of things, if we not segment traffic, we don’t really have a means to avoid, say, having Netflix streams from eating up all our inbound bandwidth.

To solve this, let’s start with outbound (upload) services. Here we can make a weighted queue that can prioritize packets depending on parameters. SSH, for instance, may get a high priority and those packets may get to “cut in line” other packets. For bulk file uploads, we might want to give an intentionally low priority to allow most other services to go out first. Ultimately, that means that we can keep using SSH at the expense of some upload speed, but otherwise both can happily coexist.

For inbound (downloads), things work a bit differently. We need to have a way to tell folks sending traffic to us to slow down. One way to do that is to leverage the features found in the TCP protocol. If we start getting too much data from a source, we can send replies to ask them to adjust the rate of transmission or can even simply drop packets as a means to slow things down. Say everyone in the house wanted to watch Netflix, for instance. We could end up opening more streams than we have bandwidth for, and the end result might be everyone streams buffer, but more importantly, all our other services will be impacted as well because our bandwidth is being overrun with Netflix. One way around this is to limit the amount of bandwidth Netflix is allowed to use. The end result is that all of us might not be able to watch our favorite Netflix shows separately, but at least it doesn’t saturate or connection and allows other services to continue to function at a reasonable level. That or, instead of us watching separate Netflix shows separately, we be the family from the 90’s and just all huddle around one TV. I suppose that would also solve the network congestion issue…

The above glosses over quite a few details - there are many ways to setup QoS depending on your needs and use case. And if all that was clear as mud, the takeaway is that we can improve, sometimes dramatically, our Internet experience by leveraging Quality of Service. You might find that your ISP-issued router may already do this, in which case, there isn’t really anything else you need to do. But if not, or if their QoS implementation is sub-par, you can do it on your own! Most higher end Home/SOHO routers offer some level of QoS, and if you really want to get down an dirty, you can setup a Linux router and use iptables and tc to set this up. It’s certainly a great way to understand, on a pretty deep level, how this all works.

In my case, I went with a RouterBoard CRS109. Of all the ones I have tried, it has worked the best, by far, with involving the least amount of configuration of QoS. I pretty much just turned it on and went. I’m a big fan of the RouterBoard hardware, but they also offer the software, RouterOS, for commodity hardware, such as standard PCs. It is Linux-based solution, albeit with some proprietary add-ons. You can think of it as DD-WRT, but on steroids.

For those thinking about RouterBoard, my particular setup is uses hardware QoS (on the switch chip) and software QoS. Both are configured for the WAN (Internet) port. The hardware QoS is set to a rate of 5M, bit metering, and a single queue (queue 0). The software QoS setup for uploads is set to 5M using the ‘default-small’ queue type; and downloads set to 45M using the ‘RED-default’ queue type. And that’s really all I needed to do. It works like a champ. The above scenario I mentioned is fictional, but I have had a similar with no problems. I’ve hosted LAN parties at my house where, before, downloading games from Steam could nuke online games, but is no longer a problem.

RouterBoard/RouterOS isn’t the most user-friendly experience for someone that doesn’t like to mess around with networking stuff, but it is very powerful. There are also many alternatives, such as Astaro (also a Linux-based solution), TPLink, ASUS, etc. In doing research I found that ASUS seemed to have the easiest to use QoS solution for non-network peeps, though I have not used it directly.

The TL;DR is, if you often run into slow network performance and overall network congestion, it might be time to bust out the manual of your router and see if it has QoS. Or find a router that does, spend a little time setting it up, and then be happy! (Do make sure your router has enough horsepower to handle your Internet speeds, however!)