Bufferbloat explained: why bandwidth isn’t the key to QoE for ISPs

Bufferbloat is high latency (lag) that appears when there’s other traffic on the network. It happens when buffers in routers and other network equipment are too large and nothing manages the queue in them. A big download or upload fills them, and every other packet, including your video call or game, waits in line. A faster connection doesn’t fix it. Queue management does: algorithms such as FQ-CoDel and CAKE keep the queue short.

Many people misunderstand bandwidth and network speed and think latency doesn’t matter over fiber. But that’s not how it works. You’ve got a 10Gbps fiber connection shared with a few others and not even coming close to maxing out the total connection bandwidth. But despite that, latency is high and highly variable. The key here is to remember that bandwidth is not the only or even the main QoE factor these days.

Below we look at latency, bufferbloat and jitter, how to test a connection for bufferbloat, and what a broadband business can do about it on its own network.

It’s all about bandwidth, but not only it

Bandwidth is often the first thing people look at when comparing network connections. But, like you probably didn’t choose your last car solely based on horsepower—or even thought much about it—it’s rarely the most important factor in networking technology.

The truth is that bandwidth has never been a measure of network speed, even though these terms are often (but not correctly!) used interchangeably. It’s like the width of a water pipe—it measures how much data can flow through a network per second. Higher bandwidth lets more data flow at once, but it is only one of the factors affecting overall network performance and speed. A few others are bufferbloat, packet loss, latency and jitter. All of them can make a high-capacity link feel slower than its bandwidth suggests. When the delay comes from bufferbloat, buying more bandwidth won’t make Netflix, Fortnite, YouTube, or regular web pages load any faster.

An end-to-end network path usually consists of multiple link connections, each with a different bandwidth capacity. As a result, a link with the lowest bandwidth is frequently described as the bottleneck because it can limit the overall capacity of all connections in the path. The time between the start and end of each signal transmission adds latency over an individual link. In fact, loss, latency, jitter and out-of-order delivery are the four quality criteria for bandwidth.

Latency matters—congestion and “last mile” problem

If bandwidth is the amount of information sent per second, latency is nothing but time a data packet travels from one point to another across a network. It’s basically a lag/ delay you experience while waiting for something to load. And it’s typically measured in milliseconds (ms).

What causes latency? Typically, there are two main factors: congestion and/or distance. The “last mile,” which is the final leg of the network that connects end users to the internet provider’s infrastructure, matters most. NetForecast’s residential latency benchmark puts it plainly: “latency in the last mile is the major contributor to user frustration.” Fiber optic networks generally have lower latency compared to copper-based technologies like DSL or cable.

Understanding and addressing last-mile latency can be crucial for anyone looking to improve their internet performance. This often requires solutions like lower-latency connections or ISP-managed services prioritizing stable latency rather than just increasing raw bandwidth, especially in real-time communications and systems requiring quick response time.

Network congestion is another common cause of network latency, and here bufferbloat and jitter come into play.

What is bufferbloat and why it hurts calls, games and streaming

Do your video or audio calls sometimes stutter? Do you experience a lag in video games, or does your web browsing slow down? If so, bufferbloat may be to blame.

Internet traffic travels over diverse paths with varying times and bandwidth. To deal with these traffic flow variations and bursts, we use buffers as temporary storage areas. They are everywhere: in routers, NICs, and network boundaries. When someone sends a large file, many packets are sent at once, so the router temporarily holds all the packets in a queue, causing new packets to wait until the others go through.

Bufferbloat happens when buffers in networking equipment (like a router or switch) are too large and nothing manages the queue in them. A large upload or download fills them up (they become “bloated”) and keeps them full, so every new packet waits behind the queue. That causes spikes in your Internet connection’s latency when a device on the network uploads or downloads files. The most straightforward analogy here is sink (your router) and narrow drain (your connection to the ISP, since it’s probably the slowest link in your network). That is what happens when a flow uses more than its fair share of the bottleneck. This might be very familiar to those often stuck in traffic jams in big cities or in line at the grocery store.

If your connection suffers from bufferbloat, you’ll often also see higher jitter

And while packets travel through today’s statistically multiplexed networks, timely packet loss is natural. When packets are dropped, TCP congestion control methods manage network congestion. With large buffers, packets enter instead of being dropped and bufferbloat prevents TCP from controlling flow. This can have detrimental results, including higher latency and jitter.

A jitter is a submeasurement of latency (variation in latency over time) that reflects the difference between delays in the transmission series of packets through a network over time. It indicates how stable this connection is.

As mentioned earlier, high latency, bufferbloat, and jitter all lead to delays and interruptions in network performance. Bufferbloat can increase latency, and inconsistent latency is jitter, which impacts the overall smoothness and speed of your online experience. Jitter usually disrupts VoIP calls, streaming, and online gaming by causing choppy audio, video issues, or lag. In severe cases, high jitter can lead to packet loss (when packets arrive too late to be useful), which results in errors, missing data, or the need to reload pages. Even ordinary web browsing may feel choppy or inconsistent.

Buying a faster connection when link speed isn’t the problem wastes time and money. As bufferbloat.net puts it, if your router buffers more data than necessary, this adds delay that can never be cured by faster transmission rates.

How to test for bufferbloat

A bufferbloat test compares latency in two states: when the line is idle, and when it’s busy with a download or upload. If latency jumps under load, the connection has bufferbloat. Several free web tests measure both:

• Waveform Bufferbloat Test: gives a letter grade for your result.
• Cloudflare Speed Test: detailed statistics of data rates and latency.
• LibreQoS Internet Quality Test: a multi-phase test with a bufferbloat grade.
• Speedtest: web, iOS and Android apps that measure latency.
• Fast.com: click “Show more info” to see latency under load.

Only make sure you’re connected over Ethernet cable, not over WiFi (significantly distorts results).

How to read the result

According to bufferbloat.net, if latency under load stays below 15–25 ms, bufferbloat is under control. If latency during the download or upload part of a test is high (say, above 50 ms) or a grade lower than B, you likely have bufferbloat.

No web test at hand? Start a ping to any well-known host in a terminal window, then run a speed test. If the ping times jump while the test uploads or downloads, the router on that path is probably bloated.

For network engineers, bufferbloat.net recommends Flent, and in particular its RRUL test, which loads the link in both directions and shows download speed, upload speed and latency in one set of charts. See Tests for Bufferbloat for more tools.

How broadband businesses fix bufferbloat: smart queue management

The quickest route to addressing bufferbloat, latency and jitter at the ISP level is to implement a Quality of Experience (QoE) middlebox. These middleboxes are designed to manage network traffic in ways that prioritize time-sensitive packets, ensuring smoother network performance even during high-traffic periods. By leveraging Smart Queue Management (SQM) in QoE middleboxes, broadband businesses worldwide can help mitigate the impact of bufferbloat and maintain stable latency alongside jitter levels. There are many different versions of SQM, and some are more effective than others. Popular algorithms for SQM include CAKE, FQ-CoDel, and PIE, each optimizing packet flow to reduce congestion and improve the end-user experience.

Two terms often get mixed up here:

• Active queue management (AQM) controls the length of a queue, dropping or marking packets before the buffer fills up.
• Smart queue management (SQM) is broader. bufferbloat.net defines it as an integrated system that combines per-flow packet scheduling, AQM, traffic shaping (rate limiting) and QoS prioritization.

The common algorithms are documented by the IETF and the bufferbloat project:

• CoDel (Controlled Delay, RFC 8289): an AQM that controls the excess delay bufferbloat creates.
• FQ-CoDel (RFC 8290): combines flow queueing (packet scheduling) with CoDel’s AQM in one algorithm.
• PIE (RFC 8033): a lightweight AQM designed to address the bufferbloat problem.
• CAKE: the shaper and queue manager that LibreQoS uses to keep busy networks responsive.

QoE middleboxes with SQM implementation tailored for broadband businesses include solutions like LibreQoS, Preseem and Bequant:

• LibreQoS: its shaping core is open source. It gives subscribers per-flow CAKE fairness and keeps shared capacity at access points, PONs, towers, sites and backhaul under hierarchical control.
• Preseem: a QoE monitoring and optimization platform for wireless providers, down to tower, sector and subscriber level.
• Bequant (BQN): its TCP optimization uses a transparent TCP proxy to reduce data latency.

Routers can run smarter queues too. MikroTik RouterOS, for example, offers `fq-codel` and `cake` among its [queue types](https://manual.mikrotik.com/docs/cli-reference/queue/type/), next to the classic `pfifo`, `sfq` and `pcq`.

On the subscriber’s side, bufferbloat.net’s advice is to enable SQM on the home router if it has it, set to the measured line speed, then adjusted until latency stays low under load.

Where Splynx fits

Splynx connects to the QoE platforms above and shows you what subscribers actually get:

• LibreQoS add-on: in Splynx 6.0 it pulls live bandwidth from LibreQoS into the Live usage widget on the customer profile and the internet service. That works for any access technology (IPoE, PPPoE, DHCP, fiber, fixed wireless) as long as the traffic passes through a LibreQoS shaper. A LibreQoS action on each service and a LibreQoS tab on the customer profile open the LibreQoS Insight statistics without leaving Splynx. With Enable live bandwidth usage on, customers see live bandwidth in their portal too. See the LibreQoS add-on documentation.
• Preseem add-on: integrates your Splynx customers, services and network infrastructure with Preseem. See the Preseem add-on documentation.
• Bequant add-on: BQN retrieves customer tariffs, speed limits and burst settings from Splynx. See the Bequant add-on documentation.
• MikroTik: Splynx 6.0 adds live bandwidth for DHCP and IPoE subscribers. Splynx reads per-customer simple-queue statistics directly from the router (API access enabled), with roughly one-second updates; where LibreQoS is enabled, it stays the preferred source.

For speed limits built with simple queues, you choose the queue type Splynx assigns for download and upload under `Config → Networking → MikroTik API`. For plans, speed limits, burst and FUP in Splynx, see bandwidth management in Splynx and our network management overview.

Additionally, the effectiveness of such tools in real-world scenarios is demonstrated in the CTN Malawi case, where LibreQoS integrated with Splynx provided burst speeds when needed, improving user experience without significantly increasing overall throughput.  This approach also strengthened customer loyalty, as subscribers became advocates for the service on social media. For a closer look, check the video below with our Client Relationship Manager, Heather Lombard, and CEO & Co-Founder of CTN,  Brian Longwe.

How to build a strong ISP foundation with Splynx and LibreQoS | CTN Malawi’s Success Story

We need more IXPs for the fastest and cheapest bandwidth

Latency is more noticeable over greater distances; we can’t defeat the laws of physics. This explains why cloud content delivery networks (CDNs) are spread worldwide and data is replicated between data centers, so users can access services closest to their geolocation for faster content delivery.

So, in addition to contributing to last-mile latency and network congestion efforts to improve QoE for users, we need more local IXPs. IXPs are physical infrastructure through which CDNs and ISPs exchange internet traffic between their networks. Think of them as the building blocks around which the internet is built.

Bill Woodcock of Packet Clearing House explained how, where, why, and when an ISP should build IXPs (short answer: to get up to 45% of the traffic for free) at one of their recent workshops. Here are some main points from it:

• Speed times distance equals cost, so a fast local connection is also the cheapest connection (the shortest path costs the least money).
• There are four stages of internet network growth. The largest one connects to a few hundred IXPs out of the roughly 800 in the world, and there are two types of connections: peering (horizontal) and transit (vertical).
• Exchange points can be built for about $8,000 and the average time to repay that in terms of bandwidth savings is typically less than a month.
• Europe is the best place because the closest you are to exchange points is the fastest and cheapest bandwidth.
• The trick to making an exchange point perform really well is to have it cost as close to zero as possible (it’s unnecessary to have $8,000, but you can get an exchange going for a few hundred with a little Ethernet switch).

You can view it in full at the link.