1 What is MQ-ECN

MQ-ECN is a new ECN marking scheme to enable ECN for multi-service multi-queue production data center networks. MQ-ECN can achieve high throughput, low latency and weighted fair sharing simultaneously.

For more details about MQ-ECN, please refer to our NSDI 2016 paper.

2 How to use

2.1 Compiling

MQ-ECN software prototype is implemented as a Linux queuing discipline (qdisc) kernel module, running on multi-NIC servers to emulate switch hardware behaviors. So you need the kernel headers to compile it. You can find available headers on your system in /lib/modules. To install the kernel headers, you just need to use the following command：

$ apt-get install linux-headers-$(uname -r)

Then you can compile MQ-ECN kernel module:

$ cd sch_dwrr
$ make

This will produce a kernel module called sch_dwrr.ko. I have tested it with Linux kernel 3.18.11. MQ-ECN kernel module is built on the top of Token Bucket Filter (tbf) and Deficit Round Robin (drr) scheduler in Linux kernel.

2.2 Installing

MQ-ECN replaces token bucket rate limiter module. Hence, you need to remove sch_tbf before installing MQ-ECN. To install MQ-ECN on a device (e.g., eth1):

$ rmmod sch_tbf
$ insmod sch_dwrr.ko
$ tc qdisc add dev eth1 root tbf rate 995mbit limit 1000k burst 1000k mtu 66000 peakrate 1000mbit

To remove MQ-ECN on a device (e.g., eth1):

$ tc qdisc del dev eth1 root
$ rmmod sch_dwrr

In above example, we install MQ-ECN on eth1. The shaping rate is 995Mbps (line rate is 1000Mbps). To accurately reflect switch buffer occupancy, we usually trade a little bandwidth.

2.3 Note

To better emulate real switch hardware behaviors, we should avoid large segments on server-emulated software switches. Hence, we need to disable related offloading techniques on all involved NICs. For example, to disable offloading on eth0:

$ ethtool -K eth0 tso off
$ ethtool -K eth0 gso off
$ ethtool -K eth0 gro off

2.4 Configuring

Except for shaping rate, all the parameters of MQ-ECN are configured through sysctl interfaces. Here, I only show several important parameters. For the rest, see params.h and params.c for more details.

• ECN marking scheme:
  

  $ sysctl dwrr.ecn_scheme
  

  
  To enable per-queue ECN marking:
  

  $ sysctl -w dwrr.ecn_scheme=1
  

  
  To enable per-port ECN marking:
  

  $ sysctl -w dwrr.ecn_scheme=2
  

  
  To enable MQ-ECN:
  

  $ sysctl -w dwrr.ecn_scheme=3
  

  
  



• Per-port ECN marking threshold (bytes):
  

  $ sysctl dwrr.port_thresh_bytes
  

  
  



• Per-queue ECN marking threshold (bytes) (i is queue index):
  

  $ sysctl dwrr.queue_thresh_bytes_i
  

  
  



• Per-port shared buffer size (bytes):
  

  $ sysctl dwrr.shared_buffer_bytes
  

  
  

2.5 WRR

By default, MQ-ECN kernel module performs Deficit Weighted Round Robin (DWRR) scheduling algorithm. You can also enable Weighted Round Robin (WRR) as follows:

$ sysctl -w dwrr.enable_wrr=1