I have been digging into how Kubernetes uses networking on Linux and came across Linux virtual server (LVS). LVS is a kernel-level feature that provides a virtual, network front end. The tool to manage LVS is ipvsadm.

The tool is usually not available by default and must be installed. It is possible to use yum with previous versions of Amazon Linux, but while Amazon Linux 2023 has LVS built into the kernel (check by running ls -la /proc/net/ip_vs), the “userspace interface” (I’m seeing this term becoming more pervasive in Linux and Kubernetes docs. Wasn’t it simply called a command, back in the day?) is not installed and is not (yet) part of the dnf (which has replaced yum) repository. There also appears to be no pre-compiled executable for this utility. Therefore it has to be compiled. Unfortunately, the project is scattered in a few different sites: the source package repository, the git repository and the wiki page (which, incredibly in 2023, has no https). In order to build the executable, there are several other components that need to be installed. The recipe to install ipvsadm is:

wget https://mirrors.edge.kernel.org/pub/linux/utils/kernel/ipvsadm/ipvsadm-1.31.tar.gz
tar xf ipvsadm-1.31.tar.gz
cd ipvsadm-1.31
dnf install make gcc libnl* popt*
make
install ipvsadm /usr/sbin/ipvsadmCode language: Bash (bash)

Let’s use LVS in a simple example to see how it works. To try this example you will need to install Flask (pip install flask). Here’s a simple Flask app:

from flask import Flask
import os
app = Flask(__name__)

@app.route('/')
def hello_world():
    port = os.environ['FLASK_RUN_PORT']
    return f'Hello, {port}!'Code language: Python (python)

When the app receives an HTTP request, it will respond with the Hello message, including the port. Save this to a file called hello.py. Let’s launch two web apps on different ports:

$ FLASK_APP=hello.py
$ FLASK_RUN_PORT=8001 flask run &
$ FLASK_RUN_PORT=8002 flask run &

Test out that they are working with curl:

$ curl http://localhost:8001
Hello, 8001!
$ curl http://localhost:8002
Hello, 8002!Code language: JavaScript (javascript)

Next, set up ipvsadm. From here on, you need to be root or prefix the commands with sudo. First, define the virtual server:

$ ipvsadm -A -t 127.0.0.1:8000 -s rr

This tells LVS to create a virtual server that uses TCP (rather than UDP) on IP 127.0.0.1 and port 8000. It also specifies the load balancing algorithm (-s for scheduler) should be round-robin. Next add the two real servers:

$ ipvsadm -a -t 127.0.0.1:8000 -r 127.0.0.1:8001 -m
$ ipvsadm -a -t 127.0.0.1:8000 -r 127.0.0.1:8002 -m

Similar to the first command, this is adding a virtual server using TCP on 127.0.0.1:8000 (-t 127.0.0.1:8000), and real servers at 127.0.0.1:8001 (-r 127.0.0.1:8001) and 127.0.0.1:8002 (-r 127.0.0.1:8002). It also tells LVS to use NAT (-m for masquerading) to communicate between the front end and back end. Now test LVS using the front-end address (port 8000):

$ curl http://localhost:8000
Hello, 8001!
$ curl http://localhost:8000
Hello, 8002!Code language: JavaScript (javascript)

The request cycles between the two instances because of the round-robin scheduler.

Certainly, we can achieve the same result by setting up a reverse proxy such as nginx. While LVS uses some different terminology, such as virtual server and real server, the net effect is the same. The IPVS virtual server is the proxy, and the real servers are the upstream. HAProxy would also do the same job. However, the interesting aspect of LVS is that it operates in the kernel at the OSI layer 4 (transport) level, while nginx and others operate at layer 7 (application level). Thus LVS can route traffic based on IP address and port for any type of traffic (SSH, FTP, or your own proprietary protocol), but not based on HTTP URL components such as path or parameters like a web proxy can do. Nginx can be run on any Linux instance, but IPVS can only work if the kernel has LVS support built-in.

I noticed that the AWS application load balancer (ALB) is actually Nginx. Learning about LVS makes me wonder whether the network load balancer (NLB) is a linux instance using LVS.