SSL Reverse Proxy
Introduction
A proxy server is an intermediary server that forwards requests for content from clients to servers across a network. A SSL reverse proxy is a type of proxy server that controls Secure Sockets Layer (SSL) traffic to ensure secure transmission of data between clients and servers. It acts as an intermediary, performing SSL encryption and decryption between the client and the server. For client, it acts as a server. For server, it acts as a client.
wrk2 is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. In this guide it is used to measure the maximum requests per second (RPS) of the SSL reverse proxy on DUT node. NGINX is open source software for web serving, reverse proxying, caching, load balancing, media streaming, and more. In this guide one NGINX instance acts as reverse proxy, and another NGINX instance acts as HTTPS server.
This guide explains in detail on how to integrate wrk2 and NGINX with VPP’s host stack for SSL proxy cases. The integration is done via LD_PRELOAD which intercepts syscalls that are supposed to go into the kernel and reinjects them into VPP. Users can execute bundled scripts in dataplane-stack repo to quickly establish the SSL proxy cases or manually run the use cases by following detailed guidelines step by step.
Network Stack Layers
VPP’s host stack provides alternatives to kernel-based sockets so that applications can take full advantage of VPP’s high performance. It implements a clean slate TCP that supports vectorized packet processing and follows VPP’s highly scalable threading model. The implementation is RFC compliant and supports many high-speed TCP protocol features. VPP’s host stack also provides a pluggable transport session layer that abstracts the interaction between applications and transports using a custom-built shared memory infrastructure. There is also VPP Comms Library (VCL) included to ease the consumability of the stack from application perspective. VCL manages the interaction with the session layer, abstracts session to integer session handles and exposes its own asynchronous communication functions.
This guide demonstrates two kinds of SSL reverse proxy connection:
Loopback connection on DUT node
Ethernet connection between DUT and client/server nodes
Loopback Connection
The loopback interface is a software virtual interface that is always up and available after it has been configured. In this setup, NGINX HTTPS server, NGINX reverse proxy and wrk2 client run over VPP’s host stack on DUT and communicate with each other through VPP loopback interfaces.
Note
This setup requires four isolated cores. Cores 1-4 are assumed to be isolated in this guide. VPP, wrk2, NGINX HTTPS server and NGINX reverse proxy each require an isolated core.
Automated Execution
Quickly set up VPP & NGINX and test SSL reverse proxy case:
cd $NW_DS_WORKSPACE/dataplane-stack
./usecase/ssl_reverse_proxy/run_vpp.sh -l -c 1
./usecase/ssl_reverse_proxy/run_nginx_server.sh -l -c 2
./usecase/ssl_reverse_proxy/run_nginx_proxy.sh -l -c 3
./usecase/ssl_reverse_proxy/run_wrk2.sh -l -c 4
Note
You will be asked a series of questions in order to embed the information correctly in the certificate. Fill out the prompts appropriately.
Use
-h
to check scripts supported options.
If the case runs successfully, the measurement results will be printed:
Initialised 1 threads in 0 ms.
Running 1m test @ https://172.16.2.1:8089/1kb
1 threads and 12 connections
Thread calibration: mean lat.: 4989.234ms, rate sampling interval: 18006ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 35.07s 14.40s 1.00m 57.95%
Req/Sec 44.74k 112.00 44.86k 50.00%
2691248 requests in 1.00m, 3.23GB read
Requests/sec: 44854.12
Transfer/sec: 55.05MB
Stop VPP and NGINX:
./usecase/ssl_reverse_proxy/stop.sh
Manual Execution
Users can also set up VPP & NGINX and test SSL reverse proxy case step by step.
VPP Setup
Declare a variable to hold the CLI socket for VPP:
export sockfile="/run/vpp/cli.sock"
Run VPP as a daemon service on core 1 with session layer enabled.
cd $NW_DS_WORKSPACE/dataplane-stack/components/vpp/build-root/install-vpp-native/vpp/bin
sudo ./vpp unix {cli-listen ${sockfile}} cpu {main-core 1} tcp {cc-algo cubic} session {enable use-app-socket-api} plugins {plugin dpdk_plugin.so {disable}}
For more configuration parameters, refer to VPP configuration reference.
Create loopback interfaces and routes by following VPP commands:
sudo ./vppctl -s ${sockfile} create loopback interface
sudo ./vppctl -s ${sockfile} set interface state loop0 up
sudo ./vppctl -s ${sockfile} create loopback interface
sudo ./vppctl -s ${sockfile} set interface state loop1 up
sudo ./vppctl -s ${sockfile} create loopback interface
sudo ./vppctl -s ${sockfile} set interface state loop2 up
sudo ./vppctl -s ${sockfile} ip table add 1
sudo ./vppctl -s ${sockfile} set interface ip table loop0 1
sudo ./vppctl -s ${sockfile} ip table add 2
sudo ./vppctl -s ${sockfile} set interface ip table loop1 2
sudo ./vppctl -s ${sockfile} ip table add 3
sudo ./vppctl -s ${sockfile} set interface ip table loop2 3
sudo ./vppctl -s ${sockfile} set interface ip address loop0 172.16.1.1/24
sudo ./vppctl -s ${sockfile} set interface ip address loop1 172.16.2.1/24
sudo ./vppctl -s ${sockfile} set interface ip address loop2 172.16.3.1/24
sudo ./vppctl -s ${sockfile} app ns add id server secret 1234 if loop0
sudo ./vppctl -s ${sockfile} app ns add id proxy secret 1234 if loop1
sudo ./vppctl -s ${sockfile} app ns add id client secret 1234 if loop2
sudo ./vppctl -s ${sockfile} ip route add 172.16.1.1/32 table 2 via lookup in table 1
sudo ./vppctl -s ${sockfile} ip route add 172.16.3.1/32 table 2 via lookup in table 3
sudo ./vppctl -s ${sockfile} ip route add 172.16.2.1/32 table 1 via lookup in table 2
sudo ./vppctl -s ${sockfile} ip route add 172.16.2.1/32 table 3 via lookup in table 2
For more detailed usage on above commands, refer to the following links:
To explore more on VPP’s available commands, please review VPP CLI reference.
Declare a variable to hold the LD_PRELOAD
library for VCL:
export LDP_PATH="$NW_DS_WORKSPACE/dataplane-stack/components/vpp/build-root/install-vpp-native/vpp/lib/aarch64-linux-gnu/libvcl_ldpreload.so"
Note
Any use of ~
in LDP_PATH
will expand to the root user’s home directory when used later in the guide.
NGINX Setup
VCL parameters can be configured through VCL configuration file. A VCL configuration sample for NGINX HTTPS server is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_server.conf
with the following contents:vcl { heapsize 64M segment-size 4000000000 add-segment-size 4000000000 rx-fifo-size 4000000 tx-fifo-size 4000000 namespace-id server namespace-secret 1234 app-scope-global app-socket-api /var/run/vpp/app_ns_sockets/server }
A VCL configuration sample for NGINX reverse proxy is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_proxy.conf
with the following contents:vcl { heapsize 64M segment-size 4000000000 add-segment-size 4000000000 rx-fifo-size 4000000 tx-fifo-size 4000000 namespace-id proxy namespace-secret 1234 app-scope-global app-socket-api /var/run/vpp/app_ns_sockets/proxy }
The above files configure VCL to request 4MB receive and transmit FIFO sizes and access to global session scope. They also provide the path to VPP’s session layer socket API for NGINX instances. Refer to VPP VCL reference for more usage information on VCL parameters.
Create SSL private keys and certificates for NGINX HTTPS server and reverse proxy:
sudo mkdir -p /etc/nginx/certs sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/certs/server.key -out /etc/nginx/certs/server.crt sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/certs/proxy.key -out /etc/nginx/certs/proxy.crt
Note
You will be asked a series of questions in order to embed the information correctly in the certificate. Fill out the prompts appropriately.
The way NGINX works is determined in the NGINX configuration file. A NGINX configuration sample for NGINX HTTPS server is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_server.conf
with the following contents:worker_processes 1; pid /run/nginx_server.pid; events {} http { sendfile on; tcp_nopush on; keepalive_requests 1000000000; default_type application/octet-stream; access_log off; error_log /dev/null crit; server { listen 8445 ssl; server_name $hostname; ssl_protocols TLSv1.3; ssl_prefer_server_ciphers on; ssl_certificate /etc/nginx/certs/server.crt; ssl_certificate_key /etc/nginx/certs/server.key; root /var/www/html; location / { try_files $uri $uri/ =404; } } }
A NGINX configuration sample for NGINX reverse proxy is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_proxy.conf
with the following contents:worker_processes 1; pid /run/nginx_proxy.pid; events {} http { sendfile on; tcp_nopush on; keepalive_requests 1000000000; default_type application/octet-stream; access_log off; error_log /dev/null crit; upstream ssl_file_server_com { server 172.16.1.1:8445; keepalive 1024; } server { listen 8089 ssl; server_name $hostname; ssl_protocols TLSv1.3; ssl_prefer_server_ciphers on; ssl_certificate /etc/nginx/certs/proxy.crt; ssl_certificate_key /etc/nginx/certs/proxy.key; location / { limit_except GET { deny all; } proxy_pass https://ssl_file_server_com; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_ssl_protocols TLSv1.3; } } }
For more detailed usage on above NGINX configuration, refer to the following links:
Create a 1kb file in NGINX HTTPS server root directory for downloading:
sudo mkdir -p /var/www/html sudo dd if=/dev/urandom of=/var/www/html/1kb bs=1024 count=1
VCL_CONFIG
provides VCL with a configuration file to read during startup. Start the NGINX HTTPS server on core 2 over VPP’s host stack, providing it with the VCL and NGINX configuration files:sudo taskset -c 2 sh -c "LD_PRELOAD=${LDP_PATH} \ VCL_CONFIG=$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_server.conf \ nginx -c $NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_server.conf"
Start the NGINX reverse proxy on core 3 over VPP’s host stack, providing it with the VCL and NGINX configuration files:
sudo taskset -c 3 sh -c "LD_PRELOAD=${LDP_PATH} \ VCL_CONFIG=$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_proxy.conf \ nginx -c $NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_proxy.conf"
To examine the NGINX sessions in VPP, use the command
sudo ./vppctl -s ${sockfile} show session verbose
. Here is a sample output for NGINX sessions:Connection State Rx-f Tx-f [0:0][T] 172.16.2.1:8089->0.0.0.0:0 LISTEN 0 0 [0:1][T] 172.16.1.1:8445->0.0.0.0:0 LISTEN 0 0 Thread 0: active sessions 2
Test
A VCL configuration sample for wrk2 HTTPS client is provided at $NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_wrk2.conf
with the following contents:
vcl {
heapsize 64M
segment-size 4000000000
add-segment-size 4000000000
rx-fifo-size 4000000
tx-fifo-size 4000000
namespace-id client
namespace-secret 1234
use-mq-eventfd
app-scope-global
app-socket-api /var/run/vpp/app_ns_sockets/client
}
Run wrk2 client on core 4 over VPP’s host stack to test SSL reverse proxy with 1kb file downloading:
cd $NW_DS_WORKSPACE/dataplane-stack/tools/traffic-gen/wrk2-aarch64
sudo taskset -c 4 sh -c "LD_PRELOAD=${LDP_PATH} \
VCL_CONFIG=$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_wrk2.conf \
./wrk --rate 100000000 -t 1 -c 12 -d 60s https://172.16.2.1:8089/1kb"
Note
Extremely high rate
--rate
is used to ensure throughput is measured.Number of connections
-c
is set to 12 to produce high throughput. For connection number less than 12, there are some socket connect or timeout errors which may block the test.Test duration
-d
is 60 seconds to get stable and repeatable results.URL is NGINX reverse proxy’s URL to be tested.
If both wrk2 and NGINX run successfully, wrk2 will output measurement result similar to the following:
Initialised 1 threads in 0 ms.
Running 1m test @ https://172.16.2.1:8089/1kb
1 threads and 12 connections
Thread calibration: mean lat.: 4989.234ms, rate sampling interval: 18006ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 35.07s 14.40s 1.00m 57.95%
Req/Sec 44.74k 112.00 44.86k 50.00%
2691248 requests in 1.00m, 3.23GB read
Requests/sec: 44854.12
Transfer/sec: 55.05MB
Stop
Kill VPP:
sudo pkill -9 vpp
Kill NGINX instances:
sudo pkill -9 nginx
Ethernet Connection
In this SSL reverse proxy scenario, NGINX HTTPS server, NGINX reverse proxy and wrk2 HTTPS client run on separated hardware platforms. The DUT has one NIC interface connected with the server node, and another NIC interface connected with the client node. NGINX reverse proxy runs over VPP’s host stack on DUT. NGINX HTTPS server runs over Linux kernel stack on server node. wrk2 HTTPS client runs over Linux kernel stack on client node.
To find out which DUT interfaces are connected with HTTPS client/server nodes,
sudo ethtool --identify <interface_name>
will typically blink a light on the
NIC to help identify the physical port associated with the interface.
Get interface names and PCIe addresses from lshw
command:
sudo lshw -c net -businfo
The output will look similar to:
Bus info Device Class Description
====================================================
pci@0000:07:00.0 eth0 network RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
pci@0001:01:00.0 enP1p1s0f0 network MT27800 Family [ConnectX-5]
pci@0001:01:00.1 enP1p1s0f1 network MT27800 Family [ConnectX-5]
In this setup example, enP1p1s0f0
at PCIe address 0001:01:00.0
is used to
connect with the client node. The IP address of this NIC interface in VPP is configured
as 172.16.2.1/24. enP1p1s0f1
at PCIe address 0001:01:00.1
is used to connect
with the server node. The IP address of this NIC interface in VPP is configured
as 172.16.1.2/24. The IP address of the client node NIC is 172.16.2.2/24. The IP address
of the server node NIC is 172.16.1.1/24. Both IP addresses can be configured by command ifconfig <interface_name> <ip_address>
.
Install git and building tools on the client node. If running Ubuntu 22.04 or later,
apt install git build-essential
will be sufficient. For other Linux distributions,
please consult the package manager for equivalent one.
Install NGINX on the server node. If running Ubuntu 22.04 or later, apt install nginx
will be sufficient. For other Linux distributions, please consult the package manager.
Follow Download Source Code to download dataplane-stack repo on the server node to
ease NGINX HTTPS server setup.
Automated Execution
Quickly set up VPP and NGINX reverse proxy on the DUT:
cd $NW_DS_WORKSPACE/dataplane-stack
./usecase/ssl_reverse_proxy/run_vpp.sh -p 0001:01:00.0,0001:01:00.1 -c 1
./usecase/ssl_reverse_proxy/run_nginx_proxy.sh -p -c 2
Note
Replace sample addresses in above command with desired PCIe addresses on DUT.
You will be asked a series of questions in order to embed the information correctly in the certificate. Fill out the prompts appropriately.
On the server node, start NGINX HTTPS server:
cd $NW_DS_WORKSPACE/dataplane-stack
./usecase/ssl_reverse_proxy/run_nginx_server.sh -p -c 1
Note
Core 1 is assumed to be isolated on the server node.
On the client node, download, build and run wrk2 to test SSL reverse proxy:
git clone https://github.com/giltene/wrk2.git
cd wrk2
make all
sudo taskset -c 1 ./wrk --rate 100000000 -t 1 -c 12 -d 60s https://172.16.2.1:8089/1kb
git clone https://github.com/AmpereTravis/wrk2-aarch64.git
cd wrk2-aarch64
make all
sudo taskset -c 1 ./wrk --rate 100000000 -t 1 -c 12 -d 60s https://172.16.2.1:8089/1kb
Note
Core 1 is assumed to be isolated on the client node.
If the case runs successfully, the measurement results will be printed by wrk2 client:
Initialised 1 threads in 0 ms.
Running 1m test @ https://172.16.2.1:8089/1kb
1 threads and 12 connections
Thread calibration: mean lat.: 5043.897ms, rate sampling interval: 18104ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 35.19s 14.41s 1.00m 57.64%
Req/Sec 35.97k 144.00 36.12k 50.00%
2165247 requests in 1.00m, 2.60GB read
Requests/sec: 36087.22
Transfer/sec: 44.29MB
Stop VPP and NGINX on DUT:
./usecase/ssl_reverse_proxy/stop.sh
Stop NGINX on server node:
./usecase/ssl_reverse_proxy/stop.sh
Manual Execution
Users can also set up VPP & NGINX and test SSL reverse proxy case step by step.
DUT VPP Setup
Declare a variable to hold the CLI socket for VPP:
export sockfile="/run/vpp/cli.sock"
Run VPP as a daemon service on core 1 with interface PCIe addresses and session layer enabled:
cd $NW_DS_WORKSPACE/dataplane-stack/components/vpp/build-root/install-vpp-native/vpp/bin
sudo ./vpp unix {cli-listen ${sockfile}} cpu {main-core 1} tcp {cc-algo cubic} dpdk {dev 0001:01:00.0 {name eth0} dev 0001:01:00.1 {name eth1}} session {enable use-app-socket-api}
Note
Replace sample addresses in above command with desired PCIe addresses on DUT.
Bring two VPP Ethernet interfaces up and set IP addresses:
sudo ./vppctl -s ${sockfile} set interface state eth0 up
sudo ./vppctl -s ${sockfile} set interface ip address eth0 172.16.2.1/24
sudo ./vppctl -s ${sockfile} set interface state eth1 up
sudo ./vppctl -s ${sockfile} set interface ip address eth1 172.16.1.2/24
Declare a variable to hold the VPP library for LD_PRELOAD
:
export LDP_PATH="$NW_DS_WORKSPACE/dataplane-stack/components/vpp/build-root/install-vpp-native/vpp/lib/aarch64-linux-gnu/libvcl_ldpreload.so"
Note
Any use of ~
in LDP_PATH
will expand to the root user’s home directory when used later in the guide.
DUT NGINX Setup
Create SSL private key and certificate for NGINX reverse proxy:
sudo mkdir -p /etc/nginx/certs sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/certs/proxy.key -out /etc/nginx/certs/proxy.crt
Note
You will be asked a series of questions in order to embed the information correctly in the certificate. Fill out the prompts appropriately.
A NGINX configuration sample for NGINX reverse proxy is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_proxy.conf
with these contents.Note
The NGINX HTTPS server
IP address:listening port
should be used as the server field in upstream section of above configuration file. For this setup example,172.16.1.1:8445
is used.A VCL configuration sample for NGINX reverse proxy is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_proxy_pn.conf
with the following contents:vcl { heapsize 64M segment-size 4000000000 add-segment-size 4000000000 rx-fifo-size 4000000 tx-fifo-size 4000000 app-socket-api /var/run/vpp/app_ns_sockets/default }
The above configures VCL to request 4MB receive and transmit FIFO sizes and provides the path to VPP’s session layer socket API. Refer to VPP VCL reference for more usage information on VCL parameters.
VCL_CONFIG
provides VCL with a configuration file to read during startup. Start the NGINX reverse proxy on core 2 over VPP’s host stack, providing it with the VCL and NGINX configuration files:sudo taskset -c 2 sh -c "LD_PRELOAD=${LDP_PATH} \ VCL_CONFIG=$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/vcl_nginx_proxy_pn.conf \ nginx -c $NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_proxy.conf"
To examine the NGINX proxy session in VPP, run the command
sudo ./vppctl -s ${sockfile} show session verbose
. Here is a sample output for NGINX proxy session:
Connection State Rx-f Tx-f [0:0][T] 0.0.0.0:8089->0.0.0.0:0 LISTEN 0 0 Thread 0: active sessions 1
NGINX Setup on Server Node
On the server node, create SSL private key and certificate for NGINX HTTPS server:
sudo mkdir -p /etc/nginx/certs sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/certs/server.key -out /etc/nginx/certs/server.crt
Note
You will be asked a series of questions in order to embed the information correctly in the certificate. Fill out the prompts appropriately.
Create a 1kb file in NGINX HTTPS server root directory for downloading:
sudo mkdir -p /var/www/html sudo dd if=/dev/urandom of=/var/www/html/1kb bs=1024 count=1
A NGINX configuration sample for NGINX HTTPS server is provided at
$NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_server.conf
with these contents.Start NGINX HTTPS server on core 1:
sudo taskset -c 1 nginx -c $NW_DS_WORKSPACE/dataplane-stack/usecase/ssl_reverse_proxy/nginx_server.conf
Note
Core 1 is assumed to be isolated on the server node.
Test
On the client node, download, build and run wrk2 to test SSL reverse proxy case:
git clone https://github.com/giltene/wrk2.git
cd wrk2
make all
sudo taskset -c 1 ./wrk --rate 100000000 -t 1 -c 12 -d 60s https://172.16.2.1:8089/1kb
git clone https://github.com/AmpereTravis/wrk2-aarch64.git
cd wrk2-aarch64
make all
sudo taskset -c 1 ./wrk --rate 100000000 -t 1 -c 12 -d 60s https://172.16.2.1:8089/1kb
Note
Core 1 is assumed to be isolated on the client node.
If both wrk2 and NGINX run successfully, wrk2 will output measurement result similar to the following:
Initialised 1 threads in 0 ms.
Running 1m test @ https://172.16.2.1:8089/1kb
1 threads and 12 connections
Thread calibration: mean lat.: 5043.897ms, rate sampling interval: 18104ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 35.19s 14.41s 1.00m 57.64%
Req/Sec 35.97k 144.00 36.12k 50.00%
2165247 requests in 1.00m, 2.60GB read
Requests/sec: 36087.22
Transfer/sec: 44.29MB
Stop
Kill VPP on DUT:
sudo pkill -9 vpp
Kill NGINX on DUT and server nodes:
sudo pkill -9 nginx