Quickstart Guide

Introduction

Welcome to the Dataplane Stack reference solution quickstart guide. This guide provides instructions on how to fetch the source code, build it and run sample applications.

By following the steps in this quickstart guide to the end, you will set up two user space programs for high-throughput packet forwarding. The programs accept packets from NIC port 0 and forward them out on the same NIC port based on their destination IP address.

Users and Essential Skills

The reference solutions are targeted for a networking software development or performance analysis engineer who has an in-depth networking knowledge, but does not know about Arm necessarily.

Mastering knowledge on certain user space open source networking projects, e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide and the reference solutions more easily.

Setup

The sample applications described in this guide require the following setup.

+------------------+                              +-------------------+
|                  |                              |                   |
|  Traffic         |                         +----|       DUT         |
|  Generator       |   Ethernet Connection   | N  |                   |
|                  |<----------------------->| I  |                   |
|                  |                 (Port 0)| C  |                   |
|                  |                         +----|                   |
+------------------+                              +-------------------+

As shown, the Device Under Test (DUT) should have at least one NIC connected to the traffic generator on port 0. The user can use any traffic generator. The DUT is also used to download the solution repository and build the code. Cross compilation is not supported currently.

Tested Platforms

The sample applications are tested on the following platforms.

DUT

  • Ampere Altra (Neoverse-N1)

    • Ubuntu 22.04.3 LTS (Jammy Jellyfish)

NIC

  • Mellanox ConnectX-6 Dx

    • OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0

    • Firmware version: 22.39.2048 (MT_0000000359)

  • Mellanox ConnectX-7

    • OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0

    • Firmware version: 28.38.1002 (MT_0000000834).

Note

To use a Mellanox NIC, install the OFED driver and update the NIC firmware by following the guidance in the FAQ.

Preparing the DUT

Requirements

The DUT needs a minimum hardware configuration as below.

  • Processor: Minimum 1 GHz and 5 CPU cores

  • Hard Drive: Minimum 32 GB

  • Memory (RAM): Minimum 8 GB

  • Network Interface Controller: Minimum 10G port connecting to Traffic Generator

  • Connection to internet to download the source code and dependent packages

This documentation assumes the user has installed Ubuntu 22.04 (Jammy) on the DUT.

  • Admin (root) privileges are required to run the software and set up the DUT.

  • Access to the internet is mandatory for downloading solution source code and installing all dependent packages and libraries.

  • Scripts are provided to install the dependent packages and libraries.

  • Mellanox OFED driver is installed and NIC firmware is updated.

  • GCC 9.4.0 or newer version is required to compile the software.

  • The provided scripts must be run in a bash shell.

The following utilities must be available on the DUT:

  • git

  • curl

  • python

  • python3

To configure Git, run:

git config --global user.email "[email protected]"
git config --global user.name "Your Name"

Follow the instructions provided in git-repo to install the repo tool manually.

Download Source Code

Create a new folder that will be the workspace, henceforth referred to and used in commands with the $NW_DS_WORKSPACE environment variable declared in these instructions:

export NW_DS_WORKSPACE=/path/to/workspace
mkdir -p $NW_DS_WORKSPACE
cd $NW_DS_WORKSPACE
export NW_DS_RELEASE=refs/tags/NW-DS-2024.03.29

Note

Sometimes new features and additional bug fixes are made available in the git repositories, but are not tagged yet as part of a release. To pick up these latest changes, remove the -b <release tag> option from the repo init command below. However, please be aware that such untagged changes may not be formally verified and should be considered unstable until they are tagged in an official release.

To clone the repository, run the following commands:

repo init \
    -u https://git.gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest.git \
    -b ${NW_DS_RELEASE} \
    -m dataplane-stack.xml
repo sync

Setup

This solution includes a setup.sh bash script which installs and upgrades the required packages. To use it:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo ./setup.sh

Note

The NGINX package is installed in setup.sh. NGINX systemd service will be started automatically, which spawns some NGINX worker processes. To stop the auto-started NGINX service, run sudo systemctl stop nginx. To disable the NGINX service from starting automatically on system boot, run sudo systemctl disable nginx.

Set the value of GRUB_CMDLINE_LINUX in /etc/default/grub to:

GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512 iommu.passthrough=1 isolcpus=1-4 rcu_nocbs=1-4 nohz_full=1-4 cpufreq.off=1 cpuidle.off=1 kpti=0"

If you already had some boot parameters append these to yours. You may need to tailor these parameters to your machine - read the main user guide for details on the parameters.

To restart the machine with new parameters, run:

sudo update-grub
sudo reboot

Build

This solution uses Makefile to build all the components.

The Makefile:

  • Builds DPDK and the L3fwd sample application

  • Builds VPP

  • Builds traffic generator tools, e.g., wrk2

To build Dataplane Stack, run the following on DUT:

cd $NW_DS_WORKSPACE/dataplane-stack
make all

Note

The compilation might take some time to complete.

Reboot

After setting up DUT and building the software, reboot the DUT. This ensures the setup changes are reflected before running the sample applications.

Get NIC Information

Identify the interface and PCIe address of the NIC port attached to the traffic generator. sudo ethtool --identify <interface name> will help identify which NIC port is associated with a given interface name. sudo lshw -c net -businfo will identify the PCIe address for the interface.

For example, if enP1p1s0f0 is attached to the traffic generator, then running lshw -c net -businfo will show the PCIe address as 0001:01:00.0:

$ sudo lshw -c net -businfo
Bus info          Device      Class      Description
====================================================
pci@0000:07:00.0  eth0        network    RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
pci@0001:01:00.0  enP1p1s0f0  network    MT27800 Family [ConnectX-5]
pci@0001:01:00.1  enP1p1s0f1  network    MT27800 Family [ConnectX-5]

DPDK L3fwd

Bind NIC to Proper Linux Driver

For NICs that support bifurcated drivers, like Mellanox NICs, please skip this step.

For other NICs to be used by DPDK, the NIC needs to be bound to the appropriate driver. In practice, vfio-pci driver is sufficient. Before using vfio-pci, be sure to load the kernel module with modprobe vfio-pci. For more information, review DPDK’s Linux Drivers Guide.

To bind the NIC to the appropriate driver, run:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo modprobe vfio-pci # ensure kernel module is loaded
sudo components/dpdk/usertools/dpdk-devbind.py -b vfio-pci <pcie_address>

For example, to bind PCIe address 0000:06:00.1 to vfio-pci:

sudo modprobe vfio-pci # ensure kernel module is loaded
sudo components/dpdk/usertools/dpdk-devbind.py -b vfio-pci 0000:06:00.1

Run

To run DPDK L3fwd application:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo components/dpdk/build/examples/dpdk-l3fwd -n 4 -l 2 -a <pcie_address> -- -P -p 0x1 --config='(0,0,2)'

For example, run dpdk-l3fwd with 0001:01:00.0:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo components/dpdk/build/examples/dpdk-l3fwd -n 4 -l 2 -a 0001:01:00.0 -- -P -p 0x1 --config='(0,0,2)'

Test

For example, the typical output contains:

Initializing port 0 ... Creating queues: nb_rxq=1 nb_txq=1...
Address:98:03:9B:71:24:2E, Destination:02:00:00:00:00:00, Allocated mbuf pool on socket 0
LPM: Adding route 198.18.0.0 / 24 (0) [0001:01:00.0]
LPM: Adding route 2001:200:: / 64 (0) [0001:01:00.0]
txq=2,0,0

These logs show port 0 has MAC address 98:03:9B:71:24:2E with PCIe address 0001:01:00.0 on the DUT. 1 IPv4 route matching the subnet 198.18.0.0/24 is added.

Configure the traffic generator to send packets to the NIC port, using the MAC and IP address displayed in the logs. In this example, use a destination MAC address of 98:03:9B:71:24:2E and a destination IP of 198.18.0.21. Then, dpdk-l3fwd will forward those packets out on port 0.

Stop

Stop the dpdk-l3fwd process with Control-C or kill. If the NIC had been bound to a different Linux driver, rebind it to its original driver. Find the original driver by running dpdk-devbind.py -s, and notice the unused= part of the PCIe address.

For example, sample output from dpdk-devbind.py -s may look like:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo components/dpdk/usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:07:00.0 'Ethernet Controller XL710 for 40GbE QSFP+ 1583' drv=vfio-pci unused=i40e
...

In this example, bind 0000:07:00.0 to the i40e Linux driver with the following command:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo components/dpdk/usertools/dpdk-devbind.py -b i40e 0000:07:00.0

VPP L3fwd

Run

Run VPP using PCIe address 0001:01:00.0 obtained from above lshw command and alias the interface name as eth0 for short:

cd $NW_DS_WORKSPACE/dataplane-stack
sudo ./components/vpp/build-root/install-vpp-native/vpp/bin/vpp unix {interactive} dpdk { dev 0001:01:00.0 { name eth0 } }

Note

It is possible that VPP may throw warnings and errors during initialization. These can be ignored safely.

Configure VPP with L3 interface and routes in VPP command prompt

vpp# set interface mac address eth0 00:11:22:33:44:55
vpp# set interface ip address eth0 1.1.1.2/24
vpp# set interface state eth0 up
vpp# ip route add 198.18.0.0/24 via 1.1.1.1
vpp# set ip neighbor eth0 1.1.1.1 02:00:00:00:00:00

Test

After running the above command, configure the traffic generator to send packets to port 0 with a destination MAC address of 00:11:22:33:44:55 and an IP in the subnet 198.18.0.0/24. vpp will forward those packets out on port 0.

Stop

To stop VPP, enter quit in VPP command line prompt:

vpp# quit

Changelog & Known Issues

To check newly added features, feature changes, and known issues in each of the releases, please refer to CHANGELOG.