User Guide

Welcome to the Dataplane Stack reference solution user guide. This guide provides the detailed guidelines to end users on how to download, build and execute the solution on Arm platforms. Guidelines on experimenting with various parameters that affect performance of the use cases are described in detail. This guide is intended to describe complex and practical use cases which require complex test setup.

Users and Essential Skills

The reference solutions are targeted for a networking software development or performance analysis engineer who has in-depth networking knowledge, but does not know about Arm necessarily.

Using this guide requires in-depth knowledge on networking use cases. Mastering knowledge on certain user space open source networking projects, e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide and the reference solutions more easily.

Tested Platforms

The sample applications are tested on the following platforms.

DUT

  • Ampere Altra (Neoverse-N1)

    • Ubuntu 22.04.3 LTS (Jammy Jellyfish)

NIC

  • Mellanox ConnectX-6 Dx

    • OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0

    • Firmware version: 22.39.2048 (MT_0000000359)

  • Mellanox ConnectX-7

    • OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0

    • Firmware version: 28.38.1002 (MT_0000000834).

Note

To use a Mellanox NIC, install the OFED driver and update the NIC firmware by following the guidance in the FAQ.

Preparing the DUT

The DUT is also used to download the solution repository and build the code. Cross compilation is not supported currently.

Requirements

The DUT needs to have a minimum hardware configuration as below.

  • Processor: Minimum 1 GHz and 5 CPU cores

  • Hard Drive: Minimum 32 GB

  • Memory (RAM): Minimum 8 GB

  • Network Interface Controller: Minimum 10G port connecting to Traffic Generator

  • Ethernet connection to internet

This documentation assumes the user has installed Ubuntu 22.04 (Jammy) on the DUT.

  • Admin (root) privileges are required to run the software and set up the DUT.

  • Access to the internet is mandatory for downloading the solution’s source code and installing all the dependent packages and libraries.

  • Scripts are provided to install the dependent packages and libraries.

  • Mellanox OFED driver is installed and the NIC firmware is updated.

  • gcc 9.4.0 or newer version is required to compile the software.

  • The provided scripts must be run in a bash shell.

The following utilities must be available on the DUT:

  • git

  • curl

  • python

  • python3

To configure Git, run:

git config --global user.email "[email protected]"
git config --global user.name "Your Name"

Follow the instructions provided in git-repo to install the repo tool manually.

Download Source Code

Create a new folder that will be the workspace, henceforth referred to and used in commands with the $NW_DS_WORKSPACE environment variable declared in these instructions:

export NW_DS_WORKSPACE=/path/to/workspace
mkdir -p $NW_DS_WORKSPACE
cd $NW_DS_WORKSPACE
export NW_DS_RELEASE=refs/tags/NW-DS-2024.03.29

Note

Sometimes new features and additional bug fixes will be made available in the git repositories and have not yet been tagged as part of a release. To pick up these latest changes, remove the -b <release tag> option from the repo init command below. However, please be aware that such untagged changes have not yet been formally verified and should be considered unstable until they are tagged in an official release.

To clone the repository, run the following commands:

repo init \
    -u https://git.gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest.git \
    -b ${NW_DS_RELEASE} \
    -m dataplane-stack.xml
repo sync

Setup

This solution includes a setup.sh Bash script responsible for installing and upgrading the following packages:

net-tools build-essential manpages-dev libnuma-dev python python3-venv cmake meson pkg-config python3-pyelftools lshw util-linux iperf3 nginx

To run it:

cd <nw_ds_workspace>/dataplane-stack
sudo ./setup.sh

Note

NGINX package is installed in setup.sh. NGINX systemd service will be started automatically, which spawns some NGINX worker processes. To stop the auto-started NGINX service, run sudo systemctl stop nginx. To disable the NGINX service from starting automatically on system boot, run sudo systemctl disable nginx.

Additionally, some platform level parameters are required to run the applications. These are set by grub at boot time.

To edit them you must edit /etc/default/grub as root. This file should have a line specifying the GRUB command line like this:

GRUB_CMDLINE_LINUX=""

This may already contain some parameters or be empty. If it’s missing add it. The string specifies the kernel command line parameters. Alter it according to the guide below.

Parameters required by dataplane stack:

  • reserve one 1G hugepage (via hugepagesz=1G hugepages=1)

  • reserve 512 2M hugepages (via hugepagesz=2M hugepages=512)

  • set IOMMU into passthrough mode (via iommu.passthrough=1)

  • isolate CPUs 1-4 from the Linux scheduler (via isolcpus=1-4)

Optional parameters for better performance:

  • isolate CPUs 1-4 from processing RCU callbacks (via rcu_nocbs=1-4)

  • set CPUs 1-4 to omit scheduling clock ticks (via nohz_full=1-4)

  • disable cpufreq and cpuidle subsystems (via cpufreq.off=1 and cpuidle.off=1)

  • disable control page table isolation of user and kernel address spaces (via kpti=0)

These options might need to be tailored to your specific machine. Make sure the selection of isolated CPUs is correct. Keep in mind that the isolated CPUs will not be available for other tasks (like compilation).

An example complete line using all these parameters would look like this:

GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512 iommu.passthrough=1 isolcpus=1-4 rcu_nocbs=1-4 nohz_full=1-4 cpufreq.off=1 cpuidle.off=1 kpti=0"

After saving the /etc/default/grub file, to restart the machine with new parameters, run:

sudo update-grub
sudo reboot
Why reserve hugepages?
Hugepages help prevent TLB misses and are commonly used by networking
applications to manage memory. The larger the hugepage, the greater the
performance increase due to fewer TLB misses and fewer page table walks.

1GB hugepages are easiest to reserve at boot time, as reservation during
run time is likely to not be possible. Run time reservation would require
1GB of available contiguous memory, which is typically not available.

Why isolate CPUs?
The combination of the kernel parameters mentioned above ensure that the
CPUs run only the desired application. They never process kernel RCU
callbacks, don't generate scheduling ticks as there is only one process
running on them, and are isolated from running any processes other than
the ones pinned to them (the desired application).

Build

This solution uses Makefile to build all the components.

The Makefile:

  • Builds DPDK and the L3fwd sample applications

  • Builds VPP

  • Builds traffic generator tools, e.g., wrk2

To build the Dataplane Stack, run the following on the DUT:

cd $NW_DS_WORKSPACE/dataplane-stack
make all

It is also possible to compile the components individually by specifying make dpdk or make vpp. Run make help to view a list of all Makefile targets.

The above mentioned make commands can also be used to rebuild after modifying the code.

Reboot

After setting up the DUT and building the software, reboot the DUT. This ensures the setup changes are reflected before running the sample applications.

Get the NIC Interface

Next, identify the interface(s) on the NIC(s) connected to the traffic generator.

How to find the right interfaces?
$ sudo ethtool --identify <interface>
will typically blink a light on the NIC to help identify the
physical port associated with the interface.

Porting/Integrating to another Arm platform

Although the solution is tested on limited hardware platforms, the solution might work just fine on other Arm platforms. However, such platforms should support Armv8 architecture at least and should be supported by the underlying components.

Use Cases