User Guide
Introduction
Welcome to the Dataplane Stack reference solution user guide. This guide provides the detailed guidelines to end users on how to download, build and execute the solution on Arm platforms. Guidelines on experimenting with various parameters that affect performance of the use cases are described in detail. This guide is intended to describe complex and practical use cases which require complex test setup.
Users and Essential Skills
The reference solutions are targeted for a networking software development or performance analysis engineer who has an in-depth networking knowledge, but does not know about Arm necessarily.
Familiarity with user-space open-source networking projects, e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide and the reference solutions more easily.
Setup
Most sample applications described in this guide require the following setup.
+---------------+ +-----------+
| | +----| |
| Traffic | Ethernet Connection | N | DUT |
| Generator |<----------------------->| I | |
| | (Port 0)| C | |
| | +----| |
+---------------+ +-----------+
Some will work with only one machine, though use of virtual interfaces.
As shown, the Device Under Test (DUT) should have at least one NIC connected to the traffic generator on port 0. The user can use any traffic generator. The DUT is also used to download the solution repository and build the code. Cross compilation is not supported currently.
Tested Platforms
The sample applications are tested on the following platforms.
DUT
Ampere Altra (Neoverse-N1)
Ubuntu 22.04.3 LTS (Jammy Jellyfish)
NIC
-
OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
Firmware version: 22.39.2048 (MT_0000000359)
-
OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
Firmware version: 28.38.1002 (MT_0000000834)
Note
To use a Mellanox NIC, install the OFED driver and update the NIC firmware by following the guidance in the FAQ.
Preparing the DUT
Requirements
The DUT needs a minimum hardware configuration as below.
Processor: Minimum 1 GHz and 5 CPU cores
Hard Drive: Minimum 32 GB
Memory (RAM): Minimum 8 GB
Network Interface Controller: Minimum 10G port connecting to Traffic Generator
Connection to internet to download the source code and dependent packages
This documentation assumes the user has installed Ubuntu 22.04 (Jammy) on the DUT.
Admin (root) privileges are required to run the software and set up the DUT.
Access to the internet is mandatory for downloading the solution’s source code and installing all the dependent packages and libraries.
Scripts are provided to install the dependent packages and libraries.
Mellanox OFED driver is installed and the NIC firmware is updated.
GCC 9.4.0 or newer version is required to compile the software.
The provided scripts must be run in a bash shell.
To configure Git, run:
git config --global user.email "[email protected]"
git config --global user.name "Your Name"
Download Source Code
Create a new folder that will be the workspace, henceforth referred to
and used in commands with the $NW_DS_WORKSPACE environment variable
declared in these instructions:
export NW_DS_WORKSPACE=/path/to/workspace
mkdir -p $NW_DS_WORKSPACE
cd $NW_DS_WORKSPACE
export NW_DS_RELEASE=refs/tags/NW-DS-2025.06.30
To clone the repository, run the following commands:
git clone [email protected]:arm-reference-solutions/dataplane-stack.git
cd dataplane-stack
git checkout ${NW_DS_RELEASE}
git submodule update --init
Note
refs/tags/NW-DS-2025.06.30 is illustrative, please check for the latest tag.
Tags prior to NW-DS-2025.06.30 should follow instructions in previous README versions.
Using other versions of Dataplane Stack
The latest features and bug fixes are made available in the git repository, but may not be tagged as part of a release. By default, we recommend checking out a tagged release using to ensure stability.
However, if you want to try the most recent changes and bugfixes, you can checkout the main branch. Please keep in mind that this version may not have been fully tested or officially validated, and could be unstable. Use it at your own discretion.
Setup
This solution includes a setup.sh bash script which installs and upgrades the
required packages. To use it:
cd $NW_DS_WORKSPACE/dataplane-stack
sudo ./setup.sh
How to stop NGINX starting automatically?
The NGINX package is installed in setup.sh. NGINX systemd service will be started automatically, which spawns some NGINX worker processes.
To stop NGINX service, run:
sudo systemctl stop nginx
To disable the NGINX service from starting automatically on system boot, run:
sudo systemctl disable nginx
Additionally, some platform level parameters are required to run the applications. These are set by grub at boot time.
To edit them you must edit /etc/default/grub as root. This file should have a
line specifying the GRUB command line like this:
GRUB_CMDLINE_LINUX=""
This may already contain some parameters or be empty. If it’s missing add it. An example value of GRUB_CMDLINE_LINUX could be:
GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512 iommu.passthrough=1 isolcpus=1-4 rcu_nocbs=1-4 nohz_full=1-4 cpufreq.off=1 cpuidle.off=1 kpti=0"
The string specifies the kernel command line parameters. Alter it according to the guide below.
Parameters required by dataplane stack:
reserve one 1G hugepage (via
hugepagesz=1G hugepages=1)reserve 512 2M hugepages (via
hugepagesz=2M hugepages=512)set IOMMU into passthrough mode (via
iommu.passthrough=1)isolate CPUs 1-4 from the Linux scheduler (via
isolcpus=1-4)
Optional parameters for better performance:
isolate CPUs 1-4 from processing RCU callbacks (via
rcu_nocbs=1-4)set CPUs 1-4 to omit scheduling clock ticks (via
nohz_full=1-4)disable cpufreq and cpuidle subsystems (via
cpufreq.off=1andcpuidle.off=1)disable control page table isolation of user and kernel address spaces (via
kpti=0)
These options might need to be tailored to your specific machine. Make sure the selection of isolated CPUs is correct. Keep in mind that the isolated CPUs will not be available for other tasks (like compilation).
Why reserve hugepages?
Hugepages help prevent TLB misses and are commonly used by networking applications to manage memory. The larger the hugepage, the greater the performance increase due to fewer TLB misses and fewer page table walks. 1GB hugepages are easiest to reserve at boot time, as reservation during run time is likely to not be possible. Run time reservation would require 1GB of available contiguous memory, which is typically not available.
Why isolate CPUs?
The combination of the kernel parameters mentioned above ensure that the CPUs run only the desired application. They never process kernel RCU callbacks, don't generate scheduling ticks as there is only one process running on them, and are isolated from running any processes other than the ones pinned to them (the desired application).
Reboot
After finishing setup, reboot the DUT. This ensures the setup changes are reflected before running the sample applications. Run:
sudo update-grub
sudo reboot
Build
This solution uses Makefile to build all the components. To build the Dataplane Stack, run the following on the DUT:
cd $NW_DS_WORKSPACE/dataplane-stack
make all
It is also possible to compile the components individually by specifying make dpdk or make vpp.
Run make help to view a list of all Makefile targets.
The above mentioned make commands can also be used to rebuild after modifying the code.
Troubleshooting
If during link time you run into a missing symbols error:
plugin/load [error ]: .[...]/dpdk_plugin.so:
undefined symbol: armv8_dec_aes_cbc_sha1_128
This is fixed by modifying the /components/vpp/build/external/packages/dpdk.mk file
and adding crypto/armv8 to the list of DPDK_DRIVERS_DISABLED.Running Use Cases
Please follow the documentation for each individual use case.
The first two on the list are good ones to start with.
Get NIC Information
If you have multiple NIC ports you may connect PCIe addresses with physical ports
by first using ethtool which can blink the light associated with the interface:
sudo ethtool --identify <interface name>
And then listing the PCIe addresses of the interfaces:
sudo lshw -c net -businfo
Example
For example, if enP1p1s0f0 is attached to the traffic generator, then running the command will show the PCIe address (0001:01:00.0).
$ sudo lshw -c net -businfo
Bus info Device Class Description
====================================================
pci@0001:01:00.0 enP1p1s0f0 network MT27800 Family [ConnectX-5]
pci@0001:01:00.1 enP1p1s0f1 network MT27800 Family [ConnectX-5]
Porting/Integrating to another Arm platform
Although the solution is tested on limited hardware platforms, the solution might work just fine on other Arm platforms. However, such platforms should support Armv8 architecture at least and should be supported by the underlying components.
Changelog & Known Issues
To check newly added features, feature changes, and known issues in each of the releases, please refer to CHANGELOG.