User Guide¶
Welcome to the Dataplane Stack reference solution user guide. This guide provides the detailed guidelines to end users on how to download, build and execute the solution on Arm platforms. Guidelines on experimenting with various parameters that affect performance of the use cases are described in detail. This guide is intended to describe complex and practical use cases which require complex test setup.
Users and Essential Skills¶
The reference solutions are targeted for a networking software development or performance analysis engineer who has in-depth networking knowledge, but does not know about Arm necessarily.
Using this guide requires in-depth knowledge on networking use cases. Mastering knowledge on certain user space open source networking projects, e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide and the reference solutions more easily.
Tested Platforms¶
The sample applications are tested on the following platforms.
DUT¶
Ampere Altra (Neoverse-N1)
Ubuntu 20.04.3 LTS (Focal Fossa)
Preparing the DUT¶
The DUT is also used to download the solution repository and build the code. Cross compilation is not supported currently.
Requirements¶
The DUT needs to have a minimum hardware configuration as below.
Processor: Minimum 1 GHz and 5 CPU cores
Hard Drive: Minimum 32 GB
Memory (RAM): Minimum 8 GB
Network Interface Controller: Minimum 10G port connecting to Traffic Generator
Ethernet connection to internet
This documentation assumes the user has installed Ubuntu 20.04 (Focal) on the DUT.
Admin (root) privileges are required to run the software and set up the DUT.
Access to the internet is mandatory for downloading solution source code and installing all dependent packages and libraries.
Scripts are provided to install the dependent packages and libraries.
Mellanox OFED driver is installed and NIC firmware is updated.
gcc 9.4.0 or newer version is required to compile the software.
The provided scripts must be run in a bash shell.
- The following utilities must be available on the DUT:
git
curl
python
python3
To configure Git, run:
git config --global user.email "[email protected]"
git config --global user.name "Your Name"
Follow the instructions provided in
git-repo to install the
repo tool manually.
Download Source Code¶
Create a new folder that will be the workspace, henceforth referred to as
<nw_ds_workspace> in these instructions:
mkdir <nw_ds_workspace>
cd <nw_ds_workspace>
export NW_DS_RELEASE=refs/tags/NW-DS-2023.05.31-RC0
Note
Sometimes new features and additional bug fixes will be made available in
the git repositories and will not yet have been tagged as part of a
release. To pick up these latest changes, remove the
-b <release tag> option from the repo init command below. However,
please be aware that such untagged changes have not yet been formally
verified and should be considered unstable until they are tagged in an
official release.
To clone the repository, run the following commands:
repo init \
-u https://git.gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest.git \
-b ${NW_DS_RELEASE} \
-m dataplane-stack.xml
repo sync
Setup¶
This solution includes a setup.sh Bash script responsible for the setup
process.
The setup script:
Installs and upgrades the following packages
net-tools build-essential manpages-dev libnuma-dev python python3-venv cmake meson pkg-config python3-pyelftools lshw util-linux iperf3 nginx
Configures platform level parameters required to run the applications
Specifically, it overwrites GRUB_CMDLINE_LINUX in /etc/default/grub to
reserve one 1G hugepage (via
hugepagesz=1G hugepages=1)reserve 512 2M hugepages (via
hugepagesz=2M hugepages=512)set IOMMU into passthrough mode (via
iommu.passthrough=1)isolate CPUs 1-4 from the Linux scheduler (via
isolcpus=1-4)isolate CPUs 1-4 from processing RCU callbacks (via
rcu_nocbs=1-4)set CPUs 1-4 to omit scheduling clock ticks (via
nohz_full=1-4)disable cpufreq and cpuidle subsystems (via
cpufreq.off=1andcpuidle.off=1)
To isolate more or different CPUs, edit setup.sh accordingly. To help speedup
compilation, ensure that there are sufficient number of CPUs (at least 4 if possible)
that are not isolated.
Why reserve hugepages?
Hugepages help prevent TLB misses and are commonly used by networking
applications to manage memory. The larger the hugepage, the greater the
performance increase due to fewer TLB misses and fewer page table walks.
1GB hugepages are easiest to reserve at boot time, as reservation during
run time is likely to not be possible. Run time reservation would require
1GB of available contiguous memory, which typically is not available.
Why isolate CPUs?
The combination of the kernel parameters mentioned above ensure that the
CPUs run only the desired application. They never process kernel RCU
callbacks, don't generate schedling ticks as there is only one process
running on them, and are isolated from running any processes other than
the ones pinned to them (the desired application).
To set up the DUT:
cd <nw_ds_workspace>/dataplane-stack
sudo ./setup.sh
Note
NGINX package is installed in setup.sh. NGINX systemd service will be started
automatically, which spawns some NGINX worker processes. To stop the
auto-started NGINX service, run sudo systemctl stop nginx. To disable the
NGINX service from starting automatically on system boot, run sudo systemctl disable nginx.
Build¶
This solution uses Makefile to build all the components.
The Makefile:
Builds DPDK and the L3fwd sample applications
Builds VPP
Builds traffic generator tools, e.g., wrk2
To build Dataplane Stack, run the following on DUT:
cd <nw_ds_workspace>/dataplane-stack
make all
It is also possible to compile the components individually by specifying make dpdk or make vpp.
Run make help to view a list of all Makefile targets.
The above mentioned make commands can also be used to rebuild after modifying the code.
Reboot¶
After setting up DUT and building the software, reboot the DUT. This ensures the setup changes are reflected before running the sample applications.
Get NIC Interface¶
Next, identify the interface(s) on the NIC(s) connected to the traffic generator.
How to find the right interfaces?
$ sudo ethtool --identify <interface>
will typically blink a light on the NIC to help identify the
physical port associated with the interface.
Porting/Integrating to another Arm platform¶
Although the solution is tested on limited hardware platforms, the solution might work just fine on other Arm platforms. However, such platforms should support Armv8 architecture at least and should be supported by the underlying components.