..
  # Copyright (c) 2022-2025, Arm Limited.
  #
  # SPDX-License-Identifier: Apache-2.0

##########
User Guide
##########

************
Introduction
************

Welcome to the Dataplane Stack reference solution user guide. This guide provides
the detailed guidelines to end users on how to download, build and execute the
solution on Arm platforms. Guidelines on experimenting with various parameters
that affect performance of the use cases are described in detail. This guide
is intended to describe complex and practical use cases which require complex
test setup.

Users and Essential Skills
==========================

The reference solutions are targeted for a networking software development or
performance analysis engineer who has an in-depth networking knowledge, but does
not know about Arm necessarily.

Familiarity with user-space open-source networking projects,
e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide
and the reference solutions more easily.

*****
Setup
*****

Most sample applications described in this guide require the following setup.

.. code-block:: none

    +---------------+                              +-----------+
    |               |                         +----|           |
    |    Traffic    |   Ethernet Connection   | N  |    DUT    |
    |   Generator   |<----------------------->| I  |           |
    |               |                 (Port 0)| C  |           |
    |               |                         +----|           |
    +---------------+                              +-----------+

Some will work with only one machine, though use of virtual interfaces.

As shown, the Device Under Test (DUT) should have at least one NIC connected
to the traffic generator on port 0. The user can use any traffic
generator. The DUT is also used to download the solution repository and
build the code. Cross compilation is not supported currently.

****************
Tested Platforms
****************

The sample applications are tested on the following platforms.

DUT
===

-  Ampere Altra (Neoverse-N1)

   -  Ubuntu 22.04.3 LTS (Jammy Jellyfish)

NIC
===

- `Mellanox ConnectX-6 Dx <https://www.nvidia.com/en-us/networking/ethernet-adapters/>`__

   - OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
   - Firmware version: 22.39.2048 (MT_0000000359)

- `Mellanox ConnectX-7 <https://www.nvidia.com/en-us/networking/ethernet-adapters/>`__

   - OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
   - Firmware version: 28.38.1002 (MT_0000000834)

.. note::

    To use a Mellanox NIC, install the OFED driver and update the NIC firmware by following the guidance in the :doc:`FAQ <../faq>`.

*****************
Preparing the DUT
*****************

Requirements
============

The DUT needs a minimum hardware configuration as below.

 * Processor: Minimum 1 GHz and 5 CPU cores
 * Hard Drive: Minimum 32 GB
 * Memory (RAM): Minimum 8 GB
 * Network Interface Controller: Minimum 10G port connecting to Traffic Generator
 * Connection to internet to download the source code and dependent packages

This documentation assumes the user has installed Ubuntu 22.04 (Jammy) on
the DUT.

 * Admin (root) privileges are required to run the software and set up the DUT.
 * Access to the internet is mandatory for downloading the solution's source code
   and installing all the dependent packages and libraries.
 * Scripts are provided to install the dependent packages and libraries.
 * Mellanox OFED driver is installed and the NIC firmware is updated.
 * GCC 9.4.0 or newer version is required to compile the software.
 * The provided scripts must be run in a bash shell.

To configure Git, run:

.. code-block:: shell

    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"

Download Source Code
====================

Create a new folder that will be the workspace, henceforth referred to
and used in commands with the ``$NW_DS_WORKSPACE`` environment variable
declared in these instructions:

.. code-block:: shell

    export NW_DS_WORKSPACE=/path/to/workspace
    mkdir -p $NW_DS_WORKSPACE
    cd $NW_DS_WORKSPACE
    export NW_DS_RELEASE=refs/tags/NW-DS-2025.06.30

To clone the repository, run the following commands:

.. code-block:: shell

    git clone git@git.gitlab.arm.com:arm-reference-solutions/dataplane-stack.git
    cd dataplane-stack
    git checkout ${NW_DS_RELEASE}
    git submodule update --init

.. note::

    ``refs/tags/NW-DS-2025.06.30`` is illustrative, please check for the latest tag.

    Tags prior to ``NW-DS-2025.06.30`` should follow instructions in previous README versions.

.. raw:: html

    <details>
    <summary><a>Using other versions of Dataplane Stack</a></summary>
    <br />
    <p>The latest features and bug fixes are made available in the git repository, but may not be
    tagged as part of a release. By default, we recommend checking out a tagged release using
    to ensure stability.</p>

    <p>However, if you want to try the most recent changes and bugfixes, you can checkout the main
    branch. Please keep in mind that this version may not have been fully tested or officially
    validated, and could be unstable. Use it at your own discretion.</p>

.. raw:: html

    </details>
    <br />

Setup
=====

This solution includes a ``setup.sh`` bash script which installs and upgrades the
required packages. To use it:

.. code-block:: shell

    cd $NW_DS_WORKSPACE/dataplane-stack
    sudo ./setup.sh

.. raw:: html

    <details>
    <summary><a>How to stop NGINX starting automatically?</a></summary>
    <br />
    <p>The NGINX package is installed in setup.sh.
    NGINX systemd service will be started automatically,
    which spawns some NGINX worker processes.</p>

    <p>To stop NGINX service, run: </p>

.. code-block:: shell

    sudo systemctl stop nginx

.. raw:: html

    <p>To disable the NGINX service from starting automatically
    on system boot, run:</p>

.. code-block:: shell

    sudo systemctl disable nginx

.. raw:: html

    </details>
    <br />

Additionally, some platform level parameters are required to run the applications.
These are set by grub at boot time.

To edit them you must edit ``/etc/default/grub`` as root. This file should have a
line specifying the GRUB command line like this:

.. code-block:: none

    GRUB_CMDLINE_LINUX=""

This may already contain some parameters or be empty. If it's missing add it.
An example value of GRUB_CMDLINE_LINUX could be:

.. code-block:: none

    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512 iommu.passthrough=1 isolcpus=1-4 rcu_nocbs=1-4 nohz_full=1-4 cpufreq.off=1 cpuidle.off=1 kpti=0"

The string specifies the kernel command line parameters. Alter it according to the guide below.

Parameters required by dataplane stack:

- reserve one 1G hugepage (via ``hugepagesz=1G hugepages=1``)
- reserve 512 2M hugepages (via ``hugepagesz=2M hugepages=512``)
- set IOMMU into passthrough mode (via ``iommu.passthrough=1``)
- isolate CPUs 1-4 from the Linux scheduler (via ``isolcpus=1-4``)

Optional parameters for better performance:

- isolate CPUs 1-4 from processing RCU callbacks (via ``rcu_nocbs=1-4``)
- set CPUs 1-4 to omit scheduling clock ticks  (via ``nohz_full=1-4``)
- disable cpufreq and cpuidle subsystems (via ``cpufreq.off=1`` and ``cpuidle.off=1``)
- disable control page table isolation of user and kernel address spaces (via ``kpti=0``)

These options might need to be tailored to your specific machine. Make sure
the selection of isolated CPUs is correct. Keep in mind that the isolated CPUs
will not be available for other tasks (like compilation).

.. raw:: html

    <details>
    <summary><a>Why reserve hugepages?</a></summary>
    <br />
    <p>Hugepages help prevent TLB misses and are commonly used by networking
    applications to manage memory. The larger the hugepage, the greater the
    performance increase due to fewer TLB misses and fewer page table walks.

    1GB hugepages are easiest to reserve at boot time, as reservation during
    run time is likely to not be possible. Run time reservation would require
    1GB of available contiguous memory, which is typically not available.</p>

.. raw:: html

    </details>
    <br />

.. raw:: html

    <details>
    <summary><a>Why isolate CPUs?</a></summary>
    <br />
    <p>The combination of the kernel parameters mentioned above ensure that the
    CPUs run only the desired application. They never process kernel RCU
    callbacks, don't generate scheduling ticks as there is only one process
    running on them, and are isolated from running any processes other than
    the ones pinned to them (the desired application).</p>

.. raw:: html

    </details>
    <br />

Reboot
======

After finishing setup, reboot the DUT. This ensures the setup changes are reflected before
running the sample applications. Run:

.. code-block:: none

    sudo update-grub
    sudo reboot

*****
Build
*****

This solution uses Makefile to build all the components.
To build the Dataplane Stack, run the following on the DUT:

.. code-block:: shell

    cd $NW_DS_WORKSPACE/dataplane-stack
    make all

It is also possible to compile the components individually by specifying ``make dpdk`` or ``make vpp``.
Run ``make help`` to view a list of all Makefile targets.

The above mentioned ``make`` commands can also be used to rebuild after modifying the code.

.. raw:: html

    <details>
    <summary><a>Troubleshooting</a></summary>
    <br />
    <p>If during link time you run into a missing symbols error:

    <pre>
    plugin/load [error ]: .[...]/dpdk_plugin.so:
        undefined symbol: armv8_dec_aes_cbc_sha1_128
    </pre>

    This is fixed by modifying the /components/vpp/build/external/packages/dpdk.mk file
    and adding crypto/armv8 to the list of DPDK_DRIVERS_DISABLED.</p>

.. raw:: html

    </details>
    <br />

*****************
Running Use Cases
*****************

Please follow the documentation for each individual use case.

The first two on the list are good ones to start with.

.. toctree::
   :titlesonly:
   :maxdepth: 2

   ipv4_l3fwd
   l3_forwarding_vpp
   l2_switching_vpp
   tcp_term
   ssl_reverse_proxy
   ipsec_vpp
   ngfw

Get NIC Information
===================

If you have multiple NIC ports you may connect PCIe addresses with physical ports
by first using ``ethtool`` which can blink the light associated with the interface:

.. code-block:: shell

    sudo ethtool --identify <interface name>

And then listing the PCIe addresses of the interfaces:

.. code-block:: shell

    sudo lshw -c net -businfo

.. raw:: html

    <details>
    <summary><a>Example</a></summary>
    <br />
    <p>For example, if enP1p1s0f0 is attached to the traffic generator, then running
    the command will show the PCIe address (0001:01:00.0).</p>

.. code-block:: none

    $ sudo lshw -c net -businfo
    Bus info          Device      Class      Description
    ====================================================
    pci@0001:01:00.0  enP1p1s0f0  network    MT27800 Family [ConnectX-5]
    pci@0001:01:00.1  enP1p1s0f1  network    MT27800 Family [ConnectX-5]

.. raw:: html

    </details>
    <br />

*******************************************
Porting/Integrating to another Arm platform
*******************************************

Although the solution is tested on limited hardware platforms, the solution
might work just fine on other Arm platforms. However, such platforms should
support Armv8 architecture at least and should be supported by the underlying
components.

************************
Changelog & Known Issues
************************

To check newly added features, feature changes, and known issues in each of
the releases, please refer to :doc:`CHANGELOG <../changelog>`.