..
  # Copyright (c) 2022-2024, Arm Limited.
  #
  # SPDX-License-Identifier: Apache-2.0

##########
User Guide
##########

Welcome to the Dataplane Stack reference solution user guide. This guide provides
the detailed guidelines to end users on how to download, build and execute the
solution on Arm platforms. Guidelines on experimenting with various parameters
that affect performance of the use cases are described in detail. This guide
is intended to describe complex and practical use cases which require complex
test setup.

**************************
Users and Essential Skills
**************************

The reference solutions are targeted for a networking software development or
performance analysis engineer who has in-depth networking knowledge, but does
not know about Arm necessarily.

Using this guide requires in-depth knowledge on networking use cases. Mastering
knowledge on certain user space open source networking projects,
e.g., DPDK, VPP, ODP, will help gain deeper understanding of this guide
and the reference solutions more easily.

****************
Tested Platforms
****************

The sample applications are tested on the following platforms.

DUT
~~~

-  Ampere Altra (Neoverse-N1)

   -  Ubuntu 22.04.3 LTS (Jammy Jellyfish)

NIC
~~~

- `Mellanox ConnectX-6 Dx <https://www.nvidia.com/en-us/networking/ethernet-adapters/>`__

   - OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
   - Firmware version: 22.39.2048 (MT_0000000359)

- `Mellanox ConnectX-7 <https://www.nvidia.com/en-us/networking/ethernet-adapters/>`__

   - OFED driver: MLNX_OFED_LINUX-23.10-1.1.9.0
   -  Firmware version: 28.38.1002 (MT_0000000834).

.. note::

    To use a Mellanox NIC, install the OFED driver and update the NIC firmware by following the guidance in the :doc:`FAQ <../faq>`.

*****************
Preparing the DUT
*****************

The DUT is also used to download the solution repository and
build the code. Cross compilation is not supported currently.

Requirements
~~~~~~~~~~~~

The DUT needs to have a minimum hardware configuration as below.

 * Processor: Minimum 1 GHz and 5 CPU cores
 * Hard Drive: Minimum 32 GB
 * Memory (RAM): Minimum 8 GB
 * Network Interface Controller: Minimum 10G port connecting to Traffic
   Generator
 * Ethernet connection to internet

This documentation assumes the user has installed Ubuntu 22.04 (Jammy) on
the DUT.

 * Admin (root) privileges are required to run the software and set up the
   DUT.
 * Access to the internet is mandatory for downloading the solution's source code
   and installing all the dependent packages and libraries.
 * Scripts are provided to install the dependent packages and libraries.
 * Mellanox OFED driver is installed and the NIC firmware is updated.
 * gcc 9.4.0 or newer version is required to compile the software.
 * The provided scripts must be run in a bash shell.

The following utilities must be available on the DUT:
 * git
 * curl
 * python
 * python3

To configure Git, run:

::

    git config --global user.email "you@example.com"
    git config --global user.name "Your Name"

Follow the instructions provided in
`git-repo <https://gerrit.googlesource.com/git-repo>`__ to install the
``repo`` tool manually.

.. _repo download target:

Download Source Code
~~~~~~~~~~~~~~~~~~~~

Create a new folder that will be the workspace, henceforth referred to
and used in commands with the ``$NW_DS_WORKSPACE`` environment variable
declared in these instructions:

::

    export NW_DS_WORKSPACE=/path/to/workspace
    mkdir -p $NW_DS_WORKSPACE
    cd $NW_DS_WORKSPACE
    export NW_DS_RELEASE=refs/tags/NW-DS-2024.03.29

.. note::

  Sometimes new features and additional bug fixes will be made available in
  the git repositories and have not yet been tagged as part of a
  release. To pick up these latest changes, remove the
  ``-b <release tag>`` option from the ``repo init`` command below. However,
  please be aware that such untagged changes have not yet been formally
  verified and should be considered unstable until they are tagged in an
  official release.

To clone the repository, run the following commands:

::

    repo init \
        -u https://git.gitlab.arm.com/arm-reference-solutions/arm-reference-solutions-manifest.git \
        -b ${NW_DS_RELEASE} \
        -m dataplane-stack.xml
    repo sync


Setup
~~~~~

This solution includes a ``setup.sh`` Bash script responsible for installing and
upgrading the following packages:

.. code-block:: none

    net-tools build-essential manpages-dev libnuma-dev python python3-venv cmake meson pkg-config python3-pyelftools lshw util-linux iperf3 nginx

To run it:

::

    cd <nw_ds_workspace>/dataplane-stack
    sudo ./setup.sh

.. note::

    NGINX package is installed in setup.sh. NGINX systemd service will be started
    automatically, which spawns some NGINX worker processes. To stop the
    auto-started NGINX service, run ``sudo systemctl stop nginx``. To disable the
    NGINX service from starting automatically on system boot, run ``sudo systemctl disable nginx``.

Additionally, some platform level parameters are required to run the applications.
These are set by grub at boot time.

To edit them you must edit ``/etc/default/grub`` as root. This file should have a
line specifying the GRUB command line like this:

.. code-block:: none

    GRUB_CMDLINE_LINUX=""

This may already contain some parameters or be empty. If it's missing add it.
The string specifies the kernel command line parameters. Alter it according to the guide below.

Parameters required by dataplane stack:

- reserve one 1G hugepage (via ``hugepagesz=1G hugepages=1``)
- reserve 512 2M hugepages (via ``hugepagesz=2M hugepages=512``)
- set IOMMU into passthrough mode (via ``iommu.passthrough=1``)
- isolate CPUs 1-4 from the Linux scheduler (via ``isolcpus=1-4``)

Optional parameters for better performance:

- isolate CPUs 1-4 from processing RCU callbacks (via ``rcu_nocbs=1-4``)
- set CPUs 1-4 to omit scheduling clock ticks  (via ``nohz_full=1-4``)
- disable cpufreq and cpuidle subsystems (via ``cpufreq.off=1`` and
  ``cpuidle.off=1``)
- disable control page table isolation of user and kernel address spaces
  (via ``kpti=0``)

These options might need to be tailored to your specific machine. Make sure
the selection of isolated CPUs is correct. Keep in mind that the isolated CPUs
will not be available for other tasks (like compilation).

An example complete line using all these parameters would look like this:

.. code-block:: none

    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=1 hugepagesz=2M hugepages=512 iommu.passthrough=1 isolcpus=1-4 rcu_nocbs=1-4 nohz_full=1-4 cpufreq.off=1 cpuidle.off=1 kpti=0"

After saving the ``/etc/default/grub`` file, to restart the machine with new
parameters, run:

::

    sudo update-grub
    sudo reboot

.. raw:: html

   <details>
   <summary><a>Why reserve hugepages?</a></summary>

.. code-block:: none

    Hugepages help prevent TLB misses and are commonly used by networking
    applications to manage memory. The larger the hugepage, the greater the
    performance increase due to fewer TLB misses and fewer page table walks.

    1GB hugepages are easiest to reserve at boot time, as reservation during
    run time is likely to not be possible. Run time reservation would require
    1GB of available contiguous memory, which is typically not available.

.. raw:: html

   </details>
   <br />

.. raw:: html

   <details>
   <summary><a>Why isolate CPUs?</a></summary>

.. code-block:: none

    The combination of the kernel parameters mentioned above ensure that the
    CPUs run only the desired application. They never process kernel RCU
    callbacks, don't generate scheduling ticks as there is only one process
    running on them, and are isolated from running any processes other than
    the ones pinned to them (the desired application).


.. raw:: html

   </details>
   <br />


Build
~~~~~

This solution uses Makefile to build all the components.

The Makefile:

- Builds DPDK and the L3fwd sample applications
- Builds VPP
- Builds traffic generator tools, e.g., wrk2

To build the Dataplane Stack, run the following on the DUT:

::

    cd $NW_DS_WORKSPACE/dataplane-stack
    make all

It is also possible to compile the components individually by specifying ``make dpdk`` or ``make vpp``.
Run ``make help`` to view a list of all Makefile targets.

The above mentioned ``make`` commands can also be used to rebuild after modifying the code.

Reboot
~~~~~~
After setting up the DUT and building the software, reboot the DUT. This ensures the setup changes are reflected before
running the sample applications.


Get the NIC Interface
~~~~~~~~~~~~~~~~~~~~~
Next, identify the interface(s) on the NIC(s) connected to the traffic
generator.

.. raw:: html

   <details>
   <summary><a>How to find the right interfaces?</a></summary>

.. code-block:: none

    $ sudo ethtool --identify <interface>
    will typically blink a light on the NIC to help identify the
    physical port associated with the interface.

.. raw:: html

   </details>
   <br />

*******************************************
Porting/Integrating to another Arm platform
*******************************************

Although the solution is tested on limited hardware platforms, the solution
might work just fine on other Arm platforms. However, such platforms should
support Armv8 architecture at least and should be supported by the underlying
components.


*********
Use Cases
*********


.. toctree::
   :titlesonly:
   :maxdepth: 2

   ipv4_l3fwd
   l3_forwarding_vpp
   l2_switching_vpp
   tcp_term
   ssl_reverse_proxy
   ipsec_vpp