VPP IPv4 L3 Forwarding
Introduction
VPP IPv4 L3 forwarding implements the typical routing function based on 32-bit IPv4 address. It forwards packets using Longest Prefix Match algorithm based on the mtrie forwarding table.
The L3 forwarding table starts with entries for directly connected network. Static table entries can be added manually. Additionally, the VPP router can dynamically learn table entries through dynamic routing protocols (such as OSPF, BGP, etc.).
When an IP packet arrives at a VPP router, the L3 forwarding table is looked up to determine the appropriate next-hop for the packet based on its destination IP address. Then the packet is sent out through the determined outgoing interface.
This guide explains how to use the VPP based L3 forwarding using either memif or NIC interfaces. Users can execute bundled scripts in dataplane-stack repo to quickly establish the L3 forwarding cases.
Memif Connection
Shared memory packet interface (memif) is software emulated Ethernet interface, which provides high performance packet transmit and receive between VPP and user application or multiple VPP instances.
In this setup, two pairs of memif interfaces are configured to connect VPP L3 router instance and VPP based traffic generator. On VPP router side, DPDK zero-copy memif interfaces are used for testing VPP + DPDK stack. On VPP traffic generator side, VPP’s native memif interfaces are used for performance reason.
Memif connection
The user can quickly run VPP L3 forwarding over memif connection using the scripts
located at: $NW_DS_WORKSPACE/dataplane-stack/usecase/l3_forwarding.
Overview
In memif connection scenario, the main operations of each script are as follows:
run_vpp_tg.sh
Run a VPP instance for the traffic generator on the specified CPU cores
Create two VPP memif interfaces in server role
Configure a software traffic generator
Start the traffic generator
run_vpp_rt.sh
Run a VPP instance for the router on the specified CPU cores
Create two DPDK memif interfaces in client role with zero-copy enabled
Bring interfaces up and set interfaces IP addresses
Add IP route entries in the L3 forwarding table
traffic_monitor.sh
Monitor VPP router interface counters and L3 forwarding throughput with VPP
show interfaceandshow runtimecommands
stop.sh
Stop both VPP router and traffic generator instances
For detailed usage on VPP CLI commands used in scripts, refer to the following links:
Execution
Note
This execution requires at least three isolated cores for VPP workers. Cores 2-4 are assumed to be isolated in this guide.
Quickly set up VPP router/traffic generator and test L3 forwarding use case for 1 flow with 60 byte packets:
cd $NW_DS_WORKSPACE/dataplane-stack
./usecase/l3_forwarding/run_vpp_tg.sh -c 1,2,3 -f 1 -l 60
./usecase/l3_forwarding/run_vpp_rt.sh -m -c 1,4 -f 1
Note
Use
-hto check scripts supported options.Larger packets may reduce VPP traffic generator performance. Changing the packet length via
-loption should not affect VPP router performance.
Examine VPP router’s interfaces rx/tx counters and packet processing runtime:
./usecase/l3_forwarding/traffic_monitor.sh
Below is key output:
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
Ethernet0 1 up 9000/0/0/0 rx packets 33345792
rx bytes 2000747520
ip4 33345792
Ethernet1 2 up 9000/0/0/0 tx packets 33345792
tx bytes 2000747520
local0 0 down 0/0/0/0
Thread 1 vpp_wk_0 (lcore 4)
Time 3.0, 10 sec internal node vector rate 256.00 loops/sec 42266.90
vector rates in 1.1013e7, out 1.1013e7, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
Ethernet1-output active 130174 33324544 0 6.09e-2 256.00
Ethernet1-tx active 130174 33324544 0 7.44e-1 256.00
dpdk-input polling 130174 33324544 0 4.65e-1 256.00
ethernet-input active 130174 33324544 0 2.57e-1 256.00
ip4-input-no-checksum active 130174 33324544 0 1.57e-1 256.00
ip4-lookup active 130174 33324544 0 2.26e-1 256.00
ip4-rewrite active 130174 33324544 0 3.56e-1 256.00
unix-epoll-input polling 127 0 0 3.49e1 0.00
Note
VPP
Ethernet0is the aliased name of the input memif interface in the example.VPP
Ethernet1is the aliased name of the output memif interface in the example.vector ratesprovide insights into the packet processing throughput of a specific node or function in VPP.Vectors/Callmeasures packet processing efficiency in VPP as operations per function call for a specific node or function.
Stop both VPP router and traffic generator instances:
./usecase/l3_forwarding/stop.sh
Ethernet Connection
In this L3 forwarding scenario, DUT and traffic generator run on separated hardware platforms and are connected with Ethernet adapters and cables. The traffic generator could be software-based, e.g., VPP/TRex/TrafficGen running on regular servers, or hardware platforms, e.g., IXIA/Spirent Smartbits.
Ethernet connection
Find out which DUT interfaces are connected to the traffic generator.
sudo ethtool --identify <interface_name> will typically blink a light on the NIC
to help identify the physical port associated with the interface.
Get interface names and PCIe addresses from lshw command:
sudo lshw -c net -businfo
The output will look similar to:
Bus info Device Class Description
====================================================
pci@0000:07:00.0 eth0 network RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
pci@0001:01:00.0 enP1p1s0f0 network MT27800 Family [ConnectX-5]
pci@0001:01:00.1 enP1p1s0f1 network MT27800 Family [ConnectX-5]
Of the two interfaces connected to the traffic generator, arbitrarily choose one
to be the input interface and the other to be the output interface. In this setup
example, enP1p1s0f0 at PCIe address 0001:01:00.0 is the input interface,
and enP1p1s0f1 at PCIe address 0001:01:00.1 is the output interface.
Get the MAC address of the input interface enP1p1s0f0 connected to the traffic
generator via ip link show enP1p1s0f0. This MAC address will be used as
destination MAC address of generated traffic. Below is a sample output:
1: enP1p1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8996 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b8:ce:f6:10:e4:6c brd ff:ff:ff:ff:ff:ff
The user can quickly run VPP L3 forwarding over NIC connection using the scripts located
at: $NW_DS_WORKSPACE/dataplane-stack/usecase/l3_forwarding.
Overview
In Ethernet connection scenario, the main operations of each script are as follows:
run_vpp_rt.sh
Run a VPP instance for the router on the specified CPU cores
Attach two DPDK interfaces on the specified PCIe addresses to VPP
Bring the interfaces up and set their IP addresses
Add IP route entries in the L3 forwarding table
traffic_monitor.sh
Monitor VPP router interface counters and L3 forwarding throughput via VPP
show interfaceandshow runtimecommands
stop.sh
Stop VPP router instance
Execution
Note
This execution requires at least one isolated core for VPP worker. Core 2 is assumed to be isolated in this guide.
Quickly set up VPP router with input/output interface PCIe addresses on specified cores for 1 flow:
cd $NW_DS_WORKSPACE/dataplane-stack
./usecase/l3_forwarding/run_vpp_rt.sh -p 0001:01:00.0,0001:01:00.1 -c 1,2 -f 1
Note
Replace sample addresses in above command with desired PCIe addresses on DUT.
Configure traffic generator to send 1 flow to VPP input interface with destination
IP address of 1.0.0.1 and destination MAC address of b8:ce:f6:10:e4:6c, then VPP
router will forward those packets out on VPP output interface.
Examine VPP router’s interfaces rx/tx counters and packet processing runtime:
./usecase/l3_forwarding/traffic_monitor.sh
Here is key output:
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter Count
Ethernet0 1 up 9000/0/0/0 rx packets 17201368
rx bytes 1032082080
ip4 17201392
Ethernet1 2 up 9000/0/0/0 tx packets 17201484
tx bytes 1032089040
local0 0 down 0/0/0/0
Thread 1 vpp_wk_0 (lcore 2)
Time 3.0, 10 sec internal node vector rate 13.51 loops/sec 460679.66
vector rates in 5.6887e6, out 5.6887e6, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
Ethernet1-output active 1272245 17205204 0 1.52e-1 13.52
Ethernet1-tx active 1272245 17205204 0 5.67e-1 13.52
dpdk-input polling 1441728 17205204 0 1.53e0 11.93
ethernet-input active 1272245 17205204 0 1.00e0 13.52
ip4-input-no-checksum active 1272245 17205204 0 3.32e-1 13.52
ip4-lookup active 1272245 17205204 0 3.84e-1 13.52
ip4-rewrite active 1272245 17205204 0 3.66e-1 13.52
unix-epoll-input polling 1406 0 0 1.47e1 0.00
Note
VPP
Ethernet0is the aliased name of the input interface, which is at PCIe address0001:01:00.0in the example.VPP
Ethernet1is the aliased name of the output interface, which is at PCIe address0001:01:00.1in the example.
Stop VPP router:
./usecase/l3_forwarding/stop.sh
Suggested Experiments
Add more CPU cores as VPP workers
To add more CPU cores for VPP data plane to achieve better performance:
./usecase/l3_forwarding/run_vpp_tg.sh -c 1,2-7 -f 1 -l 60
./usecase/l3_forwarding/run_vpp_rt.sh -m -c 1,8-10 -f 1
./usecase/l3_forwarding/run_vpp_rt.sh -p 0001:01:00.0,0001:01:00.1 -c 1,2-4 -f 1
Note
In order to ensure that the performance of traffic generator in memif connection scenario is powerful enough, the number of worker cores of VPP traffic generator needs to be twice the number of worker cores of VPP router.
Generate and forward more packet flows
To generate and forward more packet flows with different destination IP addresses, e.g. 10000 flows:
./usecase/l3_forwarding/run_vpp_tg.sh -c 1,2,3 -f 10000 -l 60
./usecase/l3_forwarding/run_vpp_rt.sh -m -c 1,4 -f 10000
./usecase/l3_forwarding/run_vpp_rt.sh -p 0001:01:00.0,0001:01:00.1 -c 1,2 -f 10000
Configure the traffic generator to send packets with destination IP addresses
starting from 1.0.0.1 and incrementing by 1 with 10000 increments.
Change number of descriptors in receive ring and transmit ring
Number of descriptors in receive ring and transmit ring can impact performance. The default number is 1024; refer to VPP configuration num-rx-desc num-tx-desc.
To change descriptors size to a non-default value such as 512, first locate the VPP
starting up line of the script run_vpp_rt.sh. It looks like below:
sudo "${vpp_binary}" ...
Then add below configuration into dev default subsection of dpdk section:
num-rx-desc 512 num-tx-desc 512
Use faster DPDK vector PMDs
It is possible to use faster DPDK vector PMDs by disabling multi-segment buffers and UDP/TCP TX checksum offload. This improves performance but does not support Jumbo MTU. To utilize the DPDK vector PMDs, refer to VPP configuration no-multi-seg.
To add this configuration, first locate the VPP starting up line of the script run_vpp_rt.sh. It looks like below:
sudo "${vpp_binary}" ...
Then add below configuration into dpdk section:
no-multi-seg no-tx-checksum-offload
Note
Above configuration only works for NIC connection.
Use other types of device drivers
Besides Mellanox ConnectX-5, VPP supports NICs from other vendors as well. VPP is integrated with NICs using the following 2 methods: