mirror of
https://github.com/saymrwulf/uhd.git
synced 2026-05-16 21:10:10 +00:00
206 lines
9 KiB
Text
206 lines
9 KiB
Text
/*! \page page_dpdk DPDK, Data Plane Development Kit
|
|
|
|
\tableofcontents
|
|
|
|
\section dpdk_overview DPDK Overview
|
|
|
|
Data Plane Development Kit (DPDK) is a set of libraries that allows network
|
|
interface controller (NIC) drivers to use user space memory buffers to send and
|
|
receive data over a network. These libraries underpin one of the network
|
|
transport options in UHD.
|
|
|
|
In UHD, the DPDK-based transport will fork off I/O threads that implement the
|
|
network services, and these I/O threads will service the NICs on cores provided
|
|
in your configuration. The cores will be completely consumed by the I/O thread.
|
|
Together with DPDK's polling-mode drivers, this virtually eliminates context
|
|
switching in UHD's transport layer, which enables us to stream higher sample
|
|
rates.
|
|
|
|
\section dpdk_setup DPDK Setup
|
|
|
|
DPDK is currently only available on Linux platforms, requires an input-output
|
|
memory management unit (IOMMU), and must be run on a multicore processor. The
|
|
following subsections will talk through the steps required to setup DPDK on your
|
|
computer.
|
|
|
|
\subsection dpdk_installation DPDK Installation Instructions
|
|
|
|
As a new and developing technology, the DPDK APIs are unstable. UHD requires
|
|
version 18.11.
|
|
|
|
On Ubuntu 19.04/19.10, Fedora 31, or Debian Buster/Stretch (via backports),
|
|
DPDK is available in your distribution's repositories. For example, on Debian
|
|
systems, it can be obtained with the following command:
|
|
|
|
sudo apt install dpdk dpdk-dev
|
|
|
|
Otherwise, you'll need to follow the build guide at
|
|
https://doc.dpdk.org/guides-18.11/linux_gsg/build_dpdk.html . The software
|
|
releases can be found at https://core.dpdk.org/download/. Note that if you
|
|
are building and installing DPDK from source, you will need to change its
|
|
configuration to build shared libraries in order for UHD to link successfully.
|
|
After running `make config` or `make defconfig`, open the `build/.config` file
|
|
in an editor and find the following line:
|
|
|
|
CONFIG_RTE_BUILD_SHARED_LIB=n
|
|
|
|
Change the `n` to a `y` to enable the building of shared libraries, then type
|
|
`make` to start the build.
|
|
|
|
\subsection dpdk_system_configuration System Configuration
|
|
|
|
The official documentation regarding system configuration can be found at
|
|
https://doc.dpdk.org/guides-18.11/linux_gsg/sys_reqs.html.
|
|
|
|
First, you'll need to enable the IOMMU and set up some hugepages. DPDK will
|
|
completely take over all available hugepages, so don't allocate all your memory
|
|
to them- the rest of UHD and the application need memory too.
|
|
|
|
For example, on a system with 16 GB of RAM, a generous appropriation of
|
|
512x 2 MiB pages was more than sufficient, and you likely won't need that much.
|
|
|
|
For best results, hugepages should be enabled at boot. For example, using an
|
|
Intel IOMMU with Ubuntu 19.04 IOMMU drivers, the following line was needed in
|
|
our Grub config.
|
|
|
|
iommu=pt intel_iommu=on hugepages=2048
|
|
|
|
The setup of the IOMMU and hugepages is system-specific, so consult the kernel
|
|
documentation for more info. After you reboot, you should see
|
|
`/sys/kernel/iommu_groups` populated.
|
|
|
|
Next, many of the NIC drivers are implemented atop `vfio-pci`, so you'll need to
|
|
load that driver with the following command:
|
|
|
|
modprobe vfio-pci
|
|
|
|
For NICs that require vfio-pci (like Intel's X520), you'll want to use the
|
|
`dpdk-devbind.py` script to the vfio-pci driver. This script is shipped with
|
|
DPDK and installed to `$prefix/share/dpdk/usertools`. If the NIC uses the
|
|
vfio-pci driver, and the package was installed apt-get, then a typical invocation
|
|
might be
|
|
|
|
/usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci ens6f0
|
|
|
|
If successful, the script might provide an updated status like this:
|
|
|
|
/usr/share/dpdk/usertools/dpdk-devbind.py -s
|
|
|
|
Network devices using DPDK-compatible driver
|
|
============================================
|
|
0000:02:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe
|
|
[...]
|
|
|
|
|
|
See https://doc.dpdk.org/guides-18.11/linux_gsg/linux_drivers.html#binding-and-unbinding-network-ports-to-from-the-kernel-modules
|
|
for more details.
|
|
|
|
With the hugepages, IOMMU, and drivers set up, the system is ready for DPDK to
|
|
use.
|
|
|
|
\subsection dpdk_nic_config NIC Configuration
|
|
|
|
Configuration of the NIC can be controlled via device arguments via the usual
|
|
methods, but the \ref page_configfiles "UHD configuration file" is the
|
|
recommended location.
|
|
|
|
In order to run, you'll need to set the permissions for your user to take over
|
|
the vfio-pci devices, the hugepages, and the scheduler's settings for the
|
|
threads (at a minimum). You may consider running you applications as root, at
|
|
least while becoming familiar with DPDK. If you use a per-user config file, make
|
|
sure it's in the correct location.
|
|
|
|
The config file will have 2 different components. First are the global DPDK
|
|
options:
|
|
|
|
;When present in device args, use_dpdk indicates you want DPDK to take over the UDP transports
|
|
;The value here represents a config, so you could have another section labeled use_dpdk=myconf
|
|
;instead and swap between them
|
|
[use_dpdk=1]
|
|
;dpdk_mtu is the NIC's MTU setting
|
|
;This is separate from MPM's maximum packet size
|
|
dpdk_mtu=9000
|
|
;dpdk_driver is the -d flag for the DPDK EAL. If DPDK doesn't pick up the driver for your NIC
|
|
;automatically, you may need this argument to point it to the folder where it can find the drivers
|
|
;Note that DPDK will attempt to load _everything_ in that folder as a driver, so you may want to
|
|
;create a separate folder with symlinks to the librte_pmd_* and librte_mempool_* libraries.
|
|
dpdk_driver=/usr/local/lib/dpdk-pmds/
|
|
;dpdk_corelist is the -l flag for the DPDK EAL. See more at the link
|
|
; https://doc.dpdk.org/guides-18.11/linux_gsg/build_sample_apps.html#running-a-sample-application
|
|
dpdk_corelist=0,1
|
|
;dpdk_num_mbufs is the total number of packet buffers allocated
|
|
;to each direction's packet buffer pool
|
|
;This will be multiplied by the number of NICs, but NICs on the same
|
|
;CPU socket share a pool.
|
|
dpdk_num_mbufs=4096
|
|
;dpdk_mbuf_cache_size is the number of buffers to cache for a CPU
|
|
;The cache reduces the interaction with the global pool
|
|
dpdk_mbuf_cache_size=64
|
|
|
|
|
|
The other sections fall under per-NIC arguments. The key for NICs is the MAC
|
|
address, and it must be in a particular format. Hex digits must all be lower
|
|
case, and octets must be separated by colons. Here is an example:
|
|
|
|
[dpdk_mac=3c:fd:fe:a2:a9:09]
|
|
;dpdk_lcore selects the lcore that this NIC's driver will run on
|
|
;Multiple NICs may occupy one lcore, but the I/O thread will completely
|
|
;consume that lcore's CPU. Also, 0 is reserved for the master thread (i.e.
|
|
;the initial UHD thread that calls init() for DPDK). Attempting to
|
|
;use it as an I/O thread will only result in hanging.
|
|
;Note also that by default, the lcore ID will be the same as the CPU ID.
|
|
dpdk_lcore = 1
|
|
;dpdk_ipv4 specifies the IPv4 address, and both the address and
|
|
;subnet mask are required (and in this format!). DPDK uses the
|
|
;netmask to create a basic routing table. Routing to other networks
|
|
;(i.e. via gateways) is not permitted.
|
|
dpdk_ipv4 = 192.168.10.1/24
|
|
;dpdk_num_desc is the number of descriptors in each DMA ring.
|
|
;Must be a power of 2.
|
|
dpdk_num_desc=4096
|
|
|
|
[dpdk_mac=3c:fd:fe:a2:a9:0a]
|
|
dpdk_lcore = 1
|
|
dpdk_ipv4 = 192.168.20.1/24
|
|
|
|
\section dpdk_using Using DPDK in UHD
|
|
|
|
Once DPDK is installed and configured on your system, it can be used with UHD.
|
|
The following steps will describe how to stream using DPDK. DPDK is currently
|
|
only available on the following devices:
|
|
|
|
- \ref page_usrp_e3xx "E320 (but not E31x) device"
|
|
- \ref page_usrp_n3xx "N3xx devices"
|
|
- \ref page_usrp_x3x0 "X3xx devices"
|
|
|
|
\subsection dpdk_device_args Enabling DPDK with UHD Device Args
|
|
|
|
Add the following to your device args in order to indicate that a DPDK-based UDP
|
|
transport shall be used instead of the kernel's UDP stack.
|
|
|
|
--args="use_dpdk=1"
|
|
|
|
Device discovery via DPDK is not currently implemented, so the device args
|
|
`mgmt_addr`, `addr`, and `second_addr` (if applicable) must all be specified at
|
|
runtime. There is no mechanism for MPM's TCP/IP control traffic to flow over a
|
|
link that is occupied by DPDK, so mgmt_addr must point to a link that is not
|
|
used for CHDR, such as N310's RJ45 port.
|
|
|
|
\subsection dpdk_link_detection DPDK Link Detection
|
|
|
|
When DPDK is enabled and the driver is initializing, the status of all
|
|
DPDK-enabled links is checked to verify that all links are active before the
|
|
driver proceeds. The links are checked every 250 ms until all links have
|
|
reported that they are up or until the link timeout expires, which by default
|
|
is 1000 ms. If any of the links have not reported as being up by the time the
|
|
timeout expires, a runtime error is thrown.
|
|
|
|
Users may choose to override the link timeout in cases where particular
|
|
systems and/or network cards take longer to establish stable DPDK links.
|
|
The timeout can be overridden by passing `dpdk_link_timeout=N` in the device
|
|
arguments, where N is the desired timeout time in <b>milliseconds</b>, or
|
|
by adding a `dpdk_link_timeout` entry to the
|
|
\ref page_configfiles "UHD configuration file".
|
|
|
|
*/
|
|
// vim:ft=doxygen:
|