DPDK

From VoIPmonitor.org
Revision as of 19:42, 6 January 2026 by Admin (talk | contribs) (Add troubleshooting section for imissed packet loss with dpdk_nb_rx, dpdk_nb_rxq_rss, and dpdk_pkt_burst parameters)


This is an expert-level guide for configuring the VoIPmonitor sensor to use the Data Plane Development Kit (DPDK) for ultra-high-performance packet capture. This setup is intended for multi-gigabit traffic loads where the standard Linux network stack becomes a bottleneck.

What is DPDK and Why Use It?

The Data Plane Development Kit (DPDK) is a set of libraries and drivers that allows an application, like the VoIPmonitor sensor, to bypass the operating system's kernel and interact directly with the network card hardware.

  • Standard Kernel Method: In a normal setup, every incoming packet (or group of packets) triggers a CPU interrupt (IRQ), telling the kernel to process it. This interrupt-driven model is reliable but creates significant overhead, typically maxing out around 2-3 Gbit/s on a 10Gbit NIC, as it is limited by the performance of a single CPU core.
  • DPDK Method: DPDK uses poll-mode drivers. A dedicated CPU core is assigned to constantly poll the network card for new packets, completely avoiding the overhead of kernel interrupts and context switching. This allows for much higher packet throughput, enabling VoIPmonitor to handle 6 Gbit/s or more on a single server.

The trade-off is that this setup requires careful system tuning and dedicated CPU cores to avoid scheduler delays, which can cause packet drops in a high-throughput environment.

Step 1: System and Hardware Prerequisites

  • Supported NIC: You must use a network card supported by DPDK. A list of compatible hardware can be found on the DPDK supported hardware page. Intel 10-Gigabit cards (like the X540 or X710 series) are a common choice.
  • BIOS/UEFI Settings: VT-d (for Intel) or AMD-Vi (for AMD) virtualization technology must be enabled in your server's BIOS/UEFI. IOMMU must also be enabled.
  • DPDK Version: VoIPmonitor requires DPDK version 21.08.0 or newer. It is recommended to download the latest stable release from the official DPDK website.

Step 2: System Preparation (HugePages & IOMMU)

DPDK requires specific kernel features and pre-allocated memory to function.

A. Configure HugePages

DPDK uses large, contiguous blocks of memory called HugePages for its packet buffers (mbufs).

To temporarily allocate 16GB of 1G-sized HugePages on NUMA node 0
echo 16 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages

ℹ️ Note: You must allocate HugePages on the same NUMA node (CPU socket) that your network card is physically connected to.

To make this permanent, edit the GRUB configuration file (/etc/default/grub)
# This example allocates 16 1GB pages at boot
# CRITICAL: transparent_hugepage=never is required to prevent "Permission denied" errors after reboot
GRUB_CMDLINE_LINUX_DEFAULT="... transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16"

After editing, run update-grub and reboot.

⚠️ Warning: If DPDK fails to start after a system reboot with a "Permission denied" error, verify that your GRUB configuration includes transparent_hugepage=never. This parameter disables transparent hugepages, which can conflict with the explicit 1GB hugepages required by DPDK and cause permission issues when the service starts.

B. Enable IOMMU

The IOMMU (Input-Output Memory Management Unit) is required for the VFIO driver used to bind the NIC to DPDK.

Edit /etc/default/grub and add the following to GRUB_CMDLINE_LINUX_DEFAULT
# For Intel CPUs
GRUB_CMDLINE_LINUX_DEFAULT="... iommu=pt intel_iommu=on"

# For AMD CPUs
GRUB_CMDLINE_LINUX_DEFAULT="... iommu=pt amd_iommu=on"

After editing, run update-grub and reboot. After rebooting, verify that the /sys/kernel/iommu_groups/ directory is populated with subdirectories.

Step 3: Bind the Network Interface to DPDK

Once the system is prepared, you must unbind the network interface you want to use for sniffing from the kernel driver and bind it to a DPDK-compatible driver. This means the OS will no longer see or be able to use this interface (e.g., it will not appear in ifconfig or ip a).

1. Find the PCI address of your network card
# This script is included with the DPDK source package
dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:1f:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=ens3f0 drv=ixgbe unused=
0000:1f:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=ens3f1 drv=ixgbe unused=
2. Load the VFIO-PCI driver
modprobe vfio-pci
3. Bind the interface to the driver using its PCI address
# This example binds both ports of the X540 card
dpdk-devbind.py -b vfio-pci 0000:1f:00.0 0000:1f:00.1
To unbind it and return control to the kernel
dpdk-devbind.py -u 0000:1f:00.1
dpdk-devbind.py -b ixgbe 0000:1f:00.1

ℹ️ Note: On some systems, vfio-pci may not work correctly. An alternative is the igb_uio driver, which may need to be compiled manually. See the official DPDK driver documentation for more details.

Step 4: Configure VoIPmonitor

Finally, configure your voipmonitor.conf file to use the DPDK interface.

Mandatory Parameters

# /etc/voipmonitor.conf

# Enable DPDK mode
dpdk = yes

# Tell the sniffer to use the DPDK interface instead of a kernel interface like eth0
interface = dpdk:0

# The PCI address of the network card to use for sniffing
dpdk_pci_device = 0000:1f:00.0

# Assign dedicated CPU cores for the DPDK polling threads.
# These cores should be on the same NUMA node as the NIC.
dpdk_read_thread_cpu_affinity = 2
dpdk_worker_thread_cpu_affinity = 30

Optional Performance Parameters

# Number of receive queues on the NIC. Default is 2.
dpdk_nb_rxq = 4

# Receive Side Scaling (RSS) for distributing packets across multiple queues.
# Recommended to enable for better packet distribution on multi-core systems.
dpdk_nb_rxq_rss = yes

# Size of the receive ring buffer on the NIC (number of descriptor entries).
# Larger values reduce packet drops (imissed) by buffering more packets
# before they must be processed. Default is 512; increase to 16384 if you
# experience imissed packet loss even during low network load.
dpdk_nb_rx = 16384

# Number of packets to read in a single burst. Default is 32.
# Increasing this to 512 can help reduce imissed packet drops by processing
# packets more efficiently. Only increase if you have CPU resources available.
dpdk_pkt_burst = 512

# Number of mbuf segments (x1024) in the memory pool between reader/worker threads.
# Default of 1024 allocates about 2GB of RAM. Increase for >5Gbit traffic.
dpdk_nb_mbufs = 4096

# Restrict all other voipmonitor threads to specific cores, leaving the
# DPDK cores isolated.
# By default, voipmonitor will automatically use all cores EXCEPT those
# assigned to the DPDK reader/worker.
thread_affinity = 1,3-29,31-59

Troubleshooting: Packet Loss Reported as 'imissed'

If you see the imissed counter increasing in your VoIPmonitor logs, this indicates packets are being dropped by the Network Interface Card (NIC) because they could not be read from the receive ring buffer fast enough. This can happen even during periods of low network load if the DPDK polling thread is interrupted or if the ring buffer is too small.

⚠️ Warning: imissed drops are caused by the NIC, not by VoIPmonitor. The NIC's small hardware buffer fills up faster than packets are read, causing it to drop packets.

    • Common Causes and Solutions:**

ℹ️ Note: Before applying configuration changes, ensure your CPU cores are properly isolated from the OS scheduler using the isolcpus= kernel parameter (see "Advanced OS Tuning" above).

  • RX Ring Buffer Too Small (dpdk_nb_rx)':: The default ring buffer size may be insufficient. Increase it to allow more packets to be buffered on the NIC:
dpdk_nb_rx = 16384
  • Burst Size Too Low (dpdk_pkt_burst)':: Reading packets in larger batches can improve efficiency and reduce drops:
dpdk_pkt_burst = 512
  • Poor Packet Distribution (dpdk_nb_rxq_rss)':: If using multiple receive queues, ensure RSS is enabled to distribute packets evenly:
dpdk_nb_rxq_rss = yes
dpdk_nb_rxq = 4
    • Verification Steps:**

After applying the above changes and restarting VoIPmonitor:

1. Monitor the imissed counter in your VoIPmonitor logs: 

tail -f /var/log/voipmonitor.log | grep imissed

2. The counter should stop increasing or increase much more slowly under the same network load.

3. For persistent imissed drops, verify CPU isolation and check that no other processes are running on your DPDK-affinity cores.

Advanced OS Tuning for Maximum Performance

For the most demanding environments (6Gbit+), isolating dedicated cores from the Linux scheduler is critical.

1. Isolate CPU Cores

Edit /etc/default/grub and add isolcpus=2,30 to the kernel command line. This tells the Linux scheduler to avoid scheduling any general tasks on cores 2 and 30, reserving them exclusively for our DPDK threads. 

GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=2,30"
2. Enable Tickless Kernel (NOHZ_FULL)

To prevent even periodic timer interrupts from disturbing the polling threads, you can configure the kernel to be "tickless." This is an advanced option that may require compiling a custom kernel with CONFIG_NO_HZ_FULL=y. Add the following to your GRUB configuration: 

GRUB_CMDLINE_LINUX_DEFAULT="... nohz=on nohz_full=2,30 rcu_nocbs=2,30"

AI Summary for RAG

Summary: This guide provides an expert-level walkthrough for configuring VoIPmonitor with DPDK (Data Plane Development Kit) for high-performance packet capture on multi-gigabit networks. It explains that DPDK bypasses the standard, interrupt-driven Linux kernel stack and uses dedicated CPU cores in "poll-mode" to read packets directly from the NIC, achieving significantly higher throughput. The guide details a four-step process: 1) Ensuring system prerequisites are met (supported NIC, BIOS settings). 2) Preparing the OS by allocating HugePages and enabling IOMMU via GRUB parameters. 3) Using the dpdk-devbind.py script to unbind the target network interface from its kernel driver and bind it to a DPDK driver like vfio-pci. 4) Configuring voipmonitor.conf with mandatory parameters, including dpdk=yes, interface=dpdk:0, dpdk_pci_device, and setting dedicated CPU cores with dpdk_read_thread_cpu_affinity. The article also covers advanced OS tuning, such as isolating CPU cores (isolcpus) and using a tickless kernel (nohz_full) for maximum, jitter-free performance.

Keywords: dpdk, performance, high throughput, packet capture, kernel bypass, poll-mode, dpdk-devbind, vfio-pci, igb_uio, hugepages, iommu, cpu affinity, isolcpus, nohz_full, tickless kernel, dpdk_pci_device, dpdk_read_thread_cpu_affinity, t0CPU, packet loss

Key Questions:

  • How can I capture more than 3 Gbit/s of traffic with VoIPmonitor?
  • What is DPDK and why should I use it?
  • How do I configure DPDK for VoIPmonitor?
  • What are HugePages and how do I configure them?
  • How do I bind a network card to the DPDK driver?
  • What does the dpdk-devbind.py script do?
  • What are the mandatory voipmonitor.conf settings for DPDK?
  • How do I isolate CPU cores for maximum packet capture performance?
  • What is a "tickless kernel" (nohz_full)?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=DPDK&oldid=9376"