DPDK: Difference between revisions

From VoIPmonitor.org
(Rewrite: consolidated GRUB config, added parameter reference table, See Also section, improved structure)
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= What is DPDK =
{{DISPLAYTITLE:High-Performance Packet Capture with DPDK}}


DPDK is the Data Plane Development Kit that consists of libraries to accelerate packet processing workloads running on a wide variety of CPU architectures. Designed to run on x86, POWER and ARM processors. Polling-mode drivers skips packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher packet throughput than is possible using the interrupt-driven processing provided in the kernel.
'''This is an expert-level guide for configuring the VoIPmonitor sensor to use the Data Plane Development Kit (DPDK) for ultra-high-performance packet capture. This setup is intended for multi-gigabit traffic loads where the standard Linux network stack becomes a bottleneck.'''


= Why DPDK for voipmonitor =  
== What is DPDK and Why Use It? ==


Sniffing packets by kernel linux is driven by IRQ interrupts - every packet (or if driver supports every set of packets) needs to be handled by interrupt which has limitation around 3Gbit on 10Gbit cards (it depends on CPU). DPDK allows to read pacekts directly in userspace not using interrupts which allows faster packet reading (so called poll-mode reading). It needs some tweaks to the operating system (cpu affinity / NOHZ kernel) as the reader thread is sensitive to any scheduler delays which can occur on overloaded or misconfigured system. For 6Gbit packet rate with 3 000 000 packets / second any slight delays can cause packet drops.  
The '''Data Plane Development Kit (DPDK)''' is a set of libraries and drivers that allows an application to bypass the operating system's kernel and interact directly with the network card hardware.


* '''Standard Kernel Method:''' Every incoming packet triggers a CPU interrupt (IRQ), telling the kernel to process it. This interrupt-driven model is reliable but creates significant overhead, typically maxing out around 2-3 Gbit/s on a 10Gbit NIC.
* '''DPDK Method:''' Uses '''poll-mode drivers'''. A dedicated CPU core constantly polls the NIC for new packets, completely avoiding interrupt overhead and context switching. This enables throughput of 6+ Gbit/s on a single server.


= installation =
<kroki lang="mermaid">
%%{init: {'flowchart': {'nodeSpacing': 15, 'rankSpacing': 40}}}%%
flowchart LR
    subgraph Standard["Standard Kernel Method"]
        direction TB
        NIC1[NIC] -->|IRQ| Kernel1[Linux Kernel]
        Kernel1 -->|Context Switch| App1[VoIPmonitor]
    end


Version >= DPDK 21.08.0 is requried - download the latest version from:
    subgraph DPDK["DPDK Poll-Mode"]
        direction TB
        NIC2[NIC] -->|Direct Access| PMD[Poll-Mode Driver]
        PMD -->|HugePages| App2[VoIPmonitor]
    end


https://core.dpdk.org/download/
    Standard -.->|"~2-3 Gbit/s"| Limit1((Bottleneck))
    DPDK -.->|"6+ Gbit/s"| Limit2((High Throughput))
</kroki>


{{Note|1=The trade-off is that DPDK requires careful system tuning and dedicated CPU cores. Scheduler delays on polling cores cause packet drops.}}


== Step 1: Hardware and System Prerequisites ==


= How it works =
{| class="wikitable"
|-
! Requirement !! Details
|-
| '''Supported NIC''' || Must be [https://core.dpdk.org/supported/ DPDK-compatible]. Intel X540/X710 series are common choices.
|-
| '''BIOS/UEFI''' || Enable '''VT-d''' (Intel) or '''AMD-Vi''' (AMD) virtualization technology. Enable '''IOMMU'''.
|-
| '''DPDK Version''' || VoIPmonitor requires DPDK 21.08.0 or newer. Download from [https://core.dpdk.org/download/ dpdk.org].
|}


On supported NIC cards (https://core.dpdk.org/supported/) the ethernet port needs to be unbinded from kernel and binded to DPDK, the command for it is:
== Step 2: System Preparation (GRUB Configuration) ==


* no special driver is needed - debian 10/11 already has support for this out of the box
DPDK requires HugePages for memory allocation and IOMMU for the VFIO driver.
* bind/unbind means that when you undind NIC port from the kernel you cannot use it within the operating system - the port dissapears (you will not see eth1 for example)
* you can unbind from dpdk and bind back to kernel so eth1 can be used again
* dpdk is referencing NIC port by the PCI address which you can get from the "dpdk-devbind.py -s" command for example


list of available network devices:
=== Complete GRUB Configuration ===


dpdk-devbind.py -s
Edit <code>/etc/default/grub</code> and add all required parameters:
Network devices using kernel driver
===================================
0000:0b:00.0 'NetXtreme II BCM5709 Gigabit Ethernet 1639' if=enp11s0f0 drv=bnx2 unused= *Active*
0000:0b:00.1 'NetXtreme II BCM5709 Gigabit Ethernet 1639' if=enp11s0f1 drv=bnx2 unused=
0000:1f:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=ens3f0 drv=ixgbe unused=
0000:1f:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' if=ens3f1 drv=ixgbe unused=


bind both 10gbit ports to vfio-pci driver (this driver is available by default on >= debian10)
<syntaxhighlight lang="bash">
# For Intel CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16 iommu=pt intel_iommu=on"


modprobe vfio-pci
# For AMD CPUs:
dpdk-devbind.py -b vfio-pci 0000:1f:00.0 0000:1f:00.1
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16 iommu=pt amd_iommu=on"
</syntaxhighlight>


bind B port back to kernel:  
Apply and reboot:
<syntaxhighlight lang="bash">
update-grub
reboot
</syntaxhighlight>


dpdk-devbind.py -b ixgbe 0000:1f:00.1
{{Warning|1=The <code>transparent_hugepage=never</code> parameter is '''critical'''. Without it, DPDK may fail with "Permission denied" errors after reboot.}}


On some systems vfio-pci does not work for 10Gbit card - instead igb_uio (for Intel cards) needs to be loaded alongside with special kernel parameters:
=== Verify Configuration ===


/etc/default/grub:
After reboot, verify:
<syntaxhighlight lang="bash">
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt intel_iommu=on"
# Check HugePages are allocated
cat /proc/meminfo | grep HugePages


Loading igb_uio for X540-AT2 4 port 10Gbit card (if vfio does not work)  
# Check IOMMU is active (should show subdirectories)
ls /sys/kernel/iommu_groups/
</syntaxhighlight>


modprobe igb_uio
{{Note|1=Allocate HugePages on the same NUMA node as your NIC. For runtime allocation: <code>echo 16 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages</code>}}


More information about drivers:  
== Step 3: Bind the Network Interface to DPDK ==


https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
Once the system is prepared, unbind the NIC from the kernel driver and bind it to DPDK. The interface will no longer be visible to the OS.


<syntaxhighlight lang="bash">
# 1. Find the PCI address of your NIC
dpdk-devbind.py -s


dpdk is now ready to be used by voipmonitor
# Output example:
# 0000:1f:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens3f0 drv=ixgbe unused=


== Some useful basic settings ==
# 2. Load the VFIO-PCI driver
modprobe vfio-pci


* by default you don't need to change anything. Just when things don't go well
# 3. Bind the NIC to DPDK
dpdk-devbind.py -b vfio-pci 0000:1f:00.0


Options for the sensor config:
# To unbind and return to kernel:
dpdk-devbind.py -u 0000:1f:00.0
dpdk-devbind.py -b ixgbe 0000:1f:00.0
</syntaxhighlight>


dpdk_nb_rxq ... number of queues where the packets from the dpdk interface is stored. Default is 2.
{{Note|1=If <code>vfio-pci</code> doesn't work, try the <code>igb_uio</code> driver (may require manual compilation). See [https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html DPDK driver documentation].}}


dpdk_pkt_burst ... number of packets processed in one run from the dpdk interface. Default is 128. If the value >= 1024 then the second worker thread is enabled automatically.
== Step 4: Configure VoIPmonitor ==


dpdk_worker_slave_thread ... second worker thead. Default is no. If enabled then the dpdk_pkt_burst option is set to 2048
=== Mandatory Parameters ===


* for more tweaks see OS tweaks' below
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf


= Troubleshooting =
dpdk = yes
interface = dpdk:0
dpdk_pci_device = 0000:1f:00.0


=== DPDK interface bind problem ===
# CPU cores must be on the same NUMA node as the NIC
dpdk_read_thread_cpu_affinity = 2
dpdk_worker_thread_cpu_affinity = 30
</syntaxhighlight>


Jul 23 07:51:02 voipmon kernel: [267244.930194] vfio-pci: probe of 0000:43:00.0 failed with error -22
=== Performance Tuning ===
Jul 23 07:51:02 voipmon kernel: [267244.930245] vfio-pci: probe of 0000:43:00.0 failed with error -22
Jul 23 07:51:06 voipmon kernel: [267248.595082] vfio-pci: probe of 0000:43:00.1 failed with error -22
Jul 23 07:51:06 voipmon kernel: [267248.595129] vfio-pci: probe of 0000:43:00.1 failed with error -22


* be sure that IOMMU is working. Try to add "iommu=pt intel_iommu=on" kernel's options to the Grub's config on the Intel's host. For the AMD's host adjust iommu options accordingly.
<syntaxhighlight lang="ini">
# Increase ring buffer to reduce imissed drops
dpdk_nb_rx = 16384


* be sure that VT-d is enabled in the BIOS
# Larger burst size for efficiency
dpdk_pkt_burst = 512


* when IOMMU is working correctly then directory /sys/kernel/iommu_groups/ should contain directories
# Enable RSS for multi-queue distribution
dpdk_nb_rxq = 4
dpdk_nb_rxq_rss = yes


root@voipmon:~# ls /sys/kernel/iommu_groups/
# Increase memory pool for >5Gbit traffic
0    102  107  111  116  120  125  13  134  139  143  148  152  157  161  166  170  175  18  22  27  31  36  40  45  5  54  59  63  68  72  77  81  86  90  95
dpdk_nb_mbufs = 4096
1    103  108  112  117  121  126  130  135  14  144  149  153  158  162  167  171  176  19  23  28  32  37  41  46  50  55  6  64  69  73  78  82  87  91  96
10  104  109  113  118  122  127  131  136  140  145  15  154  159  163  168  172  177  2  24  29  33  38  42  47  51  56  60  65  7  74  79  83  88  92  97
100  105  11  114  119  123  128  132  137  141  146  150  155  16  164  169  173  178  20  25  3  34  39  43  48  52  57  61  66  70  75  8  84  89  93  98
101  106  110  115  12  124  129  133  138  142  147  151  156  160  165  17  174  179  21  26  30  35  4  44  49  53  58  62  67  71  76  80  85  9  94  99


=== Hugepages mount problem ===
# Restrict other threads to non-DPDK cores
thread_affinity = 1,3-29,31-59
</syntaxhighlight>


* be sure that the hugepages are mounted in your system. You should see something like this.
== Troubleshooting: imissed Packet Drops ==


root@voipmon:~# mount | grep -i huge
The <code>imissed</code> counter indicates packets dropped by the NIC because they couldn't be read fast enough.
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)


= huge pages =
{{Warning|1=<code>imissed</code> drops occur at the NIC level, not in VoIPmonitor. The NIC's hardware buffer fills faster than packets are read.}}


DPDK requires huge pages which can be configured in two ways:  
'''Solutions:'''


* 16 GB huge pages allocated to numa node0 (first CPU which handles NIC card)  
{| class="wikitable"
|-
! Cause !! Solution
|-
| Ring buffer too small || <code>dpdk_nb_rx = 16384</code>
|-
| Burst size too low || <code>dpdk_pkt_burst = 512</code>
|-
| Poor packet distribution || Enable <code>dpdk_nb_rxq_rss = yes</code>
|-
| CPU scheduler interference || Use <code>isolcpus</code> (see below)
|}


echo 16 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
Monitor with: <code>tail -f /var/log/voipmonitor.log | grep imissed</code>


or permanantly add to /etc/default/grub: default_hugepagesz=1G hugepagesz=16G hugepages=100
== Advanced OS Tuning ==
but this will add hugepages evenly for all numanodes which you might not need as the DPDK needs huge pages only for its mbuffer which has to be allocated only on the numa node handling NIC card.


= OS tweaks =
For 6+ Gbit/s environments, isolate DPDK cores from the Linux scheduler:


== memory ==  
<syntaxhighlight lang="bash">
# Add to /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=2,30 nohz=on nohz_full=2,30 rcu_nocbs=2,30"
</syntaxhighlight>


In case of more physical CPU turn off numa balancing which causes memory latency
* <code>isolcpus</code>: Prevents scheduler from using these cores
* <code>nohz_full</code>: Disables timer interrupts on isolated cores (tickless)
* <code>rcu_nocbs</code>: Moves RCU callbacks off isolated cores


echo 0 > /proc/sys/kernel/numa_balancing
{{Note|1=<code>nohz_full</code> may require kernel compiled with <code>CONFIG_NO_HZ_FULL=y</code>.}}


Disable transparent huge pages which can cause latency or high TLB shootdowns
== Parameter Quick Reference ==


echo never > /sys/kernel/mm/transparent_hugepage/enabled
{| class="wikitable"
|-
or permanently - add transparent_hugepage=never to /etc/default/grub
! Parameter !! Default !! Description
|-
| <code>dpdk</code> || no || Enable DPDK mode
|-
| <code>interface</code> || - || Set to <code>dpdk:0</code> for DPDK
|-
| <code>dpdk_pci_device</code> || - || PCI address of NIC (e.g., 0000:1f:00.0)
|-
| <code>dpdk_read_thread_cpu_affinity</code> || - || CPU core for polling thread
|-
| <code>dpdk_worker_thread_cpu_affinity</code> || - || CPU core for worker thread
|-
| <code>dpdk_nb_rx</code> || 512 || Receive ring buffer size (increase to 16384 for high traffic)
|-
| <code>dpdk_pkt_burst</code> || 32 || Packets per burst (increase to 512 for efficiency)
|-
| <code>dpdk_nb_rxq</code> || 2 || Number of receive queues
|-
| <code>dpdk_nb_rxq_rss</code> || no || Enable RSS for multi-queue distribution
|-
| <code>dpdk_nb_mbufs</code> || 1024 || Memory pool size (x1024 segments, ~2GB default)
|}


== cpu affinity ==
== See Also ==


* Ideal configuration is that the sniffer dpdkd reader and worker thread will run on standalone CPU cores and denies all other processes to ever touch or run on those cores. This can be configured fully manually for every processes or system wide by kernel parameters:
* [[Scaling]] - General performance tuning guide
* [[Napatech]] - Alternative high-performance capture using Napatech SmartNICs
* [[Sniffer_configuration]] - Complete sniffer configuration reference


add to /etc/default/grub
== AI Summary for RAG ==


isolcpus=2,30
'''Summary:''' This guide provides an expert-level walkthrough for configuring VoIPmonitor with DPDK (Data Plane Development Kit) for high-performance packet capture on multi-gigabit networks (6+ Gbit/s). DPDK bypasses the interrupt-driven Linux kernel stack using dedicated CPU cores in "poll-mode" to read packets directly from the NIC. The setup requires: (1) DPDK-compatible NIC with VT-d/AMD-Vi enabled in BIOS; (2) GRUB configuration for HugePages (<code>hugepages=16</code>), IOMMU (<code>intel_iommu=on</code>), and critically <code>transparent_hugepage=never</code> to prevent permission errors; (3) Binding NIC to DPDK using <code>dpdk-devbind.py -b vfio-pci</code>; (4) VoIPmonitor config with <code>dpdk=yes</code>, <code>interface=dpdk:0</code>, <code>dpdk_pci_device</code>, and CPU affinity settings. For <code>imissed</code> packet drops, increase <code>dpdk_nb_rx=16384</code> and <code>dpdk_pkt_burst=512</code>. Advanced tuning uses <code>isolcpus</code> and <code>nohz_full</code> for complete CPU isolation.


'''Keywords:''' dpdk, performance, high throughput, packet capture, kernel bypass, poll-mode, dpdk-devbind, vfio-pci, igb_uio, hugepages, transparent_hugepage, iommu, cpu affinity, isolcpus, nohz_full, tickless kernel, dpdk_pci_device, dpdk_read_thread_cpu_affinity, dpdk_nb_rx, dpdk_pkt_burst, dpdk_nb_rxq_rss, imissed, packet loss, 10gbit, multi-gigabit


* we tell the kernel that no processes can run on 2 and 30 cores (in our case 2 and 30 is one physical core and hyperthread sybling. In voipmontior.conf : dpdk_read_thread_cpu_affinity = 2, dpdk_worker_thread_cpu_affinity = 30
'''Key Questions:'''
* How can I capture more than 3 Gbit/s of traffic with VoIPmonitor?
* What is DPDK and why should I use it?
* How do I configure DPDK for VoIPmonitor?
* What are HugePages and how do I configure them for DPDK?
* How do I bind a network card to the DPDK driver?
* What does the dpdk-devbind.py script do?
* What are the mandatory voipmonitor.conf settings for DPDK?
* How do I fix imissed packet drops with DPDK?
* What does transparent_hugepage=never do and why is it required?
* How do I isolate CPU cores for maximum packet capture performance?
* What is a tickless kernel (nohz_full)?
* What is the difference between vfio-pci and igb_uio drivers?


this was proven to work for 6Gbit traffic on  Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. For higher traffic or less powerfull CPU you might need to set reader and worker thread to two physical cores
[[Category:Configuration]]
 
[[Category:Installation]]
 
[[Category:Performance]]
* NOHZ kernel
 
For Best performance, use the data cores as isolated cpus and operate them in tickless mode on kernel version 4.4 above. For this compile the Kernel with CONFIG_NO_HZ_FULL=y (default debian kernels does not have this option)
We were able to achieve stable non packet loss reading from the NIC (6Gbit / 3000000 packets / sec)  but for high traffic this might bee needed.
 
The CONFIG_NO_HZ_FULL linux kernel build option is used to configure a tickless kernel. The idea is to configure certain processor cores to operate in tickless mode and these cores do not receive any periodic interrupts. These cores will run dedicated tasks (and no other tasks will be schedules on such cores obviating the need to send a scheduling tick). A CONFIG_HZ based timer interrupt will invalidate L1 cache on the core and this can degrade dataplane performance by a few % points (to be quantified, but estimated to be 1-3%). Running tickless typically means getting 1 timer interrupt/sec instead of 1000/sec.
 
Add to /etc/default/grub
 
nohz=on nohz_full=2,30 rcu_nocbs=2,30 rcu_nocb_poll clocksource=tsc
 
== other kernel tweaks ==
 
Add to /etc/default/grub
 
cpuidle.off=1 skew_tick=1 acpi_irq_nobalance idle=poll transparent_hugepage=never audit=0 nosoftlockup mce=ignore_ce mitigations=off selinux=0 nmi_watchdog=0
 
We are not sure if these has any impact but was recommended during DPDK implementations and testing (be aware that mitigations=off turns off security patches for discovered CPU security flaws)
 
 
= Sniffer configuration =
 
== mandatory parameters ==
 
*dpdk_read_thread_cpu_affinity sets on which CPU core will reader (polling NIC for packets) run.
*dpdk_worker_thread_cpu_affinity sets on which CPU core will worker run - it should run on hyperthread sibbling to core you set for dpdk_read_thread_core
*dpdk_pci_device  - what interface will be used for sniffing packets
*it is important to lock reader and worker threads to particular CPU cores so that sniffer will not use those cores for other threads
*in case of more NUMA nodes (two or more physical CPUs) always chose CPU cores for reader and worker thread which are on the same NUMA node for the NIC PCI card
 
 
voipmonitor.conf:  
interface = dpdk:0
dpdk = yes
dpdk_read_thread_cpu_affinity = 2
dpdk_worker_thread_cpu_affinity = 30
dpdk_pci_device = 0000:04:00.0
 
== optional parameters ==
 
thread_affinity = 1,3-5,4 ; this sets cpu affinity for the voipmonitor. It is automatically set to all cpu cores except dpdk_read_thread_core and dpdk_worker_thread_core. Using this option will override automatic cpu cores. You normally do not want to change this unless you decide to leave some cores dedicated to some other important processes on your server or if you want to hold sniffer on particular NUMA node.
dpdk_nb_rx = 4096 ; default size is 4096 if not specified. This is ring buffer on the NIC port. Maximum for Intel X540 is 4096 but it can be larger for others. You can get what is maximum by using ethtool -g eth1
dpdk_nb_tx = 1024 (we do not need ring buffer for sending, but dpdk wants to have this - default is 1024
dpdk_nb_mbufs = 1024 ; number of packets multiplied by 1024 between reader and worker (buffer size). Each packet size is around 2kb which means that it will allocate 2GB of RAM by default. Higher mbuf is recommended (4096) for >=5Gbit traffic
dpdk_pkt_burst = 32 ; do not change this unless you exactly know what you are doing
dpdk_mempool_cache_size = 512; size of the cache size for dpdk mempool (do not change this until you exactly know what you are doing)
dpdk_memory_channels = 4; number of memory bank channels - if not specified, dpdk uses default value (TODO: we are not sure if it tries to guess it or what is the default)
dpdk_force_max_simd_bitwidth = 512; default is not set - if you have CPU which supports AVX 512 and you have compiled dpdk with AVX 512 support you can try to enable this and set 512
 
== experimental parameters ==
 
dpdk_ring_size = ; number of packets * 1024 in ring buffer which holds references to mbuf structures between worker thread and voipmonitor's packet buffer. If not specified it equels to dpdk_nb_mbufs

Latest revision as of 16:48, 8 January 2026


This is an expert-level guide for configuring the VoIPmonitor sensor to use the Data Plane Development Kit (DPDK) for ultra-high-performance packet capture. This setup is intended for multi-gigabit traffic loads where the standard Linux network stack becomes a bottleneck.

What is DPDK and Why Use It?

The Data Plane Development Kit (DPDK) is a set of libraries and drivers that allows an application to bypass the operating system's kernel and interact directly with the network card hardware.

  • Standard Kernel Method: Every incoming packet triggers a CPU interrupt (IRQ), telling the kernel to process it. This interrupt-driven model is reliable but creates significant overhead, typically maxing out around 2-3 Gbit/s on a 10Gbit NIC.
  • DPDK Method: Uses poll-mode drivers. A dedicated CPU core constantly polls the NIC for new packets, completely avoiding interrupt overhead and context switching. This enables throughput of 6+ Gbit/s on a single server.

ℹ️ Note: The trade-off is that DPDK requires careful system tuning and dedicated CPU cores. Scheduler delays on polling cores cause packet drops.

Step 1: Hardware and System Prerequisites

Requirement Details
Supported NIC Must be DPDK-compatible. Intel X540/X710 series are common choices.
BIOS/UEFI Enable VT-d (Intel) or AMD-Vi (AMD) virtualization technology. Enable IOMMU.
DPDK Version VoIPmonitor requires DPDK 21.08.0 or newer. Download from dpdk.org.

Step 2: System Preparation (GRUB Configuration)

DPDK requires HugePages for memory allocation and IOMMU for the VFIO driver.

Complete GRUB Configuration

Edit /etc/default/grub and add all required parameters: 

# For Intel CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16 iommu=pt intel_iommu=on"

# For AMD CPUs:
GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never default_hugepagesz=1G hugepagesz=1G hugepages=16 iommu=pt amd_iommu=on"

Apply and reboot: 

update-grub
reboot

⚠️ Warning: The transparent_hugepage=never parameter is critical. Without it, DPDK may fail with "Permission denied" errors after reboot.

Verify Configuration

After reboot, verify: 

# Check HugePages are allocated
cat /proc/meminfo | grep HugePages

# Check IOMMU is active (should show subdirectories)
ls /sys/kernel/iommu_groups/

ℹ️ Note: Allocate HugePages on the same NUMA node as your NIC. For runtime allocation: echo 16 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages

Step 3: Bind the Network Interface to DPDK

Once the system is prepared, unbind the NIC from the kernel driver and bind it to DPDK. The interface will no longer be visible to the OS. 

# 1. Find the PCI address of your NIC
dpdk-devbind.py -s

# Output example:
# 0000:1f:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens3f0 drv=ixgbe unused=

# 2. Load the VFIO-PCI driver
modprobe vfio-pci

# 3. Bind the NIC to DPDK
dpdk-devbind.py -b vfio-pci 0000:1f:00.0

# To unbind and return to kernel:
dpdk-devbind.py -u 0000:1f:00.0
dpdk-devbind.py -b ixgbe 0000:1f:00.0

ℹ️ Note: If vfio-pci doesn't work, try the igb_uio driver (may require manual compilation). See DPDK driver documentation.

Step 4: Configure VoIPmonitor

Mandatory Parameters

# /etc/voipmonitor.conf

dpdk = yes
interface = dpdk:0
dpdk_pci_device = 0000:1f:00.0

# CPU cores must be on the same NUMA node as the NIC
dpdk_read_thread_cpu_affinity = 2
dpdk_worker_thread_cpu_affinity = 30

Performance Tuning

# Increase ring buffer to reduce imissed drops
dpdk_nb_rx = 16384

# Larger burst size for efficiency
dpdk_pkt_burst = 512

# Enable RSS for multi-queue distribution
dpdk_nb_rxq = 4
dpdk_nb_rxq_rss = yes

# Increase memory pool for >5Gbit traffic
dpdk_nb_mbufs = 4096

# Restrict other threads to non-DPDK cores
thread_affinity = 1,3-29,31-59

Troubleshooting: imissed Packet Drops

The imissed counter indicates packets dropped by the NIC because they couldn't be read fast enough.

⚠️ Warning: imissed drops occur at the NIC level, not in VoIPmonitor. The NIC's hardware buffer fills faster than packets are read.

Solutions:

Cause Solution
Ring buffer too small dpdk_nb_rx = 16384
Burst size too low dpdk_pkt_burst = 512
Poor packet distribution Enable dpdk_nb_rxq_rss = yes
CPU scheduler interference Use isolcpus (see below)

Monitor with: tail -f /var/log/voipmonitor.log | grep imissed

Advanced OS Tuning

For 6+ Gbit/s environments, isolate DPDK cores from the Linux scheduler: 

# Add to /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="... isolcpus=2,30 nohz=on nohz_full=2,30 rcu_nocbs=2,30"
  • isolcpus: Prevents scheduler from using these cores
  • nohz_full: Disables timer interrupts on isolated cores (tickless)
  • rcu_nocbs: Moves RCU callbacks off isolated cores

ℹ️ Note: nohz_full may require kernel compiled with CONFIG_NO_HZ_FULL=y.

Parameter Quick Reference

Parameter Default Description
dpdk no Enable DPDK mode
interface - Set to dpdk:0 for DPDK
dpdk_pci_device - PCI address of NIC (e.g., 0000:1f:00.0)
dpdk_read_thread_cpu_affinity - CPU core for polling thread
dpdk_worker_thread_cpu_affinity - CPU core for worker thread
dpdk_nb_rx 512 Receive ring buffer size (increase to 16384 for high traffic)
dpdk_pkt_burst 32 Packets per burst (increase to 512 for efficiency)
dpdk_nb_rxq 2 Number of receive queues
dpdk_nb_rxq_rss no Enable RSS for multi-queue distribution
dpdk_nb_mbufs 1024 Memory pool size (x1024 segments, ~2GB default)

See Also

  • Scaling - General performance tuning guide
  • Napatech - Alternative high-performance capture using Napatech SmartNICs
  • Sniffer_configuration - Complete sniffer configuration reference

AI Summary for RAG

Summary: This guide provides an expert-level walkthrough for configuring VoIPmonitor with DPDK (Data Plane Development Kit) for high-performance packet capture on multi-gigabit networks (6+ Gbit/s). DPDK bypasses the interrupt-driven Linux kernel stack using dedicated CPU cores in "poll-mode" to read packets directly from the NIC. The setup requires: (1) DPDK-compatible NIC with VT-d/AMD-Vi enabled in BIOS; (2) GRUB configuration for HugePages (hugepages=16), IOMMU (intel_iommu=on), and critically transparent_hugepage=never to prevent permission errors; (3) Binding NIC to DPDK using dpdk-devbind.py -b vfio-pci; (4) VoIPmonitor config with dpdk=yes, interface=dpdk:0, dpdk_pci_device, and CPU affinity settings. For imissed packet drops, increase dpdk_nb_rx=16384 and dpdk_pkt_burst=512. Advanced tuning uses isolcpus and nohz_full for complete CPU isolation.

Keywords: dpdk, performance, high throughput, packet capture, kernel bypass, poll-mode, dpdk-devbind, vfio-pci, igb_uio, hugepages, transparent_hugepage, iommu, cpu affinity, isolcpus, nohz_full, tickless kernel, dpdk_pci_device, dpdk_read_thread_cpu_affinity, dpdk_nb_rx, dpdk_pkt_burst, dpdk_nb_rxq_rss, imissed, packet loss, 10gbit, multi-gigabit

Key Questions:

  • How can I capture more than 3 Gbit/s of traffic with VoIPmonitor?
  • What is DPDK and why should I use it?
  • How do I configure DPDK for VoIPmonitor?
  • What are HugePages and how do I configure them for DPDK?
  • How do I bind a network card to the DPDK driver?
  • What does the dpdk-devbind.py script do?
  • What are the mandatory voipmonitor.conf settings for DPDK?
  • How do I fix imissed packet drops with DPDK?
  • What does transparent_hugepage=never do and why is it required?
  • How do I isolate CPU cores for maximum packet capture performance?
  • What is a tickless kernel (nohz_full)?
  • What is the difference between vfio-pci and igb_uio drivers?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=DPDK&oldid=10102"