CountryGrouping: Difference between revisions

From VoIPmonitor.org
(Add common configuration errors section: country prefixes/rules tab, /tmp noexec mount, and multi-sensor settings)
(Add documentation for country_code and country_code_prefix database tables)
Line 1: Line 1:
= Country grouping =
{{DISPLAYTITLE:Scaling and Performance Tuning}}
Category:Administration


== Settings ==
This guide provides a comprehensive overview of performance tuning and scaling for VoIPmonitor. It covers the three primary system bottlenecks and offers practical, expert-level advice for optimizing your deployment for high traffic loads.


Country assign is implemented since sniffer 18.2 and GUI 16.11 and it assigns country to SIP source IP / destination IP and to called / caller number included in the first SIP INVITE.
== Understanding Performance Bottlenecks ==
A VoIPmonitor deployment's maximum capacity is determined by three potential bottlenecks. Identifying and addressing the correct one is key to achieving high performance.


=== Required Sniffer Configuration ===
<kroki lang="plantuml">
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #4A90E2
  BackgroundColor #FFFFFF
}


For country flags to appear in the CDR view, you must enable country code processing on each sniffer sensor. Edit the sniffer configuration file (typically <code>/etc/voipmonitor.conf</code>)
title VoIPmonitor Performance Bottlenecks


<pre>
rectangle "Network\nInterface" as NIC #E8F4FD
cdr_country_code = yes
rectangle "Packet Capture\n(t0 thread)" as T0 #FFE6E6
</pre>
rectangle "RTP/SIP\nProcessing" as PROC #E6FFE6
rectangle "PCAP Files\nStorage" as DISK #FFF3E6
database "MySQL/MariaDB\nDatabase" as DB #E6E6FF


If this is set to <code>no</code> or commented out, country flags will not appear in the CDR view even if database number lookup is enabled and country prefixes are configured. After changing this setting, restart the sniffer service on the sensor for the changes to take effect.
NIC -right-> T0 : "1. CPU\nBottleneck"
T0 -right-> PROC
PROC -down-> DISK : "2. I/O\nBottleneck"
PROC -right-> DB : "3. Database\nBottleneck"


=== GUI Configuration ===
note bottom of T0
  Monitor: t0CPU in syslog
  Limit: Single CPU core
end note


For prefix matching, you need to set local country in Settings -> common settings:
note bottom of DISK
  Monitor: iostat, ioping
  Solution: SSD, TAR archives
end note


*international prefixes - if calling to international destinations is prefixed with for example "00" put there 00
note bottom of DB
*min. international length - if you do not use prefix for calling to international destinations you might want to set here 10 which will treat every numbers longer than >= 10 as international number
  Monitor: SQLq in syslog
*local numbers are in - choose to which country belongs number which does not match international prefix or min. international length
  Solution: Partitioning, tuning
*trim prefixes - if you are using prefixes to call to special destinations or trunks (like 999 prefix) you should list it there (delimited with space)
end note
@enduml
</kroki>


=== Troubleshooting ===
The three bottlenecks are:
# '''Packet Capturing (CPU & Network Stack):''' The ability of a single CPU core to read packets from the network interface. This is often the first limit encountered.
# '''Disk I/O (Storage):''' The speed at which the sensor can write PCAP files to disk. Critical when call recording is enabled.
# '''Database Performance (MySQL/MariaDB):''' The rate at which the database can ingest CDRs and serve data to the GUI.


If country flags are not appearing in the CDR view:
On a modern, well-tuned server (e.g., 24-core Xeon, 10Gbit NIC), a single VoIPmonitor instance can handle up to '''10,000 concurrent calls''' with full RTP analysis and recording, or over '''60,000 concurrent calls''' with SIP-only analysis.


# Verify <code>cdr_country_code = yes</code> is set in <code>/etc/voipmonitor.conf</code> on each sensor
== Optimizing Packet Capturing (CPU & Network) ==
# Restart the sniffer service after changing the configuration
The most performance-critical task is the initial packet capture, handled by a single, highly optimized thread (t0). If this thread's CPU usage (<code>t0CPU</code> in logs) approaches 100%, you are hitting the capture limit.
# Check that "database number lookup" is enabled in Settings -> System Configuration
# Verify country prefixes are configured correctly in the GUI
# Make sure new calls are processed after the setting change (older CDRs will not show flags retroactively)


=== Common Configuration Errors ===
=== Use a Modern Linux Kernel & VoIPmonitor Build ===
Modern Linux kernels (3.2+) and VoIPmonitor builds include '''TPACKET_V3''' support, a high-speed packet capture mechanism. This is the single most important factor for high performance.


==== Incorrect Use of Country Prefixes/Rules Tab ===
'''Recommendation:''' Always use a recent Linux distribution (AlmaLinux, Rocky Linux, or Debian) and the latest VoIPmonitor static binary. With this combination, a standard Intel 10Gbit NIC can often handle up to 2 Gbit/s of VoIP traffic without special drivers.


The "Country prefixes/rules" tab in the GUI is designed for '''exceptions only'''. Do not place standard country codes (e.g., "32" for Belgium, "44" for UK, etc.) in this tab. Standard country codes belong in the main "country prefixes" table.
=== Network Stack & Driver Tuning ===
For high-traffic environments (>500 Mbit/s), fine-tuning the network driver and kernel parameters is essential.


Placing standard country codes in the "prefixes/rules" tab can interfere with proper country code assignment and filtering. Only use this tab for special cases or numbering rules that deviate from standard country code patterns.
==== NIC Ring Buffer ====
The ring buffer is a queue between the network card driver and VoIPmonitor. A larger buffer prevents packet loss during short CPU usage spikes.


==== /tmp Directory Mounted with noexec ===
<syntaxhighlight lang="bash">
# Check maximum size
ethtool -g eth0
 
# Set to maximum (e.g., 16384)
ethtool -G eth0 rx 16384
</syntaxhighlight>
 
==== Interrupt Coalescing ====
This setting batches multiple hardware interrupts into one, reducing CPU overhead.
 
<syntaxhighlight lang="bash">
ethtool -C eth0 rx-usecs 1022
</syntaxhighlight>
 
==== Applying Settings Persistently ====
To make these settings permanent, add them to your network configuration. For Debian/Ubuntu using <code>/etc/network/interfaces</code>:
 
<syntaxhighlight lang="ini">
auto eth0
iface eth0 inet manual
    up ip link set $IFACE up
    up ip link set $IFACE promisc on
    up ethtool -G $IFACE rx 16384
    up ethtool -C $IFACE rx-usecs 1022
</syntaxhighlight>
 
Note: Modern systems using NetworkManager or systemd-networkd require different configuration methods.
 
==== Configuration-Level Optimizations ====
Before investing in kernel-bypass solutions, ensure your <code>voipmonitor.conf</code> is optimized for performance. Several configuration parameters can significantly reduce CPU load and improve packet capture efficiency.
 
;Use interface_ip_filter Instead of filter
:If you need to filter by IP address or subnet, use <code>interface_ip_filter</code> instead of the general BPF <code>filter</code> option. The <code>interface_ip_filter</code> directive is more efficient and reduces CPU overhead compared to complex BPF filters.
 
<syntaxhighlight lang="ini">
# More efficient IP-based filtering
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8
 
# Less efficient BPF filtering (avoid if possible)
# filter = udp and (host 192.168.0.0/24 or host 10.0.0.0/8)
</syntaxhighlight>
 
{{Note|See [[Sniffer_configuration]] for the complete reference and description of <code>interface_ip_filter</code> (Interface Selection section).}}
 
;Optimize PCAP Compression Threads
:For systems with high call recording rates, PCAP compression can become CPU-intensive. VoIPmonitor can automatically scale compression threads.
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Initial compression threads (auto-scales based on load)
pcap_dump_writethreads = 1
 
# Maximum compression threads (adjust based on CPU cores)
pcap_dump_writethreads_max = 32
 
# Asynchronous PCAP writing (enabled by default)
pcap_dump_asyncwrite = yes
</syntaxhighlight>
 
{{Tip|Set <code>pcap_dump_writethreads_max</code> to the number of CPU cores available for best performance on multi-core systems. Monitor <code>t0CPU</code> to ensure compression threads are not competing with the capture thread.}}
 
;Adjust Jitterbuffer Settings Based on Traffic Patterns
:Jitterbuffer simulation adds CPU overhead. For production environments with stable networks, consider adjusting jitterbuffer settings to balance accuracy with performance.
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Fixed 50ms jitterbuffer (default: yes)
jitterbuffer_f1 = yes
 
# Fixed 200ms jitterbuffer (default: yes)
jitterbuffer_f2 = yes
 
# Adaptive jitterbuffer up to 500ms (default: yes)
jitterbuffer_adapt = yes
</syntaxhighlight>
 
{{Warning|Disabling jitterbuffer analysis reduces CPU load but removes MOS and jitter quality metrics from CDRs. Only disable if you do not require voice quality monitoring.}}
 
=== Advanced Kernel-Bypass Solutions ===
If kernel and driver tuning are insufficient, you can offload the capture process entirely by bypassing the kernel's network stack.
 
{| class="wikitable"
|-
! Solution !! Type !! CPU Reduction !! Use Case
|-
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
|-
| '''PF_RING ZC/DNA''' || Commercial || 90% → 20% || High-volume enterprise
|-
| '''Napatech SmartNICs''' || Hardware || <3% at 10 Gbit/s || Extreme performance requirements
|}
 
;DPDK (Data Plane Development Kit)
:A set of libraries and drivers for fast packet processing. VoIPmonitor can leverage DPDK to read packets directly from the network card, completely bypassing the kernel. See [[DPDK|DPDK guide]] for details.
 
;PF_RING ZC/DNA
:A commercial software driver from ntop.org that dramatically reduces CPU load by bypassing the kernel.
 
;Napatech SmartNICs
:Specialized hardware acceleration cards that deliver packets with near-zero CPU overhead.
 
== Optimizing Disk I/O ==
VoIPmonitor's modern storage engine is highly optimized to minimize random disk access, which is the primary cause of I/O bottlenecks.
 
=== VoIPmonitor Storage Strategy ===
Instead of writing a separate PCAP file for each call (which causes massive I/O load), VoIPmonitor groups all calls starting within the same minute into a single compressed <code>.tar</code> archive. This changes the I/O pattern from thousands of small, random writes to a few large, sequential writes, reducing IOPS by a factor of 10 or more.
 
'''Typical capacity:''' A standard 7200 RPM SATA drive can handle up to 2,000 concurrent calls with full recording.


If you cannot restart or manage the sniffer service from the GUI (e.g., service restarts or upgrades fail), check the <code>/tmp</code> directory mount options:
=== Filesystem Tuning (ext4) ===
For the spool directory (<code>/var/spool/voipmonitor</code>), using an optimized ext4 filesystem can improve performance.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mount | grep /tmp
# Format partition without a journal (use with caution, requires battery-backed RAID controller)
mke2fs -t ext4 -O ^has_journal /dev/sda2
 
# Add to /etc/fstab for optimal performance
/dev/sda2  /var/spool/voipmonitor  ext4    errors=remount-ro,noatime,data=writeback,barrier=0 0 0
</syntaxhighlight>
 
{{Warning|Disabling the journal removes protection against filesystem corruption after crashes. Only use this with a battery-backed RAID controller.}}
 
=== RAID Controller Cache Policy ===
A misconfigured RAID controller is a common bottleneck. For database and spool workloads, the cache policy should be set to '''WriteBack''', not WriteThrough. This applies for RPM disks, not fast SSDs.
 
'''Requirements:'''
* A healthy Battery Backup Unit (BBU) is required
* Specific commands vary by vendor (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>)
* Refer to vendor documentation for LSI, HP, and Dell controllers
 
== Optimizing Database Performance (MySQL/MariaDB) ==
A well-tuned database is critical for both data ingestion from the sensor and GUI responsiveness.
 
{{Note|For extreme scenarios (4,000+ concurrent calls, UI lag, high SQL queue, or 1000+ CDRs/sec), see [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, and centralized writer architecture.}}
 
=== Memory Configuration ===
The most critical database parameter is <code>innodb_buffer_pool_size</code>, which defines how much memory InnoDB uses to cache data and indexes.
 
{{Warning|On servers running both VoIPmonitor and MySQL, setting <code>innodb_buffer_pool_size</code> too high causes OOM (Out of Memory) killer events, resulting in CDR delays, crashes, and instability. See [[Sniffer_troubleshooting#Check_for_OOM_.28Out_of_Memory.29_Issues|OOM Troubleshooting]] for details.}}
 
==== Buffer Pool Sizing ====
 
{| class="wikitable"
|-
! Server Type !! Calculation !! Example (32GB RAM)
|-
| '''Shared''' (VoIPmonitor + MySQL) || (Total RAM - VoIPmonitor - OS overhead) / 2 || 14GB
|-
| '''Dedicated''' MySQL server || 50-70% of total RAM || 20-22GB
|}
 
For shared servers, use this formula:
<syntaxhighlight lang="text">
innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead - safety margin) / 2
 
Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory: ~2GB (check with ps aux)
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G
</syntaxhighlight>
</syntaxhighlight>


If the output shows <code>noexec</code> flag (e.g., <code>tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec)</code>), the GUI cannot execute management scripts. Remount <code>/tmp</code> without the <code>noexec</code> flag:
==== RAM Recommendations ====
{| class="wikitable"
|-
! Deployment Size !! Minimum RAM !! Recommended RAM
|-
| Small (<500 concurrent calls) || 8GB || 16GB
|-
| Medium (500-2000 calls) || 16GB || 32GB
|-
| Large (>2000 calls) || 32GB || 64GB+
|}
 
==== Disable Graphical Desktop ====
A graphical desktop environment consumes 1-2GB of RAM unnecessarily. VoIPmonitor is managed through a web interface and does not require a desktop.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mount -o remount,exec /tmp
# Disable display manager
systemctl stop gdm          # Ubuntu/Debian with GDM
systemctl disable gdm
 
# Set default to multi-user (no GUI)
systemctl set-default multi-user.target
 
# Verify memory freed
free -h
</syntaxhighlight>
</syntaxhighlight>


To make this change persistent, edit your <code>/etc/fstab</code> and remove the <code>noexec</code> option from the <code>/tmp</code> mount entry.
=== Other Key Parameters ===
 
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
 
[mysqld]
# Buffer pool size (calculate per above)
innodb_buffer_pool_size = 14G
 
# Flush logs to OS cache, write to disk once per second (faster, minimal data loss risk)
innodb_flush_log_at_trx_commit = 2
 
# Store each table in its own file (essential for partitioning)
innodb_file_per_table = 1


==== Multi-Sensor Setup Recommendations ===
# LZ4 compression for modern MariaDB
innodb_compression_algorithm = lz4
</syntaxhighlight>


In distributed architecture with multiple sensor probes and a central GUI host, consider adding these settings to the sensor probes' <code>/etc/voipmonitor.conf</code>:
{{Warning|For deployments with 4,000+ concurrent calls experiencing UI unresponsiveness, database queue growth (SQLq), or extremely high CDR insertion rates (1000+ CDRs/sec), the intermediate settings above may not be sufficient. See [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for extreme performance configurations optimized for 10,000+ concurrent calls, including innodb_flush_log_at_trx_commit=0, hourly partitioning, and centralized writer architecture.}}


<pre>
=== Slow Query Log ===
disable_partition_operations = yes
disable_dbupgradecheck = yes
</pre>


These settings prevent redundant database operations on sensors, leaving the GUI host to manage partition management and database upgrades. This is especially important in setups where the GUI host has write access to the database while sensors only collect data.
The MySQL slow query log can consume significant memory and disk I/O on high-traffic systems. If you are experiencing high memory utilization alerts or performance issues with the database server, consider adjusting or disabling the slow query log.


= CDR =
{{Warning|Disabling the slow query log removes the ability to analyze slow queries for performance optimization. Only disable it temporarily or if you are certain you do not need it.}}


== Filtering ==
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/my.cnf.d/mysql-server.cnf


You can now filter by country IP addresses or caller/called country prefixes
[mysqld]
# Disable slow query log (set to 1 to enable)
slow_query_log = 0


[[File:country_cdr_filter.png]]
# Alternative: Increase threshold to only log extremely slow queries (e.g., 600 seconds = 10 minutes)
long_query_time = 600
</syntaxhighlight>


== CDR grid ==
After changing MySQL configuration, restart the database and dependent services:


In CDR section you should be able to see country flags in Caller and Called columns next to source and destination IP addresses and next to source / destination numbers.
<syntaxhighlight lang="bash">
# Restart MySQL/MariaDB
systemctl restart mariadb  # or mysql


[[File:country_cdr_brief.png]]
# Restart VoIPmonitor sniffer (depends on database)
systemctl restart voipmonitor
</syntaxhighlight>


== CDR group panel ==
=== Database Partitioning ===
VoIPmonitor automatically splits large tables (like <code>cdr</code>) into daily partitions. This is enabled by default and '''highly recommended'''.


There are new grouping options - country by numbers or by ip address (src/dst)  
'''Benefits:'''
* Massively improves GUI query performance (only relevant partitions are scanned)
* Allows instant deletion of old data by dropping partitions (thousands of times faster than DELETE)


[[File:country_cdr_group.png]]
See [[Data_Cleaning#The_Modern_Method:_Partitioning_.28Recommended.29|Database Partitioning]] for configuration details.


== CDR top calls ==
== Monitoring Live Performance ==
VoIPmonitor logs detailed performance metrics every 10 seconds to syslog.


In CDR group panel tab -> top calls there are also grouping by country src/dst IP / numbers
<syntaxhighlight lang="bash">
# Debian/Ubuntu
tail -f /var/log/syslog | grep voipmonitor


[[File:Country_cdr_group_topcalls.png]]
# CentOS/RHEL
tail -f /var/log/messages | grep voipmonitor
</syntaxhighlight>


= Dashboard =
=== Understanding the Log Output ===
Sample log line:
<syntaxhighlight lang="text">
voipmonitor[15567]: calls[315][355] PS[C:4 S:29/29 R:6354 A:6484] SQLq[0] heap[0|0|0] comp[54] [12.6Mb/s] t0CPU[5.2%] ... RSS/VSZ[323|752]MB
</syntaxhighlight>


In dashboard you can create custom CDR grid grouped by country IP/number
{| class="wikitable"
|-
! Metric !! Description !! Warning Threshold
|-
| <code>calls[X][Y]</code> || X = active calls, Y = total calls in memory || -
|-
| <code>SQLq[C]</code> || SQL queries waiting to be sent to database || Growing consistently = DB bottleneck
|-
| <code>heap[A{{!}}B{{!}}C]</code> || Memory usage % for internal buffers || A = 100% → packet drops
|-
| <code>t0CPU[X%]</code> || '''Main packet capture thread CPU usage''' || >90-95% = capture limit reached
|-
| <code>RSS/VSZ[X{{!}}Y]MB</code> || Resident/Virtual memory usage || RSS growing = memory leak
|}


[[File:country_dashboard_customgrid.png]]
=== Performance Diagrams ===


= Report =
The following diagrams illustrate the difference between standard kernel packet capture and optimized solutions:


You can create report which will show the same table as in dashboard
[[File:kernelstandarddiagram.png|thumb|center|600px|Standard kernel packet capture path - packets traverse multiple kernel layers before reaching VoIPmonitor]]


[[File:country_report_form.png]]
[[File:ntop.png|thumb|center|600px|PF_RING/DPDK bypass mode - packets are delivered directly to VoIPmonitor, bypassing the kernel network stack]]


[[File:country_report.png]]
== See Also ==
* [[Sniffer_troubleshooting]] - Troubleshooting guide including OOM issues
* [[Data_Cleaning]] - Database and spool retention configuration
* [[Sniffer_configuration]] - Complete configuration reference
* [[DPDK]] - DPDK setup guide
* [[IO_Measurement]] - Disk I/O benchmarking tools


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' Country grouping assigns country codes to SIP source/destination IP addresses and to called/caller numbers based on international prefixes. This feature requires both sniffer configuration and GUI setup. The most critical requirement is setting <code>cdr_country_code = yes</code> in /etc/voipmonitor.conf on each sensor - without this setting, country flags will NOT appear in the CDR view even if all other settings are configured. Additionally, enable "database number lookup" in the GUI, and configure the number classification rules: international prefixes (e.g., "00" or "+"), minimum international length (for numbers without prefixes), local country assignment for unmatched numbers, and trim prefixes to remove trunk codes (e.g., stripping leading "0"). Common configuration errors to avoid: placing standard country codes in the "Country prefixes/rules" tab (designed for exceptions only), mounting /tmp with noexec flag (prevents GUI from managing sniffer services), and not restarting sniffer after configuration changes. In multi-sensor setups, add <code>disable_partition_operations = yes</code> and <code>disable_dbupgradecheck = yes</code> to sensor probes to prevent redundant database operations.
'''Summary:''' Expert guide to scaling VoIPmonitor for high-traffic environments. Covers three main bottlenecks: (1) Packet Capturing - optimized via TPACKET_V3, NIC tuning with ethtool (ring buffer, interrupt coalescing), configuration-level optimizations (interface_ip_filter more efficient than BPF filter, pcap_dump_writethreads for compression thread tuning, jitterbuffer settings for CPU/performance balance), and kernel-bypass solutions (DPDK, PF_RING, Napatech); (2) Disk I/O - VoIPmonitor uses TAR-based storage to reduce IOPS, with ext4 tuning and RAID WriteBack cache; (3) Database - critical innodb_buffer_pool_size tuning with formula for shared servers: (Total RAM - VoIPmonitor - OS overhead) / 2. For 32GB shared server, recommend 14GB buffer pool. Dedicated servers can use 50-70% of RAM. Covers slow query log as a memory/I/O consumer and disabling it for memory optimization. Covers partitioning benefits and syslog monitoring (t0CPU, SQLq, heap metrics). For extreme scenarios (4,000+ concurrent calls, UI lag, unresponsive GUI, high SQL queue), see High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, centralized writer architecture, RTP thread tuning (rtpthreads, rtpthreads_start), and MySQL optimization settings (innodb_thread_concurrency, innodb_io_capacity, innodb_flush_method=O_DIRECT).


'''Keywords:''' country grouping, country flags, CDR view, cdr_country_code, international prefixes, local country, trim prefixes, database number lookup, country filtering, number classification, voipmonitor.conf, sniffer configuration, country prefixes/rules tab, noexec mount, disable_partition_operations, multi-sensor setup, troubleshooting
'''Keywords:''' scaling, performance tuning, bottleneck, t0CPU, TPACKET_V3, DPDK, PF_RING, ethtool, ring buffer, interface_ip_filter, BPF filter, pcap_dump_writethreads, jitterbuffer, jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt, compression threads, PCAP async write, innodb_buffer_pool_size, OOM killer, shared server memory, database partitioning, SQLq monitoring, slow query log, slow_query_log, long_query_time, UI lag, unresponsive GUI, high performance, 4000 concurrent calls, 5000 concurrent calls, innodb_flush_log_at_trx_commit=0, hourly partitioning, rtpthreads, rtpthreads_start, RTP threads, innodb_io_capacity, innodb_thread_concurrency, innodb_flush_method, extreme performance, High-Performance Manual


'''Key Questions:'''
'''Key Questions:'''
* Why are country flags not appearing in the CDR view?
* How do I scale VoIPmonitor for thousands of concurrent calls?
* How do I enable country flags in VoIPmonitor?
* What are the main performance bottlenecks in VoIPmonitor?
* What is the cdr_country_code setting?
* How do I fix high t0CPU usage?
* How do I configure international prefixes and local country?
* What is DPDK and when should I use it?
* What is the difference between international prefixes and local country?
* How do I calculate innodb_buffer_pool_size for a shared server?
* How do I filter CDRs by country?
* What happens if innodb_buffer_pool_size is set too high?
* How do I enable country grouping for SIP IP addresses?
* How do I interpret the performance metrics in syslog?
* How do I configure country classification for phone numbers?
* Should I use a dedicated database server for VoIPmonitor?
* Why is the Country prefixes/rules tab for exceptions only?
* How much RAM does a VoIPmonitor server need?
* Can I place standard country codes in the.Country prefixes/rules tab?
* How can the slow query log affect memory utilization?
* How do I fix /tmp mounted with noexec flag for GUI service management?
* How do I disable or adjust the MySQL slow query log?
* What settings should I add to sensors in a multi-sensor setup?
* Is interface_ip_filter more efficient than the filter option?
* How do I prevent redundant database operations on sensor probes?
* How do I optimize PCAP compression threads for high traffic?
* Why is GUI unable to restart sniffer service from web interface?
* Which jitterbuffer settings affect CPU load the most?
* What configuration options reduce CPU overhead?

Revision as of 04:09, 6 January 2026

Category:Administration

This guide provides a comprehensive overview of performance tuning and scaling for VoIPmonitor. It covers the three primary system bottlenecks and offers practical, expert-level advice for optimizing your deployment for high traffic loads.

Understanding Performance Bottlenecks

A VoIPmonitor deployment's maximum capacity is determined by three potential bottlenecks. Identifying and addressing the correct one is key to achieving high performance.

The three bottlenecks are:

  1. Packet Capturing (CPU & Network Stack): The ability of a single CPU core to read packets from the network interface. This is often the first limit encountered.
  2. Disk I/O (Storage): The speed at which the sensor can write PCAP files to disk. Critical when call recording is enabled.
  3. Database Performance (MySQL/MariaDB): The rate at which the database can ingest CDRs and serve data to the GUI.

On a modern, well-tuned server (e.g., 24-core Xeon, 10Gbit NIC), a single VoIPmonitor instance can handle up to 10,000 concurrent calls with full RTP analysis and recording, or over 60,000 concurrent calls with SIP-only analysis.

Optimizing Packet Capturing (CPU & Network)

The most performance-critical task is the initial packet capture, handled by a single, highly optimized thread (t0). If this thread's CPU usage (t0CPU in logs) approaches 100%, you are hitting the capture limit.

Use a Modern Linux Kernel & VoIPmonitor Build

Modern Linux kernels (3.2+) and VoIPmonitor builds include TPACKET_V3 support, a high-speed packet capture mechanism. This is the single most important factor for high performance.

Recommendation: Always use a recent Linux distribution (AlmaLinux, Rocky Linux, or Debian) and the latest VoIPmonitor static binary. With this combination, a standard Intel 10Gbit NIC can often handle up to 2 Gbit/s of VoIP traffic without special drivers.

Network Stack & Driver Tuning

For high-traffic environments (>500 Mbit/s), fine-tuning the network driver and kernel parameters is essential.

NIC Ring Buffer

The ring buffer is a queue between the network card driver and VoIPmonitor. A larger buffer prevents packet loss during short CPU usage spikes. 

# Check maximum size
ethtool -g eth0

# Set to maximum (e.g., 16384)
ethtool -G eth0 rx 16384

Interrupt Coalescing

This setting batches multiple hardware interrupts into one, reducing CPU overhead. 

ethtool -C eth0 rx-usecs 1022

Applying Settings Persistently

To make these settings permanent, add them to your network configuration. For Debian/Ubuntu using /etc/network/interfaces: 

auto eth0
iface eth0 inet manual
    up ip link set $IFACE up
    up ip link set $IFACE promisc on
    up ethtool -G $IFACE rx 16384
    up ethtool -C $IFACE rx-usecs 1022

Note: Modern systems using NetworkManager or systemd-networkd require different configuration methods.

Configuration-Level Optimizations

Before investing in kernel-bypass solutions, ensure your voipmonitor.conf is optimized for performance. Several configuration parameters can significantly reduce CPU load and improve packet capture efficiency.

Use interface_ip_filter Instead of filter
If you need to filter by IP address or subnet, use interface_ip_filter instead of the general BPF filter option. The interface_ip_filter directive is more efficient and reduces CPU overhead compared to complex BPF filters.
# More efficient IP-based filtering
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8

# Less efficient BPF filtering (avoid if possible)
# filter = udp and (host 192.168.0.0/24 or host 10.0.0.0/8)

ℹ️ Note: See Sniffer_configuration for the complete reference and description of interface_ip_filter (Interface Selection section).

Optimize PCAP Compression Threads
For systems with high call recording rates, PCAP compression can become CPU-intensive. VoIPmonitor can automatically scale compression threads.
# /etc/voipmonitor.conf
# Initial compression threads (auto-scales based on load)
pcap_dump_writethreads = 1

# Maximum compression threads (adjust based on CPU cores)
pcap_dump_writethreads_max = 32

# Asynchronous PCAP writing (enabled by default)
pcap_dump_asyncwrite = yes

💡 Tip: Set pcap_dump_writethreads_max to the number of CPU cores available for best performance on multi-core systems. Monitor t0CPU to ensure compression threads are not competing with the capture thread.

Adjust Jitterbuffer Settings Based on Traffic Patterns
Jitterbuffer simulation adds CPU overhead. For production environments with stable networks, consider adjusting jitterbuffer settings to balance accuracy with performance.
# /etc/voipmonitor.conf
# Fixed 50ms jitterbuffer (default: yes)
jitterbuffer_f1 = yes

# Fixed 200ms jitterbuffer (default: yes)
jitterbuffer_f2 = yes

# Adaptive jitterbuffer up to 500ms (default: yes)
jitterbuffer_adapt = yes

⚠️ Warning: Disabling jitterbuffer analysis reduces CPU load but removes MOS and jitter quality metrics from CDRs. Only disable if you do not require voice quality monitoring.

Advanced Kernel-Bypass Solutions

If kernel and driver tuning are insufficient, you can offload the capture process entirely by bypassing the kernel's network stack.

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
PF_RING ZC/DNA Commercial 90% → 20% High-volume enterprise
Napatech SmartNICs Hardware <3% at 10 Gbit/s Extreme performance requirements
DPDK (Data Plane Development Kit)
A set of libraries and drivers for fast packet processing. VoIPmonitor can leverage DPDK to read packets directly from the network card, completely bypassing the kernel. See DPDK guide for details.
PF_RING ZC/DNA
A commercial software driver from ntop.org that dramatically reduces CPU load by bypassing the kernel.
Napatech SmartNICs
Specialized hardware acceleration cards that deliver packets with near-zero CPU overhead.

Optimizing Disk I/O

VoIPmonitor's modern storage engine is highly optimized to minimize random disk access, which is the primary cause of I/O bottlenecks.

VoIPmonitor Storage Strategy

Instead of writing a separate PCAP file for each call (which causes massive I/O load), VoIPmonitor groups all calls starting within the same minute into a single compressed .tar archive. This changes the I/O pattern from thousands of small, random writes to a few large, sequential writes, reducing IOPS by a factor of 10 or more.

Typical capacity: A standard 7200 RPM SATA drive can handle up to 2,000 concurrent calls with full recording.

Filesystem Tuning (ext4)

For the spool directory (/var/spool/voipmonitor), using an optimized ext4 filesystem can improve performance. 

# Format partition without a journal (use with caution, requires battery-backed RAID controller)
mke2fs -t ext4 -O ^has_journal /dev/sda2

# Add to /etc/fstab for optimal performance
/dev/sda2   /var/spool/voipmonitor  ext4    errors=remount-ro,noatime,data=writeback,barrier=0 0 0

⚠️ Warning: Disabling the journal removes protection against filesystem corruption after crashes. Only use this with a battery-backed RAID controller.

RAID Controller Cache Policy

A misconfigured RAID controller is a common bottleneck. For database and spool workloads, the cache policy should be set to WriteBack, not WriteThrough. This applies for RPM disks, not fast SSDs.

Requirements:

  • A healthy Battery Backup Unit (BBU) is required
  • Specific commands vary by vendor (megacli, ssacli, perccli)
  • Refer to vendor documentation for LSI, HP, and Dell controllers

Optimizing Database Performance (MySQL/MariaDB)

A well-tuned database is critical for both data ingestion from the sensor and GUI responsiveness.

ℹ️ Note:

Memory Configuration

The most critical database parameter is innodb_buffer_pool_size, which defines how much memory InnoDB uses to cache data and indexes.

⚠️ Warning: On servers running both VoIPmonitor and MySQL, setting innodb_buffer_pool_size too high causes OOM (Out of Memory) killer events, resulting in CDR delays, crashes, and instability. See OOM Troubleshooting for details.

Buffer Pool Sizing

Server Type Calculation Example (32GB RAM)
Shared (VoIPmonitor + MySQL) (Total RAM - VoIPmonitor - OS overhead) / 2 14GB
Dedicated MySQL server 50-70% of total RAM 20-22GB

For shared servers, use this formula: 

innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead - safety margin) / 2

Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory: ~2GB (check with ps aux)
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G

RAM Recommendations

Deployment Size Minimum RAM Recommended RAM
Small (<500 concurrent calls) 8GB 16GB
Medium (500-2000 calls) 16GB 32GB
Large (>2000 calls) 32GB 64GB+

Disable Graphical Desktop

A graphical desktop environment consumes 1-2GB of RAM unnecessarily. VoIPmonitor is managed through a web interface and does not require a desktop. 

# Disable display manager
systemctl stop gdm          # Ubuntu/Debian with GDM
systemctl disable gdm

# Set default to multi-user (no GUI)
systemctl set-default multi-user.target

# Verify memory freed
free -h

Other Key Parameters

# /etc/mysql/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf

[mysqld]
# Buffer pool size (calculate per above)
innodb_buffer_pool_size = 14G

# Flush logs to OS cache, write to disk once per second (faster, minimal data loss risk)
innodb_flush_log_at_trx_commit = 2

# Store each table in its own file (essential for partitioning)
innodb_file_per_table = 1

# LZ4 compression for modern MariaDB
innodb_compression_algorithm = lz4

⚠️ Warning:

Slow Query Log

The MySQL slow query log can consume significant memory and disk I/O on high-traffic systems. If you are experiencing high memory utilization alerts or performance issues with the database server, consider adjusting or disabling the slow query log.

⚠️ Warning: Disabling the slow query log removes the ability to analyze slow queries for performance optimization. Only disable it temporarily or if you are certain you do not need it. 

# /etc/mysql/my.cnf or /etc/my.cnf.d/mysql-server.cnf

[mysqld]
# Disable slow query log (set to 1 to enable)
slow_query_log = 0

# Alternative: Increase threshold to only log extremely slow queries (e.g., 600 seconds = 10 minutes)
long_query_time = 600

After changing MySQL configuration, restart the database and dependent services: 

# Restart MySQL/MariaDB
systemctl restart mariadb  # or mysql

# Restart VoIPmonitor sniffer (depends on database)
systemctl restart voipmonitor

Database Partitioning

VoIPmonitor automatically splits large tables (like cdr) into daily partitions. This is enabled by default and highly recommended.

Benefits:

  • Massively improves GUI query performance (only relevant partitions are scanned)
  • Allows instant deletion of old data by dropping partitions (thousands of times faster than DELETE)

See Database Partitioning for configuration details.

Monitoring Live Performance

VoIPmonitor logs detailed performance metrics every 10 seconds to syslog. 

# Debian/Ubuntu
tail -f /var/log/syslog | grep voipmonitor

# CentOS/RHEL
tail -f /var/log/messages | grep voipmonitor

Understanding the Log Output

Sample log line: 

voipmonitor[15567]: calls[315][355] PS[C:4 S:29/29 R:6354 A:6484] SQLq[0] heap[0|0|0] comp[54] [12.6Mb/s] t0CPU[5.2%] ... RSS/VSZ[323|752]MB
Metric Description Warning Threshold
calls[X][Y] X = active calls, Y = total calls in memory -
SQLq[C] SQL queries waiting to be sent to database Growing consistently = DB bottleneck
B|C] Memory usage % for internal buffers A = 100% → packet drops
t0CPU[X%] Main packet capture thread CPU usage >90-95% = capture limit reached
Y]MB Resident/Virtual memory usage RSS growing = memory leak

Performance Diagrams

The following diagrams illustrate the difference between standard kernel packet capture and optimized solutions:

Standard kernel packet capture path - packets traverse multiple kernel layers before reaching VoIPmonitor
PF_RING/DPDK bypass mode - packets are delivered directly to VoIPmonitor, bypassing the kernel network stack

See Also

AI Summary for RAG

Summary: Expert guide to scaling VoIPmonitor for high-traffic environments. Covers three main bottlenecks: (1) Packet Capturing - optimized via TPACKET_V3, NIC tuning with ethtool (ring buffer, interrupt coalescing), configuration-level optimizations (interface_ip_filter more efficient than BPF filter, pcap_dump_writethreads for compression thread tuning, jitterbuffer settings for CPU/performance balance), and kernel-bypass solutions (DPDK, PF_RING, Napatech); (2) Disk I/O - VoIPmonitor uses TAR-based storage to reduce IOPS, with ext4 tuning and RAID WriteBack cache; (3) Database - critical innodb_buffer_pool_size tuning with formula for shared servers: (Total RAM - VoIPmonitor - OS overhead) / 2. For 32GB shared server, recommend 14GB buffer pool. Dedicated servers can use 50-70% of RAM. Covers slow query log as a memory/I/O consumer and disabling it for memory optimization. Covers partitioning benefits and syslog monitoring (t0CPU, SQLq, heap metrics). For extreme scenarios (4,000+ concurrent calls, UI lag, unresponsive GUI, high SQL queue), see High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, centralized writer architecture, RTP thread tuning (rtpthreads, rtpthreads_start), and MySQL optimization settings (innodb_thread_concurrency, innodb_io_capacity, innodb_flush_method=O_DIRECT).

Keywords: scaling, performance tuning, bottleneck, t0CPU, TPACKET_V3, DPDK, PF_RING, ethtool, ring buffer, interface_ip_filter, BPF filter, pcap_dump_writethreads, jitterbuffer, jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt, compression threads, PCAP async write, innodb_buffer_pool_size, OOM killer, shared server memory, database partitioning, SQLq monitoring, slow query log, slow_query_log, long_query_time, UI lag, unresponsive GUI, high performance, 4000 concurrent calls, 5000 concurrent calls, innodb_flush_log_at_trx_commit=0, hourly partitioning, rtpthreads, rtpthreads_start, RTP threads, innodb_io_capacity, innodb_thread_concurrency, innodb_flush_method, extreme performance, High-Performance Manual

Key Questions:

  • How do I scale VoIPmonitor for thousands of concurrent calls?
  • What are the main performance bottlenecks in VoIPmonitor?
  • How do I fix high t0CPU usage?
  • What is DPDK and when should I use it?
  • How do I calculate innodb_buffer_pool_size for a shared server?
  • What happens if innodb_buffer_pool_size is set too high?
  • How do I interpret the performance metrics in syslog?
  • Should I use a dedicated database server for VoIPmonitor?
  • How much RAM does a VoIPmonitor server need?
  • How can the slow query log affect memory utilization?
  • How do I disable or adjust the MySQL slow query log?
  • Is interface_ip_filter more efficient than the filter option?
  • How do I optimize PCAP compression threads for high traffic?
  • Which jitterbuffer settings affect CPU load the most?
  • What configuration options reduce CPU overhead?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=CountryGrouping&oldid=7747"