Scaling: Difference between revisions

From VoIPmonitor.org
(Add configuration-level performance optimization section (interface_ip_filter, pcap_dump_writethreads, jitterbuffer settings))
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Scaling and Performance Tuning}}
{{DISPLAYTITLE:Scaling and Performance Tuning}}
Category:Administration
[[Category:Administration]]


This guide provides a comprehensive overview of performance tuning and scaling for VoIPmonitor. It covers the three primary system bottlenecks and offers practical, expert-level advice for optimizing your deployment for high traffic loads.
This guide covers performance tuning for high-traffic VoIPmonitor deployments, addressing the three primary system bottlenecks.


== Understanding Performance Bottlenecks ==
== Understanding Performance Bottlenecks ==
A VoIPmonitor deployment's maximum capacity is determined by three potential bottlenecks. Identifying and addressing the correct one is key to achieving high performance.
 
A VoIPmonitor deployment's capacity is limited by three potential bottlenecks:


<kroki lang="plantuml">
<kroki lang="plantuml">
Line 11: Line 12:
skinparam shadowing false
skinparam shadowing false
skinparam defaultFontName Arial
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #4A90E2
  BackgroundColor #FFFFFF
}


title VoIPmonitor Performance Bottlenecks
title VoIPmonitor Performance Bottlenecks
Line 22: Line 19:
rectangle "RTP/SIP\nProcessing" as PROC #E6FFE6
rectangle "RTP/SIP\nProcessing" as PROC #E6FFE6
rectangle "PCAP Files\nStorage" as DISK #FFF3E6
rectangle "PCAP Files\nStorage" as DISK #FFF3E6
database "MySQL/MariaDB\nDatabase" as DB #E6E6FF
database "MySQL/MariaDB" as DB #E6E6FF


NIC -right-> T0 : "1. CPU\nBottleneck"
NIC -right-> T0 : "1. CPU"
T0 -right-> PROC
T0 -right-> PROC
PROC -down-> DISK : "2. I/O\nBottleneck"
PROC -down-> DISK : "2. I/O"
PROC -right-> DB : "3. Database\nBottleneck"
PROC -right-> DB : "3. Database"


note bottom of T0
note bottom of T0
   Monitor: t0CPU in syslog
   Monitor: t0CPU
   Limit: Single CPU core
   Limit: 1 CPU core
end note
end note


note bottom of DISK
note bottom of DISK
   Monitor: iostat, ioping
   Monitor: iostat
   Solution: SSD, TAR archives
   Solution: SSD, TAR
end note
end note


note bottom of DB
note bottom of DB
   Monitor: SQLq in syslog
   Monitor: SQLq
   Solution: Partitioning, tuning
   Solution: RAM, tuning
end note
end note
@enduml
@enduml
</kroki>
</kroki>


The three bottlenecks are:
{| class="wikitable"
# '''Packet Capturing (CPU & Network Stack):''' The ability of a single CPU core to read packets from the network interface. This is often the first limit encountered.
|-
# '''Disk I/O (Storage):''' The speed at which the sensor can write PCAP files to disk. Critical when call recording is enabled.
! Bottleneck !! Description !! Monitor
# '''Database Performance (MySQL/MariaDB):''' The rate at which the database can ingest CDRs and serve data to the GUI.
|-
| '''1. Packet Capture''' || Single CPU core reading packets from NIC || <code>t0CPU</code> in syslog
|-
| '''2. Disk I/O''' || Writing PCAP files to storage || <code>iostat</code>, <code>ioping</code>
|-
| '''3. Database''' || CDR ingestion and GUI queries || <code>SQLq</code> in syslog
|}


On a modern, well-tuned server (e.g., 24-core Xeon, 10Gbit NIC), a single VoIPmonitor instance can handle up to '''10,000 concurrent calls''' with full RTP analysis and recording, or over '''60,000 concurrent calls''' with SIP-only analysis.
'''Capacity:''' A modern server (24-core Xeon, 10Gbit NIC) can handle '''~10,000 concurrent calls''' with full RTP recording, or '''60,000+''' with SIP-only analysis.


== Optimizing Packet Capturing (CPU & Network) ==
== Optimizing Packet Capture (CPU & Network) ==
The most performance-critical task is the initial packet capture, handled by a single, highly optimized thread (t0). If this thread's CPU usage (<code>t0CPU</code> in logs) approaches 100%, you are hitting the capture limit.


=== Use a Modern Linux Kernel & VoIPmonitor Build ===
The packet capture thread (t0) runs on a single CPU core. If <code>t0CPU</code> approaches 100%, you've hit the capture limit.
Modern Linux kernels (3.2+) and VoIPmonitor builds include '''TPACKET_V3''' support, a high-speed packet capture mechanism. This is the single most important factor for high performance.


'''Recommendation:''' Always use a recent Linux distribution (AlmaLinux, Rocky Linux, or Debian) and the latest VoIPmonitor static binary. With this combination, a standard Intel 10Gbit NIC can often handle up to 2 Gbit/s of VoIP traffic without special drivers.
With a modern kernel and VoIPmonitor build, a standard Intel 10Gbit NIC handles up to 3 Gbit/s VoIP traffic without special drivers and almost full 10Gbit rate with [[DPDK]]


=== Network Stack & Driver Tuning ===
=== Threading (Automatic) ===
For high-traffic environments (>500 Mbit/s), fine-tuning the network driver and kernel parameters is essential.


==== NIC Ring Buffer ====
Since version 2023.11, VoIPmonitor uses <code>threading_expanded=yes</code> by default, which automatically spawns threads based on CPU load. '''No manual threading configuration is needed.'''
The ring buffer is a queue between the network card driver and VoIPmonitor. A larger buffer prevents packet loss during short CPU usage spikes.


<syntaxhighlight lang="bash">
For very high traffic (≥1500 Mbit/s), set:
# Check maximum size
<syntaxhighlight lang="ini">
ethtool -g eth0
threading_expanded = high_traffic
</syntaxhighlight>


# Set to maximum (e.g., 16384)
See [[Sniffer_configuration#Threading_Model|Threading Model]] for details.
ethtool -G eth0 rx 16384
</syntaxhighlight>


==== Interrupt Coalescing ====
=== NIC Tuning (>500 Mbit/s) ===
This setting batches multiple hardware interrupts into one, reducing CPU overhead.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Increase ring buffer (prevents packet loss during CPU spikes)
ethtool -g eth0                  # Check max size
ethtool -G eth0 rx 16384        # Set to max
# Enable interrupt coalescing (reduces CPU overhead)
ethtool -C eth0 rx-usecs 1022
ethtool -C eth0 rx-usecs 1022
</syntaxhighlight>
</syntaxhighlight>


==== Applying Settings Persistently ====
'''Persistent settings''' (Debian/Ubuntu <code>/etc/network/interfaces</code>):
To make these settings permanent, add them to your network configuration. For Debian/Ubuntu using <code>/etc/network/interfaces</code>:
 
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
auto eth0
auto eth0
Line 94: Line 94:
</syntaxhighlight>
</syntaxhighlight>


Note: Modern systems using NetworkManager or systemd-networkd require different configuration methods.
=== Configuration Optimizations ===


==== Configuration-Level Optimizations ====
{| class="wikitable"
Before investing in kernel-bypass solutions, ensure your <code>voipmonitor.conf</code> is optimized for performance. Several configuration parameters can significantly reduce CPU load and improve packet capture efficiency.
|-
! Parameter !! Purpose !! Recommendation
|-
| <code>interface_ip_filter</code> || IP-based filtering || More efficient than BPF <code>filter</code>
|-
| <code>pcap_dump_writethreads_max</code> || Compression threads || Set to CPU core count
|-
| <code>jitterbuffer_f1/f2/adapt</code> || Jitter simulation || Keep <code>f2=yes</code>, disable f1 and adapt to save CPU while keeping MOS
|}


;Use interface_ip_filter Instead of filter
<syntaxhighlight lang="ini">
:If you need to filter by IP address or subnet, use <code>interface_ip_filter</code> instead of the general BPF <code>filter</code> option. The <code>interface_ip_filter</code> directive is more efficient and reduces CPU overhead compared to complex BPF filters.
# /etc/voipmonitor.conf


<syntaxhighlight lang="ini">
# Efficient IP filtering (replaces BPF filter)
# More efficient IP-based filtering
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8
interface_ip_filter = 10.0.0.0/8


# Less efficient BPF filtering (avoid if possible)
# Compression scaling
# filter = udp and (host 192.168.0.0/24 or host 10.0.0.0/8)
</syntaxhighlight>
 
{{Note|See [[Sniffer_configuration]] for the complete reference and description of <code>interface_ip_filter</code> (Interface Selection section).}}
 
;Optimize PCAP Compression Threads
:For systems with high call recording rates, PCAP compression can become CPU-intensive. VoIPmonitor can automatically scale compression threads.
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Initial compression threads (auto-scales based on load)
pcap_dump_writethreads = 1
pcap_dump_writethreads = 1
# Maximum compression threads (adjust based on CPU cores)
pcap_dump_writethreads_max = 32
pcap_dump_writethreads_max = 32
# Asynchronous PCAP writing (enabled by default)
pcap_dump_asyncwrite = yes
pcap_dump_asyncwrite = yes
</syntaxhighlight>
</syntaxhighlight>


{{Tip|Set <code>pcap_dump_writethreads_max</code> to the number of CPU cores available for best performance on multi-core systems. Monitor <code>t0CPU</code> to ensure compression threads are not competing with the capture thread.}}
{{Note|1=Recommended: <code>jitterbuffer_f1=no</code>, <code>jitterbuffer_f2=yes</code>, <code>jitterbuffer_adapt=no</code>. This saves CPU while preserving MOS-F2 metrics. Only disable f2 if you don't need quality monitoring at all.}}
 
;Adjust Jitterbuffer Settings Based on Traffic Patterns
:Jitterbuffer simulation adds CPU overhead. For production environments with stable networks, consider adjusting jitterbuffer settings to balance accuracy with performance.
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Fixed 50ms jitterbuffer (default: yes)
jitterbuffer_f1 = yes
 
# Fixed 200ms jitterbuffer (default: yes)
jitterbuffer_f2 = yes
 
# Adaptive jitterbuffer up to 500ms (default: yes)
jitterbuffer_adapt = yes
</syntaxhighlight>


{{Warning|Disabling jitterbuffer analysis reduces CPU load but removes MOS and jitter quality metrics from CDRs. Only disable if you do not require voice quality monitoring.}}
=== Kernel-Bypass Solutions ===


=== Advanced Kernel-Bypass Solutions ===
For extreme loads, bypass the kernel network stack entirely:
If kernel and driver tuning are insufficient, you can offload the capture process entirely by bypassing the kernel's network stack.


{| class="wikitable"
{| class="wikitable"
Line 156: Line 132:
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
|-
|-
| '''PF_RING ZC/DNA''' || Commercial || 90% → 20% || High-volume enterprise
| '''PF_RING ZC''' || Commercial || 90% → 20% || High-volume enterprise
|-
|-
| '''Napatech SmartNICs''' || Hardware || <3% at 10 Gbit/s || Extreme performance requirements
| '''[[Napatech|Napatech SmartNICs]]''' || Hardware || <3% at 10Gbit/s || Extreme performance
|}
|}
;DPDK (Data Plane Development Kit)
:A set of libraries and drivers for fast packet processing. VoIPmonitor can leverage DPDK to read packets directly from the network card, completely bypassing the kernel. See [[DPDK|DPDK guide]] for details.
;PF_RING ZC/DNA
:A commercial software driver from ntop.org that dramatically reduces CPU load by bypassing the kernel.
;Napatech SmartNICs
:Specialized hardware acceleration cards that deliver packets with near-zero CPU overhead.


== Optimizing Disk I/O ==
== Optimizing Disk I/O ==
VoIPmonitor's modern storage engine is highly optimized to minimize random disk access, which is the primary cause of I/O bottlenecks.


=== VoIPmonitor Storage Strategy ===
=== VoIPmonitor Storage Strategy ===
Instead of writing a separate PCAP file for each call (which causes massive I/O load), VoIPmonitor groups all calls starting within the same minute into a single compressed <code>.tar</code> archive. This changes the I/O pattern from thousands of small, random writes to a few large, sequential writes, reducing IOPS by a factor of 10 or more.


'''Typical capacity:''' A standard 7200 RPM SATA drive can handle up to 2,000 concurrent calls with full recording.
VoIPmonitor groups all calls starting within the same minute into a single compressed <code>.tar</code> archive. This changes thousands of random writes into few sequential writes, reducing IOPS by 10x+.
 
'''Typical capacity:''' 7200 RPM SATA handles ~2,000 concurrent calls with full recording.


=== Filesystem Tuning (ext4) ===
=== Filesystem Tuning (ext4) ===
For the spool directory (<code>/var/spool/voipmonitor</code>), using an optimized ext4 filesystem can improve performance.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Format partition without a journal (use with caution, requires battery-backed RAID controller)
# Format without journal (requires battery-backed RAID)
mke2fs -t ext4 -O ^has_journal /dev/sda2
mke2fs -t ext4 -O ^has_journal /dev/sda2
</syntaxhighlight>


# Add to /etc/fstab for optimal performance
<syntaxhighlight lang="ini">
/dev/sda2   /var/spool/voipmonitor  ext4   errors=remount-ro,noatime,data=writeback,barrier=0 0 0
# /etc/fstab
/dev/sda2 /var/spool/voipmonitor  ext4 errors=remount-ro,noatime,data=writeback,barrier=0 0 0
</syntaxhighlight>
</syntaxhighlight>


{{Warning|Disabling the journal removes protection against filesystem corruption after crashes. Only use this with a battery-backed RAID controller.}}
{{Warning|1=Disabling journal removes crash protection. Only use with battery-backed RAID controller (BBU).}}


=== RAID Controller Cache Policy ===
=== RAID Controller ===
A misconfigured RAID controller is a common bottleneck. For database and spool workloads, the cache policy should be set to '''WriteBack''', not WriteThrough. This applies for RPM disks, not fast SSDs.


'''Requirements:'''
Set cache policy to '''WriteBack''' (not WriteThrough). Requires healthy BBU. Commands vary by vendor (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>).
* A healthy Battery Backup Unit (BBU) is required
* Specific commands vary by vendor (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>)
* Refer to vendor documentation for LSI, HP, and Dell controllers


== Optimizing Database Performance (MySQL/MariaDB) ==
== Optimizing Database Performance ==
A well-tuned database is critical for both data ingestion from the sensor and GUI responsiveness.


=== Memory Configuration ===
=== Memory Configuration ===
The most critical database parameter is <code>innodb_buffer_pool_size</code>, which defines how much memory InnoDB uses to cache data and indexes.


{{Warning|On servers running both VoIPmonitor and MySQL, setting <code>innodb_buffer_pool_size</code> too high causes OOM (Out of Memory) killer events, resulting in CDR delays, crashes, and instability. See [[Sniffer_troubleshooting#Check_for_OOM_.28Out_of_Memory.29_Issues|OOM Troubleshooting]] for details.}}
The most critical parameter is <code>innodb_buffer_pool_size</code>.
 
{{Warning|1=Setting too high causes OOM killer events, CDR delays, and crashes. See [[Sniffer_troubleshooting#Check_for_OOM_.28Out_of_Memory.29_Issues|OOM Troubleshooting]].}}


==== Buffer Pool Sizing ====
'''Buffer Pool Sizing:'''


{| class="wikitable"
{| class="wikitable"
|-
|-
! Server Type !! Calculation !! Example (32GB RAM)
! Server Type !! Formula !! Example (32GB RAM)
|-
|-
| '''Shared''' (VoIPmonitor + MySQL) || (Total RAM - VoIPmonitor - OS overhead) / 2 || 14GB
| '''Shared''' (VoIPmonitor + MySQL) || (Total RAM - VoIPmonitor - OS) / 2 || 14GB
|-
|-
| '''Dedicated''' MySQL server || 50-70% of total RAM || 20-22GB
| '''Dedicated''' MySQL server || 50-70% of total RAM || 20-22GB
|}
|}


For shared servers, use this formula:
'''RAM Recommendations:'''
<syntaxhighlight lang="text">
innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead - safety margin) / 2
 
Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory: ~2GB (check with ps aux)
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G
</syntaxhighlight>


==== RAM Recommendations ====
{| class="wikitable"
{| class="wikitable"
|-
|-
! Deployment Size !! Minimum RAM !! Recommended RAM
! Deployment Size !! Minimum !! Recommended
|-
|-
| Small (<500 concurrent calls) || 8GB || 16GB
| Small (<500 calls) || 8GB || 16GB
|-
|-
| Medium (500-2000 calls) || 16GB || 32GB
| Medium (500-2000) || 16GB || 32GB
|-
|-
| Large (>2000 calls) || 32GB || 64GB+
| Large (>2000) || 32GB || 64GB+
|}
|}


==== Disable Graphical Desktop ====
=== Key MySQL Parameters ===
A graphical desktop environment consumes 1-2GB of RAM unnecessarily. VoIPmonitor is managed through a web interface and does not require a desktop.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
# Disable display manager
# /etc/mysql/my.cnf or mariadb.conf.d/50-server.cnf
systemctl stop gdm          # Ubuntu/Debian with GDM
[mysqld]
systemctl disable gdm
innodb_buffer_pool_size = 14G
innodb_flush_log_at_trx_commit = 2  # Faster, minimal data loss risk
innodb_file_per_table = 1          # Essential for partitioning
innodb_compression_algorithm = lz4  # MariaDB only
</syntaxhighlight>


# Set default to multi-user (no GUI)
=== Slow Query Log ===
systemctl set-default multi-user.target


# Verify memory freed
The slow query log can consume significant memory. Consider disabling on high-traffic systems:
free -h
</syntaxhighlight>
 
=== Other Key Parameters ===


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]
slow_query_log = 0
# Or increase threshold: long_query_time = 600
</syntaxhighlight>


[mysqld]
=== Database Partitioning ===
# Buffer pool size (calculate per above)
innodb_buffer_pool_size = 14G


# Flush logs to OS cache, write to disk once per second (faster, minimal data loss risk)
VoIPmonitor automatically partitions large tables (like <code>cdr</code>) by day. This is enabled by default and '''highly recommended'''.
innodb_flush_log_at_trx_commit = 2


# Store each table in its own file (essential for partitioning)
See [[Data_Cleaning#Database_Cleaning_.28CDR_Retention.29|Database Partitioning]] for details.
innodb_file_per_table = 1


# LZ4 compression for modern MariaDB
=== Troubleshooting: Connection Refused ===
innodb_compression_algorithm = lz4
</syntaxhighlight>


=== Slow Query Log ===
'''Symptoms:''' GUI crashes, "Connection refused" errors, intermittent issues during peak volumes.


The MySQL slow query log can consume significant memory and disk I/O on high-traffic systems. If you are experiencing high memory utilization alerts or performance issues with the database server, consider adjusting or disabling the slow query log.
'''Cause:''' <code>innodb_buffer_pool_size</code> too low (default 128M is insufficient).


{{Warning|Disabling the slow query log removes the ability to analyze slow queries for performance optimization. Only disable it temporarily or if you are certain you do not need it.}}
'''Solution:''' Increase to 6G+ based on available RAM:


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/my.cnf.d/mysql-server.cnf
[mysqld]
[mysqld]
# Disable slow query log (set to 1 to enable)
innodb_buffer_pool_size = 6G
slow_query_log = 0
 
# Alternative: Increase threshold to only log extremely slow queries (e.g., 600 seconds = 10 minutes)
long_query_time = 600
</syntaxhighlight>
</syntaxhighlight>
After changing MySQL configuration, restart the database and dependent services:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Restart MySQL/MariaDB
systemctl restart mariadb
systemctl restart mariadb # or mysql
 
# Restart VoIPmonitor sniffer (depends on database)
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


=== Database Partitioning ===
== Component Separation (Multi-Host Architecture) ==
VoIPmonitor automatically splits large tables (like <code>cdr</code>) into daily partitions. This is enabled by default and '''highly recommended'''.


'''Benefits:'''
For deployments exceeding 5,000-10,000 concurrent calls, separate VoIPmonitor components onto dedicated hosts.
* Massively improves GUI query performance (only relevant partitions are scanned)
* Allows instant deletion of old data by dropping partitions (thousands of times faster than DELETE)


See [[Data_Cleaning#The_Modern_Method:_Partitioning_.28Recommended.29|Database Partitioning]] for configuration details.
=== Architecture Overview ===


== Monitoring Live Performance ==
{| class="wikitable"
VoIPmonitor logs detailed performance metrics every 10 seconds to syslog.
|-
! Host !! Component !! Primary Resources !! Scaling Strategy
|-
| '''Host 1''' || MySQL Database || RAM, fast SSD || Add RAM, read replicas
|-
| '''Host 2''' || Sensor(s) || CPU (t0 thread), network || DPDK/PF_RING, more sensors
|-
| '''Host 3''' || GUI || CPU, network || Load balancer, caching
|}


<syntaxhighlight lang="bash">
=== Configuration ===
# Debian/Ubuntu
 
tail -f /var/log/syslog | grep voipmonitor
'''MySQL Server:'''
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf
[mysqld]
bind-address = 0.0.0.0
innodb_buffer_pool_size = 50G  # 50-70% RAM for dedicated server
</syntaxhighlight>


# CentOS/RHEL
<syntaxhighlight lang="sql">
tail -f /var/log/messages | grep voipmonitor
CREATE USER 'voipmonitor'@'%' IDENTIFIED BY 'strong_password';
GRANT ALL PRIVILEGES ON voipmonitor.* TO 'voipmonitor'@'%';
</syntaxhighlight>
</syntaxhighlight>


=== Understanding the Log Output ===
'''Sensor:'''
Sample log line:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="text">
# /etc/voipmonitor.conf
voipmonitor[15567]: calls[315][355] PS[C:4 S:29/29 R:6354 A:6484] SQLq[0] heap[0|0|0] comp[54] [12.6Mb/s] t0CPU[5.2%] ... RSS/VSZ[323|752]MB
id_sensor = 1
mysqlhost = mysql.server.ip
mysqldb = voipmonitor
mysqlusername = voipmonitor
mysqlpassword = strong_password
</syntaxhighlight>
</syntaxhighlight>


'''GUI:''' Configure via Settings > System Configuration > Database, or edit <code>config/system_configuration.php</code>.
'''Firewall Rules:'''
{| class="wikitable"
{| class="wikitable"
|-
|-
! Metric !! Description !! Warning Threshold
! Source !! Destination !! Port !! Purpose
|-
| Sensor || MySQL || 3306 || CDR writes
|-
| GUI || MySQL || 3306 || Queries
|-
| GUI || Sensor(s) || 5029 || PCAP retrieval
|-
| Users || GUI || 80, 443 || Web access
|}
 
{{Note|1=Component separation can be combined with [[Sniffer_distributed_architecture|Client-Server mode]] for multi-site deployments.}}
 
== Monitoring Performance ==
 
VoIPmonitor logs performance metrics every 10 seconds to syslog. Key metrics to watch:
 
{| class="wikitable"
|-
|-
| <code>calls[X][Y]</code> || X = active calls, Y = total calls in memory || -
! Metric !! Warning Sign !! Bottleneck Type
|-
|-
| <code>SQLq[C]</code> || SQL queries waiting to be sent to database || Growing consistently = DB bottleneck
| <code>t0CPU</code> || >90% || CPU (packet capture limit)
|-
|-
| <code>heap[A{{!}}B{{!}}C]</code> || Memory usage % for internal buffers || A = 100% → packet drops
| <code>heap[A&#124;B&#124;C]</code> || A >50% || I/O or CPU (buffer filling)
|-
|-
| <code>t0CPU[X%]</code> || '''Main packet capture thread CPU usage''' || >90-95% = capture limit reached
| <code>SQLq</code> || Growing || Database
|-
|-
| <code>RSS/VSZ[X{{!}}Y]MB</code> || Resident/Virtual memory usage || RSS growing = memory leak
| <code>comp</code> || Maxed out || I/O (compression waiting for disk)
|}
|}


=== Performance Diagrams ===
<syntaxhighlight lang="bash">
# Monitor in real-time
journalctl -u voipmonitor -f
</syntaxhighlight>


The following diagrams illustrate the difference between standard kernel packet capture and optimized solutions:
'''Main article: [[Syslog_Status_Line]]''' - Complete reference for all metrics with detailed explanations and troubleshooting guidance.


[[File:kernelstandarddiagram.png|thumb|center|600px|Standard kernel packet capture path - packets traverse multiple kernel layers before reaching VoIPmonitor]]
'''For bottleneck diagnosis:''' See [[Sniffer_troubleshooting#Diagnose:_I.2FO_vs_CPU_Bottleneck|I/O vs CPU Bottleneck Diagnosis]] for step-by-step diagnostic procedure using syslog metrics and Linux tools.


[[File:ntop.png|thumb|center|600px|PF_RING/DPDK bypass mode - packets are delivered directly to VoIPmonitor, bypassing the kernel network stack]]
== See Also ==


== See Also ==
* [[Sniffer_troubleshooting]] - Troubleshooting including OOM issues
* [[Sniffer_troubleshooting]] - Troubleshooting guide including OOM issues
* [[Data_Cleaning]] - Database and spool retention
* [[Data_Cleaning]] - Database and spool retention configuration
* [[Sniffer_configuration]] - Complete configuration reference
* [[Sniffer_configuration]] - Complete configuration reference
* [[DPDK]] - DPDK setup guide
* [[DPDK]] - DPDK setup guide
* [[IO_Measurement]] - Disk I/O benchmarking tools
* [[Sniffer_distributed_architecture]] - Client-Server mode


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' Expert guide to scaling VoIPmonitor for high-traffic environments. Covers three main bottlenecks: (1) Packet Capturing - optimized via TPACKET_V3, NIC tuning with ethtool (ring buffer, interrupt coalescing), configuration-level optimizations (interface_ip_filter more efficient than BPF filter, pcap_dump_writethreads for compression thread tuning, jitterbuffer settings for CPU/performance balance), and kernel-bypass solutions (DPDK, PF_RING, Napatech); (2) Disk I/O - VoIPmonitor uses TAR-based storage to reduce IOPS, with ext4 tuning and RAID WriteBack cache; (3) Database - critical innodb_buffer_pool_size tuning with formula for shared servers: (Total RAM - VoIPmonitor - OS overhead) / 2. For 32GB shared server, recommend 14GB buffer pool. Dedicated servers can use 50-70% of RAM. Covers slow query log as a memory/I/O consumer and disabling it for memory optimization. Covers partitioning benefits and syslog monitoring (t0CPU, SQLq, heap metrics).


'''Keywords:''' scaling, performance tuning, bottleneck, t0CPU, TPACKET_V3, DPDK, PF_RING, ethtool, ring buffer, interface_ip_filter, BPF filter, pcap_dump_writethreads, jitterbuffer, jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt, compression threads, PCAP async write, innodb_buffer_pool_size, OOM killer, shared server memory, database partitioning, SQLq monitoring, slow query log, slow_query_log, long_query_time
<!-- This section is for AI/RAG systems. Do not edit manually. -->
 
=== Summary ===
Performance tuning guide for high-traffic VoIPmonitor deployments. Covers three main bottlenecks: CPU (t0 packet capture thread, single-core limit), Disk I/O (PCAP storage), and Database (MySQL/MariaDB). Threading is automatic since 2023.11 via threading_expanded=yes (use high_traffic for ≥1500 Mbit/s). NIC tuning: ethtool ring buffer and interrupt coalescing. CPU optimization: interface_ip_filter instead of BPF, jitterbuffer_f2=yes with f1/adapt disabled. Kernel bypass solutions: DPDK (~70% CPU reduction), PF_RING ZC, Napatech SmartNICs (<3% CPU at 10Gbit). Disk I/O: TAR archives reduce IOPS 10x, ext4 tuning (noatime, writeback), RAID WriteBack cache with BBU. Database: innodb_buffer_pool_size (50-70% RAM for dedicated server), innodb_flush_log_at_trx_commit=2, partitioning. Multi-host architecture for >5000-10000 concurrent calls separating MySQL, sensors, and GUI.
 
=== Keywords ===
scaling, performance, tuning, optimization, high traffic, bottleneck, CPU, t0CPU, t0 thread, single-core, disk I/O, storage, database, MySQL, MariaDB, threading_expanded, high_traffic, NIC tuning, ethtool, ring buffer, interrupt coalescing, interface_ip_filter, jitterbuffer, DPDK, PF_RING, Napatech, kernel bypass, TAR archive, ext4, noatime, writeback, RAID, WriteBack cache, BBU, innodb_buffer_pool_size, innodb_flush_log_at_trx_commit, partitioning, multi-host, component separation, concurrent calls, capacity, 10000 calls, heap, SQLq, compression threads, pcap_dump_writethreads


'''Key Questions:'''
=== Key Questions ===
* How do I scale VoIPmonitor for thousands of concurrent calls?
* How to tune VoIPmonitor for high traffic?
* What are the main performance bottlenecks in VoIPmonitor?
* How many concurrent calls can VoIPmonitor handle?
* How do I fix high t0CPU usage?
* What are the main performance bottlenecks?
* What is DPDK and when should I use it?
* How to optimize CPU usage for packet capture?
* How do I calculate innodb_buffer_pool_size for a shared server?
* What is threading_expanded and when to use high_traffic?
* What happens if innodb_buffer_pool_size is set too high?
* How to tune NIC for VoIPmonitor?
* How do I interpret the performance metrics in syslog?
* How to reduce CPU with jitterbuffer settings?
* Should I use a dedicated database server for VoIPmonitor?
* What is DPDK and when to use it?
* How much RAM does a VoIPmonitor server need?
* How to optimize disk I/O for PCAP storage?
* How can the slow query log affect memory utilization?
* How to tune ext4 filesystem for VoIPmonitor?
* How do I disable or adjust the MySQL slow query log?
* What is the recommended innodb_buffer_pool_size?
* Is interface_ip_filter more efficient than the filter option?
* How to configure MySQL for VoIPmonitor performance?
* How do I optimize PCAP compression threads for high traffic?
* When to separate VoIPmonitor components to multiple hosts?
* Which jitterbuffer settings affect CPU load the most?
* How to monitor VoIPmonitor performance metrics?
* What configuration options reduce CPU overhead?
* What do t0CPU, heap, SQLq metrics mean?

Latest revision as of 21:52, 20 January 2026


This guide covers performance tuning for high-traffic VoIPmonitor deployments, addressing the three primary system bottlenecks.

Understanding Performance Bottlenecks

A VoIPmonitor deployment's capacity is limited by three potential bottlenecks:

Bottleneck Description Monitor
1. Packet Capture Single CPU core reading packets from NIC t0CPU in syslog
2. Disk I/O Writing PCAP files to storage iostat, ioping
3. Database CDR ingestion and GUI queries SQLq in syslog

Capacity: A modern server (24-core Xeon, 10Gbit NIC) can handle ~10,000 concurrent calls with full RTP recording, or 60,000+ with SIP-only analysis.

Optimizing Packet Capture (CPU & Network)

The packet capture thread (t0) runs on a single CPU core. If t0CPU approaches 100%, you've hit the capture limit.

With a modern kernel and VoIPmonitor build, a standard Intel 10Gbit NIC handles up to 3 Gbit/s VoIP traffic without special drivers and almost full 10Gbit rate with DPDK

Threading (Automatic)

Since version 2023.11, VoIPmonitor uses threading_expanded=yes by default, which automatically spawns threads based on CPU load. No manual threading configuration is needed.

For very high traffic (≥1500 Mbit/s), set: 

threading_expanded = high_traffic

See Threading Model for details.

NIC Tuning (>500 Mbit/s)

# Increase ring buffer (prevents packet loss during CPU spikes)
ethtool -g eth0                  # Check max size
ethtool -G eth0 rx 16384         # Set to max

# Enable interrupt coalescing (reduces CPU overhead)
ethtool -C eth0 rx-usecs 1022

Persistent settings (Debian/Ubuntu /etc/network/interfaces): 

auto eth0
iface eth0 inet manual
    up ip link set $IFACE up
    up ip link set $IFACE promisc on
    up ethtool -G $IFACE rx 16384
    up ethtool -C $IFACE rx-usecs 1022

Configuration Optimizations

Parameter Purpose Recommendation
interface_ip_filter IP-based filtering More efficient than BPF filter
pcap_dump_writethreads_max Compression threads Set to CPU core count
jitterbuffer_f1/f2/adapt Jitter simulation Keep f2=yes, disable f1 and adapt to save CPU while keeping MOS 
# /etc/voipmonitor.conf

# Efficient IP filtering (replaces BPF filter)
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8

# Compression scaling
pcap_dump_writethreads = 1
pcap_dump_writethreads_max = 32
pcap_dump_asyncwrite = yes

ℹ️ Note: Recommended: jitterbuffer_f1=no, jitterbuffer_f2=yes, jitterbuffer_adapt=no. This saves CPU while preserving MOS-F2 metrics. Only disable f2 if you don't need quality monitoring at all.

Kernel-Bypass Solutions

For extreme loads, bypass the kernel network stack entirely:

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
PF_RING ZC Commercial 90% → 20% High-volume enterprise
Napatech SmartNICs Hardware <3% at 10Gbit/s Extreme performance

Optimizing Disk I/O

VoIPmonitor Storage Strategy

VoIPmonitor groups all calls starting within the same minute into a single compressed .tar archive. This changes thousands of random writes into few sequential writes, reducing IOPS by 10x+.

Typical capacity: 7200 RPM SATA handles ~2,000 concurrent calls with full recording.

Filesystem Tuning (ext4)

# Format without journal (requires battery-backed RAID)
mke2fs -t ext4 -O ^has_journal /dev/sda2

# /etc/fstab
/dev/sda2  /var/spool/voipmonitor  ext4  errors=remount-ro,noatime,data=writeback,barrier=0  0 0

⚠️ Warning: Disabling journal removes crash protection. Only use with battery-backed RAID controller (BBU).

RAID Controller

Set cache policy to WriteBack (not WriteThrough). Requires healthy BBU. Commands vary by vendor (megacli, ssacli, perccli).

Optimizing Database Performance

Memory Configuration

The most critical parameter is innodb_buffer_pool_size.

⚠️ Warning: Setting too high causes OOM killer events, CDR delays, and crashes. See OOM Troubleshooting.

Buffer Pool Sizing:

Server Type Formula Example (32GB RAM)
Shared (VoIPmonitor + MySQL) (Total RAM - VoIPmonitor - OS) / 2 14GB
Dedicated MySQL server 50-70% of total RAM 20-22GB

RAM Recommendations:

Deployment Size Minimum Recommended
Small (<500 calls) 8GB 16GB
Medium (500-2000) 16GB 32GB
Large (>2000) 32GB 64GB+

Key MySQL Parameters

# /etc/mysql/my.cnf or mariadb.conf.d/50-server.cnf
[mysqld]
innodb_buffer_pool_size = 14G
innodb_flush_log_at_trx_commit = 2  # Faster, minimal data loss risk
innodb_file_per_table = 1           # Essential for partitioning
innodb_compression_algorithm = lz4  # MariaDB only

Slow Query Log

The slow query log can consume significant memory. Consider disabling on high-traffic systems: 

[mysqld]
slow_query_log = 0
# Or increase threshold: long_query_time = 600

Database Partitioning

VoIPmonitor automatically partitions large tables (like cdr) by day. This is enabled by default and highly recommended.

See Database Partitioning for details.

Troubleshooting: Connection Refused

Symptoms: GUI crashes, "Connection refused" errors, intermittent issues during peak volumes.

Cause: innodb_buffer_pool_size too low (default 128M is insufficient).

Solution: Increase to 6G+ based on available RAM: 

[mysqld]
innodb_buffer_pool_size = 6G

systemctl restart mariadb

Component Separation (Multi-Host Architecture)

For deployments exceeding 5,000-10,000 concurrent calls, separate VoIPmonitor components onto dedicated hosts.

Architecture Overview

Host Component Primary Resources Scaling Strategy
Host 1 MySQL Database RAM, fast SSD Add RAM, read replicas
Host 2 Sensor(s) CPU (t0 thread), network DPDK/PF_RING, more sensors
Host 3 GUI CPU, network Load balancer, caching

Configuration

MySQL Server: 

# /etc/mysql/my.cnf
[mysqld]
bind-address = 0.0.0.0
innodb_buffer_pool_size = 50G  # 50-70% RAM for dedicated server

CREATE USER 'voipmonitor'@'%' IDENTIFIED BY 'strong_password';
GRANT ALL PRIVILEGES ON voipmonitor.* TO 'voipmonitor'@'%';

Sensor: 

# /etc/voipmonitor.conf
id_sensor = 1
mysqlhost = mysql.server.ip
mysqldb = voipmonitor
mysqlusername = voipmonitor
mysqlpassword = strong_password

GUI: Configure via Settings > System Configuration > Database, or edit config/system_configuration.php.

Firewall Rules:

Source Destination Port Purpose
Sensor MySQL 3306 CDR writes
GUI MySQL 3306 Queries
GUI Sensor(s) 5029 PCAP retrieval
Users GUI 80, 443 Web access

ℹ️ Note: Component separation can be combined with Client-Server mode for multi-site deployments.

Monitoring Performance

VoIPmonitor logs performance metrics every 10 seconds to syslog. Key metrics to watch:

Metric Warning Sign Bottleneck Type
t0CPU >90% CPU (packet capture limit)
heap[A|B|C] A >50% I/O or CPU (buffer filling)
SQLq Growing Database
comp Maxed out I/O (compression waiting for disk) 
# Monitor in real-time
journalctl -u voipmonitor -f

Main article: Syslog_Status_Line - Complete reference for all metrics with detailed explanations and troubleshooting guidance.

For bottleneck diagnosis: See I/O vs CPU Bottleneck Diagnosis for step-by-step diagnostic procedure using syslog metrics and Linux tools.

See Also

AI Summary for RAG

Summary

Performance tuning guide for high-traffic VoIPmonitor deployments. Covers three main bottlenecks: CPU (t0 packet capture thread, single-core limit), Disk I/O (PCAP storage), and Database (MySQL/MariaDB). Threading is automatic since 2023.11 via threading_expanded=yes (use high_traffic for ≥1500 Mbit/s). NIC tuning: ethtool ring buffer and interrupt coalescing. CPU optimization: interface_ip_filter instead of BPF, jitterbuffer_f2=yes with f1/adapt disabled. Kernel bypass solutions: DPDK (~70% CPU reduction), PF_RING ZC, Napatech SmartNICs (<3% CPU at 10Gbit). Disk I/O: TAR archives reduce IOPS 10x, ext4 tuning (noatime, writeback), RAID WriteBack cache with BBU. Database: innodb_buffer_pool_size (50-70% RAM for dedicated server), innodb_flush_log_at_trx_commit=2, partitioning. Multi-host architecture for >5000-10000 concurrent calls separating MySQL, sensors, and GUI.

Keywords

scaling, performance, tuning, optimization, high traffic, bottleneck, CPU, t0CPU, t0 thread, single-core, disk I/O, storage, database, MySQL, MariaDB, threading_expanded, high_traffic, NIC tuning, ethtool, ring buffer, interrupt coalescing, interface_ip_filter, jitterbuffer, DPDK, PF_RING, Napatech, kernel bypass, TAR archive, ext4, noatime, writeback, RAID, WriteBack cache, BBU, innodb_buffer_pool_size, innodb_flush_log_at_trx_commit, partitioning, multi-host, component separation, concurrent calls, capacity, 10000 calls, heap, SQLq, compression threads, pcap_dump_writethreads

Key Questions

  • How to tune VoIPmonitor for high traffic?
  • How many concurrent calls can VoIPmonitor handle?
  • What are the main performance bottlenecks?
  • How to optimize CPU usage for packet capture?
  • What is threading_expanded and when to use high_traffic?
  • How to tune NIC for VoIPmonitor?
  • How to reduce CPU with jitterbuffer settings?
  • What is DPDK and when to use it?
  • How to optimize disk I/O for PCAP storage?
  • How to tune ext4 filesystem for VoIPmonitor?
  • What is the recommended innodb_buffer_pool_size?
  • How to configure MySQL for VoIPmonitor performance?
  • When to separate VoIPmonitor components to multiple hosts?
  • How to monitor VoIPmonitor performance metrics?
  • What do t0CPU, heap, SQLq metrics mean?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Scaling&oldid=10993"