Billing: Difference between revisions

From VoIPmonitor.org
(Add Number Matching Type section explaining normalized vs original prefix matching)
(Review: oprava struktury nadpisů, přidání kategorie, oprava překlepu)
Tag: Reverted
Line 1: Line 1:
== Billing Feature Overview ==
{{DISPLAYTITLE:Scaling and Performance Tuning}}
[[Category:Administration]]


* Two price types in CDR: Customer and operator (for comparing operators).
This guide provides a comprehensive overview of performance tuning and scaling for VoIPmonitor. It covers the three primary system bottlenecks and offers practical, expert-level advice for optimizing your deployment for high traffic loads.
* Multiple billing tables: For each operator and customer.
* Billing watchdog rules: Quotas per hour/day/week/month aggregated by src IP or tel. number.
* Billing watchdog rules: For expensive destinations.


== Settings > Billing Configuration ==
== Understanding Performance Bottlenecks ==
A VoIPmonitor deployment's maximum capacity is determined by three potential bottlenecks. Identifying and addressing the correct one is key to achieving high performance.


Sections: Billing type/rules, state holidays, operator assignment, customer assignment, exclude, aggregation, currency, settings.
<kroki lang="plantuml">
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #4A90E2
  BackgroundColor #FFFFFF
}


=== Billing Type / Rules ===
title VoIPmonitor Performance Bottlenecks


Create at least one billing table for customer/operator prices. For operator-only, create per operator and assign in "operator assignment" without "Default customer billing".
rectangle "Network\nInterface" as NIC #E8F4FD
rectangle "Packet Capture\n(t0 thread)" as T0 #FFE6E6
rectangle "RTP/SIP\nProcessing" as PROC #E6FFE6
rectangle "PCAP Files\nStorage" as DISK #FFF3E6
database "MySQL/MariaDB\nDatabase" as DB #E6E6FF


* Name: Billing table name.
NIC -right-> T0 : "1. CPU\nBottleneck"
* State holidays: Assign holidays table for peak/offpeak.
T0 -right-> PROC
* Default price: Per-minute (offpeak if peak/offpeak enabled).
PROC -down-> DISK : "2. I/O\nBottleneck"
* Enable peak/offpeak: For time-based billing.
PROC -right-> DB : "3. Database\nBottleneck"
* Default price peak: Peak per-minute.
* Default T1: First impulse seconds.
* Default T2: Period length (e.g., 30s call, T1=10, T2=6: Bill 10 + 4*6 = 34s).
* Default customer billing: When enabled, applies the customer price (C:) from this table to all source IPs, even if not assigned to a specific customer. This is useful for reporting purposes when you want to ensure all calls show customer pricing, while operator prices (O:) are still determined by operator assignment rules based on destination IP.


Add billing rules (CSV export/import):
note bottom of T0
  Monitor: t0CPU in syslog
  Limit: Single CPU core
end note


* Name: Rule name.
note bottom of DISK
* Prefix number: Match prefix.
  Monitor: iostat, ioping
* Fixed number: Match exact.
  Solution: SSD, TAR archives
* Price: Destination (offpeak if enabled).
end note
* Price peak: Peak destination.
* Override default peak/offpeak: Custom times.
* T1: Custom first impulse.
* T2: Custom period.


==== Number Matching Type ====
note bottom of DB
  Monitor: SQLq in syslog
  Solution: Partitioning, tuning
end note
@enduml
</kroki>


When creating prefix rules, choose how VoIPmonitor matches international prefixes:
The three bottlenecks are:
# '''Packet Capturing (CPU & Network Stack):''' The ability of a single CPU core to read packets from the network interface. This is often the first limit encountered.
# '''Disk I/O (Storage):''' The speed at which the sensor can write PCAP files to disk. Critical when call recording is enabled.
# '''Database Performance (MySQL/MariaDB):''' The rate at which the database can ingest CDRs and serve data to the GUI.


* '''Original:''' Requires separate rules for each prefix format. For example, if users dial both <code>00420</code> and <code>+420</code>, you need two rules: one for <code>00420</code> and another for <code>+420</code>.
On a modern, well-tuned server (e.g., 24-core Xeon, 10Gbit NIC), a single VoIPmonitor instance can handle up to '''10,000 concurrent calls''' with full RTP analysis and recording, or over '''60,000 concurrent calls''' with SIP-only analysis.


* '''Normalized:''' International prefixes are stripped before matching. A single rule for <code>420</code> will match calls to <code>00420</code>, <code>+420</code>, and just <code>420</code>. This is recommended for simpler configuration when international prefixes vary.
== Optimizing Packet Capturing (CPU & Network) ==
The most performance-critical task is the initial packet capture, handled by a single, highly optimized thread (t0). If this thread's CPU usage (<code>t0CPU</code> in logs) approaches 100%, you are hitting the capture limit.


The matching behavior is controlled by the "International prefixes" setting in the global Settings section (default: <code>00</code> and <code>+</code>).
=== Use a Modern Linux Kernel & VoIPmonitor Build ===
Modern Linux kernels (3.2+) and VoIPmonitor builds include '''TPACKET_V3''' support, a high-speed packet capture mechanism. This is the single most important factor for high performance.


=== State Holidays ===
'''Recommendation:''' Always use a recent Linux distribution (AlmaLinux, Rocky Linux, or Debian) and the latest VoIPmonitor static binary. With this combination, a standard Intel 10Gbit NIC can often handle up to 2 Gbit/s of VoIP traffic without special drivers.


Holidays table for offpeak treatment. Define only for peak/offpeak billing.
=== Network Stack & Driver Tuning ===
For high-traffic environments (>500 Mbit/s), fine-tuning the network driver and kernel parameters is essential.


=== Operator Assignment ===
==== NIC Ring Buffer ====
The ring buffer is a queue between the network card driver and VoIPmonitor. A larger buffer prevents packet loss during short CPU usage spikes.


Rules per operator with billing table. Excludes customer price for operator calls (bills outgoing only). For both prices: One default customer table, assign operator tables here.
<syntaxhighlight lang="bash">
# Check maximum size
ethtool -g eth0


Override international prefixes per operator (see [[#settings]]).
# Set to maximum (e.g., 16384)
ethtool -G eth0 rx 16384
</syntaxhighlight>


=== Customer Assignment ===
==== Interrupt Coalescing ====
This setting batches multiple hardware interrupts into one, reducing CPU overhead.


Apply different tables to customers by src IP or tel. numbers/prefixes if default insufficient.
<syntaxhighlight lang="bash">
ethtool -C eth0 rx-usecs 1022
</syntaxhighlight>


Override international prefixes per customer (see [[#settings]]).
==== Applying Settings Persistently ====
To make these settings permanent, add them to your network configuration. For Debian/Ubuntu using <code>/etc/network/interfaces</code>:


=== Exclude ===
<syntaxhighlight lang="ini">
auto eth0
iface eth0 inet manual
    up ip link set $IFACE up
    up ip link set $IFACE promisc on
    up ethtool -G $IFACE rx 16384
    up ethtool -C $IFACE rx-usecs 1022
</syntaxhighlight>


Rules to exclude billing by src/dst IP or tel. prefixes/numbers.
Note: Modern systems using NetworkManager or systemd-networkd require different configuration methods.


=== Aggregation ===
==== Configuration-Level Optimizations ====
Before investing in kernel-bypass solutions, ensure your <code>voipmonitor.conf</code> is optimized for performance. Several configuration parameters can significantly reduce CPU load and improve packet capture efficiency.


Summarizes costs by src IP/tel. number for hour/day/week/month/year. Used in fraud/alerts for quota rules.
;Use interface_ip_filter Instead of filter
:If you need to filter by IP address or subnet, use <code>interface_ip_filter</code> instead of the general BPF <code>filter</code> option. The <code>interface_ip_filter</code> directive is more efficient and reduces CPU overhead compared to complex BPF filters.


==== Settings ====
<syntaxhighlight lang="ini">
# More efficient IP-based filtering
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8


* Enable by addresses: Per src IP.
# Less efficient BPF filtering (avoid if possible)
* Enable by numbers: Per tel. number.
# filter = udp and (host 192.168.0.0/24 or host 10.0.0.0/8)
* Week start.
</syntaxhighlight>
* Hours/days/weeks/months/years: Retention periods.


==== Exclude Addresses ====
{{Note|See [[Sniffer_configuration]] for the complete reference and description of <code>interface_ip_filter</code> (Interface Selection section).}}


Exclude src/dst IPs (e.g., operators/SIP proxies) for customer-only data.
;Optimize PCAP Compression Threads
:For systems with high call recording rates, PCAP compression can become CPU-intensive. VoIPmonitor can automatically scale compression threads.


==== Exclude Numbers ====
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Initial compression threads (auto-scales based on load)
pcap_dump_writethreads = 1


Exclude src/dst numbers. Prioritize addresses to avoid unnecessary data.
# Maximum compression threads (adjust based on CPU cores)
pcap_dump_writethreads_max = 32


=== Currency ===
# Asynchronous PCAP writing (enabled by default)
pcap_dump_asyncwrite = yes
</syntaxhighlight>


Optional for single currency. Multiple: Recalculates to main via Google rates (view-only, not stored).
{{Tip|Set <code>pcap_dump_writethreads_max</code> to the number of CPU cores available for best performance on multi-core systems. Monitor <code>t0CPU</code> to ensure compression threads are not competing with the capture thread.}}


* Code: e.g., USD.
;Adjust Jitterbuffer Settings Based on Traffic Patterns
* Custom code: Symbol, e.g., "$".
:Jitterbuffer simulation adds CPU overhead. For production environments with stable networks, consider adjusting jitterbuffer settings to balance accuracy with performance.
* Name: Currency name.
* Country: Select.
* Main currency: Recalculate all to this.
* Description.


=== Settings ===
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# Fixed 50ms jitterbuffer (default: yes)
jitterbuffer_f1 = yes


* International prefixes: Distinguish local/international (default: 00, +).
# Fixed 200ms jitterbuffer (default: yes)
* Min. international length: Treat longer numbers as international if no prefix.
jitterbuffer_f2 = yes
* Local numbers are in: Prefix local numbers with international code for selected country.


== Watchdog ==
# Adaptive jitterbuffer up to 500ms (default: yes)
jitterbuffer_adapt = yes
</syntaxhighlight>


In Alerts/Anti Fraud: Two rules requiring aggregation enabled.
{{Warning|Disabling jitterbuffer analysis reduces CPU load but removes MOS and jitter quality metrics from CDRs. Only disable if you do not require voice quality monitoring.}}


=== Billing Over Quota ===
=== Advanced Kernel-Bypass Solutions ===
If kernel and driver tuning are insufficient, you can offload the capture process entirely by bypassing the kernel's network stack.


Checks billed sums per src IP/tel. number or groups for periods. Alerts on thresholds.
{| class="wikitable"
|-
! Solution !! Type !! CPU Reduction !! Use Case
|-
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
|-
| '''PF_RING ZC/DNA''' || Commercial || 90% → 20% || High-volume enterprise
|-
| '''Napatech SmartNICs''' || Hardware || <3% at 10 Gbit/s || Extreme performance requirements
|}


* Enabled: On/off.
;DPDK (Data Plane Development Kit)
* Every N minutes: Check frequency.
:A set of libraries and drivers for fast packet processing. VoIPmonitor can leverage DPDK to read packets directly from the network card, completely bypassing the kernel. See [[DPDK|DPDK guide]] for details.
* Enable hyperlinks: Links in emails.
* Hour/day/week/month: Max costs.
* Summary: Aggregate included callers/IPs.
* Include/Exclude IP/numbers: Filter calls. None: Per individual IP/number.


=== Billing High Rate ===
;PF_RING ZC/DNA
:A commercial software driver from ntop.org that dramatically reduces CPU load by bypassing the kernel.


Alerts on destinations > cost/minute threshold.
;Napatech SmartNICs
:Specialized hardware acceleration cards that deliver packets with near-zero CPU overhead.


* Enabled: On/off.
== Optimizing Disk I/O ==
* Every N minutes: Check frequency.
VoIPmonitor's modern storage engine is highly optimized to minimize random disk access, which is the primary cause of I/O bottlenecks.
* Enable hyperlinks: Links in emails.
* Limit: Cost/minute threshold.
* Include/Exclude IP/numbers: Filter calls. None: Per individual IP/number.


=== Viewing Costs in Reports and CDR View ===
=== VoIPmonitor Storage Strategy ===
Instead of writing a separate PCAP file for each call (which causes massive I/O load), VoIPmonitor groups all calls starting within the same minute into a single compressed <code>.tar</code> archive. This changes the I/O pattern from thousands of small, random writes to a few large, sequential writes, reducing IOPS by a factor of 10 or more.


After configuring billing, you may want to verify that costs are being calculated and then include them in reports.
'''Typical capacity:''' A standard 7200 RPM SATA drive can handle up to 2,000 concurrent calls with full recording.


==== Verifying Costs in CDR View ====
=== Filesystem Tuning (ext4) ===
For the spool directory (<code>/var/spool/voipmonitor</code>), using an optimized ext4 filesystem can improve performance.


To verify that billing is working correctly, check if costs appear in your CDR records:
<syntaxhighlight lang="bash">
# Format partition without a journal (use with caution, requires battery-backed RAID controller)
mke2fs -t ext4 -O ^has_journal /dev/sda2


# Navigate to the '''CDR''' view in the GUI
# Add to /etc/fstab for optimal performance
# Click the dropdown menu on the first column (column selection icon)
/dev/sda2  /var/spool/voipmonitor  ext4    errors=remount-ro,noatime,data=writeback,barrier=0 0 0
# Select '''PRICE''' from the list of available columns
</syntaxhighlight>
# The PRICE column will now show Customer (C:) and Operator (O:) prices for each call


If prices show as zero or empty, check:
{{Warning|Disabling the journal removes protection against filesystem corruption after crashes. Only use this with a battery-backed RAID controller.}}
* The billing system is enabled (.Settings > Billing Configuration > Settings > Enable by addresses" or "Enable by numbers")
* The source or destination IP is assigned to a billing table
* The called number matches a prefix or rule in the billing table
* The "Default customer billing" checkbox is enabled if you want customer prices applied to all calls


==== Including Costs in Daily Reports ====
=== RAID Controller Cache Policy ===
A misconfigured RAID controller is a common bottleneck. For database and spool workloads, the cache policy should be set to '''WriteBack''', not WriteThrough. This applies for RPM disks, not fast SSDs.


To generate reports with call costs:
'''Requirements:'''
* A healthy Battery Backup Unit (BBU) is required
* Specific commands vary by vendor (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>)
* Refer to vendor documentation for LSI, HP, and Dell controllers


# Navigate to '''GUI > Reports > Configure Daily Reports'''
== Optimizing Database Performance (MySQL/MariaDB) ==
# Select '''CDR summary''' as the report type
A well-tuned database is critical for both data ingestion from the sensor and GUI responsiveness.
# Enable the '''Price columns''' option
# Configure the date range and any filters (e.g., specific trunks, IP addresses)
# Save the report configuration


The report will include cost information based on your billing configuration. See [[Reports|Reports]] for more details on report types.
{{Note|For extreme scenarios (4,000+ concurrent calls, UI lag, high SQL queue, or 1000+ CDRs/sec), see [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, and centralized writer architecture.}}


=== AI Summary for RAG ===
=== Memory Configuration ===
The most critical database parameter is <code>innodb_buffer_pool_size</code>, which defines how much memory InnoDB uses to cache data and indexes.


'''Summary:''' This article overviews VoIPmonitor's billing features: Price types, tables, watchdog rules for quotas/expensive destinations; configuration sections for types/rules, holidays, assignments, excludes, aggregation, currency, settings; number matching type (original vs normalized) for prefix rules; viewing costs in CDR view and daily reports; and watchdog details.
{{Warning|On servers running both VoIPmonitor and MySQL, setting <code>innodb_buffer_pool_size</code> too high causes OOM (Out of Memory) killer events, resulting in CDR delays, crashes, and instability. See [[Sniffer_troubleshooting#Check_for_OOM_.28Out_of_Memory.29_Issues|OOM Troubleshooting]] for details.}}


'''Keywords:''' billing, customer price, operator price, billing tables, peak/offpeak, currency, watchdog, quota, high rate, PRICE column, daily reports, CDR summary, default customer billing, prefix number, normalized, original, international prefixes, number matching
==== Buffer Pool Sizing ====
 
{| class="wikitable"
|-
! Server Type !! Calculation !! Example (32GB RAM)
|-
| '''Shared''' (VoIPmonitor + MySQL) || (Total RAM - VoIPmonitor - OS overhead) / 2 || 14GB
|-
| '''Dedicated''' MySQL server || 50-70% of total RAM || 20-22GB
|}
 
For shared servers, use this formula:
<syntaxhighlight lang="text">
innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead - safety margin) / 2
 
Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory: ~2GB (check with ps aux)
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G
</syntaxhighlight>
 
==== RAM Recommendations ====
{| class="wikitable"
|-
! Deployment Size !! Minimum RAM !! Recommended RAM
|-
| Small (<500 concurrent calls) || 8GB || 16GB
|-
| Medium (500-2000 calls) || 16GB || 32GB
|-
| Large (>2000 calls) || 32GB || 64GB+
|}
 
==== Disable Graphical Desktop ====
A graphical desktop environment consumes 1-2GB of RAM unnecessarily. VoIPmonitor is managed through a web interface and does not require a desktop.
 
<syntaxhighlight lang="bash">
# Disable display manager
systemctl stop gdm          # Ubuntu/Debian with GDM
systemctl disable gdm
 
# Set default to multi-user (no GUI)
systemctl set-default multi-user.target
 
# Verify memory freed
free -h
</syntaxhighlight>
 
=== Other Key Parameters ===
 
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
 
[mysqld]
# Buffer pool size (calculate per above)
innodb_buffer_pool_size = 14G
 
# Flush logs to OS cache, write to disk once per second (faster, minimal data loss risk)
innodb_flush_log_at_trx_commit = 2
 
# Store each table in its own file (essential for partitioning)
innodb_file_per_table = 1
 
# LZ4 compression for modern MariaDB
innodb_compression_algorithm = lz4
</syntaxhighlight>
 
{{Warning|For deployments with 4,000+ concurrent calls experiencing UI unresponsiveness, database queue growth (SQLq), or extremely high CDR insertion rates (1000+ CDRs/sec), the intermediate settings above may not be sufficient. See [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for extreme performance configurations optimized for 10,000+ concurrent calls, including innodb_flush_log_at_trx_commit=0, hourly partitioning, and centralized writer architecture.}}
 
=== Slow Query Log ===
 
The MySQL slow query log can consume significant memory and disk I/O on high-traffic systems. If you are experiencing high memory utilization alerts or performance issues with the database server, consider adjusting or disabling the slow query log.
 
{{Warning|Disabling the slow query log removes the ability to analyze slow queries for performance optimization. Only disable it temporarily or if you are certain you do not need it.}}
 
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf or /etc/my.cnf.d/mysql-server.cnf
 
[mysqld]
# Disable slow query log (set to 1 to enable)
slow_query_log = 0
 
# Alternative: Increase threshold to only log extremely slow queries (e.g., 600 seconds = 10 minutes)
long_query_time = 600
</syntaxhighlight>
 
After changing MySQL configuration, restart the database and dependent services:
 
<syntaxhighlight lang="bash">
# Restart MySQL/MariaDB
systemctl restart mariadb  # or mysql
 
# Restart VoIPmonitor sniffer (depends on database)
systemctl restart voipmonitor
</syntaxhighlight>
 
=== Troubleshooting: Connection Refused Errors ===
 
If the VoIPmonitor GUI or sniffer experiences crashes with "Can't connect to MySQL server" or "Connection refused" errors, verify that MariaDB is configured with adequate memory.
 
'''Symptoms:'''
* GUI server crashes repeatedly
* MariaDB connection refused errors in logs
* MariaDB service appears to be running (`systemctl status mariadb` shows active)
* Issues occur intermittently during peak call volumes
 
'''Cause:'''
When <code>innodb_buffer_pool_size</code> is set too low (e.g., default 128M or 256M), MariaDB may run out of memory for the InnoDB buffer pool during high call volumes or complex database queries. This can cause connection failures even while the database service remains technically running.
 
'''Solution:'''
Edit the MariaDB configuration file (path varies by distribution):
 
<syntaxhighlight lang="bash">
# Common configuration file locations:
# Debian/Ubuntu: /etc/mysql/mariadb.conf.d/50-server.cnf
# RHEL/CentOS: /etc/my.cnf.d/mariadb-server.cnf
# Generic: /etc/mysql/my.cnf
 
# Edit the file and increase innodb_buffer_pool_size
nano /etc/mysql/mariadb.conf.d/50-server.cnf
</syntaxhighlight>
 
Set a higher buffer pool size (example for a server with 8-16GB RAM):
 
<syntaxhighlight lang="ini">
[mysqld]
# Increase from default 128M to 6G or higher, depending on available RAM
innodb_buffer_pool_size = 6G
</syntaxhighlight>
 
Restart MariaDB to apply the change:
 
<syntaxhighlight lang="bash">
systemctl restart mariadb
</syntaxhighlight>
 
{{Tip|Use the [[#Buffer Pool Sizing|Buffer Pool Sizing formula]] above to calculate the optimal value for your server. For most production deployments with 500-2000 concurrent calls, values between 6G and 14G are typical.}}
 
=== Database Partitioning ===
VoIPmonitor automatically splits large tables (like <code>cdr</code>) into daily partitions. This is enabled by default and '''highly recommended'''.
 
'''Benefits:'''
* Massively improves GUI query performance (only relevant partitions are scanned)
* Allows instant deletion of old data by dropping partitions (thousands of times faster than DELETE)
 
See [[Data_Cleaning#The_Modern_Method:_Partitioning_.28Recommended.29|Database Partitioning]] for configuration details.
 
== Scaling Through Component Separation ==
 
For deployments requiring high availability and independent scaling of different workloads, VoIPmonitor's three core components (MySQL database, Sensor, and GUI) can be deployed on separate dedicated hosts. This architecture allows each component to be scaled based on its specific resource requirements.
 
=== When to Use Component Separation ===
 
Consider separating components when:
* You need to scale packet capture without affecting GUI responsiveness
* Database queries are becoming a bottleneck during peak hours
* You have multiple teams managing different aspects of the system (DBAs, network engineers, web administrators)
* You want to implement different high availability strategies for each component
* Your traffic volume exceeds 5,000-10,000 concurrent calls
 
=== Three-Host Architecture ===
 
In this model, each core VoIPmonitor component runs on a dedicated server:
 
{| class="wikitable"
|-
! Host !! Component !! Primary Resources !! Scaling Strategy
|-
| '''Host 1: MySQL Database''' || MariaDB/MySQL server || RAM (50-70% for buffer pool), fast I/O (SSD/NVMe) || Add RAM, upgrade storage, add read replicas for reporting
|-
| '''Host 2: Sensor(s)''' || Packet capture and analysis || Single CPU core (t0 thread), network I/O || Add more sensors for distributed capture, use DPDK/PF_RING
|-
| '''Host 3: GUI''' || Web interface (PHP + Apache/Nginx) || CPU for database queries, network bandwidth || Horizontal scaling with load balancer, caching
|}
 
=== Configuration Steps ===
 
==== Step 1: Deploy Dedicated MySQL Server ====
 
Configure the database server with proper MySQL/MariaDB tuning for VoIPmonitor. Use the dedicated server buffer pool sizing from [[#Memory_Configuration|Memory Configuration]].
 
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf on MySQL Host
[mysqld]
bind-address = 0.0.0.0  # Allow remote connections
innodb_buffer_pool_size = 50G  # 50-70% of total RAM for dedicated DB server
innodb_flush_log_at_trx_commit = 2
innodb_file_per_table = 1
innodb_compression_algorithm = lz4
</syntaxhighlight>
 
Grant access for remote connections:
 
<syntaxhighlight lang="sql">
CREATE USER 'voipmonitor'@'%' IDENTIFIED BY 'strong_password';
GRANT ALL PRIVILEGES ON voipmonitor.* TO 'voipmonitor'@'%';
FLUSH PRIVILEGES;
</syntaxhighlight>
 
==== Step 2: Deploy Sensor(s) ====
 
Install the VoIPmonitor sensor on one or more dedicated capture servers. Configure the sensor to connect to the remote MySQL database.
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on Sensor Host
[general]
id_sensor = 1  # Unique per sensor
# Remote MySQL connection
mysqlhost = mysql.server.ip  # IP of dedicated MySQL server
mysqldb = voipmonitor
mysqluser = voipmonitor
mysqlpassword = strong_password
 
# Capture settings
interface = eth0
sipport = 5060,5061
 
# Performance tuning
pcap_dump_writethreads = 1
pcap_dump_writethreads_max = 32
</syntaxhighlight>
 
==== Step 3: Deploy Dedicated GUI ====
 
Install the VoIPmonitor GUI on a separate web server and configure it to connect to the remote MySQL database.
 
<syntaxhighlight lang="bash">
# Extract GUI package
tar -xzf voipmonitor-gui-latest.tar.gz -C /var/www/html/
 
# Edit GUI configuration
/var/www/html/voipmonitor/config/system_configuration.php
</syntaxhighlight>
 
Configure database connection through the GUI interface:
 
# Open <nowiki>http://gui.server.ip/voipmonitor</nowiki> in a browser
# Navigate to '''Settings > System Configuration > Database'''
# Configure:
# * Host: mysql.server.ip
# * User: voipmonitor
# * Password: strong_password
# * Database: voipmonitor
 
Alternatively, edit the configuration file directly:
 
<syntaxhighlight lang="php">
// /var/www/html/voipmonitor/config/system_configuration.php
define('DB_HOST', 'mysql.server.ip');
define('DB_USER', 'voipmonitor');
define('DB_PASS', 'strong_password');
define('DB_NAME', 'voipmonitor');
</syntaxhighlight>
 
==== Network Connectivity Requirements ====
 
Ensure firewall rules allow:
 
{| class="wikitable"
|-
! Source !! Destination !! Port !! Purpose
|-
| Sensor host || MySQL server || 3306 || Write CDRs to database
|-
| GUI host || MySQL server || 3306 || Read/Write for queries and configuration
|-
| User browsers || GUI host || 80, 443 || Access web interface
|-
| (Optional) GUI host || Sensor host(s) || 5029 || Retrieve PCAP files (if using local processing mode)
|}
 
=== Advantages of Component Separation ===
 
* '''Independent Scaling''' - Each component can be upgraded independently based on load
* '''Resource Optimization''' - Database servers need more RAM, sensors need fast CPUs, GUI needs network and query performance
* '''Isolation''' - Issues on one component do not affect others (e.g., GUI upgrade does not stop packet capture)
* '''Team Specialization''' - DBAs manage database, network engineers manage sensors, web team manages GUI
* '''High Availability Options''' - Different HA strategies for each component:
** MySQL: Galera cluster for multi-master replication
** Sensors: Client-server mode with failover
** GUI: Load balancer with multiple web servers
 
=== Comparison with Distributed Client-Server Mode ===
 
The three-host component separation architecture differs from [[Sniffer_distributed_architecture|Client-Server mode]]:
 
{| class="wikitable"
|-
! Architecture !! Scenario !! Best For
|-
| '''Component Separation (this section)''' || 3 dedicated hosts: MySQL + GUI + Sensor(s) || Single data center, independent scaling needs, centralized database
|-
| '''Client-Server Mode''' || Central server + remote sensors forwarding packets || Multi-site deployments, geographic distribution, low-resource remote sensors
|}
 
Both models can be combined: Use component separation at the central site, while remote sites use client-server mode to forward data.
 
=== Monitoring Component Health ===
 
Monitor each component separately:
 
* '''MySQL Server:''' Check <code>innodb_buffer_pool_size</code>, query throughput, slow queries
* '''Sensor(s):''' Monitor <code>t0CPU</code>, packet drops, SQL queue depth
* '''GUI Server:''' Monitor web server load, PHP-FPM statistics, page response times
 
Use the [[Sniffer_troubleshooting|Sniffer Troubleshooting]] guide for sensor issues and database performance tools for MySQL tuning.
 
== Monitoring Live Performance ==
VoIPmonitor logs detailed performance metrics every 10 seconds to syslog.
 
<syntaxhighlight lang="bash">
# Debian/Ubuntu
tail -f /var/log/syslog | grep voipmonitor
 
# CentOS/RHEL
tail -f /var/log/messages | grep voipmonitor
</syntaxhighlight>
 
=== Understanding the Log Output ===
Sample log line:
<syntaxhighlight lang="text">
voipmonitor[15567]: calls[315][355] PS[C:4 S:29/29 R:6354 A:6484] SQLq[0] heap[0|0|0] comp[54] [12.6Mb/s] t0CPU[5.2%] ... RSS/VSZ[323|752]MB
</syntaxhighlight>
 
{| class="wikitable"
|-
! Metric !! Description !! Warning Threshold
|-
| <code>calls[X][Y]</code> || X = active calls, Y = total calls in memory || -
|-
| <code>SQLq[C]</code> || SQL queries waiting to be sent to database || Growing consistently = DB bottleneck
|-
| <code>heap[A{{!}}B{{!}}C]</code> || Memory usage % for internal buffers || A = 100% → packet drops
|-
| <code>t0CPU[X%]</code> || '''Main packet capture thread CPU usage''' || >90-95% = capture limit reached
|-
| <code>RSS/VSZ[X{{!}}Y]MB</code> || Resident/Virtual memory usage || RSS growing = memory leak
|}
 
=== Performance Diagrams ===
 
The following diagrams illustrate the difference between standard kernel packet capture and optimized solutions:
 
[[File:kernelstandarddiagram.png|thumb|center|600px|Standard kernel packet capture path - packets traverse multiple kernel layers before reaching VoIPmonitor]]
 
[[File:ntop.png|thumb|center|600px|PF_RING/DPDK bypass mode - packets are delivered directly to VoIPmonitor, bypassing the kernel network stack]]
 
== See Also ==
* [[Sniffer_troubleshooting]] - Troubleshooting guide including OOM issues
* [[Data_Cleaning]] - Database and spool retention configuration
* [[Sniffer_configuration]] - Complete configuration reference
* [[DPDK]] - DPDK setup guide
* [[IO_Measurement]] - Disk I/O benchmarking tools
 
== AI Summary for RAG ==
'''Summary:''' Expert guide to scaling VoIPmonitor for high-traffic environments. Covers three main bottlenecks: (1) Packet Capturing - optimized via TPACKET_V3, NIC tuning with ethtool (ring buffer, interrupt coalescing), configuration-level optimizations (interface_ip_filter more efficient than BPF filter, pcap_dump_writethreads for compression thread tuning, jitterbuffer settings for CPU/performance balance), and kernel-bypass solutions (DPDK, PF_RING, Napatech); (2) Disk I/O - VoIPmonitor uses TAR-based storage to reduce IOPS, with ext4 tuning and RAID WriteBack cache; (3) Database - critical innodb_buffer_pool_size tuning with formula for shared servers: (Total RAM - VoIPmonitor - OS overhead) / 2. For 32GB shared server, recommend 14GB buffer pool. Dedicated servers can use 50-70% of RAM. Covers slow query log as a memory/I/O consumer and disabling it for memory optimization. Covers partitioning benefits and syslog monitoring (t0CPU, SQLq, heap metrics). TROUBLESHOOTING: GUI or sniffer crashes with "MariaDB connection refused" errors indicate innodb_buffer_pool_size is set too low (e.g., default 128M or 256M). This causes MariaDB to run out of memory for the InnoDB buffer pool during high call volumes, resulting in connection failures even while the service remains active. Solution: Edit MariaDB config file (typically /etc/mysql/mariadb.conf.d/50-server.cnf or /etc/my.cnf.d/mariadb-server.cnf) and increase innodb_buffer_pool_size to 6G or higher based on available RAM. For most production deployments with 500-2000 concurrent calls, values between 6G and 14G are typical. Restart MariaDB with systemctl restart mariadb after changing the configuration. For extreme scenarios (4,000+ concurrent calls, UI lag, unresponsive GUI, high SQL queue), see High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, centralized writer architecture, RTP thread tuning (rtpthreads, rtpthreads_start), and MySQL optimization settings (innodb_thread_concurrency, innodb_io_capacity, innodb_flush_method=O_DIRECT).
 
COMPONENT SEPARATION ARCHITECTURE: For deployments requiring independent scaling, VoIPmonitor's three core components (MySQL database, Sensor, and GUI) can be deployed on separate dedicated hosts. This architecture is recommended when needing to scale packet capture without affecting GUI responsiveness, database queries are becoming a bottleneck, multiple teams manage different aspects (DBAs, network engineers, web administrators), traffic volume exceeds 5,000-10,000 concurrent calls, or different HA strategies are needed. Three-Host Architecture: Host 1 is MySQL Database (needs 50-70% RAM for buffer pool, fast SSD/NVMe I/O, scale by adding RAM or read replicas, consider Galera cluster for HA); Host 2 is Sensor(s) (needs fast CPU single core for t0 thread and network I/O, scale by adding more sensors for distributed capture or use DPDK/PF_RING); Host 3 is GUI web interface (needs CPU for database queries and network bandwidth, can be horizontally scaled with load balancer and caching, use multiple web servers with load balancer for HA). Configuration: MySQL server with bind-address=0.0.0.0 and remote user access via CREATE USER 'voipmonitor'@'%' with GRANT ALL PRIVILEGES. Sensor configured with mysqlhost pointing to MySQL server IP. GUI configured via Settings > System Configuration > Database or by editing config/system_configuration.php to set DB_HOST, DB_USER, DB_PASS, DB_NAME. Network connectivity requires firewall rules: Sensor→MySQL (port 3306), GUI→MySQL (port 3306), Users→GUI (ports 80/443), GUI→Sensor(s) (port 5029 for PCAP retrieval if using local processing). Advantages: independent scaling, resource optimization (database needs RAM, sensors need CPU, GUI needs network), isolation (issues on one component don't affect others), team specialization, different HA strategies per component. Comparison with Client-Server Mode: Component Separation (3 dedicated hosts MySQL+GUI+Sensor) is best for single data center and independent scaling needs while Client-Server Mode (central server+remote sensors forwarding packets) is best for multi-site and geographic distribution with low-resource remote sensors. Both models can be combined. Monitor components separately: MySQL (innodb_buffer_pool_size, query throughput, slow queries), Sensors (t0CPU, packet drops, SQL queue), GUI (web server load, PHP-FPM, response times).
 
'''Keywords:''' scaling, performance tuning, bottleneck, t0CPU, TPACKET_V3, DPDK, PF_RING, ethtool, ring buffer, interface_ip_filter, BPF filter, pcap_dump_writethreads, jitterbuffer, jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt, compression threads, PCAP async write, innodb_buffer_pool_size, OOM killer, shared server memory, database partitioning, SQLq monitoring, slow query log, slow_query_log, long_query_time, UI lag, unresponsive GUI, high performance, 4000 concurrent calls, 5000 concurrent calls, innodb_flush_log_at_trx_commit=0, hourly partitioning, rtpthreads, rtpthreads_start, RTP threads, innodb_io_capacity, innodb_thread_concurrency, innodb_flush_method, extreme performance, High-Performance Manual, component separation, three host architecture, dedicated MySQL server, dedicated GUI server, remote database, mysqlhost configuration, GUI remote database, independent scaling, high availability, Galera cluster, load balancer, DB_HOST, DB_USER, DB_PASS, DB_NAME, bind-address, firewall ports, GUI settings database configuration, connection refused, MariaDB connection refused, GUI crashes, database connection errors, mariadb.conf.d/50-server.cnf, MySQL crash, MariaDB crash, can't connect to MySQL


'''Key Questions:'''
'''Key Questions:'''
* What are the key billing features in VoIPmonitor?
* How do I scale VoIPmonitor for thousands of concurrent calls?
* How to create and configure billing tables/rules?
* What are the main performance bottlenecks in VoIPmonitor?
* What is state holidays for in billing?
* How do I fix high t0CPU usage?
* How to exclude calls from billing?
* What is DPDK and when should I use it?
* How to handle multiple currencies?
* How do I calculate innodb_buffer_pool_size for a shared server?
* What are international prefix settings?
* What happens if innodb_buffer_pool_size is set too high?
* How do billing watchdog rules function?
* What causes "MariaDB connection refused" errors in the GUI or sniffer?
* How to view costs in the CDR view?
* How do I fix GUI crashes due to MariaDB connection refused errors?
* How to include costs in daily reports?
* How do I troubleshoot database connection issues with MariaDB?
* What does Default customer billing do?
* How do I interpret the performance metrics in syslog?
* What is the difference between normalized and original number matching for billing prefixes?
* Should I use a dedicated database server for VoIPmonitor?
* How do I configure billing rates based on specific call number prefixes?
* How much RAM does a VoIPmonitor server need?
* How can the slow query log affect memory utilization?
* How do I disable or adjust the MySQL slow query log?
* Is interface_ip_filter more efficient than the filter option?
* How do I optimize PCAP compression threads for high traffic?
* Which jitterbuffer settings affect CPU load the most?
* What configuration options reduce CPU overhead?
* How do I deploy VoIPmonitor with MySQL, GUI, and Sensor on separate servers?
* When should I use component separation architecture for VoIPmonitor?
* What is the three host architecture for VoIPmonitor scaling?
* How do I configure the GUI to connect to a remote MySQL database?
* How do I configure sensors to write to a remote MySQL server?
* What are the advantages of separating VoIPmonitor components onto different hosts?
* How do I scale the MySQL database independently from sensors and GUI?
* How do I scale the GUI independently from database and sensors?
* What is the difference between component separation and client-server mode?
* Can I combine component separation with client-server mode for VoIPmonitor?
* How do I configure firewall rules for component separation architecture?
* When should I use a dedicated GUI server for VoIPmonitor?

Revision as of 11:24, 6 January 2026


This guide provides a comprehensive overview of performance tuning and scaling for VoIPmonitor. It covers the three primary system bottlenecks and offers practical, expert-level advice for optimizing your deployment for high traffic loads.

Understanding Performance Bottlenecks

A VoIPmonitor deployment's maximum capacity is determined by three potential bottlenecks. Identifying and addressing the correct one is key to achieving high performance.

The three bottlenecks are:

  1. Packet Capturing (CPU & Network Stack): The ability of a single CPU core to read packets from the network interface. This is often the first limit encountered.
  2. Disk I/O (Storage): The speed at which the sensor can write PCAP files to disk. Critical when call recording is enabled.
  3. Database Performance (MySQL/MariaDB): The rate at which the database can ingest CDRs and serve data to the GUI.

On a modern, well-tuned server (e.g., 24-core Xeon, 10Gbit NIC), a single VoIPmonitor instance can handle up to 10,000 concurrent calls with full RTP analysis and recording, or over 60,000 concurrent calls with SIP-only analysis.

Optimizing Packet Capturing (CPU & Network)

The most performance-critical task is the initial packet capture, handled by a single, highly optimized thread (t0). If this thread's CPU usage (t0CPU in logs) approaches 100%, you are hitting the capture limit.

Use a Modern Linux Kernel & VoIPmonitor Build

Modern Linux kernels (3.2+) and VoIPmonitor builds include TPACKET_V3 support, a high-speed packet capture mechanism. This is the single most important factor for high performance.

Recommendation: Always use a recent Linux distribution (AlmaLinux, Rocky Linux, or Debian) and the latest VoIPmonitor static binary. With this combination, a standard Intel 10Gbit NIC can often handle up to 2 Gbit/s of VoIP traffic without special drivers.

Network Stack & Driver Tuning

For high-traffic environments (>500 Mbit/s), fine-tuning the network driver and kernel parameters is essential.

NIC Ring Buffer

The ring buffer is a queue between the network card driver and VoIPmonitor. A larger buffer prevents packet loss during short CPU usage spikes.

# Check maximum size
ethtool -g eth0

# Set to maximum (e.g., 16384)
ethtool -G eth0 rx 16384

Interrupt Coalescing

This setting batches multiple hardware interrupts into one, reducing CPU overhead.

ethtool -C eth0 rx-usecs 1022

Applying Settings Persistently

To make these settings permanent, add them to your network configuration. For Debian/Ubuntu using /etc/network/interfaces:

auto eth0
iface eth0 inet manual
    up ip link set $IFACE up
    up ip link set $IFACE promisc on
    up ethtool -G $IFACE rx 16384
    up ethtool -C $IFACE rx-usecs 1022

Note: Modern systems using NetworkManager or systemd-networkd require different configuration methods.

Configuration-Level Optimizations

Before investing in kernel-bypass solutions, ensure your voipmonitor.conf is optimized for performance. Several configuration parameters can significantly reduce CPU load and improve packet capture efficiency.

Use interface_ip_filter Instead of filter
If you need to filter by IP address or subnet, use interface_ip_filter instead of the general BPF filter option. The interface_ip_filter directive is more efficient and reduces CPU overhead compared to complex BPF filters.
# More efficient IP-based filtering
interface_ip_filter = 192.168.0.0/24
interface_ip_filter = 10.0.0.0/8

# Less efficient BPF filtering (avoid if possible)
# filter = udp and (host 192.168.0.0/24 or host 10.0.0.0/8)

ℹ️ Note: See Sniffer_configuration for the complete reference and description of interface_ip_filter (Interface Selection section).

Optimize PCAP Compression Threads
For systems with high call recording rates, PCAP compression can become CPU-intensive. VoIPmonitor can automatically scale compression threads.
# /etc/voipmonitor.conf
# Initial compression threads (auto-scales based on load)
pcap_dump_writethreads = 1

# Maximum compression threads (adjust based on CPU cores)
pcap_dump_writethreads_max = 32

# Asynchronous PCAP writing (enabled by default)
pcap_dump_asyncwrite = yes

💡 Tip: Set pcap_dump_writethreads_max to the number of CPU cores available for best performance on multi-core systems. Monitor t0CPU to ensure compression threads are not competing with the capture thread.

Adjust Jitterbuffer Settings Based on Traffic Patterns
Jitterbuffer simulation adds CPU overhead. For production environments with stable networks, consider adjusting jitterbuffer settings to balance accuracy with performance.
# /etc/voipmonitor.conf
# Fixed 50ms jitterbuffer (default: yes)
jitterbuffer_f1 = yes

# Fixed 200ms jitterbuffer (default: yes)
jitterbuffer_f2 = yes

# Adaptive jitterbuffer up to 500ms (default: yes)
jitterbuffer_adapt = yes

⚠️ Warning: Disabling jitterbuffer analysis reduces CPU load but removes MOS and jitter quality metrics from CDRs. Only disable if you do not require voice quality monitoring.

Advanced Kernel-Bypass Solutions

If kernel and driver tuning are insufficient, you can offload the capture process entirely by bypassing the kernel's network stack.

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
PF_RING ZC/DNA Commercial 90% → 20% High-volume enterprise
Napatech SmartNICs Hardware <3% at 10 Gbit/s Extreme performance requirements
DPDK (Data Plane Development Kit)
A set of libraries and drivers for fast packet processing. VoIPmonitor can leverage DPDK to read packets directly from the network card, completely bypassing the kernel. See DPDK guide for details.
PF_RING ZC/DNA
A commercial software driver from ntop.org that dramatically reduces CPU load by bypassing the kernel.
Napatech SmartNICs
Specialized hardware acceleration cards that deliver packets with near-zero CPU overhead.

Optimizing Disk I/O

VoIPmonitor's modern storage engine is highly optimized to minimize random disk access, which is the primary cause of I/O bottlenecks.

VoIPmonitor Storage Strategy

Instead of writing a separate PCAP file for each call (which causes massive I/O load), VoIPmonitor groups all calls starting within the same minute into a single compressed .tar archive. This changes the I/O pattern from thousands of small, random writes to a few large, sequential writes, reducing IOPS by a factor of 10 or more.

Typical capacity: A standard 7200 RPM SATA drive can handle up to 2,000 concurrent calls with full recording.

Filesystem Tuning (ext4)

For the spool directory (/var/spool/voipmonitor), using an optimized ext4 filesystem can improve performance.

# Format partition without a journal (use with caution, requires battery-backed RAID controller)
mke2fs -t ext4 -O ^has_journal /dev/sda2

# Add to /etc/fstab for optimal performance
/dev/sda2   /var/spool/voipmonitor  ext4    errors=remount-ro,noatime,data=writeback,barrier=0 0 0

⚠️ Warning: Disabling the journal removes protection against filesystem corruption after crashes. Only use this with a battery-backed RAID controller.

RAID Controller Cache Policy

A misconfigured RAID controller is a common bottleneck. For database and spool workloads, the cache policy should be set to WriteBack, not WriteThrough. This applies for RPM disks, not fast SSDs.

Requirements:

  • A healthy Battery Backup Unit (BBU) is required
  • Specific commands vary by vendor (megacli, ssacli, perccli)
  • Refer to vendor documentation for LSI, HP, and Dell controllers

Optimizing Database Performance (MySQL/MariaDB)

A well-tuned database is critical for both data ingestion from the sensor and GUI responsiveness.

ℹ️ Note:

Memory Configuration

The most critical database parameter is innodb_buffer_pool_size, which defines how much memory InnoDB uses to cache data and indexes.

⚠️ Warning: On servers running both VoIPmonitor and MySQL, setting innodb_buffer_pool_size too high causes OOM (Out of Memory) killer events, resulting in CDR delays, crashes, and instability. See OOM Troubleshooting for details.

Buffer Pool Sizing

Server Type Calculation Example (32GB RAM)
Shared (VoIPmonitor + MySQL) (Total RAM - VoIPmonitor - OS overhead) / 2 14GB
Dedicated MySQL server 50-70% of total RAM 20-22GB

For shared servers, use this formula:

innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead - safety margin) / 2

Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory: ~2GB (check with ps aux)
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G

RAM Recommendations

Deployment Size Minimum RAM Recommended RAM
Small (<500 concurrent calls) 8GB 16GB
Medium (500-2000 calls) 16GB 32GB
Large (>2000 calls) 32GB 64GB+

Disable Graphical Desktop

A graphical desktop environment consumes 1-2GB of RAM unnecessarily. VoIPmonitor is managed through a web interface and does not require a desktop.

# Disable display manager
systemctl stop gdm          # Ubuntu/Debian with GDM
systemctl disable gdm

# Set default to multi-user (no GUI)
systemctl set-default multi-user.target

# Verify memory freed
free -h

Other Key Parameters

# /etc/mysql/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf

[mysqld]
# Buffer pool size (calculate per above)
innodb_buffer_pool_size = 14G

# Flush logs to OS cache, write to disk once per second (faster, minimal data loss risk)
innodb_flush_log_at_trx_commit = 2

# Store each table in its own file (essential for partitioning)
innodb_file_per_table = 1

# LZ4 compression for modern MariaDB
innodb_compression_algorithm = lz4

⚠️ Warning:

Slow Query Log

The MySQL slow query log can consume significant memory and disk I/O on high-traffic systems. If you are experiencing high memory utilization alerts or performance issues with the database server, consider adjusting or disabling the slow query log.

⚠️ Warning: Disabling the slow query log removes the ability to analyze slow queries for performance optimization. Only disable it temporarily or if you are certain you do not need it.

# /etc/mysql/my.cnf or /etc/my.cnf.d/mysql-server.cnf

[mysqld]
# Disable slow query log (set to 1 to enable)
slow_query_log = 0

# Alternative: Increase threshold to only log extremely slow queries (e.g., 600 seconds = 10 minutes)
long_query_time = 600

After changing MySQL configuration, restart the database and dependent services:

# Restart MySQL/MariaDB
systemctl restart mariadb  # or mysql

# Restart VoIPmonitor sniffer (depends on database)
systemctl restart voipmonitor

Troubleshooting: Connection Refused Errors

If the VoIPmonitor GUI or sniffer experiences crashes with "Can't connect to MySQL server" or "Connection refused" errors, verify that MariaDB is configured with adequate memory.

Symptoms:

  • GUI server crashes repeatedly
  • MariaDB connection refused errors in logs
  • MariaDB service appears to be running (`systemctl status mariadb` shows active)
  • Issues occur intermittently during peak call volumes

Cause: When innodb_buffer_pool_size is set too low (e.g., default 128M or 256M), MariaDB may run out of memory for the InnoDB buffer pool during high call volumes or complex database queries. This can cause connection failures even while the database service remains technically running.

Solution: Edit the MariaDB configuration file (path varies by distribution):

# Common configuration file locations:
# Debian/Ubuntu: /etc/mysql/mariadb.conf.d/50-server.cnf
# RHEL/CentOS: /etc/my.cnf.d/mariadb-server.cnf
# Generic: /etc/mysql/my.cnf

# Edit the file and increase innodb_buffer_pool_size
nano /etc/mysql/mariadb.conf.d/50-server.cnf

Set a higher buffer pool size (example for a server with 8-16GB RAM):

[mysqld]
# Increase from default 128M to 6G or higher, depending on available RAM
innodb_buffer_pool_size = 6G

Restart MariaDB to apply the change:

systemctl restart mariadb

💡 Tip: Use the Buffer Pool Sizing formula above to calculate the optimal value for your server. For most production deployments with 500-2000 concurrent calls, values between 6G and 14G are typical.

Database Partitioning

VoIPmonitor automatically splits large tables (like cdr) into daily partitions. This is enabled by default and highly recommended.

Benefits:

  • Massively improves GUI query performance (only relevant partitions are scanned)
  • Allows instant deletion of old data by dropping partitions (thousands of times faster than DELETE)

See Database Partitioning for configuration details.

Scaling Through Component Separation

For deployments requiring high availability and independent scaling of different workloads, VoIPmonitor's three core components (MySQL database, Sensor, and GUI) can be deployed on separate dedicated hosts. This architecture allows each component to be scaled based on its specific resource requirements.

When to Use Component Separation

Consider separating components when:

  • You need to scale packet capture without affecting GUI responsiveness
  • Database queries are becoming a bottleneck during peak hours
  • You have multiple teams managing different aspects of the system (DBAs, network engineers, web administrators)
  • You want to implement different high availability strategies for each component
  • Your traffic volume exceeds 5,000-10,000 concurrent calls

Three-Host Architecture

In this model, each core VoIPmonitor component runs on a dedicated server:

Host Component Primary Resources Scaling Strategy
Host 1: MySQL Database MariaDB/MySQL server RAM (50-70% for buffer pool), fast I/O (SSD/NVMe) Add RAM, upgrade storage, add read replicas for reporting
Host 2: Sensor(s) Packet capture and analysis Single CPU core (t0 thread), network I/O Add more sensors for distributed capture, use DPDK/PF_RING
Host 3: GUI Web interface (PHP + Apache/Nginx) CPU for database queries, network bandwidth Horizontal scaling with load balancer, caching

Configuration Steps

Step 1: Deploy Dedicated MySQL Server

Configure the database server with proper MySQL/MariaDB tuning for VoIPmonitor. Use the dedicated server buffer pool sizing from Memory Configuration.

# /etc/mysql/my.cnf on MySQL Host
[mysqld]
bind-address = 0.0.0.0  # Allow remote connections
innodb_buffer_pool_size = 50G  # 50-70% of total RAM for dedicated DB server
innodb_flush_log_at_trx_commit = 2
innodb_file_per_table = 1
innodb_compression_algorithm = lz4

Grant access for remote connections:

CREATE USER 'voipmonitor'@'%' IDENTIFIED BY 'strong_password';
GRANT ALL PRIVILEGES ON voipmonitor.* TO 'voipmonitor'@'%';
FLUSH PRIVILEGES;

Step 2: Deploy Sensor(s)

Install the VoIPmonitor sensor on one or more dedicated capture servers. Configure the sensor to connect to the remote MySQL database.

# /etc/voipmonitor.conf on Sensor Host
[general]
id_sensor = 1  # Unique per sensor
# Remote MySQL connection
mysqlhost = mysql.server.ip  # IP of dedicated MySQL server
mysqldb = voipmonitor
mysqluser = voipmonitor
mysqlpassword = strong_password

# Capture settings
interface = eth0
sipport = 5060,5061

# Performance tuning
pcap_dump_writethreads = 1
pcap_dump_writethreads_max = 32

Step 3: Deploy Dedicated GUI

Install the VoIPmonitor GUI on a separate web server and configure it to connect to the remote MySQL database.

# Extract GUI package
tar -xzf voipmonitor-gui-latest.tar.gz -C /var/www/html/

# Edit GUI configuration
/var/www/html/voipmonitor/config/system_configuration.php

Configure database connection through the GUI interface:

  1. Open http://gui.server.ip/voipmonitor in a browser
  2. Navigate to Settings > System Configuration > Database
  3. Configure:
  4. * Host: mysql.server.ip
  5. * User: voipmonitor
  6. * Password: strong_password
  7. * Database: voipmonitor

Alternatively, edit the configuration file directly:

// /var/www/html/voipmonitor/config/system_configuration.php
define('DB_HOST', 'mysql.server.ip');
define('DB_USER', 'voipmonitor');
define('DB_PASS', 'strong_password');
define('DB_NAME', 'voipmonitor');

Network Connectivity Requirements

Ensure firewall rules allow:

Source Destination Port Purpose
Sensor host MySQL server 3306 Write CDRs to database
GUI host MySQL server 3306 Read/Write for queries and configuration
User browsers GUI host 80, 443 Access web interface
(Optional) GUI host Sensor host(s) 5029 Retrieve PCAP files (if using local processing mode)

Advantages of Component Separation

  • Independent Scaling - Each component can be upgraded independently based on load
  • Resource Optimization - Database servers need more RAM, sensors need fast CPUs, GUI needs network and query performance
  • Isolation - Issues on one component do not affect others (e.g., GUI upgrade does not stop packet capture)
  • Team Specialization - DBAs manage database, network engineers manage sensors, web team manages GUI
  • High Availability Options - Different HA strategies for each component:
    • MySQL: Galera cluster for multi-master replication
    • Sensors: Client-server mode with failover
    • GUI: Load balancer with multiple web servers

Comparison with Distributed Client-Server Mode

The three-host component separation architecture differs from Client-Server mode:

Architecture Scenario Best For
Component Separation (this section) 3 dedicated hosts: MySQL + GUI + Sensor(s) Single data center, independent scaling needs, centralized database
Client-Server Mode Central server + remote sensors forwarding packets Multi-site deployments, geographic distribution, low-resource remote sensors

Both models can be combined: Use component separation at the central site, while remote sites use client-server mode to forward data.

Monitoring Component Health

Monitor each component separately:

  • MySQL Server: Check innodb_buffer_pool_size, query throughput, slow queries
  • Sensor(s): Monitor t0CPU, packet drops, SQL queue depth
  • GUI Server: Monitor web server load, PHP-FPM statistics, page response times

Use the Sniffer Troubleshooting guide for sensor issues and database performance tools for MySQL tuning.

Monitoring Live Performance

VoIPmonitor logs detailed performance metrics every 10 seconds to syslog.

# Debian/Ubuntu
tail -f /var/log/syslog | grep voipmonitor

# CentOS/RHEL
tail -f /var/log/messages | grep voipmonitor

Understanding the Log Output

Sample log line:

voipmonitor[15567]: calls[315][355] PS[C:4 S:29/29 R:6354 A:6484] SQLq[0] heap[0|0|0] comp[54] [12.6Mb/s] t0CPU[5.2%] ... RSS/VSZ[323|752]MB
Metric Description Warning Threshold
calls[X][Y] X = active calls, Y = total calls in memory -
SQLq[C] SQL queries waiting to be sent to database Growing consistently = DB bottleneck
B|C] Memory usage % for internal buffers A = 100% → packet drops
t0CPU[X%] Main packet capture thread CPU usage >90-95% = capture limit reached
Y]MB Resident/Virtual memory usage RSS growing = memory leak

Performance Diagrams

The following diagrams illustrate the difference between standard kernel packet capture and optimized solutions:

Standard kernel packet capture path - packets traverse multiple kernel layers before reaching VoIPmonitor
PF_RING/DPDK bypass mode - packets are delivered directly to VoIPmonitor, bypassing the kernel network stack

See Also

AI Summary for RAG

Summary: Expert guide to scaling VoIPmonitor for high-traffic environments. Covers three main bottlenecks: (1) Packet Capturing - optimized via TPACKET_V3, NIC tuning with ethtool (ring buffer, interrupt coalescing), configuration-level optimizations (interface_ip_filter more efficient than BPF filter, pcap_dump_writethreads for compression thread tuning, jitterbuffer settings for CPU/performance balance), and kernel-bypass solutions (DPDK, PF_RING, Napatech); (2) Disk I/O - VoIPmonitor uses TAR-based storage to reduce IOPS, with ext4 tuning and RAID WriteBack cache; (3) Database - critical innodb_buffer_pool_size tuning with formula for shared servers: (Total RAM - VoIPmonitor - OS overhead) / 2. For 32GB shared server, recommend 14GB buffer pool. Dedicated servers can use 50-70% of RAM. Covers slow query log as a memory/I/O consumer and disabling it for memory optimization. Covers partitioning benefits and syslog monitoring (t0CPU, SQLq, heap metrics). TROUBLESHOOTING: GUI or sniffer crashes with "MariaDB connection refused" errors indicate innodb_buffer_pool_size is set too low (e.g., default 128M or 256M). This causes MariaDB to run out of memory for the InnoDB buffer pool during high call volumes, resulting in connection failures even while the service remains active. Solution: Edit MariaDB config file (typically /etc/mysql/mariadb.conf.d/50-server.cnf or /etc/my.cnf.d/mariadb-server.cnf) and increase innodb_buffer_pool_size to 6G or higher based on available RAM. For most production deployments with 500-2000 concurrent calls, values between 6G and 14G are typical. Restart MariaDB with systemctl restart mariadb after changing the configuration. For extreme scenarios (4,000+ concurrent calls, UI lag, unresponsive GUI, high SQL queue), see High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for specialized configurations including innodb_flush_log_at_trx_commit=0, hourly partitioning, centralized writer architecture, RTP thread tuning (rtpthreads, rtpthreads_start), and MySQL optimization settings (innodb_thread_concurrency, innodb_io_capacity, innodb_flush_method=O_DIRECT).

COMPONENT SEPARATION ARCHITECTURE: For deployments requiring independent scaling, VoIPmonitor's three core components (MySQL database, Sensor, and GUI) can be deployed on separate dedicated hosts. This architecture is recommended when needing to scale packet capture without affecting GUI responsiveness, database queries are becoming a bottleneck, multiple teams manage different aspects (DBAs, network engineers, web administrators), traffic volume exceeds 5,000-10,000 concurrent calls, or different HA strategies are needed. Three-Host Architecture: Host 1 is MySQL Database (needs 50-70% RAM for buffer pool, fast SSD/NVMe I/O, scale by adding RAM or read replicas, consider Galera cluster for HA); Host 2 is Sensor(s) (needs fast CPU single core for t0 thread and network I/O, scale by adding more sensors for distributed capture or use DPDK/PF_RING); Host 3 is GUI web interface (needs CPU for database queries and network bandwidth, can be horizontally scaled with load balancer and caching, use multiple web servers with load balancer for HA). Configuration: MySQL server with bind-address=0.0.0.0 and remote user access via CREATE USER 'voipmonitor'@'%' with GRANT ALL PRIVILEGES. Sensor configured with mysqlhost pointing to MySQL server IP. GUI configured via Settings > System Configuration > Database or by editing config/system_configuration.php to set DB_HOST, DB_USER, DB_PASS, DB_NAME. Network connectivity requires firewall rules: Sensor→MySQL (port 3306), GUI→MySQL (port 3306), Users→GUI (ports 80/443), GUI→Sensor(s) (port 5029 for PCAP retrieval if using local processing). Advantages: independent scaling, resource optimization (database needs RAM, sensors need CPU, GUI needs network), isolation (issues on one component don't affect others), team specialization, different HA strategies per component. Comparison with Client-Server Mode: Component Separation (3 dedicated hosts MySQL+GUI+Sensor) is best for single data center and independent scaling needs while Client-Server Mode (central server+remote sensors forwarding packets) is best for multi-site and geographic distribution with low-resource remote sensors. Both models can be combined. Monitor components separately: MySQL (innodb_buffer_pool_size, query throughput, slow queries), Sensors (t0CPU, packet drops, SQL queue), GUI (web server load, PHP-FPM, response times).

Keywords: scaling, performance tuning, bottleneck, t0CPU, TPACKET_V3, DPDK, PF_RING, ethtool, ring buffer, interface_ip_filter, BPF filter, pcap_dump_writethreads, jitterbuffer, jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt, compression threads, PCAP async write, innodb_buffer_pool_size, OOM killer, shared server memory, database partitioning, SQLq monitoring, slow query log, slow_query_log, long_query_time, UI lag, unresponsive GUI, high performance, 4000 concurrent calls, 5000 concurrent calls, innodb_flush_log_at_trx_commit=0, hourly partitioning, rtpthreads, rtpthreads_start, RTP threads, innodb_io_capacity, innodb_thread_concurrency, innodb_flush_method, extreme performance, High-Performance Manual, component separation, three host architecture, dedicated MySQL server, dedicated GUI server, remote database, mysqlhost configuration, GUI remote database, independent scaling, high availability, Galera cluster, load balancer, DB_HOST, DB_USER, DB_PASS, DB_NAME, bind-address, firewall ports, GUI settings database configuration, connection refused, MariaDB connection refused, GUI crashes, database connection errors, mariadb.conf.d/50-server.cnf, MySQL crash, MariaDB crash, can't connect to MySQL

Key Questions:

  • How do I scale VoIPmonitor for thousands of concurrent calls?
  • What are the main performance bottlenecks in VoIPmonitor?
  • How do I fix high t0CPU usage?
  • What is DPDK and when should I use it?
  • How do I calculate innodb_buffer_pool_size for a shared server?
  • What happens if innodb_buffer_pool_size is set too high?
  • What causes "MariaDB connection refused" errors in the GUI or sniffer?
  • How do I fix GUI crashes due to MariaDB connection refused errors?
  • How do I troubleshoot database connection issues with MariaDB?
  • How do I interpret the performance metrics in syslog?
  • Should I use a dedicated database server for VoIPmonitor?
  • How much RAM does a VoIPmonitor server need?
  • How can the slow query log affect memory utilization?
  • How do I disable or adjust the MySQL slow query log?
  • Is interface_ip_filter more efficient than the filter option?
  • How do I optimize PCAP compression threads for high traffic?
  • Which jitterbuffer settings affect CPU load the most?
  • What configuration options reduce CPU overhead?
  • How do I deploy VoIPmonitor with MySQL, GUI, and Sensor on separate servers?
  • When should I use component separation architecture for VoIPmonitor?
  • What is the three host architecture for VoIPmonitor scaling?
  • How do I configure the GUI to connect to a remote MySQL database?
  • How do I configure sensors to write to a remote MySQL server?
  • What are the advantages of separating VoIPmonitor components onto different hosts?
  • How do I scale the MySQL database independently from sensors and GUI?
  • How do I scale the GUI independently from database and sensors?
  • What is the difference between component separation and client-server mode?
  • Can I combine component separation with client-server mode for VoIPmonitor?
  • How do I configure firewall rules for component separation architecture?
  • When should I use a dedicated GUI server for VoIPmonitor?