GUI troubleshooting: Difference between revisions

From VoIPmonitor.org
(Add MySQL max_connections connection limit troubleshooting section)
(Add spool directory ownership fix for GUI loading issues)
 
(20 intermediate revisions by the same user not shown)
Line 1: Line 1:
= High-Performance VoIPmonitor and MySQL Setup Manual =
= GUI Troubleshooting =


This manual explains the configuration for a high-call-per-second (CPS) VoIPmonitor deployment. The configuration is split into two parts: the MySQL database and the VoIPmonitor sniffer.
This guide covers common VoIPmonitor GUI issues organized by symptom category.


== Overall Concept ==
== Quick Navigation ==


This system is designed for maximum data ingestion performance based on a '''centralized writer architecture'''. It trades certain data safety and consistency guarantees for an extremely high write throughput, suitable for large-scale deployments. The architecture is composed of three distinct stages:
{| class="wikitable"
 
! Category !! Common Issues
# '''Centralized CDR Ingestion (Sniffer Role):''' The primary sniffer instance is configured to act as a central data collector. Its main purpose is to receive Call Detail Records (CDRs) from all other sniffer instances in the network. In this mode, the central sniffer may not process all network packets itself; its primary function is to prepare and queue CDRs for database insertion. It is critical that in this architecture, '''only the central sniffer writes to the database'''. All other sniffers must be configured to send their data to this central instance.
 
# '''Optimized Batch Writing (Sniffer to MySQL):''' To achieve maximum insert throughput, the central sniffer writes CDRs to MySQL in large, consolidated batches. A key optimization is that the database's automatic ID generators (<code>AUTO_INCREMENT</code>) are disabled. The sniffer pre-assigns unique IDs to the records, which allows for massive parallel inserts without the locking and serialization bottlenecks that normally occur at the database level.
 
# '''High-Speed Database Storage (MySQL):''' The MySQL database is heavily optimized to ingest these large data batches at extremely high speeds. For environments with exceptionally high CDR-per-second rates, this configuration supports '''hourly table partitioning'''. While hourly partitioning dramatically accelerates write performance by keeping the active table small, it introduces a trade-off for data retrieval. When performing a search across an entire day, the database engine must query 24 separate partitions instead of a single daily one. This can result in slower query performance for reports or searches covering broad time ranges.
 
== MySQL Configuration (my.cnf) ==
 
The goal of this configuration is to maximize the write performance of the InnoDB engine, often at the expense of strict ACID durability.
 
{| class="wikitable" style="background-color: #FFFFFF;"
! Directive
! Description
|-
|-
| <code>skip-log-bin</code>
| [[#Access Issues|Access Issues]] || HTTP 500, blank screen, single user cannot access
| Disables binary logging. This significantly increases write performance because the database doesn't have to record every data modification. The trade-off is the inability to perform point-in-time recovery or use replication.
|-
|-
| <code>innodb_flush_log_at_trx_commit=0</code>
| [[#No Data Issues|No Data Issues]] || No CDR for today, empty dashboard, missing calls
| The most critical setting for write performance. Instead of flushing the transaction log to disk after every single transaction (default: <code>1</code>), the database flushes it only once per second. This dramatically speeds up writes but can lead to the loss of the last second's worth of transactions if the server crashes. Usually innodb_flush_log_at_trx_commit = 2 is enough but for extreme CDR CPS this is crucial to set it to the innodb_flush_log_at_trx_commit=0
|-
|-
| <code>innodb_flush_log_at_timeout = 1800</code>
| [[#Database Issues|Database Issues]] || Timeouts, corruption, connections, schema errors
| Defines the timeout in seconds for the log flushing. However, it is effectively overridden by the more frequent one-second flush from the setting above.
|-
|-
| <code>max_heap_table_size = 24G</code>
| [[#GUI Upgrade Issues|GUI Upgrade Issues]] || Cannot login, blank screen, VM Binary error
| Allows for the creation of very large temporary tables in memory. This is important for complex analytical queries that the VoIPmonitor GUI might run.
|-
|-
| <code>table_open_cache=9999</code>
| [[#Permission Issues|Permission Issues]] || Ownership errors, NAS access, binary execution
| Keeps a large number of table definitions cached. Since VoIPmonitor uses hourly partitioning, it creates a vast number of tables, and this setting prevents the overhead of constantly opening and closing table files.
|-
|-
| <code>innodb_log_file_size = 5G</code>
| [[#IonCube and PHP Issues|IonCube/PHP Issues]] || Loader errors, temp directory, PHP version
| Defines a very large size for InnoDB's redo log files. Larger files reduce the frequency of I/O-intensive checkpoint operations, improving overall throughput.
|-
|-
| <code>innodb_log_buffer_size = 2G</code>
| [[#Other Issues|Other Issues]] || Timezone, fax, attachments, RRD graphs
| Allocates a large buffer in memory (2 GB) for storing transaction logs before they are flushed to disk. This allows the system to handle large bursts of write activity without immediate disk I/O.
|}
 
== Access Issues ==
 
=== HTTP 500 Error ===
 
'''Client-side steps (try first):'''
# Different browser or incognito mode
# Clear browser cache (Ctrl+Shift+Delete)
# Disable browser extensions
# Try from a different network
 
'''Server-side diagnosis:'''
 
<syntaxhighlight lang="bash">
# Check Apache error log
tail -100 /var/log/apache2/error.log    # Debian/Ubuntu
tail -100 /var/log/httpd/error_log      # RHEL/CentOS
 
# Check PHP-FPM log
tail -100 /var/log/php-fpm/www-error.log
 
# Check SELinux
getenforce
# If "Enforcing", try temporarily: setenforce 0
</syntaxhighlight>
 
{{Tip|For detailed SQL query debugging, add <code>?debug=31415</code> to any GUI URL and check browser console (F12).}}
 
 
 
=== Firefox/Waterfox: Blank Page or Hang on Login ===
 
'''Firefox/Waterfox: Blank Page or Hang on Login'''
 
If the Web GUI shows a blank page or hangs during login, but works correctly in Chrome/Edge, the cause is typically a browser extension conflict.
 
'''Diagnosis:'''
* Issue occurs in Firefox/Waterfox but not Chrome/Edge
* Works when Developer Tools (F12) is open
* Persists in private mode and through SSH tunnels
 
'''Solution:'''
# In Firefox, navigate to '''about:addons'''
# Select Extensions tab and '''disable all extensions'''
# Close and reopen Firefox completely
# Attempt to log in to the GUI
# If successful, one extension is the cause. Re-enable extensions one by one (or in pairs) restarting Firefox and testing after each change to identify the specific culprit
=== Single User Cannot Access ===
 
If one user cannot access GUI while others can:
 
# '''Check IP whitelist''': User Management > edit user > "Access from IPs" field
# '''Check network path''': Compare user's route vs working users
# '''Capture traffic''': <syntaxhighlight lang="bash" inline>tcpdump -i eth0 host USER_IP -w /tmp/debug.pcap</syntaxhighlight>
 
=== Blank Screen After Upgrade ===
 
<syntaxhighlight lang="bash">
# Check for conflicting PEAR file
ls -la /usr/share/pear/constants.php
# If exists, remove it:
rm /usr/share/pear/constants.php
 
# Check SELinux
getenforce
# If Enforcing, temporarily disable: setenforce 0
 
# Clear browser cache
# Press Ctrl+F5 in browser
</syntaxhighlight>
 
== No Data Issues ==
 
=== No CDR or Dashboard Data for Current Day ===
 
'''Step 1: Check sensor status'''
 
Navigate to '''Settings > Sensors''' in GUI:
* '''Status UP''' - Sensor connected, proceed to Step 2
* '''Status Down''' - Sensor disconnected, see [[#Sensor Down|Sensor Down]] below
 
'''Step 2: Generate debug log (if sensor UP)'''
 
Navigate to '''Tools > Generate Debug Log''' and share the link with VoIPmonitor support.
 
'''Step 3: Check common causes'''
 
{| class="wikitable"
! Cause !! Diagnosis !! Solution
|-
|-
| <code>open_files_limit = 200000</code>
| Missing partition || <code>SHOW CREATE TABLE cdr;</code> - check partition for today's date || <code>ALTER TABLE cdr ADD PARTITION (PARTITION pYYYYMMDD VALUES LESS THAN (UNIX_TIMESTAMP('YYYY-MM-DD')));</code>
| Increases the file handle limit for the MySQL process, which is necessary to support the high <code>table_open_cache</code> value.
|-
|-
| <code>key_buffer_size = 2G</code>
| Timezone mismatch || GUI shows different time than DB || Settings > System Configuration > National > Timezone
| This setting is for the MyISAM storage engine. While most of VoIPmonitor uses InnoDB, some system or temporary tables might still use MyISAM. A 2GB buffer is allocated for its indexes.
|-
|-
| <code>sort_buffer_size = 65M</code>
| CDR disabled || Check voipmonitor.conf || Ensure <code>cdr = yes</code>
| A large sort buffer for handling sorting operations (like ORDER BY or GROUP BY) for complex queries.
|-
|-
| <code>max_connections = 100000</code>
| Spool directory || Check <code>spooldir</code> path || Verify path exists and has correct permissions
| An extremely high limit for connections. It acts as a safeguard against "too many connections" errors, though the real number of active connections will be much lower.
|-
| <code>skip-name-resolve</code>
| Prevents MySQL from resolving hostnames for incoming connections. This speeds up the connection process by bypassing DNS lookups.
|-
| <code>innodb_read_io_threads = 40</code>
| Sets a high number of threads for handling read I/O operations, suitable for systems with fast multi-channel storage (like NVMe arrays).
|-
| <code>innodb_write_io_threads = 40</code>
| Sets a high number of threads for handling write I/O operations, matching the read threads for balanced I/O.
|-
| <code>innodb_purge_threads = 20</code>
| A high number of threads dedicated to purging old, unneeded data versions, which is crucial in a high-volume write environment to prevent performance degradation.
|-
| <code>innodb_flush_neighbors = 0</code>
| An optimization for SSD/NVMe drives. It disables a feature useful for rotational HDDs where adjacent blocks on disk are flushed at the same time. This is counterproductive for SSDs.
|-
| <code>innodb_io_capacity = 1000000</code>
| Informs InnoDB that it's running on a very high-performance storage system capable of 1,000,000 IOPS. This allows InnoDB to be much more aggressive with background I/O tasks.
|-
| <code>innodb_io_capacity_max = 10000000</code>
| Sets the absolute maximum IOPS the system can burst to, allowing InnoDB to use the full potential of the storage during intensive operations.
|-
| <code>innodb_doublewrite = 0</code>
| Disables the doublewrite buffer. This provides a significant performance boost but increases the risk of data corruption (a "torn page") if a power failure occurs during a write operation. It's a performance-vs-safety trade-off.
|-
| <code>innodb_buffer_pool_size = 150G</code>
| The most important parameter for InnoDB performance. A massive buffer pool size of 150 GB keeps a large portion of the database (data and indexes) in RAM, minimizing slow disk access.
|-
| <code>innodb_flush_method = O_DIRECT</code>
| Bypasses the operating system's file cache. This prevents double-caching data (once in the OS, once in InnoDB) and gives the database direct control over I/O operations.
|-
| <code>transaction-isolation = READ-UNCOMMITTED</code>
| The lowest level of transaction isolation. It allows transactions to read data that has not yet been committed. This offers maximum performance by minimizing locking.
|-
| <code>performance_schema=0</code>
| Disables the Performance Schema, a detailed performance monitoring tool. Disabling it slightly reduces overhead and can increase performance.
|}
|}


== VoIPmonitor Sniffer Configuration (voipmonitor.conf) ==
{{Note|If GUI shows yesterday's data but not today's, the most common cause is a missing database partition.}}
 
==== Sensor Down ====
 
<syntaxhighlight lang="bash">
# Check sensor service
systemctl status voipmonitor
 
# View recent logs
journalctl -u voipmonitor -n 500
 
# Check database connectivity
mysql -h DB_HOST -u voipmonitor -p -e "SELECT 1"
 
# Restart if needed
systemctl restart voipmonitor
</syntaxhighlight>
 
=== No Audio Playback ===
 
If QoS data is visible but no audio:
 
<syntaxhighlight lang="bash">
# Check voipmonitor.conf
grep -E "^savertp|^savegraph" /etc/voipmonitor.conf
 
# Should show:
# savertp = yes
# savegraph = yes (optional)
</syntaxhighlight>
 
See [[Sniffer_configuration#RTP_and_Audio_Recording|RTP and Audio Recording]] for detailed configuration.
 
== Database Issues ==
 
=== GUI Queries Timeout (40-50 seconds) ===
 
'''MySQL/MariaDB tuning:'''
 
<syntaxhighlight lang="ini">
# /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf.d/server.cnf
 
[mysqld]
# Set to 50-70% of available RAM
innodb_buffer_pool_size = 4G
 
# Reduce disk I/O (slight durability tradeoff)
innodb_flush_log_at_trx_commit = 2
 
# Increase connection timeout
wait_timeout = 28800
interactive_timeout = 28800
</syntaxhighlight>
 
'''Apache mod_fcgid timeout:'''
 
If you see "server side error - check your http server error log. - connection to the web server was lost / interrupted":
 
<syntaxhighlight lang="apache">
# In httpd.conf or apache2.conf
<IfModule mod_fcgid.c>
    FcgidIdleTimeout 900
    FcgidProcessLifeTime 7200
    FcgidConnectTimeout 300
    FcgidIOTimeout 900
</IfModule>
</syntaxhighlight>
 
<syntaxhighlight lang="bash">
systemctl restart apache2  # or httpd
</syntaxhighlight>
 
=== Too Many Connections ===
 
<syntaxhighlight lang="sql">
-- Check current connections
SHOW STATUS LIKE 'Threads_connected';
SHOW VARIABLES LIKE 'max_connections';
 
-- Increase limit (temporary)
SET GLOBAL max_connections = 500;
 
-- Permanent: add to my.cnf
-- max_connections = 500
</syntaxhighlight>
 
=== MySQL Swap Memory Warning ===
 
If you see swap memory warning on GUI login:
 
* '''Less than 50 MB swap''' - Safe to ignore, minor performance impact
* '''Hundreds of MB or GB''' - Investigate memory usage, consider increasing RAM or reducing innodb_buffer_pool_size
 
<syntaxhighlight lang="bash">
# Check MySQL memory usage
ps aux | grep mysqld
free -h
</syntaxhighlight>
 
=== Unknown Column in Field List ===
 
If GUI shows "Unknown column [table].[column] in 'field list'":
 
<syntaxhighlight lang="bash">
# Method 1: URL parameter (fastest)
# Add ?check_tables=1 to GUI URL:
# http://your-gui-ip/admin.php?check_tables=1
 
# Method 2: Command line
cd /var/www/html
php php/run.php dbcheck
 
# Method 3: Restart sensor (if disable_dbupgradecheck=no)
systemctl restart voipmonitor
</syntaxhighlight>
 
=== Database Corruption ===
 
'''Recovery methods (try in order):'''
 
'''Method 1: Table repair'''
<syntaxhighlight lang="bash">
mysqlcheck -u root -p --auto-repair voipmonitor
</syntaxhighlight>
 
'''Method 2: Drop and recreate specific table'''
<syntaxhighlight lang="sql">
-- Backup structure first
SHOW CREATE TABLE problematic_table;
 
-- Drop and let sensor recreate
DROP TABLE problematic_table;
-- Restart sensor: systemctl restart voipmonitor
</syntaxhighlight>
 
'''Method 3: InnoDB force recovery'''
<syntaxhighlight lang="ini">
# Add to my.cnf [mysqld] section:
innodb_force_recovery = 1
# Restart MySQL, export data, remove option, reimport
</syntaxhighlight>
 
{{Warning|After using innodb_force_recovery, remove the option before resuming normal operations. The database is read-only while this option is set.}}
 
'''Method 4: PCAP restore'''
 
If PCAPs are preserved, drop database and let sensor reindex:
<syntaxhighlight lang="bash">
# Edit voipmonitor.conf
reindex_all = yes  # (temporary)
 
systemctl restart voipmonitor
# Wait for reindexing to complete, then remove reindex_all
</syntaxhighlight>
 
See [[Database_troubleshooting]] for detailed recovery procedures.
 
=== MySQL 8.0 Issues ===
 
'''SYSTEM_USER privilege error:'''
<syntaxhighlight lang="sql">
GRANT SYSTEM_USER ON *.* TO 'voipmonitor'@'localhost';
GRANT ALL PRIVILEGES ON *.* TO 'voipmonitor'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;
 
-- Also set:
SET GLOBAL log_bin_trust_function_creators = 1;
</syntaxhighlight>
 
'''MySQL 8.0.42 crashes:'''
 
{{Warning|MySQL 8.0.42 has known bugs causing VoIPmonitor crashes. Upgrade to 8.0.43 or later.}}
 
<syntaxhighlight lang="bash">
mysql --version
# If 8.0.42:
sudo apt install --only-upgrade mysql-server-8.0  # Debian/Ubuntu
sudo dnf update mysql-server                      # RHEL
</syntaxhighlight>
 
=== Managed Database: Primary Key Error ===
 
For DigitalOcean, AWS RDS, Google Cloud SQL:
 
<syntaxhighlight lang="sql">
-- Disable in managed DB console or parameter group:
sql_require_primary_key = OFF
</syntaxhighlight>
 
== GUI Upgrade Issues ==
 
=== Cannot Login After Upgrade ===
 
'''Reset password via database:'''
<syntaxhighlight lang="sql">
UPDATE users SET password = MD5('newpassword') WHERE username = 'admin';
</syntaxhighlight>
 
'''Redis session issue (login loop):'''
<syntaxhighlight lang="bash">
redis-cli FLUSHALL
systemctl restart php-fpm
</syntaxhighlight>
 
=== VM Binary Error ===
 
<syntaxhighlight lang="bash">
cd /var/www/html
php php/run.php upgrade -f
</syntaxhighlight>
 
=== Invalid Compressed Data ===
 
Usually caused by incorrect web root ownership:
<syntaxhighlight lang="bash">
chown -R www-data:www-data /var/www/html  # Debian/Ubuntu
chown -R apache:apache /var/www/html      # RHEL/CentOS
</syntaxhighlight>
 
=== Unable to Remove Old Files ===
 
<syntaxhighlight lang="bash">
# Fix tshark ownership
chown www-data:www-data /var/www/html/bin/tshark*
 
# Retry upgrade from GUI
</syntaxhighlight>
 
=== Upgrade Not Visible ===
 
Check internet connectivity from GUI server:
<syntaxhighlight lang="bash">
curl -I https://www.voipmonitor.org
ping www.voipmonitor.org
</syntaxhighlight>


This configuration is tuned to enable the sniffer to process packets as quickly as possible, queue data efficiently, and avoid becoming a bottleneck.
== Permission Issues ==


{| class="wikitable" style="background-color: #FFFFFF;"
=== Incorrect Ownership Error ===
! Directive
 
! Description
Often caused by missing execute permissions on binaries:
<syntaxhighlight lang="bash">
# Fix bin directory
chmod +x /var/www/html/bin/*
 
# Verify
ls -la /var/www/html/bin/
</syntaxhighlight>
 
=== NAS Spool Permission Denied ===
 
<syntaxhighlight lang="bash">
# Use ACLs (recommended)
setfacl -R -m u:www-data:rwx /path/to/nas/spool
 
# CRITICAL: Set on ALL parent directories too
setfacl -m u:www-data:rx /NAS
setfacl -m u:www-data:rx /NAS/voipmonitor
 
# Verify
sudo -u www-data ls /path/to/nas/spool
</syntaxhighlight>
 
=== SELinux Blocking mscgen (SIP Call Flow Diagrams) ===
 
If SIP call flow diagram shows "convert msc to svg (mscgen): unknown error":
 
<syntaxhighlight lang="bash">
# Check SELinux denials
ausearch -m avc -ts recent | grep mscgen
 
# Create policy module
grep mscgen /var/log/audit/audit.log | audit2allow -M mscgen_local
semodule -i mscgen_local.pp
</syntaxhighlight>
 
== IonCube and PHP Issues ==
 
=== Temp Directory Not Writable ===
 
<syntaxhighlight lang="bash">
# Check and fix /tmp permissions
ls -ld /tmp
chmod 1777 /tmp
 
# Check SELinux
getenforce
# If Enforcing, try: setenforce 0
 
# Check systemd PrivateTmp
systemctl show httpd | grep PrivateTmp
# If true, create override:
# /etc/systemd/system/httpd.service.d/privatetmp.conf
# [Service]
# PrivateTmp=false
</syntaxhighlight>
 
=== PHP CLI vs Web Server Mismatch ===
 
<syntaxhighlight lang="bash">
# Test as web server user
sudo -u www-data php -r 'echo extension_loaded("ionCube Loader")?"yes":"no";'
 
# Compare configs
php --ini                    # CLI config
# Create phpinfo: echo '<?php phpinfo();' > /var/www/html/info.php
# Check "Loaded Configuration File" in browser
# Remove after: rm /var/www/html/info.php
</syntaxhighlight>
 
=== License Activation: Server Side Error - Forbidden ===
 
This indicates ionCube PHP version incompatibility:
 
{| class="wikitable"
! GUI Version !! PHP Version
|-
|-
| <code>mysqlstore_concat_limit = 10000</code>
| v26.37+ || PHP 8.3 supported
| Batches up to 10,000 CDRs into a single <code>INSERT</code> statement, dramatically reducing database overhead.
|-
|-
| <code>mysqlstore_max_threads_cdr = 40</code>
| Older || PHP 8.2 or earlier required
| Sets 40 threads for writing CDRs to the database, essential for high-volume parallel processing.
|-
| <code>mysql_enable_new_store = per_query</code>
| Activates an alternative internal queueing mechanism for SQL queries, which can improve parallelization.
|-
| <code>mysql_enable_set_id = yes</code>
| The sniffer generates the primary key (ID) for records itself, rather than relying on <code>AUTO_INCREMENT</code> in the database. This reduces database locking.
|-
| <code>server_sql_queue_limit = 1000000</code>
| Sets a massive in-memory queue (1,000,000) for SQL commands to buffer against database slowdowns.
|-
| <code>partition_operations_enable_fromto = 4-6</code>
| Schedules database partition management to run only between 4:00 and 6:00 AM, avoiding impact during peak hours.
|-
| <code>disable_partition_operations = yes</code>
| Disables partition management operations (creating new partitions, dropping old ones) on this sniffer instance. Use this on all sniffer instances '''except one''' when running multiple sniffers that write to the same database. Only a single designated sniffer should manage the database partition structure to avoid metadata table locks and database unresponsiveness.
|-
| <code>cleandatabase = N</code>
| Reduces the data retention period for cleaning the database to N days.
|-
| <code>mysql_redirect_cdr_queue = yes</code>
| Enables a separate, dedicated queue for CDRs being redirected from other sniffers.
|-
| <code>partition_operations_drop_first = yes</code>
| Drops the oldest partition first before creating a new one during cleanup, which can be faster.
|-
| <code>cdr_partition_by_hours = yes</code>
| Partitions CDR tables hourly instead of daily. This keeps the active write table extremely small, which is critical for high performance.
|-
| <code>disable_cdr_indexes_rtp = yes</code>
| Disables some indexes on the <code>cdr_rtp</code> table to speed up writes, at the cost of slower searches on those fields.
|-
| <code>sql_errors_skip = 1136,1054</code>
| Instructs the sniffer to ignore specific, non-critical database errors to prevent it from stopping during database maintenance.
|}
|}
<syntaxhighlight lang="bash">
# Check Apache error log
tail -50 /var/log/apache2/error.log | grep -i ioncube
# Solution: Either upgrade GUI to v26.37+ or downgrade PHP to 8.2
</syntaxhighlight>
== Other Issues ==
=== Invalid Timezone (GUI Inaccessible) ===
<syntaxhighlight lang="bash">
# Edit directly
nano /var/www/html/config/configuration.php
# Find and fix:
define('TIMEZONE', 'UTC');  # Use valid timezone
define('SENSORS_TIMEZONE', 'UTC');
systemctl restart php-fpm
</syntaxhighlight>
Valid timezones: <code>UTC</code>, <code>Europe/London</code>, <code>America/New_York</code>, <code>Europe/Berlin</code>, etc.
=== Faxes Not Displayed ===
<syntaxhighlight lang="bash">
# Install libtiff-tools
apt-get install libtiff-tools    # Debian/Ubuntu
yum install libtiff-tools        # RHEL/CentOS
# Verify
which tiff2pdf
</syntaxhighlight>
=== File Attachment Blocked ===
Files like .xlsx, .docx are blocked for security. Compress to .zip or .gz:
<syntaxhighlight lang="bash">
zip archive.zip your_file.xlsx
</syntaxhighlight>
=== RRD Graph Errors After Hardware Change ===
<syntaxhighlight lang="bash">
# Delete corrupted RRD files
rm -rf /var/spool/voipmonitor/rrd/*
# Sensor will recreate them
systemctl restart voipmonitor
</syntaxhighlight>
=== /dev/shm Full (Session Logout Issues) ===
<syntaxhighlight lang="bash">
# Check usage
df -h /dev/shm
# Temporary increase
mount -o remount,size=1G /dev/shm
# Permanent: edit /etc/fstab
# tmpfs /dev/shm tmpfs defaults,size=1G 0 0
</syntaxhighlight>
=== Concurrent Calls Stats Error ===
Usually caused by duplicate cron entries:
<syntaxhighlight lang="bash">
# Check for duplicates
grep "voipmonitor\|run.php" /etc/crontab
# Comment out duplicates, keep only one
</syntaxhighlight>
=== TShark CVE Security Updates ===
The bundled tshark may have known CVEs. To update:
<syntaxhighlight lang="bash">
cd /var/www/html/bin
# Download updated binary from Wireshark
wget https://www.wireshark.org/download/automated/linux64/tshark
chmod +x tshark
chown www-data:www-data tshark
</syntaxhighlight>
=== Dashboard Edit Icon Grayed Out ===
User lacks edit permission. Grant via '''User Management''' > edit user > enable "Edit Dashboard" permission.
== GUI Not Loading (Spool Directory Ownership) ==
If the GUI is not loading or shows generic error messages, the cause may be incorrect ownership of the spool directory by the web server user.
<syntaxhighlight lang="bash">
# Fix spool directory ownership
chown -R www-data:www-data /var/spool/voipmonitor    # Debian/Ubuntu
# OR
chown -R apache:apache /var/spool/voipmonitor        # RHEL/CentOS
</syntaxhighlight>
{{Note|The spool directory path is defined by the <code>spooldir</code> parameter in <code>/etc/voipmonitor.conf</code>. Use the correct path for your configuration.}}


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' This document provides an expert-level guide to configuring a VoIPmonitor deployment for extreme high-call-per-second (CPS) rates (10,000+ concurrent calls). It describes a "centralized writer architecture" with three stages: 1) A central sniffer that collects CDRs from all other sensors. 2) The central sniffer performs optimized batch writing to the database, notably by pre-assigning primary keys instead of using AUTO_INCREMENT (`mysql_enable_set_id=yes`). 3) The MySQL/MariaDB database is heavily tuned for write ingestion. This configuration is critical for resolving UI lag, unresponsive GUI, and database queue (SQLq) issues under high load. The guide provides and annotates two complete configuration examples. The `my.cnf` configuration focuses on maximizing write performance by disabling binary logging, relaxing transaction durability (`innodb_flush_log_at_trx_commit=0`), disabling the doublewrite buffer, and allocating massive memory buffers (`innodb_buffer_pool_size=150G`). Key MySQL optimization parameters include `innodb_io_capacity=1000000`, `innodb_thread_concurrency=40`, `innodb_flush_method=O_DIRECT`, `innodb_read_io_threads=20`, `innodb_write_io_threads=20`, `innodb_purge_threads=20`, and LZ4 compression. The `voipmonitor.conf` for the central sniffer is tuned for large-scale batching (`mysqlstore_concat_limit=10000`), high thread counts (`mysqlstore_max_threads_cdr=40`), and hourly partitioning (`cdr_partition_by_hours=yes`) to keep active tables small. This setup can handle 3000+ CDRs/second, making it suitable for deployments with 4000-5000+ concurrent calls experiencing performance issues.
 
'''Keywords:''' high performance, high cps, calls per second, scaling, tuning, database, mysql, mariadb, my.cnf, voipmonitor.conf, centralized writer, batch writing, `mysql_enable_set_id`, `innodb_flush_log_at_trx_commit`, `innodb_buffer_pool_size`, hourly partitioning, `cdr_partition_by_hours`, `mysqlstore_concat_limit`, UI lag, unresponsive GUI, SQL queue, SQLq growing, database bottleneck, extreme performance, 4000 concurrent calls, 5000 concurrent calls, `innodb_io_capacity`, `innodb_thread_concurrency`, `innodb_flush_method=O_DIRECT`, `innodb_read_io_threads`, `innodb_write_io_threads`, `innodb_purge_threads`, LZ4 compression, `innodb_compression_algorithm=lz4`, `skip-external-locking`, `skip-name-resolve`, `performance_schema=0`, `innodb_flush_neighbors=0`, `innodb_doublewrite=0`, `open_files_limit=200000`, `innodb_log_file_size=5G`, `innodb_log_buffer_size=2G`, `max_heap_table_size=24G`, `sort_buffer_size=65M`
'''Summary:''' VoIPmonitor GUI troubleshooting guide covering: (1) HTTP 500 errors - try client-side steps first (different browser, incognito, clear cache), then check Apache/PHP error logs and SELinux; (2) No CDR for current day - check sensor status in Settings > Sensors, generate debug log if UP, check partitions/timezone/cdr config if Down; (3) GUI debug mode - add ?debug=31415 to URL; (4) Database timeouts - tune innodb_buffer_pool_size (50-70% RAM), increase mod_fcgid timeouts; (5) Database corruption - use mysqlcheck, drop/recreate tables, or innodb_force_recovery; (6) GUI upgrade issues - reset password via database, fix ownership, run php php/run.php upgrade -f; (7) Permission errors - chmod +x bin/*, use setfacl for NAS; (8) IonCube errors - check SELinux, PHP version compatibility (GUI v26.37+ for PHP 8.3); (9) Schema errors - use ?check_tables=1 URL parameter.
 
'''Keywords:''' HTTP 500, no CDR, sensor status, debug mode, ?debug=31415, innodb_buffer_pool_size, MySQL tuning, database corruption, mysqlcheck, innodb_force_recovery, GUI upgrade, password reset, IonCube, SELinux, PHP 8.3, ?check_tables=1, NAS permissions, setfacl, mod_fcgid timeout, FcgidIOTimeout, mscgen, SIP call flow, tshark CVE, MySQL 8.0.42, SYSTEM_USER privilege, /dev/shm, timezone error, libtiff-tools, file attachment blocked
 
'''Key Questions:'''
'''Key Questions:'''
* How do I configure VoIPmonitor for very high call volumes (4000-5000+ concurrent calls)?
* HTTP 500 error on GUI - what to try first?
* How do I fix unresponsive UI or GUI lag under high load?
* No CDR or dashboard data for current day - how to diagnose?
* What causes SQL queue (SQLq) growth and how do I resolve it?
* How to enable GUI debug mode?
* What is the centralized writer architecture?
* GUI queries timeout - how to tune MySQL?
* What are the recommended MySQL settings for high CDR ingestion rates?
* Database corrupted - which recovery method to use?
* How does `mysql_enable_set_id=yes` improve performance?
* Cannot login after GUI upgrade - how to reset password?
* What are the benefits and drawbacks of hourly partitioning?
* Permission errors with NAS spool - how to fix?
* Why set `innodb_flush_log_at_trx_commit` to 0?
* IonCube temp directory errors - what causes them?
* How can I configure the sniffer to act as a central data collector?
* GUI shows "Unknown column in field list" - how to fix?
* What are the key `voipmonitor.conf` parameters for a high-performance setup?
* SIP call flow diagram shows error - how to fix SELinux blocking?
* What MySQL parameters help with extreme CDR insertion rates (1000+ CDRs/sec)?
* MySQL 8.0.42 causing crashes - what to do?
* How do I tune `innodb_io_capacity` and `innodb_thread_concurrency` for high performance?
* License activation fails with Forbidden error - what causes it?
* How does LZ4 compression improve MySQL performance on large datasets?

Latest revision as of 01:37, 10 January 2026

GUI Troubleshooting

This guide covers common VoIPmonitor GUI issues organized by symptom category.

Quick Navigation

Category Common Issues
Access Issues HTTP 500, blank screen, single user cannot access
No Data Issues No CDR for today, empty dashboard, missing calls
Database Issues Timeouts, corruption, connections, schema errors
GUI Upgrade Issues Cannot login, blank screen, VM Binary error
Permission Issues Ownership errors, NAS access, binary execution
IonCube/PHP Issues Loader errors, temp directory, PHP version
Other Issues Timezone, fax, attachments, RRD graphs

Access Issues

HTTP 500 Error

Client-side steps (try first):

  1. Different browser or incognito mode
  2. Clear browser cache (Ctrl+Shift+Delete)
  3. Disable browser extensions
  4. Try from a different network

Server-side diagnosis: 

# Check Apache error log
tail -100 /var/log/apache2/error.log    # Debian/Ubuntu
tail -100 /var/log/httpd/error_log      # RHEL/CentOS

# Check PHP-FPM log
tail -100 /var/log/php-fpm/www-error.log

# Check SELinux
getenforce
# If "Enforcing", try temporarily: setenforce 0

💡 Tip:


Firefox/Waterfox: Blank Page or Hang on Login

Firefox/Waterfox: Blank Page or Hang on Login

If the Web GUI shows a blank page or hangs during login, but works correctly in Chrome/Edge, the cause is typically a browser extension conflict.

Diagnosis:

  • Issue occurs in Firefox/Waterfox but not Chrome/Edge
  • Works when Developer Tools (F12) is open
  • Persists in private mode and through SSH tunnels

Solution:

  1. In Firefox, navigate to about:addons
  2. Select Extensions tab and disable all extensions
  3. Close and reopen Firefox completely
  4. Attempt to log in to the GUI
  5. If successful, one extension is the cause. Re-enable extensions one by one (or in pairs) restarting Firefox and testing after each change to identify the specific culprit

Single User Cannot Access

If one user cannot access GUI while others can:

  1. Check IP whitelist: User Management > edit user > "Access from IPs" field
  2. Check network path: Compare user's route vs working users
  3. Capture traffic: tcpdump -i eth0 host USER_IP -w /tmp/debug.pcap

Blank Screen After Upgrade

# Check for conflicting PEAR file
ls -la /usr/share/pear/constants.php
# If exists, remove it:
rm /usr/share/pear/constants.php

# Check SELinux
getenforce
# If Enforcing, temporarily disable: setenforce 0

# Clear browser cache
# Press Ctrl+F5 in browser

No Data Issues

No CDR or Dashboard Data for Current Day

Step 1: Check sensor status

Navigate to Settings > Sensors in GUI:

  • Status UP - Sensor connected, proceed to Step 2
  • Status Down - Sensor disconnected, see Sensor Down below

Step 2: Generate debug log (if sensor UP)

Navigate to Tools > Generate Debug Log and share the link with VoIPmonitor support.

Step 3: Check common causes

Cause Diagnosis Solution
Missing partition SHOW CREATE TABLE cdr; - check partition for today's date ALTER TABLE cdr ADD PARTITION (PARTITION pYYYYMMDD VALUES LESS THAN (UNIX_TIMESTAMP('YYYY-MM-DD')));
Timezone mismatch GUI shows different time than DB Settings > System Configuration > National > Timezone
CDR disabled Check voipmonitor.conf Ensure cdr = yes
Spool directory Check spooldir path Verify path exists and has correct permissions

ℹ️ Note: If GUI shows yesterday's data but not today's, the most common cause is a missing database partition.

Sensor Down

# Check sensor service
systemctl status voipmonitor

# View recent logs
journalctl -u voipmonitor -n 500

# Check database connectivity
mysql -h DB_HOST -u voipmonitor -p -e "SELECT 1"

# Restart if needed
systemctl restart voipmonitor

No Audio Playback

If QoS data is visible but no audio: 

# Check voipmonitor.conf
grep -E "^savertp|^savegraph" /etc/voipmonitor.conf

# Should show:
# savertp = yes
# savegraph = yes (optional)

See RTP and Audio Recording for detailed configuration.

Database Issues

GUI Queries Timeout (40-50 seconds)

MySQL/MariaDB tuning: 

# /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf.d/server.cnf

[mysqld]
# Set to 50-70% of available RAM
innodb_buffer_pool_size = 4G

# Reduce disk I/O (slight durability tradeoff)
innodb_flush_log_at_trx_commit = 2

# Increase connection timeout
wait_timeout = 28800
interactive_timeout = 28800

Apache mod_fcgid timeout:

If you see "server side error - check your http server error log. - connection to the web server was lost / interrupted": 

# In httpd.conf or apache2.conf
<IfModule mod_fcgid.c>
    FcgidIdleTimeout 900
    FcgidProcessLifeTime 7200
    FcgidConnectTimeout 300
    FcgidIOTimeout 900
</IfModule>

systemctl restart apache2  # or httpd

Too Many Connections

-- Check current connections
SHOW STATUS LIKE 'Threads_connected';
SHOW VARIABLES LIKE 'max_connections';

-- Increase limit (temporary)
SET GLOBAL max_connections = 500;

-- Permanent: add to my.cnf
-- max_connections = 500

MySQL Swap Memory Warning

If you see swap memory warning on GUI login:

  • Less than 50 MB swap - Safe to ignore, minor performance impact
  • Hundreds of MB or GB - Investigate memory usage, consider increasing RAM or reducing innodb_buffer_pool_size
# Check MySQL memory usage
ps aux | grep mysqld
free -h

Unknown Column in Field List

If GUI shows "Unknown column [table].[column] in 'field list'": 

# Method 1: URL parameter (fastest)
# Add ?check_tables=1 to GUI URL:
# http://your-gui-ip/admin.php?check_tables=1

# Method 2: Command line
cd /var/www/html
php php/run.php dbcheck

# Method 3: Restart sensor (if disable_dbupgradecheck=no)
systemctl restart voipmonitor

Database Corruption

Recovery methods (try in order):

Method 1: Table repair 

mysqlcheck -u root -p --auto-repair voipmonitor

Method 2: Drop and recreate specific table 

-- Backup structure first
SHOW CREATE TABLE problematic_table;

-- Drop and let sensor recreate
DROP TABLE problematic_table;
-- Restart sensor: systemctl restart voipmonitor

Method 3: InnoDB force recovery 

# Add to my.cnf [mysqld] section:
innodb_force_recovery = 1
# Restart MySQL, export data, remove option, reimport

⚠️ Warning: After using innodb_force_recovery, remove the option before resuming normal operations. The database is read-only while this option is set.

Method 4: PCAP restore

If PCAPs are preserved, drop database and let sensor reindex: 

# Edit voipmonitor.conf
reindex_all = yes  # (temporary)

systemctl restart voipmonitor
# Wait for reindexing to complete, then remove reindex_all

See Database_troubleshooting for detailed recovery procedures.

MySQL 8.0 Issues

SYSTEM_USER privilege error: 

GRANT SYSTEM_USER ON *.* TO 'voipmonitor'@'localhost';
GRANT ALL PRIVILEGES ON *.* TO 'voipmonitor'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;

-- Also set:
SET GLOBAL log_bin_trust_function_creators = 1;

MySQL 8.0.42 crashes:

⚠️ Warning: MySQL 8.0.42 has known bugs causing VoIPmonitor crashes. Upgrade to 8.0.43 or later. 

mysql --version
# If 8.0.42:
sudo apt install --only-upgrade mysql-server-8.0  # Debian/Ubuntu
sudo dnf update mysql-server                       # RHEL

Managed Database: Primary Key Error

For DigitalOcean, AWS RDS, Google Cloud SQL: 

-- Disable in managed DB console or parameter group:
sql_require_primary_key = OFF

GUI Upgrade Issues

Cannot Login After Upgrade

Reset password via database: 

UPDATE users SET password = MD5('newpassword') WHERE username = 'admin';

Redis session issue (login loop): 

redis-cli FLUSHALL
systemctl restart php-fpm

VM Binary Error

cd /var/www/html
php php/run.php upgrade -f

Invalid Compressed Data

Usually caused by incorrect web root ownership: 

chown -R www-data:www-data /var/www/html  # Debian/Ubuntu
chown -R apache:apache /var/www/html       # RHEL/CentOS

Unable to Remove Old Files

# Fix tshark ownership
chown www-data:www-data /var/www/html/bin/tshark*

# Retry upgrade from GUI

Upgrade Not Visible

Check internet connectivity from GUI server: 

curl -I https://www.voipmonitor.org
ping www.voipmonitor.org

Permission Issues

Incorrect Ownership Error

Often caused by missing execute permissions on binaries: 

# Fix bin directory
chmod +x /var/www/html/bin/*

# Verify
ls -la /var/www/html/bin/

NAS Spool Permission Denied

# Use ACLs (recommended)
setfacl -R -m u:www-data:rwx /path/to/nas/spool

# CRITICAL: Set on ALL parent directories too
setfacl -m u:www-data:rx /NAS
setfacl -m u:www-data:rx /NAS/voipmonitor

# Verify
sudo -u www-data ls /path/to/nas/spool

SELinux Blocking mscgen (SIP Call Flow Diagrams)

If SIP call flow diagram shows "convert msc to svg (mscgen): unknown error": 

# Check SELinux denials
ausearch -m avc -ts recent | grep mscgen

# Create policy module
grep mscgen /var/log/audit/audit.log | audit2allow -M mscgen_local
semodule -i mscgen_local.pp

IonCube and PHP Issues

Temp Directory Not Writable

# Check and fix /tmp permissions
ls -ld /tmp
chmod 1777 /tmp

# Check SELinux
getenforce
# If Enforcing, try: setenforce 0

# Check systemd PrivateTmp
systemctl show httpd | grep PrivateTmp
# If true, create override:
# /etc/systemd/system/httpd.service.d/privatetmp.conf
# [Service]
# PrivateTmp=false

PHP CLI vs Web Server Mismatch

# Test as web server user
sudo -u www-data php -r 'echo extension_loaded("ionCube Loader")?"yes":"no";'

# Compare configs
php --ini                    # CLI config
# Create phpinfo: echo '<?php phpinfo();' > /var/www/html/info.php
# Check "Loaded Configuration File" in browser
# Remove after: rm /var/www/html/info.php

License Activation: Server Side Error - Forbidden

This indicates ionCube PHP version incompatibility:

GUI Version PHP Version
v26.37+ PHP 8.3 supported
Older PHP 8.2 or earlier required 
# Check Apache error log
tail -50 /var/log/apache2/error.log | grep -i ioncube

# Solution: Either upgrade GUI to v26.37+ or downgrade PHP to 8.2

Other Issues

Invalid Timezone (GUI Inaccessible)

# Edit directly
nano /var/www/html/config/configuration.php

# Find and fix:
define('TIMEZONE', 'UTC');  # Use valid timezone
define('SENSORS_TIMEZONE', 'UTC');

systemctl restart php-fpm

Valid timezones: UTC, Europe/London, America/New_York, Europe/Berlin, etc.

Faxes Not Displayed

# Install libtiff-tools
apt-get install libtiff-tools    # Debian/Ubuntu
yum install libtiff-tools        # RHEL/CentOS

# Verify
which tiff2pdf

File Attachment Blocked

Files like .xlsx, .docx are blocked for security. Compress to .zip or .gz: 

zip archive.zip your_file.xlsx

RRD Graph Errors After Hardware Change

# Delete corrupted RRD files
rm -rf /var/spool/voipmonitor/rrd/*

# Sensor will recreate them
systemctl restart voipmonitor

/dev/shm Full (Session Logout Issues)

# Check usage
df -h /dev/shm

# Temporary increase
mount -o remount,size=1G /dev/shm

# Permanent: edit /etc/fstab
# tmpfs /dev/shm tmpfs defaults,size=1G 0 0

Concurrent Calls Stats Error

Usually caused by duplicate cron entries: 

# Check for duplicates
grep "voipmonitor\|run.php" /etc/crontab

# Comment out duplicates, keep only one

TShark CVE Security Updates

The bundled tshark may have known CVEs. To update: 

cd /var/www/html/bin

# Download updated binary from Wireshark
wget https://www.wireshark.org/download/automated/linux64/tshark
chmod +x tshark
chown www-data:www-data tshark

Dashboard Edit Icon Grayed Out

User lacks edit permission. Grant via User Management > edit user > enable "Edit Dashboard" permission.

GUI Not Loading (Spool Directory Ownership)

If the GUI is not loading or shows generic error messages, the cause may be incorrect ownership of the spool directory by the web server user. 

# Fix spool directory ownership
chown -R www-data:www-data /var/spool/voipmonitor    # Debian/Ubuntu
# OR
chown -R apache:apache /var/spool/voipmonitor        # RHEL/CentOS

ℹ️ Note: The spool directory path is defined by the spooldir parameter in /etc/voipmonitor.conf. Use the correct path for your configuration.



AI Summary for RAG

Summary: VoIPmonitor GUI troubleshooting guide covering: (1) HTTP 500 errors - try client-side steps first (different browser, incognito, clear cache), then check Apache/PHP error logs and SELinux; (2) No CDR for current day - check sensor status in Settings > Sensors, generate debug log if UP, check partitions/timezone/cdr config if Down; (3) GUI debug mode - add ?debug=31415 to URL; (4) Database timeouts - tune innodb_buffer_pool_size (50-70% RAM), increase mod_fcgid timeouts; (5) Database corruption - use mysqlcheck, drop/recreate tables, or innodb_force_recovery; (6) GUI upgrade issues - reset password via database, fix ownership, run php php/run.php upgrade -f; (7) Permission errors - chmod +x bin/*, use setfacl for NAS; (8) IonCube errors - check SELinux, PHP version compatibility (GUI v26.37+ for PHP 8.3); (9) Schema errors - use ?check_tables=1 URL parameter.

Keywords: HTTP 500, no CDR, sensor status, debug mode, ?debug=31415, innodb_buffer_pool_size, MySQL tuning, database corruption, mysqlcheck, innodb_force_recovery, GUI upgrade, password reset, IonCube, SELinux, PHP 8.3, ?check_tables=1, NAS permissions, setfacl, mod_fcgid timeout, FcgidIOTimeout, mscgen, SIP call flow, tshark CVE, MySQL 8.0.42, SYSTEM_USER privilege, /dev/shm, timezone error, libtiff-tools, file attachment blocked

Key Questions:

  • HTTP 500 error on GUI - what to try first?
  • No CDR or dashboard data for current day - how to diagnose?
  • How to enable GUI debug mode?
  • GUI queries timeout - how to tune MySQL?
  • Database corrupted - which recovery method to use?
  • Cannot login after GUI upgrade - how to reset password?
  • Permission errors with NAS spool - how to fix?
  • IonCube temp directory errors - what causes them?
  • GUI shows "Unknown column in field list" - how to fix?
  • SIP call flow diagram shows error - how to fix SELinux blocking?
  • MySQL 8.0.42 causing crashes - what to do?
  • License activation fails with Forbidden error - what causes it?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=GUI_troubleshooting&oldid=10531"