SQL Queue Growing During Peak Time

When the SQL queue (SQLq/SQLf) grows during peak traffic, the database cannot keep up with the CDR insertion rate. This causes delayed CDR visibility in the GUI and can lead to memory exhaustion.

Quick Diagnosis

Monitor the queue:

journalctl -u voipmonitor -f | grep SQLf

Check RRD graphs: Settings > Sensors > click graph icon > select peak time period. Look for growing "SQL Cache" or "SQL Cache Files".

ℹ️ Note: SQLf value should normally stay below 10. If it consistently exceeds 10 or grows during peak time, apply mitigations.

Storage Speed is Critical

The most common cause of SQL queue growth is slow storage. MySQL write performance depends heavily on disk I/O speed.

Storage Type	Typical IOPS	CDR Write Capacity	Recommendation
HDD (7200 RPM)	~150	10-50 CDR/s	Not suitable for production
SATA SSD	~50,000	200-500 CDR/s	Suitable for small deployments
NVMe SSD	~500,000+	1000+ CDR/s	Recommended for production

Check your current I/O performance:

# Check disk utilization during peak time
iostat -x 1 5 | grep -E "Device|sda|nvme"

# If %util stays above 80%, storage is the bottleneck

VoIPmonitor Tuning

These parameters in /etc/voipmonitor.conf control how the sniffer writes to the database:

Parameter	Default	Standard	Medium-High	Description
`mysqlstore_max_threads_cdr`	1	4-8	16-24	Parallel CDR write threads. Auto-scales when queue >1000.
`mysqlstore_concat_limit`	400	1000-2000	5000-10000	CDRs batched per INSERT. Higher values require larger `max_allowed_packet` in MySQL.
`query_cache`	no	yes	yes	Write queries to disk when queue grows (prevents OOM)

Example — standard deployment:

# /etc/voipmonitor.conf
mysqlstore_max_threads_cdr = 8
mysqlstore_concat_limit = 2000
query_cache = yes

Example — medium-high deployment (CDR Summary enabled, 200-500 CDR/s):

mysqlstore_max_threads_cdr = 18
mysqlstore_concat_limit = 10000
query_cache = yes

ℹ️ Note: CDR Summary impact: Enabling cdr_summary = yes increases database write load. If SQL queue starts growing after enabling it, increase mysqlstore_max_threads_cdr and mysqlstore_concat_limit as shown above. High concat_limit values (>2000) require max_allowed_packet ≥ 256M in MySQL/MariaDB.

Restart after changes: systemctl restart voipmonitor

MySQL Tuning

Essential Settings

These settings are safe for most deployments and significantly improve write performance:

# /etc/mysql/mysql.conf.d/mysqld.cnf

# Buffer pool - use 50-70% of available RAM
innodb_buffer_pool_size = 8G

# Faster writes - flush log once per second instead of every transaction
# Small risk: up to 1 second of data loss on crash
innodb_flush_log_at_trx_commit = 2

# I/O threads - adjust to your CPU core count
innodb_read_io_threads = 4
innodb_write_io_threads = 4
innodb_purge_threads = 2

# Skip DNS lookups for faster connections
skip-name-resolve

# Larger log files = fewer checkpoints = better performance
innodb_log_file_size = 1G
innodb_log_buffer_size = 256M

SSD/NVMe Specific Settings

If using SSD or NVMe storage, add these optimizations:

# Disable HDD optimization (not needed for SSD)
innodb_flush_neighbors = 0

# Disable synchronous flushing during checkpoints
# Prevents periodic write stalls on SSD (works with innodb_flush_neighbors=0)
innodb_flush_sync = off

# Tell InnoDB about fast storage (adjust to your disk specs)
# SATA SSD: 2000-4000 / NVMe: 10000-50000
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000

# Bypass OS cache - better for dedicated DB servers
innodb_flush_method = O_DIRECT

Restart MySQL after changes: systemctl restart mysql

ProxySQL: max_allowed_packet Synchronization

When using ProxySQL in front of MySQL/MariaDB (common with Galera clusters), max_allowed_packet must match between ProxySQL and the backend. Mismatched values cause ERROR 1153: Got a packet bigger than 'max_allowed_packet' bytes when using high mysqlstore_concat_limit.

-- Check current value in ProxySQL
SELECT @@global.max_allowed_packet;

-- Set to match MariaDB backend (e.g. 256MB)
SET GLOBAL max_allowed_packet=268435456;

Persist in /etc/proxysql.cnf:

mysql_variables =
{
    max_allowed_packet=268435456
}

Verify Settings

# Check current buffer pool size
mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"

# Check if buffer pool is fully utilized
mysql -e "SHOW STATUS LIKE 'Innodb_buffer_pool_pages%';"

Prevention

Schedule heavy operations off-peak: Configure alerts and reports to run at night
Separate components: Use dedicated database server for high-traffic deployments
Monitor trends: Set up alerts for SQLq growth before it becomes critical

Extreme High-Volume Deployments

For deployments processing hundreds or thousands of CDRs per second (10,000+ concurrent calls), standard tuning is not sufficient. These scenarios require:

Disabling AUTO_INCREMENT and letting the sniffer generate IDs
Hourly table partitioning instead of daily
Batch sizes of 10,000+ CDRs
Aggressive MySQL settings that trade durability for performance

See High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for detailed configuration.

AI Summary for RAG

Summary: Troubleshooting guide for SQL queue (SQLq/SQLf) growth during peak traffic periods. The queue grows when the database cannot keep up with CDR insertion rate. Diagnosis involves checking mysqld CPU usage, iowait, and RRD graphs in GUI (Settings > Sensors > graph icon). Immediate mitigations: enable query_cache=yes to prevent OOM, quick_save_cdr=yes to reduce GUI delay, mysqlstore_max_threads_cdr=8 for parallel writes. Root causes are typically I/O bottleneck (upgrade HDD to SSD/NVMe), RAM bottleneck (increase innodb_buffer_pool_size to 50-70% of RAM), or CPU bottleneck (hardware upgrade needed). Prevention includes scheduling heavy operations off-peak and using dedicated database servers.

Keywords: SQL queue, SQLq, SQLf, peak time, database bottleneck, query_cache, quick_save_cdr, mysqlstore_max_threads_cdr, innodb_buffer_pool_size, I/O bottleneck, SSD upgrade, OOM, CDR delay, database performance, peak traffic

Key Questions:

Why is the SQL queue growing during peak hours?
How do I monitor SQL queue size in VoIPmonitor?
What causes CDR delays in the GUI?
How do I prevent out-of-memory errors when SQL queue grows?
Should I upgrade to SSD for VoIPmonitor database?
What is the recommended innodb_buffer_pool_size setting?
How do I speed up CDR insertion during high traffic?

SQL queue is growing in a peaktime