Data Cleaning

This guide explains how VoIPmonitor manages data retention for PCAP files and database records.

Overview

VoIPmonitor generates two types of data requiring periodic cleanup:

Data Type	Storage	Cleanup Mechanism	Key Parameters
PCAP Files	Filesystem (spool directory)	Cleanspool process (every 5 min)	maxpoolsize, maxpooldays
CDR Records	MySQL database	Partition dropping (instant)	cleandatabase

ℹ️ Note: These are independent systems - filesystem cleanup does not affect database records and vice versa.

Quick Reference

Purpose	Parameter	Example
Limit total PCAP size	maxpoolsize	maxpoolsize = 512000 (500 GB)
Limit RTP specifically	maxpoolrtpsize	maxpoolrtpsize = 102400 (100 GB)
Limit PCAP age	maxpooldays	maxpooldays = 30
CDR retention (days)	cleandatabase	cleandatabase = 30
CDR retention (size)	cleandatabase_size	cleandatabase_size = 512000 (+ cleandatabase_size_force = true)
Save only RTP stats	savertp	savertp = header (saves ~90% space)

Filesystem Cleaning (PCAP Files)

How Cleanspool Works

The sniffer maintains a file index in memory. Every 5 minutes, the cleanspool thread checks retention limits and deletes the oldest files when limits are exceeded.

The .cleanspool_cache files in hourly directories serve as persistent storage for fast restart - they allow quick index reload without scanning the entire filesystem.

ℹ️ Note: Legacy Indexing: Old VoIPmonitor versions stored metadata in MySQL files table. This can be enabled with cleanspool_use_files = yes (deprecated, do not use).

Retention Configuration

Limits can be set by size (MB) or age (days). When both are configured, the first limit reached triggers cleanup.

Global Limits

Parameter	Default	Description
maxpoolsize	102400 (100 GB)	Maximum total size for all PCAP data
maxpooldays	(unset)	Maximum age in days for all PCAP data

Per-Type Limits

Data Type	Size Parameter	Days Parameter
SIP signaling	maxpoolsipsize	maxpoolsipdays
RTP audio	maxpoolrtpsize	maxpoolrtpdays
Quality graphs	maxpoolgraphsize	maxpoolgraphdays
Converted audio (WAV/OGG)	maxpoolaudiosize	maxpoolaudiodays

Recommended Configuration

Limit RTP (largest files) while keeping SIP longer for troubleshooting:

# /etc/voipmonitor.conf
maxpoolrtpsize = 102400   # 100 GB limit for RTP
maxpoolsize = 512000      # 500 GB overall limit

Secondary Storage Limits (spooldir_2)

When using spooldir_2 for secondary storage (configured via capture rules with "Store pcaps to second spooldir" enabled), use _2 suffix parameters for cleaning:

maxpoolsize_2, maxpooldays_2, maxpoolrtpsize_2, etc.

ℹ️ Note: spooldir_2 is independent from tar_move. The _2 suffix parameters apply only to the secondary spooldir configured via capture rules, not to tar_move destination storage.

Emergency Cleanup

⚠️ Warning: Emergency cleanup activates when disk is nearly full and ignores all maxpool* settings.

Parameter	Default	Triggers When
autocleanspoolminpercent	1	Disk usage reaches 99%
autocleanmingb	5	Free space below 5 GB

When triggered, oldest data is deleted aggressively until thresholds are cleared. The cleanspool_enable_fromto time window is ignored.

Control Parameters

Parameter	Default	Description
cleanspool	yes	Enable/disable spool cleaning
cleanspool_enable_fromto	0-24	Restrict cleaning to hours (e.g., 1-5 for 1-5 AM)
maxpool_clean_obsolete	no	Delete files not in index (use with caution)
all_unlink_log	no	Log all file deletions

Reducing Data at Source

Save RTP Headers Only

If you only need quality metrics (MOS, jitter, packet loss) without audio playback:

savertp = header

This reduces storage by up to 90% while preserving all quality statistics.

Disable RTP Entirely

If SIP signaling retention is the priority and you do not need RTP data at all:

savertp = no

This disables RTP packet storage entirely, reducing PCAP size by approximately 20x. Use this when:

• Storage limits are insufficient for even header-only RTP
• SIP troubleshooting is the primary need (no audio analysis required)
• You need to retain SIP signaling for much longer periods

⚠️ Warning:

Selective Recording

To record full audio only for specific calls:

1. Set savertp = header globally
2. Create capture rules with recordRTP=ON for exceptions

See Capture_rules for details.

Database Cleaning (CDR Records)

Partitioning Method

VoIPmonitor uses daily partitioning. Dropping old partitions is instant (milliseconds) regardless of row count.

# Keep CDR records for 30 days
cleandatabase = 30

Database Cleaning Parameters

Parameter	Default	Description
cleandatabase	0 (disabled)	Global retention for CDR, register_state, register_failed, sip_msg
cleandatabase_cdr	0	CDR table (includes message table)
cleandatabase_rtp_stat	2	RTP statistics table
cleandatabase_register_state	0	Registration state
cleandatabase_register_failed	0	Failed registrations
cleandatabase_register_time_info	0	Registration timing (NOT covered by global)
cleandatabase_sip_msg	0	SIP messages (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase_ss7	0	SS7 records
partition_operations_enable_fromto	1-5	Time window for partition operations (24h format)

⚠️ Warning: register_time_info is NOT covered by global cleandatabase. Set cleandatabase_register_time_info explicitly.

Size-Based Cleaning

To limit database by size instead of time:

cleandatabase_size = 512000        # 500 GB limit in MB
cleandatabase_size_force = true    # Required to enable

Multi-Sensor Environments

When multiple sensors share a database, only ONE sensor should manage partitions:

# On all sensors EXCEPT one:
disable_partition_operations = yes

# On the designated sensor:
partition_operations_enable_fromto = 4-6

Limits

• MySQL/MariaDB partition limit: 8,192 partitions per table including subpartitions (~22 years with daily partitioning). This limit applies since MySQL 5.6.7+ and MariaDB 10.0.4+ (older versions: 1,024).
• Aurora DB partition limit: 8,192 partitions per table (same as standard MySQL)
• CDR record limit: No practical limit - uses BIGINT

Pre-Creating Partitions (High Traffic / Aurora DB)

Partition operations (ALTER TABLE ... ADD/DROP PARTITION) require an exclusive metadata lock on the entire table. Under high traffic, this blocks all concurrent queries (SELECT, INSERT, UPDATE) until the partition operation completes. On Aurora DB, partition DDL can additionally cause replica reboots and replication lag spikes.

To avoid these disruptions, you can pre-create partitions before starting the sniffer and prevent runtime partition creation:

# Pre-create partitions up to a specific date before starting
create_new_partitions_until = 2026-01-31

# Disable only partition creation (dropping still works for cleanup)
disable_partition_operations_create = yes

# Or disable only partition dropping (creation still works)
# disable_partition_operations_drop = yes

# Number of partitions to create ahead (default 2)
create_new_partitions = 2

ℹ️ Note: Aurora DB supports up to 8,192 partitions per table (same as standard MySQL). With daily partitions, this allows ~22 years of data. Plan retention accordingly using cleandatabase.

Advanced Topics

Spool Directory Location

Default: /var/spool/voipmonitor

Structure: YYYY-MM-DD/HH/MM/{SIP|RTP|GRAPH|AUDIO}/files...

Relocating the Spool

# 1. Create new directory
mkdir -p /mnt/storage/voipmonitor
chown voipmonitor:voipmonitor /mnt/storage/voipmonitor

# 2. Update sniffer config
# /etc/voipmonitor.conf:
# spooldir = /mnt/storage/voipmonitor

# 3. Update GUI config (config/configuration.php):
# define('SNIFFER_DATA_PATH', '/mnt/storage/voipmonitor');

# 4. Restart
systemctl restart voipmonitor

Tiered Storage (tar_move)

Extend retention using secondary storage:

spooldir = /var/spool/voipmonitor
tar_move = yes
tar_move_destination_path = /mnt/archive/voipmonitor

Files in secondary storage remain accessible via GUI.

S3 Cloud Storage

Use rclone instead of s3fs to avoid GUI unresponsiveness:

rclone mount bucket-name /mnt/s3-archive \
  --allow-other --dir-cache-time 30s --vfs-cache-mode off

Custom Autocleaning (GUI)

For one-time cleanup of specific recordings (by IP, phone number, etc.):

1. Navigate to Settings > Custom Autocleaning
2. Create rule with filters
3. Apply and remove rule after completion

Troubleshooting

Files Disappearing Faster Than Expected

1. Check if emergency cleanup is active

df -h /var/spool/voipmonitor
# If >95% full, emergency cleanup is running

2. Check GUI configuration override

When mysqlloadconfig = yes (default), GUI settings override config file. Check: Settings > Sensors > wrench icon.

3. Set appropriate limits

Set maxpoolsize to 90-95% of disk capacity to leave buffer for growth.

Disk Space Not Reclaimed After Database Cleanup

Check innodb_file_per_table:

SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';

If OFF, enable in my.cnf:

[mysqld]
innodb_file_per_table = 1

MySQL Error 28: No Space Left

Enable size-based cleaning:

cleandatabase_size = 512000
cleandatabase_size_force = true

Other causes: Inode exhaustion (df -i), MySQL tmpdir full.

Verify Database Cleaning

SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'cdr'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;

If partition count matches your cleandatabase setting, cleaning IS working.

For SQL queue issues and database performance, see Database_troubleshooting.

See Also

• Sniffer_configuration - Full parameter reference
• Database_troubleshooting - SQL queue, performance issues
• Scaling - Performance tuning
• Capture_rules - Selective recording

AI Summary for RAG

Summary: VoIPmonitor has two independent data retention systems: (1) Cleanspool for PCAP files using maxpoolsize/maxpooldays parameters, running every 5 minutes with in-memory file index; (2) Database cleaning using cleandatabase with instant partition dropping. Key space-saving option: savertp = header reduces storage by 90% while keeping quality metrics. Emergency cleanup (autocleanspoolminpercent=1, autocleanmingb=5) activates when disk nearly full and ignores normal limits. GUI settings override config file when mysqlloadconfig=yes. Size-based database cleaning requires BOTH cleandatabase_size AND cleandatabase_size_force=true. In multi-sensor environments, only one sensor should manage partitions (disable_partition_operations=yes on others).

Keywords: data retention, maxpoolsize, maxpooldays, maxpoolrtpsize, cleandatabase, cleandatabase_size, cleandatabase_size_force, cleanspool, autocleanspoolminpercent, autocleanmingb, tar_move, tiered storage, savertp header, innodb_file_per_table, partition dropping, emergency cleanup, mysqlloadconfig, disable_partition_operations

Key Questions:

• How does VoIPmonitor cleanspool work?
• How to configure PCAP file retention?
• How to limit database size?
• Why are files being deleted faster than expected?
• How to fix MySQL Error 28 no space left?
• How to reduce storage usage (savertp header)?
• How to configure tiered storage?
• Why is disk space not reclaimed after cleanup?
• How to manage partitions in multi-sensor environment?