Data Cleaning

This guide explains how VoIPmonitor manages data retention for both captured packets (PCAP files) and Call Detail Records (CDRs) in the database. Proper configuration is essential for managing disk space and maintaining long-term database performance.

Overview

VoIPmonitor generates two primary types of data that require periodic cleaning:

PCAP Files: Raw packet captures of SIP/RTP/GRAPH data stored on the filesystem in the spool directory. These can consume significant disk space.
CDR Data: Call metadata stored in the MySQL database. Large tables can slow down GUI performance if not managed properly.

The system uses two separate, independent mechanisms to manage the retention of this data:

Filesystem Cleaning (PCAP Spool Directory)

The sensor stores captured call data in a structured directory tree on the local filesystem.

How Cleanspool Works: Modern vs Legacy Mode

VoIPmonitor has two distinct modes for tracking and cleaning PCAP files. Understanding which mode your installation uses is critical for troubleshooting cleaning issues.

Important: Two Cleanspool Modes
Legacy Mode (Default):	`cleanspool_use_files = no` Uses a filesystem-based index stored in `.cleanspool_cache` file in the spool directory. The cleanspool process scans the directory structure and maintains this cache file locally. Why this is the default: The `pcap_dump_tar = yes` setting (TAR archive mode) is enabled by default. When TAR mode is active, VoIPmonitor automatically switches to legacy cleanspool mode unless you explicitly set `cleanspool_use_files = yes`.
Modern Mode (Database-Indexed):	`cleanspool_use_files = yes` Uses the MySQL `files` table to track all created PCAP files. The sniffer records every file it creates in this database table, and cleanspool deletes only files that are indexed in the `files` table. Key behavior: Files that exist on the filesystem but are NOT in the `files` table will NOT be automatically deleted (unless `maxpool_clean_obsolete` is enabled). To enable modern mode: cleanspool_use_files = yes

Legacy Mode: Filesystem Cache (Default)

In legacy mode (cleanspool_use_files = no), which is active by default when TAR mode is enabled:

HTTP-Response: 
Error 400: Syntax Error? (Assumed diagram type: sequence) (line: 5)

Diagram-Code:
@startuml
skinparam shadowing false
participant "Sniffer" as S
participant "Cleanspool\nThread" as C
collections "Filesystem" as FS
file ".cleanspool_cache" as Cache

S -> FS: Write PCAP file\n(into TAR archive)

... Every 5 minutes ...

C -> FS: Scan directory structure
C -> Cache: Update file index
C -> Cache: Read oldest files\nwhere age > maxpooldays\nOR total size > maxpoolsize
loop For each file
    C -> FS: DELETE file
    C -> Cache: Update index
end
@enduml

Characteristics of Legacy Mode:

Uses .cleanspool_cache file in spool directory for tracking
Scans filesystem to build and update the index
Default behavior when pcap_dump_tar = yes (TAR archive mode is default)
Files not in cache are ignored (unless maxpool_clean_obsolete = yes)

Checking which mode is active:

# Check TAR mode (if yes, likely legacy cleanspool)
grep pcap_dump_tar /etc/voipmonitor.conf
grep tar /etc/voipmonitor.conf | head -5

# Check explicit cleanspool_use_files setting
grep cleanspool_use_files /etc/voipmonitor.conf

# Check if .cleanspool_cache exists (indicates legacy mode is/was used)
ls -la /var/spool/voipmonitor/.cleanspool_cache

Modern Mode: Database-Indexed Cleaning

In modern mode (cleanspool_use_files = yes), which must be explicitly enabled:

Advantages of Modern Mode:

Faster cleanup (no filesystem scanning)
Accurate size tracking via database
Supports distributed deployments with shared storage
Files not in database are protected from accidental deletion

To enable Modern Mode:

# /etc/voipmonitor.conf
cleanspool_use_files = yes

The maxpool_clean_obsolete Parameter

This parameter controls how cleanspool handles files that exist on the filesystem but are NOT in its index (either database files table or .cleanspool_cache).

Setting	Behavior	Use Case
`maxpool_clean_obsolete = no` (default)	Only delete files that are indexed. Unknown files are ignored.	Safe default - protects manually added files
`maxpool_clean_obsolete = yes`	Delete ALL files in spool directory, including those not in index	Clean up orphaned files, recover from index corruption

Warning: maxpool_clean_obsolete
Caution:	When `maxpool_clean_obsolete = yes` is enabled, cleanspool will scan the entire filesystem and delete any files not found in its index. This includes: Files manually copied into the spool directory Files from backup restores Files from other sensors (in shared storage scenarios) Only enable this if you understand the implications and want aggressive cleanup of unindexed files.

Reducing Data Collection at Source

Before configuring cleanup policies, consider reducing the amount of data captured. This is often the most effective long-term solution for storage management.

Save Only RTP Headers (Major Space Saver)

RTP packets typically contain the full audio payload, which consumes the majority of disk space. If you only need call quality statistics (MOS, jitter, packet loss) and not actual audio playback, switch to saving RTP headers only.

Edit /etc/voipmonitor.conf:

# Change from full RTP to headers only
savertp = header

Setting	Storage Impact	Use Case
`savertp = yes`	High (~10x more)	Requires ability to play back audio from PCAPs
`savertp = header`	Low	Only CDR statistics needed, no audio playback required

With savertp = header, VoIPmonitor still captures all necessary metadata for MOS scoring, jitter analysis, packet loss statistics, and quality graphs, but does not store the actual audio payload. This can reduce storage consumption by up to 90%.

Important: After changing from savertp = yes to savertp = header, existing PCAP files will remain playable. New calls will only contain RTP headers.

Critical: Distributed Architecture Consideration

packetbuffer_sender Architecture:

If you are using the Packet Mirroring/Client-Server mode (packetbuffer_sender=yes on remote sensors), the savertp setting must be applied on the central server where packet analysis is performed, not on the individual sensors.

Architecture	Where to Apply `savertp = header`
`packetbuffer_sender = no` (Local Processing)	On each sensor/probe's configuration file
`packetbuffer_sender = yes` (Packet Mirroring)	On the central server's configuration file (not on sensors)

For distributed deployment details, see Client-Server Architecture.

Enabling Selective Audio Recording with Capture Rules

For compliance scenarios where you need to record full audio for specific calls while keeping most calls with headers only (or disabled), use GUI capture rules to create exceptions:

Workflow:

1. **Set global default to headers only:**

savertp = header

2. **Create capture rules in the GUI for exceptions:**

  * Navigate to Control Panel > Capture Rules
  * For selective audio recording, set the recordRTP option to ON
  * Rules with recordRTP=ON will capture full RTP audio, overriding the global savertp=header setting for matched calls

Use Cases:

Scenario	Configuration	File Storage
Legal Holds	Create IP rule for subject party with `recordRTP=ON`	Full audio recorded
VIP Customers	Create phone number prefix rules for specific ranges with `recordRTP=ON`	Full audio recorded
All Other Calls	Global `savertp = header`	Headers only (no audio)

This approach allows you to comply with data retention laws (e.g., GDPR) by minimizing audio recording while still meeting specific legal or business requirements for certain call categories.

For detailed capture rule configuration options, see Capture_rules. For more configuration options, see Sniffer Configuration - Saving Options.

Spool Directory Structure

The spool directory uses a hierarchical structure organized by time and data type.

Structure	Description
`YY-mm-dd/HH/MM/{TYPE}/files...`	Date → Hour → Minute → Data Type

The {TYPE} subdirectory can be:

SIP - SIP signaling PCAP files
RTP - RTP audio PCAP files
AUDIO - Converted audio files (WAV/OGG)
GRAPH - Quality graph image files

Example path:

/var/spool/voipmonitor/25-01-06/10/45/RTP/rtp_2025-01-06-10-45.tar

The timestamps are based on UTC or the timezone configured in voipmonitor.conf.

Important: The cleanup process operates at the minute level, not hour level. When cleaning is triggered, it removes the oldest minute's worth of data (e.g., 25-01-06/08/57/) rather than deleting an entire hour's worth of data at once.

Spool Directory Location

By default, all data is stored in /var/spool/voipmonitor. This location can be changed by setting the spooldir option in voipmonitor.conf.

Relocating the Spool Directory to a Different Partition

If your default partition is running out of space, you can move the spool directory to a larger partition or dedicated disk. This is particularly useful when MySQL Error 28 ("No space left on device") occurs despite retention settings being in place.

Situation	Recommended Action
`/var/lib/mysql` full but other partitions available	Move spool to larger partition and update `spooldir` in `voipmonitor.conf`
Multiple disks available	Use dedicated partition for PCAP storage
GUI on same server	Sync GUI `SNIFFER_DATA_PATH` to match new `spooldir`

Procedure to Relocate Spooldir

Step 1: Identify Available Space

Check all partitions to find a suitable location:

df -h

Step 2: Create New Spool Directory

Create the directory on the destination partition:

# Example: Use /mnt/pcaps partition
mkdir -p /mnt/pcaps/voipmonitor

# Set correct ownership for both sniffer and GUI access
chown voipmonitor:voipmonitor /mnt/pcaps/voipmonitor
chmod 755 /mnt/pcaps/voipmonitor

Step 3: Update VoIPmonitor Configuration

Edit /etc/voipmonitor.conf to point to the new location:

spooldir = /mnt/pcaps/voipmonitor

Step 4: Update GUI Configuration (Critical)

If the GUI runs on the same server, you MUST update the path the GUI uses to access the spool. This is defined in the GUI configuration file:

Debian/Ubuntu: /var/www/html/voipmonitor/config/configuration.php
RHEL/CentOS: /var/www/voipmonitor/config/configuration.php

Edit the GUI configuration file:

// Find and update this line to match your new spooldir
define('SNIFFER_DATA_PATH', '/mnt/pcaps/voipmonitor');

Important: The SNIFFER_DATA_PATH in the GUI config MUST match the spooldir setting in voipmonitor.conf. If these are mismatched, the GUI will be unable to locate and display call recordings.

Step 5: Restart Services

Apply all changes by restarting the services:

# Restart the VoIPmonitor sensor
systemctl restart voipmonitor

# If using Apache/Nginx, reload web server
systemctl reload apache2    # Debian/Ubuntu
# or
systemctl reload nginx       # RHEL/CentOS

Step 6: Verify Configuration

# Check if new spool directory is being used
ls -ld /mnt/pcaps/voipmonitor

# Verify GUI can access the path
grep SNIFFER_DATA_PATH /path/to/gui/config/configuration.php

# Check sensor logs for any spool-related errors
tail -f /var/log/voipmonitor.log

Optional: Move Existing Data

If you want to migrate existing PCAP files to the new location:

# Stop the sensor to prevent writes during migration
systemctl stop voipmonitor

# Move existing data (this may take a long time)
mv /var/spool/voipmonitor/* /mnt/pcaps/voipmonitor/

# Restart the sensor
systemctl start voipmonitor

Retention Mechanism

The cleaning process runs automatically every 5 minutes and removes data based on the configuration in voipmonitor.conf.

Mechanism	Check Interval	Purpose
*Retention Policy (`maxpool`)**	Every 5 minutes	Capacity control - maintains storage below configured limits
*Emergency Cleanup (`autoclean`)**	Every 5 minutes	Disk space "fuse" - prevents disk exhaustion when free space falls below threshold

Retention Policy: maxpool* Parameters

When triggered by a maxpool* setting (e.g., maxpoolsize, maxpoolrtpsize):

1. The cleanup identifies the oldest minute of data 2. It removes the entire YY-mm-dd/HH/MM/ directory tree (e.g., 25-01-06/08/57/) 3. If maxpoolsize is used (overall limit), it removes all data types from that minute 4. If type-specific limits are used (e.g., maxpoolaudiosize), it removes only that data type

Important: The cleanup removes ONE minute of data at a time, not an entire hour's worth. This ensures gradual, controlled cleanup instead of sudden large deletions.

Emergency Cleanup: autoclean* Parameters

The autoclean* parameters act as a safety "fuse" to prevent the disk from filling up:

Parameter	Default	Behavior
`autocleanspoolminpercent`	1	When disk usage reaches 99%, trigger emergency cleanup regardless of `maxpool*` settings
`autocleanmingb`	5	When free space falls below 5 GB, trigger emergency cleanup

When either threshold is breached:

Oldest data is deleted AGGRESSIVELY until free space is restored
This happens regardless of your maxpool* or maxpooldays settings
Normal retention behavior resumes once thresholds are cleared

Performance Warning: maxpoolaudiosize

The maxpoolaudiosize parameter controls the size of converted audio files (WAV/OGG) when saveaudio is enabled.

Performance Consideration
Warning	Audio conversion (PCAP → WAV/OGG) is CPU-intensive. For deployments with high call volumes (>200 concurrent calls or high CPS), using `saveaudio` can overload the system and cause packet loss or performance degradation.

Recommended: For high-volume deployments, use savertp = yes and convert on-demand via the GUI instead
Alternative: Consider savertp = header if you only need statistics, not audio playback

Retention Configuration

The cleaning configuration allows you to set limits based on total size (in Megabytes) or age (in days).

Important: Default Retention Behavior
Default Behavior:	By default, PCAP files are deleted based on size only (using `maxpoolsize`), not based on time. The default `maxpoolsize` is 100 GB (102400 MB). When the spool directory reaches 100 GB, the oldest PCAP files are automatically deleted to free up space. Time-based retention (`maxpooldays`) is disabled by default and must be explicitly configured if you want to limit retention by days instead of size.

You can set limits based on total size (in Megabytes) or age (in days). If both a size and day limit are set for the same data type, the first limit that is reached will trigger the cleaning.

Parameter	Default Value	Description
`maxpoolsize`	102400 (100 GB)	The total maximum disk space for all captured data (SIP, RTP, GRAPH, AUDIO).
`maxpooldays`	(unset)	The maximum number of days to keep all captured data.
`maxpoolsipsize`	(unset)	A specific size limit for SIP PCAP files only.
`maxpoolsipdays`	(unset)	A specific age limit for SIP PCAP files only.
`maxpoolrtpsize`	(unset)	A specific size limit for RTP PCAP files only.
`maxpoolrtpdays`	(unset)	A specific age limit for RTP PCAP files only.
`maxpoolgraphsize`	(unset)	A specific size limit for GRAPH files only.
`maxpoolgraphdays`	(unset)	A specific age limit for GRAPH files only.
`maxpoolaudiosize`	(unset)	A specific size limit for converted audio files (WAV/OGG) only.
`maxpoolaudiodays`	(unset)	An age limit for converted audio files (WAV/OGG) only.

Recommended Configuration Strategy: Mixed Size-Based Retention

For optimal disk space management while preserving SIP signaling for long-term analysis, use a **mixed retention strategy**:

Data Type	Recommended Setting	Rationale
RTP (audio payload)	Size-based (`maxpoolrtpsize`)	RTP consumes most disk space (5-10x more than SIP)
SIP (signaling)	Overall only (`maxpoolsize`, no `maxpoolsipdays`)	SIP files are small; keep as long as total space allows
Charts/Audio	Overall only (`maxpoolsize`)	Supplementary data, keep as space permits

Configuration Example: Long-Term SIP with Limited RTP
Scenario:	You want to keep SIP signaling and charts for as long as possible, but limit RTP (audio) to a reasonable size to prevent disk exhaustion.
Configuration:	Edit `/etc/voipmonitor.conf`: # Set size limit for RTP only (keeps audio limited) maxpoolrtpsize = 102400 # 100 GB for RTP # DO NOT set maxpoolsipdays - let SIP be controlled by maxpoolsize only # Set overall limit for all data together maxpoolsize = 512000 # 500 GB total limit (includes SIP + RTP + GRAPH + AUDIO)
Behavior:	RTP is deleted when it exceeds `maxpoolrtpsize` (100 GB). SIP, chart, and audio files are kept as long as the total `maxpoolsize` (500 GB) allows. If the disk fills up (total exceeds 500 GB), oldest data is deleted regardless of type.
Why this works:	RTP files are huge (GBs per day) but rarely needed for troubleshooting. SIP files are small (MBs per day) and essential for analyzing call setup/teardown, call flows, and configuration issues. By limiting RTP size but not SIP size, you maximize useful long-term data retention.

Alternative Configuration: Time-Based Retention

If you need to keep data for a specific number of days (such as compliance requirements), you can use time-based retention parameters.

Important: Time-Based Retention Basics
Understanding Maximum vs Minimum:	`maxpoolsipdays` sets a maximum retention period for SIP files, not a guaranteed minimum. To ensure files are kept for at least the specified period, you must allocate sufficient disk space using `maxpoolsize` to accommodate the expected volume of data for that period.

Configuration Example: 14-Day SIP Retention

Scenario	Configuration	Behavior
Keep SIP for 14 days, limit RTP	Edit `/etc/voipmonitor.conf`: # Set maximum age for SIP files maxpoolsipdays = 14 maxpoolgraphdays = 14 # Set maximum age for RTP files (can be different) maxpoolrtpdays = 7 # OR set maximum size for RTP files maxpoolrtpsize = 102400 # Allocate sufficient disk space to accommodate 14 days of data maxpoolsize = 300000 # Ensure this is large enough for 14 days of SIP + graph data	SIP files are deleted after 14 days (maximum). RTP files are deleted after 7 days (if using `maxpoolrtpdays`) or when `maxpoolrtpsize` limit is reached. If `maxpoolsize` is reached before the 14-day period, data may be deleted regardless of age. Monitor disk usage to ensure `maxpoolsize` is sufficient.

Verifying Which Cleaning Rule is Active

To debug which cleaning rule is currently applied and confirm your configuration is working:

# Check systemd journal for clean triggers
journalctl -u voipmonitor --since='YYYY-MM-DD' | grep clean -i

The log messages indicate which rule triggered cleanup:

clean_maxpoolsize - Cleaning triggered by total size limit
clean_maxpooldays - Cleaning triggered by overall age limit
clean_maxpoolsipdays - Cleaning triggered by SIP age limit
clean_maxpoolrtpdays - Cleaning triggered by RTP age limit

If you see multiple triggers, VoIPmonitor applies the first limit that is reached, deleting data until all limits are satisfied.

Best Practice: Configure maxpoolsize with Disk Capacity Buffer

When setting maxpoolsize, configure a buffer below the total disk capacity. This ensures there is sufficient space during periods when the cleaning process is not active (such as when cleanspool_enable_fromto restricts cleaning to specific hours).

Important: Disk Capacity Buffer
Why a Buffer is Needed:	The cleaning process runs every 5 minutes, but `cleanspool_enable_fromto` may restrict cleaning to specific hours (e.g., 1-5 AM). During non-cleaning hours, data will continue to accumulate. Setting `maxpoolsize` close to total disk capacity risks filling the disk outside the active cleaning window.
Recommended Configuration:	Set `maxpoolsize` to approximately 90-95% of total disk capacity to create a buffer.
Example:	For a 7 TB disk: # 6.5 TB = 6656000 MB (approximately 93% of 7 TB) maxpoolsize = 6656000 # Restrict cleaning to 1-5 AM to avoid I/O impact during peak traffic cleanspool_enable_fromto = 1-5 This configuration provides a 0.5 TB buffer for data accumulation during the 19-hour cleaning-off window.

Manual File Deletion

Manual deletion of files from the spool directory is generally safe, but there are important considerations.

Is It Safe?

Scenario	Safe?	Notes
Deleting old directories (prior days/hours)	Yes	The index is recomputed automatically
Deleting current hour while service running	No risk to service, but may cause errors for active captures if file handles are still open
Deleting current MINUTE while recording in progress	Not recommended	May cause file descriptor errors temporarily

When you manually delete files:

In modern mode (cleanspool_use_files = yes): The files database table still contains records for deleted files. These orphaned records are cleaned up during the next cleanup cycle.
In legacy mode: The .cleanspool_cache is recomputed automatically during the next cleanup cycle.

Orphaned Database Records

When you manually delete PCAP files, the corresponding records in the MySQL database (CDR tables) remain. This means:

You will see calls in the GUI
Clicking "Play" or "Download" will show "File not found" errors

Manual Deletion: Database Records Remain
After deleting PCAP files	Database CDR records are NOT automatically deleted. Calls will appear in the GUI but show "File not found" for missing files.
Solution	To clean orphaned records, run the database cleaner: `voipmonitor --clean-db` or use the GUI "Maintenance" tab.

When Is Manual Deletion Recommended?

**Emergency Space Recovery:** If disk is 99% full and autoclean is not processing fast enough, manually deleting old directories (e.g., yesterday's folders) provides immediate space relief
**Compliance Requests:** Legal requests requiring immediate data removal without waiting for automated cleanup
**Targeted Cleanup:** Removing data for specific dates/time periods

Recovering from Index Issues

If the cleanspool index becomes corrupted or out of sync:

For Modern Mode (cleanspool_use_files = yes):

The files table can be rebuilt by rescanning the spool directory:

# Trigger a reindex via manager API
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
rm /tmp/vmsck

For Legacy Mode (.cleanspool_cache):

If the .cleanspool_cache file is deleted or corrupted:

# Same reindex command works for legacy mode
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
rm /tmp/vmsck

The reindexfiles command rescans the spool directory structure and rebuilds the index.

If your deployment uses the secured manager API with SSL/TLS, use the secured endpoint instead of the Unix socket method. See Manager API documentation for the secured API usage.

Alternative: Secured Manager API

Custom Autocleaning: One-Time Cleanup with Filters

The GUI provides a custom autocleaning feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.

Use Case: Cleaning Old Recordings for a Specific IP

After configuring capture rules to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.

Example scenario:

You configured a capture rule to discard RTP for IP 192.168.1.50
Only new calls will be affected by this rule
Existing recordings for this IP must be cleaned up manually

GUI cleanup steps:

Navigate to Settings > Custom Autocleaning
Create a new autocleaning rule
Set the criteria (e.g., "Delete RTP older than 1 day")
In the Common Filter section, specify the target IP address (192.168.1.50)
Save and apply the rule
Once the cleanup is complete, remove the autocleaning rule

This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.

Comparison with Global Retention

Feature	Custom Autocleaning	Global Retention
Scope	Targeted (specific IP, number, filter)	All calls
Purpose	One-time cleanup of existing data	Ongoing automated cleanup
Configuration	GUI with CDR filters	`maxpoolsize`, `maxpooldays`
Flexibility	High - can use any CDR filter criteria	Low - time/size only

Diagnosing Disk Space Usage

To properly configure retention limits, first analyze your actual disk usage:

Check total disk space

df -h

Check total spool directory size

du -hs /var/spool/voipmonitor

Analyze per-day utilization (helps identify growth patterns)

du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2

Example output interpretation:

# 80G    ./2024-12
# 120G   ./2024-11
# 150G   ./2024-10

This shows recent months consume 80-150 GB of data. Use this data to estimate appropriate size limits.

Understanding Directory Size Differences (SIP vs RTP Retention)

If you notice significant size differences between directories in your spool, this is usually expected behavior when you have different retention periods for SIP and RTP data.

Age of Directory	Contents	Size
0-30 days	Both SIP and RTP PCAP files	Large
30-90 days	SIP PCAP files only (RTP deleted)	Smaller
90+ days	None (both SIP and RTP deleted)	Empty or absent

This behavior is expected and by design when using different retention periods (e.g., maxpoolsipdays = 90 and maxpoolrtpdays = 30).

Why RTP Directories Exist After Disabling RTP Saving

If you have configured savertp = no or savertp = header but still see RTP subdirectories, this occurs because:

When savertcp = yes, RTCP (RTP Control Protocol) packets are saved in the rtp_YYYY-MM-DD-HH-MM.tar files
The RTP directory name refers to the broad UDP port range used for RTP/RTCP traffic
These files contain RTCP control traffic (receiver reports, sender reports) but not the full RTP audio payload

Configuration	Result	Expected Directory Size
`savertp = yes` + `savertcp = yes`	Full RTP audio + RTCP controls saved	Very large (GBs)
`savertp = no` + `savertcp = yes`	Only RTCP controls saved	Small (MBs)
`savertp = header` + `savertcp = yes`	RTP headers + RTCP controls saved	Small (MBs)

Filesystem Troubleshooting

Files Disappearing Faster Than Expected

If PCAP files are being deleted sooner than your maxpooldays setting suggests, check for these common causes:

Emergency Cleanup Triggers

Emergency cleanup can override your retention settings:

Parameter	Default	Triggers When
`autocleanspoolminpercent`	1	Disk usage reaches 99%
`autocleanmingb`	5	Free space falls below 5 GB

Diagnosis:

# Check disk usage
df -h /var/spool/voipmonitor

# Check emergency trigger settings
grep -E "autocleanspoolminpercent|autocleanmingb" /etc/voipmonitor.conf

Resolution:

# Increase emergency thresholds (allow more data before emergency cleanup)
autocleanspoolminpercent = 5   # Allow 95% usage before emergency
autocleanmingb = 20            # Trigger at 20 GB free instead of 5 GB

GUI Configuration Override

If mysqlloadconfig = yes (default), GUI sensor settings override voipmonitor.conf:

Diagnosis:

# Check if database config is enabled
grep mysqlloadconfig /etc/voipmonitor.conf

# Check GUI: Settings > Sensors > wrench icon > search "maxpool"

Resolution: Update settings via GUI, or set mysqlloadconfig = no to use file-based config only.

Diagnostic Checklist

Before applying fixes, run through this diagnostic checklist:

# 1. Check all retention-related settings
cat /etc/voipmonitor.conf | grep -E '^spooldir|^maxpool|^cleandatabase|^autoclean'

# 2. Measure total spool usage
du -hs /var/spool/voipmonitor

# 3. Analyze per-day usage patterns
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2

# 4. Check disk capacity
df -h

# 5. Check for emergency cleanup in logs
journalctl -u voipmonitor | grep -i "clean\|autoclean"

Spool Directory Filling Due to Database Performance

If /var/spool/voipmonitor/ is filling up rapidly after a disk swap, MySQL upgrade, or configuration change, the issue may be caused by database performance problems.

Root Cause: Slow Database Causing Query File Queuing
Symptoms	Spool directory fills exponentially, sometimes within hours, even with correct `maxpoolsize` settings.
Root Cause	MySQL cannot process database writes quickly enough, causing query files to queue.

Diagnosis:

# Watch for SQLq (SQL queue) metrics in real-time
tail -f /var/log/syslog | grep voipmonitor | grep SQLq

# Check MySQL InnoDB settings
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"

Solution - Tune MySQL:

[mysqld]
# 50-70% of server RAM
innodb_buffer_pool_size = 8G

# Fast writes (acceptable data loss risk on crash)
innodb_flush_log_at_trx_commit = 2

"low spool disk space" Message

This message indicates that your configured maxpoolsize exceeds the physical disk capacity. VoIPmonitor may automatically adjust maxpoolsize to a lower value.

Resolution:

# Check actual disk capacity
df -h /var/spool/voipmonitor

# Set realistic maxpoolsize (leave 5-10% buffer)
# For a 500 GB partition, use ~450 GB:
maxpoolsize = 460800

Missing Data Due to NFS/Storage Server Issues

If using remote storage (NFS, SSHFS), missing data may be caused by network connectivity issues rather than retention policies.

Diagnosis:

# Check system logs for NFS errors
grep -i "nfs" /var/log/syslog | grep "not responding\|timed out"

# Test connectivity
ping nfs-server.example.com
nc -zv nfs-server.example.com 2049

Tiered Storage and Archival Options

If you need to extend PCAP retention beyond your fast local disk capacity, there are three recommended approaches.

Option 1: Use tar_move Feature (Recommended)

The tar_move feature automatically archives PCAP files to secondary storage after they are closed.

# Use local fast storage for live capture
spooldir = /var/spool/voipmonitor

# Enable automatic archival
tar_move = yes
tar_move_destination_path = /mnt/archive/voipmonitor
tar_move_max_threads = 2

Parameter	Description
`tar_move = yes`	Move files to archive after capture completes
`tar_move = copy`	Copy files (keep original)
`tar_move_destination_path`	Target directory for archived files

Important: Files in tar_move_destination_path remain accessible via GUI. VoIPmonitor searches both local spooldir and tar_move_destination_path.

S3 Cloud Storage Considerations

When using S3 storage, use rclone instead of s3fs to avoid GUI unresponsiveness:

# Recommended: rclone mount
/usr/bin/rclone mount bucket-name /mnt/s3-archive \
  --allow-other --dir-cache-time 30s --poll-interval 0 \
  --vfs-cache-mode off --buffer-size 0 --use-server-modtime \
  --no-modtime --s3-no-head --log-level INFO

Option 2: Manual Archival with Additional Spool Directories

Move old files manually and configure GUI to access multiple directories:

Create archive script to move files older than X days
Add archive path to GUI: Settings > System configuration > Basic > Additional spool directories

Option 3: LVM Single Logical Volume

Combine fast SSD + large HDD into a single logical volume using LVM. VoIPmonitor sees only one spool directory.

Database Cleaning (CDR Retention)

Managing the size of the cdr table and other large tables is critical for GUI performance.

Partitioning Method (Recommended)

Since version 7, VoIPmonitor uses database partitioning, which splits large tables into smaller, daily segments.

Aspect	Description
How it works	Set `cleandatabase = 30` to keep the last 30 days of data
Why it's better	Dropping old partitions is instantaneous (milliseconds), regardless of row count
Partition limit	~8000 partitions per table (~22 years with daily partitioning)
CDR record limit	No practical limit - modern installations use `BIGINT` for `cdr.ID` (up to 18 quintillion records). See Migrating to BIGINT.

Quick Start: Global Retention

# Keep all records for 30 days
cleandatabase = 30

Retention Parameters

Parameter	Default	Description
`cleandatabase`	0 (disabled)	Master retention setting in days
`cleandatabase_cdr`	0	Specific retention for CDR tables
`cleandatabase_rtp_stat`	2	Retention for RTP statistics
`cleandatabase_sip_msg`	0	Retention for SIP messages
`cleandatabase_register_state`	0	Retention for registration states
`cleandatabase_register_time_info`	0	Must be set explicitly (not covered by global setting)
`cleandatabase_size`	(unset)	Size-based limit in MB
`cleandatabase_size_force`	false	Required for size-based cleanup
`partition_operations_enable_fromto`	1-5	Time window for partition operations

Important: register_time_info Table
Critical:	The `register_time_info` table is NOT covered by the global `cleandatabase` setting. You MUST configure `cleandatabase_register_time_info` explicitly.

Size-Based Database Cleaning

Use cleandatabase_size to limit database by size rather than time:

# Limit database to 50 GB
cleandatabase_size = 51200
cleandatabase_size_force = true

# Optional: Start cleaning when disk free space drops below threshold
cleandatabase_min_free_size = 30720

Multi-Sniffer Environments

When multiple sensors share the same database, partition operations should be managed by only ONE sensor:

# On all sniffers EXCEPT one:
disable_partition_operations = yes

# On the ONE designated sniffer:
partition_operations_enable_fromto = 4-6

Database Troubleshooting

MySQL Error 28: No Space Left on Device

If MySQL crashes with Error 28 even when cleandatabase is configured:

Primary Solution - Enable Size-Based Cleaning:

cleandatabase_size = 512000        # 500 GB limit
cleandatabase_size_force = true    # Required

Alternative Causes:

Inode exhaustion: Check with df -i
MySQL tmpdir full: Check with mysql -e "SHOW VARIABLES LIKE 'tmpdir';"

Disk Space Not Reclaimed After Cleanup

Check innodb_file_per_table setting:

SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';

If OFF, space is not reclaimed when partitions are dropped. Enable for future tables:

[mysqld]
innodb_file_per_table = 1

Database Not Cleaning (Verify Partitioning)

Before assuming cleaning is broken, verify tables are partitioned:

-- Check if CDR table is partitioned
SHOW CREATE TABLE cdr\G

-- Check partition list and row counts
SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'cdr'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;

If only expected partitions exist (matching your cleandatabase setting), cleaning IS working - you may simply have high data volume.

MySQL Performance Settings

For high-performance operation:

[mysqld]
# Use 50-70% of available RAM for caching
innodb_buffer_pool_size = 4G

# Flush logs to OS every second (faster, safe for VoIPmonitor)
innodb_flush_log_at_trx_commit = 2

# Enable per-table filespace for easy space reclamation
innodb_file_per_table = 1

AI Summary for RAG

Summary: VoIPmonitor has two independent data retention systems: (1) Filesystem cleaning for PCAP files using maxpoolsize/maxpooldays parameters, running every 5 minutes; (2) Database cleaning using cleandatabase with daily partitioning for instant partition drops.

Cleanspool modes: By default, VoIPmonitor uses legacy mode (cleanspool_use_files=no) with .cleanspool_cache filesystem index. This is because pcap_dump_tar=yes (TAR archive mode) is the default, which automatically disables database-indexed cleaning. To enable modern mode (database files table), explicitly set cleanspool_use_files=yes in config.

The maxpool_clean_obsolete parameter (default: no) controls whether files not in the index are deleted - when disabled (default), files outside the index are protected. Emergency cleanup (autocleanspoolminpercent=1, autocleanmingb=5) can override retention settings when disk is nearly full. GUI settings via mysqlloadconfig=yes override voipmonitor.conf. For tiered storage, use tar_move to archive to secondary storage. Database partition limit is ~8000 per table (~22 years); CDR record count uses BIGINT (18 quintillion limit). Size-based database cleaning requires cleandatabase_size AND cleandatabase_size_force=true.

Keywords: data retention, cleaning, maxpoolsize, maxpooldays, cleandatabase, cleanspool_use_files, maxpool_clean_obsolete, files table, .cleanspool_cache, pcap_dump_tar, autocleanspoolminpercent, autocleanmingb, emergency cleanup, tar_move, tiered storage, partitioning, cleandatabase_size, innodb_file_per_table, savertp header, packetbuffer_sender, BIGINT cdr.ID

Key Questions:

What is the default cleanspool mode and why?
How does legacy cleanspool (.cleanspool_cache) differ from modern mode (files table)?
What is cleanspool_use_files and how do I enable database-indexed cleaning?
Will cleanspool delete files not in the index/database?
What is maxpool_clean_obsolete?
Why are PCAP files being deleted faster than expected?
How do I fix MySQL Error 28?
How do I configure size-based database cleaning?
How do I extend retention with tiered storage?
Why is disk space not reclaimed after cleanup?
How do I check if database partitioning is working?
What is the maximum database size and CDR count limit?

Overview

Filesystem Cleaning (PCAP Spool Directory)

How Cleanspool Works: Modern vs Legacy Mode

Legacy Mode: Filesystem Cache (Default)

Modern Mode: Database-Indexed Cleaning

The maxpool_clean_obsolete Parameter

Reducing Data Collection at Source

Save Only RTP Headers (Major Space Saver)

Enabling Selective Audio Recording with Capture Rules

Spool Directory Structure

Spool Directory Location

Relocating the Spool Directory to a Different Partition

Procedure to Relocate Spooldir

Optional: Move Existing Data

Retention Mechanism

Retention Policy: maxpool* Parameters

Emergency Cleanup: autoclean* Parameters

Performance Warning: maxpoolaudiosize

Retention Configuration

Recommended Configuration Strategy: Mixed Size-Based Retention

Alternative Configuration: Time-Based Retention

Configuration Example: 14-Day SIP Retention

Verifying Which Cleaning Rule is Active

Best Practice: Configure maxpoolsize with Disk Capacity Buffer

Manual File Deletion

Is It Safe?

Orphaned Database Records

When Is Manual Deletion Recommended?

Recovering from Index Issues

Custom Autocleaning: One-Time Cleanup with Filters

Use Case: Cleaning Old Recordings for a Specific IP

Comparison with Global Retention

Diagnosing Disk Space Usage

Understanding Directory Size Differences (SIP vs RTP Retention)

Why RTP Directories Exist After Disabling RTP Saving

Filesystem Troubleshooting

Files Disappearing Faster Than Expected

Emergency Cleanup Triggers

GUI Configuration Override

Diagnostic Checklist

Spool Directory Filling Due to Database Performance

"low spool disk space" Message

Missing Data Due to NFS/Storage Server Issues

Tiered Storage and Archival Options

Option 1: Use tar_move Feature (Recommended)

S3 Cloud Storage Considerations

Option 2: Manual Archival with Additional Spool Directories

Option 3: LVM Single Logical Volume

Database Cleaning (CDR Retention)

Partitioning Method (Recommended)

Quick Start: Global Retention

Retention Parameters

Size-Based Database Cleaning

Multi-Sniffer Environments

Database Troubleshooting

MySQL Error 28: No Space Left on Device

Disk Space Not Reclaimed After Cleanup

Database Not Cleaning (Verify Partitioning)

MySQL Performance Settings

See Also

AI Summary for RAG