Data Cleaning: Difference between revisions

From VoIPmonitor.org
(Clarify that cleanspool operates on filesystem, not database)
(Add custom autocleaning section with IP filter cleanup workflow)
Line 441: Line 441:


For the complete parameter reference, see [[Sniffer_configuration#Spool_Cleaning|Sniffer Configuration - Spool Cleaning]].
For the complete parameter reference, see [[Sniffer_configuration#Spool_Cleaning|Sniffer Configuration - Spool Cleaning]].
=== Custom Autocleaning: One-Time Cleanup with Filters ===
The GUI provides a '''custom autocleaning''' feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.
==== Use Case: Cleaning Old Recordings for a Specific IP ===
After configuring [[Capture_rules|capture rules]] to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.
Example scenario:
* You configured a capture rule to discard RTP for IP <code>192.168.1.50</code>
* Only new calls will be affected by this rule
* Existing recordings for this IP must be cleaned up manually
GUI cleanup steps:
# Navigate to '''Settings > Custom Autocleaning'''
# Create a new autocleaning rule
# Set the criteria (e.g., "Delete RTP older than 1 day")
# In the '''Common Filter''' section, specify the target IP address (<code>192.168.1.50</code>)
# Save and apply the rule
# Once the cleanup is complete, remove the autocleaning rule
This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.
==== Comparison with Global Retention ===
{| class="wikitable"
|-
! Feature !! Custom Autocleaning !! Global Retention
|-
| '''Scope''' || Targeted (specific IP, number, filter) || All calls
|-
| '''Purpose''' || One-time cleanup of existing data || Ongoing automated cleanup
|-
| '''Configuration''' || GUI with CDR filters | <code>maxpoolsize</code>, <code>maxpooldays</code>
|-
| '''Flexibility''' || High - can use any CDR filter criteria | Low - time/size only
|}
==== Filters Available in Custom Autocleaning ===
The custom autocleaning interface supports the same filter options as the CDR view, including:
* '''IP filters''' - Source or destination IP addresses
* '''Telephone number filters''' - Caller or called numbers
* '''Time-based filters''' - Calls older than X days
* '''Other CDR criteria''' - Duration, MOS, packet loss, etc.
For details on using these filters, see [[Call_Detail_Record_-_CDR#Filtering_Calls|CDR Filtering]].
==== Important Notes ===
* Custom autocleaning rules should typically be deleted after they have run once
* Leaving a rule active will continuously clean up matching data, which may not be desired
* The cleanup affects PCAP files stored on disk, not CDR records in the database
* For database cleanup, use the global <code>cleandatabase</code> parameters


=== How Cleanspool Works: Filesystem Index vs Database ===
=== How Cleanspool Works: Filesystem Index vs Database ===
Line 713: Line 769:


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' VoIPmonitor has two independent data retention mechanisms: (1) Filesystem cleaning for PCAP files using <code>maxpoolsize</code>/<code>maxpooldays</code> parameters, running every 5 minutes to delete oldest data first; (2) Database cleaning using <code>cleandatabase</code> parameter with daily partitioning for instant partition drops. Key behavior: directories within <code>maxpoolrtpdays</code> window contain both SIP and RTP (large), while directories within <code>maxpoolsipdays</code> but beyond <code>maxpoolrtpdays</code> contain only SIP (smaller) - this is expected. Cleanspool operates on the filesystem, NOT the database. The cleanspool process maintains its own index in <code>.cleanspool_cache</code> file and deletes files based solely on this filesystem index and <code>maxpoolsize</code>/<code>maxpooldays</code> limits. Files are deleted even if they are not in the database. The <code>maxpool_clean_obsolete</code> parameter (default: no) controls whether cleanspool deletes files not found in the cache. Troubleshooting covers disk full scenarios (check with <code>du -h --max-depth=1</code>, increase <code>maxpoolsize</code>), space not reclaimed issues (<code>innodb_file_per_table</code> setting), and database health monitoring (SQLq/SQLf metrics). Additional topics: relocating spooldir to different partition, syncing GUI <code>SNIFFER_DATA_PATH</code> configuration, size-based database cleaning via <code>cleandatabase_size</code> and <code>cleandatabase_size_force</code>, saving only RTP headers with <code>savertp=header</code> (up to 90% storage reduction), critical architecture consideration for <code>packetbuffer_sender</code> mode (apply settings on central server, not sensors). Emergency cleanup triggers: <code>autocleanspoolminpercent</code> (default 1%, triggers at 99% disk usage) and <code>autocleanmingb</code> (default 5GB, triggers when free space below 5GB) can override retention settings, causing premature file deletion even when <code>maxpooldays</code> is set correctly.
'''Summary:''' VoIPmonitor has multiple data retention mechanisms: (1) Filesystem cleaning for PCAP files using <code>maxpoolsize</code>/<code>maxpooldays</code> parameters, running every 5 minutes to delete oldest data first; (2) Database cleaning using <code>cleandatabase</code> parameter with daily partitioning for instant partition drops; (3) Custom autocleaning in GUI for one-time targeted cleanup of specific recordings based on filters (IP, phone number, duration, etc.). Key behavior: directories within <code>maxpoolrtpdays</code> window contain both SIP and RTP (large), while directories within <code>maxpoolsipdays</code> but beyond <code>maxpoolrtpdays</code> contain only SIP (smaller) - this is expected. Custom autocleaning is useful for cleaning up old recordings for specific IPs after configuring capture rules to stop future recording (capture rules only affect new calls, existing recordings must be cleaned separately). Cleanspool operates on the filesystem, NOT the database. The cleanspool process maintains its own index in <code>.cleanspool_cache</code> file and deletes files based solely on this filesystem index and <code>maxpoolsize</code>/<code>maxpooldays</code> limits. Files are deleted even if they are not in the database. The <code>maxpool_clean_obsolete</code> parameter (default: no) controls whether cleanspool deletes files not found in the cache. Troubleshooting covers disk full scenarios (check with <code>du -h --max-depth=1</code>, increase <code>maxpoolsize</code>), space not reclaimed issues (<code>innodb_file_per_table</code> setting), and database health monitoring (SQLq/SQLf metrics). Additional topics: relocating spooldir to different partition, syncing GUI <code>SNIFFER_DATA_PATH</code> configuration, size-based database cleaning via <code>cleandatabase_size</code> and <code>cleandatabase_size_force</code>, saving only RTP headers with <code>savertp=header</code> (up to 90% storage reduction), critical architecture consideration for <code>packetbuffer_sender</code> mode (apply settings on central server, not sensors). Emergency cleanup triggers: <code>autocleanspoolminpercent</code> (default 1%, triggers at 99% disk usage) and <code>autocleanmingb</code> (default 5GB, triggers when free space below 5GB) can override retention settings, causing premature file deletion even when <code>maxpooldays</code> is set correctly.


'''Keywords:''' data retention, cleaning, delete old calls, disk space, spooldir, maxpoolsize, maxpooldays, maxpoolsipdays, maxpoolrtpdays, cleandatabase, partitioning, cleandatabase_size, cleandatabase_size_force, reindexfiles, innodb_file_per_table, SQLq, SQLf, directory size difference, SIP vs RTP retention, relocate spooldir, SNIFFER_DATA_PATH, MySQL Error 28, savertp header, packetbuffer_sender, client server mode, distributed architecture, autocleanspoolminpercent, autocleanmingb, emergency cleanup, retention not working, premature deletion, cleanspool_cache, maxpool_clean_obsolete, filesystem index, database vs filesystem, orphaned files
'''Keywords:''' data retention, cleaning, delete old calls, disk space, spooldir, maxpoolsize, maxpooldays, maxpoolsipdays, maxpoolrtpdays, cleandatabase, partitioning, cleandatabase_size, cleandatabase_size_force, reindexfiles, innodb_file_per_table, SQLq, SQLf, directory size difference, SIP vs RTP retention, relocate spooldir, SNIFFER_DATA_PATH, MySQL Error 28, savertp header, packetbuffer_sender, client server mode, distributed architecture, autocleanspoolminpercent, autocleanmingb, emergency cleanup, retention not working, premature deletion, cleanspool_cache, maxpool_clean_obsolete, filesystem index, database vs filesystem, orphaned files, custom autocleaning, one-time cleanup, targeted cleanup, IP filter cleanup, clean specific IP recordings


'''Key Questions:'''
'''Key Questions:'''
Line 740: Line 796:
* How does cleanspool filesystem index differ from database records?
* How does cleanspool filesystem index differ from database records?
* Will cleanspool delete files that are not indexed in the database?
* Will cleanspool delete files that are not indexed in the database?
* How do I clean up old recordings for a specific IP address?
* What is custom autocleaning in VoIPmonitor GUI?
* How do I delete old recordings after configuring capture rules to stop recording?
* Can I clean up recordings based on specific filters like IP or phone number?
* What is the difference between custom autocleaning and global retention settings?

Revision as of 08:19, 5 January 2026

This guide explains how VoIPmonitor manages data retention for both captured packets (PCAP files) and Call Detail Records (CDRs) in the database. Proper configuration is essential for managing disk space and maintaining long-term database performance.

Overview

VoIPmonitor generates two primary types of data that require periodic cleaning:

  • PCAP Files: Raw packet captures of SIP/RTP/GRAPH data stored on the filesystem in the spool directory. These can consume significant disk space.
  • CDR Data: Call metadata stored in the MySQL database. Large tables can slow down GUI performance if not managed properly.

The system uses two separate, independent mechanisms to manage the retention of this data:

Filesystem Cleaning (PCAP Spool Directory)

The sensor stores captured call data in a structured directory tree on the local filesystem.

Reducing Data Collection at Source

Before configuring cleanup policies, consider reducing the amount of data captured. This is often the most effective long-term solution for storage management.

Save Only RTP Headers (Major Space Saver)

RTP packets typically contain the full audio payload, which consumes the majority of disk space. If you only need call quality statistics (MOS, jitter, packet loss) and not actual audio playback, switch to saving RTP headers only.

Edit /etc/voipmonitor.conf:

# Change from full RTP to headers only
savertp = header
Setting Storage Impact Use Case
savertp = yes High (~10x more) Requires ability to play back audio from PCAPs
savertp = header Low Only CDR statistics needed, no audio playback required

With savertp = header, VoIPmonitor still captures all necessary metadata for MOS scoring, jitter analysis, packet loss statistics, and quality graphs, but does not store the actual audio payload. This can reduce storage consumption by up to 90%.

Important: After changing from savertp = yes to savertp = header, existing PCAP files will remain playable. New calls will only contain RTP headers.

Critical: Distributed Architecture Consideration
packetbuffer_sender Architecture:

If you are using the Packet Mirroring/Client-Server mode (packetbuffer_sender=yes on remote sensors), the savertp setting must be applied on the central server where packet analysis is performed, not on the individual sensors.

Architecture Where to Apply savertp = header
packetbuffer_sender = no (Local Processing) On each sensor/probe's configuration file
packetbuffer_sender = yes (Packet Mirroring) On the central server's configuration file (not on sensors)

For distributed deployment details, see Client-Server Architecture.

For more configuration options, see Sniffer Configuration - Saving Options.

Spool Directory Location

By default, all data is stored in /var/spool/voipmonitor. This location can be changed by setting the spooldir option in voipmonitor.conf.

Relocating the Spool Directory to a Different Partition

If your default partition is running out of space, you can move the spool directory to a larger partition or dedicated disk. This is particularly useful when MySQL Error 28 ("No space left on device") occurs despite retention settings being in place.

Situation Recommended Action
/var/lib/mysql full but other partitions available Move spool to larger partition and update spooldir in voipmonitor.conf
Multiple disks available Use dedicated partition for PCAP storage
GUI on same server Sync GUI SNIFFER_DATA_PATH to match new spooldir
Procedure to Relocate Spooldir
Step 1
Identify Available Space

Check all partitions to find a suitable location:

df -h
Step 2
Create New Spool Directory

Create the directory on the destination partition:

# Example: Use /mnt/pcaps partition
mkdir -p /mnt/pcaps/voipmonitor

# Set correct ownership for both sniffer and GUI access
chown voipmonitor:voipmonitor /mnt/pcaps/voipmonitor
chmod 755 /mnt/pcaps/voipmonitor
Step 3
Update VoIPmonitor Configuration

Edit /etc/voipmonitor.conf to point to the new location:

[general]
spooldir = /mnt/pcaps/voipmonitor
Step 4
Update GUI Configuration (Critical)

If the GUI runs on the same server, you MUST update the path the GUI uses to access the spool. This is defined in the GUI configuration file:

  • Debian/Ubuntu: /var/www/html/voipmonitor/config/configuration.php
  • RHEL/CentOS: /var/www/voipmonitor/config/configuration.php

Edit the GUI configuration file:

// Find and update this line to match your new spooldir
define('SNIFFER_DATA_PATH', '/mnt/pcaps/voipmonitor');

Important: The SNIFFER_DATA_PATH in the GUI config MUST match the spooldir setting in voipmonitor.conf. If these are mismatched, the GUI will be unable to locate and display call recordings.

Step 5
Restart Services

Apply all changes by restarting the services:

# Restart the VoIPmonitor sensor
systemctl restart voipmonitor

# If using Apache/Nginx, reload web server
systemctl reload apache2    # Debian/Ubuntu
# or
systemctl reload nginx       # RHEL/CentOS
Step 6
Verify Configuration
# Check if new spool directory is being used
ls -ld /mnt/pcaps/voipmonitor

# Verify GUI can access the path
grep SNIFFER_DATA_PATH /path/to/gui/config/configuration.php

# Check sensor logs for any spool-related errors
tail -f /var/log/voipmonitor.log
Optional: Move Existing Data

If you want to migrate existing PCAP files to the new location:

# Stop the sensor to prevent writes during migration
systemctl stop voipmonitor

# Move existing data (this may take a long time)
mv /var/spool/voipmonitor/* /mnt/pcaps/voipmonitor/

# Restart the sensor
systemctl start voipmonitor

Retention Configuration

The cleaning process runs automatically every 5 minutes and removes the oldest data based on the rules you define in voipmonitor.conf. You can set limits based on total size (in Megabytes) or age (in days). If both a size and day limit are set for the same data type, the first limit that is reached will trigger the cleaning.

Parameter Default Value Description
maxpoolsize 102400 (100 GB) The total maximum disk space for all captured data (SIP, RTP, GRAPH, AUDIO).
maxpooldays (unset) The maximum number of days to keep all captured data.
maxpoolsipsize (unset) A specific size limit for SIP PCAP files only.
maxpoolsipdays (unset) A specific age limit for SIP PCAP files only.
maxpoolrtpsize (unset) A specific size limit for RTP PCAP files only.
maxpoolrtpdays (unset) A specific age limit for RTP PCAP files only.
maxpoolgraphsize (unset) A specific size limit for GRAPH files only.
maxpoolgraphdays (unset) A specific age limit for GRAPH files only.
maxpoolaudiosize (unset) A specific size limit for converted audio files (WAV/OGG) only.
maxpoolaudiodays (unset) An age limit for converted audio files (WAV/OGG) only.

Understanding Directory Size Differences (SIP vs RTP Retention)

If you configure different retention periods for SIP and RTP data (for example, maxpoolsipdays = 90 and maxpoolrtpdays = 30), you will notice a significant size difference between directories within different time windows.

Age of Directory Contents Size
0-30 days Both SIP and RTP PCAP files Large
30-90 days SIP PCAP files only (RTP deleted) Smaller
90+ days None (both SIP and RTP deleted) Empty or absent

This behavior is expected and by design:

  • Directories within the maxpoolrtpdays retention window contain both SIP and RTP data, making them significantly larger (often 5-10x larger).
  • Directories older than maxpoolrtpdays but within maxpoolsipdays contain only SIP data, so they are much smaller.
  • The automatic cleanup process removes RTP files after maxpoolrtpdays and SIP files after maxpoolsipdays.

This is not an error or configuration issue. It is the expected result of having different retention periods for different data types.

Diagnosis: Compare Directory Sizes

To verify this behavior, check your spool directory:

cd /var/spool/voipmonitor

# Show directories sorted by size
du -h --max-depth=1 ./ | sort -rh | head -20

# Example output:
# 80G    ./2025-01          # Current month (has SIP+RTP)
# 15G    ./2024-12          # Previous month (SIP only, RTP deleted)
# 120G   .

Compare with your configuration:

grep -E "maxpoolsip|maxpoolrtp" /etc/voipmonitor.conf

# Example configuration:
# maxpoolsipdays = 90     # SIP kept for 90 days
# maxpoolrtpdays = 30     # RTP kept for 30 days

If the size difference matches your retention configuration, this is expected behavior and no action is needed.

Troubleshooting: Disk Full / Files Disappearing

If you see errors when attempting to extract older calls from the GUI, or if call files are disappearing too quickly, your spool directory may have reached its size limit.

Diagnosis: Check Disk Usage

  1. Identify the sensor/probe responsible for the missing data.
  2. SSH into the sensor/probe and navigate to the spooldir.
  3. Check the disk usage:
cd /var/spool/voipmonitor
du -h --max-depth=1 ./

# Example output:
# 150G    ./2025-01
# 120G    ./2024-12
# 90G     ./2024-11
# 360G    .
  1. Compare with the configured limit:
grep maxpoolsize /etc/voipmonitor.conf
# Example output: maxpoolsize = 102400  (100 GB in MB)

Resolution: Increase Spooldir Size

If the actual usage exceeds the configured limit, increase maxpoolsize:

# Edit /etc/voipmonitor.conf
[general]
maxpoolsize = 716800   # 700 GB in MB
maxpooldays = 90       # Optional: Keep data for last 90 days

Apply changes:

systemctl restart voipmonitor

Emergency Cleanup Triggers

Even when your maxpooldays setting appears correct, PCAP files may still be deleted prematurely due to emergency cleanup triggers. These are safety mechanisms that override normal retention settings to prevent system crashes from disk exhaustion.

Common Cause: Retention Not Working as Expected
Symptom: You set maxpooldays = 60, but files are deleted after 2 weeks (14 days).
Root Cause: Emergency cleanup triggered by autocleanspoolminpercent or autocleanmingb.

Emergency cleanup is activated when either of these conditions is met:

Parameter Default Description Behavior
autocleanspoolminpercent 1 When the partition reaches this threshold (e.g., 99% full), aggressive cleaning starts immediately regardless of maxpooldays setting.
autocleanmingb 5 When free space falls below this value (e.g., less than 5 GB left), emergency cleaning is triggered.
How Emergency Cleanup Works

The cleaning process checks disk space before each operation:

# Check current disk usage
df -h /var/spool/voipmonitor

# Example output:
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sdb1       2.0T  1.96T  40G  98% /var/spool/voipmonitor

If Use%</code exceeds 100 - autocleanspoolminpercent (e.g., 99%), emergency cleanup is triggered.

Emergency cleanup behavior:

  • Files are deleted starting from the oldest until the emergency threshold is cleared
  • This happens regardless of your maxpooldays or maxpoolsize settings
  • The normal cleaning cycle (every 5 minutes) is triggered immediately
  • Once the threshold is cleared, normal retention rules resume
Diagnosis: Check Emergency Cleanup Triggers
# Check if emergency triggers are set
grep -E "autocleanspoolminpercent|autocleanmingb" /etc/voipmonitor.conf

# Check current disk usage
df -h /var/spool/voipmonitor

# Check available space on the partition
# If Avail is close to autocleanmingb (5GB), emergency cleanup may be triggering
Resolution: Adjust Emergency Triggers or Increase Disk Space

If emergency cleanup is causing premature deletion, you have two options:

Option 1
Adjust Emergency Triggers

Increase the thresholds to allow more data before emergency cleanup triggers:

# Edit /etc/voipmonitor.conf
[general]
maxpooldays = 60

# Reduce emergency sensitivity (allow partition to fill more before trigger)
autocleanspoolminpercent = 5   # Default is 1, increase to 5 to allow 95% usage
autocleanmingb = 20           # Default is 5, increase to 20 GB

Apply changes:

systemctl restart voipmonitor
Option 2
Increase Disk Space

If the partition is genuinely full, add more storage or move the spooldir to a larger partition (see Relocating Spooldir).

Example Troubleshooting Scenario

User report: "I set maxpooldays = 60 for 60-day retention, but PCAPs are deleted after only 14 days."

Investigation:

# 1. Check retention settings
grep maxpooldays /etc/voipmonitor.conf
# maxpooldays = 60  ✓ Correct

# 2. Check disk usage
df -h /var/spool/voipmonitor
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sdb1       500G  498G  2G  99.6% /var/spool/voipmonitor

# 3. Check emergency triggers
grep autoclean /etc/voipmonitor.conf
# autocleanspoolminpercent = 1 (default)
# autocleanmingb = 5 (default)

Root Cause:

  • Partition is 99.6% full, which exceeds the 99% threshold (100 - 1)
  • Only 2 GB free space remains, below the 5 GB autocleanmingb limit
  • Emergency cleanup triggers, deleting files until at least 5 GB is free

Solution:

  • Increase autocleanspoolminpercent = 5 to allow 95% usage before emergency trigger
  • Or increase autocleanmingb = 20 to allow more data before cleanup
  • Or add more disk space to the partition

For the complete parameter reference, see Sniffer Configuration - Spool Cleaning.

Custom Autocleaning: One-Time Cleanup with Filters

The GUI provides a custom autocleaning feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.

= Use Case: Cleaning Old Recordings for a Specific IP

After configuring capture rules to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.

Example scenario:

  • You configured a capture rule to discard RTP for IP 192.168.1.50
  • Only new calls will be affected by this rule
  • Existing recordings for this IP must be cleaned up manually

GUI cleanup steps:

  1. Navigate to Settings > Custom Autocleaning
  2. Create a new autocleaning rule
  3. Set the criteria (e.g., "Delete RTP older than 1 day")
  4. In the Common Filter section, specify the target IP address (192.168.1.50)
  5. Save and apply the rule
  6. Once the cleanup is complete, remove the autocleaning rule

This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.

= Comparison with Global Retention

Feature Custom Autocleaning Global Retention
Scope Targeted (specific IP, number, filter) All calls
Purpose One-time cleanup of existing data Ongoing automated cleanup
Configuration maxpoolsize, maxpooldays
Flexibility Low - time/size only

= Filters Available in Custom Autocleaning

The custom autocleaning interface supports the same filter options as the CDR view, including:

  • IP filters - Source or destination IP addresses
  • Telephone number filters - Caller or called numbers
  • Time-based filters - Calls older than X days
  • Other CDR criteria - Duration, MOS, packet loss, etc.

For details on using these filters, see CDR Filtering.

= Important Notes

  • Custom autocleaning rules should typically be deleted after they have run once
  • Leaving a rule active will continuously clean up matching data, which may not be desired
  • The cleanup affects PCAP files stored on disk, not CDR records in the database
  • For database cleanup, use the global cleandatabase parameters

How Cleanspool Works: Filesystem Index vs Database

It is important to understand that the cleanspool feature operates on the filesystem, not the database. This is a critical distinction for configuring data retention correctly.

Filesystem-Based Cleaning

The cleanspool process maintains its own index in the .cleanspool_cache file located in the spool directory. This filesystem index is completely independent of the VoIPmonitor database.

Aspect Cleanspool (Filesystem) Database Cleaning
Where it operates Deletes PCAP files from disk Deletes CDR records from MySQL
Index used .cleanspool_cache (filesystem scan) Database tables/partitions
Retention limits maxpoolsize, maxpooldays cleandatabase, cleandatabase_size
File selection criteria Based on filesystem index, NOT database records Based on calldate in cdr table

Key Behavior: Files Not in Database

The cleanspool process will delete files based solely on the filesystem cache index and the maxpoolsize/maxpooldays limits. This means:

  • Files will be deleted even if they are not in the database (orphaned PCAPs)
  • Files will be deleted even if database records still exist (e.g., after database backup/restore)
  • To protect specific data, you must move it outside the spool directory entirely, not just to a different subfolder

The .cleanspool_cache File

When cleanspool runs, it scans the spool directory structure (YYYY-MM-DD/HH) and indexes all PCAP files into .cleanspool_cache. During the next cleaning cycle, it uses this cache to:

1. Sort files by age (oldest first) 2. Delete files until maxpoolsize or maxpooldays limits are met

Important behaviors:

  • By default (maxpool_clean_obsolete = no), cleanspool only deletes files that are indexed in .cleanspool_cache
  • Files that appear but are not in the cache are ignored until the next full scan updates the cache
  • With maxpool_clean_obsolete = yes, cleanspool will also delete any files in the spool directory that are not in the cache
Important: Protect Data Moves
maxpool_clean_obsolete = no (default):

If you manually move files into the spool directory, they will not be indexed immediately. Cleanspool will ignore them until a full scan updates the cache. To prevent deletion, move data outside the spool directory entirely, not to a different subfolder within the spool.

Configuration Parameter Reference

maxpool_clean_obsolete = no (default)
Cleanspool only deletes files that are indexed in the .cleanspool_cache filesystem index. Files that are not in the cache are ignored.
maxpool_clean_obsolete = yes
Cleanspool will delete ALL files in the spool directory, including those not found in the cache. This is useful if you have manually moved files or want to clean orphaned files.

Warning: This parameter controls filesystem cache behavior, not database record matching. Cleanspool always operates independently of the database.

Maintenance: Re-indexing the Spool Directory

VoIPmonitor maintains an index of all created PCAP files to perform cleaning efficiently without scanning the entire directory tree. If this index becomes corrupt, or if you manually move files into the spool, old data may not be deleted correctly.

To trigger a manual re-index via the manager API:

# Open a manager API session
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029

# Send the re-index command
echo reindexfiles | nc -U /tmp/vmsck

Note: This command requires netcat with support for UNIX sockets (-U). For alternative methods, see the Manager API documentation.

Database Cleaning (CDR Retention)

Managing the size of the cdr table and other large tables is critical for GUI performance.

Partitioning Method (Recommended)

Since version 7, VoIPmonitor utilizes database partitioning, which splits large tables into smaller, daily segments. This is the recommended method for managing database retention.

Aspect Description
How it works Set cleandatabase = 30 in voipmonitor.conf to keep the last 30 days of data.
Why it's better Dropping old partitions is instantaneous (milliseconds), regardless of row count. Zero database load.
Requirement Partitioning is enabled by default on new installations.

Quick Start: Global Retention

For most deployments, configure one parameter in voipmonitor.conf:

# Keep all records for 30 days
cleandatabase = 30

The cleandatabase parameter acts as a global default for all cleandatabase_* options and applies to:

  • cdr - Call Detail Records
  • message - SIP MESSAGE texts
  • sip_msg - SIP OPTIONS/SUBSCRIBE/NOTIFY messages
  • register_state - SIP registration states
  • register_failed - Failed registration attempts

Retention Parameters

Parameter Default Description
cleandatabase 0 (disabled) Master retention setting in days.
cleandatabase_cdr 0 Specific retention for cdr and message tables.
cleandatabase_rtp_stat 2 Retention for detailed RTP statistics.
cleandatabase_sip_msg 0 Retention for OPTIONS/SUBSCRIBE/NOTIFY.
cleandatabase_size (unset) Alternative: size-based limit in MB (requires version 2024.05.1+).
partition_operations_enable_fromto 1-5 Time window for partition operations (e.g., 1-5 AM).
Size-Based Database Cleaning

In addition to time-based retention, VoIPmonitor supports size-based database cleanup using cleandatabase_size. This is useful when you want to limit the database size regardless of data age.

# Limit database to 50 GB
cleandatabase_size = 51200

# Force immediate cleanup if size exceeded (required for size-based)
cleandatabase_size_force = true
Parameter Description
cleandatabase_size Maximum database size in MB. When exceeded, the oldest partitions are dropped until the size is below this limit.
cleandatabase_size_force Must be set to true to enable size-based cleanup. Without this, cleandatabase_size will not trigger cleanup.

Behavior:

  • The size check is performed daily during partition operations
  • When the size exceeds cleandatabase_size + 5% buffer, the oldest partitions are dropped
  • This ensures the database stays within the specified size limit
  • Works independently of cleandatabase (time-based retention)

Note: Size-based cleanup only affects partitioned tables. Non-partitioned tables are not affected.

More details: Sniffer Configuration - Database Cleaning.

Legacy Method: Manual Deletion (Not Recommended)

For very old, non-partitioned databases, you would need custom scripts with DELETE FROM cdr WHERE calldate < ... queries.

Warning: Manual DELETE on large tables is extremely slow and resource-intensive. A single operation on millions of rows can take hours and impact GUI performance.

Troubleshooting Disk Space Issues

Disk Space Not Reclaimed After Cleanup

If automatic cleanup runs but disk space is not freed from the MySQL data directory, check the innodb_file_per_table setting:

SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';
Value Behavior
ON Each table/partition has its own .ibd file. Dropping partitions reclaims space immediately.
OFF All data in shared ibdata1 file. Dropping partitions does not reduce file size.

Solutions

Option 1
Enable for Future Tables

Add to /etc/my.cnf or /etc/mysql/my.cnf:

[mysqld]
innodb_file_per_table = 1
systemctl restart mysql

Note: This only affects NEW tables/partitions. Existing data in ibdata1 remains.

Option 2
Reclaim Space from Existing Tables
OPTIMIZE TABLE cdr;

Warning: Requires significant free disk space to duplicate table data. May crash if disk is nearly full.

Option 3
Export and Re-import
mysqldump -u root -p voipmonitor > voipmonitor_backup.sql
mysql -u root -p -e "DROP DATABASE voipmonitor; CREATE DATABASE voipmonitor;"
mysql -u root -p voipmonitor < voipmonitor_backup.sql

Monitoring Database Health

SQL Queue Metrics

The sensor tracks queue metrics visible in GUI → Settings → Sensors → Status:

Metric Description Healthy Range
SQLq CDRs waiting to be written to database Near 0, sporadic spikes OK
SQLf Failed database write attempts Zero (not growing)
  • Consistently high/growing SQLq → database cannot keep up
  • Non-zero/growing SQLf → database errors or connectivity issues

See SQL Queue Troubleshooting for details.

System Monitoring

# Check system load
cat /proc/loadavg

# Monitor disk I/O (shows only active processes)
iotop -o

High I/O from mysqld processes may indicate slow storage or poorly tuned MySQL settings.

MySQL Performance Settings

For high-performance operation with partitioning:

[mysqld]
# Use 50-70% of available RAM for caching
innodb_buffer_pool_size = 4G

# Flush logs to OS every second (faster, safe for VoIPmonitor)
innodb_flush_log_at_trx_commit = 2

# Enable per-table filespace for easy space reclamation
innodb_file_per_table = 1

For comprehensive tuning, see Scaling and Performance Guide.

See Also

AI Summary for RAG

Summary: VoIPmonitor has multiple data retention mechanisms: (1) Filesystem cleaning for PCAP files using maxpoolsize/maxpooldays parameters, running every 5 minutes to delete oldest data first; (2) Database cleaning using cleandatabase parameter with daily partitioning for instant partition drops; (3) Custom autocleaning in GUI for one-time targeted cleanup of specific recordings based on filters (IP, phone number, duration, etc.). Key behavior: directories within maxpoolrtpdays window contain both SIP and RTP (large), while directories within maxpoolsipdays but beyond maxpoolrtpdays contain only SIP (smaller) - this is expected. Custom autocleaning is useful for cleaning up old recordings for specific IPs after configuring capture rules to stop future recording (capture rules only affect new calls, existing recordings must be cleaned separately). Cleanspool operates on the filesystem, NOT the database. The cleanspool process maintains its own index in .cleanspool_cache file and deletes files based solely on this filesystem index and maxpoolsize/maxpooldays limits. Files are deleted even if they are not in the database. The maxpool_clean_obsolete parameter (default: no) controls whether cleanspool deletes files not found in the cache. Troubleshooting covers disk full scenarios (check with du -h --max-depth=1, increase maxpoolsize), space not reclaimed issues (innodb_file_per_table setting), and database health monitoring (SQLq/SQLf metrics). Additional topics: relocating spooldir to different partition, syncing GUI SNIFFER_DATA_PATH configuration, size-based database cleaning via cleandatabase_size and cleandatabase_size_force, saving only RTP headers with savertp=header (up to 90% storage reduction), critical architecture consideration for packetbuffer_sender mode (apply settings on central server, not sensors). Emergency cleanup triggers: autocleanspoolminpercent (default 1%, triggers at 99% disk usage) and autocleanmingb (default 5GB, triggers when free space below 5GB) can override retention settings, causing premature file deletion even when maxpooldays is set correctly.

Keywords: data retention, cleaning, delete old calls, disk space, spooldir, maxpoolsize, maxpooldays, maxpoolsipdays, maxpoolrtpdays, cleandatabase, partitioning, cleandatabase_size, cleandatabase_size_force, reindexfiles, innodb_file_per_table, SQLq, SQLf, directory size difference, SIP vs RTP retention, relocate spooldir, SNIFFER_DATA_PATH, MySQL Error 28, savertp header, packetbuffer_sender, client server mode, distributed architecture, autocleanspoolminpercent, autocleanmingb, emergency cleanup, retention not working, premature deletion, cleanspool_cache, maxpool_clean_obsolete, filesystem index, database vs filesystem, orphaned files, custom autocleaning, one-time cleanup, targeted cleanup, IP filter cleanup, clean specific IP recordings

Key Questions:

  • How do I automatically delete old PCAP files?
  • What is the difference between maxpoolsize and maxpooldays?
  • My spool directory is full but old files are not deleted - how to fix?
  • How do I configure database retention with cleandatabase?
  • Why is disk space not reclaimed after MySQL cleanup?
  • What do SQLq and SQLf metrics mean?
  • Why are recent directories much larger than old ones in my spool?
  • Directory size difference with maxpoolsipdays and maxpoolrtpdays
  • How do I move the spooldir to a larger partition?
  • What is cleandatabase_size and cleandatabase_size_force?
  • How do I fix MySQL Error 28 "No space left on device"?
  • How do I stop audio recording and save only RTP headers?
  • Where should I apply savertp=header in client-server mode with packetbuffer_sender?
  • How do I reduce storage by saving only RTP headers instead of full audio?
  • Why are my PCAP files being deleted even though maxpooldays is set to 60?
  • What are autocleanspoolminpercent and autocleanmingb?
  • How do emergency cleanup triggers override retention settings?
  • How do I fix premature file deletion in spool directory?
  • Does cleanspool delete files not in the database?
  • What is maxpool_clean_obsolete?
  • How does cleanspool filesystem index differ from database records?
  • Will cleanspool delete files that are not indexed in the database?
  • How do I clean up old recordings for a specific IP address?
  • What is custom autocleaning in VoIPmonitor GUI?
  • How do I delete old recordings after configuring capture rules to stop recording?
  • Can I clean up recordings based on specific filters like IP or phone number?
  • What is the difference between custom autocleaning and global retention settings?