Data Cleaning: Difference between revisions

From VoIPmonitor.org
No edit summary
(Fix incorrect association of _2 parameters with tar_move - they are for spooldir_2 used with capture rules (TP-38))
 
(58 intermediate revisions by the same user not shown)
Line 1: Line 1:
'''This guide explains how VoIPmonitor manages data retention for both captured packets (PCAP files) and Call Detail Records (CDRs) in the database. Proper configuration is essential for managing disk space and maintaining long-term database performance.  '''
{{DISPLAYTITLE:Data Cleaning and Retention}}
[[Category:Configuration]]
[[Category:Administration]]


== Overview of Data Cleaning ==
This guide explains how VoIPmonitor manages data retention for PCAP files and database records.
VoIPmonitor generates two primary types of data that require periodic cleaning:
*'''PCAP Files:''' Raw packet captures of SIP/RTP/GRAPH data stored on the filesystem in the spool directory. These can consume significant disk space.
*'''CDR Data:''' Call metadata stored in the MySQL database. Large tables can slow down GUI performance if not managed properly.


The system uses two separate, independent mechanisms to manage the retention of this data.
== Overview ==


== 1. Filesystem Cleaning (PCAP Spool Directory) ==
VoIPmonitor generates two types of data requiring periodic cleanup:
The sensor stores captured call data in a structured directory tree on the local filesystem.


=== Spool Directory Location ===
{| class="wikitable"
By default, all data is stored in `/var/spool/voipmonitor`. This location can be changed by setting the `spooldir` option in `voipmonitor.conf`.
|-
! Data Type !! Storage !! Cleanup Mechanism !! Key Parameters
|-
| '''PCAP Files''' || Filesystem (spool directory) || Cleanspool process (every 5 min) || <code>maxpoolsize</code>, <code>maxpooldays</code>
|-
| '''CDR Records''' || MySQL database || Partition dropping (instant) || <code>cleandatabase</code>
|}
 
{{Note|These are '''independent systems''' - filesystem cleanup does not affect database records and vice versa.}}
 
=== Quick Reference ===
 
{| class="wikitable"
|-
! Purpose !! Parameter !! Example
|-
| Limit total PCAP size || <code>maxpoolsize</code> || <code>maxpoolsize = 512000</code> (500 GB)
|-
| Limit RTP specifically || <code>maxpoolrtpsize</code> || <code>maxpoolrtpsize = 102400</code> (100 GB)
|-
| Limit PCAP age || <code>maxpooldays</code> || <code>maxpooldays = 30</code>
|-
| CDR retention (days) || <code>cleandatabase</code> || <code>cleandatabase = 30</code>
|-
| CDR retention (size) || <code>cleandatabase_size</code> || <code>cleandatabase_size = 512000</code> (+ <code>cleandatabase_size_force = true</code>)
|-
| Save only RTP stats || <code>savertp</code> || <code>savertp = header</code> (saves ~90% space)
|}
 
== Filesystem Cleaning (PCAP Files) ==
 
=== How Cleanspool Works ===
 
The sniffer maintains a file index '''in memory'''. Every 5 minutes, the cleanspool thread checks retention limits and deletes the oldest files when limits are exceeded.
 
<kroki lang="plantuml">
@startuml
skinparam shadowing false
skinparam defaultFontSize 11
participant "Sniffer" as S
participant "Cleanspool" as C
database "In-Memory\nIndex" as MEM
collections "Filesystem" as FS
 
S -> FS: Write PCAP
S -> MEM: Update index
... Every 5 minutes ...
C -> MEM: Find oldest files\nexceeding limits
loop Delete old files
    C -> FS: DELETE
    C -> MEM: Remove entry
end
note over MEM : Persisted to\n.cleanspool_cache
@enduml
</kroki>
 
The <code>.cleanspool_cache</code> files in hourly directories serve as '''persistent storage for fast restart''' - they allow quick index reload without scanning the entire filesystem.
 
{{Note|1='''Legacy Indexing:''' Old VoIPmonitor versions stored metadata in MySQL <code>files</code> table. This can be enabled with <code>cleanspool_use_files = yes</code> (deprecated, do not use).}}


=== Retention Configuration ===
=== Retention Configuration ===
The cleaning process runs automatically every 5 minutes and removes the oldest data based on the rules you define in `voipmonitor.conf`. You can set limits based on total size (in Megabytes) or age (in days). If both a size and day limit are set for the same data type, the first limit that is reached will trigger the cleaning.


The following options are available:
Limits can be set by '''size''' (MB) or '''age''' (days). When both are configured, the first limit reached triggers cleanup.
 
==== Global Limits ====
 
{| class="wikitable"
|-
! Parameter !! Default !! Description
|-
| <code>maxpoolsize</code> || 102400 (100 GB) || Maximum total size for all PCAP data
|-
| <code>maxpooldays</code> || (unset) || Maximum age in days for all PCAP data
|}
 
==== Per-Type Limits ====
 
{| class="wikitable"
|-
! Data Type !! Size Parameter !! Days Parameter
|-
| SIP signaling || <code>maxpoolsipsize</code> || <code>maxpoolsipdays</code>
|-
| RTP audio || <code>maxpoolrtpsize</code> || <code>maxpoolrtpdays</code>
|-
| Quality graphs || <code>maxpoolgraphsize</code> || <code>maxpoolgraphdays</code>
|-
| Converted audio (WAV/OGG) || <code>maxpoolaudiosize</code> || <code>maxpoolaudiodays</code>
|}
 
==== Recommended Configuration ====
 
Limit RTP (largest files) while keeping SIP longer for troubleshooting:
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
maxpoolrtpsize = 102400  # 100 GB limit for RTP
maxpoolsize = 512000      # 500 GB overall limit
</syntaxhighlight>
 
==== Secondary Storage Limits (spooldir_2) ====
 
When using <code>spooldir_2</code> for secondary storage (configured via [[Capture_rules|capture rules]] with "Store pcaps to second spooldir" enabled), use <code>_2</code> suffix parameters for cleaning:
 
<code>maxpoolsize_2</code>, <code>maxpooldays_2</code>, <code>maxpoolrtpsize_2</code>, etc.
 
{{Note|1=<code>spooldir_2</code> is '''independent''' from <code>tar_move</code>. The <code>_2</code> suffix parameters apply only to the secondary spooldir configured via capture rules, not to <code>tar_move</code> destination storage.}}
 
=== Emergency Cleanup ===
 
{{Warning|1=Emergency cleanup activates when disk is nearly full and '''ignores all <code>maxpool*</code> settings'''.}}
 
{| class="wikitable"
{| class="wikitable"
|-
|-
! Parameter !! Default Value !! Description
! Parameter !! Default !! Triggers When
|-
| <code>autocleanspoolminpercent</code> || 1 || Disk usage reaches 99%
|-
| <code>autocleanmingb</code> || 5 || Free space below 5 GB
|}
 
When triggered, oldest data is deleted aggressively until thresholds are cleared. The <code>cleanspool_enable_fromto</code> time window is ignored.
 
=== Control Parameters ===
 
{| class="wikitable"
|-
! Parameter !! Default !! Description
|-
| <code>cleanspool</code> || yes || Enable/disable spool cleaning
|-
| <code>cleanspool_enable_fromto</code> || 0-24 || Restrict cleaning to hours (e.g., <code>1-5</code> for 1-5 AM)
|-
| <code>maxpool_clean_obsolete</code> || no || Delete files not in index (use with caution)
|-
| <code>all_unlink_log</code> || no || Log all file deletions
|}
 
=== Reducing Data at Source ===
 
==== Save RTP Headers Only ====
 
If you only need quality metrics (MOS, jitter, packet loss) without audio playback:
 
<syntaxhighlight lang="ini">
savertp = header
</syntaxhighlight>
 
This reduces storage by up to '''90%''' while preserving all quality statistics.
 
 
 
==== Disable RTP Entirely ====
 
If SIP signaling retention is the priority and you do not need RTP data at all:
 
<syntaxhighlight lang="ini">
savertp = no
</syntaxhighlight>
 
This disables RTP packet storage entirely, reducing PCAP size by approximately '''20x'''. Use this when:
* Storage limits are insufficient for even header-only RTP
* SIP troubleshooting is the primary need (no audio analysis required)
* You need to retain SIP signaling for much longer periods
 
{{Warning|Setting <code>savertp = no</code> removes all RTP data including quality statistics. Use <code>savertp = header</code> instead if you need MOS, jitter, or packet loss metrics.}}
==== Selective Recording ====
 
To record full audio only for specific calls:
# Set <code>savertp = header</code> globally
# Create capture rules with <code>recordRTP=ON</code> for exceptions
 
See [[Capture_rules]] for details.
 
== Database Cleaning (CDR Records) ==
 
=== Partitioning Method ===
 
VoIPmonitor uses daily partitioning. Dropping old partitions is instant (milliseconds) regardless of row count.
 
<syntaxhighlight lang="ini">
# Keep CDR records for 30 days
cleandatabase = 30
</syntaxhighlight>
 
=== Database Cleaning Parameters ===
 
{| class="wikitable"
|-
|-
| `maxpoolsize` || `102400` (100 GB) || The total maximum disk space for '''all''' captured data (SIP, RTP, GRAPH, AUDIO).
! Parameter !! Default !! Description
|-
|-
| `maxpooldays` || (unset) || The maximum number of days to keep '''all''' captured data.
| <code>cleandatabase</code> || 0 (disabled) || Global retention for CDR, register_state, register_failed, sip_msg
|-
|-
| `maxpoolsipsize` || (unset) || A specific size limit for SIP PCAP files only.
| <code>cleandatabase_cdr</code> || 0 || CDR table (includes <code>message</code> table)
|-
|-
| `maxpoolsipdays` || (unset) || A specific age limit for SIP PCAP files only.
| <code>cleandatabase_rtp_stat</code> || 2 || RTP statistics table
|-
|-
| `maxpoolrtpsize` || (unset) || A specific size limit for RTP PCAP files only.
| <code>cleandatabase_register_state</code> || 0 || Registration state
|-
|-
| `maxpoolrtpdays` || (unset) || A specific age limit for RTP PCAP files only.
| <code>cleandatabase_register_failed</code> || 0 || Failed registrations
|-
|-
| `maxpoolgraphsize` || (unset) || A specific size limit for GRAPH files only.
| <code>cleandatabase_register_time_info</code> || 0 || Registration timing ('''NOT''' covered by global)
|-
|-
| `maxpoolgraphdays` || (unset) || A specific age limit for GRAPH files only.
| <code>cleandatabase_sip_msg</code> || 0 || SIP messages (OPTIONS/SUBSCRIBE/NOTIFY)
|-
|-
| `maxpoolaudiosize` || (unset) || A specific size limit for converted audio files (WAV/OGG) only.
| <code>cleandatabase_ss7</code> || 0 || SS7 records
|-
|-
| `maxpoolaudiodays` || (unset) || A specific age limit for converted audio files (WAV/OGG) only.
| <code>partition_operations_enable_fromto</code> || 1-5 || Time window for partition operations (24h format)
|}
|}


=== Maintenance: Re-indexing the Spool Directory ===
{{Warning|1=<code>register_time_info</code> is NOT covered by global <code>cleandatabase</code>. Set <code>cleandatabase_register_time_info</code> explicitly.}}
VoIPmonitor maintains an index of all created PCAP files to perform cleaning efficiently without scanning the entire directory tree. If this index becomes corrupt, or if you manually move files into the spool, old data may not be deleted correctly.
 
=== Size-Based Cleaning ===
 
To limit database by size instead of time:
 
<syntaxhighlight lang="ini">
cleandatabase_size = 512000        # 500 GB limit in MB
cleandatabase_size_force = true    # Required to enable
</syntaxhighlight>
 
=== Multi-Sensor Environments ===
 
When multiple sensors share a database, only '''ONE''' sensor should manage partitions:
 
<syntaxhighlight lang="ini">
# On all sensors EXCEPT one:
disable_partition_operations = yes
 
# On the designated sensor:
partition_operations_enable_fromto = 4-6
</syntaxhighlight>
 
=== Limits ===
 
* '''MySQL/MariaDB partition limit:''' ~8000 partitions per table (~22 years with daily partitioning)
* '''Aurora DB partition limit:''' ~800 partitions per table - much lower than standard MySQL
* '''CDR record limit:''' No practical limit - uses BIGINT
 
=== Aurora DB / Limited Partition Environments ===
 
For databases with low partition limits (like Amazon Aurora), you can pre-create partitions before starting the sniffer and prevent runtime partition creation:
 
<syntaxhighlight lang="ini">
# Pre-create partitions up to a specific date before starting
create_new_partitions_until = 2026-01-31
 
# Disable only partition creation (dropping still works for cleanup)
disable_partition_operations_create = yes
 
# Or disable only partition dropping (creation still works)
# disable_partition_operations_drop = yes
 
# Number of partitions to create ahead (default 2)
create_new_partitions = 2
</syntaxhighlight>
 
{{Note|1=For Aurora DB with ~800 partition limit: with daily partitions, this gives ~2 years of data. Plan retention accordingly using <code>cleandatabase</code>.}}
 
== Advanced Topics ==
 
=== Spool Directory Location ===
 
Default: <code>/var/spool/voipmonitor</code>
 
Structure: <code>YYYY-MM-DD/HH/MM/{SIP|RTP|GRAPH|AUDIO}/files...</code>
 
==== Relocating the Spool ====
 
<syntaxhighlight lang="bash">
# 1. Create new directory
mkdir -p /mnt/storage/voipmonitor
chown voipmonitor:voipmonitor /mnt/storage/voipmonitor
 
# 2. Update sniffer config
# /etc/voipmonitor.conf:
# spooldir = /mnt/storage/voipmonitor
 
# 3. Update GUI config (config/configuration.php):
# define('SNIFFER_DATA_PATH', '/mnt/storage/voipmonitor');
 
# 4. Restart
systemctl restart voipmonitor
</syntaxhighlight>
 
=== Tiered Storage (tar_move) ===
 
Extend retention using secondary storage:
 
<syntaxhighlight lang="ini">
spooldir = /var/spool/voipmonitor
tar_move = yes
tar_move_destination_path = /mnt/archive/voipmonitor
</syntaxhighlight>
 
Files in secondary storage remain accessible via GUI.


In this case, you must trigger a manual re-index. This can be done via the sniffer's manager API.
==== S3 Cloud Storage ====
# '''Open a manager API session:'''
#<pre>echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029</pre>
# '''Send the re-index command:'''
#<pre>echo reindexfiles | nc -U /tmp/vmsck</pre>
''Note: This command requires `netcat` with support for UNIX sockets (`-U`). For alternative methods, see the [[Encryption_in_manager_api_customer|Manager API documentation]].''


== 2. Database Cleaning (CDR Retention) ==
Use <code>rclone</code> instead of <code>s3fs</code> to avoid GUI unresponsiveness:
Managing the size of the `cdr` table and other large tables is critical for GUI performance.


=== The Modern Method: Partitioning (Recommended) ===
<syntaxhighlight lang="bash">
Since version 7, VoIPmonitor utilizes **database partitioning**, which splits large tables into smaller, daily segments. This is the highly recommended method for managing database retention.
rclone mount bucket-name /mnt/s3-archive \
* '''How it works:''' You set a single parameter, `cleandatabase`, in `voipmonitor.conf`. This defines the number of days to keep CDRs. For example, `cleandatabase = 30` will keep the last 30 days of data.
  --allow-other --dir-cache-time 30s --vfs-cache-mode off
* '''Why it's better:''' The sniffer automatically drops old daily partitions, which is an instantaneous operation that takes milliseconds, regardless of how many millions of rows it contains. This puts zero load on the database.
</syntaxhighlight>
* '''Requirement:''' Partitioning is enabled by default on all new installations. If you are upgrading from a very old version, it may require starting with a fresh database.
 
=== Custom Autocleaning (GUI) ===
 
For one-time cleanup of specific recordings (by IP, phone number, etc.):
# Navigate to Settings > Custom Autocleaning
# Create rule with filters
# Apply and remove rule after completion
 
== Troubleshooting ==
 
=== Files Disappearing Faster Than Expected ===
 
;1. Check if emergency cleanup is active
<syntaxhighlight lang="bash">
df -h /var/spool/voipmonitor
# If >95% full, emergency cleanup is running
</syntaxhighlight>
 
;2. Check GUI configuration override
When <code>mysqlloadconfig = yes</code> (default), GUI settings override config file. Check: Settings > Sensors > wrench icon.
 
;3. Set appropriate limits
Set <code>maxpoolsize</code> to 90-95% of disk capacity to leave buffer for growth.
 
=== Disk Space Not Reclaimed After Database Cleanup ===
 
Check <code>innodb_file_per_table</code>:
 
<syntaxhighlight lang="sql">
SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';
</syntaxhighlight>
 
If OFF, enable in <code>my.cnf</code>:
 
<syntaxhighlight lang="ini">
[mysqld]
innodb_file_per_table = 1
</syntaxhighlight>
 
=== MySQL Error 28: No Space Left ===
 
Enable size-based cleaning:
 
<syntaxhighlight lang="ini">
cleandatabase_size = 512000
cleandatabase_size_force = true
</syntaxhighlight>
 
Other causes: Inode exhaustion (<code>df -i</code>), MySQL tmpdir full.
 
=== Verify Database Cleaning ===
 
<syntaxhighlight lang="sql">
SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'cdr'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;
</syntaxhighlight>
 
If partition count matches your <code>cleandatabase</code> setting, cleaning IS working.
 
For SQL queue issues and database performance, see [[Database_troubleshooting]].
 
== See Also ==
 
* [[Sniffer_configuration]] - Full parameter reference
* [[Database_troubleshooting]] - SQL queue, performance issues
* [[Scaling]] - Performance tuning
* [[Capture_rules]] - Selective recording


More details can be found in the [[Sniffer_configuration#cleandatabase|Sniffer Configuration guide]].


=== The Legacy Method: Manual Deletion (Not Recommended) ===
If you are running a very old, non-partitioned database, you cannot use the `cleandatabase` option. You would need to create a custom script that runs a `DELETE FROM cdr WHERE calldate < ...` query.
* '''Warning:''' This method is extremely slow and resource-intensive on large tables. A single `DELETE` operation on millions of rows can take hours and generate significant I/O load on your database server, potentially impacting GUI performance.


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' This article explains the two distinct data retention mechanisms in VoIPmonitor: filesystem cleaning for PCAP files and database cleaning for CDRs. For filesystem storage in the `spooldir`, it details the various `maxpoolsize` and `maxpooldays` configuration options that control the retention of SIP, RTP, and other files based on size or age. It also describes the `reindexfiles` manager command for troubleshooting cases where old files are not being deleted. For database retention, it strongly recommends the modern, default method of using MySQL/MariaDB partitioning combined with the `cleandatabase` setting in `voipmonitor.conf`, explaining that this allows for instantaneous deletion of old data. It contrasts this with the slow and inefficient legacy method of using `DELETE` queries on non-partitioned tables.
 
'''Keywords:''' data retention, cleaning, delete old calls, purge data, disk space, spooldir, maxpoolsize, maxpooldays, pcap, filesystem, database, cdr, cleandatabase, partitioning, reindexfiles, manager api
'''Summary:''' VoIPmonitor has two independent data retention systems: (1) Cleanspool for PCAP files using <code>maxpoolsize</code>/<code>maxpooldays</code> parameters, running every 5 minutes with in-memory file index; (2) Database cleaning using <code>cleandatabase</code> with instant partition dropping. Key space-saving option: <code>savertp = header</code> reduces storage by 90% while keeping quality metrics. Emergency cleanup (<code>autocleanspoolminpercent=1</code>, <code>autocleanmingb=5</code>) activates when disk nearly full and ignores normal limits. GUI settings override config file when <code>mysqlloadconfig=yes</code>. Size-based database cleaning requires BOTH <code>cleandatabase_size</code> AND <code>cleandatabase_size_force=true</code>. In multi-sensor environments, only one sensor should manage partitions (<code>disable_partition_operations=yes</code> on others).
 
'''Keywords:''' data retention, maxpoolsize, maxpooldays, maxpoolrtpsize, cleandatabase, cleandatabase_size, cleandatabase_size_force, cleanspool, autocleanspoolminpercent, autocleanmingb, tar_move, tiered storage, savertp header, innodb_file_per_table, partition dropping, emergency cleanup, mysqlloadconfig, disable_partition_operations
 
'''Key Questions:'''
'''Key Questions:'''
* How do I automatically delete old PCAP files to free up disk space?
* How does VoIPmonitor cleanspool work?
* What is the difference between `maxpoolsize` and `maxpooldays`?
* How to configure PCAP file retention?
* My spool directory is full, but old files are not being deleted. How do I fix it?
* How to limit database size?
* How do I automatically delete old CDRs from the database?
* Why are files being deleted faster than expected?
* What is the `cleandatabase` option and how does it work?
* How to fix MySQL Error 28 no space left?
* Why is database partitioning important for VoIPmonitor?
* How to reduce storage usage (savertp header)?
* What is the best way to manage data retention in VoIPmonitor?
* How to configure tiered storage?
* Why is disk space not reclaimed after cleanup?
* How to manage partitions in multi-sensor environment?

Latest revision as of 12:10, 22 January 2026


This guide explains how VoIPmonitor manages data retention for PCAP files and database records.

Overview

VoIPmonitor generates two types of data requiring periodic cleanup:

Data Type Storage Cleanup Mechanism Key Parameters
PCAP Files Filesystem (spool directory) Cleanspool process (every 5 min) maxpoolsize, maxpooldays
CDR Records MySQL database Partition dropping (instant) cleandatabase

ℹ️ Note: These are independent systems - filesystem cleanup does not affect database records and vice versa.

Quick Reference

Purpose Parameter Example
Limit total PCAP size maxpoolsize maxpoolsize = 512000 (500 GB)
Limit RTP specifically maxpoolrtpsize maxpoolrtpsize = 102400 (100 GB)
Limit PCAP age maxpooldays maxpooldays = 30
CDR retention (days) cleandatabase cleandatabase = 30
CDR retention (size) cleandatabase_size cleandatabase_size = 512000 (+ cleandatabase_size_force = true)
Save only RTP stats savertp savertp = header (saves ~90% space)

Filesystem Cleaning (PCAP Files)

How Cleanspool Works

The sniffer maintains a file index in memory. Every 5 minutes, the cleanspool thread checks retention limits and deletes the oldest files when limits are exceeded.

The .cleanspool_cache files in hourly directories serve as persistent storage for fast restart - they allow quick index reload without scanning the entire filesystem.

ℹ️ Note: Legacy Indexing: Old VoIPmonitor versions stored metadata in MySQL files table. This can be enabled with cleanspool_use_files = yes (deprecated, do not use).

Retention Configuration

Limits can be set by size (MB) or age (days). When both are configured, the first limit reached triggers cleanup.

Global Limits

Parameter Default Description
maxpoolsize 102400 (100 GB) Maximum total size for all PCAP data
maxpooldays (unset) Maximum age in days for all PCAP data

Per-Type Limits

Data Type Size Parameter Days Parameter
SIP signaling maxpoolsipsize maxpoolsipdays
RTP audio maxpoolrtpsize maxpoolrtpdays
Quality graphs maxpoolgraphsize maxpoolgraphdays
Converted audio (WAV/OGG) maxpoolaudiosize maxpoolaudiodays

Recommended Configuration

Limit RTP (largest files) while keeping SIP longer for troubleshooting: 

# /etc/voipmonitor.conf
maxpoolrtpsize = 102400   # 100 GB limit for RTP
maxpoolsize = 512000      # 500 GB overall limit

Secondary Storage Limits (spooldir_2)

When using spooldir_2 for secondary storage (configured via capture rules with "Store pcaps to second spooldir" enabled), use _2 suffix parameters for cleaning:

maxpoolsize_2, maxpooldays_2, maxpoolrtpsize_2, etc.

ℹ️ Note: spooldir_2 is independent from tar_move. The _2 suffix parameters apply only to the secondary spooldir configured via capture rules, not to tar_move destination storage.

Emergency Cleanup

⚠️ Warning: Emergency cleanup activates when disk is nearly full and ignores all maxpool* settings.

Parameter Default Triggers When
autocleanspoolminpercent 1 Disk usage reaches 99%
autocleanmingb 5 Free space below 5 GB

When triggered, oldest data is deleted aggressively until thresholds are cleared. The cleanspool_enable_fromto time window is ignored.

Control Parameters

Parameter Default Description
cleanspool yes Enable/disable spool cleaning
cleanspool_enable_fromto 0-24 Restrict cleaning to hours (e.g., 1-5 for 1-5 AM)
maxpool_clean_obsolete no Delete files not in index (use with caution)
all_unlink_log no Log all file deletions

Reducing Data at Source

Save RTP Headers Only

If you only need quality metrics (MOS, jitter, packet loss) without audio playback: 

savertp = header

This reduces storage by up to 90% while preserving all quality statistics.


Disable RTP Entirely

If SIP signaling retention is the priority and you do not need RTP data at all: 

savertp = no

This disables RTP packet storage entirely, reducing PCAP size by approximately 20x. Use this when:

  • Storage limits are insufficient for even header-only RTP
  • SIP troubleshooting is the primary need (no audio analysis required)
  • You need to retain SIP signaling for much longer periods

⚠️ Warning:

Selective Recording

To record full audio only for specific calls:

  1. Set savertp = header globally
  2. Create capture rules with recordRTP=ON for exceptions

See Capture_rules for details.

Database Cleaning (CDR Records)

Partitioning Method

VoIPmonitor uses daily partitioning. Dropping old partitions is instant (milliseconds) regardless of row count. 

# Keep CDR records for 30 days
cleandatabase = 30

Database Cleaning Parameters

Parameter Default Description
cleandatabase 0 (disabled) Global retention for CDR, register_state, register_failed, sip_msg
cleandatabase_cdr 0 CDR table (includes message table)
cleandatabase_rtp_stat 2 RTP statistics table
cleandatabase_register_state 0 Registration state
cleandatabase_register_failed 0 Failed registrations
cleandatabase_register_time_info 0 Registration timing (NOT covered by global)
cleandatabase_sip_msg 0 SIP messages (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase_ss7 0 SS7 records
partition_operations_enable_fromto 1-5 Time window for partition operations (24h format)

⚠️ Warning: register_time_info is NOT covered by global cleandatabase. Set cleandatabase_register_time_info explicitly.

Size-Based Cleaning

To limit database by size instead of time: 

cleandatabase_size = 512000        # 500 GB limit in MB
cleandatabase_size_force = true    # Required to enable

Multi-Sensor Environments

When multiple sensors share a database, only ONE sensor should manage partitions: 

# On all sensors EXCEPT one:
disable_partition_operations = yes

# On the designated sensor:
partition_operations_enable_fromto = 4-6

Limits

  • MySQL/MariaDB partition limit: ~8000 partitions per table (~22 years with daily partitioning)
  • Aurora DB partition limit: ~800 partitions per table - much lower than standard MySQL
  • CDR record limit: No practical limit - uses BIGINT

Aurora DB / Limited Partition Environments

For databases with low partition limits (like Amazon Aurora), you can pre-create partitions before starting the sniffer and prevent runtime partition creation: 

# Pre-create partitions up to a specific date before starting
create_new_partitions_until = 2026-01-31

# Disable only partition creation (dropping still works for cleanup)
disable_partition_operations_create = yes

# Or disable only partition dropping (creation still works)
# disable_partition_operations_drop = yes

# Number of partitions to create ahead (default 2)
create_new_partitions = 2

ℹ️ Note: For Aurora DB with ~800 partition limit: with daily partitions, this gives ~2 years of data. Plan retention accordingly using cleandatabase.

Advanced Topics

Spool Directory Location

Default: /var/spool/voipmonitor

Structure: YYYY-MM-DD/HH/MM/{SIP|RTP|GRAPH|AUDIO}/files...

Relocating the Spool

# 1. Create new directory
mkdir -p /mnt/storage/voipmonitor
chown voipmonitor:voipmonitor /mnt/storage/voipmonitor

# 2. Update sniffer config
# /etc/voipmonitor.conf:
# spooldir = /mnt/storage/voipmonitor

# 3. Update GUI config (config/configuration.php):
# define('SNIFFER_DATA_PATH', '/mnt/storage/voipmonitor');

# 4. Restart
systemctl restart voipmonitor

Tiered Storage (tar_move)

Extend retention using secondary storage: 

spooldir = /var/spool/voipmonitor
tar_move = yes
tar_move_destination_path = /mnt/archive/voipmonitor

Files in secondary storage remain accessible via GUI.

S3 Cloud Storage

Use rclone instead of s3fs to avoid GUI unresponsiveness: 

rclone mount bucket-name /mnt/s3-archive \
  --allow-other --dir-cache-time 30s --vfs-cache-mode off

Custom Autocleaning (GUI)

For one-time cleanup of specific recordings (by IP, phone number, etc.):

  1. Navigate to Settings > Custom Autocleaning
  2. Create rule with filters
  3. Apply and remove rule after completion

Troubleshooting

Files Disappearing Faster Than Expected

1. Check if emergency cleanup is active
df -h /var/spool/voipmonitor
# If >95% full, emergency cleanup is running
2. Check GUI configuration override

When mysqlloadconfig = yes (default), GUI settings override config file. Check: Settings > Sensors > wrench icon.

3. Set appropriate limits

Set maxpoolsize to 90-95% of disk capacity to leave buffer for growth.

Disk Space Not Reclaimed After Database Cleanup

Check innodb_file_per_table: 

SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';

If OFF, enable in my.cnf: 

[mysqld]
innodb_file_per_table = 1

MySQL Error 28: No Space Left

Enable size-based cleaning: 

cleandatabase_size = 512000
cleandatabase_size_force = true

Other causes: Inode exhaustion (df -i), MySQL tmpdir full.

Verify Database Cleaning

SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'cdr'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;

If partition count matches your cleandatabase setting, cleaning IS working.

For SQL queue issues and database performance, see Database_troubleshooting.

See Also


AI Summary for RAG

Summary: VoIPmonitor has two independent data retention systems: (1) Cleanspool for PCAP files using maxpoolsize/maxpooldays parameters, running every 5 minutes with in-memory file index; (2) Database cleaning using cleandatabase with instant partition dropping. Key space-saving option: savertp = header reduces storage by 90% while keeping quality metrics. Emergency cleanup (autocleanspoolminpercent=1, autocleanmingb=5) activates when disk nearly full and ignores normal limits. GUI settings override config file when mysqlloadconfig=yes. Size-based database cleaning requires BOTH cleandatabase_size AND cleandatabase_size_force=true. In multi-sensor environments, only one sensor should manage partitions (disable_partition_operations=yes on others).

Keywords: data retention, maxpoolsize, maxpooldays, maxpoolrtpsize, cleandatabase, cleandatabase_size, cleandatabase_size_force, cleanspool, autocleanspoolminpercent, autocleanmingb, tar_move, tiered storage, savertp header, innodb_file_per_table, partition dropping, emergency cleanup, mysqlloadconfig, disable_partition_operations

Key Questions:

  • How does VoIPmonitor cleanspool work?
  • How to configure PCAP file retention?
  • How to limit database size?
  • Why are files being deleted faster than expected?
  • How to fix MySQL Error 28 no space left?
  • How to reduce storage usage (savertp header)?
  • How to configure tiered storage?
  • Why is disk space not reclaimed after cleanup?
  • How to manage partitions in multi-sensor environment?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Data_Cleaning&oldid=11005"