Data Cleaning: Difference between revisions

From VoIPmonitor.org
(Add rclone solution to AI Summary for RAG - fixes GUI unresponsiveness with s3fs (ticket 84601))
(Consolidate article: Add cleanspool modes (legacy/modern), fix defaults (legacy is de facto default due to pcap_dump_tar=yes), add maxpool_clean_obsolete, add BIGINT info, shorten AI Summary from 145 to 25 lines)
Line 34: Line 34:
   collections "register_state" as reg
   collections "register_state" as reg
   collections "sip_msg" as sipmsg
   collections "sip_msg" as sipmsg
  collections "files" as filesdb
}
}


sensor --> spool : writes
sensor --> spool : writes
sensor --> cdr : writes
sensor --> cdr : writes
sensor --> filesdb : indexes files


rectangle "Filesystem Cleaner\n(maxpoolsize/maxpooldays)" as fscleaner #E8F5E9
rectangle "Filesystem Cleaner\n(maxpoolsize/maxpooldays)" as fscleaner #E8F5E9
Line 43: Line 45:


fscleaner --> spool : deletes old files
fscleaner --> spool : deletes old files
fscleaner --> filesdb : reads file index
dbcleaner --> cdr : drops partitions
dbcleaner --> cdr : drops partitions


Line 53: Line 56:


The sensor stores captured call data in a structured directory tree on the local filesystem.
The sensor stores captured call data in a structured directory tree on the local filesystem.
=== How Cleanspool Works: Modern vs Legacy Mode ===
VoIPmonitor has two distinct modes for tracking and cleaning PCAP files. Understanding which mode your installation uses is critical for troubleshooting cleaning issues.
{| class="wikitable" style="background-color: #E8F5E9;"
|-
! colspan="2" style="background: #4CAF50; color: white;" | Important: Two Cleanspool Modes
|-
| style="vertical-align: top;" | '''Legacy Mode (Default):'''
|
<code>cleanspool_use_files = no</code>
Uses a filesystem-based index stored in <code>.cleanspool_cache</code> file in the spool directory. The cleanspool process scans the directory structure and maintains this cache file locally.
'''Why this is the default:''' The <code>pcap_dump_tar = yes</code> setting (TAR archive mode) is enabled by default. When TAR mode is active, VoIPmonitor automatically switches to legacy cleanspool mode unless you explicitly set <code>cleanspool_use_files = yes</code>.
|-
| style="vertical-align: top;" | '''Modern Mode (Database-Indexed):'''
|
<code>cleanspool_use_files = yes</code>
Uses the MySQL <code>files</code> table to track all created PCAP files. The sniffer records every file it creates in this database table, and cleanspool deletes '''only files that are indexed in the <code>files</code> table'''.
'''Key behavior:''' Files that exist on the filesystem but are NOT in the <code>files</code> table will NOT be automatically deleted (unless <code>maxpool_clean_obsolete</code> is enabled).
'''To enable modern mode:'''
<syntaxhighlight lang="ini">
cleanspool_use_files = yes
</syntaxhighlight>
|}
==== Legacy Mode: Filesystem Cache (Default) ====
In legacy mode (<code>cleanspool_use_files = no</code>), which is active by default when TAR mode is enabled:
<kroki lang="plantuml">
@startuml
skinparam shadowing false
participant "Sniffer" as S
participant "Cleanspool\nThread" as C
collections "Filesystem" as FS
file ".cleanspool_cache" as Cache
S -> FS: Write PCAP file\n(into TAR archive)
... Every 5 minutes ...
C -> FS: Scan directory structure
C -> Cache: Update file index
C -> Cache: Read oldest files\nwhere age > maxpooldays\nOR total size > maxpoolsize
loop For each file
    C -> FS: DELETE file
    C -> Cache: Update index
end
@enduml
</kroki>
'''Characteristics of Legacy Mode:'''
* Uses <code>.cleanspool_cache</code> file in spool directory for tracking
* Scans filesystem to build and update the index
* Default behavior when <code>pcap_dump_tar = yes</code> (TAR archive mode is default)
* Files not in cache are ignored (unless <code>maxpool_clean_obsolete = yes</code>)
'''Checking which mode is active:'''
<syntaxhighlight lang="bash">
# Check TAR mode (if yes, likely legacy cleanspool)
grep pcap_dump_tar /etc/voipmonitor.conf
grep tar /etc/voipmonitor.conf | head -5
# Check explicit cleanspool_use_files setting
grep cleanspool_use_files /etc/voipmonitor.conf
# Check if .cleanspool_cache exists (indicates legacy mode is/was used)
ls -la /var/spool/voipmonitor/.cleanspool_cache
</syntaxhighlight>
==== Modern Mode: Database-Indexed Cleaning ====
In modern mode (<code>cleanspool_use_files = yes</code>), which must be explicitly enabled:
<kroki lang="plantuml">
@startuml
skinparam shadowing false
participant "Sniffer" as S
database "MySQL\nfiles table" as DB
participant "Cleanspool\nThread" as C
collections "Filesystem" as FS
S -> FS: Write PCAP file
S -> DB: INSERT file record\n(path, size, timestamp)
... Every 5 minutes ...
C -> DB: SELECT oldest files\nwhere age > maxpooldays\nOR total size > maxpoolsize
DB -> C: Return file list
loop For each file
    C -> FS: DELETE file
    C -> DB: DELETE record
end
@enduml
</kroki>
'''Advantages of Modern Mode:'''
* Faster cleanup (no filesystem scanning)
* Accurate size tracking via database
* Supports distributed deployments with shared storage
* Files not in database are protected from accidental deletion
'''To enable Modern Mode:'''
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
cleanspool_use_files = yes
</syntaxhighlight>
==== The maxpool_clean_obsolete Parameter ====
This parameter controls how cleanspool handles files that exist on the filesystem but are NOT in its index (either database <code>files</code> table or <code>.cleanspool_cache</code>).
{| class="wikitable"
|-
! Setting !! Behavior !! Use Case
|-
| <code>maxpool_clean_obsolete = no</code> (default) || Only delete files that are indexed. Unknown files are ignored. || Safe default - protects manually added files
|-
| <code>maxpool_clean_obsolete = yes</code> || Delete ALL files in spool directory, including those not in index || Clean up orphaned files, recover from index corruption
|}
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Warning: maxpool_clean_obsolete
|-
| style="vertical-align: top;" | '''Caution:'''
|
When <code>maxpool_clean_obsolete = yes</code> is enabled, cleanspool will scan the entire filesystem and delete any files not found in its index. This includes:
* Files manually copied into the spool directory
* Files from backup restores
* Files from other sensors (in shared storage scenarios)
Only enable this if you understand the implications and want aggressive cleanup of unindexed files.
|}


=== Reducing Data Collection at Source ===
=== Reducing Data Collection at Source ===
Line 82: Line 225:
'''Important:''' After changing from <code>savertp = yes</code> to <code>savertp = header</code>, existing PCAP files will remain playable. New calls will only contain RTP headers.
'''Important:''' After changing from <code>savertp = yes</code> to <code>savertp = header</code>, existing PCAP files will remain playable. New calls will only contain RTP headers.


{| class="wikitable" style="background-color: #FFF3CD;"
{| class="wikitable" style="background-color: #FFF3CD;"
|-
|-
! colspan="2" style="background: #856404; color: white;" | Critical: Distributed Architecture Consideration
! colspan="2" style="background: #856404; color: white;" | Critical: Distributed Architecture Consideration
Line 265: Line 408:
</syntaxhighlight>
</syntaxhighlight>


=== How Cleanup Works ===
=== Retention Mechanism ===


The cleaning process runs automatically every 5 minutes and removes data based on the configuration in <code>voipmonitor.conf</code>.
The cleaning process runs automatically every 5 minutes and removes data based on the configuration in <code>voipmonitor.conf</code>.
Line 306: Line 449:
* This happens regardless of your <code>maxpool*</code> or <code>maxpooldays</code> settings
* This happens regardless of your <code>maxpool*</code> or <code>maxpooldays</code> settings
* Normal retention behavior resumes once thresholds are cleared
* Normal retention behavior resumes once thresholds are cleared
For emergency cleanup details, see [[#Emergency_Cleanup_Triggers|Emergency Cleanup Triggers]].


==== Performance Warning: maxpoolaudiosize ====
==== Performance Warning: maxpoolaudiosize ====
Line 321: Line 462:
|}
|}


* '''Recommended:** For high-volume deployments, use <code>savertp = yes</code> and convert on-demand via the GUI instead
* '''Recommended:''' For high-volume deployments, use <code>savertp = yes</code> and convert on-demand via the GUI instead
* '''Alternative:** Consider <code>savertp = header</code> if you only need statistics, not audio playback
* '''Alternative:''' Consider <code>savertp = header</code> if you only need statistics, not audio playback


=== Retention Configuration ===
=== Retention Configuration ===
Line 407: Line 548:
|}
|}


=== Alternative Configuration: Time-Based SIP Retention ===
=== Alternative Configuration: Time-Based Retention ===


If you need to keep SIP PCAP files for a specific number of days (such as compliance requirements mandating 14+ days of SIP data), you can use the <code>maxpoolsipdays</code> parameter.
If you need to keep data for a specific number of days (such as compliance requirements), you can use time-based retention parameters.


{| class="wikitable" style="background-color: #FFF3CD;"
{| class="wikitable" style="background-color: #FFF3CD;"
Line 460: Line 601:


If you see multiple triggers, VoIPmonitor applies the first limit that is reached, deleting data until all limits are satisfied.
If you see multiple triggers, VoIPmonitor applies the first limit that is reached, deleting data until all limits are satisfied.
==== Best Practice: Configure maxpoolsize with Disk Capacity Buffer ====
When setting <code>maxpoolsize</code>, configure a buffer below the total disk capacity. This ensures there is sufficient space during periods when the cleaning process is not active (such as when <code>cleanspool_enable_fromto</code> restricts cleaning to specific hours).
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Important: Disk Capacity Buffer
|-
| style="vertical-align: top;" | '''Why a Buffer is Needed:'''
|
The cleaning process runs every 5 minutes, but <code>cleanspool_enable_fromto</code> may restrict cleaning to specific hours (e.g., 1-5 AM). During non-cleaning hours, data will continue to accumulate. Setting <code>maxpoolsize</code> close to total disk capacity risks filling the disk outside the active cleaning window.
|-
| style="vertical-align: top;" | '''Recommended Configuration:'''
|
Set <code>maxpoolsize</code> to approximately 90-95% of total disk capacity to create a buffer.
|-
| style="vertical-align: top;" | '''Example:'''
|
For a 7 TB disk:
<syntaxhighlight lang="ini">
# 6.5 TB = 6656000 MB (approximately 93% of 7 TB)
maxpoolsize = 6656000
# Restrict cleaning to 1-5 AM to avoid I/O impact during peak traffic
cleanspool_enable_fromto = 1-5
</syntaxhighlight>
This configuration provides a 0.5 TB buffer for data accumulation during the 19-hour cleaning-off window.
|}


=== Manual File Deletion ===
=== Manual File Deletion ===
Line 478: Line 648:
|}
|}


When you manually delete files, VoIPmonitor:
When you manually delete files:
* Maintains its own filesystem index in <code>.cleanspool_cache</code>
* In '''modern mode''' (<code>cleanspool_use_files = yes</code>): The <code>files</code> database table still contains records for deleted files. These orphaned records are cleaned up during the next cleanup cycle.
* Recomputes the index automatically during the next cleanup cycle
* In '''legacy mode''': The <code>.cleanspool_cache</code> is recomputed automatically during the next cleanup cycle.
* Continues to write new data to ongoing captures


==== Orphaned Database Records ====
==== Orphaned Database Records ====
Line 506: Line 675:
* **Targeted Cleanup:** Removing data for specific dates/time periods
* **Targeted Cleanup:** Removing data for specific dates/time periods


==== Recovering if .cleanspool_cache is Deleted ====
==== Recovering from Index Issues ====
 
If the cleanspool index becomes corrupted or out of sync:


If the <code>.cleanspool_cache</code> file in the spool directory is accidentally deleted, the spool index will need to be rebuilt. You can trigger a reindex via the manager API:
'''For Modern Mode (cleanspool_use_files = yes):'''


The <code>files</code> table can be rebuilt by rescanning the spool directory:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Create a temporary manager API session socket
# Trigger a reindex via manager API
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
rm /tmp/vmsck
</syntaxhighlight>
'''For Legacy Mode (.cleanspool_cache):'''


# Send the reindex command
If the <code>.cleanspool_cache</code> file is deleted or corrupted:
<syntaxhighlight lang="bash">
# Same reindex command works for legacy mode
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
echo reindexfiles | nc -U /tmp/vmsck
# Clean up the temporary socket when done
rm /tmp/vmsck
rm /tmp/vmsck
</syntaxhighlight>
</syntaxhighlight>


The <code>reindexfiles</code> command rescans the spool directory structure and rebuilds the <code>.cleanspool_cache</code> file.
The <code>reindexfiles</code> command rescans the spool directory structure and rebuilds the index.


{| class="wikitable" style="background-color: #FFF3CD;"
{| class="wikitable" style="background-color: #FFF3CD;"
Line 530: Line 708:
|}
|}


==== Managing Multiple Sensors with Shared Storage ====
=== Custom Autocleaning: One-Time Cleanup with Filters ===


If you use <code>tar_move_destination_path</code> with shared central storage across multiple sensors, ensure each sensor has a unique path. PCAP filenames are based on timestamps, so multiple sensors writing to the same directory will cause filename collisions and data overwrites.
The GUI provides a '''custom autocleaning''' feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.


Example:
==== Use Case: Cleaning Old Recordings for a Specific IP ====
<syntaxhighlight lang="ini">
 
# Sensor 01
After configuring [[Capture_rules|capture rules]] to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.
tar_move_destination_path = /mnt/central_storage/sensor01
 
Example scenario:
* You configured a capture rule to discard RTP for IP <code>192.168.1.50</code>
* Only new calls will be affected by this rule
* Existing recordings for this IP must be cleaned up manually


# Sensor 02
GUI cleanup steps:
tar_move_destination_path = /mnt/central_storage/sensor02
# Navigate to '''Settings > Custom Autocleaning'''
</syntaxhighlight>
# Create a new autocleaning rule
# Set the criteria (e.g., "Delete RTP older than 1 day")
# In the '''Common Filter''' section, specify the target IP address (<code>192.168.1.50</code>)
# Save and apply the rule
# Once the cleanup is complete, remove the autocleaning rule


For more details, see [[#Multiple_Sensors_Sharing_Central_Storage|Multiple Sensors Sharing Central Storage]].
This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.


==== When to Use Time-Based vs Size-Based Retention ====
==== Comparison with Global Retention ====


{| class="wikitable"
{| class="wikitable"
|-
|-
! Requirement !! Recommended Approach !! Example Parameters
! Feature !! Custom Autocleaning !! Global Retention
|-
| '''Scope''' || Targeted (specific IP, number, filter) || All calls
|-
|-
| Regulatory compliance (14+ days SIP)
| '''Purpose''' || One-time cleanup of existing data || Ongoing automated cleanup
| Time-based SIP retention + size-based RTP limit
| <code>maxpoolsipdays = 14</code>, <code>maxpoolrtpsize = 100000</code>
|-
|-
| Preserve SIP indefinitely, limit disk
| '''Configuration''' || GUI with CDR filters || <code>maxpoolsize</code>, <code>maxpooldays</code>
| Size-based RTP limit only (no SIP limit)
| <code>maxpoolrtpsize = 100000</code>, <code>maxpoolsize = 500000</code>
|-
|-
| Simple overall retention
| '''Flexibility''' || High - can use any CDR filter criteria || Low - time/size only
| Single global limit
| <code>maxpooldays = 30</code>
|}
|}
For the flexible "keep SIP as long as space allows" approach, use the [[#Recommended_Configuration_Strategy:_Mixed_Size-Based_Retention|Mixed Size-Based Retention]] strategy above.


=== Diagnosing Disk Space Usage ===
=== Diagnosing Disk Space Usage ===
Line 592: Line 772:
</syntaxhighlight>
</syntaxhighlight>
This shows recent months consume 80-150 GB of data. Use this data to estimate appropriate size limits.
This shows recent months consume 80-150 GB of data. Use this data to estimate appropriate size limits.
==== Best Practice: Configure maxpoolsize with Disk Capacity Buffer ====
When setting <code>maxpoolsize</code>, it is important to configure a buffer below the total disk capacity. This ensures there is sufficient space during periods when the cleaning process is not active (such as when <code>cleanspool_enable_fromto</code> restricts cleaning to specific hours).
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Important: Disk Capacity Buffer
|-
| style="vertical-align: top;" | '''Why a Buffer is Needed:'''
|
The cleaning process runs every 5 minutes, but <code>cleanspool_enable_fromto</code> may restrict cleaning to specific hours (e.g., 1-5 AM). During non-cleaning hours, data will continue to accumulate. Setting <code>maxpoolsize</code> close to total disk capacity risks filling the disk outside the active cleaning window.
|-
| style="vertical-align: top;" | '''Recommended Configuration:'''
|
Set <code>maxpoolsize</code> to approximately 90-95% of total disk capacity to create a buffer.
|-
| style="vertical-align: top;" | '''Example:'''
|
For a 7 TB disk:
<syntaxhighlight lang="ini">
# 6.5 TB = 6656000 MB (approximately 93% of 7 TB)
maxpoolsize = 6656000
# Restrict cleaning to 1-5 AM to avoid I/O impact during peak traffic
cleanspool_enable_fromto = 1-5
</syntaxhighlight>
This configuration provides a 0.5 TB buffer for data accumulation during the 19-hour cleaning-off window.
|}
If your spool directory fills to capacity despite having autoclean enabled, verify your <code>maxpoolsize</code> setting accounts for this buffer, especially if you have restricted cleaning hours.


=== Understanding Directory Size Differences (SIP vs RTP Retention) ===
=== Understanding Directory Size Differences (SIP vs RTP Retention) ===


If you notice an unexpected increase in disk space used by CDRs (PCAP files) in the spool directory (often mounted on <code>/log</code> partition), check your SIP and RTP retention settings. If you configure different retention periods for SIP and RTP data (for example, <code>maxpoolsipdays = 90</code> and <code>maxpoolrtpdays = 30</code>), you will notice a significant size difference between directories within different time windows.
If you notice significant size differences between directories in your spool, this is usually expected behavior when you have different retention periods for SIP and RTP data.


{| class="wikitable"
{| class="wikitable"
Line 639: Line 788:
|}
|}


This behavior is '''expected and by design''':
This behavior is '''expected and by design''' when using different retention periods (e.g., <code>maxpoolsipdays = 90</code> and <code>maxpoolrtpdays = 30</code>).


==== Why RTP Directories Exist After Disabling RTP Saving ====
==== Why RTP Directories Exist After Disabling RTP Saving ====


If you have configured <code>savertp = no</code> or <code>savertp = header</code> (to stop saving RTP audio) but still have <code>savertcp = yes</code> enabled (to continue saving RTCP control packets), you will still see <code>YYYY-MM-DD-HH-MM.tar</code> files within the <code>RTP</code> subdirectory.
If you have configured <code>savertp = no</code> or <code>savertp = header</code> but still see <code>RTP</code> subdirectories, this occurs because:


This occurs because:
* When <code>savertcp = yes</code>, RTCP (RTP Control Protocol) packets are saved in the <code>rtp_YYYY-MM-DD-HH-MM.tar</code> files
 
* The <code>RTP</code> directory name refers to the broad UDP port range used for RTP/RTCP traffic
* When <code>savertcp = yes</code>, RTCP (RTP Control Protocol) packets are saved in the <code>rtp_YYYY-MM-DD-HH-MM.tar</code> files within the <code>RTP</code> directory
* The <code>RTP</code> directory name refers to the broad UDP port range used for RTP/RTCP traffic, not just the presence of audio payload
* These files contain RTCP control traffic (receiver reports, sender reports) but not the full RTP audio payload
* These files contain RTCP control traffic (receiver reports, sender reports) but not the full RTP audio payload


Line 662: Line 809:
|}
|}


==== Identifying When RTP Audio Saving Changed ====
== Filesystem Troubleshooting ==


To find the oldest and most recent dates for which calls captured RTP audio payload after changing your configuration:
=== Files Disappearing Faster Than Expected ===


*'''Analyze directory sizes:''' Use the <code>du</code> command to identify which date directories contain full RTP audio.
If PCAP files are being deleted sooner than your <code>maxpooldays</code> setting suggests, check for these common causes:
<syntaxhighlight lang="bash">
# Daily overview - look for large vs small directories
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2


# Hourly overview for a specific day
==== Emergency Cleanup Triggers ====
du -h --max-depth=1 /var/spool/voipmonitor/YYYY-MM-DD | sort -k2,2
</syntaxhighlight>
*'''Verify with .pcap download:''' Download a <code>.pcap</code> file from the GUI for a call from the suspected date/time and check its contents with tools like <code>tcpdump</code> or Wireshark to confirm whether audio payload is present.
 
See also: [[Sniffer_configuration#Saving_Options|Sniffer Configuration - Saving Options]] for details on <code>savertp</code> and <code>savertcp</code> configuration.
 
* Directories within the <code>maxpoolrtpdays</code> retention window contain both SIP and RTP data, making them significantly larger (often 5-10x larger).
* Directories older than <code>maxpoolrtpdays</code> but within <code>maxpoolsipdays</code> contain only SIP data, so they are much smaller.
* The automatic cleanup process removes RTP files after <code>maxpoolrtpdays</code> and SIP files after <code>maxpoolsipdays</code>.
 
This is not an error or configuration issue. It is the expected result of having different retention periods for different data types.


==== Diagnosis: Compare Directory Sizes ====
Emergency cleanup can override your retention settings:
 
To verify this behavior, check your spool directory:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Step !! Method !! Purpose
! Parameter !! Default !! Triggers When
|-
|-
| 1 || '''Directory size analysis''' || Initial identification of date range
| <code>autocleanspoolminpercent</code> || 1 || Disk usage reaches 99%
|-
|-
| 2 || '''RTCP configuration check''' || Understand why RTP directories exist when RTCP is saved
| <code>autocleanmingb</code> || 5 || Free space falls below 5 GB
|-
| 3 || '''PCAP verification''' || Definitive confirmation of audio payload presence
|}
|}


'''Diagnosis:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cd /var/spool/voipmonitor
# Check disk usage
df -h /var/spool/voipmonitor


# Show directories sorted by size
# Check emergency trigger settings
du -h --max-depth=1 ./ | sort -rh | head -20
grep -E "autocleanspoolminpercent|autocleanmingb" /etc/voipmonitor.conf
 
# Example output:
# 80G    ./2025-01          # Current month (has SIP+RTP)
# 15G    ./2024-12          # Previous month (SIP only, RTP deleted)
# 120G  .
</syntaxhighlight>
</syntaxhighlight>


==== Step 2: Check RTCP Configuration ====
'''Resolution:'''
 
<syntaxhighlight lang="ini">
If you see <code>RTP</code> directories but have configured <code>savertp = no</code> or <code>savertp = header</code> to stop saving RTP audio, the directories may contain RTCP control packets instead:
# Increase emergency thresholds (allow more data before emergency cleanup)
 
autocleanspoolminpercent = 5  # Allow 95% usage before emergency
<syntaxhighlight lang="bash">
autocleanmingb = 20            # Trigger at 20 GB free instead of 5 GB
# Check if RTCP saving is enabled
grep savertcp /etc/voipmonitor.conf
 
# Expected: savertcp = yes
</syntaxhighlight>
</syntaxhighlight>


When <code>savertcp = yes</code>, RTCP (RTP Control Protocol) packets are saved in the <code>rtp_YYYY-MM-DD-HH-MM.tar</code> files within the <code>RTP</code> directory. These RTCP files contain control traffic (receiver reports, sender reports) but do NOT contain the full RTP audio payload.
==== GUI Configuration Override ====


For detailed information on RTCP behavior and different configuration scenarios, see [[#Why_RTP_Directories_Exist_After_Disabling_RTP_Saving|Why RTP Directories Exist After Disabling RTP Saving]].
If <code>mysqlloadconfig = yes</code> (default), GUI sensor settings override <code>voipmonitor.conf</code>:
 
==== Step 3: Definitive Verification with PCAP ====
 
To definitively verify whether a specific directory contains actual RTP audio payload (not just RTCP or headers), download a PCAP file from the GUI and analyze it:


'''Diagnosis:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Hourly overview for a specific day to identify suspicious date
# Check if database config is enabled
du -h --max-depth=1 /var/spool/voipmonitor/YYYY-MM-DD | sort -k2,2
grep mysqlloadconfig /etc/voipmonitor.conf
</syntaxhighlight>
 
Then in the VoIPmonitor GUI:
 
# Navigate to the CDR for a call from the suspected date/time
# Click the "PCAP" download button to save the .pcap file
# Analyze the PCAP locally using one of the methods below
 
'''Method 1: Using tcpdump (quick check)'''
<syntaxhighlight lang="bash">
tcpdump -r downloaded.pcap -nn 'rtp[1] & 0x7f == 0' -c 5
</syntaxhighlight>
 
'''Method 2: Using Wireshark (detailed analysis)'''
* Open the PCAP in Wireshark and filter: <code>rtp</code>
* Check for actual audio payload (large RTP packets with payload)
* If only small packets, likely only RTCP or RTP headers
 
For detailed information on this three-step verification process, see [[#Identifying_When_RTP_Audio_Saving_Changed|Identifying When RTP Audio Saving Changed]].
 
Compare with your configuration:
 
<syntaxhighlight lang="bash">
grep -E "maxpoolsip|maxpoolrtp" /etc/voipmonitor.conf
 
# Example configuration:
# maxpoolsipdays = 90    # SIP kept for 90 days
# maxpoolrtpdays = 30    # RTP kept for 30 days
</syntaxhighlight>
 
If the size difference matches your retention configuration, this is expected behavior and no action is needed.
 
=== Troubleshooting: Disk Full / Files Disappearing ===
 
If you see errors when attempting to extract older calls from the GUI, or if call files are disappearing too quickly, your spool directory may have reached its size limit.
 
==== Diagnosis: Check Disk Usage ====
 
# Identify the sensor/probe responsible for the missing data.
# SSH into the sensor/probe and navigate to the spooldir.
# Check the disk usage:
<syntaxhighlight lang="bash">
cd /var/spool/voipmonitor
du -h --max-depth=1 ./


# Example output:
# Check GUI: Settings > Sensors > wrench icon > search "maxpool"
# 150G    ./2025-01
# 120G    ./2024-12
# 90G    ./2024-11
# 360G    .
</syntaxhighlight>
</syntaxhighlight>


# Compare with the configured limit:
'''Resolution:''' Update settings via GUI, or set <code>mysqlloadconfig = no</code> to use file-based config only.
<syntaxhighlight lang="bash">
grep maxpoolsize /etc/voipmonitor.conf
# Example output: maxpoolsize = 102400  (100 GB in MB)
</syntaxhighlight>


==== Diagnostic Checklist: Understand Your System ====
==== Diagnostic Checklist ====


Before applying fixes, run through this diagnostic checklist to understand your current configuration and usage patterns. This systematic approach helps identify the root cause of fast PCAP rotation.
Before applying fixes, run through this diagnostic checklist:
 
;Step 1: Check Retention Configuration
Verify your current retention settings for both spool directory and database:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check all retention-related settings in one command
# 1. Check all retention-related settings
cat /etc/voipmonitor.conf | grep -E '^spooldir|^maxpool[^_]|^cleandatabase'
cat /etc/voipmonitor.conf | grep -E '^spooldir|^maxpool|^cleandatabase|^autoclean'
</syntaxhighlight>


This single command shows:
# 2. Measure total spool usage
* <code>spooldir</code> - Location of PCAP storage (default: <code>/var/spool/voipmonitor</code>)
* <code>maxpoolsize</code> - Maximum spool directory size in MB (default: 102400 = 100 GB)
* <code>maxpooldays</code> - Maximum age in days (unset by default, meaning size-based only)
* <code>cleandatabase</code> - Database retention in days
 
;Step 2: Measure Total Spool Usage
Check the actual size of your spool directory to see if it exceeds your configured limit:
 
<syntaxhighlight lang="bash">
du -hs /var/spool/voipmonitor
du -hs /var/spool/voipmonitor
</syntaxhighlight>


Compare this output with your <code>maxpoolsize</code> setting. If the actual size exceeds the configured limit, files are being deleted according to the size limit regardless of any age-based settings.
# 3. Analyze per-day usage patterns
 
;Step 3: Analyze Usage Patterns (Per-Day and Per-Hour)
To identify growth patterns and calculate retention requirements:
 
<syntaxhighlight lang="bash">
# Per-day overview (long-term pattern)
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2


# Per-hour breakdown for specific day (today's date)
# 4. Check disk capacity
du -h --max-depth=1 /var/spool/voipmonitor/$(date +%Y-%m-%d) | sort -k2,2
</syntaxhighlight>
 
Per-hour analysis helps you:
* Identify peak traffic hours (largest hourly directories)
* Calculate daily data growth rate
* Estimate if disk capacity can sustain your desired retention period
 
;Step 4: Check Database Size Contribution
The MySQL database also consumes disk space for CDR records, SIP messages, and other metadata:
 
<syntaxhighlight lang="bash">
du -hs /var/lib/mysql
</syntaxhighlight>
 
If database is consuming significant space (e.g., >50 GB), consider adjusting <code>cleandatabase</code> or using <code>cleandatabase_size</code> to limit database growth.
 
;Step 5: Check Overall Disk Capacity
Verify total available space on all partitions:
 
<syntaxhighlight lang="bash">
df -h
df -h
</syntaxhighlight>
Pay special attention to:
* The partition hosting your spool directory (often <code>/var</code> or dedicated partition)
* The partition hosting MySQL data (often <code>/var/lib/mysql</code>)
* Free space vs total capacity
* Whether emergency cleanup triggers might be at risk (see [[#Emergency_Cleanup_Triggers|Emergency Cleanup Triggers]])
{| class="wikitable" style="background-color: #E3F2FD;"
|-
! colspan="2" style="background: #1976D2; color: white;" | Understanding Diagnostic Output
|-
| style="vertical-align: top;" | '''Size exceeds maxpoolsize:'''
| Files are being deleted by size limit. Resolution: Increase <code>maxpoolsize</code> OR reduce capture rate.
|-
| style="vertical-align: top;" | '''Size under limit but files disappear:'''
| Likely emergency cleanup (<code>autocleanspoolminpercent</code> or <code>autocleanmingb</code>). Resolution: Adjust emergency thresholds.
|-
| style="vertical-align: top;" | '''Database growing faster than expected:'''
| High call volume or retaining extensive metadata. Resolution: Configure <code>cleandatabase_size</code> with <code>cleandatabase_size_force = true</code>.
|-
| style="vertical-align: top;" | '''Per-hour directories vary significantly:'''
| Traffic peaks during specific hours. Resolution: Ensure buffer disk capacity for peak hours or consider time-based retention.
|}
After completing this diagnostic checklist, proceed to the appropriate resolution section (Emergency Cleanup Triggers, Spooldir Size, or Database Cleaning) based on your findings.
==== Resolution: Increase Spooldir Size ====
If the actual usage exceeds the configured limit, increase <code>maxpoolsize</code>:
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
maxpoolsize = 716800  # 700 GB in MB
maxpooldays = 90      # Optional: Keep data for last 90 days
</syntaxhighlight>
Apply changes:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
==== Emergency Cleanup Triggers ====
Even when your <code>maxpooldays</code> setting appears correct, PCAP files may still be deleted prematurely due to '''emergency cleanup triggers'''. These are safety mechanisms that override normal retention settings to prevent system crashes from disk exhaustion.
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Common Cause: Retention Not Working as Expected
|-
| style="vertical-align: top;" | '''Symptom:'''
| You set <code>maxpooldays = 60</code>, but files are deleted after 2 weeks (14 days).
|-
| style="vertical-align: top;" | '''Root Cause:'''
| Emergency cleanup triggered by <code>autocleanspoolminpercent</code> or <code>autocleanmingb</code>.
|}
Emergency cleanup is activated when either of these conditions is met:
{| class="wikitable"
|-
! Parameter !! Default !! Description !! Behavior
|-
| <code>autocleanspoolminpercent</code> || 1 || Minimum free disk percentage that triggers emergency cleanup | When the partition reaches this threshold (e.g., 99% full), aggressive cleaning starts ''immediately'' regardless of <code>maxpooldays</code> setting.
|-
| <code>autocleanmingb</code> || 5 || Minimum free disk space in GB that triggers emergency cleanup | When free space falls below this value (e.g., less than 5 GB left), emergency cleaning is triggered.
|}
===== How Emergency Cleanup Works =====
The cleaning process checks disk space before each operation:
<syntaxhighlight lang="bash">
# Check current disk usage
df -h /var/spool/voipmonitor
# Example output:
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sdb1      2.0T  1.96T  40G  98% /var/spool/voipmonitor
</syntaxhighlight>
If <code>Use%</code> exceeds <code>100 - autocleanspoolminpercent</code> (e.g., 99%), emergency cleanup is triggered.
'''Emergency cleanup behavior:'''
* Files are deleted starting from the oldest until the emergency threshold is cleared
* This happens regardless of your <code>maxpooldays</code> or <code>maxpoolsize</code> settings
* The normal cleaning cycle (every 5 minutes) is triggered immediately
* Once the threshold is cleared, normal retention rules resume
===== Diagnosis: Check Emergency Cleanup Triggers =====
<syntaxhighlight lang="bash">
# Check if emergency triggers are set
grep -E "autocleanspoolminpercent|autocleanmingb" /etc/voipmonitor.conf
# Check current disk usage
df -h /var/spool/voipmonitor
# Check available space on the partition
# If Avail is close to autocleanmingb (5GB), emergency cleanup may be triggering
</syntaxhighlight>
===== Resolution: Adjust Emergency Triggers or Increase Disk Space =====
If emergency cleanup is causing premature deletion, you have two options:
;Option 1: Adjust Emergency Triggers
Increase the thresholds to allow more data before emergency cleanup triggers:
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
maxpooldays = 60
# Reduce emergency sensitivity (allow partition to fill more before trigger)
autocleanspoolminpercent = 5  # Default is 1, increase to 5 to allow 95% usage
autocleanmingb = 20            # Default is 5, increase to 20 GB
</syntaxhighlight>
Apply changes:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
;Option 2: Increase Disk Space
If the partition is genuinely full, add more storage or move the spooldir to a larger partition (see [[#Relocating_the_Spool_Directory_to_a_Different_Partition|Relocating Spooldir]]).


===== Example Troubleshooting Scenario =====
# 5. Check for emergency cleanup in logs
 
journalctl -u voipmonitor | grep -i "clean\|autoclean"
'''User report:''' "I set <code>maxpooldays = 60</code> for 60-day retention, but PCAPs are deleted after only 14 days."
 
'''Investigation:'''
<syntaxhighlight lang="bash">
# 1. Check retention settings
grep maxpooldays /etc/voipmonitor.conf
# maxpooldays = 60  ✓ Correct
 
# 2. Check disk usage
df -h /var/spool/voipmonitor
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sdb1      500G  498G  2G  99.6% /var/spool/voipmonitor
 
# 3. Check emergency triggers
grep autoclean /etc/voipmonitor.conf
# autocleanspoolminpercent = 1 (default)
# autocleanmingb = 5 (default)
</syntaxhighlight>
</syntaxhighlight>


'''Root Cause:'''
=== Spool Directory Filling Due to Database Performance ===
* Partition is 99.6% full, which exceeds the 99% threshold (100 - 1)
* Only 2 GB free space remains, below the 5 GB <code>autocleanmingb</code> limit
* Emergency cleanup triggers, deleting files until at least 5 GB is free
 
'''Solution:'''
* Increase <code>autocleanspoolminpercent = 5</code> to allow 95% usage before emergency trigger
* Or increase <code>autocleanmingb = 20</code> to allow more data before cleanup
* Or add more disk space to the partition


For the complete parameter reference, see [[Sniffer_configuration#Spool_Cleaning|Sniffer Configuration - Spool Cleaning]].
If <code>/var/spool/voipmonitor/</code> is filling up rapidly after a disk swap, MySQL upgrade, or configuration change, the issue may be caused by database performance problems.
 
=== Troubleshooting: Spool Directory Filling Due to Database Performance ===
 
If <code>/var/spool/voipmonitor/</code> is filling up rapidly after a disk swap, MySQL upgrade, or configuration change, the issue may be caused by **database performance problems causing query files to queue**, not retention policy issues.


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
Line 1,015: Line 888:
|-
|-
! Symptoms
! Symptoms
| Spool directory fills exponentially, sometimes within hours, even with correct <code>maxpoolsize</code> settings. The growth rate exceeds expected PCAP storage.
| Spool directory fills exponentially, sometimes within hours, even with correct <code>maxpoolsize</code> settings.
|-
|-
! Root Cause
! Root Cause
| MySQL cannot process database writes quickly enough, causing query files to queue in the spool directory. Common after disk swaps (which may reset MySQL settings) or MySQL upgrades (which may change default configurations).
| MySQL cannot process database writes quickly enough, causing query files to queue.
|}
|}


==== Diagnosis: Check for Database Bottleneck ====
'''Diagnosis:'''
 
;Step 1: Monitor SQL Queue in Logs
 
Check the sensor logs for a growing SQL queue, which indicates database write performance issues:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Watch for SQLq (SQL queue) metrics in real-time
# Watch for SQLq (SQL queue) metrics in real-time
tail -f /var/log/syslog | grep voipmonitor | grep SQLq
tail -f /var/log/syslog | grep voipmonitor | grep SQLq


# Example output shows queue depth:
# Check MySQL InnoDB settings
# voipmonitor[12345]: ... SQLq[150] ...
#                  ^^^^^^ A growing number indicates database bottleneck
</syntaxhighlight>
 
;Step 2: Verify MySQL InnoDB Settings
 
Check if the critical InnoDB parameters are configured for performance:
 
<syntaxhighlight lang="bash">
# Check InnoDB flush log setting
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';"
# Expected for performance: innodb_flush_log_at_trx_commit = 2
# Default (1) is very slow for high-traffic deployments
# Check InnoDB buffer pool size
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"
# Expected: 50-70% of available RAM (e.g., 4G, 8G, 14G)
# Default (128M or 256M) is insufficient for production
</syntaxhighlight>
</syntaxhighlight>


{| class="wikitable"
'''Solution - Tune MySQL:'''
|-
! Setting !! Default !! Recommended !! Impact
|-
| <code>innodb_flush_log_at_trx_commit</code> || 1 (safest, slowest) || 2 (fast, minimal risk) || 5-10x faster writes
|-
| <code>innodb_buffer_pool_size</code> || 128M-256M || 50-70% of RAM || Reduces disk I/O by caching data
|}
 
==== Solution: Tune MySQL for High Performance ====
 
Configure MySQL InnoDB settings for VoIPmonitor workloads:
 
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Edit MySQL configuration
# Common locations:
# - Debian/Ubuntu: /etc/mysql/mariadb.conf.d/50-server.cnf
# - RHEL/CentOS: /etc/my.cnf.d/mariadb-server.cnf
# - Generic: /etc/my.cnf or /etc/mysql/my.cnf
[mysqld]
[mysqld]
# Buffer pool: Use 50-70% of server RAM
# 50-70% of server RAM
# Example for a 16GB server: 8-11GB
# Example for a 32GB server: 16-22GB
innodb_buffer_pool_size = 8G
innodb_buffer_pool_size = 8G


# Flush logs: Write to OS cache every second (fast, acceptable data loss risk)
# Fast writes (acceptable data loss risk on crash)
# For maximum safety and lower performance, set to 1 (write to disk on every commit)
innodb_flush_log_at_trx_commit = 2
innodb_flush_log_at_trx_commit = 2
# Recommended but not critical for immediate fix
innodb_file_per_table = 1
innodb_compression_algorithm = lz4
</syntaxhighlight>
Restart MySQL to apply changes:
<syntaxhighlight lang="bash">
# Restart MySQL/MariaDB
systemctl restart mariadb  # or mysql
# Restart VoIPmonitor (depends on database)
systemctl restart voipmonitor
</syntaxhighlight>
==== Understanding the Settings ====
{| class="wikitable"
|-
! Setting !! What It Does !! Why It Matters for VoIPmonitor
|-
| <code>innodb_buffer_pool_size</code> || Cache for database data and indexes in RAM || VoIPmonitor performs frequent INSERT operations. Without sufficient buffer pool, MySQL must read/write disk for every CDR, causing massive I/O bottleneck and queue buildup.
|-
| <code>innodb_flush_log_at_trx_commit = 2</code> || Writes transaction logs to OS cache instead of disk on every commit || VoIPmonitor writes thousands of CDRs per second. Setting to 1 (default) writes to disk on EVERY transaction, which is extremely slow. Setting to 2 writes to RAM cache and flushes to disk once per second, which is 5-10x faster with minimal data loss risk (at most 1 second of data in event of power loss).
|}
==== Calculation Example ====
For a server with 16GB RAM:
<syntaxhighlight lang="text">
Total RAM: 16GB
- VoIPmonitor process: ~1-2GB
- OS and other services: ~1-2GB
- Available for MySQL: ~12-14GB
InnoDB buffer pool: 50-70% of available RAM
= 6-10GB (round to 8G for safety)
</syntaxhighlight>
==== Verification After Changes ====
After applying the MySQL changes, verify that the issue is resolved:
<syntaxhighlight lang="bash">
# 1. Check SQL queue is not growing
tail -f /var/log/syslog | grep "SQLq\["
# Should show SQLq[0] or low single-digit numbers
# Growing numbers indicate database still slow
# 2. Monitor disk usage growth rate
du -h --max-depth=1 /var/spool/voipmonitor | tail -5
# Should grow at expected rate (PCAP volume only), not exponentially
# 3. Check MySQL performance metrics
mysql -e "SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read%';"
# High read_requests with low reads indicates effective caching
</syntaxhighlight>
For comprehensive performance tuning guidance, see [[Scaling|Scaling and Performance Guide]], particularly the "Database Performance" section which covers buffer pool sizing calculations and advanced tuning options.
=== Troubleshooting: "low spool disk space" Message ===
If you see a '''"low spool disk space"''' log message from VoIPmonitor, this indicates a different condition than emergency cleanup. This message is independent of the <code>autocleanspool</code> directive.
{| class="wikitable" style="background-color: #E3F2FD;"
|-
! colspan="2" style="background: #1976D2; color: white;" | Understanding "low spool disk space"
|-
| style="vertical-align: top;" | '''What this message means:'''
| The system is checking if your configured <code>maxpoolsize</code> (maximum allowed disk space) can be accommodated on the physical disk. When the total allowed size exceeds the current disk usage plus free space, this warning is logged.
|-
| style="vertical-align: top;" | '''Automatic response:'''
| When this condition is met, VoIPmonitor may automatically adjust <code>maxpoolsize</code> to a lower value to prevent the disk from filling up.
|-
| style="vertical-align: top;" | '''Relationship to autocleanspool:'''
| This is independent of <code>autocleanspool</code>. Emergency cleanup (<code>autocleanspoolminpercent</code>, <code>autocleanmingb</code>) triggers when the disk is nearly full. The "low spool disk space" message triggers when <code>maxpoolsize</code> exceeds physical capacity.
|}
==== What Happens When This Message Appears ====
When VoIPmonitor detects that <code>maxpoolsize</code> > (current disk usage + available free space):
# The system logs the "low spool disk space" warning
# VoIPmonitor automatically reduces <code>maxpoolsize</code> to a safe value
# Normal cleaning continues using the adjusted, lower limit
# This prevents the disk from filling up beyond what is possible
==== Monitoring Cleaning Actions ====
To monitor which triggers are causing cleanup operations, grep the system log for 'clean' messages from the voipmonitor process:
<syntaxhighlight lang="bash">
# Monitor cleaning triggers in real-time
journalctl -u voipmonitor -f | grep clean
# Or search through existing logs
journalctl -u voipmonitor | grep clean
</syntaxhighlight>
</syntaxhighlight>


The log messages will specify the trigger, for example:
=== "low spool disk space" Message ===
* <code>clean_maxpooldays</code> - Cleaning triggered by <code>maxpooldays</code> age limit
* <code>clean_maxpoolsize</code> - Cleaning triggered by <code>maxpoolsize</code> size limit
* Other triggers indicating the specific cleanup condition
 
==== Resolution ====


To prevent the "low spool disk space" message, ensure your <code>maxpoolsize</code> setting is realistic for your disk capacity:
This message indicates that your configured <code>maxpoolsize</code> exceeds the physical disk capacity. VoIPmonitor may automatically adjust <code>maxpoolsize</code> to a lower value.


'''Resolution:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check current disk capacity
# Check actual disk capacity
df -h /var/spool/voipmonitor
df -h /var/spool/voipmonitor


# Check configured maxpoolsize
# Set realistic maxpoolsize (leave 5-10% buffer)
grep maxpoolsize /etc/voipmonitor.conf
# For a 500 GB partition, use ~450 GB:
maxpoolsize = 460800
</syntaxhighlight>
</syntaxhighlight>


Calculate appropriate <code>maxpoolsize</code>:
=== Missing Data Due to NFS/Storage Server Issues ===
* '''Current usage''' (from <code>du -hs</code>) + '''Free space available''' + '''Buffer for growth'''
* The buffer should account for data accumulation between cleaning cycles


If <code>maxpoolsize</code> is set higher than the actual disk can accommodate, reduce it:
If using remote storage (NFS, SSHFS), missing data may be caused by network connectivity issues rather than retention policies.
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
# Set a realistic limit based on actual disk capacity
maxpoolsize = 512000      # 500 GB (adjust to your actual disk capacity)
</syntaxhighlight>


Apply changes:
'''Diagnosis:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
'''Note:''' The automatic adjustment of <code>maxpoolsize</code> by VoIPmonitor is a protective mechanism. Setting an appropriate value manually prevents frequent automatic adjustments and ensures predictable behavior.
=== Troubleshooting: Missing Data Due to NFS/Storage Server Issues ===
If you are using a remote storage solution (NFS, SSHFS, or other network-mounted filesystem) for the spool directory, missing CDRs, PCAP files, or gaps in data for specific time periods may be caused by network connectivity issues between the VoIPmonitor probe and the storage server, rather than data deletion policies.
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Root Cause: Storage Server Connectivity Problems
|-
| style="vertical-align: top;" | '''Symptoms:'''
| Missing CDRs and PCAP files for specific time periods despite retention settings being correct. No deletion-related logs from emergency cleanup or retention policies.
|-
| style="vertical-align: top;" | '''Common causes:'''
| Network interruptions between probe and NFS server, NFS server timeouts, storage server not responding during the missing data period.
|}
==== Step 1: Check System Logs for NFS Errors ====
Examine the system logs on the VoIPmonitor probe for NFS-specific error messages around the time of the missing data.
<syntaxhighlight lang="bash">
# Check systemd journal for voipmonitor service
journalctl -u voipmonitor --since "YYYY-MM-DD HH:MM:SS" --until "YYYY-MM-DD HH:MM:SS"
# Check system logs for NFS errors
# Check system logs for NFS errors
grep -i "nfs" /var/log/syslog | grep "not responding\|timed out"
grep -i "nfs" /var/log/syslog | grep "not responding\|timed out"


# Also check dmesg and kernel logs
# Test connectivity
dmesg | grep -i "nfs"
</syntaxhighlight>
 
Look for specific NFS error messages:
* <code>nfs: server [IP_ADDRESS] not responding, timed out</code>
* <code>nfs: server [IP_ADDRESS] OK</code> (indicates the connection recovered)
* <code>Input/output error</code> when accessing the spool directory
 
If you see these errors during the time period where data is missing, the probe was unable to write to the storage server, resulting in lost data.
 
==== Step 2: Verify Network Connectivity ====
 
Test network connectivity between the VoIPmonitor probe and the NFS storage server.
 
<syntaxhighlight lang="bash">
# Test basic connectivity
ping nfs-server.example.com
ping nfs-server.example.com
# Test with packet loss detection
ping -c 100 nfs-server.example.com
# Trace the network path
traceroute nfs-server.example.com
# Check if NFS port is reachable (NFS typically uses port 2049)
nc -zv nfs-server.example.com 2049
nc -zv nfs-server.example.com 2049
</syntaxhighlight>
</syntaxhighlight>
Pay attention to:
* High packet loss or latency (intermittent connectivity)
* Route failures or timeouts in traceroute
* Firewall blocking the NFS port
==== Step 3: Verify NFS Server Status ====
Ensure the NFS server is running and accessible during the period when data collection should occur.
<syntaxhighlight lang="bash">
# From the probe, check if NFS share is mounted
mount | grep nfs
# Check mount status for specific mount point
mountpoint -q /var/spool/voipmonitor
# Test write access to NFS mount
touch /var/spool/voipmonitor/.test_write && rm /var/spool/voipmonitor/.test_write
</syntaxhighlight>
If the mount is stale (the NFS server went down and came back), you may need to remount:
<syntaxhighlight lang="bash">
# Unmount and remount (caution: this may affect active writes)
umount /var/spool/voipmonitor
mount /var/spool/voipmonitor
# Or use lazy unmount if needed (only if safe to interrupt)
umount -l /var/spool/voipmonitor
mount /var/spool/voipmonitor
</syntaxhighlight>
==== Step 4: Investigate Network Infrastructure ====
Common causes of intermittent NFS connectivity issues:
* '''Network equipment:''' Switch or router problems between probe and NFS server
* '''Network congestion:''' High traffic loads causing packet loss
* '''Firewall rules:''' Intermittent blocking of NFS ports
* '''NFS server load:''' Storage server overloaded and unable to respond
Work on these issues with your network administrator:
* Verify network stability with continuous monitoring tools
* Check switch logs for errors or port flapping
* Review firewall rules for NFS traffic (port 2049 TCP, plus RPC ports)
* Monitor NFS server performance and load
==== Step 5: Mitigation Strategies ====
To prevent future data loss due to NFS connectivity issues:
* '''Use local spool where possible:''' Store PCAPs locally and use VoIPmonitor Client-Server mode to send CDRs to a central server, rather than depending on NFS for real-time writes.
* '''Implement network monitoring:''' Set up alerts for NFS server availability and network packet loss.
* '''Consider dedicated storage:''' Use a storage solution designed for high availability and continuous access.
* '''NFS mount options:''' For improved reliability, consider adding NFS mount options like <code>soft,timeo=50,retrans=3</code> for faster timeout detection, though this may result in some I/O errors being reported earlier.
For local storage instead of NFS, see the [[#Relocating_the_Spool_Directory_to_a_Different_Partition|Relocating Spooldir]] and [[Sniffer_distributed_architecture|Distributed Architecture]] guides.
=== Troubleshooting: Configuration Not Applied (GUI Override) ===
If your <code>voipmonitor.conf</code> file shows <code>cleanspool = yes</code> but cleaning is still not running, the setting may be overridden by GUI-based configuration.
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Common Cause: GUI Settings Override File Configuration
|-
| style="vertical-align: top;" | '''Symptom:'''
| Configuration set correctly in <code>/etc/voipmonitor.conf</code> (e.g., <code>cleanspool = yes</code>) but the setting is not taking effect.
|-
| style="vertical-align: top;" | '''Root Cause:'''
| <code>mysqlloadconfig = yes</code> (default) loads configuration from the <code>sensor_config</code> database table, which stores GUI settings. These GUI settings override file-based configuration.
|}
==== How Configuration Priority Works ====
When <code>mysqlloadconfig</code> is enabled (which is the default), VoIPmonitor applies configuration in this order:
{| class="wikitable"
|-
! Priority !! Source !! Description
|-
| '''1 (Highest)''' || GUI Settings || <code>sensor_config</code> database table, configured via '''Settings > Sensors > wrench icon'''
|-
| '''2 (Medium)''' || Database || Values stored in database but not GUI-managed
|-
| '''3 (Lowest)''' || <code>/etc/voipmonitor.conf</code> || File-based configuration (only used if not set elsewhere)
|}
If a parameter is set in the GUI (<code>sensor_config</code> table), '''it will override the same parameter in <code>voipmonitor.conf</code>'''. The file is effectively ignored for that specific parameter.
==== Diagnosis: Check for GUI Configuration Override ====
<syntaxhighlight lang="bash">
# 1. Check if mysqlloadconfig is enabled
grep mysqlloadconfig /etc/voipmonitor.conf
# Should show: mysqlloadconfig = yes (this is the default)
# 2. Check your file-based cleanspool setting
grep cleanspool /etc/voipmonitor.conf
# Should show: cleanspool = yes
# 3. Verify sensor configuration in GUI
# Navigate to GUI > Settings > Sensors and click the wrench icon for your sensor
# Search for "cleanspool" in the sensor settings dialog
</syntaxhighlight>
If the GUI shows <code>cleanspool</code> set to a different value than your config file, the GUI setting is being used.
==== Resolution Options ====
You have three ways to resolve this conflict:
;Option 1: Update via GUI (Recommended)
Fix the configuration in the GUI where the sensor stores its settings:
# Log in to the VoIPmonitor GUI
# Navigate to '''Settings > Sensors'''
# Click the '''wrench icon''' for the sensor
# In the search field, type <code>cleanspool</code>
# Set <code>cleanspool</code> to <code>yes</code>
# Save the changes (GUI writes to <code>sensor_config</code> database table)
;Option 2: Disable Database Configuration
Disable <code>mysqlloadconfig</code> to rely entirely on file-based configuration:
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
# Disable loading config from database
mysqlloadconfig = no
</syntaxhighlight>
Apply changes:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
'''Note:''' With <code>mysqlloadconfig = no</code>, GUI sensor settings will no longer apply. You must configure everything in <code>voipmonitor.conf</code>.
;Option 3: Remove Conflicting Database Entry
Manually remove the conflicting setting from the database:
<syntaxhighlight lang="sql">
-- Connect to voipmonitor database
mysql -u root -p voipmonitor
-- Check current sensor_config entries
SELECT * FROM sensor_config WHERE name = 'cleanspool';
-- Delete the conflicting entry (this falls back to voipmonitor.conf)
DELETE FROM sensor_config WHERE name = 'cleanspool';
-- Verify the deletion
SELECT * FROM sensor_config WHERE name = 'cleanspool';
</syntaxhighlight>
Apply changes:
<syntaxhighlight lang="bash">
# Either restart the sensor or use GUI reload button
systemctl restart voipmonitor
</syntaxhighlight>
==== Why This Happens ====
The GUI's GUI sensor configuration feature allows administrators to manage sensor settings without editing config files. When you save settings via '''Settings > Sensors > wrench icon''', the GUI writes those values to the <code>sensor_config</code> table. The sensor reads these table values first before checking <code>voipmonitor.conf</code>.
This design provides flexibility (GUI-based configuration vs file-based) but can cause confusion if both sources contain conflicting values for the same parameter.
==== Example Troubleshooting Scenario ====
'''User report:''' "I have <code>cleanspool = yes</code> in <code>voipmonitor.conf</code>, but old files are not being deleted."
'''Investigation:'''
<syntaxhighlight lang="bash">
# 1. Check configuration file
grep cleanspool /etc/voipmonitor.conf
# Output: cleanspool = yes  ✓ Correct
# 2. Check if mysqlloadconfig is enabled
grep mysqlloadconfig /etc/voipmonitor.conf
# Output: mysqlloadconfig = yes  (default)  - CONFIG LOADED FROM DB
# 3. Check GUI Settings > Sensors > wrench icon > search "cleanspool"
# GUI shows: cleanspool = no  ✗ CONFLICT
</syntaxhighlight>
'''Root Cause:'''
* <code>mysqlloadconfig = yes</code> enables database-based configuration
* Previous GUI set <code>cleanspool = no</code> and stored it in <code>sensor_config</code> table
* Database value overrides file value, so <code>cleanspool</code> is disabled
'''Solution:'''
* Update setting via GUI (<code>Settings > Sensors > wrench > cleanspool = yes</code>)
* Or set <code>mysqlloadconfig = no</code> in <code>voipmonitor.conf</code>
==== Related Configuration Parameters ====
The following sniffer configuration parameters can be overridden by GUI sensor settings when <code>mysqlloadconfig = yes</code>:
* <code>cleanspool</code> - Enable/disable automatic file cleaning
* <code>maxpoolsize</code> - Maximum disk space for captured data
* <code>maxpooldays</code> - Maximum age for captured data
* <code>maxpoolrtpdays</code> - Maximum age for RTP files
* <code>maxpoolsipdays</code> - Maximum age for SIP files
* <code>spooldir</code> - Spool directory location
* Any other sensor-specific configuration parameters
For the complete sniffer configuration reference, see [[Sniffer_configuration|Sniffer Configuration]].
=== Custom Autocleaning: One-Time Cleanup with Filters ===
The GUI provides a '''custom autocleaning''' feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.
==== Use Case: Cleaning Old Recordings for a Specific IP ====
After configuring [[Capture_rules|capture rules]] to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.
Example scenario:
* You configured a capture rule to discard RTP for IP <code>192.168.1.50</code>
* Only new calls will be affected by this rule
* Existing recordings for this IP must be cleaned up manually
GUI cleanup steps:
# Navigate to '''Settings > Custom Autocleaning'''
# Create a new autocleaning rule
# Set the criteria (e.g., "Delete RTP older than 1 day")
# In the '''Common Filter''' section, specify the target IP address (<code>192.168.1.50</code>)
# Save and apply the rule
# Once the cleanup is complete, remove the autocleaning rule
This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.
==== Comparison with Global Retention ====
{| class="wikitable"
|-
! Feature !! Custom Autocleaning !! Global Retention
|-
| '''Scope''' || Targeted (specific IP, number, filter) || All calls
|-
| '''Purpose''' || One-time cleanup of existing data || Ongoing automated cleanup
|-
| '''Configuration''' || GUI with CDR filters || <code>maxpoolsize</code>, <code>maxpooldays</code>
|-
| '''Flexibility''' || High - can use any CDR filter criteria || Low - time/size only
|}
==== Filters Available in Custom Autocleaning ====
The custom autocleaning interface supports the same filter options as the CDR view, including:
* '''IP filters''' - Source or destination IP addresses
* '''Telephone number filters''' - Caller or called numbers
* '''Time-based filters''' - Calls older than X days
* '''Other CDR criteria''' - Duration, MOS, packet loss, etc.
For details on using these filters, see [[Call_Detail_Record_-_CDR#Filtering_Calls|CDR Filtering]].
==== Important Notes ====
* Custom autocleaning rules should typically be deleted after they have run once
* Leaving a rule active will continuously clean up matching data, which may not be desired
* The cleanup affects PCAP files stored on disk, not CDR records in the database
* For database cleanup, use the global <code>cleandatabase</code> parameters
=== How Cleanspool Works: Filesystem Index vs Database ===
It is important to understand that the <code>cleanspool</code> feature operates on the '''filesystem, not the database'''. This is a critical distinction for configuring data retention correctly.
==== Filesystem-Based Cleaning ====
The cleanspool process maintains its own index in the <code>.cleanspool_cache</code> file located in the spool directory. This filesystem index is completely independent of the VoIPmonitor database.
{| class="wikitable"
|-
! Aspect !! Cleanspool (Filesystem) !! Database Cleaning
|-
| '''Where it operates''' || Deletes PCAP files from disk || Deletes CDR records from MySQL
|-
| '''Index used''' || <code>.cleanspool_cache</code> (filesystem scan) || Database tables/partitions
|-
| '''Retention limits''' || <code>maxpoolsize</code>, <code>maxpooldays</code> || <code>cleandatabase</code>, <code>cleandatabase_size</code>
|-
| '''File selection criteria''' || Based on filesystem index, NOT database records || Based on <code>calldate</code> in cdr table
|}
==== Key Behavior: Files Not in Database ====
The cleanspool process will delete files based solely on the filesystem cache index and the <code>maxpoolsize</code>/<code>maxpooldays</code> limits. This means:
* Files '''will be deleted even if they are not in the database''' (orphaned PCAPs)
* Files '''will be deleted even if database records still exist''' (e.g., after database backup/restore)
* To protect specific data, you must move it '''outside the spool directory entirely''', not just to a different subfolder
==== The .cleanspool_cache File ====
When cleanspool runs, it scans the spool directory structure (<code>YYYY-MM-DD/HH</code>) and indexes all PCAP files into <code>.cleanspool_cache</code>. During the next cleaning cycle, it uses this cache to:
1. Sort files by age (oldest first)
2. Delete files until <code>maxpoolsize</code> or <code>maxpooldays</code> limits are met
Important behaviors:
* By default (<code>maxpool_clean_obsolete = no</code>), cleanspool only deletes files that are indexed in <code>.cleanspool_cache</code>
* Files that appear but are not in the cache are '''ignored''' until the next full scan updates the cache
* With <code>maxpool_clean_obsolete = yes</code>, cleanspool will also delete any files in the spool directory that are not in the cache
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Important: Protect Data Moves
|-
| style="vertical-align: top;" | '''<code>maxpool_clean_obsolete = no</code> (default):'''
|
If you manually move files into the spool directory, they will not be indexed immediately. Cleanspool will ignore them until a full scan updates the cache. To prevent deletion, move data outside the spool directory entirely, not to a different subfolder within the spool.
|}
==== Configuration Parameter Reference ====
; <code>maxpool_clean_obsolete = no</code> (default)
: Cleanspool only deletes files that are indexed in the <code>.cleanspool_cache</code> filesystem index. Files that are not in the cache are ignored.
; <code>maxpool_clean_obsolete = yes</code>
: Cleanspool will delete ALL files in the spool directory, including those not found in the cache. This is useful if you have manually moved files or want to clean orphaned files.
'''Warning:''' This parameter controls filesystem cache behavior, not database record matching. Cleanspool always operates independently of the database.
=== Maintenance: Re-indexing the Spool Directory ===
VoIPmonitor maintains an index of all created PCAP files to perform cleaning efficiently without scanning the entire directory tree. If this index becomes corrupt, or if you manually move files into the spool, old data may not be deleted correctly.
To trigger a manual re-index via the manager API:
<syntaxhighlight lang="bash">
# Open a manager API session
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
# Send the re-index command
echo reindexfiles | nc -U /tmp/vmsck
</syntaxhighlight>
Note: This command requires <code>netcat</code> with support for UNIX sockets (<code>-U</code>). For alternative methods, see the [[Encryption_in_manager_api_customer|Manager API documentation]].


== Tiered Storage and Archival Options ==
== Tiered Storage and Archival Options ==


If you need to extend PCAP retention beyond your fast local disk capacity, there are three recommended approaches to implement tiered storage with VoIPmonitor.
If you need to extend PCAP retention beyond your fast local disk capacity, there are three recommended approaches.


=== Option 1: Use tar_move Feature (Recommended for Automation) ===
=== Option 1: Use tar_move Feature (Recommended) ===


The <code>tar_move</code> feature allows automatic archival of PCAP files to secondary storage after the spool directory files are closed. This is ideal for moving data to a slower, larger storage device (NFS, HDD array, etc.) while keeping the fast local disk for recent captures.
The <code>tar_move</code> feature automatically archives PCAP files to secondary storage after they are closed.


==== Use Case: Preventing S3FS/Network Filesystem Startup Errors ====
If you are encountering <code>pcap_open_offline ... failed: unknown file format</code> errors when VoIPmonitor starts, and your <code>spooldir</code> points to a slow network filesystem (S3FS, NFS, SMB), the issue is caused by filesystem latency and inconsistent file metadata.
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Problem: Capturing directly to network storage causes startup errors
|-
! Symptoms
| VoIPmonitor logs <code>pcap_open_offline ... failed: unknown file format</code> on startup when <code>spooldir</code> is on S3FS or other slow network storage.
|-
! Root Cause
| Network filesystems like S3FS are not designed for high-frequency synchronous I/O required for live packet capture. Files may appear corrupted or have inconsistent metadata on startup due to filesystem latency.
|-
! Solution
| Use <code>tar_move</code> to separate '''live capture''' (local fast disk) from '''archival storage''' (slow network disk).
|}
To fix this issue:
;Step 1: Configure Local Fast Storage for Live Capture
Edit <code>/etc/voipmonitor.conf</code> to use a local SSD or NVMe partition for live capture:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Use local fast storage for live capture (avoids packet loss)
# Use local fast storage for live capture
spooldir = /var/spool/voipmonitor
spooldir = /var/spool/voipmonitor
</syntaxhighlight>


;Step 2: Enable tar_move for Network Archival
# Enable automatic archival
Add the following to <code>/etc/voipmonitor.conf</code>:
<syntaxhighlight lang="ini">
# Enable automatic archival after files are closed
tar_move = yes
tar_move = yes
 
tar_move_destination_path = /mnt/archive/voipmonitor
# Destination for archived files (your S3FS mount or network storage)
tar_move_destination_path = /mnt/s3fs/voipmonitor-archive
 
# Number of threads for moving files (optional, increase for faster archival)
tar_move_max_threads = 2
tar_move_max_threads = 2
</syntaxhighlight>
</syntaxhighlight>
;Step 3: Restart VoIPmonitor
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
'''How this works:'''
* VoIPmonitor writes PCAP files to the fast local <code>spooldir</code> during capture (zero latency, no packet loss)
* When a TAR file is closed (end of minute), <code>tar_move</code> automatically moves it to <code>tar_move_destination_path</code>
* On startup, VoIPmonitor only needs to scan the local spooldir, which is fast and reliable
* The S3FS/network storage is used only for long-term archival, not for live capture


{| class="wikitable"
{| class="wikitable"
|-
|-
! Parameter !! Description !! Example
! Parameter !! Description
|-
| <code>tar_move</code> || Options: <code>no</code> (disabled), <code>yes</code> (move and delete from spooldir), <code>copy</code> (copy with retention) || <code>tar_move = yes</code>
|-
| <code>tar_move_destination_path</code> || Target directory for archived TAR files. Can be any mounted filesystem (NFS, SMB, local HDD, S3FS, S3 via rclone). Must exist and be writable by voipmonitor user. || <code>tar_move_destination_path = /mnt/s3fs/voipmonitor-archive</code>
|-
| <code>tar_move_max_threads</code> || Number of parallel threads for moving files. Default: 2. Increase if you have high capture volume and need faster archival. || <code>tar_move_max_threads = 4</code>
|}
 
'''Important Notes:'''
* The <code>tar_move_destination_path</code> directory MUST be created before starting VoIPmonitor: <code>mkdir -p /mnt/s3fs/voipmonitor-archive && chown voipmonitor:voipmonitor /mnt/s3fs/voipmonitor-archive</code>
* For S3 cloud storage, <code>rclone</code> is recommended over <code>s3fs</code> for better performance (see [[#S3_Cloud_Storage_Considerations|S3 Considerations]] below)
* Files in <code>tar_move_destination_path</code> remain accessible via the GUI (VoIPmonitor searches both local spooldir and tar_move_destination_path)
* VoIPmonitor does NOT automatically delete files from <code>tar_move_destination_path</code>. You must implement an external cleanup mechanism (cron job) for the archive directory.
* If using network storage for <code>tar_move_destination_path</code>, ensure the mount is always available before starting VoIPmonitor
 
'''Accessing Moved Files from GUI:'''
When the VoIPmonitor GUI requests a PCAP file (e.g., user clicks play button for a call), the sniffer checks locations in this order:
1. Local spooldir (<code>/var/spool/voipmonitor</code>)
2. tar_move_destination_path (if configured and file not found locally)
 
This means files moved to <code>tar_move_destination_path</code> remain fully accessible through the GUI as long as the path is accessible to the sniffer process.
 
{| class="wikitable" style="background:#d6eaf8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | Important: S3 Cloud Storage Considerations
|-
! Issue
| When using <code>tar_move</code> with an S3 bucket mounted via <code>s3fs</code>, the manager interface (GUI) may become unresponsive when accessing archived files.
|-
! Root Cause
| The sniffer thread handles GUI file requests synchronously. If the file is in <code>tar_move_destination_path</code> (but not in local spooldir), the sniffer reads it directly from the mounted path. Because <code>s3fs</code> (FUSE over S3) has significantly higher latency than local disk or network storage, this synchronous I/O operation blocks the sniffer thread, causing GUI timeouts and unresponsiveness.
|-
! Recommended Solution
| Use <code>rclone</code> instead of <code>s3fs</code> to mount S3 buckets. <code>rclone</code> has better I/O characteristics for this use case and does not cause the same blocking behavior.
|-
! Working rclone Mount Command
| <syntaxhighlight lang="bash" inline>/usr/bin/rclone mount spr-prod0-voipmonitor /mnt1/spool-backup --allow-other --dir-cache-time 30s --poll-interval 0 --vfs-cache-mode off --buffer-size 0 --use-server-modtime --no-modtime --s3-no-head --log-level INFO</syntaxhighlight>
|-
! Why rclone Works Better
| <code>rclone</code> implements more efficient caching and non-blocking I/O for cloud storage mounts, preventing the sniffer from hanging during file access operations requested by the GUI.
|}
 
If you must use <code>s3fs</code>, expect GUI unresponsiveness when accessing archived files from S3 and consider a different archival strategy or local storage solution.
 
==== Multiple Sensors Sharing Central Storage ====
 
When deploying multiple VoIPmonitor sensors with limited local disk space that need to share a central storage device (NAS,SAN), you must use <code>tar_move_destination_path</code> with unique paths for each sensor to prevent filename collisions. VoIPmonitor generates PCAP filenames based on timestamps (e.g., <code>sip-2023-10-27-10:00:00.pcap</code>), so if multiple sensors write to the same directory without unique paths, files from different sensors will overwrite each other.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Critical: Do NOT Use spooldir on Network Storage
|-
! Warning
| '''NEVER''' set <code>spooldir</code> to point directly to network storage (NFS/SMB mounts). Writing PCAP files directly to network storage during capture causes increased CPU usage (sync I/O) and leads to '''packet loss''' on high-traffic networks. Always use local spool for live capture and <code>tar_move_destination_path</code> for archival to network storage.
|}
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | Recommended Architecture for Multiple Sensors
|-
! Approach
| Use local fast storage (SSD) for live capture via <code>spooldir</code> and <code>tar_move_destination_path</code> for moving files to shared central storage.
|-
! Why
| Avoids packet loss from network I/O during capture while centralizing long-term storage for all sensors.
|}
 
===== Architecture =====
 
<kroki lang="plantuml">
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #4A90E2
  BackgroundColor #FFFFFF
}
 
rectangle "Sensor 01" as sensor01
rectangle "Sensor 02" as sensor02
cloud "Central\nShared Storage" as storage
database "Central MySQL" as db
 
sensor01 --> storage : Writes to /sensor01/
sensor02 --> storage : Writes to /sensor02/
sensor01 --> db : CDRs (id=1)
sensor02 --> db : CDRs (id=2)
 
note right of sensor01
  Local spool: /var/spool/voipmonitor (live capture)
  tar_move_destination_path: /mnt/storage/sensor01 (archived)
end note
 
note right of sensor02
  Local spool: /var/spool/voipmonitor (live capture)
  tar_move_destination_path: /mnt/storage/sensor02 (archived)
end note
@enduml
</kroki>
 
===== Configuration Steps =====
 
;Step 1: Mount Central Storage on Each Sensor
 
On <strong>each sensor</strong>, mount the central network storage:
 
<syntaxhighlight lang="bash">
# On sensor01
mkdir -p /mnt/central_storage
mount central-storage.example.com:/voipmonitor /mnt/central_storage
 
# On sensor02
mkdir -p /mnt/central_storage
mount central-storage.example.com:/voipmonitor /mnt/central_storage
 
# Add to /etc/fstab for persistence on both sensors
central-storage.example.com:/voipmonitor /mnt/central_storage nfs defaults 0 0
</syntaxhighlight>
 
;Step 2: Create Unique Directories for Each Sensor
 
On the central storage, create separate directories for each sensor:
 
<syntaxhighlight lang="bash">
# On central storage server or via one of the sensors:
mkdir -p /mnt/central_storage/sensor01
mkdir -p /mnt/central_storage/sensor02
 
# Set ownership (voipmonitor user needs write access)
chown -R voipmonitor:voipmonitor /mnt/central_storage
</syntaxhighlight>
 
;Step 3: Configure Each Sensor's voipmonitor.conf
 
<strong>SENSOR 01 Configuration (<code>/etc/voipmonitor.conf</code>):</strong>
<syntaxhighlight lang="ini">
# Unique Sensor ID (must be different for every sensor)
id_sensor = 1
 
# Use LOCAL storage for live capture (avoids packet loss)
spooldir = /var/spool/voipmonitor
 
# CRITICAL: Unique directory for this sensor on central storage
tar_move = yes
tar_move_destination_path = /mnt/central_storage/sensor01
 
# Database configuration (shared central MySQL)
mysqlhost = central-db.example.com
mysqldb = voipmonitor
mysqluser = voipmonitor
mysqlpassword = secret
</syntaxhighlight>
 
<strong>SENSOR 02 Configuration (<code>/etc/voipmonitor.conf</code>):</strong>
<syntaxhighlight lang="ini">
# Unique Sensor ID
id_sensor = 2
 
# LOCAL storage for live capture
spooldir = /var/spool/voipmonitor
 
# CRITICAL: Different directory than Sensor 01 (prevents overwrites)
tar_move = yes
tar_move_destination_path = /mnt/central_storage/sensor02
 
# Database configuration (same as sensor 01)
mysqlhost = central-db.example.com
mysqldb = voipmonitor
mysqluser = voipmonitor
mysqlpassword = secret
</syntaxhighlight>
 
{| class="wikitable"
|-
! Component !! Sensor 01 !! Sensor 02 !! Purpose
|-
|-
| <code>id_sensor</code> || <code>1</code> || <code>2</code> || Unique identifier in database
| <code>tar_move = yes</code> || Move files to archive after capture completes
|-
|-
| <code>spooldir</code> || <code>/var/spool/voipmonitor</code> || <code>/var/spool/voipmonitor</code> || Local disk for live capture (avoid network I/O)
| <code>tar_move = copy</code> || Copy files (keep original)
|-
|-
| <code>tar_move_destination_path</code> || <code>/mnt/central_storage/sensor01</code> || <code>/mnt/central_storage/sensor02</code> || Unique shared storage path (prevents overwrites)
| <code>tar_move_destination_path</code> || Target directory for archived files
|}
|}


;Step 4: Restart Services
'''Important:''' Files in <code>tar_move_destination_path</code> remain accessible via GUI. VoIPmonitor searches both local spooldir and tar_move_destination_path.


Apply changes on all sensors:
==== S3 Cloud Storage Considerations ====
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>


===== How File Access Works =====
When using S3 storage, use <code>rclone</code> instead of <code>s3fs</code> to avoid GUI unresponsiveness:
 
When a VoIPmonitor GUI user requests a PCAP file for a call from <code>id_sensor = 1</code>:
 
1. The GUI sends the request to the sensor with <code>id_sensor = 1</code>
2. The sniffer checks its local <code>spooldir</code> first
3. If not found locally, it checks <code>tar_move_destination_path = /mnt/central_storage/sensor01</code>
4. The file is served from whichever location contains it (local or archived)
 
This provides seamless access to archived files without needing to configure "Additional spool directories" in the GUI.
 
===== Verification =====
 
Verify files are flowing correctly:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# On sensor01 - check local spooldir (quick lookups)
# Recommended: rclone mount
ls -lh /var/spool/voipmonitor
/usr/bin/rclone mount bucket-name /mnt/s3-archive \
 
  --allow-other --dir-cache-time 30s --poll-interval 0 \
# On sensor01 - check shared storage (archived files)
  --vfs-cache-mode off --buffer-size 0 --use-server-modtime \
ls -lh /mnt/central_storage/sensor01
  --no-modtime --s3-no-head --log-level INFO
 
# On sensor02 - verify separate directory
ls -lh /mnt/central_storage/sensor02
 
# On central MySQL - verify sensor IDs are unique
mysql -h central-db.example.com -u voipmonitor -p voipmonitor -e "SELECT DISTINCT id_sensor FROM cdr"
</syntaxhighlight>
</syntaxhighlight>
===== Common Pitfalls =====
{| class="wikitable" style="background:#f8d7da; border:1px solid #f5c6cb;"
|-
! colspan="2" style="background:#dc3545; color: white;" | Common Mistakes to Avoid
|-
! Setting <code>spooldir</code> to NFS/SMB directly
| This causes packet loss during high traffic as the capture thread waits for network I/O. Always use local <code>spooldir</code> for live capture.
|-
! Same <code>tar_move_destination_path</code> on multiple sensors
| Files from different sensors will overwrite each other because PCAP filenames are timestamp-based and identical across sensors.
|-
! Missing <code>id_sensor</code> uniqueness
| Without unique <code>id_sensor</code> values, database records cannot distinguish between sensors.
|-
! GUI accessing wrong sensor
| Make sure the GUI requests PCAPs from the correct sensor (based on <code>id_sensor</code> in the CDR). The sniffer only searches its own <code>tar_move_destination_path</code>.
|}
'''Prerequisites:'''
* The destination directory must exist and be accessible (e.g., mount NFS shares before starting VoIPmonitor)
* The voipmonitor service user must have read/write permissions on both directories
* For NFS mounts, ensure proper <code>/etc/fstab</code> configuration with appropriate options
* Verify connectivity: <code>mount | grep slow_storage</code>
See also: [[Sniffer_configuration#tar_move|Sniffer Configuration - tar_move parameters]]


=== Option 2: Manual Archival with Additional Spool Directories ===
=== Option 2: Manual Archival with Additional Spool Directories ===


Use a cron job or external script to manually move old PCAP files to secondary storage, then configure the GUI to access multiple directories.
Move old files manually and configure GUI to access multiple directories:
 
==== Moving Files Manually ====
 
Create a script to move files older than X days to the slower storage:
 
<syntaxhighlight lang="bash">
#!/bin/bash
# Archive old PCAP files to secondary storage
# This example keeps last 7 days on fast disk, moves older to slow storage
 
SPOOLDIR="/var/spool/voipmonitor"
ARCHIVEDIR="/mnt/slow_storage/archive"
 
# Find and move directories older than 7 days
find "$SPOOLDIR" -maxdepth 1 -type d -mtime +7 -exec mv -t "$ARCHIVEDIR" {} \;
 
echo "Archive completed: $(date)" >> /var/log/voipmonitor_archive.log
</syntaxhighlight>
 
Schedule with cron (run daily at 1 AM):
 
<syntaxhighlight lang="bash">
# Add to crontab -e
0 1 * * * /usr/local/bin/archive_spooldir.sh
</syntaxhighlight>
 
==== Accessing Archived Files in GUI ====
 
Archived files outside the main <code>spooldir</code> can still be accessed in the VoIPmonitor GUI by adding the path to the "Additional spool directories" setting.
 
1. Log in to the VoIPmonitor web GUI
2. Navigate to '''Settings > System configuration > Basic'''
3. Find the '''Additional spool directories''' field
4. Add the path to your archive directory (e.g., <code>/mnt/slow_storage/archive</code>)
5. Save changes
 
The GUI will now search both the main spool directory and any additional directories when displaying PCAP files.
 
{| class="wikitable" style="background-color: #E3F2FD;"
|-
! colspan="2" style="background: #1976D2; color: white;" | Important: File Permissions
|-
| style="vertical-align: top;" | '''Requirement:'''
|
The archive directory must be owned by the web-service user (typically <code>www-data</code> or <code>apache</code>) for GUI access.
 
<syntaxhighlight lang="bash">
# Ensure proper ownership
chown -R www-data:www-data /mnt/slow_storage/archive
 
# Set appropriate permissions
chmod -R 755 /mnt/slow_storage/archive
</syntaxhighlight>
|}
 
'''Limitations:'''
* Once files are moved to the archive directory, they are no longer subject to <code>cleanspool</code> automatic deletion
* You must implement a separate cleanup policy for the archive directory to prevent it from filling up
* Search performance may be slightly slower if scanning across multiple large directories
 
=== Option 3: LVM - Single Logical Volume (Recommended for Best Performance) ===
 
Combine two physical storage devices (e.g., fast SSD + large HDD) into a single logical volume using LVM (Logical Volume Manager). This allows the operating system to manage space across both disks transparently, and VoIPmonitor sees only one spool directory.
 
{| class="wikitable"
|-
! Pros !! Cons
|-
| Transparent to VoIPmonitor (no configuration needed) || Requires Linux LVM expertise
|-
| Automatic space balancing across disks || Degrades to slowest disk performance when full
|-
| Single spooldir - no archive management complexity || Requires careful monitoring of I/O performance
|}
 
'''Basic LVM Setup Example:'''
 
<syntaxhighlight lang="bash">
# 1. Create physical volumes from both disks
pvcreate /dev/sda1  # Fast SSD
pvcreate /dev/sdb1  # Large HDD
 
# 2. Create a volume group
vgcreate voipmonitor_vg /dev/sda1 /dev/sdb1
 
# 3. Create a logical volume spanning both disks
lvcreate -L 10T -n voipmonitor_lv voipmonitor_vg
 
# 4. Format with filesystem
mkfs.ext4 /dev/voipmonitor_vg/voipmonitor_lv
 
# 5. Mount as spooldir
mount /dev/voipmonitor_vg/voipmonitor_lv /var/spool/voipmonitor
 
# 6. Add to /etc/fstab for persistence
echo "/dev/voipmonitor_vg/voipmonitor_lv /var/spool/voipmonitor ext4 defaults 0 2" >> /etc/fstab
</syntaxhighlight>
 
For production use, consider LVM features like:
* '''Striping (RAID-0 style)''' for parallel I/O across disks
* '''Mirroring (RAID-1 style)''' for redundancy
* '''Dynamic resizing''' to add more disks without downtime


Note: This is a Linux system administration topic. LVM configuration is beyond the scope of VoIPmonitor documentation. Consult LVM guides for your distribution.
# Create archive script to move files older than X days
# Add archive path to GUI: '''Settings > System configuration > Basic > Additional spool directories'''


=== Comparison Summary ===
=== Option 3: LVM Single Logical Volume ===


{| class="wikitable"
Combine fast SSD + large HDD into a single logical volume using LVM. VoIPmonitor sees only one spool directory.
|-
! Option !! Complexity !! Automatic? !! Performance !! Best For
|-
| <code>tar_move</code> || Low || Yes || Good || Most deployments with NFS/array storage
|-
| Manual script + GUI || Medium || No (cron) || Good || Custom archival policies, regulatory compliance
|-
| LVM || High || Yes || Variable (mixed) || Single-volume simplicity, Linux expertise available
|}


== Database Cleaning (CDR Retention) ==
== Database Cleaning (CDR Retention) ==
Line 2,057: Line 1,002:
=== Partitioning Method (Recommended) ===
=== Partitioning Method (Recommended) ===


Since version 7, VoIPmonitor utilizes '''database partitioning''', which splits large tables into smaller, daily segments. This is the recommended method for managing database retention.
Since version 7, VoIPmonitor uses '''database partitioning''', which splits large tables into smaller, daily segments.


{| class="wikitable" style="width:100%;"
{| class="wikitable"
|-
|-
! Aspect !! Description
! Aspect !! Description
|-
|-
| '''How it works''' || Set <code>cleandatabase = 30</code> in <code>voipmonitor.conf</code> to keep the last 30 days of data.
| '''How it works''' || Set <code>cleandatabase = 30</code> to keep the last 30 days of data
|-
|-
| '''Why it's better''' || Dropping old partitions is instantaneous (milliseconds), regardless of row count. Zero database load.
| '''Why it's better''' || Dropping old partitions is instantaneous (milliseconds), regardless of row count
|-
|-
| '''Requirement''' || Partitioning is enabled by default on new installations.
| '''Partition limit''' || ~8000 partitions per table (~22 years with daily partitioning)
|}
 
==== Partition Limit Consideration ====
 
MySQL and MariaDB have a hard limit on the number of partitions that can be created for a single table. This limit is approximately '''8000 partitions per table'''.
 
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Important: Maximum Partitions
|-
|-
| style="vertical-align: top;" | '''MySQL/MariaDB Partition Limit:'''
| '''CDR record limit''' || No practical limit - modern installations use <code>BIGINT</code> for <code>cdr.ID</code> (up to 18 quintillion records). See [[Upgrade_to_bigint|Migrating to BIGINT]].
|
* Approximately '''8000 partitions maximum''' per table
* With daily partitioning, this equates to '''approximately 8000 days (~22 years)''' of retained data
* If you approach this limit, you cannot simply keep increasing retention indefinitely
|}
|}
For example, with daily partitioning enabled:
* <code>cleandatabase = 30</code> creates 30 active partitions (far below limit)
* <code>cleandatabase = 3650</code> creates 3650 active partitions (10 years of data)
* <code>cleandatabase = 8000</code> approaches the practical limit (~22 years of data)
'''Note on CDR Record Limits:''' While the partition limit constrains retention time, the number of CDR records is not limited by partitions. Modern VoIPmonitor installations use <code>BIGINT</code> for the <code>cdr.ID</code> column, which supports up to 18 quintillion records (far exceeding any practical deployment). For more details on the <code>ID</code> column type, see [[Upgrade_to_bigint|Migrating Database Primary Keys to BIGINT]].


==== Quick Start: Global Retention ====
==== Quick Start: Global Retention ====
For most deployments, configure one parameter in <code>voipmonitor.conf</code>:


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
Line 2,100: Line 1,023:
cleandatabase = 30
cleandatabase = 30
</syntaxhighlight>
</syntaxhighlight>
The <code>cleandatabase</code> parameter acts as a global default for all <code>cleandatabase_*</code> options and applies to:
* <code>cdr</code> - Call Detail Records
* <code>message</code> - SIP MESSAGE texts
* <code>sip_msg</code> - SIP OPTIONS/SUBSCRIBE/NOTIFY messages
* <code>register_state</code> - SIP registration states
* <code>register_failed</code> - Failed registration attempts


==== Retention Parameters ====
==== Retention Parameters ====
Line 2,114: Line 1,030:
! Parameter !! Default !! Description
! Parameter !! Default !! Description
|-
|-
| <code>cleandatabase</code> || 0 (disabled) || Master retention setting in days.
| <code>cleandatabase</code> || 0 (disabled) || Master retention setting in days
|-
|-
| <code>cleandatabase_cdr</code> || 0 || Specific retention for <code>cdr</code> and <code>message</code> tables.
| <code>cleandatabase_cdr</code> || 0 || Specific retention for CDR tables
|-
|-
| <code>cleandatabase_rtp_stat</code> || 2 || Retention for detailed RTP statistics.
| <code>cleandatabase_rtp_stat</code> || 2 || Retention for RTP statistics
|-
|-
| <code>cleandatabase_sip_msg</code> || 0 || Retention for OPTIONS/SUBSCRIBE/NOTIFY.
| <code>cleandatabase_sip_msg</code> || 0 || Retention for SIP messages
|-
|-
| <code>cleandatabase_register_failed = 0</code> || 0 || Retention for <code>register_failed</code> table.
| <code>cleandatabase_register_state</code> || 0 || Retention for registration states
|-
|-
| <code>cleandatabase_register_state = 0</code> || 0 || Retention for <code>register_state</code> table.
| <code>cleandatabase_register_time_info</code> || 0 || '''Must be set explicitly''' (not covered by global setting)
|-
|-
| <code>cleandatabase_register_time_info = 0</code> || 0 || Retention for <code>register_time_info</code> table.
| <code>cleandatabase_size</code> || (unset) || Size-based limit in MB
|-
|-
| <code>cleandatabase_size</code> || (unset) || Alternative: size-based limit in MB (requires version 2024.05.1+).
| <code>cleandatabase_size_force</code> || false || Required for size-based cleanup
|-
|-
| <code>cleandatabase_min_free_size</code> || (unset) || Optional trigger to start cleanup when free disk space drops below this threshold (in MB). Works as a safety net alongside other cleanup parameters.
| <code>partition_operations_enable_fromto</code> || 1-5 || Time window for partition operations
|-
| <code>partition_operations_enable_fromto</code> || 1-5 || Time window for partition operations (e.g., 1-5 AM).
|}
|}


===== Multi-Sensor CDR Retention Behavior =====
If you are running multiple VoIPmonitor sensors that write to the '''same database''', CDR retention behavior depends on which sensor manages database partition operations.
{| class="wikitable"
|-
! Deployment Mode !! Retention Behavior !! Configuration Impact
|-
| All sensors manage partitions independently (default) or single designated sensor manages partitions | The sensor with the '''lowest''' <code>cleandatabase</code> value will dictate CDR retention for '''all sensors''' connected to that database | If Sensor A has <code>cleandatabase = 30</code> and Sensor B has <code>cleandatabase = 90</code>, CDRs will be removed after 30 days for both sensors (the lowest value takes precedence)
|-
| Recommended: single designated sensor manages partitions | Same behavior (lowest <code>cleandatabase</code> wins) | Use <code>disable_partition_operations = yes</code> on all sensors except one, and configure the desired <code>cleandatabase</code> value on the designated sensor to avoid conflicts
|}
This design prevents inconsistent partition maintenance across sensors. All sensors connected to a shared database must have either:
* All sensors with the '''same''' <code>cleandatabase</code> value, OR
* A single designated sensor managing partitions with the desired retention value
For avoiding metadata table locks during partition operations, see [[#Multi-Sniffer_Environments:_Avoiding_Metadata_Table_Locks|Multi-Sniffer Environments]] section.
===== SIP Register Tables Cleanup =====
VoIPmonitor maintains several tables for SIP registration data when <code>sip-register = yes</code> is enabled in the configuration. Each registration table requires separate configuration for retention and partition cleanup.
{| class="wikitable"
|-
! Table !! Purpose !! Cleanup Parameter !! Default Retention
|-
| <code>register</code> || Current active registrations || No cleanup needed (temporary table) || N/A
|-
| <code>register_state</code> || Registration state history (REGISTER, UNREGISTER, EXPIRE events) || <code>cleandatabase_register_state</code> || Follows <code>cleandatabase</code> global setting
|-
| <code>register_failed</code> || Failed registration attempts || <code>cleandatabase_register_failed</code> || Follows <code>cleandatabase</code> global setting
|-
| <code>register_time_info</code> || Detailed registration timing metrics || <code>cleandatabase_register_time_info</code> || '''0 (disabled)'''
|}
{| class="wikitable" style="background-color: #FFF3CD;"
{| class="wikitable" style="background-color: #FFF3CD;"
|-
|-
! colspan="2" style="background: #856404; color: white;" | Important: register_time_info Table
! colspan="2" style="background: #856404; color: white;" | Important: register_time_info Table
|-
|-
| style="vertical-align: top;" | '''Critical Configuration Issue:'''
| style="vertical-align: top;" | '''Critical:'''
|
| The <code>register_time_info</code> table is NOT covered by the global <code>cleandatabase</code> setting. You MUST configure <code>cleandatabase_register_time_info</code> explicitly.
The <code>register_time_info</code> table is '''not covered by the global <code>cleandatabase</code> setting'''. This table must be configured explicitly with <code>cleandatabase_register_time_info</code>. If this parameter is not set (default 0), partitions for <code>register_time_info</code> will never be deleted, potentially causing excessive disk space consumption.
|}
|}
==== Enabling register_time_info Cleanup ====
To enable automatic cleanup of the <code>register_time_info</code> table partitions, add the configuration parameter to <code>/etc/voipmonitor.conf</code>:


<syntaxhighlight lang="ini">
==== Size-Based Database Cleaning ====
# Enable cleanup for register_time_info table (example: keep 7 days)
cleandatabase_register_time_info = 7
</syntaxhighlight>


The value specifies the retention period in days. Old partitions (older than the specified number of days) will be automatically dropped during the daily partition maintenance window.
Use <code>cleandatabase_size</code> to limit database by size rather than time:
 
==== Alternative: Global Setting for All Registration Tables ====
 
If you want to clean all registration-related tables with the same retention period, you can set all parameters explicitly:
 
<syntaxhighlight lang="ini">
# Keep registration data for 7 days
cleandatabase_register_state = 7
cleandatabase_register_failed = 7
cleandatabase_register_time_info = 7
 
# OR use the global setting (does NOT apply to register_time_info!)
cleandatabase = 7
</syntaxhighlight>
 
{{Warning|
Do not rely on the <code>cleandatabase</code> global setting to clean the <code>register_time_info</code> table. The global setting only applies to the tables explicitly listed in the documentation (<code>cdr</code>, <code>message</code>, <code>sip_msg</code>, <code>register_state</code>, <code>register_failed</code>). Always configure <code>cleandatabase_register_time_info</code> explicitly if the <code>register_time_info</code> table exists in your database.
}}
 
==== Verifying register_time_info Cleanup ====
 
After configuring <code>cleandatabase_register_time_info</code>, verify that partitions are being cleaned correctly:
 
<syntaxhighlight lang="sql">
-- Check current partitions for register_time_info
SELECT PARTITION_NAME,
      PARTITION_DESCRIPTION,
      TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'register_time_info'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;
</syntaxhighlight>
 
Monitor the system logs to confirm partition drop operations:
 
<syntaxhighlight lang="bash">
# Check for register_time_info partition operations in logs
journalctl -u voipmonitor | grep -i "register_time_info"
</syntaxhighlight>
 
For general information about the Register tables and their structure, see [[Register]].
 
===== Size-Based Database Cleaning =====
 
In addition to time-based retention, VoIPmonitor supports size-based database cleanup using <code>cleandatabase_size</code>. This is useful when you want to limit the database size regardless of data age.


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Limit database to 50 GB
# Limit database to 50 GB
cleandatabase_size = 51200
cleandatabase_size = 51200
# Force immediate cleanup if size exceeded (required for size-based)
cleandatabase_size_force = true
cleandatabase_size_force = true


# Optional: Start cleaning when disk free space drops below threshold (30 GB)
# Optional: Start cleaning when disk free space drops below threshold
cleandatabase_min_free_size = 30720
cleandatabase_min_free_size = 30720
</syntaxhighlight>
</syntaxhighlight>


{| class="wikitable"
==== Multi-Sniffer Environments ====
|-
! Parameter !! Description
|-
| <code>cleandatabase_size</code> || Maximum database size in MB. When exceeded, the oldest partitions are dropped until the size is below this limit.
|-
| <code>cleandatabase_size_force</code> || Must be set to <code>true</code> to enable size-based cleanup. Without this, <code>cleandatabase_size</code> will not trigger cleanup.
|-
| <code>cleandatabase_min_free_size</code> || Trigger cleanup when free disk space (in MB) drops below this threshold. Acts as a safety net to prevent the disk from filling up regardless of database age. Optional parameter that works with or without <code>cleandatabase_size</code>.
|-
| <code>mysqldatadir</code> || Path to the MySQL data directory. Required for remote databases when using size-based cleanup.
|}
 
'''Behavior:'''
* The size check is performed daily during partition operations
* When the size exceeds <code>cleandatabase_size</code> + 5% buffer, the oldest partitions are dropped
* <code>cleandatabase_min_free_size</code> provides an alternative trigger: when disk free space drops below this threshold, cleanup starts immediately
* This ensures the database stays within the specified size limit
* Works independently of <code>cleandatabase</code> (time-based retention)


=== Remote Database Considerations ===
When multiple sensors share the same database, partition operations should be managed by only ONE sensor:
 
If you are using a remote MySQL database (not localhost/127.0.0.1), additional configuration is required for size-based database cleaning to work correctly.
 
{| class="wikitable" style="background-color: #E3F2FD;"
|-
! colspan="2" style="background: #1976D2; color: white;" | Remote Database Setup for Size-Based Cleaning
|-
| style="vertical-align: top;" | '''Scenario 1: Local Database with Non-Local mysqlhost'''
|
If your database is physically local on the same server but configured with a hostname other than <code>localhost</code> or <code>127.0.0.1</code> (e.g., a Docker container or VM hostname):
 
<syntaxhighlight lang="ini">
mysqlhost = db-server.example.com
cleandatabase_size = 300000
cleandatabase_size_force = true
</syntaxhighlight>
|-
| style="vertical-align: top;" | '''Scenario 2: Truly Remote Database'''
|
If your database is on a separate server, use the <code>mysqldatadir</code> parameter to point to a remotely mounted copy of the database directory:
 
<syntaxhighlight lang="ini">
# Remote MySQL server
mysqlhost = remote-db.example.com
 
# Point to remotely mounted database directory
mysqldatadir = /mnt/remote-mysql/voipmonitor
 
# Size-based cleanup
cleandatabase_size = 300000
cleandatabase_size_force = true
</syntaxhighlight>
 
The <code>mysqldatadir</code> path must contain the actual database files, not be empty. For example, mount the remote MySQL data directory locally:
 
<syntaxhighlight lang="bash">
# Mount remote MySQL data directory (example using NFS)
mount -t nfs remote-db.example.com:/var/lib/mysql /mnt/remote-mysql
 
# Set permissions
chown voipmonitor:voipmonitor /mnt/remote-mysql/voipmonitor
</syntaxhighlight>
|}
 
'''Important Notes for Remote Databases:'''
* Size-based cleanup requires read access to the database directory on disk
* Network latency may affect cleanup timing
* For remote databases, size-based cleaning may be less reliable than time-based retention
 
'''Note:''' Size-based cleanup only affects partitioned tables. Non-partitioned tables are not affected.
 
More details: [[Sniffer_configuration#Database_Cleaning|Sniffer Configuration - Database Cleaning]].
 
===== Multi-Sniffer Environments: Avoiding Metadata Table Locks =====
 
If you are running multiple sniffer instances (or sensors) that write to the same database, partition operations on one instance can cause '''metadata table locks''' that affect all other instances, leading to:
* Blocked queries during partition creation/dropping
* Database unresponsiveness (high SQL queue values)
* GUI lag or packet capture disruption
 
To avoid this, partition operations should be managed by only '''one''' designated sniffer instance.
 
Configure your multi-sniffer deployment as follows:
 
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Important: One Sniffer for Partition Management
|-
! Instance !! Configuration
|-
| All sniffers '''except one'''
| <code>disable_partition_operations = yes</code>
|-
| Single designated sniffer
| <code>partition_operations_enable_fromto = 4-6</code>
|}


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
Line 2,352: Line 1,082:
</syntaxhighlight>
</syntaxhighlight>


This configuration ensures:
== Database Troubleshooting ==
* Only one instance handles partition creation/dropping
* The designated instance runs maintenance during off-peak hours
* Other instances continue capturing and writing data without interruption


'''Note:''' The designated sniffer must run continuously to manage partition structure.
=== MySQL Error 28: No Space Left on Device ===


=== Legacy Method: Manual Deletion (Not Recommended) ===
If MySQL crashes with Error 28 even when <code>cleandatabase</code> is configured:


For very old, non-partitioned databases, you would need custom scripts with <code>DELETE FROM cdr WHERE calldate < ...</code> queries.
'''Primary Solution - Enable Size-Based Cleaning:'''
 
<syntaxhighlight lang="ini">
'''Warning:''' Manual DELETE on large tables is extremely slow and resource-intensive. A single operation on millions of rows can take hours and impact GUI performance.
cleandatabase_size = 512000        # 500 GB limit
cleandatabase_size_force = true    # Required
</syntaxhighlight>


== Troubleshooting Disk Space Issues ==
'''Alternative Causes:'''
* '''Inode exhaustion:''' Check with <code>df -i</code>
* '''MySQL tmpdir full:''' Check with <code>mysql -e "SHOW VARIABLES LIKE 'tmpdir';"</code>


=== Disk Space Not Reclaimed After Cleanup ===
=== Disk Space Not Reclaimed After Cleanup ===


If automatic cleanup runs but disk space is not freed from the MySQL data directory, check the <code>innodb_file_per_table</code> setting:
Check <code>innodb_file_per_table</code> setting:


<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
Line 2,375: Line 1,106:
</syntaxhighlight>
</syntaxhighlight>


{| class="wikitable"
If OFF, space is not reclaimed when partitions are dropped. Enable for future tables:
|-
! Value !! Behavior
|-
| '''ON''' || Each table/partition has its own <code>.ibd</code> file. Dropping partitions reclaims space immediately.
|-
| '''OFF''' || All data in shared <code>ibdata1</code> file. Dropping partitions does '''not''' reduce file size.
|}
 
==== Solutions ====
 
;Option 1: Enable for Future Tables
Add to <code>/etc/my.cnf</code> or <code>/etc/mysql/my.cnf</code>:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
[mysqld]
[mysqld]
innodb_file_per_table = 1
innodb_file_per_table = 1
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="bash">
systemctl restart mysql
</syntaxhighlight>
Note: This only affects NEW tables/partitions. Existing data in <code>ibdata1</code> remains.
;Option 2: Reclaim Space from Existing Tables
<syntaxhighlight lang="sql">
OPTIMIZE TABLE cdr;
</syntaxhighlight>
'''Warning:''' Requires significant free disk space to duplicate table data. May crash if disk is nearly full.
;Option 3: Export and Re-import
<syntaxhighlight lang="bash">
mysqldump -u root -p voipmonitor > voipmonitor_backup.sql
mysql -u root -p -e "DROP DATABASE voipmonitor; CREATE DATABASE voipmonitor;"
mysql -u root -p voipmonitor < voipmonitor_backup.sql
</syntaxhighlight>
=== MySQL Error 28: No Space Left on Device ===
If MySQL repeatedly crashes with "No space left on device" (Error 28) even when standard <code>cleandatabase</code> retention settings are in place, the database is likely filling up faster than time-based retention can clean it.
{| class="wikitable" style="background-color: #FFF3CD;"
|-
! colspan="2" style="background: #856404; color: white;" | Critical: Time-Based vs Size-Based Retention
|-
| style="vertical-align: top;" | '''<code>cleandatabase = 30</code> (Time-based):'''
| Keeps data for 30 days. If call volume is high, the database may reach disk capacity well before reaching the 30-day limit.
|-
| style="vertical-align: top;" | '''<code>cleandatabase_size</code> (Size-based):'''
| Limits database to a specific size in MB. Oldest partitions are dropped immediately when the size limit is exceeded, preventing disk exhaustion regardless of data age.
|}
==== Diagnosis: Check Disk and Database Size ====
Identify which partition is full and check database size:
<syntaxhighlight lang="bash">
# 1. Check all partitions to find the full one
df -h
# Example output:
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1        50G  48G  2.0G  97% /
# /dev/sdb1      1.0T  400G  600G  40% /var/lib/mysql
# 2. Check database directory size
du -hs /var/lib/mysql
# 3. Check spool directory size (separate from database)
du -hs /var/spool/voipmonitor
</syntaxhighlight>
If <code>/var/lib/mysql</code> is nearly full but other partitions have free space, proceed to size-based database cleaning.


==== Alternative Causes When Standard Disk Checks Show Free Space ====
=== Database Not Cleaning (Verify Partitioning) ===


If <code>df -h</code> on the main partition shows free space but MySQL still reports Error 28, investigate these common root causes:
Before assuming cleaning is broken, verify tables are partitioned:
 
===== 1. Inode Exhaustion (df -i) =====
 
Inode exhaustion occurs when the partition has free disk space but has run out of file handles (inodes). This commonly happens on partitions with many small files (e.g., web servers with cached images, email servers with many messages). MySQL Error 28 can occur even with terabytes of free space if inodes are exhausted.
 
{| class="wikitable" style="background-color: #FFDDBB;"
|-
! colspan="2" style="background: #D9534F; color: white;" | Important: Check Inode Usage
|-
| style="vertical-align: top;" | '''When to check:'''
| Standard <code>df -h</code> shows available space but MySQL still returns "Got error 28 from storage engine"
|-
| style="vertical-align: top;" | '''Symptoms:'''
| MySQL intermittently fails during CREATE TEMPORARY TABLE or ALTER TABLE operations
|}
 
Check inode usage:
 
<syntaxhighlight lang="bash">
# Check inode usage (IUse column shows percentage)
df -i
 
# Example output showing inode exhaustion:
# Filesystem    Inodes IUsed IFree IUse% Mounted on
# /dev/sda1    1310720 1310719      1  100% /
# /dev/sdb1      524288  120000 404288  23% /var/lib/mysql
</syntaxhighlight>
 
If <code>IUse%</code> is at 100% or near 100% (inodes exhausted):
 
{| class="wikitable"
|-
! Resolution Option !! When to Use !! Procedure
|-
| Clean up excessive small files | Temporary fix, buys time | Find and delete directories with many small files (cache directories, log directories, etc.)
|-
| Reformat partition with higher inode count | Permanent fix, requires downtime | Recreate filesystem with <code>mke2fs -T news /dev/sdXN</code> or specify higher inode ratio with <code>-i size</code>
|-
| Move to new partition with larger inode limit | When downtime cannot be tolerated | Create new filesystem with optimized inode settings, migrate data, update mount points
|}
 
To find directories consuming the most inodes:
 
<syntaxhighlight lang="bash">
# Find top 20 directories by inode count
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -rn | head -20
</syntaxhighlight>
 
===== 2. MySQL Temporary Directory (tmpdir) =====
 
MySQL may use a separate temporary directory (different from <code>/var/lib/mysql</code>) for temporary tables, sorting operations, and file sorts. If this directory is on a different partition that becomes full, MySQL returns Error 28 even when the main data partition has space.
 
{| class="wikitable" style="background-color: #D6EAF8;"
|-
! colspan="2" style="background: #1976D2; color: white;" | MySQL tmpdir Configuration
|-
| style="vertical-align: top;" | '''Default location:'''
| <code>/tmp</code> (system temporary directory)
|-
| style="vertical-align: top;" | '''When checked:'''
| Inode exhaustion or partition mismatch when <code>df -h</code> shows free space
|-
| style="vertical-align: top;" | '''Common scenarios:'''
| * <code>/tmp</code> is on a separate small partition
  * Custom <code>tmpdir</code> pointing to a full partition
  * Large temporary tables during complex queries
|}
 
Check MySQL temporary directory configuration:
 
<syntaxhighlight lang="bash">
# 1. Check MySQL tmpdir setting
mysql -u root -p -e "SHOW VARIABLES LIKE 'tmpdir';"
 
# 2. Check disk space on the tmpdir partition
# (Replace /tmp with the path returned by the command above)
df -h /tmp
 
# 3. Check inode usage on tmpdir partition
df -i /tmp
</syntaxhighlight>
 
If the tmpdir partition is full:
 
{| class="wikitable"
|-
! Resolution Option !! Procedure
|-
| Clean tmpdir | Remove old temporary files (carefully - only files not in use by MySQL): <code>find /tmp -name "sql_*" -mtime +1 -delete</code>
|-
| Move tmpdir to larger partition | Set custom tmpdir in MySQL configuration
|-
| Increase tmpdir partition | Expand the filesystem hosting <code>/tmp</code> or custom tmpdir
|}
 
To change MySQL tmpdir location:
 
<syntaxhighlight lang="bash">
# 1. Edit MySQL configuration
sudo nano /etc/my.cnf
# or
sudo nano /etc/mysql/my.cnf
</syntaxhighlight>
 
Add to <code>[mysqld]</code> section:
 
<syntaxhighlight lang="ini">
[mysqld]
# Set temporary directory to a partition with sufficient space
tmpdir = /mnt/large_partition/mysql_tmp
</syntaxhighlight>
 
Create the directory and set permissions:
 
<syntaxhighlight lang="bash">
# Create the directory
sudo mkdir -p /mnt/large_partition/mysql_tmp
 
# Set ownership to mysql user
sudo chown mysql:mysql /mnt/large_partition/mysql_tmp
 
# Set appropriate permissions
sudo chmod 770 /mnt/large_partition/mysql_tmp
 
# Restart MySQL
systemctl restart mysql
</syntaxhighlight>
 
{| class="wikitable" style="background-color: #FFE6CC;"
|-
! colspan="2" style="background: #E67E22; color: white;" | Security: tmpdir Permissions
|-
| style="vertical-align: top;" | '''Important:'''
| Always ensure <code>tmpdir</code> is owned by <code>mysql:mysql</code> and has restricted permissions (770 or 750). Never use a world-writable directory. MySQL uses temporary files for database operations, and insecure permissions could pose security risks.
|}
 
==== Solution: Enable Size-Based Database Cleaning ====
 
Use <code>cleandatabase_size</code> to control database growth by size rather than time.
 
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
 
# Set maximum database size in MB (example: 500 GB = 512000 MB)
cleandatabase_size = 512000
 
# CRITICAL: Must be set to true for size-based cleanup to work
cleandatabase_size_force = true
 
# Optional: Keep time-based retention as a fallback
cleandatabase = 30
</syntaxhighlight>
 
{| class="wikitable"
|-
! Parameter !! Purpose !! Required for Size-Based?
|-
| <code>cleandatabase_size</code> || Maximum database size in MB || Yes
|-
| <code>cleandatabase_size_force</code> || Enables size-based cleanup || Yes (MUST be true)
|}
 
Apply changes:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
 
'''Behavior:'''
* Size check runs daily during partition operations
* When database exceeds <code>cleandatabase_size</code> + 5% buffer, oldest partitions are dropped until under limit
* Works independently of <code>cleandatabase</code> time-based retention
* Ensures database never grows beyond specified size limit
 
==== Additional Resolution: Move Database to Larger Partition ====
 
If the database partition cannot accommodate your data requirements, move MySQL to a larger partition.
 
<syntaxhighlight lang="bash">
# 1. Stop MySQL
systemctl stop mysql
 
# 2. Move database files to new location
mv /var/lib/mysql /mnt/storage/mysql
 
# 3. Edit MySQL configuration
# Edit /etc/mysql/my.cnf or /etc/my.cnf
[mysqld]
datadir=/mnt/storage/mysql
 
# 4. Update AppArmor (Ubuntu/Debian) or SELinux (RHEL/CentOS)
# Ubuntu/Debian:
echo '/mnt/storage/mysql/** rwk,' >> /etc/apparmor.d/local/usr.sbin.mysqld
systemctl reload apparmor
 
# RHEL/CentOS:
semanage fcontext -a -t mysqld_db_t "/mnt/storage/mysql(/.*)?"
restorecon -Rv /mnt/storage/mysql
 
# 5. Start MySQL
systemctl start mysql
</syntaxhighlight>
 
=== Troubleshooting: Database Not Cleaning (Partition Verification) ===
 
If your database fills up even though `<code>cleandatabase</code>` is enabled in the configuration, the root cause may be '''data volume exceeding expectations, not a cleaning failure'''. Before assuming cleaning is broken, verify that partitioning is working and check how much data exists in the retained time window.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Common Misdiagnosis: Cleaning Broken vs Data Volume Too High
|-
! Symptom
| Database fills disk space even with `<code>cleandatabase = 30</code>` (or similar) configured
|-
! Common Misdiagnosis
| "Cleaning is not working - partitions are not being dropped"
|-
! Actual Cause
| Cleaning IS working correctly, but it cannot keep up because you have high call volume that exceeds the configured retention capacity
|-
! Fix
| Either shorten the retention period (`<code>cleandatabase</code>`) or switch to size-based cleaning (`<code>cleandatabase_size</code>`)
|}
 
==== Step 1: Verify Tables Are Partitioned ====
 
First, confirm that your large tables are actually partitioned. If tables are not partitioned, the automatic cleaning mechanism will not work:


<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
-- Check partitioning status for CDR table
-- Check if CDR table is partitioned
SHOW CREATE TABLE cdr\G
SHOW CREATE TABLE cdr\G


-- Expected output should show PARTITION BY RANGE (calldate) or similar:
-- Check partition list and row counts
-- Create Table: cdr
-- ...
-- PARTITION BY RANGE (TO_DAYS(calldate))
-- (PARTITION p20250101 VALUES LESS THAN (TO_DAYS('2025-01-01')) ENGINE = ...)
 
-- Check other large tables if needed
SHOW CREATE TABLE sip_msg\G
SHOW CREATE TABLE register_state\G
</syntaxhighlight>
 
If you do NOT see partition information (only standard table definition), your database is not partitioned. This can happen on very old installations. In this case, automatic partition-based cleaning will not work, and you must either:
 
* Enable partitioning (see [[Database#Upgrading_to_Partitioned_Tables|Database Partitioning Upgrade]])
* Fall back to manual `DELETE` queries (not recommended for large tables)
 
==== Step 2: Check Record Counts in Retained Partitions ====
 
If tables are correctly partitioned, check how many records exist in the time window that should be retained. This will tell you whether the issue is:
 
* '''Cleaning not working''' (old partitions still exist)
* '''Cleaning working but volume too high''' (only expected partitions exist, but they contain massive amounts of data)
 
<syntaxhighlight lang="sql">
-- Check how many partitions exist for cdr table
SELECT PARTITION_NAME,
      PARTITION_DESCRIPTION,
      TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_SCHEMA = 'voipmonitor'
  AND TABLE_NAME = 'cdr'
  AND PARTITION_NAME IS NOT NULL
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 20;
</syntaxhighlight>
 
{| class="wikitable"
|-
! Partition !! Description !! Table Rows !! What This Means
|-
| <code>p20250107</code> || 2025-01-07 || 5,000,000 || Today/ Yesterday (expected)
|-
| <code>p20250106</code> || 2025-01-06 || 4,800,000 || 1 day ago (expected)
|-
| <code>p20250101</code> || 2025-01-01 || 5,200,000 || 6 days ago (expected with `<code>cleandatabase = 7</code>`)
|-
| <code>p20241231</code> || 2024-12-31 || 5,100,000 || ✗ '''OLD - should be dropped if retention is 7 days'''
|}
 
If you see partitions older than your configured `<code>cleandatabase</code>` setting, then cleaning is genuinely not working. See [[#Troubleshooting:_Configuration_Not_Applied_(GUI_Override)|Configuration Not Applied]] section.
 
If you see ONLY the expected number of partitions (e.g., partitions for the last 7 days when `<code>cleandatabase = 7</code>), then cleaning IS working correctly.
 
==== Step 3: Diagnose Data Volume vs Cleaning Failure ====
 
If the partition list shows only expected recent partitions but the database is still filling disk space:
 
<syntaxhighlight lang="sql">
-- Count total records in ALL current partitions
SELECT COUNT(*) FROM cdr;
 
-- Count records for a specific large table
SELECT COUNT(*) FROM sip_msg;
</syntaxhighlight>
 
{| class="wikitable"
|-
! Result !! Diagnosis !! Resolution
|-
| Record count is low (< 1 million per day) || May be unrelated issue (check `innodb_file_per_table`, partition operations) || See [[#Disk_Space_Not_Reclaimed_After_Cleanup|Disk Space Not Reclaimed]]
|-
| Record count is high (millions per day) AND only recent partitions exist || '''Cleaning IS working - data volume exceeds retention capacity''' || Reduce `<code>cleandatabase</code>` or use `<code>cleandatabase_size</code>`
|-
| Old partitions exist (older than `<code>cleandatabase</code>`) || Cleaning is NOT working (configuration issue) || Check GUI override, configuration priority
|}
 
==== Example: High Volume Diagnosis =====
 
'''Scenario:''' User reports: "My database fills disk space even though `<code>cleandatabase = 7</code>` is set. Cleaning must be broken."
 
'''Investigation:'''
<syntaxhighlight lang="sql">
-- Step 1: Verify partitioning
SHOW CREATE TABLE cdr\G
-- Result: Partitioned by RANGE (calldate) ✓
 
-- Step 2: Check partition list
SELECT PARTITION_NAME, TABLE_ROWS
SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
FROM information_schema.PARTITIONS
Line 2,773: Line 1,126:
ORDER BY PARTITION_ORDINAL_POSITION DESC
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;
LIMIT 10;
-- Result shows ONLY partitions for last 7 days:
-- p20250107: 8,000,000 rows
-- p20250106: 7,900,000 rows
-- p20250105: 8,100,000 rows
-- ... (total 7 partitions only)
-- Step 3: Check total records
SELECT COUNT(*) FROM cdr;
-- Result: 56,000,000 total records (8 million x 7 days)
</syntaxhighlight>
</syntaxhighlight>


'''Diagnosis:'''
If only expected partitions exist (matching your <code>cleandatabase</code> setting), cleaning IS working - you may simply have high data volume.
* Cleaning IS working - only last 7 days of partitions exist (no old partitions)
* Each day has 8 million CDR records - this is extremely high volume
* 8 million records/day × 7 days = 56 million total records
* Total database disk usage may already exceed capacity with just 7 days of data
 
'''Resolution Options:'''
 
*Option 1: Reduce retention period*
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
cleandatabase = 3  # Keep only 3 days instead of 7
</syntaxhighlight>
 
*Option 2: Switch to size-based cleaning*
<syntaxhighlight lang="ini">
# Edit /etc/voipmonitor.conf
cleandatabase_size = 100000        # 100 GB limit
cleandatabase_size_force = true    # Required for size-based cleaning
</syntaxhighlight>
 
*Option 3: Reduce data collection rate*
* Disable optional logging parameters
* Increase `<code>savertp</code>` to `<code>header</code>` for RTP-only storage
* Adjust capture rules to reduce data collected
 
=== Monitoring Database Health ===
 
==== SQL Queue Metrics ====
 
The sensor tracks queue metrics visible in GUI → Settings → Sensors → Status:
 
{| class="wikitable"
|-
! Metric !! Description !! Healthy Range
|-
| '''SQLq''' || CDRs waiting to be written to database || Near 0, sporadic spikes OK
|-
| '''SQLf''' || Failed database write attempts || Zero (not growing)
|}
 
* Consistently high/growing SQLq → database cannot keep up
* Non-zero/growing SQLf → database errors or connectivity issues
 
See [[SQL_queue_is_growing_in_a_peaktime|SQL Queue Troubleshooting]] for details.
 
==== System Monitoring ====
 
<syntaxhighlight lang="bash">
# Check system load
cat /proc/loadavg
 
# Monitor disk I/O (shows only active processes)
iotop -o
</syntaxhighlight>
 
High I/O from <code>mysqld</code> processes may indicate slow storage or poorly tuned MySQL settings.


== MySQL Performance Settings ==
== MySQL Performance Settings ==


For high-performance operation with partitioning:
For high-performance operation:


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
Line 2,858: Line 1,145:
innodb_file_per_table = 1
innodb_file_per_table = 1
</syntaxhighlight>
</syntaxhighlight>
For comprehensive tuning, see [[Scaling|Scaling and Performance Guide]].


== See Also ==
== See Also ==
Line 2,869: Line 1,154:


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' VoIPmonitor has multiple data retention mechanisms: (1) Filesystem cleaning for PCAP files using <code>maxpoolsize</code>/<code>maxpooldays</code> parameters, running every 5 minutes to delete oldest data first; (2) Database cleaning using <code>cleandatabase</code> parameter with daily partitioning for instant partition drops. '''Database size is theoretically unlimited but limited by MySQL/MariaDB partition limit of approximately 8000 partitions per table (~8000 days or ~22 years with daily partitioning)'''. CDR record count is not limited by partitions - modern installations use <code>BIGINT</code> for the <code>cdr.ID</code> column supporting up to 18 quintillion records (see [[Upgrade_to_bigint]]). (3) Custom autocleaning in GUI for one-time targeted cleanup of specific recordings based on filters (IP, phone number, duration, etc.). Key behavior: directories within <code>maxpoolrtpdays</code> window contain both SIP and RTP (large), while directories within <code>maxpoolsipdays</code> but beyond <code>maxpoolrtpdays</code> contain only SIP (smaller) - this is expected. '''Unexpected increase in disk space used by CDRs in /log partition: If you notice unexpected disk space changes for PCAP files (often on /log partition where spooldir is mounted), check <code>maxpoolsipdays</code> and <code>maxpoolrtpdays</code> settings. Directory size differences are expected behavior: directories within maxpoolrtpdays window contain both SIP and RTP (much larger), while directories within maxpoolsipdays but outside maxpoolrtpdays contain only SIP (smaller). This is NOT an error - it is the expected result of having different retention periods for different data types.''' Custom autocleaning is useful for cleaning up old recordings for specific IPs after configuring capture rules to stop future recording (capture rules only affect new calls, existing recordings must be cleaned separately). Cleanspool operates on the filesystem, NOT the database. The cleanspool process maintains its own index in <code>.cleanspool_cache</code> file and deletes files based solely on this filesystem index and <code>maxpoolsize</code>/<code>maxpooldays</code> limits. Files are deleted even if they are not in the database. The <code>maxpool_clean_obsolete</code> parameter (default: no) controls whether cleanspool deletes files not found in the cache. Troubleshooting covers disk full scenarios (check with <code>df -h</code> and <code>du -hs</code>, analyze which partition is full), space not reclaimed issues (<code>innodb_file_per_table</code> setting), and database health monitoring (SQLq/SQLf metrics). '''Spool directory filling exponentially after disk swap or MySQL upgrade: diagnose with <code>tail -f /var/log/syslog | grep voipmonitor | grep SQLq</code> - a growing SQL queue indicates database bottleneck. Solution: tune MySQL InnoDB settings: <code>innodb_flush_log_at_trx_commit = 2</code> (writes to OS cache every second instead of disk on every commit, 5-10x faster) and <code>innodb_buffer_pool_size</code> to 50-70% of available RAM (reduces disk I/O by caching data). Default settings (commit=1, buffer=128M) are too slow for production. Common after disk swaps which reset MySQL configuration. Check settings with <code>SHOW VARIABLES LIKE 'innodb_*'</code>.''' '''MySQL Error 28 "No space left on device" troubleshooting: when standard <code>cleandatabase</code> time-based retention fails, use <code>cleandatabase_size</code> and <code>cleandatabase_size_force = true</code> for size-based database cleaning as the primary solution'''. Additional topics: relocating spooldir to different partition, syncing GUI <code>SNIFFER_DATA_PATH</code> configuration, size-based database cleaning via <code>cleandatabase_size</code> and <code>cleandatabase_size_force</code>, saving only RTP headers with <code>savertp=header</code> (up to 90% storage reduction), critical architecture consideration for <code>packetbuffer_sender</code> mode (apply settings on central server, not sensors). A new section explains enabling selective audio recording with capture rules: set global <code>savertp=header</code> to save only headers, then create GUI capture rules with <code>recordRTP=ON</code> for exceptions (e.g., legal holds, VIP customers) - this allows compliance with data retention laws while meeting specific legal/business requirements. Emergency cleanup triggers: <code>autocleanspoolminpercent</code> (default 1%, triggers at 99% disk usage) and <code>autocleanmingb</code> (default 5GB, triggers when free space below 5GB) can override retention settings, causing premature file deletion even when <code>maxpooldays</code> is set correctly. GUI configuration override: when <code>mysqlloadconfig = yes</code> (default), GUI settings in the <code>sensor_config</code> database table override <code>voipmonitor.conf</code> file settings. If <code>cleanspool = yes</code> in config file but cleaning is not running, check GUI Settings > Sensors > wrench icon for conflicting values. Resolution options: update via GUI, disable <code>mysqlloadconfig</code>, or remove conflicting entry from <code>sensor_config</code> table. '''For tiered storage to extend retention beyond fast disk capacity, use the <code>tar_move</code> feature to automatically archive PCAP files to secondary storage after collection, or manually move files and add the archive path to GUI Settings > System configuration > Basic > Additional spool directories. Alternatively, use LVM to combine fast and slow disks into a single logical volume.'''
'''IMPORTANT: Files moved outside the main spooldir using manual archival CAN still be accessed in the GUI by adding the archive directory path to "Additional spool directories" setting, contrary to some outdated documentation.''' '''S3 cloud storage for tar_move: When using <code>tar_move</code> with an S3 bucket mounted via <code>s3fs</code>, the manager interface (GUI) may become unresponsive when accessing archived files. This occurs because the sniffer thread handles GUI file requests synchronously, and <code>s3fs</code> (FUSE over S3) has significantly higher latency than local disk, causing the sniffer thread to block during file access operations. Recommended solution: use <code>rclone</code> instead of <code>s3fs</code> to mount S3 buckets. Working rclone mount command: <code>/usr/bin/rclone mount spr-prod0-voipmonitor /mnt1/spool-backup --allow-other --dir-cache-time 30s --poll-interval 0 --vfs-cache-mode off --buffer-size 0 --use-server-modtime --no-modtime --s3-no-head --log-level INFO</code>.''' '''S3FS/Network Filesystem startup errors: If encountering <code>pcap_open_offline ... failed: unknown file format</code> errors when VoIPmonitor starts and <code>spooldir</code> points to S3FS, NFS, or SMB, use <code>tar_move</code> to separate live capture (local fast disk) from archival storage (slow network disk).''' '''Troubleshooting PCAP rotation: use the 5-step diagnostic checklist to systematically investigate fast PCAP rotation issues before applying fixes. The checklist includes: (1) Check retention configuration with <code>cat /etc/voipmonitor.conf | grep -E '^spooldir|^maxpool[^_]|^cleandatabase'</code> - shows spooldir location, maxpoolsize, maxpooldays, cleandatabase settings; (2) Measure total spool usage with <code>du -hs /var/spool/voipmonitor</code>; (3) Analyze usage patterns with per-day <code>du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2</code> and per-hour <code>du -h --max-depth=1 /var/spool/voipmonitor/YYYY-MM-DD | sort -k2,2</code> - identify peak hours and growth rate; (4) Check database size contribution with <code>du -hs /var/lib/mysql</code>; (5) Check overall disk capacity with <code>df -h</code>.'''


'''Keywords:''' data retention, cleaning, delete old calls, disk space, spooldir, maxpoolsize, maxpooldays, maxpoolsipdays, maxpoolrtpdays, cleandatabase, partitioning, cleandatabase_size, cleandatabase_size_force, reindexfiles, innodb_file_per_table, SQLq, SQLf, directory size difference, SIP vs RTP retention, relocate spooldir, SNIFFER_DATA_PATH, MySQL Error 28, savertp header, packetbuffer_sender, client server mode, distributed architecture, autocleanspoolminpercent, autocleanmingb, emergency cleanup, retention not working, premature deletion, cleanspool_cache, maxpool_clean_obsolete, filesystem index, database vs filesystem, orphaned files, custom autocleaning, one-time cleanup, targeted cleanup, IP filter cleanup, clean specific IP recordings, mysqlloadconfig, sensor_config, GUI configuration, configuration override, database override, Settings > Sensors, wrench icon, config priority, partition operations, metadata table locks, multiple sensors, disable_partition_operations, partition_operations_enable_fromto, unresponsive database, distributed architecture partition management, maxpoolsize buffer, disk capacity buffer, cleanspool_enable_fromto, restrict cleaning hours, spool directory fill up despite autoclean, cleaning time window, selective audio recording, capture rules, recordRTP, legal holds, VIP customers, compliance exceptions, low spool disk space message, automatic maxpoolsize adjustment, monitoring clean messages, clean_maxpooldays, clean_maxpoolsize, journalctl monitoring, syslog triggers, tiered storage, hot cold storage, archive old files, tar_move, tar_move_destination_path, additional spool directories, Settings > System configuration > Basic, manual archival, cron job, move old pcaps, LVM logical volume manager, extend retention period, secondary storage, slow storage disk, NFS mount, HDD array, archive directory, external storage, large disk storage, PCAP rotation diagnostic checklist, fast PCAP rotation, diagnose PCAP disappearing, check spooldir settings, per-hour usage analysis, per-day usage analysis, peak hours, growth rate calculation, database performance, MySQL tuning, innodb_flush_log_at_trx_commit, innodb_buffer_pool_size, spool directory filling exponentially, query file queue, disk swap problem, SQL queue growing, database bottleneck, MySQL slow writes, InnoDB settings, partition verification, show create table, select count partitions, database not cleaning troubleshooting, data volume misdiagnosis, cleaning working but data volume high, partition list check, record count in retained partitions, high volume diagnosis, s3fs, rclone, S3 cloud storage, manager interface unresponsive, GUI hangs with S3, S3 bucket mount, FUSE S3, tar_move S3, rclone mount S3, s3fs GUI timeout, sniffer thread blocking, network filesystem latency, pcap_open_offline unknown file format
'''Summary:''' VoIPmonitor has two independent data retention systems: (1) Filesystem cleaning for PCAP files using <code>maxpoolsize</code>/<code>maxpooldays</code> parameters, running every 5 minutes; (2) Database cleaning using <code>cleandatabase</code> with daily partitioning for instant partition drops.
 
'''Cleanspool modes:''' By default, VoIPmonitor uses '''legacy mode''' (<code>cleanspool_use_files=no</code>) with <code>.cleanspool_cache</code> filesystem index. This is because <code>pcap_dump_tar=yes</code> (TAR archive mode) is the default, which automatically disables database-indexed cleaning. To enable '''modern mode''' (database <code>files</code> table), explicitly set <code>cleanspool_use_files=yes</code> in config.
 
The <code>maxpool_clean_obsolete</code> parameter (default: no) controls whether files not in the index are deleted - when disabled (default), files outside the index are protected. Emergency cleanup (<code>autocleanspoolminpercent=1</code>, <code>autocleanmingb=5</code>) can override retention settings when disk is nearly full. GUI settings via <code>mysqlloadconfig=yes</code> override <code>voipmonitor.conf</code>. For tiered storage, use <code>tar_move</code> to archive to secondary storage. Database partition limit is ~8000 per table (~22 years); CDR record count uses BIGINT (18 quintillion limit). Size-based database cleaning requires <code>cleandatabase_size</code> AND <code>cleandatabase_size_force=true</code>.
 
'''Keywords:''' data retention, cleaning, maxpoolsize, maxpooldays, cleandatabase, cleanspool_use_files, maxpool_clean_obsolete, files table, .cleanspool_cache, pcap_dump_tar, autocleanspoolminpercent, autocleanmingb, emergency cleanup, tar_move, tiered storage, partitioning, cleandatabase_size, innodb_file_per_table, savertp header, packetbuffer_sender, BIGINT cdr.ID


'''Key Questions:'''
'''Key Questions:'''
* Why are directories within maxpoolrtpdays window much larger than older directories
* What is the default cleanspool mode and why?
* What causes unexpected increase in disk space used by CDRs in /log partition
* How does legacy cleanspool (.cleanspool_cache) differ from modern mode (files table)?
* Why do some spool directories contain both SIP and RTP while others contain only SIP
* What is cleanspool_use_files and how do I enable database-indexed cleaning?
* How do I understand directory size differences between maxpoolsipdays and maxpoolrtpdays retention windows
* Will cleanspool delete files not in the index/database?
* Why is my /log partition showing unexpected disk space changes
* How do I check maxpoolsipdays and maxpoolrtpdays settings
* How do I compare size of directories within retention period vs outside retention period
* Why are directories within maxpoolrtpdays window significantly larger
* Is it expected for directories to have different sizes based on retention periods
* How do I verify directory size difference is not an error
* How do I diagnose unexpected spool directory disk space increase
* What does it mean when directories vary significantly in size
* Why do directories within maxpoolrtpdays window contain both SIP and RTP data
* Do older directories containing only SIP data mean something is wrong
* How do I verify spool directory size behavior is expected and not an error
* How do I check if maxpoolsipdays and maxpoolrtpdays are configured correctly
* What causes spool directories to vary in size across different time periods
* How do I verify directory size difference matches my retention configuration
* How do I use du command to compare directory sizes in spooldir
* What should I check when noticing unexpected increase in disk space for PCAP files
* Why are recent CDR directories much larger than older ones in /log partition
* How do I verify maxpoolrtpdays and maxpoolsipdays retention settings
* How do I compare size of directories within maxpoolrtpdays window vs outside window
* Is directory size variation between SIP and RTP retention windows expected
* What causes spool directory size differences in /log partition
* How do I confirm directory size difference is not a configuration error
* How do I check directory sizes in spooldir using du command
* Why do some date directories contain more data than others
* How do I verify maxpool SIP and RTP retention settings are correct
* What causes unexpected disk space increase in /log partition
* How do I diagnose directory size variations in spool
* How do I check maxpoolsipdays and maxpoolrtpdays configuration
* Why are directories within retention period larger than those outside
* Understanding SIP vs RTP retention window directory size differences
* How do I verify database tables are partitioned for cleaning
* What command shows if tables are correctly partitioned
* How do I check partition list and record counts in retained partitions
* Database fills disk space despite cleandatabase being configured - is cleaning broken
* How do I diagnose if cleaning is not working vs data volume too high
* How do I check if only expected partitions exist (cleaning working) or old partitions exist (cleaning broken)
* What does it mean if partition list shows only recent partitions but database still fills disk
* How do I count records in retained partitions to check data volume
* When is database cleaning actually broken vs just high data volume
* How do I use SHOW CREATE TABLE to verify partitioning
* What query shows partition list with row counts
* If cleandatabase is configured but database fills, what should I check first
* PCAP files rotating faster than expected, how to diagnose
* What are the 5 steps to diagnose fast PCAP rotation
* How do I check my spooldir, maxpool, and cleandatabase settings in one command
* How do I measure total spool directory usage
* How do I analyze per-hour and per-day PCAP usage patterns
* How do I check database size contribution to disk usage
* How do I check overall disk capacity for VoIPmonitor
* What command shows spooldir location and retention settings together
* What does the "low spool disk space" log message mean?
* What triggers the "low spool disk space" message?
* Why is my spool directory filling up exponentially after a disk swap?
* How do I diagnose database performance problems causing spool directory growth?
* What MySQL InnoDB settings should I tune to fix disk filling issues?
* How do I check SQL queue in VoIPmonitor logs?
* What is the recommended value for innodb_flush_log_at_trx_commit?
* How should I configure innodb_buffer_pool_size for VoIPmonitor?
* Why does disk swap cause spool directory to fill up with query files?
* What are the symptoms of database bottleneck in VoIPmonitor?
* How do I verify MySQL InnoDB settings are configured correctly?
* What is the difference between innodb_flush_log_at_trx_commit values?
* How do I calculate innodb_buffer_pool_size for my server?
* What is the maximum recommended database size and CDR count for VoIPmonitor?
* How do I automatically delete old PCAP files?
* What is the difference between maxpoolsize and maxpooldays?
* My spool directory is full but old files are not deleted - how to fix?
* How do I configure database retention with cleandatabase?
* Why is disk space not reclaimed after MySQL cleanup?
* What do SQLq and SQLf metrics mean?
* Why are recent directories much larger than old ones in my spool?
* Directory size difference with maxpoolsipdays and maxpoolrtpdays
* How do I move the spooldir to a larger partition?
* What is cleandatabase_size and cleandatabase_size_force?
* How do I fix MySQL Error 28 "No space left on device" when standard cleandatabase retention is set?
* Why does MySQL crash even though cleandatabase retention is configured?
* When should I use cleandatabase_size instead of cleandatabase retention?
* What is the difference between time-based and size-based database cleaning?
* How do I stop audio recording and save only RTP headers?
* Where should I apply savertp=header in client-server mode with packetbuffer_sender?
* How do I reduce storage by saving only RTP headers instead of full audio?
* How do I enable selective audio recording for specific calls using capture rules?
* How do I use capture rules to record full audio for select calls while keeping global savertp=header?
* Can I use recordRTP=ON in capture rules to override savertp setting?
* How do I set up legal holds or VIP customer recording while complying with data retention laws?
* Why are my PCAP files being deleted even though maxpooldays is set to 60?
* What are autocleanspoolminpercent and autocleanmingb?
* How do emergency cleanup triggers override retention settings?
* How do I fix premature file deletion in spool directory?
* Does cleanspool delete files not in the database?
* What is maxpool_clean_obsolete?
* What is maxpool_clean_obsolete?
* How does cleanspool filesystem index differ from database records?
* Why are PCAP files being deleted faster than expected?
* Will cleanspool delete files that are not indexed in the database?
* How do I fix MySQL Error 28?
* How do I clean up old recordings for a specific IP address?
* How do I configure size-based database cleaning?
* What is custom autocleaning in VoIPmonitor GUI?
* How do I extend retention with tiered storage?
* How do I delete old recordings after configuring capture rules to stop recording?
* Why is disk space not reclaimed after cleanup?
* Can I clean up recordings based on specific filters like IP or phone number?
* How do I check if database partitioning is working?
* What is the difference between custom autocleaning and global retention settings?
* What is the maximum database size and CDR count limit?
* How to fix: cleanspool = yes in config file but cleaning not working?
* What is mysqlloadconfig and how does it affect cleanspool settings?
* How do GUI sensor settings override voipmonitor.conf configuration?
* What is the sensor_config database table?
* How do I resolve configuration conflicts between GUI and voipmonitor.conf?
* How do I find cleanspool setting in GUI Settings > Sensors?
* How do I prevent metadata table locks during partition operations?
* What is disable_partition_operations and when should I use it?
* How do I configure partition management for multiple sensors?
* Why does my database become unresponsive during partition operations?
* When should I use partition_operations_enable_fromto?
* My spool directory is filling up to capacity despite autoclean being enabled - what should I check?
* How do I configure maxpoolsize with a disk capacity buffer?
* What is cleanspool_enable_fromto and how does it work?
* Should I set maxpoolsize to less than total disk capacity?
* How do I create a buffer for times when cleaning is not active?
* Can multiple sensors manage the same database partitions?
* How do I extend PCAP retention with tiered storage using a second slower disk?
* What is the tar_move feature for automatic archival?
* How do I use tar_move to move files to secondary storage?
* How do I archive old PCAP files to a larger slower disk while keeping recent data on fast storage?
* How do I configure additional spool directories in the GUI Settings > System configuration > Basic?
* Can VoIPmonitor access PCAP files stored in multiple directories outside the main spooldir?
* How do I use LVM to combine fast and slow disks into a single spooldir?
* What is the best method for implementing hot/cold storage tiers with VoIPmonitor?
* How do I set up a cron job to manually archive old PCAP files?
* How do I configure tar_move_destination_path for NFS or external storage?
* What are the differences between tar_move, manual archival, and LVM for tiered storage?
* Why is the manager interface unresponsive when using tar_move with s3fs?
* How do I fix GUI unresponsiveness with S3 storage?
* What is the recommended alternative to s3fs for mounting S3 buckets?
* How do I use rclone to mount S3 buckets for tar_move?
* What rclone mount options are recommended for VoIPmonitor?
* Why does s3fs cause the GUI to hang when accessing archived files?
* How do I prevent the sniffer thread from blocking on file access operations?
* What is the working rclone mount command for tar_move with S3?
* How do I fix pcap_open_offline unknown file format errors with S3FS spooldir?
* Why does VoIPmonitor fail to start with spooldir on S3FS?

Revision as of 14:11, 8 January 2026

This guide explains how VoIPmonitor manages data retention for both captured packets (PCAP files) and Call Detail Records (CDRs) in the database. Proper configuration is essential for managing disk space and maintaining long-term database performance.

Overview

VoIPmonitor generates two primary types of data that require periodic cleaning:

  • PCAP Files: Raw packet captures of SIP/RTP/GRAPH data stored on the filesystem in the spool directory. These can consume significant disk space.
  • CDR Data: Call metadata stored in the MySQL database. Large tables can slow down GUI performance if not managed properly.

The system uses two separate, independent mechanisms to manage the retention of this data:

Filesystem Cleaning (PCAP Spool Directory)

The sensor stores captured call data in a structured directory tree on the local filesystem.

How Cleanspool Works: Modern vs Legacy Mode

VoIPmonitor has two distinct modes for tracking and cleaning PCAP files. Understanding which mode your installation uses is critical for troubleshooting cleaning issues.

Important: Two Cleanspool Modes
Legacy Mode (Default):

cleanspool_use_files = no

Uses a filesystem-based index stored in .cleanspool_cache file in the spool directory. The cleanspool process scans the directory structure and maintains this cache file locally.

Why this is the default: The pcap_dump_tar = yes setting (TAR archive mode) is enabled by default. When TAR mode is active, VoIPmonitor automatically switches to legacy cleanspool mode unless you explicitly set cleanspool_use_files = yes.

Modern Mode (Database-Indexed):

cleanspool_use_files = yes

Uses the MySQL files table to track all created PCAP files. The sniffer records every file it creates in this database table, and cleanspool deletes only files that are indexed in the files table.

Key behavior: Files that exist on the filesystem but are NOT in the files table will NOT be automatically deleted (unless maxpool_clean_obsolete is enabled).

To enable modern mode: 

cleanspool_use_files = yes

Legacy Mode: Filesystem Cache (Default)

In legacy mode (cleanspool_use_files = no), which is active by default when TAR mode is enabled: 

HTTP-Response: 
Error 400: Syntax Error? (Assumed diagram type: sequence) (line: 5)

Diagram-Code:
@startuml
skinparam shadowing false
participant "Sniffer" as S
participant "Cleanspool\nThread" as C
collections "Filesystem" as FS
file ".cleanspool_cache" as Cache

S -> FS: Write PCAP file\n(into TAR archive)

... Every 5 minutes ...

C -> FS: Scan directory structure
C -> Cache: Update file index
C -> Cache: Read oldest files\nwhere age > maxpooldays\nOR total size > maxpoolsize
loop For each file
    C -> FS: DELETE file
    C -> Cache: Update index
end
@enduml

Characteristics of Legacy Mode:

  • Uses .cleanspool_cache file in spool directory for tracking
  • Scans filesystem to build and update the index
  • Default behavior when pcap_dump_tar = yes (TAR archive mode is default)
  • Files not in cache are ignored (unless maxpool_clean_obsolete = yes)

Checking which mode is active: 

# Check TAR mode (if yes, likely legacy cleanspool)
grep pcap_dump_tar /etc/voipmonitor.conf
grep tar /etc/voipmonitor.conf | head -5

# Check explicit cleanspool_use_files setting
grep cleanspool_use_files /etc/voipmonitor.conf

# Check if .cleanspool_cache exists (indicates legacy mode is/was used)
ls -la /var/spool/voipmonitor/.cleanspool_cache

Modern Mode: Database-Indexed Cleaning

In modern mode (cleanspool_use_files = yes), which must be explicitly enabled:

Advantages of Modern Mode:

  • Faster cleanup (no filesystem scanning)
  • Accurate size tracking via database
  • Supports distributed deployments with shared storage
  • Files not in database are protected from accidental deletion

To enable Modern Mode: 

# /etc/voipmonitor.conf
cleanspool_use_files = yes

The maxpool_clean_obsolete Parameter

This parameter controls how cleanspool handles files that exist on the filesystem but are NOT in its index (either database files table or .cleanspool_cache).

Setting Behavior Use Case
maxpool_clean_obsolete = no (default) Only delete files that are indexed. Unknown files are ignored. Safe default - protects manually added files
maxpool_clean_obsolete = yes Delete ALL files in spool directory, including those not in index Clean up orphaned files, recover from index corruption
Warning: maxpool_clean_obsolete
Caution:

When maxpool_clean_obsolete = yes is enabled, cleanspool will scan the entire filesystem and delete any files not found in its index. This includes:

  • Files manually copied into the spool directory
  • Files from backup restores
  • Files from other sensors (in shared storage scenarios)

Only enable this if you understand the implications and want aggressive cleanup of unindexed files.

Reducing Data Collection at Source

Before configuring cleanup policies, consider reducing the amount of data captured. This is often the most effective long-term solution for storage management.

Save Only RTP Headers (Major Space Saver)

RTP packets typically contain the full audio payload, which consumes the majority of disk space. If you only need call quality statistics (MOS, jitter, packet loss) and not actual audio playback, switch to saving RTP headers only.

Edit /etc/voipmonitor.conf: 

# Change from full RTP to headers only
savertp = header
Setting Storage Impact Use Case
savertp = yes High (~10x more) Requires ability to play back audio from PCAPs
savertp = header Low Only CDR statistics needed, no audio playback required

With savertp = header, VoIPmonitor still captures all necessary metadata for MOS scoring, jitter analysis, packet loss statistics, and quality graphs, but does not store the actual audio payload. This can reduce storage consumption by up to 90%.

Important: After changing from savertp = yes to savertp = header, existing PCAP files will remain playable. New calls will only contain RTP headers.

Critical: Distributed Architecture Consideration
packetbuffer_sender Architecture:

If you are using the Packet Mirroring/Client-Server mode (packetbuffer_sender=yes on remote sensors), the savertp setting must be applied on the central server where packet analysis is performed, not on the individual sensors.

Architecture Where to Apply savertp = header
packetbuffer_sender = no (Local Processing) On each sensor/probe's configuration file
packetbuffer_sender = yes (Packet Mirroring) On the central server's configuration file (not on sensors)

For distributed deployment details, see Client-Server Architecture.

Enabling Selective Audio Recording with Capture Rules

For compliance scenarios where you need to record full audio for specific calls while keeping most calls with headers only (or disabled), use GUI capture rules to create exceptions:

Workflow:

1. **Set global default to headers only:** 

savertp = header

2. **Create capture rules in the GUI for exceptions:** 

  * Navigate to Control Panel > Capture Rules
  * For selective audio recording, set the recordRTP option to ON
  * Rules with recordRTP=ON will capture full RTP audio, overriding the global savertp=header setting for matched calls

Use Cases:

Scenario Configuration File Storage
Legal Holds Create IP rule for subject party with recordRTP=ON Full audio recorded
VIP Customers Create phone number prefix rules for specific ranges with recordRTP=ON Full audio recorded
All Other Calls Global savertp = header Headers only (no audio)

This approach allows you to comply with data retention laws (e.g., GDPR) by minimizing audio recording while still meeting specific legal or business requirements for certain call categories.

For detailed capture rule configuration options, see Capture_rules. For more configuration options, see Sniffer Configuration - Saving Options.

Spool Directory Structure

The spool directory uses a hierarchical structure organized by time and data type.

Structure Description
YY-mm-dd/HH/MM/{TYPE}/files... Date → Hour → Minute → Data Type

The {TYPE} subdirectory can be:

  • SIP - SIP signaling PCAP files
  • RTP - RTP audio PCAP files
  • AUDIO - Converted audio files (WAV/OGG)
  • GRAPH - Quality graph image files

Example path: 

/var/spool/voipmonitor/25-01-06/10/45/RTP/rtp_2025-01-06-10-45.tar

The timestamps are based on UTC or the timezone configured in voipmonitor.conf.

Important: The cleanup process operates at the minute level, not hour level. When cleaning is triggered, it removes the oldest minute's worth of data (e.g., 25-01-06/08/57/) rather than deleting an entire hour's worth of data at once.

Spool Directory Location

By default, all data is stored in /var/spool/voipmonitor. This location can be changed by setting the spooldir option in voipmonitor.conf.

Relocating the Spool Directory to a Different Partition

If your default partition is running out of space, you can move the spool directory to a larger partition or dedicated disk. This is particularly useful when MySQL Error 28 ("No space left on device") occurs despite retention settings being in place.

Situation Recommended Action
/var/lib/mysql full but other partitions available Move spool to larger partition and update spooldir in voipmonitor.conf
Multiple disks available Use dedicated partition for PCAP storage
GUI on same server Sync GUI SNIFFER_DATA_PATH to match new spooldir
Procedure to Relocate Spooldir
Step 1
Identify Available Space

Check all partitions to find a suitable location: 

df -h
Step 2
Create New Spool Directory

Create the directory on the destination partition: 

# Example: Use /mnt/pcaps partition
mkdir -p /mnt/pcaps/voipmonitor

# Set correct ownership for both sniffer and GUI access
chown voipmonitor:voipmonitor /mnt/pcaps/voipmonitor
chmod 755 /mnt/pcaps/voipmonitor
Step 3
Update VoIPmonitor Configuration

Edit /etc/voipmonitor.conf to point to the new location: 

spooldir = /mnt/pcaps/voipmonitor
Step 4
Update GUI Configuration (Critical)

If the GUI runs on the same server, you MUST update the path the GUI uses to access the spool. This is defined in the GUI configuration file:

  • Debian/Ubuntu: /var/www/html/voipmonitor/config/configuration.php
  • RHEL/CentOS: /var/www/voipmonitor/config/configuration.php

Edit the GUI configuration file: 

// Find and update this line to match your new spooldir
define('SNIFFER_DATA_PATH', '/mnt/pcaps/voipmonitor');

Important: The SNIFFER_DATA_PATH in the GUI config MUST match the spooldir setting in voipmonitor.conf. If these are mismatched, the GUI will be unable to locate and display call recordings.

Step 5
Restart Services

Apply all changes by restarting the services: 

# Restart the VoIPmonitor sensor
systemctl restart voipmonitor

# If using Apache/Nginx, reload web server
systemctl reload apache2    # Debian/Ubuntu
# or
systemctl reload nginx       # RHEL/CentOS
Step 6
Verify Configuration
# Check if new spool directory is being used
ls -ld /mnt/pcaps/voipmonitor

# Verify GUI can access the path
grep SNIFFER_DATA_PATH /path/to/gui/config/configuration.php

# Check sensor logs for any spool-related errors
tail -f /var/log/voipmonitor.log
Optional: Move Existing Data

If you want to migrate existing PCAP files to the new location: 

# Stop the sensor to prevent writes during migration
systemctl stop voipmonitor

# Move existing data (this may take a long time)
mv /var/spool/voipmonitor/* /mnt/pcaps/voipmonitor/

# Restart the sensor
systemctl start voipmonitor

Retention Mechanism

The cleaning process runs automatically every 5 minutes and removes data based on the configuration in voipmonitor.conf.

Mechanism Check Interval Purpose
Retention Policy (maxpool*) Every 5 minutes Capacity control - maintains storage below configured limits
Emergency Cleanup (autoclean*) Every 5 minutes Disk space "fuse" - prevents disk exhaustion when free space falls below threshold

Retention Policy: maxpool* Parameters

When triggered by a maxpool* setting (e.g., maxpoolsize, maxpoolrtpsize):

1. The cleanup identifies the oldest minute of data 2. It removes the entire YY-mm-dd/HH/MM/ directory tree (e.g., 25-01-06/08/57/) 3. If maxpoolsize is used (overall limit), it removes all data types from that minute 4. If type-specific limits are used (e.g., maxpoolaudiosize), it removes only that data type

Important: The cleanup removes ONE minute of data at a time, not an entire hour's worth. This ensures gradual, controlled cleanup instead of sudden large deletions.

Emergency Cleanup: autoclean* Parameters

The autoclean* parameters act as a safety "fuse" to prevent the disk from filling up:

Parameter Default Behavior
autocleanspoolminpercent 1 When disk usage reaches 99%, trigger emergency cleanup regardless of maxpool* settings
autocleanmingb 5 When free space falls below 5 GB, trigger emergency cleanup

When either threshold is breached:

  • Oldest data is deleted AGGRESSIVELY until free space is restored
  • This happens regardless of your maxpool* or maxpooldays settings
  • Normal retention behavior resumes once thresholds are cleared

Performance Warning: maxpoolaudiosize

The maxpoolaudiosize parameter controls the size of converted audio files (WAV/OGG) when saveaudio is enabled.

Performance Consideration
Warning Audio conversion (PCAP → WAV/OGG) is CPU-intensive. For deployments with high call volumes (>200 concurrent calls or high CPS), using saveaudio can overload the system and cause packet loss or performance degradation.
  • Recommended: For high-volume deployments, use savertp = yes and convert on-demand via the GUI instead
  • Alternative: Consider savertp = header if you only need statistics, not audio playback

Retention Configuration

The cleaning configuration allows you to set limits based on total size (in Megabytes) or age (in days).

Important: Default Retention Behavior
Default Behavior:

By default, PCAP files are deleted based on size only (using maxpoolsize), not based on time. The default maxpoolsize is 100 GB (102400 MB). When the spool directory reaches 100 GB, the oldest PCAP files are automatically deleted to free up space.

Time-based retention (maxpooldays) is disabled by default and must be explicitly configured if you want to limit retention by days instead of size.

You can set limits based on total size (in Megabytes) or age (in days). If both a size and day limit are set for the same data type, the first limit that is reached will trigger the cleaning.

Parameter Default Value Description
maxpoolsize 102400 (100 GB) The total maximum disk space for all captured data (SIP, RTP, GRAPH, AUDIO).
maxpooldays (unset) The maximum number of days to keep all captured data.
maxpoolsipsize (unset) A specific size limit for SIP PCAP files only.
maxpoolsipdays (unset) A specific age limit for SIP PCAP files only.
maxpoolrtpsize (unset) A specific size limit for RTP PCAP files only.
maxpoolrtpdays (unset) A specific age limit for RTP PCAP files only.
maxpoolgraphsize (unset) A specific size limit for GRAPH files only.
maxpoolgraphdays (unset) A specific age limit for GRAPH files only.
maxpoolaudiosize (unset) A specific size limit for converted audio files (WAV/OGG) only.
maxpoolaudiodays (unset) An age limit for converted audio files (WAV/OGG) only.

Recommended Configuration Strategy: Mixed Size-Based Retention

For optimal disk space management while preserving SIP signaling for long-term analysis, use a **mixed retention strategy**:

Data Type Recommended Setting Rationale
RTP (audio payload) Size-based (maxpoolrtpsize) RTP consumes most disk space (5-10x more than SIP)
SIP (signaling) Overall only (maxpoolsize, no maxpoolsipdays) SIP files are small; keep as long as total space allows
Charts/Audio Overall only (maxpoolsize) Supplementary data, keep as space permits
Configuration Example: Long-Term SIP with Limited RTP
Scenario: You want to keep SIP signaling and charts for as long as possible, but limit RTP (audio) to a reasonable size to prevent disk exhaustion.
Configuration: Edit /etc/voipmonitor.conf: 
# Set size limit for RTP only (keeps audio limited)
maxpoolrtpsize = 102400    # 100 GB for RTP

# DO NOT set maxpoolsipdays - let SIP be controlled by maxpoolsize only

# Set overall limit for all data together
maxpoolsize = 512000       # 500 GB total limit (includes SIP + RTP + GRAPH + AUDIO)
Behavior: RTP is deleted when it exceeds maxpoolrtpsize (100 GB). SIP, chart, and audio files are kept as long as the total maxpoolsize (500 GB) allows. If the disk fills up (total exceeds 500 GB), oldest data is deleted regardless of type.
Why this works: RTP files are huge (GBs per day) but rarely needed for troubleshooting. SIP files are small (MBs per day) and essential for analyzing call setup/teardown, call flows, and configuration issues. By limiting RTP size but not SIP size, you maximize useful long-term data retention.

Alternative Configuration: Time-Based Retention

If you need to keep data for a specific number of days (such as compliance requirements), you can use time-based retention parameters.

Important: Time-Based Retention Basics
Understanding Maximum vs Minimum: maxpoolsipdays sets a maximum retention period for SIP files, not a guaranteed minimum. To ensure files are kept for at least the specified period, you must allocate sufficient disk space using maxpoolsize to accommodate the expected volume of data for that period.

Configuration Example: 14-Day SIP Retention

Scenario Configuration Behavior
Keep SIP for 14 days, limit RTP Edit /etc/voipmonitor.conf: 
# Set maximum age for SIP files
maxpoolsipdays = 14
maxpoolgraphdays = 14

# Set maximum age for RTP files (can be different)
maxpoolrtpdays = 7

# OR set maximum size for RTP files
maxpoolrtpsize = 102400

# Allocate sufficient disk space to accommodate 14 days of data
maxpoolsize = 300000     # Ensure this is large enough for 14 days of SIP + graph data
SIP files are deleted after 14 days (maximum). RTP files are deleted after 7 days (if using maxpoolrtpdays) or when maxpoolrtpsize limit is reached. If maxpoolsize is reached before the 14-day period, data may be deleted regardless of age. Monitor disk usage to ensure maxpoolsize is sufficient.

Verifying Which Cleaning Rule is Active

To debug which cleaning rule is currently applied and confirm your configuration is working: 

# Check systemd journal for clean triggers
journalctl -u voipmonitor --since='YYYY-MM-DD' | grep clean -i

The log messages indicate which rule triggered cleanup:

  • clean_maxpoolsize - Cleaning triggered by total size limit
  • clean_maxpooldays - Cleaning triggered by overall age limit
  • clean_maxpoolsipdays - Cleaning triggered by SIP age limit
  • clean_maxpoolrtpdays - Cleaning triggered by RTP age limit

If you see multiple triggers, VoIPmonitor applies the first limit that is reached, deleting data until all limits are satisfied.

Best Practice: Configure maxpoolsize with Disk Capacity Buffer

When setting maxpoolsize, configure a buffer below the total disk capacity. This ensures there is sufficient space during periods when the cleaning process is not active (such as when cleanspool_enable_fromto restricts cleaning to specific hours).

Important: Disk Capacity Buffer
Why a Buffer is Needed:

The cleaning process runs every 5 minutes, but cleanspool_enable_fromto may restrict cleaning to specific hours (e.g., 1-5 AM). During non-cleaning hours, data will continue to accumulate. Setting maxpoolsize close to total disk capacity risks filling the disk outside the active cleaning window.

Recommended Configuration:

Set maxpoolsize to approximately 90-95% of total disk capacity to create a buffer.

Example:

For a 7 TB disk: 

# 6.5 TB = 6656000 MB (approximately 93% of 7 TB)
maxpoolsize = 6656000

# Restrict cleaning to 1-5 AM to avoid I/O impact during peak traffic
cleanspool_enable_fromto = 1-5

This configuration provides a 0.5 TB buffer for data accumulation during the 19-hour cleaning-off window.

Manual File Deletion

Manual deletion of files from the spool directory is generally safe, but there are important considerations.

Is It Safe?

Scenario Safe? Notes
Deleting old directories (prior days/hours) Yes The index is recomputed automatically
Deleting current hour while service running No risk to service, but may cause errors for active captures if file handles are still open
Deleting current MINUTE while recording in progress Not recommended May cause file descriptor errors temporarily

When you manually delete files:

  • In modern mode (cleanspool_use_files = yes): The files database table still contains records for deleted files. These orphaned records are cleaned up during the next cleanup cycle.
  • In legacy mode: The .cleanspool_cache is recomputed automatically during the next cleanup cycle.

Orphaned Database Records

When you manually delete PCAP files, the corresponding records in the MySQL database (CDR tables) remain. This means:

  • You will see calls in the GUI
  • Clicking "Play" or "Download" will show "File not found" errors
Manual Deletion: Database Records Remain
After deleting PCAP files Database CDR records are NOT automatically deleted. Calls will appear in the GUI but show "File not found" for missing files.
Solution To clean orphaned records, run the database cleaner: voipmonitor --clean-db or use the GUI "Maintenance" tab.

When Is Manual Deletion Recommended?

  • **Emergency Space Recovery:** If disk is 99% full and autoclean is not processing fast enough, manually deleting old directories (e.g., yesterday's folders) provides immediate space relief
  • **Compliance Requests:** Legal requests requiring immediate data removal without waiting for automated cleanup
  • **Targeted Cleanup:** Removing data for specific dates/time periods

Recovering from Index Issues

If the cleanspool index becomes corrupted or out of sync:

For Modern Mode (cleanspool_use_files = yes):

The files table can be rebuilt by rescanning the spool directory: 

# Trigger a reindex via manager API
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
rm /tmp/vmsck

For Legacy Mode (.cleanspool_cache):

If the .cleanspool_cache file is deleted or corrupted: 

# Same reindex command works for legacy mode
echo 'manager_file start /tmp/vmsck' | nc 127.0.0.1 5029
echo reindexfiles | nc -U /tmp/vmsck
rm /tmp/vmsck

The reindexfiles command rescans the spool directory structure and rebuilds the index.

If your deployment uses the secured manager API with SSL/TLS, use the secured endpoint instead of the Unix socket method. See Manager API documentation for the secured API usage.
Alternative: Secured Manager API

Custom Autocleaning: One-Time Cleanup with Filters

The GUI provides a custom autocleaning feature that allows you to perform one-time cleanup of existing recordings based on specific criteria, such as IP address or telephone number. This is useful when you need to clean up data for a specific subset of calls without affecting global retention settings.

Use Case: Cleaning Old Recordings for a Specific IP

After configuring capture rules to stop recording future calls from a specific IP address, you may still have existing RTP recordings from that IP. Custom autocleaning allows you to remove these old recordings.

Example scenario:

  • You configured a capture rule to discard RTP for IP 192.168.1.50
  • Only new calls will be affected by this rule
  • Existing recordings for this IP must be cleaned up manually

GUI cleanup steps:

  1. Navigate to Settings > Custom Autocleaning
  2. Create a new autocleaning rule
  3. Set the criteria (e.g., "Delete RTP older than 1 day")
  4. In the Common Filter section, specify the target IP address (192.168.1.50)
  5. Save and apply the rule
  6. Once the cleanup is complete, remove the autocleaning rule

This rule will run once and clean up all matching old recordings, then the capture rule will prevent future recordings.

Comparison with Global Retention

Feature Custom Autocleaning Global Retention
Scope Targeted (specific IP, number, filter) All calls
Purpose One-time cleanup of existing data Ongoing automated cleanup
Configuration GUI with CDR filters maxpoolsize, maxpooldays
Flexibility High - can use any CDR filter criteria Low - time/size only

Diagnosing Disk Space Usage

To properly configure retention limits, first analyze your actual disk usage:

Check total disk space
df -h
Check total spool directory size
du -hs /var/spool/voipmonitor
Analyze per-day utilization (helps identify growth patterns)
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2

Example output interpretation: 

# 80G    ./2024-12
# 120G   ./2024-11
# 150G   ./2024-10

This shows recent months consume 80-150 GB of data. Use this data to estimate appropriate size limits.

Understanding Directory Size Differences (SIP vs RTP Retention)

If you notice significant size differences between directories in your spool, this is usually expected behavior when you have different retention periods for SIP and RTP data.

Age of Directory Contents Size
0-30 days Both SIP and RTP PCAP files Large
30-90 days SIP PCAP files only (RTP deleted) Smaller
90+ days None (both SIP and RTP deleted) Empty or absent

This behavior is expected and by design when using different retention periods (e.g., maxpoolsipdays = 90 and maxpoolrtpdays = 30).

Why RTP Directories Exist After Disabling RTP Saving

If you have configured savertp = no or savertp = header but still see RTP subdirectories, this occurs because:

  • When savertcp = yes, RTCP (RTP Control Protocol) packets are saved in the rtp_YYYY-MM-DD-HH-MM.tar files
  • The RTP directory name refers to the broad UDP port range used for RTP/RTCP traffic
  • These files contain RTCP control traffic (receiver reports, sender reports) but not the full RTP audio payload
Configuration Result Expected Directory Size
savertp = yes + savertcp = yes Full RTP audio + RTCP controls saved Very large (GBs)
savertp = no + savertcp = yes Only RTCP controls saved Small (MBs)
savertp = header + savertcp = yes RTP headers + RTCP controls saved Small (MBs)

Filesystem Troubleshooting

Files Disappearing Faster Than Expected

If PCAP files are being deleted sooner than your maxpooldays setting suggests, check for these common causes:

Emergency Cleanup Triggers

Emergency cleanup can override your retention settings:

Parameter Default Triggers When
autocleanspoolminpercent 1 Disk usage reaches 99%
autocleanmingb 5 Free space falls below 5 GB

Diagnosis: 

# Check disk usage
df -h /var/spool/voipmonitor

# Check emergency trigger settings
grep -E "autocleanspoolminpercent|autocleanmingb" /etc/voipmonitor.conf

Resolution: 

# Increase emergency thresholds (allow more data before emergency cleanup)
autocleanspoolminpercent = 5   # Allow 95% usage before emergency
autocleanmingb = 20            # Trigger at 20 GB free instead of 5 GB

GUI Configuration Override

If mysqlloadconfig = yes (default), GUI sensor settings override voipmonitor.conf:

Diagnosis: 

# Check if database config is enabled
grep mysqlloadconfig /etc/voipmonitor.conf

# Check GUI: Settings > Sensors > wrench icon > search "maxpool"

Resolution: Update settings via GUI, or set mysqlloadconfig = no to use file-based config only.

Diagnostic Checklist

Before applying fixes, run through this diagnostic checklist: 

# 1. Check all retention-related settings
cat /etc/voipmonitor.conf | grep -E '^spooldir|^maxpool|^cleandatabase|^autoclean'

# 2. Measure total spool usage
du -hs /var/spool/voipmonitor

# 3. Analyze per-day usage patterns
du -h --max-depth=1 /var/spool/voipmonitor | sort -k2,2

# 4. Check disk capacity
df -h

# 5. Check for emergency cleanup in logs
journalctl -u voipmonitor | grep -i "clean\|autoclean"

Spool Directory Filling Due to Database Performance

If /var/spool/voipmonitor/ is filling up rapidly after a disk swap, MySQL upgrade, or configuration change, the issue may be caused by database performance problems.

Root Cause: Slow Database Causing Query File Queuing
Symptoms Spool directory fills exponentially, sometimes within hours, even with correct maxpoolsize settings.
Root Cause MySQL cannot process database writes quickly enough, causing query files to queue.

Diagnosis: 

# Watch for SQLq (SQL queue) metrics in real-time
tail -f /var/log/syslog | grep voipmonitor | grep SQLq

# Check MySQL InnoDB settings
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';"
mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';"

Solution - Tune MySQL: 

[mysqld]
# 50-70% of server RAM
innodb_buffer_pool_size = 8G

# Fast writes (acceptable data loss risk on crash)
innodb_flush_log_at_trx_commit = 2

"low spool disk space" Message

This message indicates that your configured maxpoolsize exceeds the physical disk capacity. VoIPmonitor may automatically adjust maxpoolsize to a lower value.

Resolution: 

# Check actual disk capacity
df -h /var/spool/voipmonitor

# Set realistic maxpoolsize (leave 5-10% buffer)
# For a 500 GB partition, use ~450 GB:
maxpoolsize = 460800

Missing Data Due to NFS/Storage Server Issues

If using remote storage (NFS, SSHFS), missing data may be caused by network connectivity issues rather than retention policies.

Diagnosis: 

# Check system logs for NFS errors
grep -i "nfs" /var/log/syslog | grep "not responding\|timed out"

# Test connectivity
ping nfs-server.example.com
nc -zv nfs-server.example.com 2049

Tiered Storage and Archival Options

If you need to extend PCAP retention beyond your fast local disk capacity, there are three recommended approaches.

Option 1: Use tar_move Feature (Recommended)

The tar_move feature automatically archives PCAP files to secondary storage after they are closed. 

# Use local fast storage for live capture
spooldir = /var/spool/voipmonitor

# Enable automatic archival
tar_move = yes
tar_move_destination_path = /mnt/archive/voipmonitor
tar_move_max_threads = 2
Parameter Description
tar_move = yes Move files to archive after capture completes
tar_move = copy Copy files (keep original)
tar_move_destination_path Target directory for archived files

Important: Files in tar_move_destination_path remain accessible via GUI. VoIPmonitor searches both local spooldir and tar_move_destination_path.

S3 Cloud Storage Considerations

When using S3 storage, use rclone instead of s3fs to avoid GUI unresponsiveness: 

# Recommended: rclone mount
/usr/bin/rclone mount bucket-name /mnt/s3-archive \
  --allow-other --dir-cache-time 30s --poll-interval 0 \
  --vfs-cache-mode off --buffer-size 0 --use-server-modtime \
  --no-modtime --s3-no-head --log-level INFO

Option 2: Manual Archival with Additional Spool Directories

Move old files manually and configure GUI to access multiple directories:

  1. Create archive script to move files older than X days
  2. Add archive path to GUI: Settings > System configuration > Basic > Additional spool directories

Option 3: LVM Single Logical Volume

Combine fast SSD + large HDD into a single logical volume using LVM. VoIPmonitor sees only one spool directory.

Database Cleaning (CDR Retention)

Managing the size of the cdr table and other large tables is critical for GUI performance.

Partitioning Method (Recommended)

Since version 7, VoIPmonitor uses database partitioning, which splits large tables into smaller, daily segments.

Aspect Description
How it works Set cleandatabase = 30 to keep the last 30 days of data
Why it's better Dropping old partitions is instantaneous (milliseconds), regardless of row count
Partition limit ~8000 partitions per table (~22 years with daily partitioning)
CDR record limit No practical limit - modern installations use BIGINT for cdr.ID (up to 18 quintillion records). See Migrating to BIGINT.

Quick Start: Global Retention

# Keep all records for 30 days
cleandatabase = 30

Retention Parameters

Parameter Default Description
cleandatabase 0 (disabled) Master retention setting in days
cleandatabase_cdr 0 Specific retention for CDR tables
cleandatabase_rtp_stat 2 Retention for RTP statistics
cleandatabase_sip_msg 0 Retention for SIP messages
cleandatabase_register_state 0 Retention for registration states
cleandatabase_register_time_info 0 Must be set explicitly (not covered by global setting)
cleandatabase_size (unset) Size-based limit in MB
cleandatabase_size_force false Required for size-based cleanup
partition_operations_enable_fromto 1-5 Time window for partition operations
Important: register_time_info Table
Critical: The register_time_info table is NOT covered by the global cleandatabase setting. You MUST configure cleandatabase_register_time_info explicitly.

Size-Based Database Cleaning

Use cleandatabase_size to limit database by size rather than time: 

# Limit database to 50 GB
cleandatabase_size = 51200
cleandatabase_size_force = true

# Optional: Start cleaning when disk free space drops below threshold
cleandatabase_min_free_size = 30720

Multi-Sniffer Environments

When multiple sensors share the same database, partition operations should be managed by only ONE sensor: 

# On all sniffers EXCEPT one:
disable_partition_operations = yes

# On the ONE designated sniffer:
partition_operations_enable_fromto = 4-6

Database Troubleshooting

MySQL Error 28: No Space Left on Device

If MySQL crashes with Error 28 even when cleandatabase is configured:

Primary Solution - Enable Size-Based Cleaning: 

cleandatabase_size = 512000        # 500 GB limit
cleandatabase_size_force = true    # Required

Alternative Causes:

  • Inode exhaustion: Check with df -i
  • MySQL tmpdir full: Check with mysql -e "SHOW VARIABLES LIKE 'tmpdir';"

Disk Space Not Reclaimed After Cleanup

Check innodb_file_per_table setting: 

SHOW GLOBAL VARIABLES LIKE 'innodb_file_per_table';

If OFF, space is not reclaimed when partitions are dropped. Enable for future tables: 

[mysqld]
innodb_file_per_table = 1

Database Not Cleaning (Verify Partitioning)

Before assuming cleaning is broken, verify tables are partitioned: 

-- Check if CDR table is partitioned
SHOW CREATE TABLE cdr\G

-- Check partition list and row counts
SELECT PARTITION_NAME, TABLE_ROWS
FROM information_schema.PARTITIONS
WHERE TABLE_NAME = 'cdr'
ORDER BY PARTITION_ORDINAL_POSITION DESC
LIMIT 10;

If only expected partitions exist (matching your cleandatabase setting), cleaning IS working - you may simply have high data volume.

MySQL Performance Settings

For high-performance operation: 

[mysqld]
# Use 50-70% of available RAM for caching
innodb_buffer_pool_size = 4G

# Flush logs to OS every second (faster, safe for VoIPmonitor)
innodb_flush_log_at_trx_commit = 2

# Enable per-table filespace for easy space reclamation
innodb_file_per_table = 1

See Also

AI Summary for RAG

Summary: VoIPmonitor has two independent data retention systems: (1) Filesystem cleaning for PCAP files using maxpoolsize/maxpooldays parameters, running every 5 minutes; (2) Database cleaning using cleandatabase with daily partitioning for instant partition drops.

Cleanspool modes: By default, VoIPmonitor uses legacy mode (cleanspool_use_files=no) with .cleanspool_cache filesystem index. This is because pcap_dump_tar=yes (TAR archive mode) is the default, which automatically disables database-indexed cleaning. To enable modern mode (database files table), explicitly set cleanspool_use_files=yes in config.

The maxpool_clean_obsolete parameter (default: no) controls whether files not in the index are deleted - when disabled (default), files outside the index are protected. Emergency cleanup (autocleanspoolminpercent=1, autocleanmingb=5) can override retention settings when disk is nearly full. GUI settings via mysqlloadconfig=yes override voipmonitor.conf. For tiered storage, use tar_move to archive to secondary storage. Database partition limit is ~8000 per table (~22 years); CDR record count uses BIGINT (18 quintillion limit). Size-based database cleaning requires cleandatabase_size AND cleandatabase_size_force=true.

Keywords: data retention, cleaning, maxpoolsize, maxpooldays, cleandatabase, cleanspool_use_files, maxpool_clean_obsolete, files table, .cleanspool_cache, pcap_dump_tar, autocleanspoolminpercent, autocleanmingb, emergency cleanup, tar_move, tiered storage, partitioning, cleandatabase_size, innodb_file_per_table, savertp header, packetbuffer_sender, BIGINT cdr.ID

Key Questions:

  • What is the default cleanspool mode and why?
  • How does legacy cleanspool (.cleanspool_cache) differ from modern mode (files table)?
  • What is cleanspool_use_files and how do I enable database-indexed cleaning?
  • Will cleanspool delete files not in the index/database?
  • What is maxpool_clean_obsolete?
  • Why are PCAP files being deleted faster than expected?
  • How do I fix MySQL Error 28?
  • How do I configure size-based database cleaning?
  • How do I extend retention with tiered storage?
  • Why is disk space not reclaimed after cleanup?
  • How do I check if database partitioning is working?
  • What is the maximum database size and CDR count limit?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Data_Cleaning&oldid=9988"