Reprocess Historical Calls

From VoIPmonitor.org
Revision as of 20:24, 7 January 2026 by Admin (talk | contribs) (Add documentation for reprocessing historical calls with process_spooldir.pl to enable features retroactively (e.g., save_sip_responses))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


This guide explains how to re-process captured packet archives (PCAP files) using a different configuration than was originally used at capture time. This is useful for enabling features retroactively, such as storing intermediate SIP responses, analyzing calls with new detection rules, or testing configuration changes without affecting live monitoring.

== Use Cases

Common scenarios where historical call reprocessing is needed:

  • Enable SIP response storage retroactively - Capture intermediate SIP responses (e.g., 491 Request Pending) from previously captured calls when save_sip_responses was not enabled
  • Test new detection algorithms - Apply updated fax/DTMF detection or jitter analysis to historical data
  • Replay with different capture rules - Test how a new capture_rules configuration would have behaved on past traffic
  • Upgrade schema compatibility - Re-process old PCAPs to take advantage of new database schema features
  • Debug analysis issues - Re-run a problematic CDR through the packet processor to debug parsing logic

== Safety Considerations

⚠️ Warning: Reprocessing historical data can be resource-intensive and should be performed on non-production equipment or during maintenance windows. The process can consume significant CPU, I/O, and database resources depending on the volume of data being reprocessed.

Important safety notes:

  • Always re-process into a separate database, not your primary production database
  • The original PCAP files remain unchanged; reprocessing only creates new CDR entries
  • Re-processing does not delete or modify data in the source PCAP archives
  • Disk space is required for both the original PCAPs and the new reprocessed CDR data
  • Network packet capture is not required during reprocessing; this is an offline batch process

== Step-by-Step Guide

=== Step 1: Create a New Empty Database

Create a separate MySQL/MariaDB database for the reprocessed data. Do not use your main production database for this process. 

-- Connect to MySQL as an administrative user
mysql -u root -p

-- Create a new empty database
CREATE DATABASE voipmonitor_reprocess CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- Create a dedicated user for this database (optional but recommended)
CREATE USER 'voipmonitor_reprocess'@'localhost' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON voipmonitor_reprocess.* TO 'voipmonitor_reprocess'@'localhost';
FLUSH PRIVILEGES;

The VoIPmonitor sensor will automatically create the required tables when it starts processing with this database.

=== Step 2: Create a Reprocess Configuration File

Create a new configuration file (e.g., /etc/voipmonitor-reprocess.conf) with your desired settings: 

# /etc/voipmonitor-reprocess.conf
[general]

# Point to the NEW reprocessing database (NOT your main production database)
mysqlhost = localhost
mysqlsocket = /var/run/mysqld/mysqld.sock
mysqlport = 3306
mysqlusername = voipmonitor_reprocess
mysqlpassword = your_secure_password
mysqldb = voipmonitor_reprocess

# Enable the features you want to apply to historical calls
# Example: Store all SIP responses including intermediate ones
save_sip_responses = yes

# Other configuration based on what you want to change
# Example: Re-run with new SIP history settings
save_sip_history = responses

# Keep other settings appropriate for your environment
spooldir = /var/spool/voipmonitor

# CRITICAL: No interface or live capture settings required
# This is an offline reprocessing job

💡 Tip: You can copy your existing /etc/voipmonitor.conf as a starting point and then modify it for reprocessing. Remove or comment out options related to live capture (like interface, sipport, etc.) since reprocessing reads from disk only.

=== Step 3: Run the Reprocessing Script

Use the process_spooldir.pl Perl script to read PCAP files from the spooldir and re-process them with your new configuration: 

# Basic syntax
perl /var/www/html/voipmonitor/scripts/process_spooldir.pl \
  --voipcfg=/etc/voipmonitor-reprocess.conf \
  --spooldir=/var/spool/voipmonitor \
  --fromdate='2024-12-01 00:00' \
  --todate='2024-12-31 23:59'

Parameters:

  • --voipcfg: Path to your reprocessing configuration file (REQUIRED)
  • --spooldir: Path to the spooldir directory containing the archived PCAP files
  • --fromdate: Start date for reprocessing (YYYY-MM-DD HH:MM format)
  • --todate: End date for reprocessing (YYYY-MM-DD HH:MM format)
  • --pcapfilter: Optional BPF filter to only reprocess matching packets (e.g., --pcapfilter='udp port 5060')
  • --verbose: Enable detailed output for troubleshooting
  • --threads: Number of parallel threads (default: 2)

ℹ️ Note: The date range selects files based on the date/time hierarchy in the spooldir structure (/var/spool/voipmonitor/YYYY-MM-DD/HH/), NOT based on call start times within PCAP files.

=== Step 4: Monitor the Reprocess Progress

Reprocessing can take time depending on your data volume and system performance. Monitor progress with: 

# In another terminal, monitor the script output
# (If you didn't run with --verbose, check logs below)

# Check database size growth
mysql -u voipmonitor_reprocess -p -e "SELECT COUNT(*) AS cdr_count FROM voipmonitor_reprocess.cdr;"

# Monitor CPU and I/O usage
htop

# Check MySQL performance
mysqladmin -u root -p processlist

The script will output progress information including files processed, calls found, and errors encountered.

=== Step 5: View Reprocessed Data in the GUI

Once reprocessing is complete, you need to configure the GUI to connect to the new database:

ℹ️ Note: You can either switch the primary GUI database connection (affecting all users) or create a dedicated read-only access for viewing reprocessed data without disrupting normal operations.

Option A: Temporarily Switch GUI Database (for quick review): 

// Edit /var/www/html/voipmonitor/configuration.php
// Change the database connection parameters:

$dbhost = 'localhost';
$dbname = 'voipmonitor_reprocess';  // Changed from 'voipmonitor'
$dbuser = 'voipmonitor_reprocess';
$dbpass = 'your_secure_password';

Refresh the GUI in your browser. You will now see the reprocessed calls with your new configuration options applied.

Option B: Use Separate Virtual Host (recommended):

Create a separate Apache/Nginx virtual host pointing to a copy of the GUI configured to connect to the reprocessing database. This allows you to view both production and reprocessed data simultaneously.

=== Step 6: Analyze the Results

Now you can search for the data that your new configuration enabled:

  • To find calls with specific intermediate SIP responses, use the SIP Messages interface in the GUI and filter by response code (e.g., 491)
  • Compare the reprocessed CDR structure with original CDRs to verify the changes are correct
  • Export reports from the reprocessed data as needed

== Common Reprocessing Scenarios

=== Enable SIP Response Storage for Historical Calls

This is the most common use case - you want to find calls containing intermediate SIP responses that were not captured originally.

Configuration: 

[general]
# ... database settings as above ...

# Store all SIP responses (not just final ones)
save_sip_responses = yes

# Also store SIP response text for searching
save_sip_history = responses

After reprocessing: 1. Navigate to CDR → SIP Messages in the GUI 2. Use the Response filter to search for specific codes (e.g., 491, 180, 183) 3. Click on entries to see the full SIP message details and associated CDR

=== Re-process with Enhanced Audio Detection

Update audio analysis settings on historical calls: 

[general]
# ... database settings as above ...

# Enable in-band DTMF detection (was disabled before)
inbanddtmf = yes

# Enable silence detection (was disabled before)
silencedetect = yes
silencethreshold = 512

# Save energy levels for analysis
save-energylevels = yes

=== Test New Capture Rules

See how a new capture_rules configuration would have affected PCAP retention: 

[general]
# ... database settings as above ...

# Define new capture rules
capture_rules = /etc/capture_rules_new.conf

# Save all PCAPs for testing (do not use this in production)
maxpooldays = 0
maxpoolsize = 0

⚠️ Warning: Setting maxpooldays = 0 and maxpoolsize = 0 disables automatic PCAP cleanup. Only use this for testing specific date ranges and clean up manually afterward.

== Troubleshooting

=== Script Errors or Hangs

If process_spooldir.pl encounters issues:

{syntaxhighlight lang="bash}

  1. Run with verbose logging to see detailed progress

perl /var/www/html/voipmonitor/scripts/process_spooldir.pl \ 

 --voipcfg=/etc/voipmonitor-reprocess.conf \
 --spooldir=/var/spool/voipmonitor \
 --fromdate='2024-12-01 00:00' \
 --todate='2024-12-31 23:59' \
 --verbose 2>&1 | tee /tmp/reprocess.log
  1. Check for common issues:
  2. - Disk space: df -h
  3. - MySQL connection: mysql -u voipmonitor_reprocess -p voipmonitor_reprocess
  4. - PCAP file permissions: ls -l /var/spool/voipmonitor

}</syntaxhighlight>

=== Database Connection Issues

Ensure the MySQL user has proper permissions: 

-- Verify user permissions
SHOW GRANTS FOR 'voipmonitor_reprocess'@'localhost';

-- The user should have ALL PRIVILEGES on the voipmonitor_reprocess database

=== No Data in Database After Reprocessing

If the reprocess script completes without errors but the database is empty:

  • Verify the --fromdate and --todate ranges match the spooldir structure
  • Check that PCAP files exist in the specified date ranges: ls -la /var/spool/voipmonitor/2024-12-01/
  • Verify the spooldir parameter points to the correct location
  • Review the script output for any skipped files or warnings

=== CDR Count Doesn't Match Expectations

If you reprocessed a known number of calls but the CDR count differs:

  • The reprocessing script processes PCAP files, not individual calls
  • Multiple CDRs may be generated from a single PCAP file
  • Some calls may have been filtered out by the new configuration settings (e.g., cdronlyanswered = yes)
  • Check the cdr_ignore_response setting which may filter certain response codes

== Performance Optimization Tips

  • Threading: Use --threads=N to parallelize processing on multi-core systems
  • Date Ranges: Process data in smaller date chunks (e.g., one day at a time) for better progress tracking
  • **Off-Peak Hours**: Run reprocessing jobs during low traffic periods to minimize impact on production systems
  • Disk I/O: If possible, place the reprocessing SQL files and PCAP archives on separate disk drives
  • Index Creation: Consider creating database indexes after reprocessing is complete rather than during the process

=syntaxhighlight lang="bash"

  1. Example: Process one month in weekly chunks to track progress better

for week in 1 2 3 4; do 

 start_date="2024-12-01"
 end_date=$(date -d "$start_date + 7 days - 1 second" +%Y-%m-%d\ %H:%M)

 perl /var/www/html/voipmonitor/scripts/process_spooldir.pl \
   --voipcfg=/etc/voipmonitor-reprocess.conf \
   --spooldir=/var/spool/voipmonitor \
   --fromdate="$start_date 00:00" \
   --todate="$end_date 23:59" \
   --verbose 2>&1 | tee "/tmp/reprocess_week$week.log"

 start_date=$end_date

done </syntaxhighlight>

== Cleanup After Reprocessing

Once you have analyzed the reprocessed data and no longer need the reprocessing database: 

# Stop any GUI connections to the reprocessing database
# Either revert GUI configuration or remove the virtual host

# Remove the database and user
mysql -u root -p -e "DROP DATABASE voipmonitor_reprocess;"
mysql -u root -p -e "DROP USER 'voipmonitor_reprocess'@'localhost';"

# Remove the temporary configuration file
rm /etc/voipmonitor-reprocess.conf

== Related Documentation

== AI Summary for RAG

Summary: This document explains how to re-process historical PCAP archives with a different VoIPmonitor configuration using the process_spooldir.pl Perl script. Common use cases include enabling save_sip_responses retroactively to find intermediate SIP response codes (like 491) in old calls, testing new detection algorithms, or applying new configuration options. The process requires creating a separate database, writing a reprocessing config file (--voipcfg), specifying date ranges (--fromdate/--todate), and optionally using threading (--threads). Reproduction is an offline batch operation that reads from spooldir TAR archives and writes new CDR entries without modifying the original PCAP files. After reprocessing, the GUI can be connected to the new database (either by temporarily switching configuration or creating a separate virtual host) to view the results.

Keywords: reprocess, historical data, process_spooldir.pl, save_sip_responses, intermediate SIP responses, 491, SIP response code, offline processing, reprocessing, retroactive analysis, PCAP re-analysis, new database, voipcfg, fromdate, todate

Key Questions:

  • How do I enable SIP response storage for calls that were already captured?
  • How can I find intermediate SIP response codes like 491 in historical data?
  • What is process_spooldir.pl and how do I use it?
  • How do I re-process captured PCAP files with a different configuration?
  • Can I apply new VoIPmonitor settings to old call data?
  • How do I create a separate database for reprocessing analysis?
  • Is it safe to re-process historical data without affecting live monitoring?
  • How do I view reprocessed CDRs in the GUI?
  • Can I test new capture rules on past traffic?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Reprocess_Historical_Calls&oldid=9826"