Manual PCAP Extraction from spooldir: Difference between revisions

From VoIPmonitor.org
Jump to navigation Jump to search
 
(30 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Notes ==
{{DISPLAYTITLE:Manual PCAP Extraction from spooldir}}
'''RTP format:''' With default config shipped with latest voipmonitor sensor, is RTP compression enabled into LZO in time of capture - those LZOed files are tared into RTP archives based on date-hourminute of a call start and its call's call-id.
option '''pcap_dump_zip_rtp = lzo'''


'''SIP format:''' With default config shipped with latest voipmonitor sensor, is SIP compression enabled after tar archive was created:
'''This is an expert-level guide for manually extracting individual call PCAP files from VoIPmonitor's TAR archives and for generating audio files directly from a PCAP. These procedures are useful for offline analysis, scripting, and advanced troubleshooting.'''
option '''tar_compress_sip = gzip'''


== Export pcap file with default config used ==
== Understanding the Storage Format ==
=== precondition ===
To efficiently store millions of calls, VoIPmonitor does not save each call as a separate file. Instead, it groups captures into `.tar` archives based on the minute they started.
call needs to be captured with sensor's compression settings like in default voipmonitor.conf
*'''PCAP Format:''' Inside the TAR archives, individual RTP PCAP files are typically compressed with '''LZO''', while SIP PCAP files are compressed with '''Gzip'''.
pcap_dump_zip_rtp = lzo
*'''Directory Structure:''' Archives are stored in a nested directory structure: `[spooldir]/YYYY-MM-DD/HH/MM/`
option tar_compress_sip = gzip


=== Information needed for export from CDR detail ===
This guide will show you how to work with this structure.
You will need:
1.CDR.id (103)
2.Date time of a call start (2016-08-23 16:37:38)
3.Call-ID (CwA8j-SNSN)
4.Location of your spooldir ('spooldir' option is defined in /etc/voipmonitor.conf)


example : [[File: cdr_detail_for_export_pcap_default.jpg]]
== Part 1: How to Manually Extract PCAP Files ==
This process allows you to pull the complete SIP and RTP packet capture for a single call out of the TAR archives.


=== export SIP pcap ===
=== Step 1: Gather Required Call Information ===
From spooldir location (by default its '/var/spool/voipmonitor' and calldate start '2016-08-23 16:37:38' in example and from CALL-ID header 'CwA8j-SNSN' you can write command:
First, you need four key pieces of information for the call you want to extract. You can find these in the GUI's Call Detail Record (CDR) view or by querying the database.
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz' 'CwA8j-SNSN.pcap*' /tmp/expsip.pcap
#'''CDR ID:''' The unique ID from the `cdr.id` column (e.g., `103`).
#'''Call Date:''' The full start time of the call from `cdr.calldate` (e.g., `2016-08-23 16:37:38`).
#'''Call-ID:''' The SIP Call-ID, stored in `cdr_next.fbasename` (e.g., `CwA8j-SNSN`).
#'''Spooldir Path:''' The path to your spool directory, defined in `voipmonitor.conf` (e.g., `/var/spool/voipmonitor`).


=== export RTP pcap ===
[[File: cdr_detail_for_export_pcap_default.jpg|A CDR detail view showing where to find the necessary information.]]
First we will need to get '''lzo positions''' from database (calldate start '2016-08-23 16:37:38'in example and from CALL-ID header 'CwA8j-SNSN' you can write a query), type=2 (means RTP filetype):
mysql> SELECT pos FROM voipmonitor.cdr_tar_part where cdr_id = 103 and type = 2 and calldate = '2016-08-23 16:37:38';


=== Step 2: Flush the TAR Cache (for recent calls) ===
If you are extracting a very recent call (from the last few minutes), its data may still be in the sniffer's memory buffer and not yet written to the TAR file on disk. You must force the sniffer to flush its cache via the manager API.


Returned:
;Find the exact path to the TAR file and send the command:
pos: 0
<pre>
pos: 164352
# Example path to a SIP tarball
pos: 328704
TAR_PATH="/var/spool/voipmonitor/2024-06-30/10/05/SIP/sip_2024-06-30-10-05.tar.gz"
pos: 493056
4 rows in set (0,00 sec)


Second we use positions returned from db to '''export RTP and unLZO''' using voipmonitor binary:
echo "flush_tar '$TAR_PATH'" | nc 127.0.0.1 5029
/usr/local/sbin/voipmonitor -kc -d /var/spool/voipmonitor/ --untar-gui='/var/spool/voipmonitor//2016-08-23/16/37/RTP/rtp_2016-08-23-16-37.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/exprtp.pcap
</pre>


=== Step 3: Extract the SIP PCAP File ===
SIP packets for a call are stored in a single compressed file within the SIP TAR archive. You can extract it using the `tar` command. The filename inside the archive is based on the SIP Call-ID.


;Construct the path to the SIP TAR file and run the command:
<pre>
# The path is constructed from the call's start time
# Example: /var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz


== Export pcap file when LZO compression disabled for RTP in config ==
# Use tar to extract the file matching the Call-ID and redirect output to a new file
=== preconditions ===
tar --wildcards -xOf '/path/to/sip.tar.gz' '*CALL-ID*.pcap.gz' > /tmp/sip.pcap.gz
call captured when sensor's compression settings changed from default voipmonitor.conf
pcap_dump_zip_rtp = no
option tar_compress_sip = gzip


=== info needed to collect from cdr ===
# Decompress the resulting file
Call-ID
gunzip /tmp/sip.pcap.gz
Date time of a call start
</pre>


=== export SIP pcap ===
=== Step 4: Extract the RTP PCAP File ===
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/SIP/sip_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > ./exportSIP.pcap
RTP streams are often split into multiple chunks within the RTP TAR archive. The most efficient way to extract them is to get their exact positions from the database.


=== export RTP pcap ===
;1. Query the database for RTP chunk positions:
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > ./exportRTP.pcap
<pre>
-- Use the CDR ID and full calldate of your target call
SELECT pos FROM voipmonitor.cdr_tar_part WHERE cdr_id = 103 AND type = 2 AND calldate = '2016-08-23 16:37:38';
</pre>
This will return a list of numeric positions (offsets).


=== merge SIP and RTP into one file ===
;2. Use the `voipmonitor` binary to extract the chunks:
mergecap -w /tmp/export.pcap /tmp/exportSIP.pcap /tmp/exportRTP.pcap
The sensor binary itself has a powerful `--untar-gui` mode that can extract multiple chunks by their offsets and combine them into a single, decompressed PCAP file.
<pre>
# Command format:
# voipmonitor -kc --untar-gui='/path/to/rtp.tar Call-ID.pcap offset1,offset2,... output.pcap'
 
/usr/local/sbin/voipmonitor -kc --untar-gui='/var/spool/voipmonitor/2016-08-23/16/37/RTP/rtp.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/rtp.pcap'
</pre>
The resulting `/tmp/rtp.pcap` file will contain all RTP packets for the call and will already be decompressed (LZO is handled internally).
 
=== Step 5: Merge SIP and RTP (Optional) ===
To create a single PCAP file containing the entire call for analysis in tools like Wireshark, use `mergecap`.
<pre>
# Install mergecap if you don't have it (part of the wireshark package)
# sudo apt-get install wireshark-common
# sudo yum install wireshark
 
mergecap -w /tmp/full_call.pcap /tmp/sip.pcap /tmp/rtp.pcap
</pre>
 
== Part 2: How to Generate an Audio File from PCAP ==
If you have a complete PCAP file (containing both SIP and RTP), you can use the `voipmonitor` binary as a command-line tool to convert it into an audio file (OGG, WAV, or MP3) without needing a running GUI or database.
 
=== Step 1: Create a Special Configuration File ===
Create a temporary configuration file that tells the sniffer to run in a special "audio conversion" mode.
;Create a file, e.g., `/tmp/voipmonitor-audio.conf`:
<pre>
# /tmp/voipmonitor-audio.conf
[general]
 
# Define an output directory for the audio files
spooldir = /tmp/audio_output
 
# Set the desired audio format (ogg, wav, or mp3)
saveaudio = ogg
 
# Disable all other features
nocdr = yes
savesip = no
savertp = no
</pre>
 
=== Step 2: Run the Conversion Command ===
Execute the `voipmonitor` binary, pointing it to your special configuration and the source PCAP file.
<pre>
# Command format:
# voipmonitor --config-file=[config] -k -v1 -r [source_pcap]
 
voipmonitor --config-file=/tmp/voipmonitor-audio.conf -k -v1 -r /tmp/full_call.pcap
</pre>
The sniffer will process the PCAP file and save the resulting audio file in the `spooldir` defined in your temporary config (e.g., `/tmp/audio_output/`). The filename will be the call's SIP Call-ID.
 
== AI Summary for RAG ==
'''Summary:''' This guide provides expert-level instructions for two advanced, command-line tasks: manually extracting call PCAP files from VoIPmonitor's TAR archives and generating audio files directly from a PCAP. The first part details a five-step process for PCAP extraction: 1) Gathering essential call information (CDR ID, calldate, Call-ID) from the database. 2) Using the `flush_tar` manager API command for recent calls. 3) Extracting the compressed SIP PCAP using the `tar` command. 4) Efficiently extracting RTP packets by first querying the `cdr_tar_part` table for chunk offsets and then using the `voipmonitor --untar-gui` command. 5) Merging the SIP and RTP PCAPs with `mergecap`. The second part of the guide explains how to convert a complete PCAP file into an audio file (OGG/WAV). This involves creating a special, minimal `voipmonitor.conf` file with `saveaudio=ogg` and `nocdr=yes`, and then running the `voipmonitor` binary with the `-r` flag to process the source PCAP file.
'''Keywords:''' extract pcap, export pcap, tar archive, `cdr_tar_part`, `--untar-gui`, `flush_tar`, mergecap, generate audio, create wav, pcap to audio, command line, cli, `saveaudio`, `nocdr`
'''Key Questions:'''
* How can I manually extract a single call's PCAP file from the TAR archives?
* How does VoIPmonitor store PCAP files on disk?
* What is the purpose of the `cdr_tar_part` table?
* How do I use the `voipmonitor --untar-gui` command?
* How can I create a WAV or OGG file from a PCAP without using the GUI?
* How to merge SIP and RTP pcap files into one?
* Why do I need to run `flush_tar` before extracting a recent call?

Latest revision as of 22:52, 30 June 2025


This is an expert-level guide for manually extracting individual call PCAP files from VoIPmonitor's TAR archives and for generating audio files directly from a PCAP. These procedures are useful for offline analysis, scripting, and advanced troubleshooting.

Understanding the Storage Format

To efficiently store millions of calls, VoIPmonitor does not save each call as a separate file. Instead, it groups captures into `.tar` archives based on the minute they started.

  • PCAP Format: Inside the TAR archives, individual RTP PCAP files are typically compressed with LZO, while SIP PCAP files are compressed with Gzip.
  • Directory Structure: Archives are stored in a nested directory structure: `[spooldir]/YYYY-MM-DD/HH/MM/`

This guide will show you how to work with this structure.

Part 1: How to Manually Extract PCAP Files

This process allows you to pull the complete SIP and RTP packet capture for a single call out of the TAR archives.

Step 1: Gather Required Call Information

First, you need four key pieces of information for the call you want to extract. You can find these in the GUI's Call Detail Record (CDR) view or by querying the database.

  1. CDR ID: The unique ID from the `cdr.id` column (e.g., `103`).
  2. Call Date: The full start time of the call from `cdr.calldate` (e.g., `2016-08-23 16:37:38`).
  3. Call-ID: The SIP Call-ID, stored in `cdr_next.fbasename` (e.g., `CwA8j-SNSN`).
  4. Spooldir Path: The path to your spool directory, defined in `voipmonitor.conf` (e.g., `/var/spool/voipmonitor`).

A CDR detail view showing where to find the necessary information.

Step 2: Flush the TAR Cache (for recent calls)

If you are extracting a very recent call (from the last few minutes), its data may still be in the sniffer's memory buffer and not yet written to the TAR file on disk. You must force the sniffer to flush its cache via the manager API.

Find the exact path to the TAR file and send the command
# Example path to a SIP tarball
TAR_PATH="/var/spool/voipmonitor/2024-06-30/10/05/SIP/sip_2024-06-30-10-05.tar.gz"

echo "flush_tar '$TAR_PATH'" | nc 127.0.0.1 5029

Step 3: Extract the SIP PCAP File

SIP packets for a call are stored in a single compressed file within the SIP TAR archive. You can extract it using the `tar` command. The filename inside the archive is based on the SIP Call-ID.

Construct the path to the SIP TAR file and run the command
# The path is constructed from the call's start time
# Example: /var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz

# Use tar to extract the file matching the Call-ID and redirect output to a new file
tar --wildcards -xOf '/path/to/sip.tar.gz' '*CALL-ID*.pcap.gz' > /tmp/sip.pcap.gz

# Decompress the resulting file
gunzip /tmp/sip.pcap.gz

Step 4: Extract the RTP PCAP File

RTP streams are often split into multiple chunks within the RTP TAR archive. The most efficient way to extract them is to get their exact positions from the database.

1. Query the database for RTP chunk positions
-- Use the CDR ID and full calldate of your target call
SELECT pos FROM voipmonitor.cdr_tar_part WHERE cdr_id = 103 AND type = 2 AND calldate = '2016-08-23 16:37:38';

This will return a list of numeric positions (offsets).

2. Use the `voipmonitor` binary to extract the chunks

The sensor binary itself has a powerful `--untar-gui` mode that can extract multiple chunks by their offsets and combine them into a single, decompressed PCAP file.

# Command format:
# voipmonitor -kc --untar-gui='/path/to/rtp.tar Call-ID.pcap offset1,offset2,... output.pcap'

/usr/local/sbin/voipmonitor -kc --untar-gui='/var/spool/voipmonitor/2016-08-23/16/37/RTP/rtp.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/rtp.pcap'

The resulting `/tmp/rtp.pcap` file will contain all RTP packets for the call and will already be decompressed (LZO is handled internally).

Step 5: Merge SIP and RTP (Optional)

To create a single PCAP file containing the entire call for analysis in tools like Wireshark, use `mergecap`.

# Install mergecap if you don't have it (part of the wireshark package)
# sudo apt-get install wireshark-common
# sudo yum install wireshark

mergecap -w /tmp/full_call.pcap /tmp/sip.pcap /tmp/rtp.pcap

Part 2: How to Generate an Audio File from PCAP

If you have a complete PCAP file (containing both SIP and RTP), you can use the `voipmonitor` binary as a command-line tool to convert it into an audio file (OGG, WAV, or MP3) without needing a running GUI or database.

Step 1: Create a Special Configuration File

Create a temporary configuration file that tells the sniffer to run in a special "audio conversion" mode.

Create a file, e.g., `/tmp/voipmonitor-audio.conf`
# /tmp/voipmonitor-audio.conf
[general]

# Define an output directory for the audio files
spooldir = /tmp/audio_output

# Set the desired audio format (ogg, wav, or mp3)
saveaudio = ogg

# Disable all other features
nocdr = yes
savesip = no
savertp = no

Step 2: Run the Conversion Command

Execute the `voipmonitor` binary, pointing it to your special configuration and the source PCAP file.

# Command format:
# voipmonitor --config-file=[config] -k -v1 -r [source_pcap]

voipmonitor --config-file=/tmp/voipmonitor-audio.conf -k -v1 -r /tmp/full_call.pcap

The sniffer will process the PCAP file and save the resulting audio file in the `spooldir` defined in your temporary config (e.g., `/tmp/audio_output/`). The filename will be the call's SIP Call-ID.

AI Summary for RAG

Summary: This guide provides expert-level instructions for two advanced, command-line tasks: manually extracting call PCAP files from VoIPmonitor's TAR archives and generating audio files directly from a PCAP. The first part details a five-step process for PCAP extraction: 1) Gathering essential call information (CDR ID, calldate, Call-ID) from the database. 2) Using the `flush_tar` manager API command for recent calls. 3) Extracting the compressed SIP PCAP using the `tar` command. 4) Efficiently extracting RTP packets by first querying the `cdr_tar_part` table for chunk offsets and then using the `voipmonitor --untar-gui` command. 5) Merging the SIP and RTP PCAPs with `mergecap`. The second part of the guide explains how to convert a complete PCAP file into an audio file (OGG/WAV). This involves creating a special, minimal `voipmonitor.conf` file with `saveaudio=ogg` and `nocdr=yes`, and then running the `voipmonitor` binary with the `-r` flag to process the source PCAP file. Keywords: extract pcap, export pcap, tar archive, `cdr_tar_part`, `--untar-gui`, `flush_tar`, mergecap, generate audio, create wav, pcap to audio, command line, cli, `saveaudio`, `nocdr` Key Questions:

  • How can I manually extract a single call's PCAP file from the TAR archives?
  • How does VoIPmonitor store PCAP files on disk?
  • What is the purpose of the `cdr_tar_part` table?
  • How do I use the `voipmonitor --untar-gui` command?
  • How can I create a WAV or OGG file from a PCAP without using the GUI?
  • How to merge SIP and RTP pcap files into one?
  • Why do I need to run `flush_tar` before extracting a recent call?