Manual PCAP Extraction from spooldir: Difference between revisions

From VoIPmonitor.org
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
== Notes ==
{{DISPLAYTITLE:Manual PCAP Extraction from spooldir}}
'''RTP format:''' With default voipmonitor.conf RTP pcap chunks are compressed by LZO which are tared and archived in directory in date-hourminute


option '''pcap_dump_zip_rtp = lzo'''
'''This is an expert-level guide for manually extracting individual call PCAP files from VoIPmonitor's TAR archives and for generating audio files directly from a PCAP. These procedures are useful for offline analysis, scripting, and advanced troubleshooting.'''


'''SIP format:''' With default voipmonitor.conf SIP compression uses gzip
== Understanding the Storage Format ==
option '''tar_compress_sip = gzip'''
To efficiently store millions of calls, VoIPmonitor does not save each call as a separate file. Instead, it groups captures into `.tar` archives based on the minute they started.
*'''PCAP Format:''' Inside the TAR archives, individual RTP PCAP files are typically compressed with '''LZO''', while SIP PCAP files are compressed with '''Gzip'''.
*'''Directory Structure:''' Archives are stored in a nested directory structure: `[spooldir]/YYYY-MM-DD/HH/MM/`


This guide will show you how to work with this structure.


== Export pcap file with default config used ==
== Part 1: How to Manually Extract PCAP Files ==
This process allows you to pull the complete SIP and RTP packet capture for a single call out of the TAR archives.


=== Get information about CDR from database ===
=== Step 1: Gather Required Call Information ===
You will need:
First, you need four key pieces of information for the call you want to extract. You can find these in the GUI's Call Detail Record (CDR) view or by querying the database.
1.cdr.id (103)
#'''CDR ID:''' The unique ID from the `cdr.id` column (e.g., `103`).
2.Date time of call start (2016-08-23 16:37:38)
#'''Call Date:''' The full start time of the call from `cdr.calldate` (e.g., `2016-08-23 16:37:38`).
3.Call-ID (CwA8j-SNSN)
#'''Call-ID:''' The SIP Call-ID, stored in `cdr_next.fbasename` (e.g., `CwA8j-SNSN`).
4.Location of your spooldir (spooldir=X)  
#'''Spooldir Path:''' The path to your spool directory, defined in `voipmonitor.conf` (e.g., `/var/spool/voipmonitor`).


example : [[File: cdr_detail_for_export_pcap_default.jpg]]
[[File: cdr_detail_for_export_pcap_default.jpg|A CDR detail view showing where to find the necessary information.]]


SQL Query:  
=== Step 2: Flush the TAR Cache (for recent calls) ===
SELECT cdr.calldate,cdr.caller,cdr.called,cdr.id as cdrID,cdr_next.fbasename as callID
If you are extracting a very recent call (from the last few minutes), its data may still be in the sniffer's memory buffer and not yet written to the TAR file on disk. You must force the sniffer to flush its cache via the manager API.
FROM cdr,cdr_next
WHERE cdr.id=cdr_next.cdr_ID AND cdr.calldate >= '2017-02-01 00:00:00' AND cdr.calldate <= '2017-02-01 23:59:59' AND cdr.caller like '+222%';


You MUST use cdr.calldate condition otherwise database will be overloaded by searching in all partitions
;Find the exact path to the TAR file and send the command:
=== flush data to tar file===
<pre>
Before you ask tar binary or voipmonitor for file extraction, you need to flush data from sniffer's cache first ( if the tarball file is still opened ) send api command to sniffer service
# Example path to a SIP tarball
echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1
TAR_PATH="/var/spool/voipmonitor/2024-06-30/10/05/SIP/sip_2024-06-30-10-05.tar.gz"


=== export SIP pcap ===
echo "flush_tar '$TAR_PATH'" | nc 127.0.0.1 5029
</pre>


tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz' 'CwA8j-SNSN.pcap*' > /tmp/expsip.pcap
=== Step 3: Extract the SIP PCAP File ===
SIP packets for a call are stored in a single compressed file within the SIP TAR archive. You can extract it using the `tar` command. The filename inside the archive is based on the SIP Call-ID.


;Construct the path to the SIP TAR file and run the command:
<pre>
# The path is constructed from the call's start time
# Example: /var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz


=== export RTP ===
# Use tar to extract the file matching the Call-ID and redirect output to a new file
tar --wildcards -xOf '/path/to/sip.tar.gz' '*CALL-ID*.pcap.gz' > /tmp/sip.pcap.gz


Get RTP positions
# Decompress the resulting file
gunzip /tmp/sip.pcap.gz
</pre>


mysql> SELECT pos FROM voipmonitor.cdr_tar_part where cdr_id = 103 and type = 2 and calldate = '2016-08-23 16:37:38';
=== Step 4: Extract the RTP PCAP File ===
RTP streams are often split into multiple chunks within the RTP TAR archive. The most efficient way to extract them is to get their exact positions from the database.


'''Returned:'''
;1. Query the database for RTP chunk positions:
pos: 0
<pre>
pos: 164352
-- Use the CDR ID and full calldate of your target call
pos: 328704
SELECT pos FROM voipmonitor.cdr_tar_part WHERE cdr_id = 103 AND type = 2 AND calldate = '2016-08-23 16:37:38';
pos: 493056
</pre>
4 rows in set (0,00 sec)
This will return a list of numeric positions (offsets).


use positions returned from db and extract pcap  
;2. Use the `voipmonitor` binary to extract the chunks:
The sensor binary itself has a powerful `--untar-gui` mode that can extract multiple chunks by their offsets and combine them into a single, decompressed PCAP file.
<pre>
# Command format:
# voipmonitor -kc --untar-gui='/path/to/rtp.tar Call-ID.pcap offset1,offset2,... output.pcap'


/usr/local/sbin/voipmonitor -kc -d /var/spool/voipmonitor/ --untar-gui='/var/spool/voipmonitor//2016-08-23/16/37/RTP/rtp_2016-08-23-16-37.tar CwA8j-SNSN.pcap 0,164352,328704,493056 rtp.pcap'
/usr/local/sbin/voipmonitor -kc --untar-gui='/var/spool/voipmonitor/2016-08-23/16/37/RTP/rtp.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/rtp.pcap'
#rtp.pcap is already decompressed (no unLZO needed)  
</pre>
The resulting `/tmp/rtp.pcap` file will contain all RTP packets for the call and will already be decompressed (LZO is handled internally).


==Alternative RTP extraction without knowing positions from database==
=== Step 5: Merge SIP and RTP (Optional) ===
this will consume more IO reads as tar file has to be fully scanned, additionally you need to have tar file closed or you need to ask sniffer to flush cashed data if the file is still opened.
To create a single PCAP file containing the entire call for analysis in tools like Wireshark, use `mergecap`.
<pre>
# Install mergecap if you don't have it (part of the wireshark package)
# sudo apt-get install wireshark-common
# sudo yum install wireshark


tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > rtp.pcap
mergecap -w /tmp/full_call.pcap /tmp/sip.pcap /tmp/rtp.pcap
# if LZO compression for RTP pcaps is enabled '''
</pre>
voipmonitor -kc --unlzo-gui='/path/to/rtp.pcap /path/to/rtp-uncompressed.pcap'
#if path to file is not absolute (/...) it is relative to the spooldir directory
=== flush partially written data into tar file ===
Before you ask tar binary for file extraction you need to flush data from sniffer first ( if the pcap file is still opened ) using sniffer's api like:
echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1


=== merge SIP and RTP into one file ===
== Part 2: How to Generate an Audio File from PCAP ==
(apt-get install tshark | yum install wireshark)  
If you have a complete PCAP file (containing both SIP and RTP), you can use the `voipmonitor` binary as a command-line tool to convert it into an audio file (OGG, WAV, or MP3) without needing a running GUI or database.
Note: the rtp.pcap needs to be de-lzoed first (lzo is default compression method used by sniffer service when rtp stored to spooldir)


mergecap -w rtp.pcap sip.pcap final.pcap
=== Step 1: Create a Special Configuration File ===
Create a temporary configuration file that tells the sniffer to run in a special "audio conversion" mode.
;Create a file, e.g., `/tmp/voipmonitor-audio.conf`:
<pre>
# /tmp/voipmonitor-audio.conf
[general]


# Define an output directory for the audio files
spooldir = /tmp/audio_output


=Create the AUDIO using pcap file=
# Set the desired audio format (ogg, wav, or mp3)
after you get/have SIP+RTP in a single pcap file you can ask sniffer's binary for create audio file without need to have GUI installed/ db used (works only for g711 calls without the GUI)
saveaudio = ogg
==prepare voipmonitor-audio.conf==
Change following options copy of the sniffer's config file - sniffer needs to be set to analyze traffic (not mirror senders):
spooldir=/myAudio
savesip=no
savertp=no
savertcp=no
savegraph=no
saveaudio=ogg
#saveaudio=wav
saveaudio_stereo = yes
nocdr=yes
#for possible to create audio also from other then g711 codecs( g711 is supported with sniffer directly) you need to have valid license for GUI and enable following option with path to gui's keycheck file
#keycheck = /var/www/html/php/lib/keycheck.php


==call sniffer's binary==
# Disable all other features
Call the sniffer's binary with file.pcap as argument to create audio from it in spooldir defined above:
nocdr = yes
voipmonitor --config-file=/etc/voipmonitor-audio.conf -k -v1 -r /tmp/final.pcap
savesip = no
savertp = no
</pre>


==Results location==
=== Step 2: Run the Conversion Command ===
location of created audio files (it's names is CALL-ID observed in INVITE packet) depends on option **spooldiroldschema**
Execute the `voipmonitor` binary, pointing it to your special configuration and the source PCAP file.
<pre>
# Command format:
# voipmonitor --config-file=[config] -k -v1 -r [source_pcap]


===spooldiroldschema = no===
voipmonitor --config-file=/tmp/voipmonitor-audio.conf -k -v1 -r /tmp/full_call.pcap
By default, the audio file will be created in directory /${SPOOLDIR}/DATE/HOUR/MINUTE/AUDIO/ (based on timestamp of packets
</pre>
/myAudio/2020-07-31/19/52/AUDIO/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)
The sniffer will process the PCAP file and save the resulting audio file in the `spooldir` defined in your temporary config (e.g., `/tmp/audio_output/`). The filename will be the call's SIP Call-ID.


===spooldiroldschema = yes===
== AI Summary for RAG ==
the audio will be made in DATE dir (based on timestamp of packets
'''Summary:''' This guide provides expert-level instructions for two advanced, command-line tasks: manually extracting call PCAP files from VoIPmonitor's TAR archives and generating audio files directly from a PCAP. The first part details a five-step process for PCAP extraction: 1) Gathering essential call information (CDR ID, calldate, Call-ID) from the database. 2) Using the `flush_tar` manager API command for recent calls. 3) Extracting the compressed SIP PCAP using the `tar` command. 4) Efficiently extracting RTP packets by first querying the `cdr_tar_part` table for chunk offsets and then using the `voipmonitor --untar-gui` command. 5) Merging the SIP and RTP PCAPs with `mergecap`. The second part of the guide explains how to convert a complete PCAP file into an audio file (OGG/WAV). This involves creating a special, minimal `voipmonitor.conf` file with `saveaudio=ogg` and `nocdr=yes`, and then running the `voipmonitor` binary with the `-r` flag to process the source PCAP file.
/myAudio/2020-07-31/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)
'''Keywords:''' extract pcap, export pcap, tar archive, `cdr_tar_part`, `--untar-gui`, `flush_tar`, mergecap, generate audio, create wav, pcap to audio, command line, cli, `saveaudio`, `nocdr`
'''Key Questions:'''
* How can I manually extract a single call's PCAP file from the TAR archives?
* How does VoIPmonitor store PCAP files on disk?
* What is the purpose of the `cdr_tar_part` table?
* How do I use the `voipmonitor --untar-gui` command?
* How can I create a WAV or OGG file from a PCAP without using the GUI?
* How to merge SIP and RTP pcap files into one?
* Why do I need to run `flush_tar` before extracting a recent call?

Revision as of 22:51, 30 June 2025


This is an expert-level guide for manually extracting individual call PCAP files from VoIPmonitor's TAR archives and for generating audio files directly from a PCAP. These procedures are useful for offline analysis, scripting, and advanced troubleshooting.

Understanding the Storage Format

To efficiently store millions of calls, VoIPmonitor does not save each call as a separate file. Instead, it groups captures into `.tar` archives based on the minute they started.

  • PCAP Format: Inside the TAR archives, individual RTP PCAP files are typically compressed with LZO, while SIP PCAP files are compressed with Gzip.
  • Directory Structure: Archives are stored in a nested directory structure: `[spooldir]/YYYY-MM-DD/HH/MM/`

This guide will show you how to work with this structure.

Part 1: How to Manually Extract PCAP Files

This process allows you to pull the complete SIP and RTP packet capture for a single call out of the TAR archives.

Step 1: Gather Required Call Information

First, you need four key pieces of information for the call you want to extract. You can find these in the GUI's Call Detail Record (CDR) view or by querying the database.

  1. CDR ID: The unique ID from the `cdr.id` column (e.g., `103`).
  2. Call Date: The full start time of the call from `cdr.calldate` (e.g., `2016-08-23 16:37:38`).
  3. Call-ID: The SIP Call-ID, stored in `cdr_next.fbasename` (e.g., `CwA8j-SNSN`).
  4. Spooldir Path: The path to your spool directory, defined in `voipmonitor.conf` (e.g., `/var/spool/voipmonitor`).

A CDR detail view showing where to find the necessary information.

Step 2: Flush the TAR Cache (for recent calls)

If you are extracting a very recent call (from the last few minutes), its data may still be in the sniffer's memory buffer and not yet written to the TAR file on disk. You must force the sniffer to flush its cache via the manager API.

Find the exact path to the TAR file and send the command
# Example path to a SIP tarball
TAR_PATH="/var/spool/voipmonitor/2024-06-30/10/05/SIP/sip_2024-06-30-10-05.tar.gz"

echo "flush_tar '$TAR_PATH'" | nc 127.0.0.1 5029

Step 3: Extract the SIP PCAP File

SIP packets for a call are stored in a single compressed file within the SIP TAR archive. You can extract it using the `tar` command. The filename inside the archive is based on the SIP Call-ID.

Construct the path to the SIP TAR file and run the command
# The path is constructed from the call's start time
# Example: /var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz

# Use tar to extract the file matching the Call-ID and redirect output to a new file
tar --wildcards -xOf '/path/to/sip.tar.gz' '*CALL-ID*.pcap.gz' > /tmp/sip.pcap.gz

# Decompress the resulting file
gunzip /tmp/sip.pcap.gz

Step 4: Extract the RTP PCAP File

RTP streams are often split into multiple chunks within the RTP TAR archive. The most efficient way to extract them is to get their exact positions from the database.

1. Query the database for RTP chunk positions
-- Use the CDR ID and full calldate of your target call
SELECT pos FROM voipmonitor.cdr_tar_part WHERE cdr_id = 103 AND type = 2 AND calldate = '2016-08-23 16:37:38';

This will return a list of numeric positions (offsets).

2. Use the `voipmonitor` binary to extract the chunks

The sensor binary itself has a powerful `--untar-gui` mode that can extract multiple chunks by their offsets and combine them into a single, decompressed PCAP file.

# Command format:
# voipmonitor -kc --untar-gui='/path/to/rtp.tar Call-ID.pcap offset1,offset2,... output.pcap'

/usr/local/sbin/voipmonitor -kc --untar-gui='/var/spool/voipmonitor/2016-08-23/16/37/RTP/rtp.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/rtp.pcap'

The resulting `/tmp/rtp.pcap` file will contain all RTP packets for the call and will already be decompressed (LZO is handled internally).

Step 5: Merge SIP and RTP (Optional)

To create a single PCAP file containing the entire call for analysis in tools like Wireshark, use `mergecap`.

# Install mergecap if you don't have it (part of the wireshark package)
# sudo apt-get install wireshark-common
# sudo yum install wireshark

mergecap -w /tmp/full_call.pcap /tmp/sip.pcap /tmp/rtp.pcap

Part 2: How to Generate an Audio File from PCAP

If you have a complete PCAP file (containing both SIP and RTP), you can use the `voipmonitor` binary as a command-line tool to convert it into an audio file (OGG, WAV, or MP3) without needing a running GUI or database.

Step 1: Create a Special Configuration File

Create a temporary configuration file that tells the sniffer to run in a special "audio conversion" mode.

Create a file, e.g., `/tmp/voipmonitor-audio.conf`
# /tmp/voipmonitor-audio.conf
[general]

# Define an output directory for the audio files
spooldir = /tmp/audio_output

# Set the desired audio format (ogg, wav, or mp3)
saveaudio = ogg

# Disable all other features
nocdr = yes
savesip = no
savertp = no

Step 2: Run the Conversion Command

Execute the `voipmonitor` binary, pointing it to your special configuration and the source PCAP file.

# Command format:
# voipmonitor --config-file=[config] -k -v1 -r [source_pcap]

voipmonitor --config-file=/tmp/voipmonitor-audio.conf -k -v1 -r /tmp/full_call.pcap

The sniffer will process the PCAP file and save the resulting audio file in the `spooldir` defined in your temporary config (e.g., `/tmp/audio_output/`). The filename will be the call's SIP Call-ID.

AI Summary for RAG

Summary: This guide provides expert-level instructions for two advanced, command-line tasks: manually extracting call PCAP files from VoIPmonitor's TAR archives and generating audio files directly from a PCAP. The first part details a five-step process for PCAP extraction: 1) Gathering essential call information (CDR ID, calldate, Call-ID) from the database. 2) Using the `flush_tar` manager API command for recent calls. 3) Extracting the compressed SIP PCAP using the `tar` command. 4) Efficiently extracting RTP packets by first querying the `cdr_tar_part` table for chunk offsets and then using the `voipmonitor --untar-gui` command. 5) Merging the SIP and RTP PCAPs with `mergecap`. The second part of the guide explains how to convert a complete PCAP file into an audio file (OGG/WAV). This involves creating a special, minimal `voipmonitor.conf` file with `saveaudio=ogg` and `nocdr=yes`, and then running the `voipmonitor` binary with the `-r` flag to process the source PCAP file. Keywords: extract pcap, export pcap, tar archive, `cdr_tar_part`, `--untar-gui`, `flush_tar`, mergecap, generate audio, create wav, pcap to audio, command line, cli, `saveaudio`, `nocdr` Key Questions:

  • How can I manually extract a single call's PCAP file from the TAR archives?
  • How does VoIPmonitor store PCAP files on disk?
  • What is the purpose of the `cdr_tar_part` table?
  • How do I use the `voipmonitor --untar-gui` command?
  • How can I create a WAV or OGG file from a PCAP without using the GUI?
  • How to merge SIP and RTP pcap files into one?
  • Why do I need to run `flush_tar` before extracting a recent call?