Difference between revisions of "Manual export of pcap files from spooldir"

From VoIPmonitor.org
Jump to navigation Jump to search
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Notes ==
 
== Notes ==
'''RTP format:''' With default config shipped with latest voipmonitor sensor, is RTP compression enabled into LZO in time of capture - those LZOed files are tared into RTP archives based on date-hourminute of a call start and its call's call-id.
+
'''RTP format:''' With default voipmonitor.conf RTP pcap chunks are compressed by LZO which are tared and archived in directory in date-hourminute
 +
 
 
  option '''pcap_dump_zip_rtp = lzo'''
 
  option '''pcap_dump_zip_rtp = lzo'''
  
'''SIP format:''' With default config shipped with latest voipmonitor sensor, is SIP compression enabled after tar archive was created:
+
'''SIP format:''' With default voipmonitor.conf SIP compression uses gzip
 
  option '''tar_compress_sip = gzip'''
 
  option '''tar_compress_sip = gzip'''
 
  
  
 
== Export pcap file with default config used ==
 
== Export pcap file with default config used ==
=== precondition ===
 
call needs to be captured with sensor's compression settings like in default voipmonitor.conf (no change to compression options)
 
pcap_dump_zip_rtp = lzo
 
option tar_compress_sip = gzip
 
  
=== information needed from CDR detail for export ===
+
=== Get information about CDR from database ===
 
You will need:
 
You will need:
  1.CDR.id (103)
+
  1.cdr.id (103)
  2.Date time of a call start (2016-08-23 16:37:38)
+
  2.Date time of call start (2016-08-23 16:37:38)
 
  3.Call-ID (CwA8j-SNSN)
 
  3.Call-ID (CwA8j-SNSN)
  4.Location of your spooldir ('spooldir' option is defined in /etc/voipmonitor.conf)
+
  4.Location of your spooldir (spooldir=X)  
  
 
example : [[File: cdr_detail_for_export_pcap_default.jpg]]
 
example : [[File: cdr_detail_for_export_pcap_default.jpg]]
 +
 +
SQL Query:
 +
SELECT cdr.calldate,cdr.caller,cdr.called,cdr.id as cdrID,cdr_next.fbasename as callID
 +
FROM cdr,cdr_next
 +
WHERE cdr.id=cdr_next.cdr_ID AND cdr.calldate >= '2017-02-01 00:00:00' AND cdr.calldate <= '2017-02-01 23:59:59' AND cdr.caller like '+222%';
 +
 +
You MUST use cdr.calldate condition otherwise database will be overloaded by searching in all partitions
 +
=== flush data to tar file===
 +
Before you ask tar binary or voipmonitor for file extraction, you need to flush data from sniffer's cache first ( if the pcap file is still opened ) send api command to sniffer service
 +
echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1
  
 
=== export SIP pcap ===
 
=== export SIP pcap ===
From spooldir location (by default its '/var/spool/voipmonitor' and calldate start '2016-08-23 16:37:38' in example and from CALL-ID header 'CwA8j-SNSN' you can write command:
+
 
 
  tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz' 'CwA8j-SNSN.pcap*' > /tmp/expsip.pcap
 
  tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz' 'CwA8j-SNSN.pcap*' > /tmp/expsip.pcap
  
=== export RTP pcap ===
+
 
First we will need to get '''lzo positions''' from database (calldate start '2016-08-23 16:37:38'in example and from CALL-ID header 'CwA8j-SNSN' you can write a query), type=2 (means RTP filetype):
+
=== export RTP ===
 +
 
 +
Get RTP positions
 +
 
 
  mysql> SELECT pos FROM voipmonitor.cdr_tar_part where cdr_id = 103 and type = 2 and calldate = '2016-08-23 16:37:38';
 
  mysql> SELECT pos FROM voipmonitor.cdr_tar_part where cdr_id = 103 and type = 2 and calldate = '2016-08-23 16:37:38';
  
Line 38: Line 47:
 
  4 rows in set (0,00 sec)
 
  4 rows in set (0,00 sec)
  
Second we use positions returned from db to '''export RTP and unLZO''' using voipmonitor binary:
+
use positions returned from db and extract pcap
  /usr/local/sbin/voipmonitor -kc -d /var/spool/voipmonitor/ --untar-gui='/var/spool/voipmonitor//2016-08-23/16/37/RTP/rtp_2016-08-23-16-37.tar CwA8j-SNSN.pcap 0,164352,328704,493056 /tmp/exprtp.pcap
+
 
 +
  /usr/local/sbin/voipmonitor -kc -d /var/spool/voipmonitor/ --untar-gui='/var/spool/voipmonitor//2016-08-23/16/37/RTP/rtp_2016-08-23-16-37.tar CwA8j-SNSN.pcap 0,164352,328704,493056 rtp.pcap'
 +
#rtp.pcap is already decompressed (no unLZO needed)
 +
 
 +
==Alternative RTP extraction without knowing positions from database==
 +
this will consume more IO reads as tar file has to be fully scanned, additionally you need to have tar file closed or you need to ask sniffer to flush cashed data if the file is still opened.
 +
 
 +
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > rtp.pcap
 +
# if LZO compression for RTP pcaps is enabled '''
 +
voipmonitor -kc --unlzo-gui='/path/to/rtp.pcap /path/to/rtp-uncompressed.pcap'
 +
#if path to file is not absolute (/...) it is relative to the spooldir directory
 +
=== flush partially written data into tar file ===
 +
Before you ask tar binary for file extraction you need to flush data from sniffer first ( if the pcap file is still opened ) using sniffer's api like:
 +
echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1
  
 
=== merge SIP and RTP into one file ===
 
=== merge SIP and RTP into one file ===
mergecap -w /tmp/export.pcap /tmp/exportSIP.pcap /tmp/exportRTP.pcap
+
(apt-get install tshark | yum install wireshark)
 +
Note: the rtp.pcap needs to be de-lzoed first (lzo is default compression method used by sniffer service when rtp stored to spooldir)
  
 +
mergecap -w rtp.pcap sip.pcap final.pcap
  
  
== Export pcap file when LZO compression disabled for RTP in config ==
+
=Create the AUDIO using pcap file=
=== preconditions ===
+
after you get/have SIP+RTP in a single pcap file you can ask sniffer's binary for create audio file without need to have GUI installed/ db used (works only for g711 calls without the GUI)
call captured when sensor's compression settings changed from default voipmonitor.conf
+
==prepare voipmonitor-audio.conf==
  pcap_dump_zip_rtp = '''no'''
+
Change following options copy of the sniffer's config file - sniffer needs to be set to analyze traffic (not mirror senders):
  option tar_compress_sip = gzip
+
spooldir=/myAudio
 +
savesip=no
 +
savertp=no
 +
savertcp=no
 +
savegraph=no
 +
saveaudio=ogg
 +
#saveaudio=wav
 +
saveaudio_stereo = yes
 +
nocdr=yes
 +
  #for possible to create audio also from other then g711 codecs( g711 is supported with sniffer directly) you need to have valid license for GUI and enable following option with path to gui's keycheck file
 +
  #keycheck = /var/www/html/php/lib/keycheck.php
  
=== information needed to collect from CDR ===
+
==call sniffer's binary==
From picture in section above you will need:
+
Call the sniffer's binary with file.pcap as argument to create audio from it in spooldir defined above:
2.Date time of a call start
+
voipmonitor --config-file=/etc/voipmonitor-audio.conf -k -v1 -r /tmp/final.pcap
3.Call-ID
 
  
=== export SIP pcap ===
+
==Results location==
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/SIP/sip_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > ./exportSIP.pcap
+
location of created audio files (it's names is CALL-ID observed in INVITE packet) depends on option **spooldiroldschema**
  
=== export RTP pcap ===
+
===spooldiroldschema = no===
tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > ./exportRTP.pcap
+
By default, the audio file will be created in directory /${SPOOLDIR}/DATE/HOUR/MINUTE/AUDIO/ (based on timestamp of packets
 +
/myAudio/2020-07-31/19/52/AUDIO/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)
  
=== merge SIP and RTP into one file ===
+
===spooldiroldschema = yes===
  mergecap -w /tmp/export.pcap /tmp/exportSIP.pcap /tmp/exportRTP.pcap
+
the audio will be made in DATE dir (based on timestamp of packets
 +
  /myAudio/2020-07-31/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)

Revision as of 15:29, 10 August 2020

Notes

RTP format: With default voipmonitor.conf RTP pcap chunks are compressed by LZO which are tared and archived in directory in date-hourminute

option pcap_dump_zip_rtp = lzo

SIP format: With default voipmonitor.conf SIP compression uses gzip

option tar_compress_sip = gzip


Export pcap file with default config used

Get information about CDR from database

You will need:

1.cdr.id (103)
2.Date time of call start (2016-08-23 16:37:38)
3.Call-ID (CwA8j-SNSN)
4.Location of your spooldir (spooldir=X) 

example : Cdr detail for export pcap default.jpg

SQL Query:

SELECT cdr.calldate,cdr.caller,cdr.called,cdr.id as cdrID,cdr_next.fbasename as callID 
FROM cdr,cdr_next 
WHERE cdr.id=cdr_next.cdr_ID AND cdr.calldate >= '2017-02-01 00:00:00' AND cdr.calldate <= '2017-02-01 23:59:59' AND cdr.caller like '+222%';

You MUST use cdr.calldate condition otherwise database will be overloaded by searching in all partitions

flush data to tar file

Before you ask tar binary or voipmonitor for file extraction, you need to flush data from sniffer's cache first ( if the pcap file is still opened ) send api command to sniffer service

echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1

export SIP pcap

tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/16/37/SIP/sip_2016-08-23-16-37.tar.gz' 'CwA8j-SNSN.pcap*' > /tmp/expsip.pcap


export RTP

Get RTP positions

mysql> SELECT pos FROM voipmonitor.cdr_tar_part where cdr_id = 103 and type = 2 and calldate = '2016-08-23 16:37:38';

Returned:

pos: 0
pos: 164352
pos: 328704
pos: 493056
4 rows in set (0,00 sec)

use positions returned from db and extract pcap

/usr/local/sbin/voipmonitor -kc -d /var/spool/voipmonitor/ --untar-gui='/var/spool/voipmonitor//2016-08-23/16/37/RTP/rtp_2016-08-23-16-37.tar CwA8j-SNSN.pcap 0,164352,328704,493056 rtp.pcap'
#rtp.pcap is already decompressed (no unLZO needed) 

Alternative RTP extraction without knowing positions from database

this will consume more IO reads as tar file has to be fully scanned, additionally you need to have tar file closed or you need to ask sniffer to flush cashed data if the file is still opened.

tar --wildcards -xOf '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar' 'R3YqlN7pnY.pcap*' > rtp.pcap
# if LZO compression for RTP pcaps is enabled 
voipmonitor -kc --unlzo-gui='/path/to/rtp.pcap /path/to/rtp-uncompressed.pcap'
#if path to file is not absolute (/...) it is relative to the spooldir directory

flush partially written data into tar file

Before you ask tar binary for file extraction you need to flush data from sniffer first ( if the pcap file is still opened ) using sniffer's api like:

echo "flush_tar '/var/spool/voipmonitor/2016-08-23/15/27/RTP/rtp_2016-08-23-15-27.tar'" |nc 127.0.0.1

merge SIP and RTP into one file

(apt-get install tshark | yum install wireshark) Note: the rtp.pcap needs to be de-lzoed first (lzo is default compression method used by sniffer service when rtp stored to spooldir)

mergecap -w rtp.pcap sip.pcap final.pcap


Create the AUDIO using pcap file

after you get/have SIP+RTP in a single pcap file you can ask sniffer's binary for create audio file without need to have GUI installed/ db used (works only for g711 calls without the GUI)

prepare voipmonitor-audio.conf

Change following options copy of the sniffer's config file - sniffer needs to be set to analyze traffic (not mirror senders):

spooldir=/myAudio
savesip=no
savertp=no
savertcp=no
savegraph=no
saveaudio=ogg
#saveaudio=wav
saveaudio_stereo = yes
nocdr=yes
#for possible to create audio also from other then g711 codecs( g711 is supported with sniffer directly) you need to have valid license for GUI and enable following option with path to gui's keycheck file
#keycheck = /var/www/html/php/lib/keycheck.php

call sniffer's binary

Call the sniffer's binary with file.pcap as argument to create audio from it in spooldir defined above:

voipmonitor --config-file=/etc/voipmonitor-audio.conf -k -v1 -r /tmp/final.pcap

Results location

location of created audio files (it's names is CALL-ID observed in INVITE packet) depends on option **spooldiroldschema**

spooldiroldschema = no

By default, the audio file will be created in directory /${SPOOLDIR}/DATE/HOUR/MINUTE/AUDIO/ (based on timestamp of packets

/myAudio/2020-07-31/19/52/AUDIO/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)

spooldiroldschema = yes

the audio will be made in DATE dir (based on timestamp of packets

/myAudio/2020-07-31/sSCCXfB1Pa.ogg: Ogg data, Vorbis audio, stereo, 8000 Hz, ~36400 bps, created by: Xiph.Org libVorbis I (1.3.2)