Internal support hints: Difference between revisions

From VoIPmonitor.org
(Add IPFIX PCAP download behavior - RTP not included, SIP is reconstructed)
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:Administration]]
{{DISPLAYTITLE:Internal Support Hints}}


==Important note: ==
{{Warning|1=This documentation is for the internal support team. Use at your own risk and test on non-production servers first.}}
Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)


= Testing configure options =
= Quick Reference =
If you need to test configure options without need to modify the config file or database settings, you can use json_config option:
--json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
(The otpions defined using '''json_config''' command line option will overrides the configfile and database settings)


== Examples ==
<kroki lang="mermaid">
This will override options '''rtpip_find_endpoints,rrd,natalias''' no matter what is set in config file.
%%{init: {'flowchart': {'nodeSpacing': 10, 'rankSpacing': 25}}}%%
./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams -rpba:/totest/test.pcap --json_config='[{"rtpip_find_endpoints","yes"},{"rrd","no"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
flowchart LR
    subgraph Test["Testing"]
        A[json_config] --> B[Regexp/Country]
        B --> C[Billing]
    end
    subgraph Debug["Debugging"]
        D[Jemalloc] --> E[Threads]
        E --> F[Charts crash]
    end
    subgraph Tools["Tools"]
        G[PCAP replay] --> H[Manager API]
        H --> I[Scripts]
    end
</kroki>


= Testing regexp / country rules / billing =
= Testing Configuration =
== Regexp test ==
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|'


== Country assignment test ==
== Override Settings with json_config ==
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345


* gui version (obsolete)
Override config file and database settings without editing files:


root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999
<syntaxhighlight lang="bash">
number:           4909699999
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
country:         DE
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'
is_international: Y
</syntaxhighlight>


== pricing / billing test ==
{{Tip|Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.}}
voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing


=== Example of test_data/billing file===
== Regexp / Country / Billing Tests ==
Columns order
  - calldate
  - connect_duration
  - caller
  - called
  - sipcallerip
  - sipcalledip
  - expected price operator - (voluntary option)
  - expected price customer - (voluntary option)


2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
{| class="wikitable"
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
! Test Type !! Command
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151
|-
| Regexp pattern || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'</code>
|-
| Country by IP || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113</code>
|-
| Country by number || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345</code>
|-
| Billing test || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing</code>
|}


= License Hints =
'''Billing test file format''' (CSV):
== Write state to db (count of cc channels for license purposse) ==
<syntaxhighlight lang="text">
php /var/www/html/php/run.php saveCallStatistics
# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
</syntaxhighlight>


== keep the license updated without processing alerts reports ==
= License Management =
(put into /etc/crontab)
04 04  * * *  root php run.php runUpdateLicenseKey


== License Details ==
{| class="wikitable"
=== detail on channels count usage ===
! Task !! Command
php /var/www/html/php/run.php checkLicense -v
|-
Less details:
| Write channel stats to DB || <code>php /var/www/html/php/run.php saveCallStatistics</code>
php /var/www/html/php/run.php checkLicense -v|grep max_peak -A3 -B1
|-
| Check license details || <code>php /var/www/html/php/run.php checkLicense -v</code>
|-
| Check max peak only || <code>php /var/www/html/php/run.php checkLicense -v <nowiki>|</nowiki> grep max_peak -A3 -B1</code>
|-
| Get token from DB || <code>SELECT content FROM voipmonitor.system WHERE type='license token';</code>
|-
| View license file || <code>cat /var/www/html/key.php</code>
|}


=== license file ===
'''Auto-update license via cron''' (without processing alerts/reports):
cat /var/www/html/key.php


=== Get current license token from db ===
<syntaxhighlight lang="bash">
mysql> select content from voipmonitor.system where type='license token';
04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey
</syntaxhighlight>


= SCRIPTs for better run control =
= Utility Scripts =
== Alerts and Report ==
When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.


Follow this link [[script-for-multi-Instances-alerts-processing]] (deprecated)
{| class="wikitable"
! Script !! Purpose !! Details
|-
| Parallel alerts || Process alerts faster || Enable in '''Settings → System Configuration → Advanced → Number of parallel tasks'''
|-
| [[script-for-big-batchdownloads|Batch download]] || Download >1000 audio files || Bypasses GUI limit of 1000 CDRs
|-
| [[script-for-GUI-cron-control-oom|RAM control]] || Prevent OOM killer || Sets <code>oom_adj_score</code> for PHP threads
|-
| [[script-for-probes-backup|Probe backup]] || Backup all probe configs || Requires SSH access without password
|}


The GUI now supports parralel run of the alerts/report - enable it in system settings->Advanced->Number of parallel tasks
= Debugging =


== Batch download for more then 1000CDRs ==
== Charts Binary Crash ==
When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.


Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) [[script-for-big-batchdownloads]]
When dmesg or Apache log shows charts crash:
<syntaxhighlight lang="text">
charts-x86_64[7321]: segfault at 10 ip 0000000000412553...
</syntaxhighlight>


== RAM (alerts / dashboards cache) ==
'''Debug procedure:'''
When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)
<syntaxhighlight lang="bash">
# Create debug folder
mkdir /tmp/vm_charts_parameters


Follow link [[script-for-GUI-cron-control-oom]]
# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
</syntaxhighlight>


== Backup the settings from all probes and GUI ==
== Memory Leak Detection (Jemalloc) ==
The script gets voipmonitor.conf from all probes and do GUI backup config tables from db. Requires ssh access without password, error redirect to /dev/null, run it using cron


Follow link [[script-for-probes-backup]]
=== Compile Jemalloc ===


= charts binary crashes debug steps =
<syntaxhighlight lang="bash">
When apachelog or dmesg reports crashes of vm's charts binary like
# Download from https://github.com/jemalloc/jemalloc
Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]
./autogen.sh
== create debug folder for charts ==
./configure --enable-prof
Create vm_charts_parameters in /tmp
make && make install
</syntaxhighlight>


after GUI will create some charts there files with arguments will be created.
=== Compile VoIPmonitor with Jemalloc ===
== test Arguments passed to charts binary ==
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files


= voipmonitor sniffer memory leaks detect and debug =
Edit <code>config.h</code>:
When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)
<syntaxhighlight lang="c">
== compile jemalloc ==
#define HAVE_LIBTCMALLOC 0
Download jemalloc from https://github.com/jemalloc/jemalloc
#define HAVE_LIBJEMALLOC 1
./autogen.sh
</syntaxhighlight>
#requires development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools'
./configure --enable-prof
make
make install


=== enable jemalloc for voipmonitor and compile it ===
Edit <code>Makefile</code>:
./configure
<syntaxhighlight lang="makefile">
edit config.h,to have there
# Change -O2 to -O0 (disable optimization)
#define HAVE_LIBTCMALLOC 0
LIBS = ${SHARED_LIBS} -ljemalloc
#define HAVE_LIBJEMALLOC 1
</syntaxhighlight>
edit Makefile and disable optimization, add ljemalloc to libs, remove -ltcmalloc
substitute there -O2 with -O0 (It is capital o followed by zero)
LIBS = ${SHARED_LIBS} -ljemalloc


Finish compilation
<syntaxhighlight lang="bash">
make clean
make clean && make -j8
make -j8
</syntaxhighlight>


=== edit /etc/init.d/voipmonitor ===
=== Run with Jemalloc ===
Ensure
ARGS="-v 1"
and in start section (2places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS


Modify <code>/etc/init.d/voipmonitor</code> start section:
<syntaxhighlight lang="bash">
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
</syntaxhighlight>


== enable X88 compatible sources and recompile ==
=== Generate Memory Report ===
Copy sniffer sources with compiled binary to new location and recreate binary with -X88 (This is necessary to get correct memory allocation for debuging usin jeprof)
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
#It will list of source files modification,then re-make
make clean
make


More detailed instructions on debuging using X88 [[X88-memoryConsumptionDebug]]
<syntaxhighlight lang="bash">
# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript


== Modify init file or run sniffer manually with extra env arguments and parameters ==
# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf
</syntaxhighlight>


=== edit /etc/init.d/voipmonitor ===
=== X88 Mode (Detailed Memory Debug) ===
ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
#and in start section (2places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS


=== or run it from konsole / when sniffer service stopped ===
For more detailed memory allocation info:
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
<syntaxhighlight lang="bash">
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make
</syntaxhighlight>


== Check results ==
Run with extra verbose options:
in some interval you should check the results of memory usage with
<syntaxhighlight lang="bash">
=== je_prof.pdf ===
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
* don't forget to install graphviz and ghostscript
  --config-file=/etc/voipmonitor.conf \
yum install graphviz ghostscript
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf
</syntaxhighlight>


=== sniffer_memory.txt ===
See: [[X88-memoryConsumptionDebug]]
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029


=== script for periodical collecting ===
== Thread Debugging ==


#!/bin/bash
{| class="wikitable"
DATE=`date '+%Y%m%d_%H:%M:%S'`
! Method !! Usage
LOGDIR=/root/vm_debug/log/$DATE
|-
mkdir -p $LOGDIR
| Log thread CPU || Add <code>-v1,threads_cpu_log</code> to command line
pdftmpname=/tmp/je_prof
|-
pdfname=$LOGDIR/je_prof.pdf
| List threads || <code>echo 'sniffer_threads' | nc 127.0.0.1 5029</code>
txtname=$LOGDIR/mem_stat.txt
|-
senname=$LOGDIR/senlog.txt
| Extended info || Set <code>sniffer_threads_ext=yes</code> in config (requires compile with <code>SNIFFER_THREADS_EXT true</code>)
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>&1
|}
/usr/local/bin/jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>&1
echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>&1
echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>&1
rm $pdftmpname


=== script for periodical mail sending ===
'''sniffer_threads options:'''
* <code>no_sort</code> - don't sort by load
* <code>only_traffic</code> - only threads with traffic
* <code>all</code> - include unloaded threads
* <code>line</code> - simple line output


* add it to cron.hourly
== SS7 Module Compilation ==


#!/bin/bash
Requires Wireshark 3.2.
mail="YOUR@MAIL.COM"
sniffer_bin="/usr/local/sbin/voipmonitor"
manager_ip="127.0.0.1"
manager_port="5029"
tmp_je_prof_pdf="/tmp/je_prof.pdf"
tmp_je_prof_out="/tmp/je_prof"
tmp_pb_log="/tmp/pb_log"
tmp_sn_stat="/tmp/sn_stat"
echo 'jemalloc_stat_full' | nc $manager_ip $manager_port > $tmp_je_prof_out; jeprof --show_bytes --pdf $sniffer_bin $tmp_je_prof_out > $tmp_je_prof_pdf 2>/dev/null
echo 'packetbuffer_log' | nc $manager_ip $manager_port > $tmp_pb_log
echo 'sniffer_stats' | nc $manager_ip $manager_port > $tmp_sn_stat
#old mail:
# echo 'jeprof' | mail -s 'jeprof' -A $tmp_je_prof_pdf -A $tmp_pb_log -A $tmp_sn_stat $mail
#new mail n-nails:
echo 'jeprof' | mail -s 'jeprof' -a $tmp_je_prof_pdf -a $tmp_pb_log -a $tmp_sn_stat $mail
rm $tmp_je_prof_out
rm $tmp_je_prof_pdf
rm $tmp_pb_log
rm $tmp_sn_stat


= info about internal threads =
Edit <code>config.h</code>:
<syntaxhighlight lang="c">
#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200
</syntaxhighlight>


== more info about threads in the log ==
Edit <code>Makefile</code>:
<syntaxhighlight lang="makefile">
INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0
</syntaxhighlight>


Add verbose option threads_cpu_log into command line (-v1,threads_cpu_log). If enabled then it prints the load of the threads to the syslog in the line output.
<syntaxhighlight lang="bash">
./configure && make clean && make ss7
</syntaxhighlight>


== basic usage of manager command 'sniffer_threads' ==
= PCAP Testing =


options are now:
== Basic Methods ==


no_sort - do not sort by load in descending order
{| class="wikitable"
only_traffic - lists only threads with recorded traffic (requires sniffer_threads_ext)
! Method !! Command
all - lists all threads (even unloaded ones)
|-
line - use simple line output
| Basic upload || <code>voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap</code>
|-
| Via packetbuffer (fast) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap</code>
|-
| Via packetbuffer (original time) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap</code>
|}


== debugging output of enhanced manager command 'sniffer_threads' ==
== IPFIX Testing ==


now includes:
'''Receiver instance:'''
- traffic passing through each thread
<syntaxhighlight lang="ini">
- the percentage of states where the buffer between threads is full when a packet is attempted to be added to the buffer
ipfix = yes
- the percentage of usleep times when waiting for buffer space to become available, out of the total time
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345
</syntaxhighlight>


required steps:
'''Sender instance:'''
- compile the sniffer with the define option #define SNIFFER_THREADS_EXT true (in voipmonitor_define.h)
<syntaxhighlight lang="bash">
- set sniffer_threads_ext=yes in the configuration, or execute echo 'setverbparam sniffer_threads_ext' | nc ...
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port
</syntaxhighlight>
=== IPFIX PCAP Download Behavior ===


output example:
'''Key fact:''' IPFIX data is internally converted to packet format for processing. This means:
* '''PCAP files CAN be downloaded''' from the GUI for IPFIX-sourced calls
* SIP signaling is reconstructed from the IPFIX data
* '''RTP streams are NOT included''' in the PCAP - only QoS metrics from IPFIX data are available, not actual audio packets


root@test:~# echo "sniffer_threads" | nc 127.0.0.1 5029
When downloading PCAP from IPFIX calls with TLS, TCP sequence numbers may be set to 0 causing Wireshark to display incorrectly. Disable TCP sequence analysis in Wireshark preferences to view all packets.
t2 sip preprocess detach                      (  21939) :  0.0                    1 /    72837 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  1.39%    0.00%
== Tcpreplay ==
t2 sip preprocess sip                        (  21940) :  0.0                    1 /    69402 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  0.01%    0.00%
t2 sip preprocess rtp                        (  21947) :  0.0                    2 /    69922 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  0.01%    0.00%
t2 rtp preprocess hash                        (  21952) :  0.0                  10 /    67685 /  0.000                521p 8.54Mb ->  --- p  --- b  bf  0.00%    0.00%
t2 out thread defrag                          (  21954) :  0.0                    1 /    77085 /  0.000                823p 8.80Mb ->    815p 8.80Mb  bf  5.88%    0.00%
pb - main queue                              (  21955) :  0.0                    0 /    26550 /  0.000                954p 10.2Mb ->    954p 10.2Mb                   
pb - write queue                              (  21956) :  0.0                    0 /    23481 /  0.000                954p 10.2Mb ->    878p 9.32Mb                   
pb - read thread eth1 read                    (  21957) :  0.0                  123 /      378 /  0.325                954p 10.2Mb ->    954p 10.2Mb                   
pb - read thread eth1 pcap_process            (  21960) :  0.0                    0 /    57547 /  0.000                954p 10.2Mb ->    954p 10.2Mb                   
CPU ALL :  0.0


= voipmonitor sniffer compile notes ss7 (wireshark) module =
Enable in <code>/etc/init.d/voipmonitor</code>:
(requires ws version 3.2):
<syntaxhighlight lang="bash">
ARGS="-v 1,dump_call_flags,tcpreplay"
</syntaxhighlight>


== modify config.h ==
'''Add Layer 2 if missing''' (for captures done on "any" interface):
#define HAVE_LIBWIRESHARK 1
<syntaxhighlight lang="bash">
#define LIBWIRESHARK_VERSION 30200
tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap
</syntaxhighlight>


== modify Makefile's INCLUDE ==
'''Rewrite IP addresses:'''
-I/usr/include/wireshark
<syntaxhighlight lang="bash">
== modify Makefile's SHARED_LIBS==
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap
-lwiretap -lwireshark -lwsutil
</syntaxhighlight>


internal build in JIRA:/opt/debian-7-64bit
== SSL/TLS Testing ==
== disable optimize in Makefile==
change -O2 to -O0


== compile ==
<syntaxhighlight lang="bash">
./configure && make clean && make ss7
voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'
</syntaxhighlight>


= testing customer pcaps =
For TLS details, see [[Tls]].
== testing special pcaps ==
=== ipfix ===
Run first instance of voipmonitor that will act as receiver for packets and will push CDRs to db
in the settings enable ipfix options:
ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345


Then run scecond instance that will act as source for replay the ipfix encapsulated pcap
= Manager API =
./voipmonitor --config-file=config/voipmonitor.conf -k -v1 --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
Params:
pcap file
2 IP address - 1.ip fix client,2.ip fix server
last IP : port telling to where to send the ipfix stream (IP:port of ipfix options set for the 1.st instance)


'''For complete reference, see [[Manager_API]].'''


This section covers quick debug commands. For encryption setup and full command list, see the main article.


== testing via basic upload aka GUI upload ==
== Via Manager Port (5029) ==
voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap
== testing via packetbuffer upload ==
=== time shift to now, increased speed of replaying ===
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap
=== no time shift via pb===
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap
=== tcpreplay to running service ===
you need to enable additional argument when starting the service in /etc/init.d/voipmonitor
ARGS="-v 1,dump_call_flags,tcpreplay"
then use tcpreplay
tcpreplay -i eth0 /tmp/l/x.pcap
when error appears like
sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)
probably captured on any interface and layer2 missing, here follow stpes on how to add it:


==== adding layer2 to pcap ====
<syntaxhighlight lang="bash">
after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay)
# These require encryption disabled OR use socket file
tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
echo 'listcalls' | nc 127.0.0.1 5029
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap
echo 'sniffer_stat' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
echo 'sniffer_threads' | nc 127.0.0.1 5029
</syntaxhighlight>


==== rewrite IPv4 address in whole pcap ====
{{Tip|For encrypted communication, use the GUI's <code>run.php send_manager_cmd</code> or Unix socket. See [[Manager_API#Usage Examples]].}}
when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses:
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap


= Rename mysql database =
== Via Server Port (60024) ==
It's not possible to do it via direct command. You need to use '''RENAME TABLE''' command.
With this SELECT you generate the commands for the rename of the all tables in the database. (Replace OLD_DB_NAME and NEW_DB_NAME)


SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
Used in distributed deployments for probe management:
    ' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';


The output:
<syntaxhighlight lang="bash">
# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024


RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
# Send command to specific probe
RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024
RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;
</syntaxhighlight>


Run these commands a you are ready.
= GUI Workarounds =


'''Attention! Sometimes you can have a problem with moving of the foreign keys.'''
== Disable Sensor Logs Popup ==


= disabling TCP sack =
{{Note|There is no built-in GUI option to disable the sensor logs popup shown at admin login.}}
On buggy kernels (all before 2019/03) following prevents SACK issue
sideback is that on lossy lines (packet lost or big delays) the tcp traffic will increase)
echo "0" > /proc/sys/net/ipv4/tcp_sack


= MacOS high sierra in virtual box (on windows) =
'''Option 1: Periodic SQL update'''
== Download Image ==
<syntaxhighlight lang="sql">
from - https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view
UPDATE voipmonitor.log_sensor SET confirmed = 1;
== Prepare ISO for install ==
</syntaxhighlight>
===Create High Sierra DMG Disk Image===
hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J


=== Mount DMG Image to your macOS ===
'''Option 2: Database trigger (permanent)'''
hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra
<syntaxhighlight lang="sql">
DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;
</syntaxhighlight>


=== Create a High Sierra ISO image ===
== IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4) ==
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra


=== Unmount Disk Image ===
'''Symptom:''' ldap.so crash at login page (CVE-2024-8932).
hdiutil detach /volumes/Install\ macOS\ High\ Sierra


=== Convert DMG to ISO ===
'''Fix:'''
hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra
<syntaxhighlight lang="bash">
# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP


=== Rename and Move ISO Image to Desktop ===
# Download and extract fixed package
mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb


== Create new virtual Image ==
# Install and restart
during preconfigure of new image set:
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
memmory 3-6GB
systemctl restart apache2
disk size 100GB
</syntaxhighlight>
then edit it's settings and additionally modify:
system -> Motherboard -> Chipset: PIIX3
system -> Processor -> count (2), Enable PAE/NX
system -> Storage -> Attributes-> Optical drive->Sata port 1, for the newly created iso
display -> Video memory -> 128MB
display -> Graphics Controler -> VMSVGA


== Add the VirtualBox macOS Code to CMD ==
= Database Operations =
cd "C:\Program Files\Oracle\VirtualBox\"
VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1


== Install macOS on virtulBox ==
== Rename MySQL Database ==
#Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
#Format VirtualBox Virtual Disk with "Mac OS Extended". (GUID:Partition map)
#Now, you can install macOS High Sierra on new virtual disk with your ISO Image.
#After your installation is completed, turn off your macOS High Sierra. Open VirtualBox, and remove High Sierra.iso file.
#Turn on your Virtual Machine. At UEFI Shell screen, boot with this commands:


>"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"
Generate RENAME TABLE commands:
<syntaxhighlight lang="sql">
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';
</syntaxhighlight>


{{Warning|1=Foreign keys may cause issues during rename.}}


== Start the High Sierra Virtual machine and Enjoy ==
== Spooldir Testing ==


= Measurement =
<syntaxhighlight lang="bash">
== Memory speed ==
voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all
</syntaxhighlight>


sysbench memory --threads=2 run
= Remote Support Access =


* should be over 5000 MiB/sec
== VPN Connection Issues ==


= SSL and keylogger =
When support staff cannot connect to client VPN (Fortinet, etc.):
== How to test it in voipmonitor ==
./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line -rpba:/tmp/keys.pcap@@/tmp/call.pcap --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'


== How to test it in wireshark/tshark ==
# '''Cause:''' Support IP not whitelisted on client firewall
# '''Solution:''' Client must add support staff's public IP to their VPN/firewall whitelist
# '''Note:''' This is client-side network administration, not a VoIPmonitor issue


=== Get the keys from keyloger's dump ===
{{Note|1=Don't confuse VPN network access with FortiToken 2FA (GUI authentication).}}
ngrep -I /tmp/keycapture.pcap |grep '^ ' |cut -d ' ' -f3- > /tmp/keys.txt


=== Decrypt with tshark ===
= System Troubleshooting =
tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'


=== Decrypt in wireshark ===
== TCP SACK (Buggy Kernels pre-2019/03) ==
In edit->preferences->protocol->TLS->pre-master secret log filename and choose the keys.txt
then find the client Hello HS packet, right click on it and follow stream - TLS


=== Check if we have the right keys for RTP/DTLS ===
<syntaxhighlight lang="bash">
In rtp you have to find dtls packets (dtls.record.content_type == 22), select 'client random' in them and find the 'Random' item in it.
echo "0" > /proc/sys/net/ipv4/tcp_sack
</syntaxhighlight>


That's e.g. e54fc0f917cdd5c0d914f24956cdfd689022709f59ca4cd7fc20296db17039d8 in our sample. And this must be included in the captured keys.
== Memory Speed Test ==


= Spooldir testing =
<syntaxhighlight lang="bash">
When there are problems with cleaning the spooldir or spooldir size not respects the settings you can use following command to test
sysbench memory --threads=2 run
voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all
# Should be >5000 MiB/sec
</syntaxhighlight>


= Manager API via server port =
= Additional Resources =
== get list of the connected clients ==
echo '{"type_connection":"manager_command","command":"active"}'|nc 127.0.0.1 60024
== send terminating manager command to a probe ==
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}'|nc 127.0.0.1 60024


= HA proxy =
{| class="wikitable"
[[sample HAproxy configuration]]
! Topic !! Link
|-
| HAProxy config || [[sample HAproxy configuration]]
|-
| Hugepages for high CPS || [[voipmonitor and hugepages]]
|-
| Manager API reference || [[Manager_API]]
|-
| Syslog status line || [[Syslog_Status_Line]]
|-
| Ribbon SBC 7k TLS || [[Ribbon7k monitoring profiles]]
|-
| Support email templates || [[common body of support emails]]
|-
| IPACCOUNT feature || [[ipaccount]]
|-
| DTLS decryption || [[DTLS]]
|-
| Rocky 9 installation || [[Rocky 9]]
|}


== High CPS config for voipmonitor with hugepages ==
'''IOPS testing:'''
[[voipmonitor and hugepages]]
<syntaxhighlight lang="bash">
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat
</syntaxhighlight>


= IOPS tests =
--rw can be randread,randwrite,randrw
  fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap --filename=/root/testW.dat --output=/root/fio_db_write.txt


= Rocky 9=
[[Rocky 9]]


= Manager api of a sniffer =
= AI Summary for RAG =
[[encryption_in_manager_api]]


= Manager api of a sniffer for customers =
'''Summary:''' Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with <code>--json_config</code> to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (<code>checkLicense</code>, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via <code>sniffer_threads</code> command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).
[[encryption_in_manager_api_customer]]


= TLS decode by Ribbon SBC 7k=
'''Keywords:''' internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed
[[Ribbon7k monitoring profiles]]


= Email templates =
'''Key Questions:'''
common requests in supports [[common body of support emails]]
* How to test configuration without modifying voipmonitor.conf?
 
* How to test regexp patterns or country assignment?
= IPACCOUNT =
* How to check license channel usage?
How to enable [[ipaccount]] feature in the voipmonitor
* How to debug memory leaks using jemalloc?
 
* How to compile voipmonitor with jemalloc or SS7 support?
= DTLS =
* How to generate jeprof PDF memory report?
[[DTLS]] SRTP decryption using voipmonitor
* How to monitor thread load with sniffer_threads?
 
* How to test IPFIX or replay PCAPs?
=ioncube is crashing for apache/https=
* How to add layer 2 to PCAP for tcpreplay?
 
* How to disable sensor logs popup in GUI?
== wrong privileges ==
* How to fix IonCube ldap.so crash in Ubuntu 22.04?
If following command is crashig under www-data used but not under root user:
* How to resolve VPN connection issues for remote support?
su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data
* How to disable TCP SACK on buggy kernels?
 
* How to test memory speed with sysbench?
it can be caused by wrong mod for the ioncube.ini file or ioncube's module, you can test with strace (search for ioncube or denied)
strace -f  su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data 2> /tmp/text.txt
 
== crashes in libc ==
It was experieced in ubuntu 22.04 with php 7.4 that ioncube causes crash in ldap.so (displays following error on the login page)
Failed check Ioncube.com PHP Loader for php cli : PHP Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/lib/php/20190902/ldap.so (/usr/lib/php/20190902/ldap.so: undefined symbol: RETURN_THROWS), /usr/lib/php/20190902/ldap.so.so (/usr/lib/php/20190902/ldap.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
The reason is new ldap library broken in ubuntu 22.04
 
https://ubuntu.com/security/CVE-2024-8932
 
https://askubuntu.com/questions/1535405/php7-4-ldap-broken-on-focal-after-13-dec-2024
 
Backup current ldap libs
cd /usr/lib/php/20190902
mkdir BKUP
cp -pv ldap.so* BKUP
 
Download and extract the package.
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
 
Copy in the fixed lib from local tmp/ to prod.
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
'./usr/lib/php/20190902/ldap.so' -> '/usr/lib/php/20190902/ldap.so'
 
Restart apache and test.
systemctl restart apache2
 
''thanks to Andrew from distributel.ca for this article''

Latest revision as of 23:55, 9 January 2026


⚠️ Warning: This documentation is for the internal support team. Use at your own risk and test on non-production servers first.

Quick Reference

Testing Configuration

Override Settings with json_config

Override config file and database settings without editing files: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'

💡 Tip: Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.

Regexp / Country / Billing Tests

Test Type Command
Regexp pattern voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'
Country by IP voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113
Country by number voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345
Billing test voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing

Billing test file format (CSV): 

# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8

License Management

Task Command
Write channel stats to DB php /var/www/html/php/run.php saveCallStatistics
Check license details php /var/www/html/php/run.php checkLicense -v
Check max peak only php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
Get token from DB SELECT content FROM voipmonitor.system WHERE type='license token';
View license file cat /var/www/html/key.php

Auto-update license via cron (without processing alerts/reports): 

04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey

Utility Scripts

Script Purpose Details
Parallel alerts Process alerts faster Enable in Settings → System Configuration → Advanced → Number of parallel tasks
Batch download Download >1000 audio files Bypasses GUI limit of 1000 CDRs
RAM control Prevent OOM killer Sets oom_adj_score for PHP threads
Probe backup Backup all probe configs Requires SSH access without password

Debugging

Charts Binary Crash

When dmesg or Apache log shows charts crash: 

charts-x86_64[7321]: segfault at 10 ip 0000000000412553...

Debug procedure: 

# Create debug folder
mkdir /tmp/vm_charts_parameters

# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

Memory Leak Detection (Jemalloc)

Compile Jemalloc

# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./configure --enable-prof
make && make install

Compile VoIPmonitor with Jemalloc

Edit config.h: 

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

Edit Makefile: 

# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc

make clean && make -j8

Run with Jemalloc

Modify /etc/init.d/voipmonitor start section: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

Generate Memory Report

# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript

# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf

X88 Mode (Detailed Memory Debug)

For more detailed memory allocation info: 

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make

Run with extra verbose options: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
  --config-file=/etc/voipmonitor.conf \
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

See: X88-memoryConsumptionDebug

Thread Debugging

Method Usage
Log thread CPU Add -v1,threads_cpu_log to command line
List threads nc 127.0.0.1 5029
Extended info Set sniffer_threads_ext=yes in config (requires compile with SNIFFER_THREADS_EXT true)

sniffer_threads options:

  • no_sort - don't sort by load
  • only_traffic - only threads with traffic
  • all - include unloaded threads
  • line - simple line output

SS7 Module Compilation

Requires Wireshark 3.2.

Edit config.h: 

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

Edit Makefile: 

INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0

./configure && make clean && make ss7

PCAP Testing

Basic Methods

Method Command
Basic upload voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap
Via packetbuffer (fast) voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap
Via packetbuffer (original time) voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap

IPFIX Testing

Receiver instance: 

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Sender instance: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port

IPFIX PCAP Download Behavior

Key fact: IPFIX data is internally converted to packet format for processing. This means:

  • PCAP files CAN be downloaded from the GUI for IPFIX-sourced calls
  • SIP signaling is reconstructed from the IPFIX data
  • RTP streams are NOT included in the PCAP - only QoS metrics from IPFIX data are available, not actual audio packets

When downloading PCAP from IPFIX calls with TLS, TCP sequence numbers may be set to 0 causing Wireshark to display incorrectly. Disable TCP sequence analysis in Wireshark preferences to view all packets.

Tcpreplay

Enable in /etc/init.d/voipmonitor: 

ARGS="-v 1,dump_call_flags,tcpreplay"

Add Layer 2 if missing (for captures done on "any" interface): 

tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap

Rewrite IP addresses: 

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap

SSL/TLS Testing

voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'

For TLS details, see Tls.

Manager API

For complete reference, see Manager_API.

This section covers quick debug commands. For encryption setup and full command list, see the main article.

Via Manager Port (5029)

# These require encryption disabled OR use socket file
echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stat' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
echo 'sniffer_threads' | nc 127.0.0.1 5029

💡 Tip: For encrypted communication, use the GUI's run.php send_manager_cmd or Unix socket. See Manager_API#Usage Examples.

Via Server Port (60024)

Used in distributed deployments for probe management: 

# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024

# Send command to specific probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024

GUI Workarounds

Disable Sensor Logs Popup

ℹ️ Note: There is no built-in GUI option to disable the sensor logs popup shown at admin login.

Option 1: Periodic SQL update 

UPDATE voipmonitor.log_sensor SET confirmed = 1;

Option 2: Database trigger (permanent) 

DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;

IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4)

Symptom: ldap.so crash at login page (CVE-2024-8932).

Fix: 

# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP

# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

# Install and restart
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2

Database Operations

Rename MySQL Database

Generate RENAME TABLE commands: 

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';

⚠️ Warning: Foreign keys may cause issues during rename.

Spooldir Testing

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Remote Support Access

VPN Connection Issues

When support staff cannot connect to client VPN (Fortinet, etc.):

  1. Cause: Support IP not whitelisted on client firewall
  2. Solution: Client must add support staff's public IP to their VPN/firewall whitelist
  3. Note: This is client-side network administration, not a VoIPmonitor issue

ℹ️ Note: Don't confuse VPN network access with FortiToken 2FA (GUI authentication).

System Troubleshooting

TCP SACK (Buggy Kernels pre-2019/03)

echo "0" > /proc/sys/net/ipv4/tcp_sack

Memory Speed Test

sysbench memory --threads=2 run
# Should be >5000 MiB/sec

Additional Resources

Topic Link
HAProxy config sample HAproxy configuration
Hugepages for high CPS voipmonitor and hugepages
Manager API reference Manager_API
Syslog status line Syslog_Status_Line
Ribbon SBC 7k TLS Ribbon7k monitoring profiles
Support email templates common body of support emails
IPACCOUNT feature ipaccount
DTLS decryption DTLS
Rocky 9 installation Rocky 9

IOPS testing: 

fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat


AI Summary for RAG

Summary: Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with --json_config to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (checkLicense, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via sniffer_threads command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).

Keywords: internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed

Key Questions:

  • How to test configuration without modifying voipmonitor.conf?
  • How to test regexp patterns or country assignment?
  • How to check license channel usage?
  • How to debug memory leaks using jemalloc?
  • How to compile voipmonitor with jemalloc or SS7 support?
  • How to generate jeprof PDF memory report?
  • How to monitor thread load with sniffer_threads?
  • How to test IPFIX or replay PCAPs?
  • How to add layer 2 to PCAP for tcpreplay?
  • How to disable sensor logs popup in GUI?
  • How to fix IonCube ldap.so crash in Ubuntu 22.04?
  • How to resolve VPN connection issues for remote support?
  • How to disable TCP SACK on buggy kernels?
  • How to test memory speed with sysbench?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Internal_support_hints&oldid=10474"