Internal support hints

From VoIPmonitor.org
Jump to navigation Jump to search

Important note:

Following articles are designed to be used by internal support team. You can find hints usauseful but please use it at your own risk (use on testing servers first)

Testing regexp / country rules / billing

Regexp test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|'

Country assignment test

/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345

pricing / billing test

voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing

Example of test_data/billing file

Columns order
 - calldate 
 - connect_duration
 - caller
 - called
 - sipcallerip
 - sipcalledip
 - expected price operator - (voluntary option)
 - expected price customer - (voluntary option) 

2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151

License Hints

Write state to db (count of cc cchannels for license purposse)

php /var/www/html/php/run.php saveCallStatistics

License Details

detail on channels count usage

php /var/www/html/php/run.php checkLicense -v
php /var/www/html/php/run.php checkLicense -v 2

license file

cat /var/www/html/key.php

Get current license token from db

mysql> select content from voipmonitor.system where type='license token';

SCRIPTs for better run control

ALERTS

When alerts taking longer time to process then interval is schedulled it can cause out of memory or high load to CPU after some time because same alert is processed multipletimes for various time ranges. Following script allows manual run of disabled alert in GUI via crontab - its benefit is that it will run multiple alerts at same time unlike normal run where is processed one alert by one.

Follow this link script-for-multi-Instances-alerts-processing

Batch download for more then 1000CDRs

When you need to download lot of audio files - you use GUI->menu->cdr batch download,but it allows only 1000calls to process with one GUI's query.

Following script allows you to create audio from longer set of CDRs, also it process in multiple instances to get audio faster (beware of HDD IO/CPU) script-for-big-batchdownloads

RAM (alerts / dashboards cache)

When there is lot of CDRs then alerts or caching of dashboard data can take lot of RAM, folloowing script will control if some php thread will not consume over limit, when it does, its oom_adj_score is set to be killed first (prevent killing of voipmonitor sniffer od db when running on same server)

Follow link script-for-GUI-cron-control-oom

voipmonitor sniffer memory leaks detect and debug

When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more CALLs (regs/messages) following steps can tell you where the RAM ends (it can be leak or some thread overload)

compile je_malloc

Download jemalloc from https://github.com/jemalloc/jemalloc 

./autogen.sh (requieres development tools Deb/ubu apt install build-essential, centos/rh yum groupinstall 'Development Tools')
./configure --enable-prof
make
make install

enable jemalloc for voipmonitor and compile it

./configure

edit config.h,to have there 

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

edit Makefile and disable optimization 

substitute there -O2 with -O0 (It is capital o followed by zero)

Finish compilation 

make clean
make -j8

enable X88 compatible sources and recompile

Copy sniffer sources with compiled binary to new location and recreate binary with -X88 

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
./voipmonitor -k -v1 -c -X88
#It will list of source files modification,then re-make
make clean
make

Modify init file or run sniffer manually with extra env arguments and parameters

edit /etc/init.d/voipmonitor

ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
#and in start section (2places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

or run it from konsole / when sniffer service stopped

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file=/etc/voipmonitor.conf -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

Check results

in some interval you should check the results of memory usage with

je_prof.pdf

yum install graphviz ghostscript
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof; jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor /tmp/je_prof > je_prof.pdf

sniffer_memory.txt

echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

script for periodical collecting

#!/bin/bash
DATE=`date '+%d_%H:%M:%S'`
LOGDIR=/root/vm_debug/log/$DATE
mkdir -p $LOGDIR
pdftmpname=/tmp/je_prof
pdfname=$LOGDIR/je_prof.pdf
txtname=$LOGDIR/mem_stat.txt
logname=$LOGDIR/runlog.txt
senname=$LOGDIR/senlog.txt
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > $pdftmpname 2>$logname
jeprof --show_bytes --pdf /usr/src/sniffer-git-jemalloc-X88/voipmonitor $pdftmpname > $pdfname 2>$logname
echo memory_stat | nc 127.0.0.1 5029 > $txtname 2>$logname
echo sniffer_stats | nc 127.0.0.1 5029 > $senname 2>$logname
rm $pdftmpname

testing customer pcaps

testing via basic upload aka GUI upload

voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap

testing via packetbuffer upload

time shift to now, increased speed of replaying

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap

no time shift via pb

voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap

tcpreplay to running service

you need to enable additional argument when starting the service in /etc/init.d/voipmonitor 

ARGS="-v 1,dump_call_flags,tcpreplay"

then use tcpreplay 

tcpreplay -i eth0 /tmp/l/x.pcap

when error appears like 

sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)

probably captured on any interface and layer2 missing, here follow stpes on how to add it:

adding layer2 to pcap

after capture was done on any interface and you need to replay it to hw interface you need to add layer2 (at least in 3.4.3 v.tcpreplay) 

tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap

rewrite IPv4 address in whole pcap

when you need to test alert based on IP adresses (also country continent alert based on IPs) you can find usefull to change IP address in pcap and use it multiple times with various addresses: 

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap
Retrieved from "https://www.voipmonitor.org//doc/index.php?title=Internal_support_hints&oldid=2656"