Internal support hints: Difference between revisions

From VoIPmonitor.org
(Add remote VPN access procedures (IP whitelisting) to address ticket #82990)
(Add IPFIX PCAP download behavior - RTP not included, SIP is reconstructed)
 
(10 intermediate revisions by 2 users not shown)
Line 2: Line 2:
{{DISPLAYTITLE:Internal Support Hints}}
{{DISPLAYTITLE:Internal Support Hints}}


{{Warning|1=Following articles are designed to be used by internal support team. You can find hints useful but please use it at your own risk (use on testing servers first).}}
{{Warning|1=This documentation is for the internal support team. Use at your own risk and test on non-production servers first.}}


== Testing Configuration Options ==
= Quick Reference =


If you need to test configuration options without modifying the config file or database settings, you can use the <code>json_config</code> option:
<kroki lang="mermaid">
%%{init: {'flowchart': {'nodeSpacing': 10, 'rankSpacing': 25}}}%%
flowchart LR
    subgraph Test["Testing"]
        A[json_config] --> B[Regexp/Country]
        B --> C[Billing]
    end
    subgraph Debug["Debugging"]
        D[Jemalloc] --> E[Threads]
        E --> F[Charts crash]
    end
    subgraph Tools["Tools"]
        G[PCAP replay] --> H[Manager API]
        H --> I[Scripts]
    end
</kroki>


<syntaxhighlight lang="bash">
= Testing Configuration =
--json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
</syntaxhighlight>
 
The options defined using '''json_config''' command line option will override the config file and database settings.
 
=== Examples ===
 
This will override options '''rtpip_find_endpoints''', '''rrd''', '''natalias''' no matter what is set in config file:
 
<syntaxhighlight lang="bash">
./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams \
  -rpba:/totest/test.pcap \
  --json_config='[{"rtpip_find_endpoints","yes"},{"rrd","no"},{"natalias","8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
</syntaxhighlight>
 
== Testing Regexp / Country Rules / Billing ==


=== Regexp Test ===
== Override Settings with json_config ==


<syntaxhighlight lang="bash">
Override config file and database settings without editing files:
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|'
</syntaxhighlight>
 
=== Country Assignment Test ===


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Find country by IP
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'
 
# Find country by phone number
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345
</syntaxhighlight>
</syntaxhighlight>


GUI version (obsolete):
{{Tip|Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.}}


<syntaxhighlight lang="bash">
== Regexp / Country / Billing Tests ==
root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999
number:          4909699999
country:          DE
is_international: Y
</syntaxhighlight>
 
=== Pricing / Billing Test ===


<syntaxhighlight lang="bash">
{| class="wikitable"
voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing
! Test Type !! Command
</syntaxhighlight>
|-
 
| Regexp pattern || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'</code>
==== Example of test_data/billing File ====
|-
 
| Country by IP || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113</code>
Column order:
|-
* calldate
| Country by number || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345</code>
* connect_duration
|-
* caller
| Billing test || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing</code>
* called
|}
* sipcallerip
* sipcalledip
* expected price operator (optional)
* expected price customer (optional)


'''Billing test file format''' (CSV):
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151
</syntaxhighlight>
</syntaxhighlight>


== License Hints ==
= License Management =


=== Write State to DB (count of concurrent channels for license purpose) ===
{| class="wikitable"
! Task !! Command
|-
| Write channel stats to DB || <code>php /var/www/html/php/run.php saveCallStatistics</code>
|-
| Check license details || <code>php /var/www/html/php/run.php checkLicense -v</code>
|-
| Check max peak only || <code>php /var/www/html/php/run.php checkLicense -v <nowiki>|</nowiki> grep max_peak -A3 -B1</code>
|-
| Get token from DB || <code>SELECT content FROM voipmonitor.system WHERE type='license token';</code>
|-
| View license file || <code>cat /var/www/html/key.php</code>
|}


<syntaxhighlight lang="bash">
'''Auto-update license via cron''' (without processing alerts/reports):
php /var/www/html/php/run.php saveCallStatistics
</syntaxhighlight>
 
=== Keep License Updated Without Processing Alerts/Reports ===
 
Add to <code>/etc/crontab</code>:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
04 04   * * *   root php /var/www/html/php/run.php runUpdateLicenseKey
04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey
</syntaxhighlight>
</syntaxhighlight>


=== License Details ===
= Utility Scripts =


==== Detail on Channels Count Usage ====
{| class="wikitable"
! Script !! Purpose !! Details
|-
| Parallel alerts || Process alerts faster || Enable in '''Settings → System Configuration → Advanced → Number of parallel tasks'''
|-
| [[script-for-big-batchdownloads|Batch download]] || Download >1000 audio files || Bypasses GUI limit of 1000 CDRs
|-
| [[script-for-GUI-cron-control-oom|RAM control]] || Prevent OOM killer || Sets <code>oom_adj_score</code> for PHP threads
|-
| [[script-for-probes-backup|Probe backup]] || Backup all probe configs || Requires SSH access without password
|}


<syntaxhighlight lang="bash">
= Debugging =
php /var/www/html/php/run.php checkLicense -v
</syntaxhighlight>


Less details:
== Charts Binary Crash ==
 
<syntaxhighlight lang="bash">
php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
</syntaxhighlight>
 
==== License File ====
 
<syntaxhighlight lang="bash">
cat /var/www/html/key.php
</syntaxhighlight>
 
==== Get Current License Token from DB ====
 
<syntaxhighlight lang="sql">
SELECT content FROM voipmonitor.system WHERE type='license token';
</syntaxhighlight>
 
 
== Scripts for Better Run Control ==
 
=== Alerts and Reports ===
 
When alerts take longer time to process than the scheduled interval, it can cause out of memory or high CPU load because the same alert is processed multiple times for various time ranges.
 
The GUI now supports parallel run of alerts/reports - enable it in '''System settings → Advanced → Number of parallel tasks'''.
 
For deprecated manual approach, see: [[script-for-multi-Instances-alerts-processing]]
 
=== Batch Download for More Than 1000 CDRs ===
 
When you need to download a lot of audio files via '''GUI → Menu → CDR batch download''', it allows only 1000 calls to process with one query.
 
Following script allows you to create audio from a longer set of CDRs, also processing in multiple instances to get audio faster (beware of HDD IO/CPU): [[script-for-big-batchdownloads]]
 
=== RAM Control (Alerts / Dashboards Cache) ===
 
When there are many CDRs, alerts or caching of dashboard data can consume a lot of RAM. Following script controls if any PHP thread consumes over the limit. When it does, its <code>oom_adj_score</code> is set to be killed first (prevents killing of voipmonitor sniffer or DB when running on the same server).
 
See: [[script-for-GUI-cron-control-oom]]
 
=== Backup Settings from All Probes and GUI ===
 
The script fetches <code>voipmonitor.conf</code> from all probes and backs up GUI config tables from the database. Requires SSH access without password. Run via cron.
 
See: [[script-for-probes-backup]]
 
== Charts Binary Crash Debug Steps ==
 
When Apache log or dmesg reports crashes of the VM charts binary like:


When dmesg or Apache log shows charts crash:
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]
charts-x86_64[7321]: segfault at 10 ip 0000000000412553...
</syntaxhighlight>
</syntaxhighlight>


=== Create Debug Folder for Charts ===
'''Debug procedure:'''
 
Create <code>vm_charts_parameters</code> in <code>/tmp</code>:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Create debug folder
mkdir /tmp/vm_charts_parameters
mkdir /tmp/vm_charts_parameters
</syntaxhighlight>


After GUI creates some charts, files with arguments will be created there.
# After GUI creates charts, test with saved arguments
 
=== Test Arguments Passed to Charts Binary ===
 
<syntaxhighlight lang="bash">
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
</syntaxhighlight>
</syntaxhighlight>


== VoIPmonitor Sniffer Memory Leaks Detection and Debug ==
== Memory Leak Detection (Jemalloc) ==


When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more calls (regs/messages), following steps can tell you where the RAM ends (it can be a leak or some thread overload).
=== Compile Jemalloc ===
 
=== Jemalloc ===
 
==== Compile Jemalloc ====
 
Download jemalloc from https://github.com/jemalloc/jemalloc


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./autogen.sh
# Requires development tools: apt install build-essential (Deb/Ubuntu), yum groupinstall 'Development Tools' (CentOS/RHEL)
./configure --enable-prof
./configure --enable-prof
make
make && make install
make install
</syntaxhighlight>
 
==== Enable Jemalloc for VoIPmonitor and Compile It ====
 
<syntaxhighlight lang="bash">
./configure
</syntaxhighlight>
</syntaxhighlight>


Edit <code>config.h</code> to have:
=== Compile VoIPmonitor with Jemalloc ===


Edit <code>config.h</code>:
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBTCMALLOC 0
Line 201: Line 129:
</syntaxhighlight>
</syntaxhighlight>


Edit <code>Makefile</code> and disable optimization, add ljemalloc to libs, remove -ltcmalloc:
Edit <code>Makefile</code>:
 
<syntaxhighlight lang="makefile">
<syntaxhighlight lang="makefile">
# Substitute -O2 with -O0 (capital O followed by zero)
# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc
LIBS = ${SHARED_LIBS} -ljemalloc
</syntaxhighlight>
</syntaxhighlight>
Finish compilation:
<syntaxhighlight lang="bash">
make clean
make -j8
</syntaxhighlight>
==== Edit /etc/init.d/voipmonitor ====
Ensure:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ARGS="-v 1"
make clean && make -j8
</syntaxhighlight>
</syntaxhighlight>


And in the start section (2 places) of the script:
=== Run with Jemalloc ===


Modify <code>/etc/init.d/voipmonitor</code> start section:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
</syntaxhighlight>
</syntaxhighlight>


==== Check Results ====
=== Generate Memory Report ===
 
At regular intervals you should check the results of memory usage.
 
===== je_prof.pdf =====
 
Don't forget to install graphviz and ghostscript:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
yum install graphviz ghostscript
# Install dependencies
# or: apt install graphviz ghostscript
apt install graphviz ghostscript # or: yum install graphviz ghostscript
</syntaxhighlight>
 
Generate the report:


<syntaxhighlight lang="bash">
# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf
</syntaxhighlight>
</syntaxhighlight>


===== sniffer_memory.txt =====
=== X88 Mode (Detailed Memory Debug) ===
 
<syntaxhighlight lang="bash">
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
</syntaxhighlight>
 
===== Script for Periodical Mail Sending =====
 
Add to <code>cron.hourly</code>:
 
<syntaxhighlight lang="bash">
#!/bin/bash
export PATH=/usr/local/bin:/usr/bin:/bin
mail="support@voipmonitor.org"
 
sniffer_bin="/usr/local/src/sniffer/voipmonitor"
manager_ip="127.0.0.1"
manager_port="5029"
 
tmp_je_prof_pdf="/tmp/je_prof.pdf"
tmp_je_prof_out="/tmp/je_prof"
tmp_pb_log="/tmp/pb_log"
tmp_sn_stat="/tmp/sn_stat"
 
echo 'jemalloc_stat_full' | nc $manager_ip $manager_port > $tmp_je_prof_out
jeprof --show_bytes --pdf $sniffer_bin $tmp_je_prof_out > $tmp_je_prof_pdf 2>/dev/null
echo 'packetbuffer_log' | nc $manager_ip $manager_port > $tmp_pb_log
echo 'sniffer_stats' | nc $manager_ip $manager_port > $tmp_sn_stat
 
# Send email with attachments
echo 'jeprof' | mail -s 'jeprof' -a $tmp_je_prof_pdf -a $tmp_pb_log -a $tmp_sn_stat $mail
 
rm $tmp_je_prof_out $tmp_je_prof_pdf $tmp_pb_log $tmp_sn_stat
</syntaxhighlight>
 
=== Enable X88 Mode ===
 
==== X88 Compatible Sources and Recompile ====
 
Copy sniffer sources with compiled binary to a new location and recreate binary with -X88 (necessary to get correct memory allocation for debugging using jeprof):


For more detailed memory allocation info:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
# It will list source files modification, then re-make
make clean && make
make clean
make
</syntaxhighlight>
 
More detailed instructions: [[X88-memoryConsumptionDebug]]
 
==== Modify Init File or Run Sniffer Manually ====
 
===== Edit /etc/init.d/voipmonitor =====
 
<syntaxhighlight lang="bash">
ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
# And in start section (2 places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
</syntaxhighlight>
</syntaxhighlight>


===== Or Run from Console (when sniffer service stopped) =====
Run with extra verbose options:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor \
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
   --config-file=/etc/voipmonitor.conf \
   --config-file=/etc/voipmonitor.conf \
   -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
   -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
</syntaxhighlight>
</syntaxhighlight>


== Info About Internal Threads ==
See: [[X88-memoryConsumptionDebug]]


=== More Info About Threads in the Log ===
== Thread Debugging ==


Add verbose option <code>threads_cpu_log</code> into command line (<code>-v1,threads_cpu_log</code>). If enabled, it prints the load of the threads to the syslog in the line output.
{| class="wikitable"
! Method !! Usage
|-
| Log thread CPU || Add <code>-v1,threads_cpu_log</code> to command line
|-
| List threads || <code>echo 'sniffer_threads' | nc 127.0.0.1 5029</code>
|-
| Extended info || Set <code>sniffer_threads_ext=yes</code> in config (requires compile with <code>SNIFFER_THREADS_EXT true</code>)
|}


=== Basic Usage of Manager Command 'sniffer_threads' ===
'''sniffer_threads options:'''
* <code>no_sort</code> - don't sort by load
* <code>only_traffic</code> - only threads with traffic
* <code>all</code> - include unloaded threads
* <code>line</code> - simple line output


<syntaxhighlight lang="bash">
== SS7 Module Compilation ==
echo 'sniffer_threads' | nc 127.0.0.1 5029
</syntaxhighlight>


Options:
Requires Wireshark 3.2.
* <code>no_sort</code> - do not sort by load in descending order
* <code>only_traffic</code> - lists only threads with recorded traffic (requires <code>sniffer_threads_ext</code>)
* <code>all</code> - lists all threads (even unloaded ones)
* <code>line</code> - use simple line output
 
=== Debugging Output of Enhanced Manager Command 'sniffer_threads' ===
 
Now includes:
* Traffic passing through each thread
* The percentage of states where the buffer between threads is full when a packet is attempted to be added to the buffer
* The percentage of usleep times when waiting for buffer space to become available, out of the total time
 
Required steps:
# Compile the sniffer with <code>#define SNIFFER_THREADS_EXT true</code> (in <code>voipmonitor_define.h</code>)
# Set <code>sniffer_threads_ext=yes</code> in the configuration, or execute <code>echo 'setverbparam sniffer_threads_ext' | nc ...</code>
 
Output example:
 
<syntaxhighlight lang="text">
root@test:~# echo "sniffer_threads" | nc 127.0.0.1 5029
t2 sip preprocess detach                      (  21939) :  0.0                    1 /    72837 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  1.39%    0.00%
t2 sip preprocess sip                        (  21940) :  0.0                    1 /    69402 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  0.01%    0.00%
t2 sip preprocess rtp                        (  21947) :  0.0                    2 /    69922 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  0.01%    0.00%
t2 rtp preprocess hash                        (  21952) :  0.0                  10 /    67685 /  0.000                521p 8.54Mb ->  --- p  --- b  bf  0.00%    0.00%
t2 out thread defrag                          (  21954) :  0.0                    1 /    77085 /  0.000                823p 8.80Mb ->    815p 8.80Mb  bf  5.88%    0.00%
pb - main queue                              (  21955) :  0.0                    0 /    26550 /  0.000                954p 10.2Mb ->    954p 10.2Mb
pb - write queue                              (  21956) :  0.0                    0 /    23481 /  0.000                954p 10.2Mb ->    878p 9.32Mb
pb - read thread eth1 read                    (  21957) :  0.0                  123 /      378 /  0.325                954p 10.2Mb ->    954p 10.2Mb
pb - read thread eth1 pcap_process            (  21960) :  0.0                    0 /    57547 /  0.000                954p 10.2Mb ->    954p 10.2Mb
 
CPU ALL :  0.0
</syntaxhighlight>
 
== VoIPmonitor Sniffer Compile Notes: SS7 (Wireshark) Module ==
 
Requires Wireshark version 3.2.
 
=== Modify config.h ===


Edit <code>config.h</code>:
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
#define HAVE_LIBWIRESHARK 1
#define HAVE_LIBWIRESHARK 1
Line 375: Line 203:
</syntaxhighlight>
</syntaxhighlight>


=== Modify Makefile ===
Edit <code>Makefile</code>:
 
Add to INCLUDE:
 
<syntaxhighlight lang="makefile">
-I/usr/include/wireshark
</syntaxhighlight>
 
Add to SHARED_LIBS:
 
<syntaxhighlight lang="makefile">
<syntaxhighlight lang="makefile">
-lwiretap -lwireshark -lwsutil
INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0
</syntaxhighlight>
</syntaxhighlight>
Internal build location: <code>JIRA:/opt/debian-7-64bit</code>
=== Disable Optimization in Makefile ===
Change <code>-O2</code> to <code>-O0</code>.
=== Compile ===


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 401: Line 214:
</syntaxhighlight>
</syntaxhighlight>


== Testing Customer PCAPs ==
= PCAP Testing =


=== Testing Special PCAPs ===
== Basic Methods ==


==== IPFIX ====
{| class="wikitable"
! Method !! Command
|-
| Basic upload || <code>voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap</code>
|-
| Via packetbuffer (fast) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap</code>
|-
| Via packetbuffer (original time) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap</code>
|}


Run first instance of voipmonitor that will act as receiver for packets and push CDRs to DB. In the settings enable IPFIX options:
== IPFIX Testing ==


'''Receiver instance:'''
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
ipfix = yes
ipfix = yes
Line 415: Line 237:
</syntaxhighlight>
</syntaxhighlight>


Then run second instance that will act as source to replay the IPFIX encapsulated PCAP:
'''Sender instance:'''
 
<syntaxhighlight lang="bash">
./voipmonitor --config-file=config/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
</syntaxhighlight>
 
Parameters:
* PCAP file path
* IPFIX client IP
* IPFIX server IP
* Destination IP:port (where to send the IPFIX stream - matching the first instance)
 
=== Testing via Basic Upload (GUI Upload) ===
 
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap
</syntaxhighlight>
 
=== Testing via Packetbuffer Upload ===
 
==== Time Shift to Now, Increased Speed of Replaying ====
 
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap
</syntaxhighlight>
 
==== No Time Shift via Packetbuffer ====
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port
</syntaxhighlight>
</syntaxhighlight>
=== IPFIX PCAP Download Behavior ===


==== Tcpreplay to Running Service ====
'''Key fact:''' IPFIX data is internally converted to packet format for processing. This means:
* '''PCAP files CAN be downloaded''' from the GUI for IPFIX-sourced calls
* SIP signaling is reconstructed from the IPFIX data
* '''RTP streams are NOT included''' in the PCAP - only QoS metrics from IPFIX data are available, not actual audio packets


Enable additional argument when starting the service in <code>/etc/init.d/voipmonitor</code>:
When downloading PCAP from IPFIX calls with TLS, TCP sequence numbers may be set to 0 causing Wireshark to display incorrectly. Disable TCP sequence analysis in Wireshark preferences to view all packets.
== Tcpreplay ==


Enable in <code>/etc/init.d/voipmonitor</code>:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ARGS="-v 1,dump_call_flags,tcpreplay"
ARGS="-v 1,dump_call_flags,tcpreplay"
</syntaxhighlight>
</syntaxhighlight>


Then use tcpreplay:
'''Add Layer 2 if missing''' (for captures done on "any" interface):
 
<syntaxhighlight lang="bash">
tcpreplay -i eth0 /tmp/l/x.pcap
</syntaxhighlight>
 
If you get an error like:
 
<syntaxhighlight lang="text">
sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)
</syntaxhighlight>
 
This means the capture was done on "any" interface and layer 2 is missing. Follow steps below to add it.
 
===== Adding Layer 2 to PCAP =====
 
After capture was done on "any" interface and you need to replay it to a hardware interface, you need to add layer 2:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
   --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap
   --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap
</syntaxhighlight>
</syntaxhighlight>


===== Rewrite IPv4 Address in Whole PCAP =====
'''Rewrite IP addresses:'''
 
When you need to test alerts based on IP addresses (also country/continent alerts based on IPs), you can change IP addresses in PCAP and use it multiple times with various addresses:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap
</syntaxhighlight>
</syntaxhighlight>


== Rename MySQL Database ==
== SSL/TLS Testing ==
 
It's not possible to do it via a direct command. You need to use '''RENAME TABLE''' command.
 
With this SELECT you generate the commands for renaming all tables in the database (replace <code>OLD_DB_NAME</code> and <code>NEW_DB_NAME</code>):
 
<syntaxhighlight lang="sql">
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
    ' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';
</syntaxhighlight>
 
The output:
 
<syntaxhighlight lang="sql">
RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;
</syntaxhighlight>
 
Run these commands and you are ready.
 
'''Attention!''' Sometimes you can have a problem with moving foreign keys.
 
== Disabling TCP SACK ==
 
On buggy kernels (all before 2019/03), following prevents the SACK issue. Side effect is that on lossy lines (packet loss or big delays) the TCP traffic will increase:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
echo "0" > /proc/sys/net/ipv4/tcp_sack
voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'
</syntaxhighlight>
</syntaxhighlight>


== MacOS High Sierra in VirtualBox (on Windows) ==
For TLS details, see [[Tls]].


=== Download Image ===
= Manager API =


From: https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view
'''For complete reference, see [[Manager_API]].'''


=== Prepare ISO for Install ===
This section covers quick debug commands. For encryption setup and full command list, see the main article.


==== Create High Sierra DMG Disk Image ====
== Via Manager Port (5029) ==


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J
# These require encryption disabled OR use socket file
echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stat' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
echo 'sniffer_threads' | nc 127.0.0.1 5029
</syntaxhighlight>
</syntaxhighlight>


==== Mount DMG Image to Your macOS ====
{{Tip|For encrypted communication, use the GUI's <code>run.php send_manager_cmd</code> or Unix socket. See [[Manager_API#Usage Examples]].}}


<syntaxhighlight lang="bash">
== Via Server Port (60024) ==
hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra
</syntaxhighlight>


==== Create a High Sierra ISO Image ====
Used in distributed deployments for probe management:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra
# List connected clients
</syntaxhighlight>
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024


==== Unmount Disk Image ====
# Send command to specific probe
 
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024
<syntaxhighlight lang="bash">
hdiutil detach /volumes/Install\ macOS\ High\ Sierra
</syntaxhighlight>
</syntaxhighlight>


==== Convert DMG to ISO ====
= GUI Workarounds =


<syntaxhighlight lang="bash">
== Disable Sensor Logs Popup ==
hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra
</syntaxhighlight>


==== Rename and Move ISO Image to Desktop ====
{{Note|There is no built-in GUI option to disable the sensor logs popup shown at admin login.}}


<syntaxhighlight lang="bash">
'''Option 1: Periodic SQL update'''
mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso
<syntaxhighlight lang="sql">
UPDATE voipmonitor.log_sensor SET confirmed = 1;
</syntaxhighlight>
</syntaxhighlight>


=== Create New Virtual Image ===
'''Option 2: Database trigger (permanent)'''
 
<syntaxhighlight lang="sql">
During pre-configure of new image set:
DELIMITER $$
* Memory: 3-6 GB
CREATE TRIGGER before_insert_log_sensor
* Disk size: 100 GB
BEFORE INSERT ON log_sensor
 
FOR EACH ROW
Then edit its settings and additionally modify:
BEGIN
* System → Motherboard → Chipset: PIIX3
    SET NEW.confirmed = 1;
* System → Processor → Count: 2, Enable PAE/NX
END$$
* System → Storage → Attributes → Optical drive → SATA port 1, for the newly created ISO
DELIMITER ;
* Display → Video memory → 128 MB
* Display → Graphics Controller → VMSVGA
 
=== Add the VirtualBox macOS Code to CMD ===
 
<syntaxhighlight lang="batch">
cd "C:\Program Files\Oracle\VirtualBox\"
VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1
</syntaxhighlight>
</syntaxhighlight>


=== Install macOS on VirtualBox ===
== IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4) ==


# Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
'''Symptom:''' ldap.so crash at login page (CVE-2024-8932).
# Format VirtualBox Virtual Disk with "Mac OS Extended" (GUID Partition Map).
# Install macOS High Sierra on new virtual disk with your ISO Image.
# After installation is completed, turn off your macOS High Sierra. Open VirtualBox and remove High Sierra.iso file.
# Turn on your Virtual Machine. At UEFI Shell screen, boot with this command:
 
<syntaxhighlight lang="text">
>"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"
</syntaxhighlight>
 
=== Start the High Sierra Virtual Machine and Enjoy ===
 
== Measurement ==
 
=== Memory Speed ===


'''Fix:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sysbench memory --threads=2 run
# Backup
</syntaxhighlight>
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP


Should be over 5000 MiB/sec.
# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb


== SSL and Keylogger ==
# Install and restart
 
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
=== How to Test It in VoIPmonitor ===
systemctl restart apache2
 
<syntaxhighlight lang="bash">
./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'
</syntaxhighlight>
</syntaxhighlight>


=== How to Test It in Wireshark/Tshark ===
= Database Operations =


==== Get the Keys from Keylogger's Dump ====
== Rename MySQL Database ==


<syntaxhighlight lang="bash">
Generate RENAME TABLE commands:
ngrep -I /tmp/keycapture.pcap | grep '^ ' | cut -d ' ' -f3- > /tmp/keys.txt
<syntaxhighlight lang="sql">
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';
</syntaxhighlight>
</syntaxhighlight>


==== Decrypt with Tshark ====
{{Warning|1=Foreign keys may cause issues during rename.}}
 
<syntaxhighlight lang="bash">
tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'
</syntaxhighlight>
 
==== Decrypt in Wireshark ====
 
In Edit → Preferences → Protocol → TLS → Pre-Master Secret log filename, choose the keys.txt file.
Then find the Client Hello HS packet, right-click on it and follow stream → TLS.
 
==== Check if We Have the Right Keys for RTP/DTLS ====
 
In RTP you have to find DTLS packets (<code>dtls.record.content_type == 22</code>), select 'client random' in them and find the 'Random' item.
 
That's e.g. <code>e54fc0f917cdd5c0d914f24956cdfd689022709f59ca4cd7fc20296db17039d8</code> in our sample. This must be included in the captured keys.


== Spooldir Testing ==
== Spooldir Testing ==
When there are problems with cleaning the spooldir or spooldir size does not respect the settings, you can use following command to test:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 660: Line 370:
</syntaxhighlight>
</syntaxhighlight>


== Manager API via Server Port ==
= Remote Support Access =


=== Get List of Connected Clients ===
== VPN Connection Issues ==


<syntaxhighlight lang="bash">
When support staff cannot connect to client VPN (Fortinet, etc.):
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024
</syntaxhighlight>


=== Send Terminating Manager Command to a Probe ===
# '''Cause:''' Support IP not whitelisted on client firewall
# '''Solution:''' Client must add support staff's public IP to their VPN/firewall whitelist
# '''Note:''' This is client-side network administration, not a VoIPmonitor issue


<syntaxhighlight lang="bash">
{{Note|1=Don't confuse VPN network access with FortiToken 2FA (GUI authentication).}}
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024
</syntaxhighlight>


== Additional Resources ==
= System Troubleshooting =


=== HA Proxy ===
== TCP SACK (Buggy Kernels pre-2019/03) ==
[[sample HAproxy configuration]]
 
=== High CPS Config for VoIPmonitor with Hugepages ===
[[voipmonitor and hugepages]]
 
=== IOPS Tests ===
 
<code>--rw</code> can be <code>randread</code>, <code>randwrite</code>, <code>randrw</code>:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
echo "0" > /proc/sys/net/ipv4/tcp_sack
  --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap \
  --filename=/root/testW.dat --output=/root/fio_db_write.txt
</syntaxhighlight>
</syntaxhighlight>


=== Rocky 9 Installation ===
== Memory Speed Test ==
[[Rocky 9]]
 
=== Manager API of a Sniffer ===
[[encryption_in_manager_api]]
 
=== Manager API for Customers ===
[[encryption_in_manager_api_customer]]
 
=== TLS Decode by Ribbon SBC 7k ===
[[Ribbon7k monitoring profiles]]
 
=== Email Templates ===
Common requests in support: [[common body of support emails]]
 
=== IPACCOUNT ===
How to enable [[ipaccount]] feature in voipmonitor.
 
=== DTLS ===
[[DTLS]] SRTP decryption using voipmonitor.
 
== Remote Support Access via Client VPN ==
 
=== VPN Connection Issues: IP Whitelisting ===
 
When support staff cannot connect to a client's VPN (Fortinet or other enterprise firewall), this is typically due to the support staff IP address not being whitelisted on the client-side firewall configuration.
 
=== Standard Procedure ===
 
1. **Identify the issue**: Connection attempt fails without prompting for OTP/token, indicating firewall-level blocking.
 
2. **Action required by the client**:
  * Obtain the public IP address of the support staff member requiring VPN access
  * Add this IP address to the whitelist on their Fortinet VPN or firewall configuration
  * This is '''client-side network administration''' - not a VoIPmonitor configuration issue
 
3. **After whitelisting**: Support staff should attempt VPN connection again
 
=== Important Notes ===
 
* This is a general networking/firewall administration task specific to each customer's security policies
* VoIPmonitor support team coordinates with customers but does not have direct access to customer firewall configurations
* Guide the customer to consult their network/firewall administrator for IP whitelisting procedures
* Do not confuse with FortiToken 2FA - that is VoIPmonitor GUI authentication, different from VPN network access
 
== IonCube Crashes for Apache/HTTPS ==
 
=== Wrong Privileges ===
 
If following command crashes under www-data user but not under root:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data
sysbench memory --threads=2 run
</syntaxhighlight>
# Should be >5000 MiB/sec
 
It can be caused by wrong permissions for the ioncube.ini file or ioncube module. You can test with strace (search for "ioncube" or "denied"):
 
<syntaxhighlight lang="bash">
strace -f su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data 2> /tmp/text.txt
</syntaxhighlight>
 
=== Crashes in libc (Ubuntu 22.04 with PHP 7.4) ===
 
It was experienced in Ubuntu 22.04 with PHP 7.4 that ioncube causes crash in ldap.so (displays following error on the login page):
 
<syntaxhighlight lang="text">
Failed check Ioncube.com PHP Loader for php cli : PHP Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/lib/php/20190902/ldap.so (/usr/lib/php/20190902/ldap.so: undefined symbol: RETURN_THROWS), /usr/lib/php/20190902/ldap.so.so (/usr/lib/php/20190902/ldap.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
</syntaxhighlight>
</syntaxhighlight>


The reason is a broken ldap library in Ubuntu 22.04:
= Additional Resources =
* https://ubuntu.com/security/CVE-2024-8932
* https://askubuntu.com/questions/1535405/php7-4-ldap-broken-on-focal-after-13-dec-2024
 
==== Fix ====


Backup current ldap libs:
{| class="wikitable"
! Topic !! Link
|-
| HAProxy config || [[sample HAproxy configuration]]
|-
| Hugepages for high CPS || [[voipmonitor and hugepages]]
|-
| Manager API reference || [[Manager_API]]
|-
| Syslog status line || [[Syslog_Status_Line]]
|-
| Ribbon SBC 7k TLS || [[Ribbon7k monitoring profiles]]
|-
| Support email templates || [[common body of support emails]]
|-
| IPACCOUNT feature || [[ipaccount]]
|-
| DTLS decryption || [[DTLS]]
|-
| Rocky 9 installation || [[Rocky 9]]
|}


'''IOPS testing:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cd /usr/lib/php/20190902
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
mkdir BKUP
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat
cp -pv ldap.so* BKUP
</syntaxhighlight>
</syntaxhighlight>


Download and extract the package:
<syntaxhighlight lang="bash">
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
</syntaxhighlight>
Copy in the fixed lib:
<syntaxhighlight lang="bash">
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
</syntaxhighlight>
Restart Apache and test:
<syntaxhighlight lang="bash">
systemctl restart apache2
</syntaxhighlight>


''Thanks to Andrew from distributel.ca for this article.''


== AI Summary for RAG ==
= AI Summary for RAG =


'''Summary:''' Internal support documentation for VoIPmonitor team. Covers testing configuration options using <code>--json_config</code> to override settings without modifying config files. Includes testing procedures for regexp patterns, country assignment (by IP or phone number), and billing/pricing. License management hints include checking channel usage (<code>checkLicense -v</code>), updating license via cron, and license portability notes (one license can be used at multiple physical sites). Scripts section covers parallel alerts processing, batch downloads for >1000 CDRs, RAM control for PHP threads, and probe backup automation. Debugging sections include charts binary crash analysis, memory leak detection using jemalloc (compile instructions, jeprof PDF generation, X88 mode), internal thread monitoring (<code>sniffer_threads</code> command), and SS7/Wireshark module compilation. PCAP testing covers IPFIX emulation, basic upload, packetbuffer replay, tcpreplay with layer 2 addition, and IP address rewriting. Remote Support section covers VPN connection issues: when support staff cannot connect to client VPN, the typical resolution is IP whitelisting on client-side firewall (Fortinet or other). This is network administration, not VoIPmonitor configuration. Additional topics: MySQL database rename procedure, TCP SACK disabling for buggy kernels, macOS High Sierra in VirtualBox setup, SSL keylogger testing, spooldir troubleshooting, Manager API commands via server port, and IonCube crash fixes for Ubuntu 22.04 PHP 7.4 ldap.so issue.
'''Summary:''' Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with <code>--json_config</code> to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (<code>checkLicense</code>, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via <code>sniffer_threads</code> command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).


'''Keywords:''' internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, MySQL rename, TCP SACK, VirtualBox macOS, SSL keylogger, spooldir, Manager API, IonCube, ldap.so crash, Ubuntu 22.04, VPN access, IP whitelisting, Fortinet VPN, remote support, firewall
'''Keywords:''' internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed


'''Key Questions:'''
'''Key Questions:'''
* How to test configuration options without modifying voipmonitor.conf?
* How to test configuration without modifying voipmonitor.conf?
* How to use json_config to override sniffer settings?
* How to test regexp patterns or country assignment?
* How to test regexp patterns in voipmonitor?
* How to check license channel usage?
* How to find country by IP or phone number?
* How to debug memory leaks using jemalloc?
* How to test billing/pricing configuration?
* How to compile voipmonitor with jemalloc or SS7 support?
* How to check license channel usage details?
* How to generate jeprof PDF memory report?
* How to keep license updated automatically via cron?
* How to monitor thread load with sniffer_threads?
* Can one VoIPmonitor license be used at multiple physical sites?
* How to test IPFIX or replay PCAPs?
* How to debug memory leaks in voipmonitor using jemalloc?
* How to compile voipmonitor with jemalloc support?
* How to generate jeprof PDF memory analysis report?
* What is X88 mode and how to enable it for memory debugging?
* How to monitor internal thread load using sniffer_threads command?
* How to compile voipmonitor with SS7/Wireshark support?
* How to test IPFIX encapsulated PCAPs?
* How to replay PCAPs using tcpreplay?
* How to add layer 2 to PCAP for tcpreplay?
* How to add layer 2 to PCAP for tcpreplay?
* How to rewrite IP addresses in PCAP files?
* How to disable sensor logs popup in GUI?
* How to rename MySQL database in voipmonitor?
* What to do when support staff cannot connect to client VPN?
* How to resolve VPN connection issues via IP whitelisting?
* What is the difference between FortiToken 2FA and Fortinet VPN access?
* How to fix IonCube ldap.so crash in Ubuntu 22.04?
* How to fix IonCube ldap.so crash in Ubuntu 22.04?
* How to resolve VPN connection issues for remote support?
* How to disable TCP SACK on buggy kernels?
* How to test memory speed with sysbench?

Latest revision as of 23:55, 9 January 2026


⚠️ Warning: This documentation is for the internal support team. Use at your own risk and test on non-production servers first.

Quick Reference

Testing Configuration

Override Settings with json_config

Override config file and database settings without editing files: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'

💡 Tip: Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.

Regexp / Country / Billing Tests

Test Type Command
Regexp pattern voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'
Country by IP voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113
Country by number voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345
Billing test voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing

Billing test file format (CSV): 

# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8

License Management

Task Command
Write channel stats to DB php /var/www/html/php/run.php saveCallStatistics
Check license details php /var/www/html/php/run.php checkLicense -v
Check max peak only php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
Get token from DB SELECT content FROM voipmonitor.system WHERE type='license token';
View license file cat /var/www/html/key.php

Auto-update license via cron (without processing alerts/reports): 

04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey

Utility Scripts

Script Purpose Details
Parallel alerts Process alerts faster Enable in Settings → System Configuration → Advanced → Number of parallel tasks
Batch download Download >1000 audio files Bypasses GUI limit of 1000 CDRs
RAM control Prevent OOM killer Sets oom_adj_score for PHP threads
Probe backup Backup all probe configs Requires SSH access without password

Debugging

Charts Binary Crash

When dmesg or Apache log shows charts crash: 

charts-x86_64[7321]: segfault at 10 ip 0000000000412553...

Debug procedure: 

# Create debug folder
mkdir /tmp/vm_charts_parameters

# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

Memory Leak Detection (Jemalloc)

Compile Jemalloc

# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./configure --enable-prof
make && make install

Compile VoIPmonitor with Jemalloc

Edit config.h: 

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

Edit Makefile: 

# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc

make clean && make -j8

Run with Jemalloc

Modify /etc/init.d/voipmonitor start section: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

Generate Memory Report

# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript

# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf

X88 Mode (Detailed Memory Debug)

For more detailed memory allocation info: 

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make

Run with extra verbose options: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
  --config-file=/etc/voipmonitor.conf \
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

See: X88-memoryConsumptionDebug

Thread Debugging

Method Usage
Log thread CPU Add -v1,threads_cpu_log to command line
List threads nc 127.0.0.1 5029
Extended info Set sniffer_threads_ext=yes in config (requires compile with SNIFFER_THREADS_EXT true)

sniffer_threads options:

  • no_sort - don't sort by load
  • only_traffic - only threads with traffic
  • all - include unloaded threads
  • line - simple line output

SS7 Module Compilation

Requires Wireshark 3.2.

Edit config.h: 

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

Edit Makefile: 

INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0

./configure && make clean && make ss7

PCAP Testing

Basic Methods

Method Command
Basic upload voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap
Via packetbuffer (fast) voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap
Via packetbuffer (original time) voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap

IPFIX Testing

Receiver instance: 

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Sender instance: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port

IPFIX PCAP Download Behavior

Key fact: IPFIX data is internally converted to packet format for processing. This means:

  • PCAP files CAN be downloaded from the GUI for IPFIX-sourced calls
  • SIP signaling is reconstructed from the IPFIX data
  • RTP streams are NOT included in the PCAP - only QoS metrics from IPFIX data are available, not actual audio packets

When downloading PCAP from IPFIX calls with TLS, TCP sequence numbers may be set to 0 causing Wireshark to display incorrectly. Disable TCP sequence analysis in Wireshark preferences to view all packets.

Tcpreplay

Enable in /etc/init.d/voipmonitor: 

ARGS="-v 1,dump_call_flags,tcpreplay"

Add Layer 2 if missing (for captures done on "any" interface): 

tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap

Rewrite IP addresses: 

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap

SSL/TLS Testing

voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'

For TLS details, see Tls.

Manager API

For complete reference, see Manager_API.

This section covers quick debug commands. For encryption setup and full command list, see the main article.

Via Manager Port (5029)

# These require encryption disabled OR use socket file
echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stat' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
echo 'sniffer_threads' | nc 127.0.0.1 5029

💡 Tip: For encrypted communication, use the GUI's run.php send_manager_cmd or Unix socket. See Manager_API#Usage Examples.

Via Server Port (60024)

Used in distributed deployments for probe management: 

# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024

# Send command to specific probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024

GUI Workarounds

Disable Sensor Logs Popup

ℹ️ Note: There is no built-in GUI option to disable the sensor logs popup shown at admin login.

Option 1: Periodic SQL update 

UPDATE voipmonitor.log_sensor SET confirmed = 1;

Option 2: Database trigger (permanent) 

DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;

IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4)

Symptom: ldap.so crash at login page (CVE-2024-8932).

Fix: 

# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP

# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

# Install and restart
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2

Database Operations

Rename MySQL Database

Generate RENAME TABLE commands: 

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';

⚠️ Warning: Foreign keys may cause issues during rename.

Spooldir Testing

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Remote Support Access

VPN Connection Issues

When support staff cannot connect to client VPN (Fortinet, etc.):

  1. Cause: Support IP not whitelisted on client firewall
  2. Solution: Client must add support staff's public IP to their VPN/firewall whitelist
  3. Note: This is client-side network administration, not a VoIPmonitor issue

ℹ️ Note: Don't confuse VPN network access with FortiToken 2FA (GUI authentication).

System Troubleshooting

TCP SACK (Buggy Kernels pre-2019/03)

echo "0" > /proc/sys/net/ipv4/tcp_sack

Memory Speed Test

sysbench memory --threads=2 run
# Should be >5000 MiB/sec

Additional Resources

Topic Link
HAProxy config sample HAproxy configuration
Hugepages for high CPS voipmonitor and hugepages
Manager API reference Manager_API
Syslog status line Syslog_Status_Line
Ribbon SBC 7k TLS Ribbon7k monitoring profiles
Support email templates common body of support emails
IPACCOUNT feature ipaccount
DTLS decryption DTLS
Rocky 9 installation Rocky 9

IOPS testing: 

fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat


AI Summary for RAG

Summary: Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with --json_config to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (checkLicense, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via sniffer_threads command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).

Keywords: internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed

Key Questions:

  • How to test configuration without modifying voipmonitor.conf?
  • How to test regexp patterns or country assignment?
  • How to check license channel usage?
  • How to debug memory leaks using jemalloc?
  • How to compile voipmonitor with jemalloc or SS7 support?
  • How to generate jeprof PDF memory report?
  • How to monitor thread load with sniffer_threads?
  • How to test IPFIX or replay PCAPs?
  • How to add layer 2 to PCAP for tcpreplay?
  • How to disable sensor logs popup in GUI?
  • How to fix IonCube ldap.so crash in Ubuntu 22.04?
  • How to resolve VPN connection issues for remote support?
  • How to disable TCP SACK on buggy kernels?
  • How to test memory speed with sysbench?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Internal_support_hints&oldid=10474"