Internal support hints: Difference between revisions

From VoIPmonitor.org
(Add workaround for disabling sensor logs popup)
(Rewrite: consolidate structure, use tables, remove obsolete content (macOS VirtualBox setup), improve organization)
Line 2: Line 2:
{{DISPLAYTITLE:Internal Support Hints}}
{{DISPLAYTITLE:Internal Support Hints}}


{{Warning|1=Following articles are designed to be used by internal support team. You can find hints useful but please use it at your own risk (use on testing servers first).}}
{{Warning|1=This documentation is for the internal support team. Use at your own risk and test on non-production servers first.}}


== Quick Reference ==
= Quick Reference =


<kroki lang="mermaid">
<kroki lang="mermaid">
Line 10: Line 10:
flowchart LR
flowchart LR
     subgraph Test["Testing"]
     subgraph Test["Testing"]
         A[json_config] --> B[Regexp]
         A[json_config] --> B[Regexp/Country]
         B --> C[Country]
         B --> C[Billing]
        C --> D[Billing]
     end
     end
     subgraph Debug["Debugging"]
     subgraph Debug["Debugging"]
         E[Memory/jemalloc] --> F[Threads]
         D[Jemalloc] --> E[Threads]
         F --> G[Charts crash]
         E --> F[Charts crash]
     end
     end
     subgraph Tools["Tools"]
     subgraph Tools["Tools"]
         H[PCAP replay] --> I[Manager API]
         G[PCAP replay] --> H[Manager API]
         I --> J[Scripts]
         H --> I[Scripts]
     end
     end
</kroki>
</kroki>


== Testing Configuration Options ==
= Testing Configuration =


If you need to test configuration options without modifying the config file or database settings, you can use the <code>json_config</code> option:
== Override Settings with json_config ==


<syntaxhighlight lang="bash">
Override config file and database settings without editing files:
--json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
</syntaxhighlight>
 
The options defined using '''json_config''' command line option will override the config file and database settings.
 
=== Examples ===
 
This will override options '''rtpip_find_endpoints''', '''rrd''', '''natalias''' no matter what is set in config file:
 
<syntaxhighlight lang="bash">
./voipmonitor --config-file=config/voipmonitor.conf -p XobmuJ -b voipmonitor_ipv6 -k -v1,pcap_stat_period=5,hash_rtp,rtp_streams \
  -rpba:/totest/test.pcap \
  --json_config='[{"rtpip_find_endpoints":"yes"},{"rrd":"no"},{"natalias":"8.8.8.8 10.10.100.50;9.9.9.9 192.168.88.33"}]'
</syntaxhighlight>
 
== Testing Regexp / Country Rules / Billing ==
 
=== Regexp Test ===
 
<syntaxhighlight lang="bash">
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --test-regexp '^800880080[1-9][0-9][*]|80088008012345|'
</syntaxhighlight>
 
=== Country Assignment Test ===
 
<syntaxhighlight lang="bash">
# Find country by IP
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-ip=102.140.68.113
 
# Find country by phone number
/usr/local/sbin/voipmonitor --config-file=/etc/voipmonitor-test.conf -k -v1 --find-country-for-number 0034*12345
</syntaxhighlight>
 
GUI version (obsolete):


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
root@vmweb:GUI_ROOT# php php/run.php testFindCountryForNumber -n 4909699999
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
number:           4909699999
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'
country:         DE
is_international: Y
</syntaxhighlight>
</syntaxhighlight>


=== Pricing / Billing Test ===
{{Tip|Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.}}


<syntaxhighlight lang="bash">
== Regexp / Country / Billing Tests ==
voipmonitor --config-file=config/voipmonitor.conf -k -v1 --test-billing test_data/billing
</syntaxhighlight>


==== Example of test_data/billing File ====
{| class="wikitable"
 
! Test Type !! Command
Column order:
|-
* calldate
| Regexp pattern || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'</code>
* connect_duration
|-
* caller
| Country by IP || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113</code>
* called
|-
* sipcallerip
| Country by number || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345</code>
* sipcalledip
|-
* expected price operator (optional)
| Billing test || <code>voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing</code>
* expected price customer (optional)
|}


'''Billing test file format''' (CSV):
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8.1
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151
</syntaxhighlight>
== License Hints ==
=== Write State to DB (count of concurrent channels for license purpose) ===
<syntaxhighlight lang="bash">
php /var/www/html/php/run.php saveCallStatistics
</syntaxhighlight>
</syntaxhighlight>


=== Keep License Updated Without Processing Alerts/Reports ===
= License Management =


Add to <code>/etc/crontab</code>:
{| class="wikitable"
 
! Task !! Command
<syntaxhighlight lang="bash">
|-
04 04  * * *  root php /var/www/html/php/run.php runUpdateLicenseKey
| Write channel stats to DB || <code>php /var/www/html/php/run.php saveCallStatistics</code>
</syntaxhighlight>
|-
 
| Check license details || <code>php /var/www/html/php/run.php checkLicense -v</code>
=== License Details ===
|-
 
| Check max peak only || <code>php /var/www/html/php/run.php checkLicense -v <nowiki>|</nowiki> grep max_peak -A3 -B1</code>
==== Detail on Channels Count Usage ====
|-
 
| Get token from DB || <code>SELECT content FROM voipmonitor.system WHERE type='license token';</code>
<syntaxhighlight lang="bash">
|-
php /var/www/html/php/run.php checkLicense -v
| View license file || <code>cat /var/www/html/key.php</code>
</syntaxhighlight>
|}
 
Less details:
 
<syntaxhighlight lang="bash">
php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
</syntaxhighlight>


==== License File ====
'''Auto-update license via cron''' (without processing alerts/reports):


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cat /var/www/html/key.php
04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey
</syntaxhighlight>
 
==== Get Current License Token from DB ====
 
<syntaxhighlight lang="sql">
SELECT content FROM voipmonitor.system WHERE type='license token';
</syntaxhighlight>
</syntaxhighlight>


= Utility Scripts =


== Scripts for Better Run Control ==
{| class="wikitable"
 
! Script !! Purpose !! Details
=== Alerts and Reports ===
|-
| Parallel alerts || Process alerts faster || Enable in '''Settings → System Configuration → Advanced → Number of parallel tasks'''
|-
| [[script-for-big-batchdownloads|Batch download]] || Download >1000 audio files || Bypasses GUI limit of 1000 CDRs
|-
| [[script-for-GUI-cron-control-oom|RAM control]] || Prevent OOM killer || Sets <code>oom_adj_score</code> for PHP threads
|-
| [[script-for-probes-backup|Probe backup]] || Backup all probe configs || Requires SSH access without password
|}


When alerts take longer time to process than the scheduled interval, it can cause out of memory or high CPU load because the same alert is processed multiple times for various time ranges.
= Debugging =


The GUI now supports parallel run of alerts/reports - enable it in '''Settings → System Configuration → Advanced → Number of parallel tasks'''.
== Charts Binary Crash ==
 
For deprecated manual approach, see: [[script-for-multi-Instances-alerts-processing]]
 
=== Batch Download for More Than 1000 CDRs ===
 
When you need to download a lot of audio files via '''GUI → Menu → CDR batch download''', it allows only 1000 calls to process with one query.
 
Following script allows you to create audio from a longer set of CDRs, also processing in multiple instances to get audio faster (beware of HDD IO/CPU): [[script-for-big-batchdownloads]]
 
=== RAM Control (Alerts / Dashboards Cache) ===
 
When there are many CDRs, alerts or caching of dashboard data can consume a lot of RAM. Following script controls if any PHP thread consumes over the limit. When it does, its <code>oom_adj_score</code> is set to be killed first (prevents killing of voipmonitor sniffer or DB when running on the same server).
 
See: [[script-for-GUI-cron-control-oom]]
 
=== Backup Settings from All Probes and GUI ===
 
The script fetches <code>voipmonitor.conf</code> from all probes and backs up GUI config tables from the database. Requires SSH access without password. Run via cron.
 
See: [[script-for-probes-backup]]
 
== Charts Binary Crash Debug Steps ==
 
When Apache log or dmesg reports crashes of the VM charts binary like:


When dmesg or Apache log shows charts crash:
<syntaxhighlight lang="text">
<syntaxhighlight lang="text">
Aug 20 10:32:30 C17VOIPMONDB02 kernel: charts-x86_64[7321]: segfault at 10 ip 0000000000412553 sp 00007ffff2638430 error 4 in charts-x86_64[400000+6c6000]
charts-x86_64[7321]: segfault at 10 ip 0000000000412553...
</syntaxhighlight>
</syntaxhighlight>


=== Create Debug Folder for Charts ===
'''Debug procedure:'''
 
Create <code>vm_charts_parameters</code> in <code>/tmp</code>:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Create debug folder
mkdir /tmp/vm_charts_parameters
mkdir /tmp/vm_charts_parameters
</syntaxhighlight>


After GUI creates some charts, files with arguments will be created there.
# After GUI creates charts, test with saved arguments
 
=== Test Arguments Passed to Charts Binary ===
 
<syntaxhighlight lang="bash">
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files
</syntaxhighlight>
</syntaxhighlight>


== VoIPmonitor Sniffer Memory Leaks Detection and Debug ==
== Memory Leak Detection (Jemalloc) ==
 
When voipmonitor uses more and more RAM (RSS/VSZ stat in syslog) without processing more calls (regs/messages), following steps can tell you where the RAM ends (it can be a leak or some thread overload).


=== Jemalloc ===
=== Compile Jemalloc ===
 
==== Compile Jemalloc ====
 
Download jemalloc from https://github.com/jemalloc/jemalloc


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./autogen.sh
# Requires development tools: apt install build-essential (Deb/Ubuntu), yum groupinstall 'Development Tools' (CentOS/RHEL)
./configure --enable-prof
./configure --enable-prof
make
make && make install
make install
</syntaxhighlight>
</syntaxhighlight>


==== Enable Jemalloc for VoIPmonitor and Compile It ====
=== Compile VoIPmonitor with Jemalloc ===
 
<syntaxhighlight lang="bash">
./configure
</syntaxhighlight>
 
Edit <code>config.h</code> to have:


Edit <code>config.h</code>:
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBTCMALLOC 0
Line 221: Line 129:
</syntaxhighlight>
</syntaxhighlight>


Edit <code>Makefile</code> and disable optimization, add ljemalloc to libs, remove -ltcmalloc:
Edit <code>Makefile</code>:
 
<syntaxhighlight lang="makefile">
<syntaxhighlight lang="makefile">
# Substitute -O2 with -O0 (capital O followed by zero)
# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc
LIBS = ${SHARED_LIBS} -ljemalloc
</syntaxhighlight>
</syntaxhighlight>
Finish compilation:
<syntaxhighlight lang="bash">
make clean
make -j8
</syntaxhighlight>
==== Edit /etc/init.d/voipmonitor ====
Ensure:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ARGS="-v 1"
make clean && make -j8
</syntaxhighlight>
</syntaxhighlight>


And in the start section (2 places) of the script:
=== Run with Jemalloc ===


Modify <code>/etc/init.d/voipmonitor</code> start section:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
</syntaxhighlight>
</syntaxhighlight>


==== Check Results ====
=== Generate Memory Report ===
 
At regular intervals you should check the results of memory usage.
 
===== je_prof.pdf =====
 
Don't forget to install graphviz and ghostscript:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
yum install graphviz ghostscript
# Install dependencies
# or: apt install graphviz ghostscript
apt install graphviz ghostscript # or: yum install graphviz ghostscript
</syntaxhighlight>
 
Generate the report:


<syntaxhighlight lang="bash">
# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf
</syntaxhighlight>
</syntaxhighlight>


===== sniffer_memory.txt =====
=== X88 Mode (Detailed Memory Debug) ===
 
<syntaxhighlight lang="bash">
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
</syntaxhighlight>
 
===== Script for Periodical Mail Sending =====
 
Add to <code>cron.hourly</code>:
 
<syntaxhighlight lang="bash">
#!/bin/bash
export PATH=/usr/local/bin:/usr/bin:/bin
mail="support@voipmonitor.org"
 
sniffer_bin="/usr/local/src/sniffer/voipmonitor"
manager_ip="127.0.0.1"
manager_port="5029"
 
tmp_je_prof_pdf="/tmp/je_prof.pdf"
tmp_je_prof_out="/tmp/je_prof"
tmp_pb_log="/tmp/pb_log"
tmp_sn_stat="/tmp/sn_stat"
 
echo 'jemalloc_stat_full' | nc $manager_ip $manager_port > $tmp_je_prof_out
jeprof --show_bytes --pdf $sniffer_bin $tmp_je_prof_out > $tmp_je_prof_pdf 2>/dev/null
echo 'packetbuffer_log' | nc $manager_ip $manager_port > $tmp_pb_log
echo 'sniffer_stats' | nc $manager_ip $manager_port > $tmp_sn_stat
 
# Send email with attachments
echo 'jeprof' | mail -s 'jeprof' -a $tmp_je_prof_pdf -a $tmp_pb_log -a $tmp_sn_stat $mail
 
rm $tmp_je_prof_out $tmp_je_prof_pdf $tmp_pb_log $tmp_sn_stat
</syntaxhighlight>
 
=== Enable X88 Mode ===
 
==== X88 Compatible Sources and Recompile ====
 
Copy sniffer sources with compiled binary to a new location and recreate binary with -X88 (necessary to get correct memory allocation for debugging using jeprof):


For more detailed memory allocation info:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
# It will list source files modification, then re-make
make clean && make
make clean
make
</syntaxhighlight>
</syntaxhighlight>


More detailed instructions: [[X88-memoryConsumptionDebug]]
Run with extra verbose options:
 
==== Modify Init File or Run Sniffer Manually ====
 
===== Edit /etc/init.d/voipmonitor =====
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ARGS="-v 1,force_log_sqlq,memory_stat,memory_stat_ignore_limit=100000"
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
# And in start section (2 places) of a script
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS
</syntaxhighlight>
 
===== Or Run from Console (when sniffer service stopped) =====
 
<syntaxhighlight lang="bash">
LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/src/sniffer-git-jemalloc-X88/voipmonitor \
   --config-file=/etc/voipmonitor.conf \
   --config-file=/etc/voipmonitor.conf \
   -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
   -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe
</syntaxhighlight>
</syntaxhighlight>


== Info About Internal Threads ==
See: [[X88-memoryConsumptionDebug]]
 
=== More Info About Threads in the Log ===
 
Add verbose option <code>threads_cpu_log</code> into command line (<code>-v1,threads_cpu_log</code>). If enabled, it prints the load of the threads to the syslog in the line output.
 
=== Basic Usage of Manager Command 'sniffer_threads' ===
 
<syntaxhighlight lang="bash">
echo 'sniffer_threads' | nc 127.0.0.1 5029
</syntaxhighlight>
 
Options:
* <code>no_sort</code> - do not sort by load in descending order
* <code>only_traffic</code> - lists only threads with recorded traffic (requires <code>sniffer_threads_ext</code>)
* <code>all</code> - lists all threads (even unloaded ones)
* <code>line</code> - use simple line output
 
=== Debugging Output of Enhanced Manager Command 'sniffer_threads' ===
 
Now includes:
* Traffic passing through each thread
* The percentage of states where the buffer between threads is full when a packet is attempted to be added to the buffer
* The percentage of usleep times when waiting for buffer space to become available, out of the total time


Required steps:
== Thread Debugging ==
# Compile the sniffer with <code>#define SNIFFER_THREADS_EXT true</code> (in <code>voipmonitor_define.h</code>)
# Set <code>sniffer_threads_ext=yes</code> in the configuration, or execute <code>echo 'setverbparam sniffer_threads_ext' | nc ...</code>


Output example:
{| class="wikitable"
 
! Method !! Usage
<syntaxhighlight lang="text">
|-
root@test:~# echo "sniffer_threads" | nc 127.0.0.1 5029
| Log thread CPU || Add <code>-v1,threads_cpu_log</code> to command line
t2 sip preprocess detach                      (  21939) :  0.0                    1 /    72837 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  1.39%    0.00%
|-
t2 sip preprocess sip                        (  21940) :  0.0                    1 /    69402 /   0.000                806p 8.70Mb ->   806p 8.70Mb  bf  0.01%    0.00%
| List threads || <code>echo 'sniffer_threads' | nc 127.0.0.1 5029</code>
t2 sip preprocess rtp                        (  21947) :  0.0                    2 /    69922 /  0.000                806p 8.70Mb ->    806p 8.70Mb  bf  0.01%    0.00%
|-
t2 rtp preprocess hash                        (  21952) :  0.0                  10 /    67685 /  0.000                521p 8.54Mb ->   --- p  --- b  bf  0.00%    0.00%
| Extended info || Set <code>sniffer_threads_ext=yes</code> in config (requires compile with <code>SNIFFER_THREADS_EXT true</code>)
t2 out thread defrag                          (  21954) :  0.0                    1 /   77085 /  0.000                823p 8.80Mb ->   815p 8.80Mb  bf  5.88%    0.00%
|}
pb - main queue                              (  21955) :  0.0                    0 /    26550 /  0.000                954p 10.2Mb ->    954p 10.2Mb
pb - write queue                              (  21956) :  0.0                    0 /    23481 /  0.000                954p 10.2Mb ->   878p 9.32Mb
pb - read thread eth1 read                    (  21957) :  0.0                  123 /     378 /  0.325                954p 10.2Mb ->   954p 10.2Mb
pb - read thread eth1 pcap_process            ( 21960) :  0.0                    0 /    57547 /  0.000                954p 10.2Mb ->   954p 10.2Mb
 
CPU ALL :  0.0
</syntaxhighlight>


== VoIPmonitor Sniffer Compile Notes: SS7 (Wireshark) Module ==
'''sniffer_threads options:'''
* <code>no_sort</code> - don't sort by load
* <code>only_traffic</code> - only threads with traffic
* <code>all</code> - include unloaded threads
* <code>line</code> - simple line output


Requires Wireshark version 3.2.
== SS7 Module Compilation ==


=== Modify config.h ===
Requires Wireshark 3.2.


Edit <code>config.h</code>:
<syntaxhighlight lang="c">
<syntaxhighlight lang="c">
#define HAVE_LIBWIRESHARK 1
#define HAVE_LIBWIRESHARK 1
Line 395: Line 203:
</syntaxhighlight>
</syntaxhighlight>


=== Modify Makefile ===
Edit <code>Makefile</code>:
 
Add to INCLUDE:
 
<syntaxhighlight lang="makefile">
<syntaxhighlight lang="makefile">
-I/usr/include/wireshark
INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0
</syntaxhighlight>
</syntaxhighlight>
Add to SHARED_LIBS:
<syntaxhighlight lang="makefile">
-lwiretap -lwireshark -lwsutil
</syntaxhighlight>
Internal build location: <code>JIRA:/opt/debian-7-64bit</code>
=== Disable Optimization in Makefile ===
Change <code>-O2</code> to <code>-O0</code>.
=== Compile ===


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
Line 421: Line 214:
</syntaxhighlight>
</syntaxhighlight>


== Testing Customer PCAPs ==
= PCAP Testing =


=== Testing Special PCAPs ===
== Basic Methods ==


==== IPFIX ====
{| class="wikitable"
! Method !! Command
|-
| Basic upload || <code>voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap</code>
|-
| Via packetbuffer (fast) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap</code>
|-
| Via packetbuffer (original time) || <code>voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap</code>
|}


Run first instance of voipmonitor that will act as receiver for packets and push CDRs to DB. In the settings enable IPFIX options:
== IPFIX Testing ==


'''Receiver instance:'''
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
ipfix = yes
ipfix = yes
Line 435: Line 237:
</syntaxhighlight>
</syntaxhighlight>


Then run second instance that will act as source to replay the IPFIX encapsulated PCAP:
'''Sender instance:'''
 
<syntaxhighlight lang="bash">
./voipmonitor --config-file=config/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/TCPdump_ipfix_stream.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
</syntaxhighlight>
 
Parameters:
* PCAP file path
* IPFIX client IP
* IPFIX server IP
* Destination IP:port (where to send the IPFIX stream - matching the first instance)
 
=== Testing via Basic Upload (GUI Upload) ===
 
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -r /tmp/file.pcap
</syntaxhighlight>
 
=== Testing via Packetbuffer Upload ===
 
==== Time Shift to Now, Increased Speed of Replaying ====
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpbsa9: /tmp/file.pcap
voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port
</syntaxhighlight>
</syntaxhighlight>


==== No Time Shift via Packetbuffer ====
== Tcpreplay ==
 
<syntaxhighlight lang="bash">
voipmonitor --config-file=/etc/voipmonitor-customer.conf -rpb: /tmp/file.pcap
</syntaxhighlight>
 
==== Tcpreplay to Running Service ====
 
Enable additional argument when starting the service in <code>/etc/init.d/voipmonitor</code>:


Enable in <code>/etc/init.d/voipmonitor</code>:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
ARGS="-v 1,dump_call_flags,tcpreplay"
ARGS="-v 1,dump_call_flags,tcpreplay"
</syntaxhighlight>
</syntaxhighlight>


Then use tcpreplay:
'''Add Layer 2 if missing''' (for captures done on "any" interface):
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
tcpreplay -i eth0 /tmp/l/x.pcap
tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
</syntaxhighlight>
 
If you get an error like:
 
<syntaxhighlight lang="text">
sending out eth0
processing file: /tmp/l/x.pcap
Warning in tcpreplay.c:replay_file() line 227:
/tmp/l/x.pcap DLT (LINUX_SLL) does not match that of the outbound interface: eth0 (EN10MB)
</syntaxhighlight>
 
This means the capture was done on "any" interface and layer 2 is missing. Follow steps below to add it.
 
===== Adding Layer 2 to PCAP =====
 
After capture was done on "any" interface and you need to replay it to a hardware interface, you need to add layer 2:
 
<syntaxhighlight lang="bash">
tcprewrite --dlt=enet -i /tmp/l/x.pcap -o /tmp/l/x_enet-encaps.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
   --infile=/tmp/l/x_enet-encaps.pcap --outfile=/tmp/l/x_enet-encaps-l2.pcap
   --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap
</syntaxhighlight>
 
===== Rewrite IPv4 Address in Whole PCAP =====
 
When you need to test alerts based on IP addresses (also country/continent alerts based on IPs), you can change IP addresses in PCAP and use it multiple times with various addresses:
 
<syntaxhighlight lang="bash">
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./7efd0e8b87a54f48ac04b7a49fdf5669.pcap -o ./7e-rew.pcap
</syntaxhighlight>
 
== Rename MySQL Database ==
 
It's not possible to do it via a direct command. You need to use '''RENAME TABLE''' command.
 
With this SELECT you generate the commands for renaming all tables in the database (replace <code>OLD_DB_NAME</code> and <code>NEW_DB_NAME</code>):
 
<syntaxhighlight lang="sql">
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,
    ' TO ','NEW_DB_NAME.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB_NAME';
</syntaxhighlight>
 
The output:
 
<syntaxhighlight lang="sql">
RENAME TABLE OLD_DB_NAME.table1 TO NEW_DB_NAME.table1;
RENAME TABLE OLD_DB_NAME.table2 TO NEW_DB_NAME.table2;
RENAME TABLE OLD_DB_NAME.tableX TO NEW_DB_NAME.tableX;
</syntaxhighlight>
 
Run these commands and you are ready.
 
'''Attention!''' Sometimes you can have a problem with moving foreign keys.
 
== Disabling TCP SACK ==
 
On buggy kernels (all before 2019/03), following prevents the SACK issue. Side effect is that on lossy lines (packet loss or big delays) the TCP traffic will increase:
 
<syntaxhighlight lang="bash">
echo "0" > /proc/sys/net/ipv4/tcp_sack
</syntaxhighlight>
 
== MacOS High Sierra in VirtualBox (on Windows) ==
 
=== Download Image ===
 
From: https://drive.google.com/file/d/0B0EsZGPuNAPvVDlicGt1NUx0Y0E/view
 
=== Prepare ISO for Install ===
 
==== Create High Sierra DMG Disk Image ====
 
<syntaxhighlight lang="bash">
hdiutil create -o /tmp/Highsierra -size 7900m -volname highsierra -layout SPUD -fs HFS+J
</syntaxhighlight>
 
==== Mount DMG Image to Your macOS ====
 
<syntaxhighlight lang="bash">
hdiutil attach /tmp/Highsierra.dmg -noverify -mountpoint /Volumes/Highsierra
</syntaxhighlight>
 
==== Create a High Sierra ISO Image ====
 
<syntaxhighlight lang="bash">
sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Highsierra
</syntaxhighlight>
 
==== Unmount Disk Image ====
 
<syntaxhighlight lang="bash">
hdiutil detach /volumes/Install\ macOS\ High\ Sierra
</syntaxhighlight>
 
==== Convert DMG to ISO ====
 
<syntaxhighlight lang="bash">
hdiutil convert /tmp/Highsierra.dmg -format UDTO -o ~/Desktop/Highsierra
</syntaxhighlight>
 
==== Rename and Move ISO Image to Desktop ====
 
<syntaxhighlight lang="bash">
mv ~/Desktop/Highsierra.cdr ~/Desktop/Highsierra.iso
</syntaxhighlight>
 
=== Create New Virtual Image ===
 
During pre-configure of new image set:
* Memory: 3-6 GB
* Disk size: 100 GB
 
Then edit its settings and additionally modify:
* System → Motherboard → Chipset: PIIX3
* System → Processor → Count: 2, Enable PAE/NX
* System → Storage → Attributes → Optical drive → SATA port 1, for the newly created ISO
* Display → Video memory → 128 MB
* Display → Graphics Controller → VMSVGA
 
=== Add the VirtualBox macOS Code to CMD ===
 
<syntaxhighlight lang="batch">
cd "C:\Program Files\Oracle\VirtualBox\"
VBoxManage.exe modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1
</syntaxhighlight>
</syntaxhighlight>


=== Install macOS on VirtualBox ===
'''Rewrite IP addresses:'''
 
# Open VirtualBox and turn on your High Sierra, then Open "Disk Utility".
# Format VirtualBox Virtual Disk with "Mac OS Extended" (GUID Partition Map).
# Install macOS High Sierra on new virtual disk with your ISO Image.
# After installation is completed, turn off your macOS High Sierra. Open VirtualBox and remove High Sierra.iso file.
# Turn on your Virtual Machine. At UEFI Shell screen, boot with this command:
 
<syntaxhighlight lang="text">
>"fs1:\macOS Install Data\Locked Files\Boot Files\boot.efi"
</syntaxhighlight>
 
=== Start the High Sierra Virtual Machine and Enjoy ===
 
== Measurement ==
 
=== Memory Speed ===
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
sysbench memory --threads=2 run
tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap
</syntaxhighlight>
</syntaxhighlight>


Should be over 5000 MiB/sec.
== SSL/TLS Testing ==
 
== SSL and Keylogger ==
 
=== How to Test It in VoIPmonitor ===


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
./voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode,ssl_sessionkey,dump_sip_line \
voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
   -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
   -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
   --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"},{"ssl_sessionkey_udp_maxwait_ms":10000},{"interface":"lo"},{"spooldir":"/var/spool/voipmonitorTLS"}]'
   --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'
</syntaxhighlight>
</syntaxhighlight>


=== How to Test It in Wireshark/Tshark ===
For TLS details, see [[Tls]].


==== Get the Keys from Keylogger's Dump ====
= Manager API =


<syntaxhighlight lang="bash">
== Via Manager Port (5029) ==
ngrep -I /tmp/keycapture.pcap | grep '^ ' | cut -d ' ' -f3- > /tmp/keys.txt
</syntaxhighlight>
 
==== Decrypt with Tshark ====


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
tshark -r '/tmp/tlscall.pcap' -o 'tls.keylog_file: /tmp/keys.txt' -o 'tls.debug_file: /dev/stdout'
echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stats' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029
</syntaxhighlight>
</syntaxhighlight>


==== Decrypt in Wireshark ====
== Via Server Port (60024) ==
 
In Edit → Preferences → Protocol → TLS → Pre-Master Secret log filename, choose the keys.txt file.
Then find the Client Hello HS packet, right-click on it and follow stream → TLS.
 
==== Check if We Have the Right Keys for RTP/DTLS ====
 
In RTP you have to find DTLS packets (<code>dtls.record.content_type == 22</code>), select 'client random' in them and find the 'Random' item.
 
That's e.g. <code>e54fc0f917cdd5c0d914f24956cdfd689022709f59ca4cd7fc20296db17039d8</code> in our sample. This must be included in the captured keys.
 
== Spooldir Testing ==
 
When there are problems with cleaning the spooldir or spooldir size does not respect the settings, you can use following command to test:
 
<syntaxhighlight lang="bash">
voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all
</syntaxhighlight>
 
== Manager API via Server Port ==
 
=== Get List of Connected Clients ===


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024
</syntaxhighlight>


=== Send Terminating Manager Command to a Probe ===
# Send command to specific probe
 
<syntaxhighlight lang="bash">
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024
</syntaxhighlight>
</syntaxhighlight>


== Additional Resources ==
= GUI Workarounds =
 
=== HA Proxy ===
[[sample HAproxy configuration]]
 
=== High CPS Config for VoIPmonitor with Hugepages ===
[[voipmonitor and hugepages]]
 
=== IOPS Tests ===
 
<code>--rw</code> can be <code>randread</code>, <code>randwrite</code>, <code>randrw</code>:
 
<syntaxhighlight lang="bash">
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --ramp_time=15s --norandommap \
  --filename=/root/testW.dat --output=/root/fio_db_write.txt
</syntaxhighlight>
 
=== Rocky 9 Installation ===
[[Rocky 9]]


=== Manager API of a Sniffer ===
== Disable Sensor Logs Popup ==
[[encryption_in_manager_api]]


=== Manager API for Customers ===
{{Note|There is no built-in GUI option to disable the sensor logs popup shown at admin login.}}
[[encryption_in_manager_api_customer]]
 
=== TLS Decode by Ribbon SBC 7k ===
[[Ribbon7k monitoring profiles]]
 
=== Email Templates ===
Common requests in support: [[common body of support emails]]
 
=== IPACCOUNT ===
How to enable [[ipaccount]] feature in voipmonitor.
 
=== DTLS ===
[[DTLS]] SRTP decryption using voipmonitor.
 
== GUI: Sensor Logs Popup Workaround ===
 
=== Issue ===
 
There is no built-in PHP or GUI option to disable the 'sensor logs' popup that appears for administrators at login. This popup is shown when there are unconfirmed sensor status messages in the <code>log_sensor</code> table.
 
=== Workaround 1: Periodic SQL Update ===
 
Run this SQL command periodically (e.g., via cron) to mark all existing sensor messages as confirmed:


'''Option 1: Periodic SQL update'''
<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
UPDATE voipmonitor.log_sensor SET confirmed = 1;
UPDATE voipmonitor.log_sensor SET confirmed = 1;
</syntaxhighlight>
</syntaxhighlight>


This will eliminate the popup but requires recurrent manual execution.
'''Option 2: Database trigger (permanent)'''
 
=== Workaround 2: Database Trigger (Permanent Solution) ===
 
Create a database trigger that automatically sets any new sensor log entry to 'confirmed' status upon insertion. This prevents the popup from ever being triggered by new messages:
 
<syntaxhighlight lang="sql">
<syntaxhighlight lang="sql">
DELIMITER $$
DELIMITER $$
Line 764: Line 316:
</syntaxhighlight>
</syntaxhighlight>


This provides a permanent solution but note that you will not receive any sensor notification popups.
== IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4) ==


=== Important Notes ===
'''Symptom:''' ldap.so crash at login page (CVE-2024-8932).
 
* These are database-level manual interventions, not GUI configuration options
* The <code>confirmed</code> field in <code>log_sensor</code> table controls whether the popup is shown
* Popup appears when there are rows with <code>confirmed = 0</code> (or NULL)
* For permanent solution, prefer the database trigger approach
 
== Remote Support Access via Client VPN ==
 
=== VPN Connection Issues: IP Whitelisting ===
 
When support staff cannot connect to a client's VPN (Fortinet or other enterprise firewall), this is typically due to the support staff IP address not being whitelisted on the client-side firewall configuration.
 
=== Standard Procedure ===
 
1. **Identify the issue**: Connection attempt fails without prompting for OTP/token, indicating firewall-level blocking.
 
2. **Action required by the client**:
  * Obtain the public IP address of the support staff member requiring VPN access
  * Add this IP address to the whitelist on their Fortinet VPN or firewall configuration
  * This is '''client-side network administration''' - not a VoIPmonitor configuration issue
 
3. **After whitelisting**: Support staff should attempt VPN connection again
 
=== Important Notes ===
 
* This is a general networking/firewall administration task specific to each customer's security policies
* VoIPmonitor support team coordinates with customers but does not have direct access to customer firewall configurations
* Guide the customer to consult their network/firewall administrator for IP whitelisting procedures
* Do not confuse with FortiToken 2FA - that is VoIPmonitor GUI authentication, different from VPN network access
 
== IonCube Crashes for Apache/HTTPS ==
 
=== Wrong Privileges ===
 
If following command crashes under www-data user but not under root:


'''Fix:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data
# Backup
</syntaxhighlight>
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP


It can be caused by wrong permissions for the ioncube.ini file or ioncube module. You can test with strace (search for "ioncube" or "denied"):
# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb


<syntaxhighlight lang="bash">
# Install and restart
strace -f su -s '/bin/bash' -c "php -r 'echo extension_loaded(\"ionCube Loader\")?\"yes\":\"no\";'" www-data 2> /tmp/text.txt
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2
</syntaxhighlight>
</syntaxhighlight>


=== Crashes in libc (Ubuntu 22.04 with PHP 7.4) ===
= Database Operations =


It was experienced in Ubuntu 22.04 with PHP 7.4 that ioncube causes crash in ldap.so (displays following error on the login page):
== Rename MySQL Database ==


<syntaxhighlight lang="text">
Generate RENAME TABLE commands:
Failed check Ioncube.com PHP Loader for php cli : PHP Warning: PHP Startup: Unable to load dynamic library 'ldap.so' (tried: /usr/lib/php/20190902/ldap.so (/usr/lib/php/20190902/ldap.so: undefined symbol: RETURN_THROWS), /usr/lib/php/20190902/ldap.so.so (/usr/lib/php/20190902/ldap.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
<syntaxhighlight lang="sql">
SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';
</syntaxhighlight>
</syntaxhighlight>


The reason is a broken ldap library in Ubuntu 22.04:
{{Warning|1=Foreign keys may cause issues during rename.}}
* https://ubuntu.com/security/CVE-2024-8932
* https://askubuntu.com/questions/1535405/php7-4-ldap-broken-on-focal-after-13-dec-2024


==== Fix ====
== Spooldir Testing ==
 
Backup current ldap libs:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cd /usr/lib/php/20190902
voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all
mkdir BKUP
cp -pv ldap.so* BKUP
</syntaxhighlight>
</syntaxhighlight>


Download and extract the package:
= Remote Support Access =
 
== VPN Connection Issues ==
 
When support staff cannot connect to client VPN (Fortinet, etc.):


<syntaxhighlight lang="bash">
# '''Cause:''' Support IP not whitelisted on client firewall
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
# '''Solution:''' Client must add support staff's public IP to their VPN/firewall whitelist
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
# '''Note:''' This is client-side network administration, not a VoIPmonitor issue
</syntaxhighlight>


Copy in the fixed lib:
{{Note|1=Don't confuse VPN network access with FortiToken 2FA (GUI authentication).}}


<syntaxhighlight lang="bash">
= Additional Resources =
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
</syntaxhighlight>


Restart Apache and test:
{| class="wikitable"
! Topic !! Link
|-
| HAProxy config || [[sample HAproxy configuration]]
|-
| Hugepages for high CPS || [[voipmonitor and hugepages]]
|-
| Manager API encryption || [[encryption_in_manager_api]]
|-
| Ribbon SBC 7k TLS || [[Ribbon7k monitoring profiles]]
|-
| Support email templates || [[common body of support emails]]
|-
| IPACCOUNT feature || [[ipaccount]]
|-
| DTLS decryption || [[DTLS]]
|-
| Rocky 9 installation || [[Rocky 9]]
|}


'''IOPS testing:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart apache2
fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat
</syntaxhighlight>
</syntaxhighlight>


''Thanks to Andrew from distributel.ca for this article.''
= AI Summary for RAG =
 
== AI Summary for RAG ==


'''Summary:''' Internal support documentation for VoIPmonitor team. Covers testing configuration options using <code>--json_config</code> to override settings without modifying config files. Includes testing procedures for regexp patterns, country assignment (by IP or phone number), and billing/pricing. License management hints include checking channel usage (<code>checkLicense -v</code>), updating license via cron, and license portability notes (one license can be used at multiple physical sites). Scripts section covers parallel alerts processing, batch downloads for >1000 CDRs, RAM control for PHP threads, and probe backup automation. Debugging sections include charts binary crash analysis, memory leak detection using jemalloc (compile instructions, jeprof PDF generation, X88 mode), internal thread monitoring (<code>sniffer_threads</code> command), and SS7/Wireshark module compilation. PCAP testing covers IPFIX emulation, basic upload, packetbuffer replay, tcpreplay with layer 2 addition, and IP address rewriting. GUI workaround section addresses disabling sensor logs popup: no built-in option exists, but two database-level workarounds are available - periodic SQL update (<code>UPDATE voipmonitor.log_sensor SET confirmed = 1;</code>) or permanent solution using database trigger to auto-set <code>confirmed = 1</code> on INSERT. Remote Support section covers VPN connection issues: when support staff cannot connect to client VPN, the typical resolution is IP whitelisting on client-side firewall (Fortinet or other). This is network administration, not VoIPmonitor configuration. Additional topics: MySQL database rename procedure, TCP SACK disabling for buggy kernels, macOS High Sierra in VirtualBox setup, SSL keylogger testing, spooldir troubleshooting, Manager API commands via server port, and IonCube crash fixes for Ubuntu 22.04 PHP 7.4 ldap.so issue.
'''Summary:''' Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with <code>--json_config</code> to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (<code>checkLicense</code>, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via <code>sniffer_threads</code> command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).


'''Keywords:''' internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, MySQL rename, TCP SACK, VirtualBox macOS, SSL keylogger, spooldir, Manager API, IonCube, ldap.so crash, Ubuntu 22.04, VPN access, IP whitelisting, Fortinet VPN, remote support, firewall, sensor logs popup, GUI popup, log_sensor, database trigger, confirmed field
'''Keywords:''' internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger


'''Key Questions:'''
'''Key Questions:'''
* How to test configuration options without modifying voipmonitor.conf?
* How to test configuration without modifying voipmonitor.conf?
* How to use json_config to override sniffer settings?
* How to test regexp patterns or country assignment?
* How to test regexp patterns in voipmonitor?
* How to check license channel usage?
* How to find country by IP or phone number?
* How to debug memory leaks using jemalloc?
* How to test billing/pricing configuration?
* How to compile voipmonitor with jemalloc or SS7 support?
* How to check license channel usage details?
* How to generate jeprof PDF memory report?
* How to keep license updated automatically via cron?
* How to monitor thread load with sniffer_threads?
* Can one VoIPmonitor license be used at multiple physical sites?
* How to test IPFIX or replay PCAPs?
* How to debug memory leaks in voipmonitor using jemalloc?
* How to compile voipmonitor with jemalloc support?
* How to generate jeprof PDF memory analysis report?
* What is X88 mode and how to enable it for memory debugging?
* How to monitor internal thread load using sniffer_threads command?
* How to compile voipmonitor with SS7/Wireshark support?
* How to test IPFIX encapsulated PCAPs?
* How to replay PCAPs using tcpreplay?
* How to add layer 2 to PCAP for tcpreplay?
* How to add layer 2 to PCAP for tcpreplay?
* How to rewrite IP addresses in PCAP files?
* How to disable sensor logs popup in GUI?
* How to rename MySQL database in voipmonitor?
* How to disable the sensor logs popup in VoIPmonitor GUI?
* Is there a GUI option to disable sensor logs popup at login?
* How to auto-confirm sensor logs to prevent popup notifications?
* What to do when support staff cannot connect to client VPN?
* How to resolve VPN connection issues via IP whitelisting?
* What is the difference between FortiToken 2FA and Fortinet VPN access?
* How to fix IonCube ldap.so crash in Ubuntu 22.04?
* How to fix IonCube ldap.so crash in Ubuntu 22.04?
* How to resolve VPN connection issues for remote support?

Revision as of 16:49, 8 January 2026


⚠️ Warning: This documentation is for the internal support team. Use at your own risk and test on non-production servers first.

Quick Reference

Testing Configuration

Override Settings with json_config

Override config file and database settings without editing files: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'

💡 Tip: Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.

Regexp / Country / Billing Tests

Test Type Command
Regexp pattern voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'
Country by IP voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113
Country by number voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345
Billing test voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing

Billing test file format (CSV): 

# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8

License Management

Task Command
Write channel stats to DB php /var/www/html/php/run.php saveCallStatistics
Check license details php /var/www/html/php/run.php checkLicense -v
Check max peak only php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
Get token from DB SELECT content FROM voipmonitor.system WHERE type='license token';
View license file cat /var/www/html/key.php

Auto-update license via cron (without processing alerts/reports): 

04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey

Utility Scripts

Script Purpose Details
Parallel alerts Process alerts faster Enable in Settings → System Configuration → Advanced → Number of parallel tasks
Batch download Download >1000 audio files Bypasses GUI limit of 1000 CDRs
RAM control Prevent OOM killer Sets oom_adj_score for PHP threads
Probe backup Backup all probe configs Requires SSH access without password

Debugging

Charts Binary Crash

When dmesg or Apache log shows charts crash: 

charts-x86_64[7321]: segfault at 10 ip 0000000000412553...

Debug procedure: 

# Create debug folder
mkdir /tmp/vm_charts_parameters

# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

Memory Leak Detection (Jemalloc)

Compile Jemalloc

# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./configure --enable-prof
make && make install

Compile VoIPmonitor with Jemalloc

Edit config.h: 

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

Edit Makefile: 

# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc

make clean && make -j8

Run with Jemalloc

Modify /etc/init.d/voipmonitor start section: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

Generate Memory Report

# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript

# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf

X88 Mode (Detailed Memory Debug)

For more detailed memory allocation info: 

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make

Run with extra verbose options: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
  --config-file=/etc/voipmonitor.conf \
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

See: X88-memoryConsumptionDebug

Thread Debugging

Method Usage
Log thread CPU Add -v1,threads_cpu_log to command line
List threads nc 127.0.0.1 5029
Extended info Set sniffer_threads_ext=yes in config (requires compile with SNIFFER_THREADS_EXT true)

sniffer_threads options:

  • no_sort - don't sort by load
  • only_traffic - only threads with traffic
  • all - include unloaded threads
  • line - simple line output

SS7 Module Compilation

Requires Wireshark 3.2.

Edit config.h: 

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

Edit Makefile: 

INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0

./configure && make clean && make ss7

PCAP Testing

Basic Methods

Method Command
Basic upload voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap
Via packetbuffer (fast) voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap
Via packetbuffer (original time) voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap

IPFIX Testing

Receiver instance: 

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Sender instance: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port

Tcpreplay

Enable in /etc/init.d/voipmonitor: 

ARGS="-v 1,dump_call_flags,tcpreplay"

Add Layer 2 if missing (for captures done on "any" interface): 

tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap

Rewrite IP addresses: 

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap

SSL/TLS Testing

voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'

For TLS details, see Tls.

Manager API

Via Manager Port (5029)

echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stats' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

Via Server Port (60024)

# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024

# Send command to specific probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024

GUI Workarounds

Disable Sensor Logs Popup

ℹ️ Note: There is no built-in GUI option to disable the sensor logs popup shown at admin login.

Option 1: Periodic SQL update 

UPDATE voipmonitor.log_sensor SET confirmed = 1;

Option 2: Database trigger (permanent) 

DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;

IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4)

Symptom: ldap.so crash at login page (CVE-2024-8932).

Fix: 

# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP

# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

# Install and restart
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2

Database Operations

Rename MySQL Database

Generate RENAME TABLE commands: 

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';

⚠️ Warning: Foreign keys may cause issues during rename.

Spooldir Testing

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Remote Support Access

VPN Connection Issues

When support staff cannot connect to client VPN (Fortinet, etc.):

  1. Cause: Support IP not whitelisted on client firewall
  2. Solution: Client must add support staff's public IP to their VPN/firewall whitelist
  3. Note: This is client-side network administration, not a VoIPmonitor issue

ℹ️ Note: Don't confuse VPN network access with FortiToken 2FA (GUI authentication).

Additional Resources

Topic Link
HAProxy config sample HAproxy configuration
Hugepages for high CPS voipmonitor and hugepages
Manager API encryption encryption_in_manager_api
Ribbon SBC 7k TLS Ribbon7k monitoring profiles
Support email templates common body of support emails
IPACCOUNT feature ipaccount
DTLS decryption DTLS
Rocky 9 installation Rocky 9

IOPS testing: 

fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat

AI Summary for RAG

Summary: Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with --json_config to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (checkLicense, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via sniffer_threads command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).

Keywords: internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger

Key Questions:

  • How to test configuration without modifying voipmonitor.conf?
  • How to test regexp patterns or country assignment?
  • How to check license channel usage?
  • How to debug memory leaks using jemalloc?
  • How to compile voipmonitor with jemalloc or SS7 support?
  • How to generate jeprof PDF memory report?
  • How to monitor thread load with sniffer_threads?
  • How to test IPFIX or replay PCAPs?
  • How to add layer 2 to PCAP for tcpreplay?
  • How to disable sensor logs popup in GUI?
  • How to fix IonCube ldap.so crash in Ubuntu 22.04?
  • How to resolve VPN connection issues for remote support?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Internal_support_hints&oldid=10138"