Internal support hints

From VoIPmonitor.org
Revision as of 16:38, 9 January 2026 by Festr (talk | contribs)


⚠️ Warning: This documentation is for the internal support team. Use at your own risk and test on non-production servers first.

Quick Reference

Testing Configuration

Override Settings with json_config

Override config file and database settings without editing files: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'

💡 Tip: Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.

Regexp / Country / Billing Tests

Test Type Command
Regexp pattern voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'
Country by IP voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113
Country by number voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345
Billing test voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing

Billing test file format (CSV): 

# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8

License Management

Task Command
Write channel stats to DB php /var/www/html/php/run.php saveCallStatistics
Check license details php /var/www/html/php/run.php checkLicense -v
Check max peak only php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
Get token from DB SELECT content FROM voipmonitor.system WHERE type='license token';
View license file cat /var/www/html/key.php

Auto-update license via cron (without processing alerts/reports): 

04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey

Utility Scripts

Script Purpose Details
Parallel alerts Process alerts faster Enable in Settings → System Configuration → Advanced → Number of parallel tasks
Batch download Download >1000 audio files Bypasses GUI limit of 1000 CDRs
RAM control Prevent OOM killer Sets oom_adj_score for PHP threads
Probe backup Backup all probe configs Requires SSH access without password

Debugging

Charts Binary Crash

When dmesg or Apache log shows charts crash: 

charts-x86_64[7321]: segfault at 10 ip 0000000000412553...

Debug procedure: 

# Create debug folder
mkdir /tmp/vm_charts_parameters

# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

Memory Leak Detection (Jemalloc)

Compile Jemalloc

# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./configure --enable-prof
make && make install

Compile VoIPmonitor with Jemalloc

Edit config.h: 

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

Edit Makefile: 

# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc

make clean && make -j8

Run with Jemalloc

Modify /etc/init.d/voipmonitor start section: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

Generate Memory Report

# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript

# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf

X88 Mode (Detailed Memory Debug)

For more detailed memory allocation info: 

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make

Run with extra verbose options: 

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
  --config-file=/etc/voipmonitor.conf \
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

See: X88-memoryConsumptionDebug

Thread Debugging

Method Usage
Log thread CPU Add -v1,threads_cpu_log to command line
List threads nc 127.0.0.1 5029
Extended info Set sniffer_threads_ext=yes in config (requires compile with SNIFFER_THREADS_EXT true)

sniffer_threads options:

  • no_sort - don't sort by load
  • only_traffic - only threads with traffic
  • all - include unloaded threads
  • line - simple line output

SS7 Module Compilation

Requires Wireshark 3.2.

Edit config.h: 

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

Edit Makefile: 

INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0

./configure && make clean && make ss7

PCAP Testing

Basic Methods

Method Command
Basic upload voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap
Via packetbuffer (fast) voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap
Via packetbuffer (original time) voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap

IPFIX Testing

Receiver instance: 

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Sender instance: 

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port

Tcpreplay

Enable in /etc/init.d/voipmonitor: 

ARGS="-v 1,dump_call_flags,tcpreplay"

Add Layer 2 if missing (for captures done on "any" interface): 

tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap

Rewrite IP addresses: 

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap

SSL/TLS Testing

voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'

For TLS details, see Tls.

Manager API

Via Manager Port (5029)

echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stats' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

Via Server Port (60024)

# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024

# Send command to specific probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024

GUI Workarounds

Disable Sensor Logs Popup

ℹ️ Note: There is no built-in GUI option to disable the sensor logs popup shown at admin login.

Option 1: Periodic SQL update 

UPDATE voipmonitor.log_sensor SET confirmed = 1;

Option 2: Database trigger (permanent) 

DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;

IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4)

Symptom: ldap.so crash at login page (CVE-2024-8932).

Fix: 

# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP

# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

# Install and restart
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2

Database Operations

Rename MySQL Database

Generate RENAME TABLE commands: 

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';

⚠️ Warning: Foreign keys may cause issues during rename.

Spooldir Testing

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Remote Support Access

VPN Connection Issues

When support staff cannot connect to client VPN (Fortinet, etc.):

  1. Cause: Support IP not whitelisted on client firewall
  2. Solution: Client must add support staff's public IP to their VPN/firewall whitelist
  3. Note: This is client-side network administration, not a VoIPmonitor issue

ℹ️ Note: Don't confuse VPN network access with FortiToken 2FA (GUI authentication).

System Troubleshooting

TCP SACK (Buggy Kernels pre-2019/03)

echo "0" > /proc/sys/net/ipv4/tcp_sack

Memory Speed Test

sysbench memory --threads=2 run
# Should be >5000 MiB/sec

Additional Resources

Topic Link
HAProxy config sample HAproxy configuration
Hugepages for high CPS voipmonitor and hugepages
Manager API encryption encryption_in_manager_api
Ribbon SBC 7k TLS Ribbon7k monitoring profiles
Support email templates common body of support emails
IPACCOUNT feature ipaccount
DTLS decryption DTLS
Rocky 9 installation Rocky 9

IOPS testing: 

fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat

AI Summary for RAG

Summary: Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with --json_config to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (checkLicense, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via sniffer_threads command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).

Keywords: internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed

Key Questions:

  • How to test configuration without modifying voipmonitor.conf?
  • How to test regexp patterns or country assignment?
  • How to check license channel usage?
  • How to debug memory leaks using jemalloc?
  • How to compile voipmonitor with jemalloc or SS7 support?
  • How to generate jeprof PDF memory report?
  • How to monitor thread load with sniffer_threads?
  • How to test IPFIX or replay PCAPs?
  • How to add layer 2 to PCAP for tcpreplay?
  • How to disable sensor logs popup in GUI?
  • How to fix IonCube ldap.so crash in Ubuntu 22.04?
  • How to resolve VPN connection issues for remote support?
  • How to disable TCP SACK on buggy kernels?
  • How to test memory speed with sysbench?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Internal_support_hints&oldid=10381"