Internal support hints

⚠️ Warning: This documentation is for the internal support team. Use at your own risk and test on non-production servers first.

Quick Reference

Testing Configuration

Override Settings with json_config

Override config file and database settings without editing files:

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --json_config='[{"id_sensor":"11"},{"interface":"lo"},{"natalias":"8.8.8.8 10.10.100.50"}]'

💡 Tip: Useful for quick testing of specific parameters. Options in json_config take precedence over config file and DB settings.

Regexp / Country / Billing Tests

Test Type	Command
Regexp pattern	voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-regexp '^800...[1-9][0-9]'
Country by IP	voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-ip=102.140.68.113
Country by number	voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --find-country-for-number 0034*12345
Billing test	voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 --test-billing test_data/billing

Billing test file format (CSV):

# calldate,connect_duration,caller,called,sipcallerip,sipcalledip,[exp_price_op],[exp_price_cust]
2018-05-09 12:00,120,+4121353333,+41792826527,192.168.101.10,192.168.101.151,800,8

License Management

Task	Command
Write channel stats to DB	php /var/www/html/php/run.php saveCallStatistics
Check license details	php /var/www/html/php/run.php checkLicense -v
Check max peak only	php /var/www/html/php/run.php checkLicense -v | grep max_peak -A3 -B1
Get token from DB	SELECT content FROM voipmonitor.system WHERE type='license token';
View license file	cat /var/www/html/key.php

Auto-update license via cron (without processing alerts/reports):

04 04 * * * root php /var/www/html/php/run.php runUpdateLicenseKey

Utility Scripts

Script	Purpose	Details
Parallel alerts	Process alerts faster	Enable in Settings → System Configuration → Advanced → Number of parallel tasks
Batch download	Download >1000 audio files	Bypasses GUI limit of 1000 CDRs
RAM control	Prevent OOM killer	Sets oom_adj_score for PHP threads
Probe backup	Backup all probe configs	Requires SSH access without password

Debugging

Charts Binary Crash

When dmesg or Apache log shows charts crash:

charts-x86_64[7321]: segfault at 10 ip 0000000000412553...

Debug procedure:

# Create debug folder
mkdir /tmp/vm_charts_parameters

# After GUI creates charts, test with saved arguments
/var/www/html/bin/charts-x86_64 -i /tmp/vm_charts_parameters/files

Memory Leak Detection (Jemalloc)

Compile Jemalloc

# Download from https://github.com/jemalloc/jemalloc
./autogen.sh
./configure --enable-prof
make && make install

Compile VoIPmonitor with Jemalloc

Edit config.h:

#define HAVE_LIBTCMALLOC 0
#define HAVE_LIBJEMALLOC 1

Edit Makefile:

# Change -O2 to -O0 (disable optimization)
LIBS = ${SHARED_LIBS} -ljemalloc

make clean && make -j8

Run with Jemalloc

Modify /etc/init.d/voipmonitor start section:

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' /usr/local/src/voipmonitor --config-file $CONFIGFILE --pid-file $PIDFILE $ARGS

Generate Memory Report

# Install dependencies
apt install graphviz ghostscript  # or: yum install graphviz ghostscript

# Generate PDF report
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029 > /tmp/je_prof
jeprof --show_bytes --pdf /usr/local/src/sniffer/voipmonitor /tmp/je_prof > je_prof.pdf

X88 Mode (Detailed Memory Debug)

For more detailed memory allocation info:

cp -a /usr/src/voipmonitor-git /usr/src/sniffer-git-jemalloc-X88
LD_PRELOAD=/usr/local/lib/libjemalloc.so ./voipmonitor -k -v1 -c -X88
make clean && make

Run with extra verbose options:

LD_PRELOAD=/usr/local/lib/libjemalloc.so MALLOC_CONF='prof:true' ./voipmonitor \
  --config-file=/etc/voipmonitor.conf \
  -v 1,force_log_sqlq,memory_stat_ex,memory_stat_ignore_limit=100000,heapsafe

See: X88-memoryConsumptionDebug

Thread Debugging

Method	Usage
Log thread CPU	Add -v1,threads_cpu_log to command line
List threads	nc 127.0.0.1 5029
Extended info	Set sniffer_threads_ext=yes in config (requires compile with SNIFFER_THREADS_EXT true)

sniffer_threads options:

• no_sort - don't sort by load
• only_traffic - only threads with traffic
• all - include unloaded threads
• line - simple line output

SS7 Module Compilation

Requires Wireshark 3.2.

Edit config.h:

#define HAVE_LIBWIRESHARK 1
#define LIBWIRESHARK_VERSION 30200

Edit Makefile:

INCLUDE += -I/usr/include/wireshark
SHARED_LIBS += -lwiretap -lwireshark -lwsutil
# Change -O2 to -O0

./configure && make clean && make ss7

PCAP Testing

Basic Methods

Method	Command
Basic upload	voipmonitor --config-file=/etc/voipmonitor.conf -r /tmp/file.pcap
Via packetbuffer (fast)	voipmonitor --config-file=/etc/voipmonitor.conf -rpbsa9: /tmp/file.pcap
Via packetbuffer (original time)	voipmonitor --config-file=/etc/voipmonitor.conf -rpb: /tmp/file.pcap

IPFIX Testing

Receiver instance:

ipfix = yes
ipfix_bind_ip = 0.0.0.0
ipfix_bind_port = 12345

Sender instance:

voipmonitor --config-file=/etc/voipmonitor.conf -k -v1 \
  --ipfix-client-emulation='/tmp/ipfix.pcap;1.2.3.4;8.8.8.8;127.0.0.1;12345'
# Parameters: pcap_file;client_ip;server_ip;dest_ip;dest_port

Tcpreplay

Enable in /etc/init.d/voipmonitor:

ARGS="-v 1,dump_call_flags,tcpreplay"

Add Layer 2 if missing (for captures done on "any" interface):

tcprewrite --dlt=enet -i /tmp/x.pcap -o /tmp/x_enet.pcap
tcprewrite --enet-dmac=00:55:22:AF:C6:37 --enet-smac=00:44:66:FC:29:AF \
  --infile=/tmp/x_enet.pcap --outfile=/tmp/x_l2.pcap

Rewrite IP addresses:

tcprewrite --pnat=8.7.6.5/32:1.2.3.4/32 -i ./original.pcap -o ./rewritten.pcap

SSL/TLS Testing

voipmonitor --config-file /etc/test.conf -k -v1,pcap_stat_period=1,_ssl,_tls,_ssldecode \
  -rpba:/tmp/keys.pcap@@/tmp/call.pcap \
  --json_config='[{"sipport":"8089"},{"ssl":"yes"},{"ssl_ipport":"192.168.0.1:8089"},{"ssl_sessionkey_udp":"yes"},{"ssl_sessionkey_udp_port":"1234"}]'

For TLS details, see Tls.

Manager API

Via Manager Port (5029)

echo 'listcalls' | nc 127.0.0.1 5029
echo 'sniffer_stats' | nc 127.0.0.1 5029
echo 'jemalloc_stat_full' | nc 127.0.0.1 5029

Via Server Port (60024)

# List connected clients
echo '{"type_connection":"manager_command","command":"active"}' | nc 127.0.0.1 60024

# Send command to specific probe
echo '{"type_connection":"gui_command","sensor_id":1011,"command":"terminating"}' | nc 127.0.0.1 60024

GUI Workarounds

Disable Sensor Logs Popup

ℹ️ Note: There is no built-in GUI option to disable the sensor logs popup shown at admin login.

Option 1: Periodic SQL update

UPDATE voipmonitor.log_sensor SET confirmed = 1;

Option 2: Database trigger (permanent)

DELIMITER $$
CREATE TRIGGER before_insert_log_sensor
BEFORE INSERT ON log_sensor
FOR EACH ROW
BEGIN
    SET NEW.confirmed = 1;
END$$
DELIMITER ;

IonCube Crash Fix (Ubuntu 22.04 + PHP 7.4)

Symptom: ldap.so crash at login page (CVE-2024-8932).

Fix:

# Backup
cd /usr/lib/php/20190902 && mkdir BKUP && cp -pv ldap.so* BKUP

# Download and extract fixed package
wget https://pl.archive.ubuntu.com/ubuntu/pool/main/p/php7.4/php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb
ar x php7.4-ldap_7.4.3-4ubuntu2.24_amd64.deb

# Install and restart
cp -v ./usr/lib/php/20190902/ldap.so /usr/lib/php/20190902/ldap.so
systemctl restart apache2

Database Operations

Rename MySQL Database

Generate RENAME TABLE commands:

SELECT CONCAT('RENAME TABLE ',table_schema,'.',table_name,' TO NEW_DB.',table_name,';')
FROM information_schema.TABLES
WHERE table_schema LIKE 'OLD_DB';

⚠️ Warning: Foreign keys may cause issues during rename.

Spooldir Testing

voipmonitor --config-file /etc/voipmonitor.conf -k -v1 --test-cleanspool-load all

Remote Support Access

VPN Connection Issues

When support staff cannot connect to client VPN (Fortinet, etc.):

1. Cause: Support IP not whitelisted on client firewall
2. Solution: Client must add support staff's public IP to their VPN/firewall whitelist
3. Note: This is client-side network administration, not a VoIPmonitor issue

ℹ️ Note: Don't confuse VPN network access with FortiToken 2FA (GUI authentication).

System Troubleshooting

TCP SACK (Buggy Kernels pre-2019/03)

echo "0" > /proc/sys/net/ipv4/tcp_sack

Memory Speed Test

sysbench memory --threads=2 run
# Should be >5000 MiB/sec

Additional Resources

Topic	Link
HAProxy config	sample HAproxy configuration
Hugepages for high CPS	voipmonitor and hugepages
Manager API encryption	encryption_in_manager_api
Ribbon SBC 7k TLS	Ribbon7k monitoring profiles
Support email templates	common body of support emails
IPACCOUNT feature	ipaccount
DTLS decryption	DTLS
Rocky 9 installation	Rocky 9

IOPS testing:

fio --name=rootTest --ioengine=libaio --rw=randwrite --bs=8k --numjobs=16 \
  --size=1G --runtime=600s --time_based --filename=/root/testW.dat

sniffer_stat Command Output Format

The sniffer_stat command (singular) returns a JSON response with sensor status information:

echo 'sniffer_stat' | nc 127.0.0.1 5029

Important output fields:

Field	Description
pbStatString	Contains SQL queue information when queue has items (e.g., SQLq[C1:145 / 0.059s / 2q/s])
version	Sensor version
active_calls	Number of active calls

ℹ️ Note: The SQLq value in pbStatString only appears when the queue has items. If the queue is empty, this specific string will not be present in the output.

Example output:

{
  "version": "2025.01.08-SVN.76",
  "active_calls": 42,
  "pbStatString": "SQLq[C:145 / 0.059s / 2q/s]",
  ...
}

AI Summary for RAG

Summary: Internal support documentation for VoIPmonitor team covering testing, debugging, and troubleshooting procedures. Key topics: (1) Testing configuration with --json_config to override settings without editing files; (2) Testing regexp, country assignment, and billing; (3) License management commands (checkLicense, cron updates); (4) Debugging tools including jemalloc for memory leaks, X88 mode for detailed allocation, thread monitoring via sniffer_threads command; (5) PCAP testing methods (basic upload, packetbuffer, IPFIX emulation, tcpreplay with L2 addition and IP rewriting); (6) SS7/Wireshark module compilation; (7) Manager API commands on ports 5029 and 60024; (8) GUI workarounds including sensor logs popup disable (database trigger solution); (9) IonCube ldap.so crash fix for Ubuntu 22.04 + PHP 7.4; (10) VPN access troubleshooting (IP whitelisting on client firewall).

Keywords: internal support, json_config, testing, regexp, country assignment, billing test, license, checkLicense, jemalloc, memory leak, X88, sniffer_threads, SS7, wireshark, IPFIX, tcpreplay, pcap testing, Manager API, IonCube, ldap.so, Ubuntu 22.04, VPN access, IP whitelisting, sensor logs popup, log_sensor, database trigger, tcp_sack, sysbench, memory speed

Key Questions:

• How to test configuration without modifying voipmonitor.conf?
• How to test regexp patterns or country assignment?
• How to check license channel usage?
• How to debug memory leaks using jemalloc?
• How to compile voipmonitor with jemalloc or SS7 support?
• How to generate jeprof PDF memory report?
• How to monitor thread load with sniffer_threads?
• How to test IPFIX or replay PCAPs?
• How to add layer 2 to PCAP for tcpreplay?
• How to disable sensor logs popup in GUI?
• How to fix IonCube ldap.so crash in Ubuntu 22.04?
• How to resolve VPN connection issues for remote support?
• How to disable TCP SACK on buggy kernels?
• How to test memory speed with sysbench?

MediaWiki Infrastructure

The VoIPmonitor documentation at https://www.voipmonitor.org/doc/ runs on MediaWiki. When MediaWiki versions reach End of Life, they must be upgraded to maintain security and compatibility.

Supported Version	Support Ends	Type
MediaWiki 1.43 LTS	December 2027 (Recommended)	Long-Term Support
MediaWiki 1.44	July 2026	Regular release
MediaWiki 1.45	December 2026	Regular release

⚠️ Warning: MediaWiki 1.39 reached EOL on December 31, 2025 and no longer receives security updates. Upgrade immediately to a supported version.

For MediaWiki upgrade procedures, refer to the official MediaWiki upgrade documentation.