Sniffer troubleshooting: Difference between revisions

From VoIPmonitor.org
(Add tcpdump PCAP capture workflow for definitive call leg troubleshooting)
(Add detailed innodb_buffer_pool_size calculation formula to OOM troubleshooting section)
Line 312: Line 312:
* '''Frequent process restarts without clear error messages'''
* '''Frequent process restarts without clear error messages'''


;3. Solution: Increase physical memory:
;3. First Fix: Check and correct innodb_buffer_pool_size:
The definitive solution for OOM-related CDR processing issues is to upgrade the server's physical RAM. After upgrading:
Before upgrading hardware, verify that <code>innodb_buffer_pool_size</code> is not set too high. This is a common cause of OOM incidents. If MySQL/MariaDB is consuming most of the available RAM, the buffer pool size is likely configured incorrectly for your system.
 
'''Calculate the correct buffer pool size:'''
For a server running both VoIPmonitor and MySQL on the same host:
<pre>
Formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS & services overhead - safety margin) / 2
 
Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory (check with ps aux): 2GB
- OS + other services overhead: 2GB
- Safety margin: ~25-30% of remaining RAM for other internal buffers
 
Calculation:
Available for buffer pool = 32GB - 2GB - 2GB = 28GB
Recommended innodb_buffer_pool_size = 14G (approximately 50% of available memory)
</pre>
 
'''Edit the MariaDB configuration file:'''
<pre>
# Common locations: /etc/mysql/my.cnf, /etc/mysql/mariadb.conf.d/50-server.cnf, /etc/my.cnf.d/
 
innodb_buffer_pool_size = 14G  # Adjust based on your calculation
</pre>
 
'''Restart MariaDB to apply:'''
<pre>systemctl restart mariadb  # or systemctl restart mysql</pre>
 
If the OOM events stop after correcting <code>innodb_buffer_pool_size</code>, no hardware upgrade is needed.
 
;4. Solution: Increase physical memory (if buffer pool tuning is insufficient):
If correcting the buffer pool size does not resolve the OOM issues, upgrade the server's physical RAM. After upgrading:
* Verify memory improvements with <code>free -h</code>
* Verify memory improvements with <code>free -h</code>
* Recalculate and adjust <code>innodb_buffer_pool_size</code> to utilize the additional memory
* Monitor for several days to ensure OOM events stop
* Monitor for several days to ensure OOM events stop
* Consider tuning `innodb_buffer_pool_size` in your MySQL configuration to use the additional memory effectively


Additional mitigation strategies (while planning for RAM upgrade):
Additional mitigation strategies (while planning for RAM upgrade or adjusting configuration):
* Reduce MySQL's memory footprint by lowering `innodb_buffer_pool_size` (e.g., from 16GB to 8GB)
* Reduce MySQL's memory footprint by lowering <code>innodb_buffer_pool_size</code> (e.g., from 16GB to 8GB)
* Disable or limit non-essential VoIPmonitor features (e.g., packet capture storage, RTP analysis)
* Disable or limit non-essential VoIPmonitor features (e.g., packet capture storage, RTP analysis)
* Ensure swap space is properly configured as a safety buffer (though swap is much slower than RAM)
* Ensure swap space is properly configured as a safety buffer (though swap is much slower than RAM)
* Use `sysctl vm.swappiness=10` to favor RAM over swap when some memory is still available
* Use <code>sysctl vm.swappiness=10</code> to favor RAM over swap when some memory is still available


== Step 9: Sensor Upgrade Fails with "Permission denied" from /tmp ==
== Step 9: Sensor Upgrade Fails with "Permission denied" from /tmp ==
Line 420: Line 451:


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' This document provides a step-by-step troubleshooting guide for when the VoIPmonitor sensor is not capturing any calls. The process is broken down into nine logical steps. Step 1 is to verify the service is running correctly using `systemctl status` and `ps`. Step 1 also includes troubleshooting for when the service fails to start with "binary not found" error after a crash: check if the binary has been renamed with an underscore suffix (e.g., `voipmonitor_`), rename it back to the standard name with `mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor`, then restart the service. Additionally, Step 1 includes a specific troubleshooting procedure for sensors that become unresponsive after GUI updates: execute `killall voipmonitor`, `systemctl stop voipmonitor`, and `systemctl start voipmonitor` to forcefully restart the service, then verify sensor status in the GUI. Step 2 is to use `tshark` to confirm if SIP/RTP traffic is actually arriving at the server's network interface. Step 3 covers network-level issues, including an important distinction between Layer 2 mirroring (SPAN/RSPAN) which requires promiscuous mode, and Layer 3 tunneling (ERSPAN/GRE/TZSP/VXLAN) which does NOT require promiscuous mode because the tunnel packets are addressed to the sensor's IP. Step 4 focuses on checking the `voipmonitor.conf` file for common misconfigurations like the `interface`, `sipport`, or `filter` parameters. Step 5 addresses GUI capture rules with the "Skip" option that can cause probes to stop processing calls even when network traffic is visible; it explains how to review, backup, remove, and systematically test capture rules via the GUI interface. Step 6 instructs the user on how to check the system logs (`syslog` or `messages`) for specific error messages from the sensor. Step 7 explains how to diagnose HEAP FULL / packetbuffer MEMORY IS FULL errors by first checking kernel message buffer (`dmesg -T`) for storage hardware errors like `ext4-fs error`, `I/O error`, or `NVRAM WARNING`. If hardware errors are present, the disk needs replacement. If dmesg is clear, the issue is a performance bottleneck that requires checking disk I/O performance and NFS latency. This step emphasizes the critical distinction between hardware failure and performance issues. Step 8 explains how to diagnose OOM (Out of Memory) killer events, which cause CDR processing to stop and require service restarts to restore. It provides commands to check for OOM events in kernel logs (grep, dmesg), monitor current memory usage (free -h, ps aux), and identify warning signs like low available memory (<500MB), high MySQL memory consumption, and frequent process restarts. The solution is to increase physical RAM, with mitigation strategies including reducing `innodb_buffer_pool_size`, disabling non-essential features, and configuring swap. Step 9 addresses a specific issue where sensor upgrade fails with "Permission denied" errors when executing scripts from `/tmp`, or the service fails to restart after upgrade. This is caused by the `/tmp` partition being mounted with the `noexec` flag. The solution involves checking mount options with `mount | grep /tmp`, remounting `/tmp` without the `noexec` flag using `mount -o remount,exec /tmp`, and making the change permanent by editing `/etc/fstab` to remove the `noexec` option from the `/tmp` mount line. The Appendix provides correct tshark display filter syntax for analyzing SIP traffic, including examples for filtering by SIP method (sip.Method == INVITE), searching for phone numbers (sip contains "number"), extracting Call-IDs (-T fields -e sip.Call-ID), and filtering by response codes (sip.Status-Code).
'''Summary:''' This document provides a step-by-step troubleshooting guide for when the VoIPmonitor sensor is not capturing any calls. The process is broken down into nine logical steps. Step 1 is to verify the service is running correctly using `systemctl status` and `ps`. Step 1 also includes troubleshooting for when the service fails to start with "binary not found" error after a crash: check if the binary has been renamed with an underscore suffix (e.g., `voipmonitor_`), rename it back to the standard name with `mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor`, then restart the service. Additionally, Step 1 includes a specific troubleshooting procedure for sensors that become unresponsive after GUI updates: execute `killall voipmonitor`, `systemctl stop voipmonitor`, and `systemctl start voipmonitor` to forcefully restart the service, then verify sensor status in the GUI. Step 2 is to use `tshark` to confirm if SIP/RTP traffic is actually arriving at the server's network interface. Step 3 covers network-level issues, including an important distinction between Layer 2 mirroring (SPAN/RSPAN) which requires promiscuous mode, and Layer 3 tunneling (ERSPAN/GRE/TZSP/VXLAN) which does NOT require promiscuous mode because the tunnel packets are addressed to the sensor's IP. Step 4 focuses on checking the `voipmonitor.conf` file for common misconfigurations like the `interface`, `sipport`, or `filter` parameters. Step 5 addresses GUI capture rules with the "Skip" option that can cause probes to stop processing calls even when network traffic is visible; it explains how to review, backup, remove, and systematically test capture rules via the GUI interface. Step 6 instructs the user on how to check the system logs (`syslog` or `messages`) for specific error messages from the sensor. Step 7 explains how to diagnose HEAP FULL / packetbuffer MEMORY IS FULL errors by first checking kernel message buffer (`dmesg -T`) for storage hardware errors like `ext4-fs error`, `I/O error`, or `NVRAM WARNING`. If hardware errors are present, the disk needs replacement. If dmesg is clear, the issue is a performance bottleneck that requires checking disk I/O performance and NFS latency. This step emphasizes the critical distinction between hardware failure and performance issues. Step 8 explains how to diagnose OOM (Out of Memory) killer events, which cause CDR processing to stop and require service restarts to restore. It provides commands to check for OOM events in kernel logs (grep, dmesg), monitor current memory usage (free -h, ps aux), and identify warning signs like low available memory (<500MB), high MySQL memory consumption, and frequent process restarts. The primary fix is to check and correct `innodb_buffer_pool_size`, which is often set too high and causes MariaDB to consume most of the available RAM. The guide provides a calculation formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS & services overhead - safety margin) / 2. For a 32GB server with VoIPmonitor using 2GB, the recommended value is 14G. If buffer pool tuning does not resolve the issue, the solution is to increase physical RAM. Additional mitigation strategies include disabling non-essential features and configuring swap. Step 9 addresses a specific issue where sensor upgrade fails with "Permission denied" errors when executing scripts from `/tmp`, or the service fails to restart after upgrade. This is caused by the `/tmp` partition being mounted with the `noexec` flag. The solution involves checking mount options with `mount | grep /tmp`, remounting `/tmp` without the `noexec` flag using `mount -o remount,exec /tmp`, and making the change permanent by editing `/etc/fstab` to remove the `noexec` option from the `/tmp` mount line. The Appendix provides correct tshark display filter syntax for analyzing SIP traffic, including examples for filtering by SIP method (sip.Method == INVITE), searching for phone numbers (sip contains "number"), extracting Call-IDs (-T fields -e sip.Call-ID), and filtering by response codes (sip.Status-Code).
'''Keywords:''' troubleshooting, no calls, not sniffing, no data, no CDRs, tshark, wireshark, promiscuous mode, promisc, ifconfig, ip link, SPAN, RSPAN, ERSPAN, GRE, TZSP, VXLAN, port mirroring, voipmonitor.conf, interface, sipport, filter, capture rules, Skip ON, GUI capture rules, reload sniffer, backup and restore, syslog, logs, permission denied, display filter, sip.Method, sip.Call-ID, sip.Status-Code, sip contains, OOM, out of memory, OOM killer, killed process, mysqld killed, free -h, memory usage, dmesg, swap, RAM upgrade, innodb_buffer_pool_size, GUI update, sensor update, killall, sensor unresponsive, service fails to start after update, /tmp, noexec, mount, fstab, upgrade fails, permission denied from tmp, sensor upgrade permission denied, tmp noexec flag, remount tmp exec, HEAP FULL, packetbuffer MEMORY IS FULL, dmesg -T, ext4-fs error, I/O error, NVRAM WARNING, storage hardware errors, disk failure, smart error, ata failed, buffer I/O error, smartctl, mdstat, RAID health, iostat, dstat, ioping, NFS latency, disk I/O performance, iowait, performance bottleneck, hardware failure vs performance, binary not found, service fails to start after crash, voipmonitor binary renaming, voipmonitor_ underscore, watchdog crash, binary renamed with underscore, mv voipmonitor_ voipmonitor, ls -l /usr/local/sbin/voipmonitor_
'''Keywords:''' troubleshooting, no calls, not sniffing, no data, no CDRs, tshark, wireshark, promiscuous mode, promisc, ifconfig, ip link, SPAN, RSPAN, ERSPAN, GRE, TZSP, VXLAN, port mirroring, voipmonitor.conf, interface, sipport, filter, capture rules, Skip ON, GUI capture rules, reload sniffer, backup and restore, syslog, logs, permission denied, display filter, sip.Method, sip.Call-ID, sip.Status-Code, sip contains, OOM, out of memory, OOM killer, killed process, mysqld killed, free -h, memory usage, dmesg, swap, RAM upgrade, innodb_buffer_pool_size, GUI update, sensor update, buffer pool calculation, innodb_buffer_pool too high, correct buffer pool size, calculate innodb_buffer_pool, killall, sensor unresponsive, service fails to start after update, /tmp, noexec, mount, fstab, upgrade fails, permission denied from tmp, sensor upgrade permission denied, tmp noexec flag, remount tmp exec, HEAP FULL, packetbuffer MEMORY IS FULL, dmesg -T, ext4-fs error, I/O error, NVRAM WARNING, storage hardware errors, disk failure, smart error, ata failed, buffer I/O error, smartctl, mdstat, RAID health, iostat, dstat, ioping, NFS latency, disk I/O performance, iowait, performance bottleneck, hardware failure vs performance, binary not found, service fails to start after crash, voipmonitor binary renaming, voipmonitor_ underscore, watchdog crash, binary renamed with underscore, mv voipmonitor_ voipmonitor, ls -l /usr/local/sbin/voipmonitor_
'''Key Questions:'''
'''Key Questions:'''
* Why is VoIPmonitor not recording any calls?
* Why is VoIPmonitor not recording any calls?
Line 460: Line 491:
* How do I fix OOM killer issues on VoIPmonitor?
* How do I fix OOM killer issues on VoIPmonitor?
* Why is mysqld getting killed on my system?
* Why is mysqld getting killed on my system?
* How do I calculate the correct innodb_buffer_pool_size for VoIPmonitor?
* What is the formula for innodb_buffer_pool_size on a shared server?
* My innodb_buffer_pool_size is too high, how do I fix OOM issues?
* What should innodb_buffer_pool_size be set to on a 32GB server?
* Does MySQL innodb_buffer_pool_size cause OOM killer?
* Sensor becomes unresponsive after GUI update, what should I do?
* Sensor becomes unresponsive after GUI update, what should I do?
* How do I restart a sensor service that fails to start after GUI update?
* How do I restart a sensor service that fails to start after GUI update?

Revision as of 05:35, 4 January 2026


This guide provides a systematic, step-by-step process to diagnose why the VoIPmonitor sensor might not be capturing any calls. Follow these steps in order to quickly identify and resolve the most common issues.

Step 1: Is the VoIPmonitor Service Running Correctly?

First, confirm the sensor process is active and loaded the correct configuration file.

1. Check the service status (for modern systemd systems)
systemctl status voipmonitor

Look for a line that says Active: active (running). If it is inactive or failed, try restarting it with `systemctl restart voipmonitor` and check the status again.

2. Service Fails to Start with "Binary Not Found" After Crash

If the VoIPmonitor service fails to start after a crash or watchdog restart with an error message indicating the binary cannot be found (e.g., "No such file or directory" for `/usr/local/sbin/voipmonitor`), the binary may have been renamed with an underscore suffix during the crash recovery process.

Check for a renamed binary:

# Check if the standard binary path exists
ls -l /usr/local/sbin/voipmonitor

# If not found, look for a renamed version with underscore suffix
ls -l /usr/local/sbin/voipmonitor_*

If you find a renamed binary (e.g., `voipmonitor_`, `voipmonitor_20250104`, etc.), rename it back to the standard name:

mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor

Then restart the service:

systemctl start voipmonitor

Verify the service starts correctly:

systemctl status voipmonitor
3. Sensor Becomes Unresponsive After GUI Update

If the sensor service fails to start or becomes unresponsive after updating a sensor through the Web GUI, the update process may have left the service in a stuck state. The solution is to forcefully stop the service and restart it using these commands:

# SSH into the sensor host and execute:
killall voipmonitor
systemctl stop voipmonitor
systemctl start voipmonitor

After running these commands, verify the sensor status in the GUI to confirm it is responding correctly. This sequence ensures: (1) Any zombie or hung processes are terminated with `killall`, (2) systemd is fully stopped, and (3) a clean start of the service.

4. Verify the running process
ps aux | grep voipmonitor

This command will show the running process and the exact command line arguments it was started with. Critically, ensure it is using the correct configuration file, for example: --config-file /etc/voipmonitor.conf. If it is not, there may be an issue with your startup script.

Step 2: Is Network Traffic Reaching the Server?

If the service is running, the next step is to verify if the VoIP packets (SIP/RTP) are actually arriving at the server's network interface. The best tool for this is `tshark` (the command-line version of Wireshark).

1. Install tshark
# For Debian/Ubuntu
apt-get update && apt-get install tshark

# For CentOS/RHEL/AlmaLinux
yum install wireshark
2. Listen for SIP traffic on the correct interface

Replace `eth0` with the interface name you have configured in `voipmonitor.conf`.

tshark -i eth0 -Y "sip || rtp" -n
  • If you see a continuous stream of SIP and RTP packets, it means traffic is reaching the server, and the problem is likely in VoIPmonitor's configuration (see Step 4).
  • If you see NO packets, the problem lies with your network configuration. Proceed to Step 3.
3. Advanced
Capture to PCAP File for Definitive Testing

Live monitoring with tshark is useful for observation, but capturing traffic to a .pcap file during a test call provides definitive evidence for troubleshooting intermittent issues or specific call legs.

Method 1: Using tcpdump (Recommended)

# Start capture on the correct interface (replace eth0)
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap port 5060

# Or capture both SIP and RTP traffic:
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap "(port 5060 or udp)"

# Let it run while you make a test call with the missing call leg
# Press Ctrl+C to stop the capture

# Analyze the capture file:
tshark -r /tmp/test_capture.pcap -Y "sip"

Method 2: Using tshark to capture to file

# Start capture:
tshark -i eth0 -w /tmp/test_capture.pcap -f "tcp port 5060 or udp"

# Make your test call, then press Ctrl+C to stop

# Analyze the capture:
tshark -r /tmp/test_capture.pcap -Y "sip" -V

Decision Tree for PCAP Analysis: After capturing a test call known to have a missing leg:

  • If SIP packets are missing from the .pcap file:
    • The problem is with your network mirroring configuration (SPAN/TAP port, AWS Traffic Mirroring, etc.)
    • The packets never reached the VoIPmonitor sensor's network interface
    • Fix the switch mirroring setup or infrastructure configuration first
  • If SIP packets ARE present in the .pcap file but missing in the VoIPmonitor GUI:**
    • The problem is with VoIPmonitor's configuration or processing
    • Packets reached the NIC but were not processed correctly
    • Review Step 4 (VoIPmonitor Configuration) and Step 5 (Capture Rules)

Example Test Call Workflow:

# 1. Start capture
tcpdump -i eth0 -s 0 -w /tmp/test.pcap "sip and host 10.0.1.100"

# 2. Make a test call from phone at 10.0.1.100 to 10.0.2.200
#    (a call that you know should have recordings but is missing)

# 3. Stop capture (Ctrl+C)

# 4. Check for the specific call's Call-ID
tshark -r /tmp/test.pcap -Y "sip" -T fields -e sip.Call-ID

# 5. Verify if packets for both A-leg and B-leg exist
tshark -r /tmp/test.pcap -Y "sip && ip.addr == 10.0.1.100"

# 6. Compare results with VoIPmonitor GUI
#    - If packets found in .pcap: VoIPmonitor software issue
#    - If packets missing from .pcap: Network mirroring issue

Step 3: Troubleshoot Network and Interface Configuration

If `tshark` shows no traffic, it means the packets are not being delivered to the operating system correctly.

1. Check if the interface is UP

Ensure the network interface is active.

ip link show eth0

The output should contain the word `UP`. If it doesn't, bring it up with:

ip link set dev eth0 up
2. Check for Promiscuous Mode (for SPAN/RSPAN Mirrored Traffic)

Important: Promiscuous mode requirements depend on your traffic mirroring method:

  • SPAN/RSPAN (Layer 2 mirroring): The network interface must be in promiscuous mode. Mirrored packets retain their original MAC addresses, so the interface would normally ignore them. Promiscuous mode forces the interface to accept all packets regardless of destination MAC.
  • ERSPAN/GRE/TZSP/VXLAN (Layer 3 tunnels): Promiscuous mode is NOT required. These tunneling protocols encapsulate the mirrored traffic inside IP packets that are addressed directly to the sensor's IP address. The operating system receives these packets normally, and VoIPmonitor automatically decapsulates them to extract the inner SIP/RTP traffic.

For SPAN/RSPAN deployments, check the current promiscuous mode status:

ip link show eth0

Look for the `PROMISC` flag.

Enable promiscuous mode manually if needed:

ip link set eth0 promisc on

If this solves the problem, you should make the change permanent. The `install-script.sh` for the sensor usually attempts to do this, but it can fail.

3. Verify Your SPAN/Mirror/TAP Configuration

This is the most common cause of no traffic. Double-check your network switch or hardware tap configuration to ensure:

  • The correct source ports (where your PBX/SBC is connected) are being monitored.
  • The correct destination port (where your VoIPmonitor sensor is connected) is configured.
  • If you are monitoring traffic across different VLANs, ensure your mirror port is configured to carry all necessary VLAN tags (often called "trunk" mode).

Step 4: Check the VoIPmonitor Configuration

If `tshark` sees traffic but VoIPmonitor does not, the problem is almost certainly in `voipmonitor.conf`.

1. Check the `interface` directive
Make sure the `interface` parameter in `/etc/voipmonitor.conf` exactly matches the interface where you see traffic with `tshark`. For example: `interface = eth0`.
2. Check the `sipport` directive
By default, VoIPmonitor only listens on port 5060. If your PBX uses a different port for SIP, you must add it. For example:
sipport = 5060,5080
3. Check for a restrictive `filter`
If you have a BPF `filter` configured, ensure it is not accidentally excluding the traffic you want to see. For debugging, try commenting out the `filter` line entirely and restarting the sensor.

Step 5: Check GUI Capture Rules (Causing Call Stops)

If `tshark` sees SIP traffic and the sniffer configuration appears correct, but the probe stops processing calls or shows traffic only on the network interface, GUI capture rules may be the culprit.

Capture rules configured in the GUI can instruct the sniffer to ignore ("skip") all processing for matched calls. This includes calls matching specific IP addresses or telephone number prefixes.

1. Review existing capture rules
Navigate to GUI → Capture rules and examine all rules for any that might be blocking your traffic.
Look specifically for rules with the Skip option set to ON (displayed as "Skip: ON"). The Skip option instructs the sniffer to completely ignore matching calls (no files, RTP analysis, or CDR creation).
2. Test by temporarily removing all capture rules
To isolate the issue, first create a backup of your GUI configuration:
  • Navigate to Tools → Backup & Restore → Backup GUI → Configuration tables
  • This saves your current settings including capture rules
  • Delete all capture rules from the GUI
  • Click the Apply button to save changes
  • Reload the sniffer by clicking the green "reload sniffer" button in the control panel
  • Test if calls are now being processed correctly
  • If resolved, restore the configuration from the backup and systematically investigate the rules to identify the problematic one
3. Identify the problematic rule
  • After restoring your configuration, remove rules one at a time and reload the sniffer after each removal
  • When calls start being processed again, you have identified the problematic rule
  • Review the rule's match criteria (IP addresses, prefixes, direction) against your actual traffic pattern
  • Adjust the rule's conditions or Skip setting as needed
4. Verify rules are reloaded
After making changes to capture rules, remember that changes are not automatically applied to the running sniffer. You must click the "reload sniffer" button in the control panel, or the rules will continue using the previous configuration.

For more information on capture rules, see Capture_rules.

Step 6: Check VoIPmonitor Logs for Errors

Finally, VoIPmonitor's own logs are the best source for clues. Check the system log for any error messages generated by the sensor on startup or during operation.

# For Debian/Ubuntu
tail -f /var/log/syslog | grep voipmonitor

# For CentOS/RHEL/AlmaLinux
tail -f /var/log/messages | grep voipmonitor

Look for errors like:

  • "pcap_open_live(eth0) error: eth0: No such device" (Wrong interface name)
  • "Permission denied" (The sensor is not running with sufficient privileges)
  • Errors related to database connectivity.
  • Messages about dropping packets.

Step 7: Check for Storage Hardware Errors (HEAP FULL / packetbuffer Issues)

If the sensor is crashing with "HEAP FULL" errors or showing "packetbuffer: MEMORY IS FULL" messages, you must distinguish between actual storage hardware failures (requires disk replacement) and performance bottlenecks (requires tuning).

1. Check kernel message buffer for storage errors
dmesg -T | grep -iE "ext4-fs error|i/o error|nvram warning|ata.*failed|sda.*error|disk failure|smart error" | tail -50

Look for these hardware error indicators:

  • ext4-fs error - Filesystem corruption or disk failure
  • I/O error or BUG: soft lockup - Disk read/write failures
  • NVRAM WARNING: nvram_check: failed - RAID controller battery/capacitor issues
  • ata.*: FAILED - Hard drive SMART failure
  • Buffer I/O error - Disk unable to complete operations

If you see ANY of these errors:

  • The storage subsystem is failing and likely needs hardware replacement
  • Do not attempt performance tuning - replace the failed disk/RAID first
  • Check SMART status: smartctl -a /dev/sda
  • Check RAID health: cat /proc/mdstat or RAID controller tools
2. If dmesg is clean of errors → Performance Bottleneck

If the kernel logs show no storage errors, the issue is a performance bottleneck (disk too slow, network latency, etc.).

Check disk I/O performance:

# Current I/O wait (should be < 10% normally)
iostat -x 5

# Detailed disk stats
dstat -d

# Real-time disk latency
ioping -c 10 .

Check NFS latency (if using NFS storage):

# Test NFS read/write latency
time dd if=/dev/zero of=/var/spool/voipmonitor/testfile bs=1M count=100
time cat /var/spool/voipmonitor/testfile > /dev/null
rm /var/spool/voipmonitor/testfile

# Check NFS mount options
mount | grep nfs

Common performance solutions:

  • Use SSD/NVMe for VoIPmonitor spool directory
  • Ensure proper NIC queue settings for high-throughput NFS
  • Check network switch port configuration for NFS
  • Review Scaling guide for detailed optimization

See also IO_Measurement for comprehensive disk benchmarking tools.

Step 8: Check for OOM (Out of Memory) Issues

If VoIPmonitor suddenly stops processing CDRs and a service restart temporarily restores functionality, the system may be experiencing OOM (Out of Memory) killer events. The Linux OOM killer terminates processes when available RAM is exhausted, and MySQL (`mysqld`) is a common target due to its memory-intensive nature.

1. Check for OOM killer events in kernel logs
# For Debian/Ubuntu
grep -i "out of memory\|killed process" /var/log/syslog | tail -20

# For CentOS/RHEL/AlmaLinux
grep -i "out of memory\|killed process" /var/log/messages | tail -20

# Also check dmesg:
dmesg | grep -i "killed process" | tail -10

Typical OOM killer messages look like:

Out of memory: Kill process 1234 (mysqld) score 123 or sacrifice child
Killed process 1234 (mysqld) total-vm: 12345678kB, anon-rss: 1234567kB
2. Monitor current memory usage
# Check available memory (look for low 'available' or 'free' values)
free -h

# Check per-process memory usage (sorted by RSS)
ps aux --sort=-%mem | head -15

# Check MySQL memory usage in bytes
cat /proc/$(pgrep mysqld)/status | grep -E "VmSize|VmRSS"

Warning signs:

  • Available memory consistently below 500MB during operation
  • MySQL consuming most of the available RAM
  • Swap usage near 100% (if swap is enabled)
  • Frequent process restarts without clear error messages
3. First Fix
Check and correct innodb_buffer_pool_size:

Before upgrading hardware, verify that innodb_buffer_pool_size is not set too high. This is a common cause of OOM incidents. If MySQL/MariaDB is consuming most of the available RAM, the buffer pool size is likely configured incorrectly for your system.

Calculate the correct buffer pool size: For a server running both VoIPmonitor and MySQL on the same host:

Formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS & services overhead - safety margin) / 2

Example for a 32GB server:
- Total RAM: 32GB
- VoIPmonitor process memory (check with ps aux): 2GB
- OS + other services overhead: 2GB
- Safety margin: ~25-30% of remaining RAM for other internal buffers

Calculation:
Available for buffer pool = 32GB - 2GB - 2GB = 28GB
Recommended innodb_buffer_pool_size = 14G (approximately 50% of available memory)

Edit the MariaDB configuration file:

# Common locations: /etc/mysql/my.cnf, /etc/mysql/mariadb.conf.d/50-server.cnf, /etc/my.cnf.d/

innodb_buffer_pool_size = 14G  # Adjust based on your calculation

Restart MariaDB to apply:

systemctl restart mariadb  # or systemctl restart mysql

If the OOM events stop after correcting innodb_buffer_pool_size, no hardware upgrade is needed.

4. Solution
Increase physical memory (if buffer pool tuning is insufficient):

If correcting the buffer pool size does not resolve the OOM issues, upgrade the server's physical RAM. After upgrading:

  • Verify memory improvements with free -h
  • Recalculate and adjust innodb_buffer_pool_size to utilize the additional memory
  • Monitor for several days to ensure OOM events stop

Additional mitigation strategies (while planning for RAM upgrade or adjusting configuration):

  • Reduce MySQL's memory footprint by lowering innodb_buffer_pool_size (e.g., from 16GB to 8GB)
  • Disable or limit non-essential VoIPmonitor features (e.g., packet capture storage, RTP analysis)
  • Ensure swap space is properly configured as a safety buffer (though swap is much slower than RAM)
  • Use sysctl vm.swappiness=10 to favor RAM over swap when some memory is still available

Step 9: Sensor Upgrade Fails with "Permission denied" from /tmp

If the sensor upgrade process fails with "Permission denied" errors when executing scripts from the `/tmp` directory, or the service fails to restart after upgrade, the `/tmp` partition may be mounted with the `noexec` flag.

The `noexec` mount option prevents execution of any script or binary from the `/tmp` directory for security reasons. However, the VoIPmonitor sensor upgrade process uses `/tmp` for temporary script execution.

1. Check the mount options for /tmp
mount | grep /tmp

Look for the `noexec` flag in the mount options. Output will show something like:

/dev/sda2 on /tmp type ext4 rw,relatime,noexec,nosuid,nodev
2. Remount /tmp without noexec (temporary fix)
mount -o remount,exec /tmp

Verify the change:

mount | grep /tmp

The output should no longer contain `noexec`.

3. Make the change permanent (edit /etc/fstab)

Open the `/etc/fstab` file and locate the line corresponding to the `/tmp` mount point. Remove the `noexec` option from that line.

nano /etc/fstab

Example:

# Before:
/dev/sda2  /tmp  ext4  rw,relatime,noexec,nosuid,nodev  0 0

# After (remove noexec):
/dev/sda2  /tmp  ext4  rw,relatime,nosuid,nodev  0 0

If `/tmp` is a separate partition, you may need to remount it for changes to take effect:

mount -o remount /tmp
4. Re-run the sensor upgrade

After fixing the mount options, retry the sensor upgrade process.

Appendix: tshark Display Filter Syntax for SIP

When using `tshark` to analyze SIP traffic, it is important to use the correct Wireshark display filter syntax. Below are common filter examples:

Basic SIP Filters

# Show all SIP INVITE messages
tshark -r capture.pcap -Y "sip.Method == INVITE"

# Show all SIP messages (any method)
tshark -r capture.pcap -Y "sip"

# Show SIP and RTP traffic
tshark -r capture.pcap -Y "sip || rtp"

Search for Specific Phone Number or Text

# Find calls containing a specific phone number (e.g., 5551234567)
tshark -r capture.pcap -Y 'sip contains "5551234567"'

# Find INVITE messages for a specific number
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"'

Extract Call-ID from Matching Calls

# Get Call-ID for calls matching a phone number
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"' -T fields -e sip.Call-ID

# Get Call-ID along with From and To headers
tshark -r capture.pcap -Y 'sip.Method == INVITE' -T fields -e sip.Call-ID -e sip.from.user -e sip.to.user

Filter by IP Address

# SIP traffic from a specific source IP
tshark -r capture.pcap -Y "sip && ip.src == 192.168.1.100"

# SIP traffic between two hosts
tshark -r capture.pcap -Y "sip && ip.addr == 192.168.1.100 && ip.addr == 10.0.0.50"

Filter by SIP Response Code

# Show all 200 OK responses
tshark -r capture.pcap -Y "sip.Status-Code == 200"

# Show all 4xx and 5xx error responses
tshark -r capture.pcap -Y "sip.Status-Code >= 400"

# Show 486 Busy Here responses
tshark -r capture.pcap -Y "sip.Status-Code == 486"

Important Syntax Notes

  • Field names are case-sensitive: Use sip.Method, sip.Call-ID, sip.Status-Code (not sip.method or sip.call-id)
  • String matching uses contains: Use sip contains "text" (not sip.contains())
  • Use double quotes for strings: sip contains "number" (not single quotes)
  • Boolean operators: Use && (and), || (or), ! (not)

For a complete reference, see the Wireshark SIP Display Filter Reference.

AI Summary for RAG

Summary: This document provides a step-by-step troubleshooting guide for when the VoIPmonitor sensor is not capturing any calls. The process is broken down into nine logical steps. Step 1 is to verify the service is running correctly using `systemctl status` and `ps`. Step 1 also includes troubleshooting for when the service fails to start with "binary not found" error after a crash: check if the binary has been renamed with an underscore suffix (e.g., `voipmonitor_`), rename it back to the standard name with `mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor`, then restart the service. Additionally, Step 1 includes a specific troubleshooting procedure for sensors that become unresponsive after GUI updates: execute `killall voipmonitor`, `systemctl stop voipmonitor`, and `systemctl start voipmonitor` to forcefully restart the service, then verify sensor status in the GUI. Step 2 is to use `tshark` to confirm if SIP/RTP traffic is actually arriving at the server's network interface. Step 3 covers network-level issues, including an important distinction between Layer 2 mirroring (SPAN/RSPAN) which requires promiscuous mode, and Layer 3 tunneling (ERSPAN/GRE/TZSP/VXLAN) which does NOT require promiscuous mode because the tunnel packets are addressed to the sensor's IP. Step 4 focuses on checking the `voipmonitor.conf` file for common misconfigurations like the `interface`, `sipport`, or `filter` parameters. Step 5 addresses GUI capture rules with the "Skip" option that can cause probes to stop processing calls even when network traffic is visible; it explains how to review, backup, remove, and systematically test capture rules via the GUI interface. Step 6 instructs the user on how to check the system logs (`syslog` or `messages`) for specific error messages from the sensor. Step 7 explains how to diagnose HEAP FULL / packetbuffer MEMORY IS FULL errors by first checking kernel message buffer (`dmesg -T`) for storage hardware errors like `ext4-fs error`, `I/O error`, or `NVRAM WARNING`. If hardware errors are present, the disk needs replacement. If dmesg is clear, the issue is a performance bottleneck that requires checking disk I/O performance and NFS latency. This step emphasizes the critical distinction between hardware failure and performance issues. Step 8 explains how to diagnose OOM (Out of Memory) killer events, which cause CDR processing to stop and require service restarts to restore. It provides commands to check for OOM events in kernel logs (grep, dmesg), monitor current memory usage (free -h, ps aux), and identify warning signs like low available memory (<500MB), high MySQL memory consumption, and frequent process restarts. The primary fix is to check and correct `innodb_buffer_pool_size`, which is often set too high and causes MariaDB to consume most of the available RAM. The guide provides a calculation formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS & services overhead - safety margin) / 2. For a 32GB server with VoIPmonitor using 2GB, the recommended value is 14G. If buffer pool tuning does not resolve the issue, the solution is to increase physical RAM. Additional mitigation strategies include disabling non-essential features and configuring swap. Step 9 addresses a specific issue where sensor upgrade fails with "Permission denied" errors when executing scripts from `/tmp`, or the service fails to restart after upgrade. This is caused by the `/tmp` partition being mounted with the `noexec` flag. The solution involves checking mount options with `mount | grep /tmp`, remounting `/tmp` without the `noexec` flag using `mount -o remount,exec /tmp`, and making the change permanent by editing `/etc/fstab` to remove the `noexec` option from the `/tmp` mount line. The Appendix provides correct tshark display filter syntax for analyzing SIP traffic, including examples for filtering by SIP method (sip.Method == INVITE), searching for phone numbers (sip contains "number"), extracting Call-IDs (-T fields -e sip.Call-ID), and filtering by response codes (sip.Status-Code). Keywords: troubleshooting, no calls, not sniffing, no data, no CDRs, tshark, wireshark, promiscuous mode, promisc, ifconfig, ip link, SPAN, RSPAN, ERSPAN, GRE, TZSP, VXLAN, port mirroring, voipmonitor.conf, interface, sipport, filter, capture rules, Skip ON, GUI capture rules, reload sniffer, backup and restore, syslog, logs, permission denied, display filter, sip.Method, sip.Call-ID, sip.Status-Code, sip contains, OOM, out of memory, OOM killer, killed process, mysqld killed, free -h, memory usage, dmesg, swap, RAM upgrade, innodb_buffer_pool_size, GUI update, sensor update, buffer pool calculation, innodb_buffer_pool too high, correct buffer pool size, calculate innodb_buffer_pool, killall, sensor unresponsive, service fails to start after update, /tmp, noexec, mount, fstab, upgrade fails, permission denied from tmp, sensor upgrade permission denied, tmp noexec flag, remount tmp exec, HEAP FULL, packetbuffer MEMORY IS FULL, dmesg -T, ext4-fs error, I/O error, NVRAM WARNING, storage hardware errors, disk failure, smart error, ata failed, buffer I/O error, smartctl, mdstat, RAID health, iostat, dstat, ioping, NFS latency, disk I/O performance, iowait, performance bottleneck, hardware failure vs performance, binary not found, service fails to start after crash, voipmonitor binary renaming, voipmonitor_ underscore, watchdog crash, binary renamed with underscore, mv voipmonitor_ voipmonitor, ls -l /usr/local/sbin/voipmonitor_ Key Questions:

  • Why is VoIPmonitor not recording any calls?
  • VoIPmonitor service fails to start after crash with binary not found error, what to do?
  • How do I fix voipmonitor binary not found error?
  • Where is the voipmonitor binary located?
  • Why is the voipmonitor binary renamed with underscore?
  • How can I check if VoIP traffic is reaching my sensor server?
  • What command can I use to see live SIP traffic on the command line?
  • How do I enable promiscuous mode on my network card?
  • Do I need promiscuous mode for ERSPAN or GRE tunnels?
  • Does ERSPAN require promiscuous mode on the receiving interface?
  • VoIPmonitor is running but I have no new calls in the GUI, what should I check first?
  • Where can I find the log files for the VoIPmonitor sniffer?
  • What are the most common reasons for VoIPmonitor not capturing data?
  • How do I filter tshark output for SIP INVITE messages?
  • What is the correct tshark filter syntax to find a specific phone number?
  • How do I extract Call-ID from a pcap file using tshark?
  • What tshark filter shows all SIP 4xx and 5xx error responses?
  • Why is my VoIPmonitor probe stopping processing calls even though network traffic is visible?
  • What should I check if the probe sees SIP packets on the interface but processes no calls?
  • How do GUI capture rules affect call processing?
  • What does the "Skip" option in capture rules do?
  • How do I troubleshoot capture rules that are blocking calls?
  • VoIPmonitor server stops processing CDRs and needs restart. What could be wrong?
  • Why does sensor upgrade fail with permission denied errors from /tmp?
  • How do I fix noexec flag on /tmp directory?
  • VoIPmonitor permission denied when upgrading sensor, what to do?
  • How do I check mount options for /tmp partition?
  • What command checks if /tmp has noexec flag?
  • How to remove noexec from /tmp mount point?
  • Sensor upgrade fails in tmp directory, why?
  • Why does MySQL crash and restart on my VoIPmonitor server?
  • How do I check for OOM killer events in Linux?
  • What does the error "Out of memory: Kill process" mean?
  • How can I monitor memory usage on my VoIPmonitor server?
  • What command shows available memory in Linux?
  • How do I fix OOM killer issues on VoIPmonitor?
  • Why is mysqld getting killed on my system?
  • How do I calculate the correct innodb_buffer_pool_size for VoIPmonitor?
  • What is the formula for innodb_buffer_pool_size on a shared server?
  • My innodb_buffer_pool_size is too high, how do I fix OOM issues?
  • What should innodb_buffer_pool_size be set to on a 32GB server?
  • Does MySQL innodb_buffer_pool_size cause OOM killer?
  • Sensor becomes unresponsive after GUI update, what should I do?
  • How do I restart a sensor service that fails to start after GUI update?
  • What is the fix for sensor service not starting after web GUI update?
  • Why is my sensor service stuck after updating through the GUI?
  • How to fix unresponsive sensor after GUI upgrade using killall?
  • Sensor service fails after GUI upgrade, what commands to run?
  • How do I force kill all voipmonitor processes?
  • VoIPmonitor crashes with HEAP FULL error, what should I check?
  • sniffer crashes with packetbuffer MEMORY IS FULL messages, how to fix?
  • How do I check for disk hardware errors in Linux?
  • What command shows storage errors in the kernel buffer?
  • How to distinguish between disk hardware failure vs performance bottleneck?
  • What are the signs of a failing hard drive in dmesg?
  • How do I check SMART status of a hard drive?
  • What does ext4-fs error mean in dmesg?
  • What is an I/O error in kernel logs?
  • What is NVRAM WARNING on RAID controller?
  • How do I check disk I/O performance on Linux?
  • How to measure NFS latency for VoIPmonitor?
  • What is iostat command and how do I use it?
  • How to test disk speed with ioping?
  • Should I replace disk or tune settings for HEAP FULL errors?
  • Voipmonitor sensor crashing with HEAP FULL, first diagnostic step?