Sniffer troubleshooting: Difference between revisions

From VoIPmonitor.org
(Remove illogical Step numbering, convert pre to syntaxhighlight, fix internal links)
(Patch: replace '=== Solution: I/O Bottleneck =...')
 
(102 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Troubleshooting: No Calls Being Sniffed}}
= Sniffer Troubleshooting =


'''This guide provides a systematic process to diagnose why the VoIPmonitor sensor might not be capturing any calls. Use it to quickly identify and resolve the most common issues.'''
This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see [[Sniffer_configuration]]. For performance tuning, see [[Scaling]].


== Is the VoIPmonitor Service Running Correctly? ==
== Critical First Step: Is Traffic Reaching the Interface? ==
First, confirm the sensor process is active and loaded the correct configuration file.


;1. Check the service status (for modern systemd systems):
{{Warning|Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help.}}
<syntaxhighlight lang="bash">systemctl status voipmonitor</syntaxhighlight>
Look for a line that says <code>Active: active (running)</code>. If it is inactive or failed, try restarting it with <code>systemctl restart voipmonitor</code> and check the status again.


;2. Service Fails to Start with "Binary Not Found" After Crash:
<syntaxhighlight lang="bash">
If the VoIPmonitor service fails to start after a crash or watchdog restart with an error message indicating the binary cannot be found (e.g., "No such file or directory" for <code>/usr/local/sbin/voipmonitor</code>), the binary may have been renamed with an underscore suffix during the crash recovery process.
# Check for SIP traffic on the capture interface
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10
 
# If no packets: Network/SPAN issue - contact network admin
# If packets visible: Proceed with sensor troubleshooting below
</syntaxhighlight>
 
<kroki lang="mermaid">
graph TD
    A[No Calls Recorded] --> B{Packets on interface?<br/>tcpdump -i eth0 port 5060}
    B -->|No packets| C[Network Issue]
    C --> C1[Check SPAN/mirror config]
    C --> C2[Verify VLAN tagging]
    C --> C3[Check cable/port]
    B -->|Packets visible| D[Sensor Issue]
    D --> D1[Check voipmonitor.conf]
    D --> D2[Check GUI Capture Rules]
    D --> D3[Check logs for errors]
</kroki>
 
== Quick Diagnostic Checklist ==
 
{| class="wikitable"
|-
! Check !! Command !! Expected Result
|-
| Service running || <code>systemctl status voipmonitor</code> || Active (running)
|-
| Traffic on interface || <code>tshark -i eth0 -c 5 -Y "sip"</code> || SIP packets displayed
|-
| Interface errors || <code>ip -s link show eth0</code> || No RX errors/drops
|-
| Promiscuous mode || <code>ip link show eth0</code> || PROMISC flag present
|-
| Logs || <code>tail -100 /var/log/syslog \| grep voip</code> || No critical errors
|-
| GUI rules || Settings → Capture Rules || No unexpected "Skip" rules
|}
 
== No Calls Being Recorded ==
 
=== Service Not Running ===


Check for a renamed binary:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check if the standard binary path exists
# Check status
ls -l /usr/local/sbin/voipmonitor
systemctl status voipmonitor


# If not found, look for a renamed version with underscore suffix
# View recent logs
ls -l /usr/local/sbin/voipmonitor_*
journalctl -u voipmonitor --since "10 minutes ago"
 
# Start/restart
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


If you find a renamed binary (e.g., <code>voipmonitor_</code>, <code>voipmonitor_20250104</code>, etc.), rename it back to the standard name:
Common startup failures:
* '''Interface not found''': Check <code>interface</code> in voipmonitor.conf matches <code>ip a</code> output
* '''Port already in use''': Another process using the management port
* '''License issue''': Check [[License]] for activation problems
 
=== Wrong Interface or Port Configuration ===
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor
# Check current config
grep -E "^interface|^sipport" /etc/voipmonitor.conf
 
# Example correct config:
# interface = eth0
# sipport = 5060
</syntaxhighlight>
</syntaxhighlight>


Then restart the service:
{{Tip|For multiple SIP ports: <code>sipport = 5060,5061,5080</code>}}
 
=== GUI Capture Rules Blocking ===
 
Navigate to '''Settings → Capture Rules''' and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.
 
See [[Capture_rules]] for detailed configuration.
 
=== SPAN/Mirror Not Configured ===
 
If <code>tcpdump</code> shows no traffic:
# Verify switch SPAN/mirror port configuration
# Check that both directions (ingress + egress) are mirrored
# Confirm VLAN tagging is preserved if needed
# Test physical connectivity (cable, port status)
 
See [[Sniffing_modes]] for SPAN, RSPAN, and ERSPAN configuration.
 
=== Filter Parameter Too Restrictive ===
 
If <code>filter</code> is set in voipmonitor.conf, it may exclude traffic:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl start voipmonitor
# Check filter
grep "^filter" /etc/voipmonitor.conf
 
# Temporarily disable to test
# Comment out the filter line and restart
</syntaxhighlight>
</syntaxhighlight>


Verify the service starts correctly:
 
 
==== Missing id_sensor Parameter ====
 
'''Symptom''': SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.
 
'''Cause''': The <code>id_sensor</code> parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.
 
'''Solution''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl status voipmonitor
# Check if id_sensor is set
grep "^id_sensor" /etc/voipmonitor.conf
 
# Add or correct the parameter
echo "id_sensor = 1" >> /etc/voipmonitor.conf
 
# Restart the service
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


;3. Sensor Becomes Unresponsive After GUI Update:
{{Tip|Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See [[Sniffer_configuration#id_sensor|id_sensor documentation]].}}
If the sensor service fails to start or becomes unresponsive after updating a sensor through the Web GUI, the update process may have left the service in a stuck state. The solution is to forcefully stop the service and restart it using these commands:
== Missing Audio / RTP Issues ==
 
=== One-Way Audio (Asymmetric Mirroring) ===
 
'''Symptom''': SIP recorded but only one RTP direction captured.
 
'''Cause''': SPAN port configured for only one direction.
 
'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# SSH into the sensor host and execute:
# Count RTP packets per direction
killall voipmonitor
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c
systemctl stop voipmonitor
systemctl start voipmonitor
</syntaxhighlight>
</syntaxhighlight>
After running these commands, verify the sensor status in the GUI to confirm it is responding correctly. This sequence ensures: (1) Any zombie or hung processes are terminated with <code>killall</code>, (2) systemd is fully stopped, and (3) a clean start of the service.


;4. Verify the running process:
If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.
<syntaxhighlight lang="bash">ps aux | grep voipmonitor</syntaxhighlight>
This command will show the running process and the exact command line arguments it was started with. Critically, ensure it is using the correct configuration file, for example: <code>--config-file /etc/voipmonitor.conf</code>. If it is not, there may be an issue with your startup script.


== Is Network Traffic Reaching the Server? ==
=== RTP Not Associated with Call ===
If the service is running, verify if the VoIP packets (SIP/RTP) are actually arriving at the server's network interface. The best tool for this is <code>tshark</code> (the command-line version of Wireshark).


;1. Install tshark:
'''Symptom''': Audio plays in sniffer but not in GUI, or RTP listed under wrong call.
<syntaxhighlight lang="bash">
 
# For Debian/Ubuntu
'''Possible causes''':
apt-get update && apt-get install tshark


# For CentOS/RHEL/AlmaLinux
'''1. SIP and RTP on different interfaces/VLANs''':
yum install wireshark
<syntaxhighlight lang="ini">
# voipmonitor.conf - enable automatic RTP association
auto_enable_use_blocks = yes
</syntaxhighlight>
</syntaxhighlight>


;2. Listen for SIP traffic on the correct interface:
'''2. NAT not configured''':
Replace <code>eth0</code> with the interface name you have configured in <code>voipmonitor.conf</code>.
<syntaxhighlight lang="ini">
<syntaxhighlight lang="bash">
# voipmonitor.conf - for NAT scenarios
tshark -i eth0 -Y "sip || rtp" -n
natalias = <public_ip> <private_ip>
 
# If not working, try reversed order:
natalias = <private_ip> <public_ip>
</syntaxhighlight>
</syntaxhighlight>
*'''If you see a continuous stream of SIP and RTP packets''', it means traffic is reaching the server, and the problem is likely in VoIPmonitor's configuration (see [[#Check the VoIPmonitor Configuration|Check the VoIPmonitor Configuration]]).
*'''If you see NO packets''', the problem lies with your network configuration. See [[#Troubleshoot Network and Interface Configuration|Troubleshoot Network and Interface Configuration]].


;3. Advanced: Capture to PCAP File for Definitive Testing
'''3. External device modifying media ports''':
Live monitoring with tshark is useful for observation, but capturing traffic to a .pcap file during a test call provides definitive evidence for troubleshooting intermittent issues or specific call legs.


'''Method 1: Using tcpdump (Recommended)'''
If SDP advertises one port but RTP arrives on different port (SBC/media server issue):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Start capture on the correct interface (replace eth0)
# Compare SDP ports vs actual RTP
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap port 5060
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u
</syntaxhighlight>
 
If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.
=== RTP Incorrectly Associated with Wrong Call (PBX Port Reuse) ===
'''Symptom''': RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.


# Or capture both SIP and RTP traffic:
'''Cause''': When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap "(port 5060 or udp)"


# Let it run while you make a test call with the missing call leg
'''Solution''': Enable <code>rtp_check_both_sides_by_sdp</code> to require verification of both source and destination IP:port against SDP:
# Press Ctrl+C to stop the capture
<syntaxhighlight lang="ini">
# voipmonitor.conf - require both source and destination to match SDP
rtp_check_both_sides_by_sdp = yes


# Analyze the capture file:
# Alternative (strict) mode - allows initial unverified packets
tshark -r /tmp/test_capture.pcap -Y "sip"
rtp_check_both_sides_by_sdp = strict
</syntaxhighlight>
</syntaxhighlight>


'''Method 2: Using tshark to capture to file'''
{{Warning|Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use <code>natalias</code> mappings or the <code>strict</code> setting to mitigate this.}}
<syntaxhighlight lang="bash">
=== Snaplen Truncation ===
# Start capture:
tshark -i eth0 -w /tmp/test_capture.pcap -f "tcp port 5060 or udp"


# Make your test call, then press Ctrl+C to stop
'''Symptom''': Large SIP messages truncated, incomplete headers.


# Analyze the capture:
'''Solution''':
tshark -r /tmp/test_capture.pcap -Y "sip" -V
<syntaxhighlight lang="ini">
# voipmonitor.conf - increase packet capture size
snaplen = 8192
</syntaxhighlight>
</syntaxhighlight>


'''Decision Tree for PCAP Analysis:'''
For Kamailio siptrace, also check <code>trace_msg_fragment_size</code> in Kamailio config. See [[Sniffer_configuration#snaplen|snaplen documentation]].
After capturing a test call known to have a missing leg:


* '''If SIP packets are missing from the .pcap file:'''
== PACKETBUFFER Saturation ==
** The problem is with your network mirroring configuration (SPAN/TAP port, AWS Traffic Mirroring, etc.)
** The packets never reached the VoIPmonitor sensor's network interface
** Fix the switch mirroring setup or infrastructure configuration first


* '''If SIP packets ARE present in the .pcap file but missing in the VoIPmonitor GUI:'''
'''Symptom''': Log shows <code>PACKETBUFFER: memory is FULL</code>, truncated RTP recordings.
** The problem is with VoIPmonitor's configuration or processing
** Packets reached the NIC but were not processed correctly
** Review [[#Check the VoIPmonitor Configuration|VoIPmonitor Configuration]] and [[#Check GUI Capture Rules (Causing Call Stops)|Capture Rules]]


'''Example Test Call Workflow:'''
{{Warning|This alert refers to VoIPmonitor's '''internal packet buffer''' (<code>max_buffer_mem</code>), '''NOT system RAM'''. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.}}
<syntaxhighlight lang="bash">
 
# 1. Start capture
'''Before testing solutions''', gather diagnostic data:
tcpdump -i eth0 -s 0 -w /tmp/test.pcap "sip and host 10.0.1.100"
* Check sensor logs: <code>/var/log/syslog</code> (Debian/Ubuntu) or <code>/var/log/messages</code> (RHEL/CentOS)
* Generate debug log via GUI: '''Tools → Generate debug log'''


# 2. Make a test call from phone at 10.0.1.100 to 10.0.2.200
=== Diagnose: I/O vs CPU Bottleneck ===
#    (a call that you know should have recordings but is missing)


# 3. Stop capture (Ctrl+C)
{{Warning|Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to '''confirm''' findings, not as the primary diagnostic method.}}


# 4. Check for the specific call's Call-ID
==== Step 1: Check IO[] Metrics (v2026.01.3+) ====
tshark -r /tmp/test.pcap -Y "sip" -T fields -e sip.Call-ID


# 5. Verify if packets for both A-leg and B-leg exist
'''Starting with version 2026.01.3''', VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status:
tshark -r /tmp/test.pcap -Y "sip && ip.addr == 10.0.1.100"


# 6. Compare results with VoIPmonitor GUI
<syntaxhighlight lang="text">
#    - If packets found in .pcap: VoIPmonitor software issue
[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]
#    - If packets missing from .pcap: Network mirroring issue
</syntaxhighlight>
</syntaxhighlight>


== Troubleshoot Network and Interface Configuration ==
'''Quick interpretation:'''
If <code>tshark</code> shows no traffic, it means the packets are not being delivered to the operating system correctly.
{| class="wikitable"
|-
! Metric !! Meaning !! Problem Indicator
|-
| '''C''' (Capacity) || % of disk's sustainable throughput used || '''C ≥ 80% = Warning''', '''C ≥ 95% = Saturated'''
|-
| '''L''' (Latency) || Current write latency in ms || '''L ≥ 3× B''' (baseline) = Saturated
|-
| '''U''' (Utilization) || % time disk is busy || '''U > 90%''' = Disk at limit
|}


;1. Check if the interface is UP:
'''If you see <code>DISK_SAT</code> or <code>WARN</code> after IO[]:'''
Ensure the network interface is active.
<syntaxhighlight lang="text">
<syntaxhighlight lang="bash">ip link show eth0</syntaxhighlight>
IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT
The output should contain the word <code>UP</code>. If it doesn't, bring it up with:
</syntaxhighlight>
<syntaxhighlight lang="bash">ip link set dev eth0 up</syntaxhighlight>


;2. Check for Promiscuous Mode (for SPAN/RSPAN Mirrored Traffic):
→ This confirms I/O bottleneck. Skip to [[#Solution:_I.2FO_Bottleneck|I/O Bottleneck Solutions]].
'''Important:''' Promiscuous mode requirements depend on your traffic mirroring method:


* '''SPAN/RSPAN (Layer 2 mirroring):''' The network interface '''must''' be in promiscuous mode. Mirrored packets retain their original MAC addresses, so the interface would normally ignore them. Promiscuous mode forces the interface to accept all packets regardless of destination MAC.
'''For older versions or additional confirmation''', continue with the steps below.


* '''ERSPAN/GRE/TZSP/VXLAN (Layer 3 tunnels):''' Promiscuous mode is '''NOT required'''. These tunneling protocols encapsulate the mirrored traffic inside IP packets that are addressed directly to the sensor's IP address. The operating system receives these packets normally, and VoIPmonitor automatically decapsulates them to extract the inner SIP/RTP traffic.
{{Note|See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|Syslog Status Line - IO[] section]] for detailed field descriptions.}}


For SPAN/RSPAN deployments, check the current promiscuous mode status:
==== Step 2: Read the Full Syslog Status Line ====
<syntaxhighlight lang="bash">ip link show eth0</syntaxhighlight>
Look for the <code>PROMISC</code> flag.


Enable promiscuous mode manually if needed:
VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool:
<syntaxhighlight lang="bash">ip link set eth0 promisc on</syntaxhighlight>
If this solves the problem, you should make the change permanent. The <code>install-script.sh</code> for the sensor usually attempts to do this, but it can fail.


;3. Verify Your SPAN/Mirror/TAP Configuration:
<syntaxhighlight lang="bash">
This is the most common cause of no traffic. Double-check your network switch or hardware tap configuration to ensure:
# Monitor in real-time
* The correct source ports (where your PBX/SBC is connected) are being monitored.
journalctl -u voipmonitor -f
* The correct destination port (where your VoIPmonitor sensor is connected) is configured.
# or
* If you are monitoring traffic across different VLANs, ensure your mirror port is configured to carry all necessary VLAN tags (often called "trunk" mode).
tail -f /var/log/syslog | grep voipmonitor
</syntaxhighlight>


== Check the VoIPmonitor Configuration ==
'''Example status line:'''
If <code>tshark</code> sees traffic but VoIPmonitor does not, the problem is almost certainly in <code>voipmonitor.conf</code>.
<syntaxhighlight lang="text">
calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB
</syntaxhighlight>


;1. Check the <code>interface</code> directive:
'''Key metrics for bottleneck identification:'''
:Make sure the <code>interface</code> parameter in <code>/etc/voipmonitor.conf</code> exactly matches the interface where you see traffic with <code>tshark</code>. For example: <code>interface = eth0</code>.


;2. Check the <code>sipport</code> directive:
{| class="wikitable"
:By default, VoIPmonitor only listens on port 5060. If your PBX uses a different port for SIP, you must add it. For example:
|-
:<code>sipport = 5060,5080</code>
! Metric !! What It Indicates !! I/O Bottleneck Sign !! CPU Bottleneck Sign
|-
| <code>heap[A&#124;B&#124;C]</code> || Buffer fill % (primary / secondary / processing) || High A with low t0CPU || High A with high t0CPU
|-
| <code>t0CPU[X%]</code> || Packet capture thread (single-core, cannot parallelize) || Low (<50%) || High (>80%)
|-
| <code>comp[X]</code> || Active compression threads || Very high (maxed out) || Normal
|-
| <code>SQLq[C:X M:Y]</code> || Pending SQL queries || Growing = database bottleneck || Stable
|-
| <code>tacCPU[...]</code> || TAR compression threads || All near 100% = compression bottleneck || Normal
|}


;3. Check for a restrictive <code>filter</code>:
'''Interpretation flowchart:'''
:If you have a BPF <code>filter</code> configured, ensure it is not accidentally excluding the traffic you want to see. For debugging, try commenting out the <code>filter</code> line entirely and restarting the sensor.


== Check GUI Capture Rules (Causing Call Stops) ==
<kroki lang="mermaid">
If <code>tshark</code> sees SIP traffic and the sniffer configuration appears correct, but the probe stops processing calls or shows traffic only on the network interface, GUI capture rules may be the culprit.
graph TD
    A[heap values rising] --> B{Check t0CPU}
    B -->|t0CPU > 80%| C[CPU Bottleneck]
    B -->|t0CPU < 50%| D{Check comp and tacCPU}
    D -->|comp maxed, tacCPU high| E[I/O Bottleneck<br/>Disk cannot keep up with writes]
    D -->|comp normal| F{Check SQLq}
    F -->|SQLq growing| G[Database Bottleneck]
    F -->|SQLq stable| H[Mixed/Other Issue]


Capture rules configured in the GUI can instruct the sniffer to ignore ("skip") all processing for matched calls. This includes calls matching specific IP addresses or telephone number prefixes.
    C --> C1[Solution: CPU optimization]
    E --> E1[Solution: Faster storage]
    G --> G1[Solution: MySQL tuning]
</kroki>


;1. Review existing capture rules:
==== Step 3: Linux I/O Diagnostics ====
:Navigate to '''GUI → Capture rules''' and examine all rules for any that might be blocking your traffic.
:Look specifically for rules with the '''Skip''' option set to '''ON''' (displayed as "Skip: ON"). The Skip option instructs the sniffer to completely ignore matching calls (no files, RTP analysis, or CDR creation).


;2. Test by temporarily removing all capture rules:
Use these standard Linux tools to confirm I/O bottleneck:
:To isolate the issue, first create a backup of your GUI configuration:
:* Navigate to '''Tools → Backup & Restore → Backup GUI → Configuration tables'''
:* This saves your current settings including capture rules
:* Delete all capture rules from the GUI
:* Click the '''Apply''' button to save changes
:* Reload the sniffer by clicking the green '''"reload sniffer"''' button in the control panel
:* Test if calls are now being processed correctly
:* If resolved, restore the configuration from the backup and systematically investigate the rules to identify the problematic one


;3. Identify the problematic rule:
'''Install required tools:'''
:* After restoring your configuration, remove rules one at a time and reload the sniffer after each removal
<syntaxhighlight lang="bash">
:* When calls start being processed again, you have identified the problematic rule
# Debian/Ubuntu
:* Review the rule's match criteria (IP addresses, prefixes, direction) against your actual traffic pattern
apt install sysstat iotop ioping
:* Adjust the rule's conditions or Skip setting as needed


;4. Verify rules are reloaded:
# CentOS/RHEL
:After making changes to capture rules, remember that changes are '''not automatically applied''' to the running sniffer. You must click the '''"reload sniffer"''' button in the control panel, or the rules will continue using the previous configuration.
yum install sysstat iotop ioping
</syntaxhighlight>


For more information on capture rules, see [[Capture_rules]].
'''2a) iostat - Disk utilization and wait times'''
<syntaxhighlight lang="bash">
# Run for 10 intervals of 2 seconds
iostat -xz 2 10
</syntaxhighlight>


== Troubleshoot MySQL/MariaDB Database Connection Errors ==
'''Key output columns:'''
If you see "Connection refused (111)" errors or the sensor cannot connect to your database server, the issue is with the MySQL/MariaDB database connection configuration in <code>/etc/voipmonitor.conf</code>.
<syntaxhighlight lang="text">
Device  r/s    w/s  rkB/s  wkB/s  await  %util
sda    12.50  245.30  50.00  1962.40  45.23  98.50
</syntaxhighlight>


Error 111 (Connection refused) indicates that the database server is reachable on the network, but no MySQL/MariaDB service is listening on the specified port, or the connection is being blocked by a firewall. This commonly happens after migrations when the database server IP address has changed.
{| class="wikitable"
|-
! Column !! Description !! Problem Indicator
|-
| <code>%util</code> || Device utilization percentage || '''> 90%''' = disk saturated
|-
| <code>await</code> || Average I/O wait time (ms) || '''> 20ms''' for SSD, '''> 50ms''' for HDD = high latency
|-
| <code>w/s</code> || Writes per second || Compare with disk's rated IOPS
|}


;1. Check for database connection errors in sensor logs:
'''2b) iotop - Per-process I/O usage'''
Verify the specific error from the sensor process:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu (systemd journal)
# Show I/O by process (run as root)
journalctl -u voipmonitor --since "1 hour ago" | grep -iE "mysql|database|connection|can.t connect"
iotop -o
</syntaxhighlight>


# For systems using traditional syslog
Look for <code>voipmonitor</code> or <code>mysqld</code> dominating I/O. If voipmonitor shows high DISK WRITE but system <code>%util</code> is 100%, disk cannot keep up.
tail -f /var/log/syslog | grep voipmonitor | grep -iE "mysql|database|connection"


# For CentOS/RHEL/AlmaLinux
'''2c) ioping - Quick latency check'''
tail -f /var/log/messages | grep voipmonitor | grep -iE "mysql|database|connection"
<syntaxhighlight lang="bash">
# Test latency on VoIPmonitor spool directory
cd /var/spool/voipmonitor
ioping -c 20 .
</syntaxhighlight>
</syntaxhighlight>


Look for errors like:
'''Expected results:'''
* <code>Can't connect to MySQL server on '192.168.1.10' (111)</code> - Connection refused (wrong host/port)
{| class="wikitable"
* <code>Access denied for user 'root'@'localhost'</code> - Wrong username/password
|-
* <code>Unknown database 'voipmonitor'</code> - Wrong database name
! Storage Type !! Healthy Latency !! Problem Indicator
|-
| NVMe SSD || < 0.5 ms || > 2 ms
|-
| SATA SSD || < 1 ms || > 5 ms
|-
| HDD (7200 RPM) || < 10 ms || > 30 ms
|}


;2. Verify database connection parameters in <code>voipmonitor.conf</code>:
==== Step 4: Linux CPU Diagnostics ====
Open <code>/etc/voipmonitor.conf</code> and check the MySQL connection settings:
 
<syntaxhighlight lang="ini">
'''3a) top - Overall CPU usage'''
# Database Connection Parameters
<syntaxhighlight lang="bash">
mysqlhost = 192.168.1.10      # IP address or hostname of MySQL/MariaDB server
# Press '1' to show per-core CPU
mysqlport = 3306              # TCP port of the database server (default: 3306)
top
mysqlusername = root          # Database username
mysqlpassword = your_password  # Database password
mysqldatabase = voipmonitor    # Database name
</syntaxhighlight>
</syntaxhighlight>


Key points:
Look for:
* <code>mysqlhost</code>: Should be the IP address or hostname of the database server. After migration, this may have changed.
* Individual CPU core at 100% (t0 thread is single-threaded)
* <code>mysqlport</code>: Port 3306 is the default, but your database might use a different port.
* High <code>%wa</code> (I/O wait) vs high <code>%us/%sy</code> (CPU-bound)
* <code>mysqlusername</code>: Database user must have proper privileges.
* <code>mysqlpassword</code>: Ensure there are no typos or special character issues (surround with single quotes if needed).
* <code>mysqldatabase</code>: Database must exist on the server.


;3. Test MySQL connectivity from the sensor host:
'''3b) Verify voipmonitor threads'''
Use the <code>mysql</code> command-line client to test if the database is reachable from the sensor:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test basic TCP connectivity (replace IP and port as needed)
# Show voipmonitor threads with CPU usage
nc -zv 192.168.1.10 3306
top -H -p $(pgrep voipmonitor)
</syntaxhighlight>


# Or using telnet
If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).
telnet 192.168.1.10 3306
 
==== Step 5: Decision Matrix ====
 
{| class="wikitable"
|-
! Observation !! Likely Cause !! Go To
|-
| <code>heap</code> high, <code>t0CPU</code> > 80%, iostat <code>%util</code> low || '''CPU Bottleneck''' || [[#Solution: CPU Bottleneck|CPU Solution]]
|-
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> > 90% || '''I/O Bottleneck''' || [[#Solution: I/O Bottleneck|I/O Solution]]
|-
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> < 50%, <code>SQLq</code> growing || '''Database Bottleneck''' || [[#SQL Queue Overload|Database Solution]]
|-
| <code>heap</code> normal, <code>comp</code> maxed, <code>tacCPU</code> all ~100% || '''Compression Bottleneck''' (type of I/O) || [[#Solution: I/O Bottleneck|I/O Solution]]
|}
 
==== Step 6: Confirmation Test (Optional) ====
 
After identifying the likely cause with the tools above, you can confirm with a storage disable test:
 
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf - temporarily disable all storage
savesip = no
savertp = no
savertcp = no
savegraph = no
</syntaxhighlight>
</syntaxhighlight>


If you see "Connection refused", the database service is not running or not listening on that port.
;4. Test MySQL authentication using credentials from <code>voipmonitor.conf</code>:
Use the same credentials configured in <code>voipmonitor.conf</code> to verify they work:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mysql -h 192.168.1.10 -P 3306 -u root -p'your_password' voipmonitor
systemctl restart voipmonitor
# Monitor for 5-10 minutes during peak traffic
journalctl -u voipmonitor -f | grep heap
</syntaxhighlight>
</syntaxhighlight>


Commands to run inside mysql client to verify:
* If <code>heap</code> values drop to near zero → confirms '''I/O bottleneck'''
<syntaxhighlight lang="sql">
* If <code>heap</code> values remain high → confirms '''CPU bottleneck'''
-- Check if connected correctly
 
SELECT USER(), CURRENT_USER();
{{Warning|Remember to re-enable storage after testing! This test causes call recordings to be lost.}}
 
=== Solution: I/O Bottleneck ===


-- Check database exists
{{Note|If you see <code>IO[...] DISK_SAT</code> or <code>WARN</code> in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|IO[] Metrics]] for details.}}
SHOW DATABASES LIKE 'voipmonitor';


-- Test write access
'''Quick confirmation (for older versions):'''
USE voipmonitor;
 
SHOW TABLES;
Temporarily save only RTP headers to reduce disk write load:
EXIT;
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
savertp = header
</syntaxhighlight>
</syntaxhighlight>


;5. Compare with a working sensor's configuration:
Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.
If you have other sensors that successfully connect to the database, compare their configuration files:
 
'''Check storage health before upgrading:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Compare database settings between working and failing sensors
# Check drive health
diff <(grep -E "^mysql" /etc/voipmonitor.conf) <(grep -E "^mysql" /path/to/working/sensor/voipmonitor.conf)
smartctl -a /dev/sda
 
# Check for I/O errors in system logs
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"
</syntaxhighlight>
</syntaxhighlight>


Common discrepancies after migration:
Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.
* Wrong database server IP address (<code>mysqlhost</code>)
 
* Wrong database port (<code>mysqlport</code>)
'''Storage controller cache settings:'''
* Different password due to migration to new database server
{| class="wikitable"
* Using <code>localhost</code> vs actual IP address
|-
! Storage Type !! Recommended Cache Mode
|-
| HDD / NAS || WriteBack (requires battery-backed cache)
|-
| SSD || WriteThrough (or WriteBack with power loss protection)
|}
 
Use vendor-specific tools to configure cache policy (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>).
 
'''Storage upgrades (in order of effectiveness):'''
{| class="wikitable"
|-
! Solution !! IOPS Improvement !! Notes
|-
| '''NVMe SSD''' || 50-100x vs HDD || Best option, handles 10,000+ concurrent calls
|-
| '''SATA SSD''' || 20-50x vs HDD || Good option, handles 5,000+ concurrent calls
|-
| '''RAID 10 with BBU''' || 5-10x vs single disk || Enable WriteBack cache (requires battery backup)
|-
| '''Separate storage server''' || Variable || Use [[Sniffer_distributed_architecture|client/server mode]]
|}


;6. Check firewall and network connectivity:
'''Filesystem tuning (ext4):'''
Ensure the sensor can reach the database server and the required port is open:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test network reachability
# Check current mount options
ping -c 4 192.168.1.10
mount | grep voipmonitor
 
# Recommended mount options for /var/spool/voipmonitor
# Add to /etc/fstab: noatime,data=writeback,barrier=0
# WARNING: barrier=0 requires battery-backed RAID
</syntaxhighlight>


# Check if MySQL port is reachable
'''Verify improvement:'''
nc -zv 192.168.1.10 3306
<syntaxhighlight lang="bash">
# After changes, monitor iostat
iostat -xz 2 10
# %util should drop below 70%, await should decrease
</syntaxhighlight>


# Check firewall rules (if using firewalld)
=== Solution: CPU Bottleneck ===
firewall-cmd --list-ports


# Check firewall rules (if using iptables)
==== Identify CPU Bottleneck Using Manager Commands ====
iptables -L -n | grep 3306
</syntaxhighlight>


If the port is blocked, you may need to:
VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.
* Open port 3306 in the firewall on the database server
* Configure network ACLs or security groups (for cloud deployments)
* Check VPN/SSH tunnel configurations


;7. Verify MySQL/MariaDB service is running:
'''Connect to manager interface:'''
On the database server, check if the service is active:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check MySQL/MariaDB service status
# Via Unix socket (local, recommended)
systemctl status mariadb    # or systemctl status mysql
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket


# Restart service if needed
# Via TCP port 5029 (remote or local)
systemctl restart mariadb
echo 'sniffer_threads' | nc 127.0.0.1 5029
 
# Monitor continuously (every 2 seconds)
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"
</syntaxhighlight>
 
{{Note|1=TCP port 5029 is encrypted by default. For unencrypted access, set <code>manager_enable_unencrypted = yes</code> in voipmonitor.conf (security risk on public networks).}}


# Check which port MySQL is listening on
'''Example output:'''
ss -tulpn | grep mysql
<syntaxhighlight lang="text">
# or
t0 - binlog1 fifo pcap read          (  12345) :  78.5  FIFO  99    1234
netstat -tulpn | grep mysql
t2 - binlog1 pb write                (  12346) :  12.3              456
rtp thread binlog1 binlog1 0        (  12347) :  8.1              234
rtp thread binlog1 binlog1 1        (  12348) :  6.2              198
t1 - binlog1 call processing        (  12349) :  4.5              567
tar binlog1 compression 0            (  12350) :  3.2                89
</syntaxhighlight>
</syntaxhighlight>


MySQL should be listening on the interface and port specified in your <code>voipmonitor.conf</code> <code>mysqlhost</code> and <code>mysqlport</code> settings.
'''Column interpretation:'''
{| class="wikitable"
|-
! Column !! Description
|-
| Thread name || Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
|-
| (TID) || Linux thread ID (useful for <code>top -H -p TID</code>)
|-
| CPU % || Current CPU usage percentage - '''key metric'''
|-
| Sched || Scheduler type (FIFO = real-time, empty = normal)
|-
| Priority || Thread priority
|-
| CS/s || Context switches per second
|}


;8. Apply configuration changes and restart the sensor:
'''Critical threads to watch:'''
After correcting the database connection settings in <code>/etc/voipmonitor.conf</code>:
{| class="wikitable"
<syntaxhighlight lang="bash">
|-
# Restart the VoIPmonitor service to apply changes
! Thread !! Role !! If at 90-100%
systemctl restart voipmonitor
|-
| '''t0''' (pcap read) || Packet capture from NIC || '''Single-core limit reached!''' Cannot parallelize. Need DPDK/Napatech.
|-
| '''t2''' (pb write) || Packet buffer processing || Processing bottleneck. Check t2CPU breakdown.
|-
| '''rtp thread''' || RTP packet processing || Threads auto-scale. If still saturated, consider DPDK/Napatech.
|-
| '''tar compression''' || PCAP archiving || I/O bottleneck (compression waiting for disk)
|-
| '''mysql store''' || Database writes || Database bottleneck. Check SQLq metric.
|}


# Alternatively, reload without full restart (if supported in your version)
{{Warning|If '''t0 thread is at 90-100%''', you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and '''cannot be parallelized'''. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:
echo 'reload' | nc 127.0.0.1 5029
* '''Reduce captured traffic''' using <code>interface_ip_filter</code> or BPF <code>filter</code>
* '''Use kernel bypass''' ([[DPDK]] or [[Napatech]]) which eliminates kernel overhead entirely}}


# Verify the service started successfully
==== Interpreting t2CPU Detailed Breakdown ====
systemctl status voipmonitor


# Check logs for database connection confirmation
The syslog status line shows <code>t2CPU</code> with detailed sub-metrics:
journalctl -u voipmonitor -n 20
<syntaxhighlight lang="text">
t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
</syntaxhighlight>
</syntaxhighlight>


Look for a successful database connection message in the logs, which typically appears within the first few seconds after startup.
{| class="wikitable"
|-
! Code !! Function !! High Value Indicates
|-
| '''pb''' || Packet buffer output || Buffer management overhead
|-
| '''d''' || Dispatch || Structure creation bottleneck
|-
| '''s''' || SIP parsing || Complex/large SIP messages
|-
| '''e''' || Entity lookup || Call table lookup overhead
|-
| '''c''' || Call processing || Call state machine processing
|-
| '''g''' || Register processing || High REGISTER volume
|-
| '''r, rm, rh, rd''' || RTP processing stages || High RTP volume (threads auto-scale)
|}


;9. Common Troubleshooting Scenarios:
'''Thread auto-scaling:''' VoIPmonitor automatically spawns additional threads when load increases:
* If '''d''' > 50% → SIP parsing thread ('''s''') starts
* If '''s''' > 50% → Entity lookup thread ('''e''') starts
* If '''e''' > 50% → Call/register/RTP threads start


<b>Scenario A: Database server IP changed after migration</b>
==== Configuration for High Traffic (>10,000 calls/sec) ====
* Symptom: "Can't connect to MySQL server on '10.1.1.10' (111)"
* Fix: Update <code>mysqlhost</code> in <code>/etc/voipmonitor.conf</code> to the new database server IP


<b>Scenario B: Wrong MySQL username or password</b>
<syntaxhighlight lang="ini">
* Symptom: "Access denied for user 'user'@'host'"
# /etc/voipmonitor.conf
* Fix: Verify credentials match the database server's user permissions, update <code>mysqlusername</code> and <code>mysqlpassword</code>


<b>Scenario C: Database service not running</b>
# Increase buffer to handle processing spikes (value in MB)
* Symptom: "Connection refused (111)" or "Connection timed out"
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
* Fix: Start MySQL/MariaDB service on the database server: <code>systemctl start mariadb</code>
# Larger buffer absorbs I/O and CPU spikes without packet loss
max_buffer_mem = 10000


<b>Scenario D: Firewall blocking port 3306</b>
# Use IP filter instead of BPF (more efficient)
* Symptom: "Connection refused" when testing with <code>nc</code>, but MySQL is running
interface_ip_filter = 10.0.0.0/8
* Fix: Open port 3306 in firewall, or configure MySQL to allow connections from the sensor's IP in <code>user</code> table
interface_ip_filter = 192.168.0.0/16
# Comment out any 'filter' parameter
</syntaxhighlight>


<b>Scenario E: Localhost vs remote connection confusion</b>
==== CPU Optimizations ====
* Symptom: Connection works locally but fails from sensor
* Fix: Ensure <code>mysqlhost</code> uses the actual IP address (not <code>localhost</code> or <code>127.0.0.1</code>) if the sensor is on a different host


For more detailed information about all <code>mysql*</code> configuration parameters, see [[Sniffer_configuration#Database_Configuration]].
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf


== Check for Storage Hardware Errors (HEAP FULL / packetbuffer Issues) ==
# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
If the sensor is crashing with "HEAP FULL" errors or showing "packetbuffer: MEMORY IS FULL" messages, you must distinguish between '''actual storage hardware failures''' (requires disk replacement) and '''performance bottlenecks''' (requires tuning).
jitterbuffer_f1 = no
jitterbuffer_f2 = yes
jitterbuffer_adapt = no


;1. Check kernel message buffer for storage errors:
# If MOS metrics are not needed at all, disable everything:
<syntaxhighlight lang="bash">dmesg -T | grep -iE "ext4-fs error|i/o error|nvram warning|ata.*failed|sda.*error|disk failure|smart error" | tail -50</syntaxhighlight>
# jitterbuffer_f1 = no
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no
</syntaxhighlight>


Look for these hardware error indicators:
==== Kernel Bypass Solutions (Extreme Loads) ====
* <code>ext4-fs error</code> - Filesystem corruption or disk failure
* <code>I/O error</code> or <code>BUG: soft lockup</code> - Disk read/write failures
* <code>NVRAM WARNING: nvram_check: failed</code> - RAID controller battery/capacitor issues
* <code>ata.*: FAILED</code> - Hard drive SMART failure
* <code>Buffer I/O error</code> - Disk unable to complete operations


If you see ANY of these errors:
When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:
* The storage subsystem is failing and likely needs hardware replacement
* Do not attempt performance tuning - replace the failed disk/RAID first
* Check SMART status: <code>smartctl -a /dev/sda</code>
* Check RAID health: <code>cat /proc/mdstat</code> or RAID controller tools


;2. If dmesg is clean of errors → Performance Bottleneck:
{| class="wikitable"
If the kernel logs show no storage errors, the issue is a performance bottleneck (disk too slow, network latency, etc.).
|-
! Solution !! Type !! CPU Reduction !! Use Case
|-
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
|-
| '''[[Napatech]]''' || Hardware SmartNIC || >97% (< 3% at 10Gbit) || Extreme performance requirements
|}
 
==== Verify Improvement ====


<b>Check disk I/O performance:</b>
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Current I/O wait (should be < 10% normally)
# Monitor thread CPU after changes
iostat -x 5
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"
 
# Or monitor syslog
journalctl -u voipmonitor -f
# t0CPU should drop, heap values should stay < 20%
</syntaxhighlight>
 
{{Note|1=After changes, monitor syslog <code>heap[A&#124;B&#124;C]</code> values - should stay below 20% during peak traffic. See [[Syslog_Status_Line]] for detailed metric explanations.}}


# Detailed disk stats
== Storage Hardware Failure ==
dstat -d


# Real-time disk latency
'''Symptom''': Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.
ioping -c 10 .
</syntaxhighlight>


<b>Check NFS latency (if using NFS storage):</b>
'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test NFS read/write latency
# Check disk health
time dd if=/dev/zero of=/var/spool/voipmonitor/testfile bs=1M count=100
smartctl -a /dev/sda
time cat /var/spool/voipmonitor/testfile > /dev/null
rm /var/spool/voipmonitor/testfile


# Check NFS mount options
# Check RAID status (if applicable)
mount | grep nfs
cat /proc/mdstat
mdadm --detail /dev/md0
</syntaxhighlight>
</syntaxhighlight>


<b>Common performance solutions:</b>
Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.
* Use SSD/NVMe for VoIPmonitor spool directory
* Ensure proper NIC queue settings for high-throughput NFS
* Check network switch port configuration for NFS
* Review [[Scaling]] guide for detailed optimization


See also [[IO_Measurement]] for comprehensive disk benchmarking tools.
== OOM (Out of Memory) ==


== Check for OOM (Out of Memory) Issues ==
=== Identify OOM Victim ===
If VoIPmonitor suddenly stops processing CDRs and a service restart temporarily restores functionality, the system may be experiencing OOM (Out of Memory) killer events. The Linux OOM killer terminates processes when available RAM is exhausted, and MySQL (<code>mysqld</code>) is a common target due to its memory-intensive nature.


;1. Check for OOM killer events in kernel logs:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Check for OOM kills
grep -i "out of memory\|killed process" /var/log/syslog | tail -20
dmesg | grep -i "out of memory\|oom\|killed process"
journalctl --since "1 hour ago" | grep -i oom
</syntaxhighlight>


# For CentOS/RHEL/AlmaLinux
=== MySQL Killed by OOM ===
grep -i "out of memory\|killed process" /var/log/messages | tail -20


# Also check dmesg:
Reduce InnoDB buffer pool:
dmesg | grep -i "killed process" | tail -10
<syntaxhighlight lang="ini">
# /etc/mysql/my.cnf
innodb_buffer_pool_size = 2G  # Reduce from default
</syntaxhighlight>
</syntaxhighlight>
Typical OOM killer messages look like:
 
<syntaxhighlight lang="text">
=== Voipmonitor Killed by OOM ===
Out of memory: Kill process 1234 (mysqld) score 123 or sacrifice child
 
Killed process 1234 (mysqld) total-vm: 12345678kB, anon-rss: 1234567kB
Reduce buffer sizes in voipmonitor.conf:
<syntaxhighlight lang="ini">
max_buffer_mem = 2000  # Reduce from default
ringbuffer = 50        # Reduce from default
</syntaxhighlight>
</syntaxhighlight>


;2. Monitor current memory usage:
=== Runaway External Process ===
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check available memory (look for low 'available' or 'free' values)
# Find memory-hungry processes
free -h
ps aux --sort=-%mem | head -20
 
# Kill orphaned/runaway process
kill -9 <PID>
</syntaxhighlight>
For servers limited to '''16GB RAM''' or when experiencing repeated MySQL OOM kills:


# Check per-process memory usage (sorted by RSS)
<syntaxhighlight lang="ini">
ps aux --sort=-%mem | head -15
# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G


# Check MySQL memory usage in bytes
# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
cat /proc/$(pgrep mysqld)/status | grep -E "VmSize|VmRSS"
innodb_flush_log_at_trx_commit = 2
</syntaxhighlight>
</syntaxhighlight>
Warning signs:
* '''Available memory consistently below 500MB during operation'''
* '''MySQL consuming most of the available RAM'''
* '''Swap usage near 100% (if swap is enabled)'''
* '''Frequent process restarts without clear error messages'''


;3. First Fix: Check and correct innodb_buffer_pool_size:
Restart MySQL after changes:
Before upgrading hardware, verify that <code>innodb_buffer_pool_size</code> is not set too high. This is a common cause of OOM incidents. If MySQL/MariaDB is consuming most of the available RAM, the buffer pool size is likely configured incorrectly for your system.
<syntaxhighlight lang="bash">
systemctl restart mysql
# or
systemctl restart mariadb
</syntaxhighlight>
=== SQL Queue Growth from Non-Call Data ===


'''Calculate the correct buffer pool size:'''
If <code>sip-register</code>, <code>sip-options</code>, or <code>sip-subscribe</code> are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.
For a server running both VoIPmonitor and MySQL on the same host:
<syntaxhighlight lang="text">
Formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS & services overhead - safety margin) / 2


Example for a 32GB server:
{{Warning|1=Even with reduced <code>innodb_buffer_pool_size</code>, SQL queue will grow indefinitely without cleanup of non-call data.}}
- Total RAM: 32GB
- VoIPmonitor process memory (check with ps aux): 2GB
- OS + other services overhead: 2GB
- Safety margin: ~25-30% of remaining RAM for other internal buffers


Calculation:
'''Solution: Enable automatic cleanup of old non-call data'''
Available for buffer pool = 32GB - 2GB - 2GB = 28GB
<syntaxhighlight lang="ini">
Recommended innodb_buffer_pool_size = 14G (approximately 50% of available memory)
# /etc/voipmonitor.conf
# cleandatabase=2555 automatically deletes partitions older than 7 years
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase = 2555
</syntaxhighlight>
</syntaxhighlight>


'''Edit the MariaDB configuration file:'''
Restart the sniffer after changes:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="bash">
# Common locations: /etc/mysql/my.cnf, /etc/mysql/mariadb.conf.d/50-server.cnf, /etc/my.cnf.d/
systemctl restart voipmonitor
</syntaxhighlight>


innodb_buffer_pool_size = 14G  # Adjust based on your calculation
{{Note|See [[Data_Cleaning]] for detailed configuration options and other <code>cleandatabase_*</code> parameters.}}
</syntaxhighlight>
== Service Startup Failures ==


'''Restart MariaDB to apply:'''
=== Interface No Longer Exists ===
<syntaxhighlight lang="bash">systemctl restart mariadb  # or systemctl restart mysql</syntaxhighlight>


If the OOM events stop after correcting <code>innodb_buffer_pool_size</code>, no hardware upgrade is needed.
After OS upgrade, interface names may change (eth0 → ensXXX):


;4. Second Fix: Reduce VoIPmonitor buffer memory usage:
<syntaxhighlight lang="bash">
In addition to MySQL memory consumption, VoIPmonitor itself allocates significant memory for packet buffers. The total buffer memory used by VoIPmonitor is calculated based on:
# Find current interface names
ip a


'''VoIPmonitor Buffer Memory Calculation:'''
# Update all config locations
* '''<code>ringbuffer</code>''': Ring buffer size in MB per interface (default: 50MB, recommended ≥500MB for >100 Mbit traffic)
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/
* '''<code>max_buffer_mem</code>''': Maximum buffer memory limit in MB (default: 2000MB)
* '''Number of sniffing interfaces''': Each interface gets its own ringbuffer allocation
* '''Total formula''': Approximate total = (ringbuffer × number of interfaces) + max_buffer_mem


If you are monitoring multiple interfaces (e.g., <code>interface = eth0,eth1,eth2</code>), each interface uses a separate ringbuffer. With the default ringbuffer of 50MB and 3 interfaces, that's 150MB plus max_buffer_mem of 2000MB, totaling approximately 2150MB.
# Also check GUI: Settings → Sensors → Configuration
</syntaxhighlight>


'''To reduce VoIPmonitor memory usage:'''
=== Missing Dependencies ===


Edit <code>/etc/voipmonitor.conf</code> and decrease buffer settings:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
# Install common missing package
# Reduce ringbuffer for each interface (e.g., from 50 to 20)
apt install libpcap0.8  # Debian/Ubuntu
ringbuffer = 20
yum install libpcap    # RHEL/CentOS
</syntaxhighlight>


# Reduce maximum buffer memory (e.g., from 2000 to 1000)
== Network Interface Issues ==
max_buffer_mem = 1000


# Alternatively, reduce the number of sniffing interfaces if not all are needed
=== Promiscuous Mode ===
interface = eth0,eth1  # Instead of eth0,eth1,eth2,eth3
</syntaxhighlight>


After making changes, restart the VoIPmonitor service:
Required for SPAN port monitoring:
<syntaxhighlight lang="bash">systemctl restart voipmonitor</syntaxhighlight>
<syntaxhighlight lang="bash">
# Enable
ip link set eth0 promisc on


'''Important notes:'''
# Verify
* Reducing <code>ringbuffer</code> may increase packet loss during traffic spikes
ip link show eth0 | grep PROMISC
* Reducing <code>max_buffer_mem</code> affects how many packets can be buffered before being written to disk
</syntaxhighlight>
* Monitor packet loss statistics in the GUI after reducing buffers to ensure acceptable performance


;5. Solution: Increase physical memory (if buffer tuning is insufficient):
{{Note|Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.}}
If correcting both MySQL and VoIPmonitor buffer settings does not resolve the OOM issues, upgrade the server's physical RAM. After upgrading:
* Verify memory improvements with <code>free -h</code>
* Recalculate and adjust <code>innodb_buffer_pool_size</code> to utilize the additional memory
* Re-tune <code>ringbuffer</code> and <code>max_buffer_mem</code> for the new memory capacity
* Monitor for several days to ensure OOM events stop


== Sensor Upgrade Fails with "Permission denied" from /tmp ==
=== Interface Drops ===
If the sensor upgrade process fails with "Permission denied" errors when executing scripts from the <code>/tmp</code> directory, or the service fails to restart after upgrade, the <code>/tmp</code> partition may be mounted with the <code>noexec</code> flag.


The <code>noexec</code> mount option prevents execution of any script or binary from the <code>/tmp</code> directory for security reasons. However, the VoIPmonitor sensor upgrade process uses <code>/tmp</code> for temporary script execution.
<syntaxhighlight lang="bash">
# Check for drops
ip -s link show eth0 | grep -i drop


;1. Check the mount options for /tmp:
# If drops present, increase ring buffer
<syntaxhighlight lang="bash">mount | grep /tmp</syntaxhighlight>
ethtool -G eth0 rx 4096
Look for the <code>noexec</code> flag in the mount options. Output will show something like:
</syntaxhighlight>
<syntaxhighlight lang="text">/dev/sda2 on /tmp type ext4 rw,relatime,noexec,nosuid,nodev</syntaxhighlight>


;2. Remount /tmp without noexec (temporary fix):
=== Bonded/EtherChannel Interfaces ===
<syntaxhighlight lang="bash">mount -o remount,exec /tmp</syntaxhighlight>
Verify the change:
<syntaxhighlight lang="bash">mount | grep /tmp</syntaxhighlight>
The output should no longer contain <code>noexec</code>.


;3. Make the change permanent (edit /etc/fstab):
'''Symptom''': False packet loss when monitoring bond0 or br0.
Open the <code>/etc/fstab</code> file and locate the line corresponding to the <code>/tmp</code> mount point. Remove the <code>noexec</code> option from that line.
<syntaxhighlight lang="bash">nano /etc/fstab</syntaxhighlight>
Example:
<syntaxhighlight lang="text">
# Before:
/dev/sda2  /tmp  ext4  rw,relatime,noexec,nosuid,nodev  0 0


# After (remove noexec):
'''Solution''': Monitor physical interfaces, not logical:
/dev/sda2  /tmp  ext4  rw,relatime,nosuid,nodev  0 0
<syntaxhighlight lang="ini">
# voipmonitor.conf - use physical interfaces
interface = eth0,eth1
</syntaxhighlight>
</syntaxhighlight>
If <code>/tmp</code> is a separate partition, you may need to remount it for changes to take effect:
<syntaxhighlight lang="bash">mount -o remount /tmp</syntaxhighlight>


;4. Re-run the sensor upgrade:
=== Network Offloading Issues ===
After fixing the mount options, retry the sensor upgrade process.


== Check VoIPmonitor Logs for General Errors ==
'''Symptom''': Kernel errors like <code>bad gso: type: 1, size: 1448</code>
After addressing the specific issues above, check the system logs for other error messages from the sensor process that may reveal additional problems.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Disable offloading on capture interface
tail -f /var/log/syslog | grep voipmonitor
ethtool -K eth0 gso off tso off gro off lro off
 
# For CentOS/RHEL/AlmaLinux
tail -f /var/log/messages | grep voipmonitor
</syntaxhighlight>
</syntaxhighlight>


Look for errors like:
== Packet Ordering Issues ==
* "pcap_open_live(eth0) error: eth0: No such device" (Wrong interface name)
* "Permission denied" (The sensor is not running with sufficient privileges)
* Messages about connection issues (see [[#Troubleshoot MySQL/MariaDB Database Connection Errors|Troubleshoot MySQL/MariaDB Database Connection Errors]])
* Messages about dropping packets


== Benign Database Errors When Features Are Disabled ==
If SIP messages appear out of sequence:
Some VoIPmonitor features may generate harmless database errors when those features are not enabled in your configuration. These errors are '''benign''' (cause no harm to the system) and can be safely ignored.


=== Common Benign Error: Missing Tables ===
'''First''': Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See [[FAQ]].
If you see MySQL errors stating that a table does not exist (e.g., "Table 'voipmonitor.ss7' doesn't exist") even though the corresponding feature is disabled in your configuration, this is expected behavior.


;Common examples:
'''If genuine reordering''': Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see [[Napatech]]).
* Errors about the <code>ss7</code> table when <code>ss7 = no</code> in <code>voipmonitor.conf</code>
{{Note|For out-of-order packets in '''client/server mode''' (multiple sniffers), see [[Sniffer_distributed_architecture]] for <code>pcap_queue_dequeu_window_length</code> configuration.}}
* Errors about the <code>register_failed</code>, <code>register_state</code>, or <code>sip_msg</code> tables when those features are disabled
* Other similar errors for optional features that are not enabled


=== Solution: Ignore or Suppress in Monitoring ===
=== Solutions for SPAN/Mirroring Reordering ===
Since these errors indicate that a feature is simply not active, they do not impact system functionality. The recommended approach is:


# '''Do not change the configuration''' to fix these errors
If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):
# '''Add monitoring exceptions''' to suppress warnings for table-not-found errors (MySQL error code 1146)
# Configure alerting systems to exclude these specific SQL errors from notifications


This prevents alert noise while keeping your logs intact for real issues that require attention.
1. '''Configure switch to preserve packet order''': Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.


=== When to Take Action ===
2. '''Replace SPAN with TAP or packet broker''': Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.
You only need to take action in the following situations:
== Database Issues ==


* If you actually want to use the feature (enable the corresponding configuration option)
=== SQL Queue Overload ===
* If errors persist about tables for features that '''are''' explicitly enabled in <code>voipmonitor.conf</code>


Otherwise, these database errors are simply informational and confirm that optional features remain inactive as configured.
'''Symptom''': Growing <code>SQLq</code> metric, potential coredumps.


== Appendix: tshark Display Filter Syntax for SIP ==
<syntaxhighlight lang="ini">
When using <code>tshark</code> to analyze SIP traffic, it is important to use the '''correct Wireshark display filter syntax'''. Below are common filter examples:
# voipmonitor.conf - increase threads
mysqlstore_concat_limit_cdr = 1000
cdr_check_exists_callid = 0
</syntaxhighlight>


=== Basic SIP Filters ===
=== Error 1062 - Lookup Table Limit ===
<syntaxhighlight lang="bash">
# Show all SIP INVITE messages
tshark -r capture.pcap -Y "sip.Method == INVITE"


# Show all SIP messages (any method)
'''Symptom''': <code>Duplicate entry '16777215' for key 'PRIMARY'</code>
tshark -r capture.pcap -Y "sip"


# Show SIP and RTP traffic
'''Quick fix''':
tshark -r capture.pcap -Y "sip || rtp"
<syntaxhighlight lang="ini">
# voipmonitor.conf
cdr_reason_string_enable = no
</syntaxhighlight>
</syntaxhighlight>


=== Search for Specific Phone Number or Text ===
See [[Database_troubleshooting#Database_Error_1062_-_Lookup_Table_Auto-Increment_Limit|Database Troubleshooting]] for complete solution.
 
== Bad Packet Errors ==
 
'''Symptom''': <code>bad packet with ether_type 0xFFFF detected on interface</code>
 
'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Find calls containing a specific phone number (e.g., 5551234567)
# Run diagnostic (let run 30-60 seconds, then kill)
tshark -r capture.pcap -Y 'sip contains "5551234567"'
voipmonitor --check_bad_ether_type=eth0


# Find INVITE messages for a specific number
# Find and kill the diagnostic process
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"'
ps ax | grep voipmonitor
kill -9 <PID>
</syntaxhighlight>
</syntaxhighlight>


=== Extract Call-ID from Matching Calls ===
Causes: corrupted packets, driver issues, VLAN tagging problems. Check <code>ethtool -S eth0</code> for interface errors.
 
== Useful Diagnostic Commands ==
 
=== tshark Filters for SIP ===
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Get Call-ID for calls matching a phone number
# All SIP INVITEs
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"' -T fields -e sip.Call-ID
tshark -r capture.pcap -Y "sip.Method == INVITE"


# Get Call-ID along with From and To headers
# Find specific phone number
tshark -r capture.pcap -Y 'sip.Method == INVITE' -T fields -e sip.Call-ID -e sip.from.user -e sip.to.user
tshark -r capture.pcap -Y 'sip contains "5551234567"'
 
# Get Call-IDs
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID
 
# SIP errors (4xx, 5xx)
tshark -r capture.pcap -Y "sip.Status-Code >= 400"
</syntaxhighlight>
</syntaxhighlight>


=== Filter by IP Address ===
=== Interface Statistics ===
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# SIP traffic from a specific source IP
# Detailed NIC stats
tshark -r capture.pcap -Y "sip && ip.src == 192.168.1.100"
ethtool -S eth0


# SIP traffic between two hosts
# Watch packet rates
tshark -r capture.pcap -Y "sip && ip.addr == 192.168.1.100 && ip.addr == 10.0.0.50"
watch -n 1 'cat /proc/net/dev | grep eth0'
</syntaxhighlight>
</syntaxhighlight>


=== Filter by SIP Response Code ===
== See Also ==
<syntaxhighlight lang="bash">
 
# Show all 200 OK responses
* [[Sniffer_configuration]] - Configuration parameter reference
tshark -r capture.pcap -Y "sip.Status-Code == 200"
* [[Sniffer_distributed_architecture]] - Client/server deployment
* [[Capture_rules]] - GUI-based recording rules
* [[Sniffing_modes]] - SPAN, ERSPAN, GRE, TZSP setup
* [[Scaling]] - Performance optimization
* [[Database_troubleshooting]] - Database issues
* [[FAQ]] - Common questions and Wireshark display issues
 
 
 


# Show all 4xx and 5xx error responses
tshark -r capture.pcap -Y "sip.Status-Code >= 400"


# Show 486 Busy Here responses
tshark -r capture.pcap -Y "sip.Status-Code == 486"
</syntaxhighlight>


=== Important Syntax Notes ===
* '''Field names are case-sensitive:''' Use <code>sip.Method</code>, <code>sip.Call-ID</code>, <code>sip.Status-Code</code> (not <code>sip.method</code> or <code>sip.call-id</code>)
* '''String matching uses <code>contains</code>:''' Use <code>sip contains "text"</code> (not <code>sip.contains()</code>)
* '''Use double quotes for strings:''' <code>sip contains "number"</code> (not single quotes)
* '''Boolean operators:''' Use <code>&&</code> (and), <code>||</code> (or), <code>!</code> (not)


For a complete reference, see the [https://www.wireshark.org/docs/dfref/s/sip.html Wireshark SIP Display Filter Reference].


== AI Summary for RAG ==
== AI Summary for RAG ==
'''Summary:''' This document provides a troubleshooting guide for when the VoIPmonitor sensor is not capturing any calls. The first section covers verifying the service is running correctly using <code>systemctl status</code> and <code>ps</code>, including troubleshooting for when the service fails to start with "binary not found" error after a crash (check if the binary has been renamed with an underscore suffix, rename it back with <code>mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor</code>), and a specific procedure for sensors that become unresponsive after GUI updates (execute <code>killall voipmonitor</code>, <code>systemctl stop voipmonitor</code>, and <code>systemctl start voipmonitor</code>). The second section covers using <code>tshark</code> to confirm if SIP/RTP traffic is actually arriving at the server's network interface. The third section covers network-level issues, including an important distinction between Layer 2 mirroring (SPAN/RSPAN) which requires promiscuous mode, and Layer 3 tunneling (ERSPAN/GRE/TZSP/VXLAN) which does NOT require promiscuous mode. The fourth section focuses on checking the <code>voipmonitor.conf</code> file for common misconfigurations like the <code>interface</code>, <code>sipport</code>, or <code>filter</code> parameters. The fifth section addresses GUI capture rules with the "Skip" option that can cause probes to stop processing calls. The sixth section is a comprehensive guide for troubleshooting MySQL/MariaDB database connection errors, specifically "Connection refused (111)" errors. The seventh section explains how to diagnose HEAP FULL / packetbuffer MEMORY IS FULL errors by checking kernel message buffer for storage hardware errors. The eighth section explains how to diagnose OOM (Out of Memory) killer events and provides fixes including correcting <code>innodb_buffer_pool_size</code> and reducing VoIPmonitor buffer settings (<code>ringbuffer</code>, <code>max_buffer_mem</code>). The ninth section addresses sensor upgrade failures with "Permission denied" errors from <code>/tmp</code> due to the <code>noexec</code> mount flag. The tenth section explains benign database errors that occur when optional features are disabled. The Appendix provides correct tshark display filter syntax for analyzing SIP traffic.
 
'''Keywords:''' troubleshooting, no calls, not sniffing, no data, no CDRs, tshark, wireshark, promiscuous mode, promisc, ifconfig, ip link, SPAN, RSPAN, ERSPAN, GRE, TZSP, VXLAN, port mirroring, voipmonitor.conf, interface, sipport, filter, capture rules, Skip ON, GUI capture rules, reload sniffer, backup and restore, syslog, logs, permission denied, display filter, sip.Method, sip.Call-ID, sip.Status-Code, sip contains, OOM, out of memory, OOM killer, killed process, mysqld killed, free -h, memory usage, dmesg, swap, RAM upgrade, innodb_buffer_pool_size, GUI update, sensor update, buffer pool calculation, innodb_buffer_pool too high, correct buffer pool size, calculate innodb_buffer_pool, killall, sensor unresponsive, service fails to start after update, /tmp, noexec, mount, fstab, upgrade fails, permission denied from tmp, sensor upgrade permission denied, tmp noexec flag, remount tmp exec, HEAP FULL, packetbuffer MEMORY IS FULL, dmesg -T, ext4-fs error, I/O error, NVRAM WARNING, storage hardware errors, disk failure, smart error, ata failed, buffer I/O error, smartctl, mdstat, RAID health, iostat, dstat, ioping, NFS latency, disk I/O performance, iowait, performance bottleneck, hardware failure vs performance, binary not found, service fails to start after crash, voipmonitor binary renaming, voipmonitor_ underscore, watchdog crash, binary renamed with underscore, mv voipmonitor_ voipmonitor, ls -l /usr/local/sbin/voipmonitor_, ringbuffer, max_buffer_mem, voipmonitor buffer memory, reduce voipmonitor memory, voipmonitor buffer calculation, sniffer buffers, packetbuffer, ring buffer size, maximum buffer memory, voipmonitor memory settings, benign database errors, harmless sql errors, table does not exist, ss7 table not found, register_failed table, monitoring exceptions, ignore mysql errors, error code 1146, optional features disabled, ss7 disabled, suppress warnings
<!-- This section is for AI/RAG systems. Do not edit manually. -->
'''Key Questions:'''
 
* Why is VoIPmonitor not recording any calls?
=== Summary ===
* VoIPmonitor service fails to start after crash with binary not found error, what to do?
Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).
* How do I fix voipmonitor binary not found error?
 
* Where is the voipmonitor binary located?
=== Keywords ===
* Why is the voipmonitor binary renamed with underscore?
troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark
* How can I check if VoIP traffic is reaching my sensor server?
 
* What command can I use to see live SIP traffic on the command line?
=== Key Questions ===
* How do I enable promiscuous mode on my network card?
* Why are no calls being recorded in VoIPmonitor?
* SS7 table does not exist error, what should I do?
* How to diagnose PACKETBUFFER FULL or memory is FULL error?
* MySQL error table does not exist when feature is disabled, is it harmful?
* How to determine if bottleneck is I/O or CPU?
* Should I fix database errors about missing tables for disabled features?
* What do heap values in syslog mean?
* How do I suppress benign MySQL error warnings in monitoring?
* What does t0CPU percentage indicate?
* What are harmless errors in VoIPmonitor logs?
* How to use sniffer_threads manager command?
* Table 1146 error in voipmonitor logs, is this a problem?
* How to connect to manager socket or port 5029?
* Why do I see errors about register_failed table?
* What to do when t0 thread is at 100%?
* Do I need promiscuous mode for ERSPAN or GRE tunnels?
* How to fix one-way audio or missing RTP?
* Does ERSPAN require promiscuous mode on the receiving interface?
* How to configure natalias for NAT?
* VoIPmonitor is running but I have no new calls in the GUI, what should I check first?
* How to increase max_buffer_mem for high traffic?
* Where can I find the log files for the VoIPmonitor sniffer?
* How to disable jitterbuffer to save CPU?
* What are the most common reasons for VoIPmonitor not capturing data?
* What causes OOM kills of voipmonitor or MySQL?
* How do I filter tshark output for SIP INVITE messages?
* How to check disk I/O performance with iostat?
* What is the correct tshark filter syntax to find a specific phone number?
* How to enable promiscuous mode on interface?
* How do I extract Call-ID from a pcap file using tshark?
* How to fix packet ordering issues with SPAN?
* What tshark filter shows all SIP 4xx and 5xx error responses?
* What is Error 1062 duplicate entry?
* Why is my VoIPmonitor probe stopping processing calls even though network traffic is visible?
* How to verify traffic reaches capture interface?
* What should I check if the probe sees SIP packets on the interface but processes no calls?
* How do GUI capture rules affect call processing?
* What does the "Skip" option in capture rules do?
* How do I troubleshoot capture rules that are blocking calls?
* VoIPmonitor server stops processing CDRs and needs restart. What could be wrong?
* Why does sensor upgrade fail with permission denied errors from /tmp?
* How do I fix noexec flag on /tmp directory?
* VoIPmonitor permission denied when upgrading sensor, what to do?
* How do I check mount options for /tmp partition?
* What command checks if /tmp has noexec flag?
* How to remove noexec from /tmp mount point?
* Sensor upgrade fails in tmp directory, why?
* Why does MySQL crash and restart on my VoIPmonitor server?
* How do I check for OOM killer events in Linux?
* What does the error "Out of memory: Kill process" mean?
* How can I monitor memory usage on my VoIPmonitor server?
* What command shows available memory in Linux?
* How do I fix OOM killer issues on VoIPmonitor?
* Why is mysqld getting killed on my system?
* How do I calculate the correct innodb_buffer_pool_size for VoIPmonitor?
* What is the formula for innodb_buffer_pool_size on a shared server?
* My innodb_buffer_pool_size is too high, how do I fix OOM issues?
* What should innodb_buffer_pool_size be set to on a 32GB server?
* Does MySQL innodb_buffer_pool_size cause OOM killer?
* Sensor becomes unresponsive after GUI update, what should I do?
* How do I restart a sensor service that fails to start after GUI update?
* What is the fix for sensor service not starting after web GUI update?
* Why is my sensor service stuck after updating through the GUI?
* How to fix unresponsive sensor after GUI upgrade using killall?
* Sensor service fails after GUI upgrade, what commands to run?
* How do I force kill all voipmonitor processes?
* VoIPmonitor crashes with HEAP FULL error, what should I check?
* sniffer crashes with packetbuffer MEMORY IS FULL messages, how to fix?
* How do I check for disk hardware errors in Linux?
* What command shows storage errors in the kernel buffer?
* How to distinguish between disk hardware failure vs performance bottleneck?
* What are the signs of a failing hard drive in dmesg?
* How do I check SMART status of a hard drive?
* What does ext4-fs error mean in dmesg?
* What is an I/O error in kernel logs?
* What is NVRAM WARNING on RAID controller?
* How do I check disk I/O performance on Linux?
* How to measure NFS latency for VoIPmonitor?
* What is iostat command and how do I use it?
* How to test disk speed with ioping?
* Should I replace disk or tune settings for HEAP FULL errors?
* Voipmonitor sensor crashing with HEAP FULL, first diagnostic step?

Latest revision as of 19:08, 22 January 2026

Sniffer Troubleshooting

This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see Sniffer_configuration. For performance tuning, see Scaling.

Critical First Step: Is Traffic Reaching the Interface?

⚠️ Warning: Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help. 

# Check for SIP traffic on the capture interface
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10

# If no packets: Network/SPAN issue - contact network admin
# If packets visible: Proceed with sensor troubleshooting below

Quick Diagnostic Checklist

Check Command Expected Result
Service running systemctl status voipmonitor Active (running)
Traffic on interface tshark -i eth0 -c 5 -Y "sip" SIP packets displayed
Interface errors ip -s link show eth0 No RX errors/drops
Promiscuous mode ip link show eth0 PROMISC flag present
Logs grep voip No critical errors
GUI rules Settings → Capture Rules No unexpected "Skip" rules

No Calls Being Recorded

Service Not Running

# Check status
systemctl status voipmonitor

# View recent logs
journalctl -u voipmonitor --since "10 minutes ago"

# Start/restart
systemctl restart voipmonitor

Common startup failures:

  • Interface not found: Check interface in voipmonitor.conf matches ip a output
  • Port already in use: Another process using the management port
  • License issue: Check License for activation problems

Wrong Interface or Port Configuration

# Check current config
grep -E "^interface|^sipport" /etc/voipmonitor.conf

# Example correct config:
# interface = eth0
# sipport = 5060

💡 Tip:

GUI Capture Rules Blocking

Navigate to Settings → Capture Rules and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.

See Capture_rules for detailed configuration.

SPAN/Mirror Not Configured

If tcpdump shows no traffic:

  1. Verify switch SPAN/mirror port configuration
  2. Check that both directions (ingress + egress) are mirrored
  3. Confirm VLAN tagging is preserved if needed
  4. Test physical connectivity (cable, port status)

See Sniffing_modes for SPAN, RSPAN, and ERSPAN configuration.

Filter Parameter Too Restrictive

If filter is set in voipmonitor.conf, it may exclude traffic: 

# Check filter
grep "^filter" /etc/voipmonitor.conf

# Temporarily disable to test
# Comment out the filter line and restart


Missing id_sensor Parameter

Symptom: SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.

Cause: The id_sensor parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.

Solution: 

# Check if id_sensor is set
grep "^id_sensor" /etc/voipmonitor.conf

# Add or correct the parameter
echo "id_sensor = 1" >> /etc/voipmonitor.conf

# Restart the service
systemctl restart voipmonitor

💡 Tip: Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See id_sensor documentation.

Missing Audio / RTP Issues

One-Way Audio (Asymmetric Mirroring)

Symptom: SIP recorded but only one RTP direction captured.

Cause: SPAN port configured for only one direction.

Diagnosis: 

# Count RTP packets per direction
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c

If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.

RTP Not Associated with Call

Symptom: Audio plays in sniffer but not in GUI, or RTP listed under wrong call.

Possible causes:

1. SIP and RTP on different interfaces/VLANs: 

# voipmonitor.conf - enable automatic RTP association
auto_enable_use_blocks = yes

2. NAT not configured: 

# voipmonitor.conf - for NAT scenarios
natalias = <public_ip> <private_ip>

# If not working, try reversed order:
natalias = <private_ip> <public_ip>

3. External device modifying media ports:

If SDP advertises one port but RTP arrives on different port (SBC/media server issue): 

# Compare SDP ports vs actual RTP
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u

If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.

RTP Incorrectly Associated with Wrong Call (PBX Port Reuse)

Symptom: RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.

Cause: When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.

Solution: Enable rtp_check_both_sides_by_sdp to require verification of both source and destination IP:port against SDP: 

# voipmonitor.conf - require both source and destination to match SDP
rtp_check_both_sides_by_sdp = yes

# Alternative (strict) mode - allows initial unverified packets
rtp_check_both_sides_by_sdp = strict

⚠️ Warning: Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use natalias mappings or the strict setting to mitigate this.

Snaplen Truncation

Symptom: Large SIP messages truncated, incomplete headers.

Solution: 

# voipmonitor.conf - increase packet capture size
snaplen = 8192

For Kamailio siptrace, also check trace_msg_fragment_size in Kamailio config. See snaplen documentation.

PACKETBUFFER Saturation

Symptom: Log shows PACKETBUFFER: memory is FULL, truncated RTP recordings.

⚠️ Warning: This alert refers to VoIPmonitor's internal packet buffer (max_buffer_mem), NOT system RAM. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.

Before testing solutions, gather diagnostic data:

  • Check sensor logs: /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS)
  • Generate debug log via GUI: Tools → Generate debug log

Diagnose: I/O vs CPU Bottleneck

⚠️ Warning: Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to confirm findings, not as the primary diagnostic method.

Step 1: Check IO[] Metrics (v2026.01.3+)

Starting with version 2026.01.3, VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status: 

[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]

Quick interpretation:

Metric Meaning Problem Indicator
C (Capacity) % of disk's sustainable throughput used C ≥ 80% = Warning, C ≥ 95% = Saturated
L (Latency) Current write latency in ms L ≥ 3× B (baseline) = Saturated
U (Utilization) % time disk is busy U > 90% = Disk at limit

If you see DISK_SAT or WARN after IO[]: 

IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT

→ This confirms I/O bottleneck. Skip to I/O Bottleneck Solutions.

For older versions or additional confirmation, continue with the steps below.

ℹ️ Note: See Syslog Status Line - IO[] section for detailed field descriptions.

Step 2: Read the Full Syslog Status Line

VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool: 

# Monitor in real-time
journalctl -u voipmonitor -f
# or
tail -f /var/log/syslog | grep voipmonitor

Example status line: 

calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB

Key metrics for bottleneck identification:

Metric What It Indicates I/O Bottleneck Sign CPU Bottleneck Sign
heap[A|B|C] Buffer fill % (primary / secondary / processing) High A with low t0CPU High A with high t0CPU
t0CPU[X%] Packet capture thread (single-core, cannot parallelize) Low (<50%) High (>80%)
comp[X] Active compression threads Very high (maxed out) Normal
SQLq[C:X M:Y] Pending SQL queries Growing = database bottleneck Stable
tacCPU[...] TAR compression threads All near 100% = compression bottleneck Normal

Interpretation flowchart:

Step 3: Linux I/O Diagnostics

Use these standard Linux tools to confirm I/O bottleneck:

Install required tools: 

# Debian/Ubuntu
apt install sysstat iotop ioping

# CentOS/RHEL
yum install sysstat iotop ioping

2a) iostat - Disk utilization and wait times 

# Run for 10 intervals of 2 seconds
iostat -xz 2 10

Key output columns: 

Device   r/s     w/s   rkB/s   wkB/s  await  %util
sda     12.50  245.30  50.00  1962.40  45.23  98.50
Column Description Problem Indicator
%util Device utilization percentage > 90% = disk saturated
await Average I/O wait time (ms) > 20ms for SSD, > 50ms for HDD = high latency
w/s Writes per second Compare with disk's rated IOPS

2b) iotop - Per-process I/O usage 

# Show I/O by process (run as root)
iotop -o

Look for voipmonitor or mysqld dominating I/O. If voipmonitor shows high DISK WRITE but system %util is 100%, disk cannot keep up.

2c) ioping - Quick latency check 

# Test latency on VoIPmonitor spool directory
cd /var/spool/voipmonitor
ioping -c 20 .

Expected results:

Storage Type Healthy Latency Problem Indicator
NVMe SSD < 0.5 ms > 2 ms
SATA SSD < 1 ms > 5 ms
HDD (7200 RPM) < 10 ms > 30 ms

Step 4: Linux CPU Diagnostics

3a) top - Overall CPU usage 

# Press '1' to show per-core CPU
top

Look for:

  • Individual CPU core at 100% (t0 thread is single-threaded)
  • High %wa (I/O wait) vs high %us/%sy (CPU-bound)

3b) Verify voipmonitor threads 

# Show voipmonitor threads with CPU usage
top -H -p $(pgrep voipmonitor)

If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).

Step 5: Decision Matrix

Observation Likely Cause Go To
heap high, t0CPU > 80%, iostat %util low CPU Bottleneck CPU Solution
heap high, t0CPU < 50%, iostat %util > 90% I/O Bottleneck I/O Solution
heap high, t0CPU < 50%, iostat %util < 50%, SQLq growing Database Bottleneck Database Solution
heap normal, comp maxed, tacCPU all ~100% Compression Bottleneck (type of I/O) I/O Solution

Step 6: Confirmation Test (Optional)

After identifying the likely cause with the tools above, you can confirm with a storage disable test: 

# /etc/voipmonitor.conf - temporarily disable all storage
savesip = no
savertp = no
savertcp = no
savegraph = no

systemctl restart voipmonitor
# Monitor for 5-10 minutes during peak traffic
journalctl -u voipmonitor -f | grep heap
  • If heap values drop to near zero → confirms I/O bottleneck
  • If heap values remain high → confirms CPU bottleneck

⚠️ Warning: Remember to re-enable storage after testing! This test causes call recordings to be lost.

Solution: I/O Bottleneck

ℹ️ Note: If you see IO[...] DISK_SAT or WARN in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See IO[] Metrics for details.

Quick confirmation (for older versions):

Temporarily save only RTP headers to reduce disk write load: 

# /etc/voipmonitor.conf
savertp = header

Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.

Check storage health before upgrading: 

# Check drive health
smartctl -a /dev/sda

# Check for I/O errors in system logs
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"

Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.

Storage controller cache settings:

Storage Type Recommended Cache Mode
HDD / NAS WriteBack (requires battery-backed cache)
SSD WriteThrough (or WriteBack with power loss protection)

Use vendor-specific tools to configure cache policy (megacli, ssacli, perccli).

Storage upgrades (in order of effectiveness):

Solution IOPS Improvement Notes
NVMe SSD 50-100x vs HDD Best option, handles 10,000+ concurrent calls
SATA SSD 20-50x vs HDD Good option, handles 5,000+ concurrent calls
RAID 10 with BBU 5-10x vs single disk Enable WriteBack cache (requires battery backup)
Separate storage server Variable Use client/server mode

Filesystem tuning (ext4): 

# Check current mount options
mount | grep voipmonitor

# Recommended mount options for /var/spool/voipmonitor
# Add to /etc/fstab: noatime,data=writeback,barrier=0
# WARNING: barrier=0 requires battery-backed RAID

Verify improvement: 

# After changes, monitor iostat
iostat -xz 2 10
# %util should drop below 70%, await should decrease

Solution: CPU Bottleneck

Identify CPU Bottleneck Using Manager Commands

VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.

Connect to manager interface: 

# Via Unix socket (local, recommended)
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket

# Via TCP port 5029 (remote or local)
echo 'sniffer_threads' | nc 127.0.0.1 5029

# Monitor continuously (every 2 seconds)
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"

ℹ️ Note: TCP port 5029 is encrypted by default. For unencrypted access, set manager_enable_unencrypted = yes in voipmonitor.conf (security risk on public networks).

Example output: 

t0 - binlog1 fifo pcap read          (  12345) :  78.5  FIFO  99     1234
t2 - binlog1 pb write                (  12346) :  12.3               456
rtp thread binlog1 binlog1 0         (  12347) :   8.1               234
rtp thread binlog1 binlog1 1         (  12348) :   6.2               198
t1 - binlog1 call processing         (  12349) :   4.5               567
tar binlog1 compression 0            (  12350) :   3.2                89

Column interpretation:

Column Description
Thread name Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
(TID) Linux thread ID (useful for top -H -p TID)
CPU % Current CPU usage percentage - key metric
Sched Scheduler type (FIFO = real-time, empty = normal)
Priority Thread priority
CS/s Context switches per second

Critical threads to watch:

Thread Role If at 90-100%
t0 (pcap read) Packet capture from NIC Single-core limit reached! Cannot parallelize. Need DPDK/Napatech.
t2 (pb write) Packet buffer processing Processing bottleneck. Check t2CPU breakdown.
rtp thread RTP packet processing Threads auto-scale. If still saturated, consider DPDK/Napatech.
tar compression PCAP archiving I/O bottleneck (compression waiting for disk)
mysql store Database writes Database bottleneck. Check SQLq metric.

⚠️ Warning: If t0 thread is at 90-100%, you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and cannot be parallelized. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:

  • Reduce captured traffic using interface_ip_filter or BPF filter
  • Use kernel bypass (DPDK or Napatech) which eliminates kernel overhead entirely

Interpreting t2CPU Detailed Breakdown

The syslog status line shows t2CPU with detailed sub-metrics: 

t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
Code Function High Value Indicates
pb Packet buffer output Buffer management overhead
d Dispatch Structure creation bottleneck
s SIP parsing Complex/large SIP messages
e Entity lookup Call table lookup overhead
c Call processing Call state machine processing
g Register processing High REGISTER volume
r, rm, rh, rd RTP processing stages High RTP volume (threads auto-scale)

Thread auto-scaling: VoIPmonitor automatically spawns additional threads when load increases:

  • If d > 50% → SIP parsing thread (s) starts
  • If s > 50% → Entity lookup thread (e) starts
  • If e > 50% → Call/register/RTP threads start

Configuration for High Traffic (>10,000 calls/sec)

# /etc/voipmonitor.conf

# Increase buffer to handle processing spikes (value in MB)
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
# Larger buffer absorbs I/O and CPU spikes without packet loss
max_buffer_mem = 10000

# Use IP filter instead of BPF (more efficient)
interface_ip_filter = 10.0.0.0/8
interface_ip_filter = 192.168.0.0/16
# Comment out any 'filter' parameter

CPU Optimizations

# /etc/voipmonitor.conf

# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
jitterbuffer_f1 = no
jitterbuffer_f2 = yes
jitterbuffer_adapt = no

# If MOS metrics are not needed at all, disable everything:
# jitterbuffer_f1 = no
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no

Kernel Bypass Solutions (Extreme Loads)

When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
Napatech Hardware SmartNIC >97% (< 3% at 10Gbit) Extreme performance requirements

Verify Improvement

# Monitor thread CPU after changes
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"

# Or monitor syslog
journalctl -u voipmonitor -f
# t0CPU should drop, heap values should stay < 20%

ℹ️ Note: After changes, monitor syslog heap[A|B|C] values - should stay below 20% during peak traffic. See Syslog_Status_Line for detailed metric explanations.

Storage Hardware Failure

Symptom: Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.

Diagnosis: 

# Check disk health
smartctl -a /dev/sda

# Check RAID status (if applicable)
cat /proc/mdstat
mdadm --detail /dev/md0

Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.

OOM (Out of Memory)

Identify OOM Victim

# Check for OOM kills
dmesg | grep -i "out of memory\|oom\|killed process"
journalctl --since "1 hour ago" | grep -i oom

MySQL Killed by OOM

Reduce InnoDB buffer pool: 

# /etc/mysql/my.cnf
innodb_buffer_pool_size = 2G  # Reduce from default

Voipmonitor Killed by OOM

Reduce buffer sizes in voipmonitor.conf: 

max_buffer_mem = 2000  # Reduce from default
ringbuffer = 50        # Reduce from default

Runaway External Process

# Find memory-hungry processes
ps aux --sort=-%mem | head -20

# Kill orphaned/runaway process
kill -9 <PID>

For servers limited to 16GB RAM or when experiencing repeated MySQL OOM kills: 

# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G

# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
innodb_flush_log_at_trx_commit = 2

Restart MySQL after changes: 

systemctl restart mysql
# or
systemctl restart mariadb

SQL Queue Growth from Non-Call Data

If sip-register, sip-options, or sip-subscribe are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.

⚠️ Warning: Even with reduced innodb_buffer_pool_size, SQL queue will grow indefinitely without cleanup of non-call data.

Solution: Enable automatic cleanup of old non-call data 

# /etc/voipmonitor.conf
# cleandatabase=2555 automatically deletes partitions older than 7 years
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase = 2555

Restart the sniffer after changes: 

systemctl restart voipmonitor

ℹ️ Note: See Data_Cleaning for detailed configuration options and other cleandatabase_* parameters.

Service Startup Failures

Interface No Longer Exists

After OS upgrade, interface names may change (eth0 → ensXXX): 

# Find current interface names
ip a

# Update all config locations
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/

# Also check GUI: Settings → Sensors → Configuration

Missing Dependencies

# Install common missing package
apt install libpcap0.8  # Debian/Ubuntu
yum install libpcap     # RHEL/CentOS

Network Interface Issues

Promiscuous Mode

Required for SPAN port monitoring: 

# Enable
ip link set eth0 promisc on

# Verify
ip link show eth0 | grep PROMISC

ℹ️ Note: Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.

Interface Drops

# Check for drops
ip -s link show eth0 | grep -i drop

# If drops present, increase ring buffer
ethtool -G eth0 rx 4096

Bonded/EtherChannel Interfaces

Symptom: False packet loss when monitoring bond0 or br0.

Solution: Monitor physical interfaces, not logical: 

# voipmonitor.conf - use physical interfaces
interface = eth0,eth1

Network Offloading Issues

Symptom: Kernel errors like bad gso: type: 1, size: 1448 

# Disable offloading on capture interface
ethtool -K eth0 gso off tso off gro off lro off

Packet Ordering Issues

If SIP messages appear out of sequence:

First: Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See FAQ.

If genuine reordering: Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see Napatech).

ℹ️ Note: For out-of-order packets in client/server mode (multiple sniffers), see Sniffer_distributed_architecture for pcap_queue_dequeu_window_length configuration.

Solutions for SPAN/Mirroring Reordering

If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):

1. Configure switch to preserve packet order: Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.

2. Replace SPAN with TAP or packet broker: Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.

Database Issues

SQL Queue Overload

Symptom: Growing SQLq metric, potential coredumps. 

# voipmonitor.conf - increase threads
mysqlstore_concat_limit_cdr = 1000
cdr_check_exists_callid = 0

Error 1062 - Lookup Table Limit

Symptom: Duplicate entry '16777215' for key 'PRIMARY'

Quick fix: 

# voipmonitor.conf
cdr_reason_string_enable = no

See Database Troubleshooting for complete solution.

Bad Packet Errors

Symptom: bad packet with ether_type 0xFFFF detected on interface

Diagnosis: 

# Run diagnostic (let run 30-60 seconds, then kill)
voipmonitor --check_bad_ether_type=eth0

# Find and kill the diagnostic process
ps ax | grep voipmonitor
kill -9 <PID>

Causes: corrupted packets, driver issues, VLAN tagging problems. Check ethtool -S eth0 for interface errors.

Useful Diagnostic Commands

tshark Filters for SIP

# All SIP INVITEs
tshark -r capture.pcap -Y "sip.Method == INVITE"

# Find specific phone number
tshark -r capture.pcap -Y 'sip contains "5551234567"'

# Get Call-IDs
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID

# SIP errors (4xx, 5xx)
tshark -r capture.pcap -Y "sip.Status-Code >= 400"

Interface Statistics

# Detailed NIC stats
ethtool -S eth0

# Watch packet rates
watch -n 1 'cat /proc/net/dev | grep eth0'

See Also





AI Summary for RAG

Summary

Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).

Keywords

troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark

Key Questions

  • Why are no calls being recorded in VoIPmonitor?
  • How to diagnose PACKETBUFFER FULL or memory is FULL error?
  • How to determine if bottleneck is I/O or CPU?
  • What do heap values in syslog mean?
  • What does t0CPU percentage indicate?
  • How to use sniffer_threads manager command?
  • How to connect to manager socket or port 5029?
  • What to do when t0 thread is at 100%?
  • How to fix one-way audio or missing RTP?
  • How to configure natalias for NAT?
  • How to increase max_buffer_mem for high traffic?
  • How to disable jitterbuffer to save CPU?
  • What causes OOM kills of voipmonitor or MySQL?
  • How to check disk I/O performance with iostat?
  • How to enable promiscuous mode on interface?
  • How to fix packet ordering issues with SPAN?
  • What is Error 1062 duplicate entry?
  • How to verify traffic reaches capture interface?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Sniffer_troubleshooting&oldid=11050"