Sniffer troubleshooting: Difference between revisions

From VoIPmonitor.org
(Compact flowchart, significantly shorten AI Summary section)
(Patch: replace '=== Solution: I/O Bottleneck =...')
 
(54 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Troubleshooting: No Calls Being Sniffed}}
= Sniffer Troubleshooting =


'''This guide provides a systematic, step-by-step process to diagnose why the VoIPmonitor sensor might not be capturing any calls. Follow these steps in order to quickly identify and resolve the most common issues.'''
This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see [[Sniffer_configuration]]. For performance tuning, see [[Scaling]].


== Troubleshooting Flowchart ==
== Critical First Step: Is Traffic Reaching the Interface? ==


<kroki lang="mermaid">
{{Warning|Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help.}}
%%{init: {'theme': 'base', 'flowchart': {'nodeSpacing': 10, 'rankSpacing': 25, 'curve': 'basis'}, 'themeVariables': {'fontSize': '11px'}}}%%
flowchart TD
    A[No Calls Being Captured] --> B{Step 1: Service Running?}
    B -->|No| B1[systemctl restart voipmonitor]
    B -->|Yes| C{Step 2: Traffic on Interface?<br/>tshark -i eth0 -Y 'sip'}


    C -->|No packets| D[Step 3: Network Issue]
<syntaxhighlight lang="bash">
    D --> D1{Interface UP?}
# Check for SIP traffic on the capture interface
    D1 -->|No| D2[ip link set dev eth0 up]
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10
    D1 -->|Yes| D3{SPAN/RSPAN?}
    D3 -->|Yes| D4[Enable promisc mode]
    D3 -->|ERSPAN/GRE/TZSP| D5[Check tunnel config]


    C -->|Packets visible| E[Step 4: VoIPmonitor Config]
# If no packets: Network/SPAN issue - contact network admin
    E --> E1{interface correct?}
# If packets visible: Proceed with sensor troubleshooting below
    E1 -->|No| E2[Fix interface in voipmonitor.conf]
</syntaxhighlight>
    E1 -->|Yes| E3{sipport correct?}
    E3 -->|No| E4[Add port: sipport = 5060,5080]
    E3 -->|Yes| E5{BPF filter blocking?}
    E5 -->|Maybe| E6[Comment out filter directive]


     E5 -->|No| F[Step 5: GUI Capture Rules]
<kroki lang="mermaid">
    F --> F1{Rules with Skip: ON?}
graph TD
     F1 -->|Yes| F2[Remove/modify rules + reload sniffer]
     A[No Calls Recorded] --> B{Packets on interface?<br/>tcpdump -i eth0 port 5060}
     F1 -->|No| G[Step 6: Check Logs]
     B -->|No packets| C[Network Issue]
 
     C --> C1[Check SPAN/mirror config]
     G --> H{OOM Events?}
     C --> C2[Verify VLAN tagging]
     H -->|Yes| H1[Step 7: Add RAM / tune MySQL]
     C --> C3[Check cable/port]
     H -->|No| I{Large SIP packets?}
     B -->|Packets visible| D[Sensor Issue]
     I -->|Yes| I1{External SIP source?<br/>Kamailio/HAProxy mirror}
     D --> D1[Check voipmonitor.conf]
    I1 -->|No| I2[Increase snaplen in voipmonitor.conf]
     D --> D2[Check GUI Capture Rules]
     I1 -->|Yes| I3[Fix external source: Kamailio siptrace or HAProxy tee]
     D --> D3[Check logs for errors]
     I2 --> I4[If snaplen change fails, recheck with tcpdump -s0]
    I4 --> I1
    I -->|No| J[Contact Support]
</kroki>
</kroki>


== Post-Reboot Verification Checklist ==
== Quick Diagnostic Checklist ==
After a planned server reboot, verify these critical items to ensure VoIPmonitor operates correctly. This check helps identify issues that may occur when configurations are not persisted across reboots.


=== Verify Firewall/Iptables Rules ===
{| class="wikitable"
|-
! Check !! Command !! Expected Result
|-
| Service running || <code>systemctl status voipmonitor</code> || Active (running)
|-
| Traffic on interface || <code>tshark -i eth0 -c 5 -Y "sip"</code> || SIP packets displayed
|-
| Interface errors || <code>ip -s link show eth0</code> || No RX errors/drops
|-
| Promiscuous mode || <code>ip link show eth0</code> || PROMISC flag present
|-
| Logs || <code>tail -100 /var/log/syslog \| grep voip</code> || No critical errors
|-
| GUI rules || Settings → Capture Rules || No unexpected "Skip" rules
|}
 
== No Calls Being Recorded ==


After a system restart, verify that firewall rules have been correctly applied and are allowing necessary traffic. Firewall rules may need to be manually re-applied if they were not made persistent.
=== Service Not Running ===


;1. Check current firewall status:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For systems using iptables
# Check status
iptables -L -n -v
systemctl status voipmonitor


# For systems using firewalld
# View recent logs
firewall-cmd --list-all
journalctl -u voipmonitor --since "10 minutes ago"


# For systems using ufw
# Start/restart
ufw status verbose
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


;2. Verify critical ports are allowed:
Common startup failures:
Ensure the firewall permits traffic on the following VoIPmonitor ports:
* '''Interface not found''': Check <code>interface</code> in voipmonitor.conf matches <code>ip a</code> output
* SIP ports (default: 5060/udp, or your configured sipport values)
* '''Port already in use''': Another process using the management port
* RTP ports (range used by your PBX)
* '''License issue''': Check [[License]] for activation problems
* GUI access (typically: 80/tcp, 443/tcp)
* Sensor management port: 5029/tcp
* Client-Server connection port: 60024/tcp (for distributed setups)


;3. Make firewall rules persistent:
=== Wrong Interface or Port Configuration ===
To prevent firewall rules from being lost after future reboots:


'''For iptables (Debian/Ubuntu):'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Save current rules
# Check current config
iptables-save > /etc/iptables/rules.v4
grep -E "^interface|^sipport" /etc/voipmonitor.conf
# Install persistent package if not present
 
apt-get install iptables-persistent
# Example correct config:
# interface = eth0
# sipport = 5060
</syntaxhighlight>
</syntaxhighlight>


'''For firewalld (CentOS/RHEL):'''
{{Tip|For multiple SIP ports: <code>sipport = 5060,5061,5080</code>}}
<syntaxhighlight lang="bash">
 
# Runtime rules automatically persist with --permanent flag
=== GUI Capture Rules Blocking ===
firewall-cmd --permanent --add-port=5060/udp
firewall-cmd --permanent --add-port=60024/tcp
firewall-cmd --reload
</syntaxhighlight>


=== Verify System Time Synchronization ===
Navigate to '''Settings → Capture Rules''' and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.


Correct system time synchronization is '''critical''', especially when using the <code>packetbuffer_sender</code> option in distributed architectures. Time mismatches between hosts and servers can cause call correlation failures and dropped packets.
See [[Capture_rules]] for detailed configuration.


;1. Check current NTP/chrony status:
=== SPAN/Mirror Not Configured ===
<syntaxhighlight lang="bash">
# For systems using NTP
ntpstat


# For systems using chrony
If <code>tcpdump</code> shows no traffic:
chronyc tracking
# Verify switch SPAN/mirror port configuration
</syntaxhighlight>
# Check that both directions (ingress + egress) are mirrored
# Confirm VLAN tagging is preserved if needed
# Test physical connectivity (cable, port status)


;2. Verify time synchronization with servers:
See [[Sniffing_modes]] for SPAN, RSPAN, and ERSPAN configuration.
<syntaxhighlight lang="bash">
# For NTP
ntpq -p


# For chrony
=== Filter Parameter Too Restrictive ===
chronyc sources -v
</syntaxhighlight>


'''Expected output:''' Time offset should be minimal (ideally under 100 milliseconds). Large offsets (several seconds) indicate synchronization problems.
If <code>filter</code> is set in voipmonitor.conf, it may exclude traffic:


;3. Manual sync if needed (temporary fix):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Force immediate NTP sync
# Check filter
sudo systemctl restart ntp
grep "^filter" /etc/voipmonitor.conf


# For chrony
# Temporarily disable to test
sudo chronyc makestep
# Comment out the filter line and restart
</syntaxhighlight>
</syntaxhighlight>


'''Critical for packetbuffer_sender mode:''' When using <code>packetbuffer_sender=yes</code> to forward raw packets from remote sensors to a central server, the host and server '''must have synchronized times'''. VoIPmonitor requires host and server times to match for proper call correlation and packet processing. Maximum allowed time difference is 2 seconds by default (configurable via <code>client_server_connect_maximum_time_diff_s</code>).


;4. Check distributed architecture time sync:
In Client-Server mode, ensure all sensors and the central server are synchronized to the same NTP servers:


<syntaxhighlight lang="bash">
==== Missing id_sensor Parameter ====
# On each sensor and central server
timedatectl status
</syntaxhighlight>


Look for: <code>System clock synchronized: yes</code>
'''Symptom''': SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.


If times are not synchronized across distributed components:
'''Cause''': The <code>id_sensor</code> parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.
* Verify all systems point to the same reliable NTP source
* Check firewall allows UDP port 123 (NTP)
* Ensure timezones are consistent across all systems


'''Troubleshooting time sync issues:'''
'''Solution''':
* Check firewall rules allow NTP (UDP port 123)
<syntaxhighlight lang="bash">
* Verify NTP servers are reachable: <code>ping pool.ntp.org</code>
# Check if id_sensor is set
* Review NTP configuration: <code>/etc/ntp.conf</code> or <code>/etc/chrony.conf</code>
grep "^id_sensor" /etc/voipmonitor.conf
* Ensure time service is enabled to start on boot: <code>systemctl enable ntp</code>


== Step 1: Is the VoIPmonitor Service Running Correctly? ==
# Add or correct the parameter
First, confirm that the sensor process is active and loaded the correct configuration file.
echo "id_sensor = 1" >> /etc/voipmonitor.conf


;1. Check the service status (for modern systemd systems):
# Restart the service
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
systemctl status voipmonitor
</syntaxhighlight>
</syntaxhighlight>
Look for a line that says <code>Active: active (running)</code>. If it is inactive or failed, try restarting it with <code>systemctl restart voipmonitor</code> and check the status again.


;2. Verify the running process:
{{Tip|Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See [[Sniffer_configuration#id_sensor|id_sensor documentation]].}}
<syntaxhighlight lang="bash">
== Missing Audio / RTP Issues ==
ps aux | grep voipmonitor
 
</syntaxhighlight>
=== One-Way Audio (Asymmetric Mirroring) ===
This command will show the running process and the exact command line arguments it was started with. Critically, ensure it is using the correct configuration file, for example: <code>--config-file /etc/voipmonitor.conf</code>. If it is not, there may be an issue with your startup script.


=== Troubleshooting: Missing Package or Library Dependencies ===
'''Symptom''': SIP recorded but only one RTP direction captured.


If the sensor service fails to start or crashes immediately with an error about a "missing package" or "missing library," it indicates that a required system dependency is not installed on the server. This is most common on newly installed sensors or fresh operating system installations.
'''Cause''': SPAN port configured for only one direction.


;1. Check the system logs for the specific error message:
'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Count RTP packets per direction
tail -f /var/log/syslog | grep voipmonitor
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c
 
# For CentOS/RHEL/AlmaLinux or systemd systems
journalctl -u voipmonitor -f
</syntaxhighlight>
</syntaxhighlight>


;2. Common missing packages for sensors:
If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.
Most sensor missing package issues are resolved by installing the <code>rrdtools</code> package. This is required for RRD (Round-Robin Database) graphing and statistics functionality.


<syntaxhighlight lang="bash">
=== RTP Not Associated with Call ===
# For Debian/Ubuntu
apt-get update && apt-get install rrdtool


# For CentOS/RHEL/AlmaLinux
'''Symptom''': Audio plays in sniffer but not in GUI, or RTP listed under wrong call.
yum install rrdtool
# OR
dnf install rrdtool
</syntaxhighlight>


;3. Other frequently missing dependencies:
'''Possible causes''':
If the error references a specific shared library or binary, install it using your package manager. Common examples:


* <code>libpcap</code> or <code>libpcap-dev</code>: Packet capture library
'''1. SIP and RTP on different interfaces/VLANs''':
* <code>libssl</code> or <code>libssl-dev</code>: SSL/TLS support
<syntaxhighlight lang="ini">
* <code>zlib</code> or <code>zlib1g-dev</code>: Compression library
# voipmonitor.conf - enable automatic RTP association
 
auto_enable_use_blocks = yes
;4. Verify shared library dependencies:
If the error mentions a specific shared library (e.g., <code>error while loading shared libraries: libxxx.so</code>), check which libraries the binary is trying to load:
 
<syntaxhighlight lang="bash">
ldd /usr/local/sbin/voipmonitor | grep pcap
</syntaxhighlight>
</syntaxhighlight>


If <code>ldd</code> reports "not found," install the missing library using your package manager.
'''2. NAT not configured''':
<syntaxhighlight lang="ini">
# voipmonitor.conf - for NAT scenarios
natalias = <public_ip> <private_ip>


;5. After installing the missing package, restart the sensor service:
# If not working, try reversed order:
<syntaxhighlight lang="bash">
natalias = <private_ip> <public_ip>
systemctl restart voipmonitor
systemctl status voipmonitor
</syntaxhighlight>
</syntaxhighlight>


Verify the service starts successfully and is now <code>Active: active (running)</code>.
'''3. External device modifying media ports''':
 
=== Troubleshooting: Cron Daemon Not Running ===
 
If the VoIPmonitor sniffer service fails to start or the sensor appears unavailable despite the systemd service being configured correctly, the cron daemon may not be running. Some VoIPmonitor deployment methods and maintenance scripts rely on cron for proper initialization and periodic tasks.


;1. Check the cron daemon status:
If SDP advertises one port but RTP arrives on different port (SBC/media server issue):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl status cron
# Compare SDP ports vs actual RTP
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u
</syntaxhighlight>
</syntaxhighlight>


Look for <code>Active: active (running)</code>. If the status shows inactive or failed, the cron daemon is not running.
If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.
=== RTP Incorrectly Associated with Wrong Call (PBX Port Reuse) ===
'''Symptom''': RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.


;2. Alternative check for systems using cron (not crond):
'''Cause''': When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.
<syntaxhighlight lang="bash">
systemctl status crond
</syntaxhighlight>


Note: On CentOS/RHEL systems, the service is typically named <code>crond</code>, while on Debian/Ubuntu systems it is named <code>cron</code>.
'''Solution''': Enable <code>rtp_check_both_sides_by_sdp</code> to require verification of both source and destination IP:port against SDP:
 
<syntaxhighlight lang="ini">
;3. Start the cron daemon if it is inactive:
# voipmonitor.conf - require both source and destination to match SDP
<syntaxhighlight lang="bash">
rtp_check_both_sides_by_sdp = yes
# For Debian/Ubuntu systems
systemctl start cron


# For CentOS/RHEL/AlmaLinux systems
# Alternative (strict) mode - allows initial unverified packets
systemctl start crond
rtp_check_both_sides_by_sdp = strict
</syntaxhighlight>
</syntaxhighlight>


;4. Enable cron to start automatically on boot:
{{Warning|Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use <code>natalias</code> mappings or the <code>strict</code> setting to mitigate this.}}
<syntaxhighlight lang="bash">
=== Snaplen Truncation ===
# For Debian/Ubuntu systems
systemctl enable cron


# For CentOS/RHEL/AlmaLinux systems
'''Symptom''': Large SIP messages truncated, incomplete headers.
systemctl enable crond
</syntaxhighlight>


;5. After starting the cron daemon, restart the VoIPmonitor service:
'''Solution''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
systemctl restart voipmonitor
# voipmonitor.conf - increase packet capture size
systemctl status voipmonitor
snaplen = 8192
</syntaxhighlight>
</syntaxhighlight>


Verify the service now shows <code>Active: active (running)</code> and the sensor becomes visible in the GUI.
For Kamailio siptrace, also check <code>trace_msg_fragment_size</code> in Kamailio config. See [[Sniffer_configuration#snaplen|snaplen documentation]].
 
=== Root Cause ===
The cron daemon being inactive can prevent VoIPmonitor from starting properly in scenarios where:
* Installation scripts use cron for post-install configuration
* Maintenance or cleanup jobs are required for proper sensor operation
* System initialization processes depend on cron-based tasks
* The sensor was recently rebooted or upgraded and cron failed to start
 
=== Long-Term Stability ===
If the cron daemon is consistently failing to start after reboots:
* Check system logs for cron startup errors: <code>journalctl -u cron -n 50</code> or <code>journalctl -u crond -n 50</code>
* Verify that the server has sufficient resources (CPU, memory) to run all required system services
* Investigate performance bottlenecks that may be causing system services to fail to start
* Ensure no other system services are conflicting or preventing cron from starting


== Step 2: Is Network Traffic Reaching the Server? ==
== PACKETBUFFER Saturation ==
If the service is running, the next step is to verify if the VoIP packets (SIP/RTP) are actually arriving at the server's network interface. The best tool for this is <code>tshark</code> (the command-line version of Wireshark).


;1. Install tshark:
'''Symptom''': Log shows <code>PACKETBUFFER: memory is FULL</code>, truncated RTP recordings.
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
apt-get update && apt-get install tshark


# For CentOS/RHEL/AlmaLinux
{{Warning|This alert refers to VoIPmonitor's '''internal packet buffer''' (<code>max_buffer_mem</code>), '''NOT system RAM'''. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.}}
yum install wireshark
</syntaxhighlight>


;2. Listen for SIP traffic on the correct interface:
'''Before testing solutions''', gather diagnostic data:
Replace <code>eth0</code> with the interface name you have configured in <code>voipmonitor.conf</code>.
* Check sensor logs: <code>/var/log/syslog</code> (Debian/Ubuntu) or <code>/var/log/messages</code> (RHEL/CentOS)
<syntaxhighlight lang="bash">
* Generate debug log via GUI: '''Tools → Generate debug log'''
tshark -i eth0 -Y "sip || rtp" -n
</syntaxhighlight>
* '''If you see a continuous stream of SIP and RTP packets''', it means traffic is reaching the server, and the problem is likely in VoIPmonitor's configuration (see Step 4).
* '''If you see NO packets''', the problem lies with your network configuration. Proceed to Step 3.


== Step 3: Troubleshoot Network and Interface Configuration ==
=== Diagnose: I/O vs CPU Bottleneck ===
If <code>tshark</code> shows no traffic, it means the packets are not being delivered to the operating system correctly.


;1. Check if the interface is UP:
{{Warning|Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to '''confirm''' findings, not as the primary diagnostic method.}}
Ensure the network interface is active.
<syntaxhighlight lang="bash">
ip link show eth0
</syntaxhighlight>
The output should contain the word <code>UP</code>. If it doesn't, bring it up with:
<syntaxhighlight lang="bash">
ip link set dev eth0 up
</syntaxhighlight>


;2. Check for Interface Packet Drops:
==== Step 1: Check IO[] Metrics (v2026.01.3+) ====
If calls are missing, showing "000" as the last response, or have silent audio, the root cause may be packets being dropped at the **network interface level** BEFORE they reach VoIPmonitor. This is different from sensor resource limitations.


Check the interface statistics for packet drops on the sniffing interface:
'''Starting with version 2026.01.3''', VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Check detailed interface statistics, packet errors, and drops
[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]
ip -s -s l l eth0
</syntaxhighlight>
</syntaxhighlight>


The output shows RX (receive) and TX (transmit) statistics. Look specifically at the <code>dropped</code> counter under the <code>RX</code> section:
'''Quick interpretation:'''
 
{| class="wikitable"
<pre>
|-
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
! Metric !! Meaning !! Problem Indicator
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    12345123  45678    12      5432    0      0
    TX: bytes  packets  errors  dropped carrier collsns
    9876543  23456    5      0      0      0
</pre>
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Critical: Interface Packet Drops vs Sensor Drops
| '''C''' (Capacity) || % of disk's sustainable throughput used || '''C ≥ 80% = Warning''', '''C ≥ 95% = Saturated'''
|-
|-
| style="vertical-align: top;" | '''Interface drops (kernel level):'''
| '''L''' (Latency) || Current write latency in ms || '''L ≥ 3× B''' (baseline) = Saturated
| Packets dropped by the network card driver BEFORE reaching VoIPmonitor. Use <code>ip -s -s l l</code> to check. Root cause is network infrastructure: switch overload, duplex mismatch, faulty NIC/switch port.
|-
|-
| style="vertical-align: top;" | '''Sensor drops (VoIPmonitor level):'''
| '''U''' (Utilization) || % time disk is busy || '''U > 90%''' = Disk at limit
| Packets received by the OS but dropped by VoIPmonitor due to high CPU load, insufficient ringbuffer, or configuration limits. Check the "# packet drops" counter in GUI Settings → Sensors and <code>t0CPU</code> metric in logs.
|}
|}


;Diagnosing Interface Packet Drops:
'''If you see <code>DISK_SAT</code> or <code>WARN</code> after IO[]:'''
 
<syntaxhighlight lang="text">
'''Step 1: Check if the dropped counter is increasing:'''
IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT
 
Run the interface statistics command multiple times while making test calls:
 
<syntaxhighlight lang="bash">
# First measurement
ip -s -s l l eth0 | grep -A 1 "RX:"
 
# Make a test call during which you expect to see dropped packets
 
# Second measurement 10-30 seconds later
ip -s -s l l eth0 | grep -A 1 "RX:"
</syntaxhighlight>
</syntaxhighlight>


If the <code>dropped</code> value increases between measurements during test calls, the interface is losing packets due to infrastructure issues.
→ This confirms I/O bottleneck. Skip to [[#Solution:_I.2FO_Bottleneck|I/O Bottleneck Solutions]].


'''Step 2: Identify the root cause of interface packet drops:'''
'''For older versions or additional confirmation''', continue with the steps below.


Common causes of network interface packet drops:
{{Note|See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|Syslog Status Line - IO[] section]] for detailed field descriptions.}}


* '''Network switch port overload:''' The switch port connected to the VoIPmonitor sensor is receiving more traffic than it can forward to the sensor. This is common during peak traffic hours.
==== Step 2: Read the Full Syslog Status Line ====
* '''Duplex/speed mismatch:''' The server NIC and switch port are configured with mismatched speed or duplex settings (e.g., NIC set to 100Mbps/half-duplex while switch is 1Gbps/full-duplex).
* '''Faulty hardware:''' Defective network interface card (NIC) or a damaged switch port.


;Specific diagnostic actions:
VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool:


* '''Check duplex/speed negotiation:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check the current speed and duplex setting
# Monitor in real-time
ethtool eth0
journalctl -u voipmonitor -f
# or
tail -f /var/log/syslog | grep voipmonitor
</syntaxhighlight>
</syntaxhighlight>
Look for the <code>Speed</code> and <code>Duplex</code> values. Ensure they match the switch port configuration (e.g., Speed: 1000Mb/s, Duplex: Full).


* '''Check for driver errors:'''
'''Example status line:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Look for NIC driver messages that indicate hardware issues
calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB
dmesg | grep -i eth0 | tail -50
</syntaxhighlight>
</syntaxhighlight>


* '''Test with a different network interface or cable:'''
'''Key metrics for bottleneck identification:'''
If the issue persists after verifying duplex/speed, try connecting the sensor to a different switch port or using a different cable to rule out hardware faults.


;Step 3: Verify bidirectional SIP traffic in SPAN/mirroring configuration:
{| class="wikitable"
|-
! Metric !! What It Indicates !! I/O Bottleneck Sign !! CPU Bottleneck Sign
|-
| <code>heap[A&#124;B&#124;C]</code> || Buffer fill % (primary / secondary / processing) || High A with low t0CPU || High A with high t0CPU
|-
| <code>t0CPU[X%]</code> || Packet capture thread (single-core, cannot parallelize) || Low (<50%) || High (>80%)
|-
| <code>comp[X]</code> || Active compression threads || Very high (maxed out) || Normal
|-
| <code>SQLq[C:X M:Y]</code> || Pending SQL queries || Growing = database bottleneck || Stable
|-
| <code>tacCPU[...]</code> || TAR compression threads || All near 100% = compression bottleneck || Normal
|}


Even if the interface shows no packet drops, verify that your network switch SPAN/mirror configuration is sending **BOTH directions** of SIP traffic to the sniffing interface. Missing one direction causes incomplete CDRs and incorrect Last Response tracking.
'''Interpretation flowchart:'''


Use <code>tshark</code> during a test call to verify bidirectional SIP flow:
<kroki lang="mermaid">
graph TD
    A[heap values rising] --> B{Check t0CPU}
    B -->|t0CPU > 80%| C[CPU Bottleneck]
    B -->|t0CPU < 50%| D{Check comp and tacCPU}
    D -->|comp maxed, tacCPU high| E[I/O Bottleneck<br/>Disk cannot keep up with writes]
    D -->|comp normal| F{Check SQLq}
    F -->|SQLq growing| G[Database Bottleneck]
    F -->|SQLq stable| H[Mixed/Other Issue]


<syntaxhighlight lang="bash">
    C --> C1[Solution: CPU optimization]
# Monitor INVITE requests and their responses to confirm bidirectional flow
    E --> E1[Solution: Faster storage]
# Replace eth0 with your sniffing interface
    G --> G1[Solution: MySQL tuning]
tshark -i eth0 -Y 'sip.CSeq.method == "INVITE" || sip.Status-Code' -n
</kroki>
</syntaxhighlight>


If you see INVITE requests but NO corresponding responses (like 200 OK, 404, 500), your SPAN/mirror configuration is only capturing one direction of traffic. This requires network switch configuration changes:
==== Step 3: Linux I/O Diagnostics ====


* **Cisco switches:** Verify SPAN source includes `both` direction:
Use these standard Linux tools to confirm I/O bottleneck:
  <syntaxhighlight lang="bash">
  show running-config | include monitor session
  # Should include: monitor session 1 source interface GigabitEthernet1/1 both
  </syntaxhighlight>


* **Other switch vendors:** Refer to switch documentation for SPAN mirroring direction configuration.
'''Install required tools:'''
<syntaxhighlight lang="bash">
# Debian/Ubuntu
apt install sysstat iotop ioping


;Summary of workflow when interface packet drops are detected:
# CentOS/RHEL
 
yum install sysstat iotop ioping
# Use <code>ip -s -s l l</code> to check for increasing <code>dropped</code> counter
</syntaxhighlight>
# Confirm drops occur during test calls with repeated measurements
# Check duplex/speed with <code>ethtool</code>
# Verify switch port configuration matches NIC settings
# Check for hardware faults (different port, different cable)
# Verify SPAN/mirror sends both directions of SIP traffic with <code>tshark -Y sip.CSeq.method == "INVITE"</code>
# Resolve the underlying network infrastructure issue before tuning VoIPmonitor configuration
 
{{Warning|Interface packet drops cannot be fixed with VoIPmonitor configuration changes. Increasing ringbuffer, adjusting pcap_queue_deque_window_length, or other sniffer tuning will NOT resolve packet drops at the kernel/interface level. You must fix the network infrastructure first.}}
 
;3B. Troubleshooting: Asymmetric Traffic Mirroring Across Multiple Interfaces or Hosts
 
If SIP packets are visible on the network but CDRs are not appearing in the GUI, the issue may be that SIP requests and responses are being mirrored to **different interfaces** or **different sniffer hosts**, rather than a single interface monitored by a single voipmonitor instance.
 
=== Diagnosis: Check Traffic on Each Interface and Host ===
 
When investigating mirroring issues, you must verify packet flow on **every network interface** of **each sniffer host**. A common misconfiguration occurs when:
 
* SIP requests (INVITE) are mirrored to interface A on sensor host 1
* SIP responses (200 OK) are mirrored to interface B on sensor host 1
* Or requests go to sensor host 1 and responses go to sensor host 2
 
This asymmetric mirroring prevents voipmonitor from correlating requests with responses, resulting in missing or incomplete CDRs.
 
=== Step 1: Identify All Sniffing Interfaces and Hosts ===
 
First, identify all interfaces configured for mirroring and all sensor hosts in your deployment:


'''2a) iostat - Disk utilization and wait times'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# On each sniffer host, list all network interfaces
# Run for 10 intervals of 2 seconds
ip a | grep -E "^[0-9]+:|state UP"
iostat -xz 2 10
</syntaxhighlight>


# Check which interfaces voipmonitor is configured to use
'''Key output columns:'''
grep "^interface" /etc/voipmonitor.conf
<syntaxhighlight lang="text">
 
Device  r/s    w/s  rkB/s  wkB/s  await  %util
# Check for multiple voipmonitor instances running on the same host
sda    12.50  245.30  50.00  1962.40  45.23  98.50
ps aux | grep voipmonitor
</syntaxhighlight>
</syntaxhighlight>


=== Step 2: Capture Traffic on Each Interface Simultaneously ===
{| class="wikitable"
 
|-
For each interface on each sniffer host, run a separate tcpdump capture during a test call:
! Column !! Description !! Problem Indicator
|-
| <code>%util</code> || Device utilization percentage || '''> 90%''' = disk saturated
|-
| <code>await</code> || Average I/O wait time (ms) || '''> 20ms''' for SSD, '''> 50ms''' for HDD = high latency
|-
| <code>w/s</code> || Writes per second || Compare with disk's rated IOPS
|}


'''2b) iotop - Per-process I/O usage'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# On sensor host 1, interface eth0:
# Show I/O by process (run as root)
tcpdump -i eth0 -nn "sip" -w /tmp/sensor1_eth0.pcap &
iotop -o
 
# On sensor host 1, interface eth1 (if exists):
tcpdump -i eth1 -nn "sip" -w /tmp/sensor1_eth1.pcap &
 
# On sensor host 2, interface eth0:
tcpdump -i eth0 -nn "sip" -w /tmp/sensor2_eth0.pcap &
</syntaxhighlight>
</syntaxhighlight>


Make a test call and stop the captures after 10-30 seconds.
Look for <code>voipmonitor</code> or <code>mysqld</code> dominating I/O. If voipmonitor shows high DISK WRITE but system <code>%util</code> is 100%, disk cannot keep up.
 
=== Step 3: Analyze Captures to Detect Asymmetric Mirroring ===
 
Using tshark, check which interface received which part of the SIP dialog:


'''2c) ioping - Quick latency check'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check capture 1 for INVITE requests
# Test latency on VoIPmonitor spool directory
tshark -r /tmp/sensor1_eth0.pcap -Y "sip.CSeq.method == INVITE" -T fields -e sip.Call-ID
cd /var/spool/voipmonitor
 
ioping -c 20 .
# Check capture 1 for SIP responses
tshark -r /tmp/sensor1_eth0.pcap -Y "sip.Status-Code" -T fields -e sip.Call-ID
 
# Check capture 2 for INVITE requests
tshark -r /tmp/sensor1_eth1.pcap -Y "sip.CSeq.method == INVITE" -T fields -e sip.Call-ID
 
# Check capture 2 for SIP responses
tshark -r /tmp/sensor1_eth1.pcap -Y "sip.Status-Code" -T fields -e sip.Call-ID
</syntaxhighlight>
</syntaxhighlight>


Compare the Call-IDs across captures. If you observe:
'''Expected results:'''
 
{| class="wikitable"
* INVITEs appear in capture 1, but responses for the same Call-ID appear only in capture 2
|-
* Or INVITEs and responses are split across different hosts
! Storage Type !! Healthy Latency !! Problem Indicator
* Or any combination where the complete SIP dialog is not on a single interface
|-
 
| NVMe SSD || < 0.5 ms || > 2 ms
Then your mirroring configuration is asymmetric.
|-
 
| SATA SSD || < 1 ms || > 5 ms
=== Step 4: Count Packets Per Interface ===
|-
| HDD (7200 RPM) || < 10 ms || > 30 ms
|}


To quantify the asymmetry, count the SIP packets on each interface:
==== Step 4: Linux CPU Diagnostics ====


'''3a) top - Overall CPU usage'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Count total SIP packets on each interface
# Press '1' to show per-core CPU
tshark -r /tmp/sensor1_eth0.pcap -Y "sip" | wc -l
top
tshark -r /tmp/sensor1_eth1.pcap -Y "sip" | wc -l
tshark -r /tmp/sensor2_eth0.pcap -Y "sip" | wc -l
</syntaxhighlight>
</syntaxhighlight>


If voipmonitor is running on only one interface but traffic is split across two, you will see partial SIP dialogs. The interface receiving only requests or only responses will not generate complete CDRs.
Look for:
 
* Individual CPU core at 100% (t0 thread is single-threaded)
=== Solution: Correct Mirroring Configuration for Symmetric Traffic Flow ===
* High <code>%wa</code> (I/O wait) vs high <code>%us/%sy</code> (CPU-bound)


VoIPmonitor requires the **complete SIP session (both requests and responses)** to be mirrored to a **SINGLE network interface** monitored by a **SINGLE voipmonitor sniffer instance**.
'''3b) Verify voipmonitor threads'''
 
==== Identify the Correct Source for Complete Mirroring ====
 
1. Determine which switch port(s), interface(s), or VLAN(s) carry the traffic you want to monitor
2. Trace the network path to understand where SIP requests and responses converge
3. Find a single point in your network where you can capture the bidirectional traffic
 
==== Configure SPAN/Mirror to Single Destination ====
 
Update your network switch's port mirroring (SPAN) configuration:
 
* Cisco switches example:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Monitor both inbound and outbound traffic from source port
# Show voipmonitor threads with CPU usage
# Send complete session to a SINGLE destination port
top -H -p $(pgrep voipmonitor)
monitor session 1 source interface GigabitEthernet1/1 both
monitor session 1 destination interface GigabitEthernet1/2
</syntaxhighlight>
</syntaxhighlight>


* Ensure the SPAN source uses the <code>both</code> keyword to capture bidirectional traffic
If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).
* Ensure the SPAN destination is a single port connected to one voipmonitor sensor interface


==== Multi-Host Deployments ====
==== Step 5: Decision Matrix ====


If you must monitor traffic that is split across multiple network segments or hosts:
{| class="wikitable"
 
1. Use VoIPmonitor's **Client-Server** mode (distributed architecture) instead of multiple independent sniffers
2. Configure remote probes to forward all packets to one central analysis server
3. See [[Sniffer_distributed_architecture|Distributed Architecture: Client-Server Mode]] for setup details
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Asymmetric Mirroring Prevention Best Practices
! Observation !! Likely Cause !! Go To
|-
|-
| style="vertical-align: top;" | '''Single Interface Rule:'''
| <code>heap</code> high, <code>t0CPU</code> > 80%, iostat <code>%util</code> low || '''CPU Bottleneck''' || [[#Solution: CPU Bottleneck|CPU Solution]]
| Always ensure complete SIP sessions (requests + responses) go to one monitored interface on one sensor instance.
|-
|-
| style="vertical-align: top;" | '''Verification Method:'''
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> > 90% || '''I/O Bottleneck''' || [[#Solution: I/O Bottleneck|I/O Solution]]
| During installation, use tshark on the target interface to verify INVITEs AND responses have matching Call-IDs.
|-
|-
| style="vertical-align: top;" | '''Switch SPAN Configuration:'''
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> < 50%, <code>SQLq</code> growing || '''Database Bottleneck''' || [[#SQL Queue Overload|Database Solution]]
| Use <code>both</code> direction in SPAN/mirror commands. Verify no duplicate or split destination ports.
|-
|-
| style="vertical-align: top;" | '''Avoid Multiple Sniffers:'''
| <code>heap</code> normal, <code>comp</code> maxed, <code>tacCPU</code> all ~100% || '''Compression Bottleneck''' (type of I/O) || [[#Solution: I/O Bottleneck|I/O Solution]]
| Do not run multiple independent voipmonitor instances on different interfaces unless using Client-Server mode with packet forwarding.
|}
|}


=== Step 5: Verify Fix After Mirroring Changes ===
==== Step 6: Confirmation Test (Optional) ====


After correcting the network mirroring configuration:
After identifying the likely cause with the tools above, you can confirm with a storage disable test:


1. Restart the voipmonitor service: <code>systemctl restart voipmonitor</code>
<syntaxhighlight lang="ini">
2. Run a test call
# /etc/voipmonitor.conf - temporarily disable all storage
3. Verify CDRs appear in the GUI within 30-60 seconds
savesip = no
4. Open the CDR and confirm it shows complete SIP history (INVITE + responses)
savertp = no
5. If using tcpdump for verification, confirm the complete dialog is now on one interface:
savertcp = no
savegraph = no
</syntaxhighlight>


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Should now show both INVITEs and responses with same Call-IDs
systemctl restart voipmonitor
tshark -i eth0 -Y "sip" -T fields -e sip.Call-ID -e sip.Method -e sip.Status-Code | head -20
# Monitor for 5-10 minutes during peak traffic
journalctl -u voipmonitor -f | grep heap
</syntaxhighlight>
</syntaxhighlight>


;3. Check for Promiscuous Mode (for SPAN/RSPAN Mirrored Traffic):
* If <code>heap</code> values drop to near zero → confirms '''I/O bottleneck'''
'''Important:''' Promiscuous mode requirements depend on your traffic mirroring method:
* If <code>heap</code> values remain high → confirms '''CPU bottleneck'''


* '''SPAN/RSPAN (Layer 2 mirroring):''' The network interface '''must''' be in promiscuous mode. Mirrored packets retain their original MAC addresses, so the interface would normally ignore them. Promiscuous mode forces the interface to accept all packets regardless of destination MAC.
{{Warning|Remember to re-enable storage after testing! This test causes call recordings to be lost.}}


* '''ERSPAN/GRE/TZSP/VXLAN (Layer 3 tunnels):''' Promiscuous mode is '''NOT required'''. These tunneling protocols encapsulate the mirrored traffic inside IP packets that are addressed directly to the sensor's IP address. The operating system receives these packets normally, and VoIPmonitor automatically decapsulates them to extract the inner SIP/RTP traffic.
=== Solution: I/O Bottleneck ===


For SPAN/RSPAN deployments, check the current promiscuous mode status:
{{Note|If you see <code>IO[...] DISK_SAT</code> or <code>WARN</code> in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|IO[] Metrics]] for details.}}
<syntaxhighlight lang="bash">
 
ip link show eth0
'''Quick confirmation (for older versions):'''
</syntaxhighlight>
Look for the <code>PROMISC</code> flag.


Enable promiscuous mode manually if needed:
Temporarily save only RTP headers to reduce disk write load:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
ip link set eth0 promisc on
# /etc/voipmonitor.conf
savertp = header
</syntaxhighlight>
</syntaxhighlight>
If this solves the problem, you should make the change permanent. The <code>install-script.sh</code> for the sensor usually attempts to do this, but it can fail.


;3A. Troubleshooting: Missing Packets for Specific IPs During High-Traffic Periods:
Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.
If calls are missing only for certain IP addresses or specific call flows (particularly during high-traffic periods), the issue is typically at the network infrastructure level (SPAN configuration) rather than sensor resource limits. Use this systematic approach:
 
=== Step 1: Use tcpdump to Verify Packet Arrival ===
 
Before tuning any sensor configuration, first verify if the missing packets are actually reaching the sensor's network interface. Use <code>tcpdump</code> for this verification:


'''Check storage health before upgrading:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Listen for SIP packets from a specific IP during the next high-traffic window
# Check drive health
# Replace eth0 with your interface and 10.1.2.3 with the problematic IP
smartctl -a /dev/sda
tcpdump -i eth0 -nn "host 10.1.2.3 and port 5060" -v


# Or capture to a file for later analysis
# Check for I/O errors in system logs
tcpdump -i eth0 -nn "host 10.1.2.3 and port 5060" -w /tmp/trace_10.1.2.3.pcap
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"
</syntaxhighlight>
</syntaxhighlight>


Interpret the results:
Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.
* '''If you see SIP packets arriving:''' The traffic reaches the sensor. The issue is likely a sensor resource bottleneck (CPU, memory, or configuration limits). Proceed to [[#Sensor_Resource_Bottlenecks|Step 4: Check Sensor Statistics]].
* '''If you see NO packets or only intermittent packets:''' The traffic is not reaching the sensor. This indicates a network infrastructure issue. Proceed to [[#SPAN_Configuration_Troubleshooting|Step 2: Check SPAN Configuration]].


=== Step 2: Check SPAN Configuration for Bidirectional Capture ===
'''Storage controller cache settings:'''
{| class="wikitable"
|-
! Storage Type !! Recommended Cache Mode
|-
| HDD / NAS || WriteBack (requires battery-backed cache)
|-
| SSD || WriteThrough (or WriteBack with power loss protection)
|}


If packets are missing at the interface level, verify your network switch's SPAN (port mirroring) configuration. During high-traffic periods, switches may have insufficient SPAN buffer capacity, causing packets to be dropped in the mirroring process itself.
Use vendor-specific tools to configure cache policy (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>).


Key verification points:
'''Storage upgrades (in order of effectiveness):'''
 
{| class="wikitable"
* '''Verify Source Ports:''' Confirm that both source IP addresses (or the switch ports they connect to) are included in the SPAN source list. Missing one direction of the call flow will result in incomplete CDRs.
|-
 
! Solution !! IOPS Improvement !! Notes
* '''Check for Bidirectional Mirroring:''' Your SPAN configuration must capture '''BOTH inbound and outbound traffic'''. On most Cisco switches, this requires specifying:
|-
  <syntaxhighlight lang="bash">
| '''NVMe SSD''' || 50-100x vs HDD || Best option, handles 10,000+ concurrent calls
  monitor session 1 source interface GigabitEthernet1/1 both
  </syntaxhighlight>
 
  Replace <code>both</code> with:
  * <code>rx</code> for incoming traffic only
  * <code>tx</code> for outgoing traffic only
  * <code>both</code> for bidirectional capture (recommended)
 
* '''Verify Destination Port:''' Confirm the SPAN destination points to the switch port where the VoIPmonitor sensor is connected.
 
* '''Check SPAN Buffer Saturation (High-Traffic Issues):''' Some switches have limited SPAN buffer capacity. When monitoring multiple high-traffic ports simultaneously, the SPAN buffer may overflow during peak usage, causing randomized packet drops. Symptoms:
  ** Drops occur only during busy hours
  ** Missing packets are inconsistent across different calls
  ** Sensor CPU usage and t0CPU metrics appear normal (no bottleneck at sensor)
 
  Solutions:
  ** Reduce the number of monitored source ports in the SPAN session
  ** Use multiple SPAN sessions if your switch supports it
  ** Consider upgrading to a switch with higher SPAN buffer capacity
 
* '''Verify Switch Interface Counters for Packet Drops:''' Check the network switch interface counters to determine if the switch itself is dropping packets during the mirroring process. This is critical when investigating false low MOS scores or packet loss reports.
 
  Cisco switches:
  <syntaxhighlight lang="bash">
  # Show general interface statistics for the SPAN source port
  show interface GigabitEthernet1/1 counters
  show interface GigabitEthernet1/1 | include drops|errors|Input queue|Output queue
 
  # Show detailed interface status (look for input errors, CRC, frame)
  show interface GigabitEthernet1/1 detail
 
  # Monitor in real-time during a high-traffic period
  show interface Gi1/1 accounting
  </syntaxhighlight>
 
  Key indicators of switch-level packet loss:
  ** Non-zero input errors or CRC errors on source/destination ports
  ** Input queue drops (indicating switch buffer overflow)
  ** Increasing drop counters during peak traffic hours
  ** Output errors on the SPAN destination port (sensor may not be accepting fast enough)
 
  If switch interface counters show drops, the issue is at the network infrastructure level (overloaded switch), not the VoIPmonitor sensor. Consult your network administrator for switch optimization or consider redistributing SPAN traffic across multiple ports.
 
* '''Verify VLAN Trunking:''' If the monitored traffic spans different VLANs, ensure the SPAN destination port is configured as a trunk to carry all necessary VLAN tags. Without trunk mode, packets from non-native VLANs will be dropped or stripped of their tags.
 
For detailed instructions on configuring SPAN/ERSPAN/GRE for different network environments, see [[Sniffing_modes]].
 
=== Step 3: Check for UDP Fragmentation Issues ===
 
When investigating missing packets, especially for SIP over UDP, a common issue is packet fragmentation. If the MTU path between systems causes SIP packets to be fragmented (typically >1480 bytes), the IP fragments will not contain UDP port information, making port-based tcpdump filters miss these fragments.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Critical: Check Defragmentation Settings FIRST
| '''SATA SSD''' || 20-50x vs HDD || Good option, handles 5,000+ concurrent calls
|-
|-
| style="vertical-align: top;" | '''Before using tcpdump:'''
| '''RAID 10 with BBU''' || 5-10x vs single disk || Enable WriteBack cache (requires battery backup)
| Verify VoIPmonitor's defragmentation is enabled in <code>/etc/voipmonitor.conf</code>. If these are disabled, VoIPmonitor cannot reassemble IP fragments before parsing SIP, causing missing packets even when traffic reaches the interface correctly.
|-
|-
| style="vertical-align: top;" | '''Required settings:'''
| '''Separate storage server''' || Variable || Use [[Sniffer_distributed_architecture|client/server mode]]
| <code>udpfrag = yes</code> (default) for UDP fragment reassembly<br/><code>sip_tcp_reassembly_ext = yes</code> (default) for TCP reassembly
|}
|}
;Step 1: Verify defragmentation is enabled:


Check your VoIPmonitor configuration:
'''Filesystem tuning (ext4):'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check if UDP fragment reassembly is enabled
# Check current mount options
grep "^udpfrag" /etc/voipmonitor.conf
mount | grep voipmonitor


# Check if TCP reassembly is enabled
# Recommended mount options for /var/spool/voipmonitor
grep "^sip_tcp_reassembly_ext" /etc/voipmonitor.conf
# Add to /etc/fstab: noatime,data=writeback,barrier=0
# WARNING: barrier=0 requires battery-backed RAID
</syntaxhighlight>
</syntaxhighlight>


Both should be set to <code>yes</code> (these are the defaults). If either is set to <code>no</code>, VoIPmonitor will not reassemble fragmented packets, causing packets larger than the MTU to be ignored or parsed incorrectly.
'''Verify improvement:'''
 
If the settings are correct and you are still experiencing issues, proceed to the following diagnostic steps using tcpdump.
 
;Step 2: Identify if fragmentation may be causing packet loss:
If you are investigating missing INVITE packets (especially those with large SDP), check if packet fragmentation is in the network path. Large SIP packets get fragmented at IP layers, and subsequent fragments only contain IP headers (no UDP port information).
 
;Step 3: Do NOT use port-based filters when investigating fragmentation:
<code>tcpdump</code> or <code>tshark</code> filters based on UDP port will miss IP fragments, because fragments only contain the IP header (no transport layer port information). Instead, filter by IP address.
 
;Step 4: Use IP-based tcpdump filters for fragmented packets:
When capturing to investigate missing packets that may be fragmented, filter by IP addresses instead of ports:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# CORRECT: Filter by IP address to capture fragments
# After changes, monitor iostat
# Replace opensips_ip and carrier_ip with actual IP addresses
iostat -xz 2 10
tcpdump -i eth0 -nn "host opensips_ip and carrier_ip" -w /tmp/fragmentation_test.pcap
# %util should drop below 70%, await should decrease
 
# INCORRECT: Port filter will miss IP fragments
tcpdump -i eth0 -nn "host opensips_ip and port 5060" -w /tmp/fragmentation_test.pcap
</syntaxhighlight>
</syntaxhighlight>


;Step 5: Test with a specific call that is missing from the GUI:
=== Solution: CPU Bottleneck ===
Make a test call that exhibits the missing packet issue, and capture the entire traffic flow between the two endpoints:
<syntaxhighlight lang="bash">
# Capture all traffic between two IPs during a test call
# Replace with your actual IP addresses
tcpdump -i eth0 -nn "host 192.168.1.100 and 203.0.113.50" -w /tmp/test_call_capture.pcap
</syntaxhighlight>


;Step 6: Analyze the capture for fragmentation:
==== Identify CPU Bottleneck Using Manager Commands ====
Open the pcap file in <code>tshark</code> or Wireshark to check for fragmented IP packets:
<syntaxhighlight lang="bash">
# Look for fragmented IP packets in the capture
tshark -r /tmp/test_call_capture.pcap -Y "ip.fragments"


# Count fragments
VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.
tshark -r /tmp/test_call_capture.pcap -Y "ip.fragments" | wc -l
</syntaxhighlight>


If you see fragmented packets, especially for large SIP INVITE messages, this confirms that port-based filters would miss fragments, which explains why the call appears incomplete in VoIPmonitor.
'''Connect to manager interface:'''
 
;{{Note|If tcpdump with IP-based filters captures the missing packets while VoIPmonitor does not see them, share the pcap file and your <code>voipmonitor.conf</code> with VoIPmonitor support for further analysis.}}
 
=== Step 3A: Check for Sensor Resource Bottlenecks ===
 
If <code>tcpdump</code> confirms that packets are arriving at the interface consistently, but VoIPmonitor is still missing them, the issue may be sensor resource limitations.
 
* '''Check Packet Drops:''' In the GUI, navigate to '''Settings → Sensors''' and look at the "# packet drops" counter. If this counter is non-zero or increasing during high traffic:
  ** Increase the <code>ringbuffer</code> size in <code>voipmonitor.conf</code> (default 50 MB, max 2000 MB)
  ** Check the <code>t0CPU</code> metric in system logs - if consistently above 90%, you may need to upgrade CPU or optimize NIC drivers
 
* '''Monitor Memory Usage:''' Check for OOM (Out of Memory) killer events:
  <syntaxhighlight lang="bash">
  grep -i "out of memory\|killed process" /var/log/syslog | tail -20
  </syntaxhighlight>
 
* '''SIP Packet Limits:''' If only long or chatty calls are affected, check the <code>max_sip_packets_in_call</code> and <code>max_invite_packets_in_call</code> limits in <code>voipmonitor.conf</code>.
 
;3. Verify Your SPAN/Mirror/TAP Configuration:
This is the most common cause of no traffic. Double-check your network switch or hardware tap configuration to ensure:
* The correct source ports (where your PBX/SBC is connected) are being monitored.
* The correct destination port (where your VoIPmonitor sensor is connected) is configured.
* If you are monitoring traffic across different VLANs, ensure your mirror port is configured to carry all necessary VLAN tags (often called "trunk" mode).
 
;4. Investigate Packet Encapsulation (If tcpdump shows traffic but VoIPmonitor does not):
If <code>tcpdump</code> or <code>tshark</code> shows packets reaching the interface but VoIPmonitor is not capturing them, the traffic may be encapsulated in a tunnel that VoIPmonitor cannot automatically process without additional configuration. Common encapsulations include VLAN tags, ERSPAN, GRE, VXLAN, and TZSP.
 
First, capture a sample of the traffic for analysis:
<syntaxhighlight lang="bash">
# Capture 100 packets of SIP traffic to a pcap file
tcpdump -i eth0 -c 100 -s0 port 5060 -w /tmp/encapsulation_check.pcap
</syntaxhighlight>
 
Then analyze the capture to identify encapsulation:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check for VLAN-tagged packets (802.1Q)
# Via Unix socket (local, recommended)
tshark -r /tmp/encapsulation_check.pcap -Y "vlan"
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket
 
# Check for GRE tunnels
tshark -r /tmp/encapsulation_check.pcap -Y "gre"
 
# Check for ERSPAN (GRE encapsulated with ERSPAN protocol)
tshark -r /tmp/encapsulation_check.pcap -Y "gre && ip.proto == 47"
 
# Check for VXLAN (UDP port 4789)
tshark -r /tmp/encapsulation_check.pcap -Y "udp.port == 4789"


# Check for TZSP (UDP ports 37008 or 37009)
# Via TCP port 5029 (remote or local)
tshark -r /tmp/encapsulation_check.pcap -Y "udp.port == 37008 || udp.port == 37009"
echo 'sniffer_threads' | nc 127.0.0.1 5029


# Show packet summary to identify any unusual protocol stacks
# Monitor continuously (every 2 seconds)
tshark -r /tmp/encapsulation_check.pcap -V | head -50
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"
</syntaxhighlight>
</syntaxhighlight>


Identifying encapsulation issues:
{{Note|1=TCP port 5029 is encrypted by default. For unencrypted access, set <code>manager_enable_unencrypted = yes</code> in voipmonitor.conf (security risk on public networks).}}
* '''VLAN tags present:''' Ensure VoIPmonitor's <code>sipport</code> filter does not use <code>udp</code> (which may drop VLAN-tagged packets). Comment out the <code>filter</code> directive in <code>voipmonitor.conf</code> to test.


* '''ERSPAN/GRE tunnels:''' Promiscuous mode is NOT required for these Layer 3 tunnels. Verify that tunneling is configured correctly on your network device and that the packets are addressed to the sensor's IP. VoIPmonitor automatically decapsulates ERSPAN and GRE.
'''Example output:'''
 
<syntaxhighlight lang="text">
* '''VXLAN/TZSP tunnels:''' These specialized tunneling protocols require proper configuration on the sending device. Consult your network device documentation for VoIPmonitor compatibility requirements.
t0 - binlog1 fifo pcap read          (  12345) : 78.5  FIFO  99    1234
 
t2 - binlog1 pb write                (  12346) :  12.3              456
If encapsulation is identified as the issue, review [[Sniffing_modes]] for detailed configuration guidance.
rtp thread binlog1 binlog1 0        ( 12347) :  8.1              234
 
rtp thread binlog1 binlog1 1         ( 12348) :  6.2              198
;3B. Troubleshooting: RTP Streams Not Displayed for Specific Provider:
t1 - binlog1 call processing        ( 12349) :  4.5              567
If SIP signaling appears correctly in the GUI for calls from a specific provider, but RTP streams (audio quality graphs, waveform visualization) are missing for that provider while working correctly for other call paths, use this systematic approach to identify the cause.
tar binlog1 compression 0           ( 12350) :  3.2               89
 
=== Step 1: Make a Test Call to Reproduce the Issue===
 
First, create a controlled test scenario to investigate the specific provider.
 
* Determine if the issue affects ALL calls from this provider or only some (e.g., specific codecs, call duration, time of day)
* Make a test call that reproduces the problem (e.g., from the problematic provider to a test number)
* Allow the call to establish and run for at least 30-60 seconds to capture meaningful RTP data
 
=== Step 2: Capture Packets on the Sniffing Interface During the Test Call ===
 
During the test call, use <code>tcpdump</code> (or <code>tshark</code>) to directly capture packets on the network interface configured in <code>voipmonitor.conf</code>. This tells you whether RTP packets are being received by the sensor.
 
<syntaxhighlight lang="bash">
# Capture SIP and RTP packets from the specific provider IP during your test call
# Replace eth0 with your interface and 1.2.3.4 with the provider's IP
sudo tcpdump -i eth0 -nn "host 1.2.3.4 and (udp port 5060 or (udp[0] & 0x78) == 0x78)" -v
 
# Capture RTP to a file for detailed analysis (recommended)
sudo tcpdump -i eth0 -nn "host 1.2.3.4 and rtp" -w /tmp/test_provider_rtp.pcap
</syntaxhighlight>
</syntaxhighlight>


Note: The RTP filter <code>(udp[0] & 0x78) == 0x78</code> matches packets with the first two bits of the first byte set to "10", which is characteristic of RTP.
'''Column interpretation:'''
{| class="wikitable"
|-
! Column !! Description
|-
| Thread name || Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
|-
| (TID) || Linux thread ID (useful for <code>top -H -p TID</code>)
|-
| CPU % || Current CPU usage percentage - '''key metric'''
|-
| Sched || Scheduler type (FIFO = real-time, empty = normal)
|-
| Priority || Thread priority
|-
| CS/s || Context switches per second
|}


=== Step 3: Compare Raw Packet Capture with Sensor Output ===
'''Critical threads to watch:'''
 
{| class="wikitable"
After the test call:
|-
! Thread !! Role !! If at 90-100%
|-
| '''t0''' (pcap read) || Packet capture from NIC || '''Single-core limit reached!''' Cannot parallelize. Need DPDK/Napatech.
|-
| '''t2''' (pb write) || Packet buffer processing || Processing bottleneck. Check t2CPU breakdown.
|-
| '''rtp thread''' || RTP packet processing || Threads auto-scale. If still saturated, consider DPDK/Napatech.
|-
| '''tar compression''' || PCAP archiving || I/O bottleneck (compression waiting for disk)
|-
| '''mysql store''' || Database writes || Database bottleneck. Check SQLq metric.
|}


* Check what tcpdump captured:
{{Warning|If '''t0 thread is at 90-100%''', you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and '''cannot be parallelized'''. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:
<syntaxhighlight lang="bash">
* '''Reduce captured traffic''' using <code>interface_ip_filter</code> or BPF <code>filter</code>
# Count SIP packets
* '''Use kernel bypass''' ([[DPDK]] or [[Napatech]]) which eliminates kernel overhead entirely}}
tshark -r /tmp/test_provider_rtp.pcap -Y "sip" | wc -l


# Count RTP packets
==== Interpreting t2CPU Detailed Breakdown ====
tshark -r /tmp/test_provider_rtp.pcap -Y "rtp" | wc -l


# View RTP stream details
The syslog status line shows <code>t2CPU</code> with detailed sub-metrics:
tshark -r /tmp/test_provider_rtp.pcap -Y "rtp" -T fields -e rtp.ssrc -e rtp.seq -e rtp.ptype -e udp.srcport -e udp.dstport | head -20
<syntaxhighlight lang="text">
t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
</syntaxhighlight>
</syntaxhighlight>


* Check what VoIPmonitor recorded:
{| class="wikitable"
  * Open the CDR for your test call in the GUI
|-
  * Verify if the "Received Packets" column shows non-zero values for the provider leg
! Code !! Function !! High Value Indicates
  * Check if the "Streams" section shows RTP quality graphs and waveform visualization
|-
 
| '''pb''' || Packet buffer output || Buffer management overhead
* Compare the results:
|-
** '''If tcpdump shows NO RTP packets:''' The RTP traffic is not reaching the sensor interface. This indicates a network-level issue (asymmetric routing, SPAN configuration missing the RTP path, or firewall). You need to troubleshoot the network infrastructure, not VoIPmonitor.
| '''d''' || Dispatch || Structure creation bottleneck
 
|-
** '''If tcpdump shows RTP packets but the GUI shows no streams or zero received packets:''' The packets are reaching the sensor but VoIPmonitor is not processing them. Check:
| '''s''' || SIP parsing || Complex/large SIP messages
* [[#Check_GUI_Capture_Rules_(Causing_Call_Stops)|Step 5: Check GUI Capture Rules]] - Look for capture rules targeting the provider's IP with RTP set to "DISCARD" or "Header Only"
|-
* [[Tls|TLS/SSL Decryption]] - Verify SRTP decryption is configured correctly if the provider uses encryption
| '''e''' || Entity lookup || Call table lookup overhead
* [[Sniffer_configuration]] - Check for any problematic <code>sipport</code> or <code>filter</code> settings
|-
 
| '''c''' || Call processing || Call state machine processing
For more information on capture rules that affect RTP storage, see [[Capture_rules]].
|-
| '''g''' || Register processing || High REGISTER volume
|-
| '''r, rm, rh, rd''' || RTP processing stages || High RTP volume (threads auto-scale)
|}


;5. Check for Non-Call SIP Traffic Only:
'''Thread auto-scaling:''' VoIPmonitor automatically spawns additional threads when load increases:
If you see SIP traffic but it consists only of OPTIONS, NOTIFY, SUBSCRIBE, or MESSAGE methods (without any INVITE packets), there are no calls to generate CDRs. This can occur in environments that use SIP for non-call purposes like heartbeat checks or instant messaging.
* If '''d''' > 50% → SIP parsing thread ('''s''') starts
* If '''s''' > 50% → Entity lookup thread ('''e''') starts
* If '''e''' > 50% → Call/register/RTP threads start


You can configure VoIPmonitor to process and store these non-call SIP messages. See [[SIP_OPTIONS/SUBSCRIBE/NOTIFY]] and [[MESSAGES]] for configuration details.
==== Configuration for High Traffic (>10,000 calls/sec) ====


Enable non-call SIP message processing in '''/etc/voipmonitor.conf''':
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Process SIP OPTIONS (qualify pings). Default: no
# /etc/voipmonitor.conf
sip-options = yes
 
# Process SIP MESSAGE (instant messaging). Default: yes
sip-message = yes
 
# Process SIP SUBSCRIBE requests. Default: no
sip-subscribe = yes
 
# Process SIP NOTIFY requests. Default: no
sip-notify = yes
</syntaxhighlight>
 
Note that enabling these for processing and storage can significantly increase database load in high-traffic scenarios. Use with caution and monitor SQL queue growth. See [[SIP_OPTIONS/SUBSCRIBE/NOTIFY#Performance_Tuning|Performance Tuning]] for optimization tips.
 
== Step 4: Check the VoIPmonitor Configuration ==
If <code>tshark</code> sees traffic but VoIPmonitor does not, the problem is almost certainly in <code>voipmonitor.conf</code>.
 
;1. Check the <code>interface</code> directive:
Make sure the <code>interface</code> parameter in <code>/etc/voipmonitor.conf</code> exactly matches the interface where you see traffic with <code>tshark</code>. For example: <code>interface = eth0</code>.
 
=== Troubleshooting: Wrong Interface Name ===
 
If the <code>interface</code> directive is set to an interface name that does not exist on the system, the sensor will fail to capture traffic completely. This is a common issue when network interface names change after system updates or hardware reconfiguration.


;Step 1: Identify the correct interface name:
# Increase buffer to handle processing spikes (value in MB)
Use either of these commands to list all available network interfaces:
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
<syntaxhighlight lang="bash">
# Larger buffer absorbs I/O and CPU spikes without packet loss
# Option 1: Modern Linux systems
max_buffer_mem = 10000
ip a


# Option 2: Older systems
# Use IP filter instead of BPF (more efficient)
ifconfig
interface_ip_filter = 10.0.0.0/8
interface_ip_filter = 192.168.0.0/16
# Comment out any 'filter' parameter
</syntaxhighlight>
</syntaxhighlight>


Look for the interface that is receiving traffic. Common interface names include:
==== CPU Optimizations ====
* <code>eth0, eth1, eth2...</code> (classic Ethernet naming)
* <code>ens33, ens34, enp0s3...</code> (predictable naming on modern systems)
* <code>enp2s0f0, enp2s0f1...</code> (multi-port NICs)
 
;Step 2: Verify the interface exists and is UP:
<syntaxhighlight lang="bash">
# Check specific interface status (replace eth0 with your interface name)
ip link show eth0
 
# The output should show "UP" and "LOWER_UP" to indicate the interface is active
</syntaxhighlight>


;Step 3: Update <code>/etc/voipmonitor.conf</code>:
Edit the <code>interface</code> directive to use the correct interface name:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
# /etc/voipmonitor.conf
interface = ens33
</syntaxhighlight>


;Step 4: Restart the VoIPmonitor service:
# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
<syntaxhighlight lang="bash">
jitterbuffer_f1 = no
systemctl restart voipmonitor
jitterbuffer_f2 = yes
systemctl status voipmonitor
jitterbuffer_adapt = no
 
# If MOS metrics are not needed at all, disable everything:
# jitterbuffer_f1 = no
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no
</syntaxhighlight>
</syntaxhighlight>


Verify the service shows <code>Active: active (running)</code> after the restart.
==== Kernel Bypass Solutions (Extreme Loads) ====
 
;2. Check the <code>sipport</code> directive:
By default, VoIPmonitor only listens on port 5060. If your PBX uses a different port for SIP, you must add it. '''Common causes of missing calls:'''
* '''Missing ports:''' Some providers use alternate SIP ports (5061, 5080, etc.). If these are not listed, calls on those ports will be ignored.
* '''Syntax errors:''' List multiple ports comma-separated without extra commas or trailing characters. Correct syntax: <code>sipport = 5060,5061</code> or <code>sipport = 5060,5080</code>
* '''Ranges:''' You can specify port ranges using dashes: <code>sipport = 5060,5070-5080</code>
Example:
<code>sipport = 5060,5080</code>
 
;3. '''Distributed/Probe Setup Considerations:'''
If you are using a remote sensor (probe) with Packet Mirroring (<code>packetbuffer_sender=yes</code>), call detection depends on configuration on '''both''' the probe and the central analysis host.


Common symptom: The probe captures traffic (visible via <code>tcpdump</code>), but the central server records incomplete or missing CDRs for calls on non-default ports.
When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
{| class="wikitable"
|-
|-
! colspan="2" style="background:#ffc107;" | Critical: Both Systems Must Have Matching sipport Configuration
! Solution !! Type !! CPU Reduction !! Use Case
|-
|-
| style="vertical-align: top;" | '''Probe side:'''
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
| The probe captures packets from the network interface. Its <code>sipport</code> setting determines which UDP ports it considers as SIP traffic to capture and forward.
|-
|-
| style="vertical-align: top;" | '''Central server side:'''
| '''[[Napatech]]''' || Hardware SmartNIC || >97% (< 3% at 10Gbit) || Extreme performance requirements
| When receiving raw packets in Packet Mirroring mode, the central server analyzes the packets locally. Its <code>sipport</code> setting determines which ports it interprets as SIP during analysis. If a port is missing here, packets are captured but not recognized as SIP, resulting in missing CDRs.
|}
|}


:'''Troubleshooting steps for distributed probe setups:'''
==== Verify Improvement ====
 
::1. Verify traffic reachability on the probe:
::Use <code>tcpdump</code> on the probe VM to confirm SIP packets for the missing calls are arriving on the expected ports.
::<pre>
::# On the probe VM
::tcpdump -i eth0 -n port 5061
::</pre>
 
::2. Check the probe's ''voipmonitor.conf'':
::Ensure the <code>sipport</code> directive on the probe includes all necessary SIP ports used in your network.
::<syntaxhighlight lang="ini">
::# /etc/voipmonitor.conf on the PROBE
::sipport = 5060,5061,5080,6060
::</syntaxhighlight>
 
::3. Check the central analysis host's ''voipmonitor.conf'':
::'''This is the most common cause of missing calls in distributed setups.''' The central analysis host (the system receiving packets via <code>server_bind</code> or legacy <code>mirror_bind</code>) must also have the <code>sipport</code> directive configured with the same list of ports used by all probes.
::<syntaxhighlight lang="ini">
::# /etc/voipmonitor.conf on the CENTRAL HOST
::sipport = 5060,5061,5080,6060
::</syntaxhighlight>
 
::4. Restart both services:
::Apply the configuration changes:
::<syntaxhighlight lang="bash">
::# On both probe and central host
::systemctl restart voipmonitor
::</syntaxhighlight>


:For more details on distributed architecture configuration and packet mirroring, see [[Sniffer_distributed_architecture|Distributed Architecture: Client-Server Mode]].
;4. Check for a restrictive <code>filter</code>:
:If you have a BPF <code>filter</code> configured, ensure it is not accidentally excluding the traffic you want to see. For debugging, try commenting out the <code>filter</code> line entirely and restarting the sensor.
== Step 5: Check GUI Capture Rules (Causing Call Stops) ==
If <code>tshark</code> sees SIP traffic and the sniffer configuration appears correct, but the probe stops processing calls or shows traffic only on the network interface, GUI capture rules may be the culprit.
Capture rules configured in the GUI can instruct the sniffer to ignore ("skip") all processing for matched calls. This includes calls matching specific IP addresses or telephone number prefixes.
;1. Review existing capture rules:
:Navigate to '''GUI -> Capture rules''' and examine all rules for any that might be blocking your traffic.
:Look specifically for rules with the '''Skip''' option set to '''ON''' (displayed as "Skip: ON"). The Skip option instructs the sniffer to completely ignore matching calls (no files, RTP analysis, or CDR creation).
;2. Test by temporarily removing all capture rules:
:To isolate the issue, first create a backup of your GUI configuration:
:* Navigate to '''Tools -> Backup & Restore -> Backup GUI -> Configuration tables'''
:* This saves your current settings including capture rules
:* Delete all capture rules from the GUI
:* Click the '''Apply''' button to save changes
:* Reload the sniffer by clicking the green '''"reload sniffer"''' button in the control panel
:* Test if calls are now being processed correctly
:* If resolved, restore the configuration from the backup and systematically investigate the rules to identify the problematic one
;3. Identify the problematic rule:
:* After restoring your configuration, remove rules one at a time and reload the sniffer after each removal
:* When calls start being processed again, you have identified the problematic rule
:* Review the rule's match criteria (IP addresses, prefixes, direction) against your actual traffic pattern
:* Adjust the rule's conditions or Skip setting as needed
;4. Verify rules are reloaded:
:After making changes to capture rules, remember that changes are '''not automatically applied''' to the running sniffer. You must click the '''"reload sniffer"''' button in the control panel, or the rules will continue using the previous configuration.
For more information on capture rules, see [[Capture_rules]].
== Troubleshooting: Service Fails to Start with "failed read rsa key" Error ==
If the VoIPmonitor sniffer service fails to start and logs the error message "failed read rsa key," this indicates that the manager key cannot be loaded from the database.
=== Cause ===
The manager_key is stored in the <code>system</code> database table (identified by <code>type='manager_key'</code>) and is required for proper manager/sensor operations in distributed deployments. This error most commonly occurs when the <code>mysqlloadconfig</code> option in <code>voipmonitor.conf</code> is set to <code>no</code>, which prevents VoIPmonitor from loading configuration (including the manager_key) from the database.
=== Troubleshooting Steps ===
;1. Check for the error in system logs:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Monitor thread CPU after changes
tail -f /var/log/syslog | grep voipmonitor
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"


# For CentOS/RHEL/AlmaLinux
# Or monitor syslog
tail -f /var/log/messages | grep voipmonitor
 
# For systemd systems
journalctl -u voipmonitor -f
journalctl -u voipmonitor -f
# t0CPU should drop, heap values should stay < 20%
</syntaxhighlight>
</syntaxhighlight>


;2. Verify mysqlloadconfig setting:
{{Note|1=After changes, monitor syslog <code>heap[A&#124;B&#124;C]</code> values - should stay below 20% during peak traffic. See [[Syslog_Status_Line]] for detailed metric explanations.}}
<syntaxhighlight lang="bash">
# Check if mysqlloadconfig is set to no in voipmonitor.conf
grep mysqlloadconfig /etc/voipmonitor.conf
</syntaxhighlight>
 
If the output shows <code>mysqlloadconfig = no</code>, this is the cause of the issue.
 
;3. Fix the mysqlloadconfig setting:
<syntaxhighlight lang="bash">
# Edit the configuration file
nano /etc/voipmonitor.conf
 
# Either remove the mysqlloadconfig line entirely (defaults to yes)
# Or uncomment/set to yes:
# mysqlloadconfig = yes
 
# Restart the sniffer service
systemctl restart voipmonitor
 
# Check if it started successfully
systemctl status voipmonitor
</syntaxhighlight>


;4. Verify manager_key exists in database:
== Storage Hardware Failure ==
<syntaxhighlight lang="sql">
-- Query the manager_key from the system table
SELECT * FROM voipmonitor.`system` WHERE type='manager_key'\G
</syntaxhighlight>


If no manager_key exists, check your VoIPmonitor installation and consider running the installer or contacting support to regenerate the key.
'''Symptom''': Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.


;5. Check database connectivity and permissions:
'''Diagnosis''':
Verify that the VoIPmonitor sniffer can connect to the database and has read access to the <code>system</code> table.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test database connectivity with the configured credentials
# Check disk health
mysql -h <mysqlhost> -u <mysqlusername> -p <mysqldb>
smartctl -a /dev/sda


# Inside MySQL, verify the user has SELECT on voipmonitor.system
# Check RAID status (if applicable)
SHOW GRANTS FOR 'voipmonitor_user'@'%';
cat /proc/mdstat
mdadm --detail /dev/md0
</syntaxhighlight>
</syntaxhighlight>


;6. Check configuration consistency between probe and server:
Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.
In distributed deployments with probe and server components, ensure that both systems have consistent configuration in <code>/etc/voipmonitor.conf</code>. Specifically, both should have the same database connection settings and <code>mysqlloadconfig</code> should be enabled on both systems.


=== Summary ===
== OOM (Out of Memory) ==


The "failed read rsa key" error is almost always caused by <code>mysqlloadconfig=no</code> in <code>voipmonitor.conf</code>. The solution is to remove or change this setting to <code>yes</code>, then restart the service.
=== Identify OOM Victim ===


== Step 6: Check VoIPmonitor Logs for Errors ==
Finally, VoIPmonitor's own logs are the best source for clues. Check the system log for any error messages generated by the sensor on startup or during operation.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Check for OOM kills
tail -f /var/log/syslog | grep voipmonitor
dmesg | grep -i "out of memory\|oom\|killed process"
 
journalctl --since "1 hour ago" | grep -i oom
# For CentOS/RHEL/AlmaLinux
tail -f /var/log/messages | grep voipmonitor
</syntaxhighlight>
</syntaxhighlight>
Look for errors like:
* "pcap_open_live(eth0) error: eth0: No such device" (Wrong interface name)
* "Permission denied" (The sensor is not running with sufficient privileges)
* Errors related to database connectivity.
* Messages about dropping packets.


== Step 7: Check for OOM (Out of Memory) Issues ==
=== MySQL Killed by OOM ===
If VoIPmonitor suddenly stops processing CDRs and a service restart temporarily restores functionality, the system may be experiencing OOM (Out of Memory) killer events. The Linux OOM killer terminates processes when available RAM is exhausted, and MySQL (<code>mysqld</code>) is a common target due to its memory-intensive nature.


;1. Check for OOM killer events in kernel logs:
Reduce InnoDB buffer pool:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
# For Debian/Ubuntu
# /etc/mysql/my.cnf
grep -i "out of memory\|killed process" /var/log/syslog | tail -20
innodb_buffer_pool_size = 2G  # Reduce from default
 
# For CentOS/RHEL/AlmaLinux
grep -i "out of memory\|killed process" /var/log/messages | tail -20
 
# Also check dmesg:
dmesg | grep -i "killed process" | tail -10
</syntaxhighlight>
</syntaxhighlight>
Typical OOM killer messages look like:
<pre>
Out of memory: Kill process 1234 (mysqld) score 123 or sacrifice child
Killed process 1234 (mysqld) total-vm: 12345678kB, anon-rss: 1234567kB
</pre>


;2. Monitor current memory usage:
=== Voipmonitor Killed by OOM ===
<syntaxhighlight lang="bash">
# Check available memory (look for low 'available' or 'free' values)
free -h


# Check per-process memory usage (sorted by RSS)
Reduce buffer sizes in voipmonitor.conf:
ps aux --sort=-%mem | head -15
<syntaxhighlight lang="ini">
 
max_buffer_mem = 2000  # Reduce from default
# Check MySQL memory usage in bytes
ringbuffer = 50        # Reduce from default
cat /proc/$(pgrep mysqld)/status | grep -E "VmSize|VmRSS"
</syntaxhighlight>
</syntaxhighlight>
Warning signs:
* '''Available memory consistently below 500MB during operation'''
* '''MySQL consuming most of the available RAM'''
* '''Swap usage near 100% (if swap is enabled)'''
* '''Frequent process restarts without clear error messages'''


;3. Solution: Increase physical memory:
=== Runaway External Process ===
The definitive solution for OOM-related CDR processing issues is to upgrade the server's physical RAM. After upgrading:
* Verify memory improvements with <code>free -h</code>
* Monitor for several days to ensure OOM events stop
* Consider tuning <code>innodb_buffer_pool_size</code> in your MySQL configuration to use the additional memory effectively


Additional mitigation strategies (while planning for RAM upgrade):
<syntaxhighlight lang="bash">
* Reduce MySQL's memory footprint by lowering <code>innodb_buffer_pool_size</code> (e.g., from 16GB to 8GB)
# Find memory-hungry processes
* Disable or limit non-essential VoIPmonitor features (e.g., packet capture storage, RTP analysis)
ps aux --sort=-%mem | head -20
* Ensure swap space is properly configured as a safety buffer (though swap is much slower than RAM)
* Use <code>sysctl vm.swappiness=10</code> to favor RAM over swap when some memory is still available
 
== Step 8: Missing CDRs for Calls with Large Packets ==
If VoIPmonitor is capturing some calls successfully but missing CDRs for specific calls (especially those that seem to have larger SIP packets like INVITEs with extensive SDP), there are two common causes to investigate.


=== Cause 1: snaplen Packet Truncation (VoIPmonitor Configuration) ===
# Kill orphaned/runaway process
The <code>snaplen</code> parameter in <code>voipmonitor.conf</code> limits how many bytes of each packet are captured. If a SIP packet exceeds <code>snaplen</code>, it is truncated and the sniffer may fail to parse the call correctly.
kill -9 <PID>
 
;1. Check your current snaplen setting:
<syntaxhighlight lang="bash">
grep snaplen /etc/voipmonitor.conf
</syntaxhighlight>
</syntaxhighlight>
Default is 3200 bytes (6000 if SSL/HTTP is enabled).
For servers limited to '''16GB RAM''' or when experiencing repeated MySQL OOM kills:
 
;2. Test if packet truncation is the issue:
Use <code>tcpdump</code> with <code>-s0</code> (snap infinite) to capture full packets:
<syntaxhighlight lang="bash">
# Capture SIP traffic with full packet length
tcpdump -i eth0 -s0 -nn port 5060 -w /tmp/test_capture.pcap


# Analyze packet sizes with Wireshark or tshark
tshark -r /tmp/test_capture.pcap -T fields -e frame.len -Y "sip" | sort -n | tail -10
</syntaxhighlight>
If you see SIP packets larger than your <code>snaplen</code> value (e.g., 4000+ bytes), increase <code>snaplen</code> in <code>voipmonitor.conf</code>:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
snaplen = 65535
# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
</syntaxhighlight>
[mysqld]
Then restart the sniffer: <code>systemctl restart voipmonitor</code>.
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G


=== Cause 2: MTU Mismatch (Network Infrastructure) ===
# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
If packets are being lost or fragmented due to MTU mismatches in the network path, VoIPmonitor may never receive the complete packets, regardless of <code>snaplen</code> settings.
innodb_flush_log_at_trx_commit = 2
 
;1. Diagnose MTU-related packet loss:
Capture traffic with tcpdump and analyze in Wireshark:
<syntaxhighlight lang="bash">
# Capture traffic on the VoIPmonitor host
tcpdump -i eth0 -s0 host <pbx_ip_address> -w /tmp/mtu_test.pcap
</syntaxhighlight>
</syntaxhighlight>
Open the pcap in Wireshark and look for:
* Reassembled PDUs marked as incomplete
* TCP retransmissions for the same packet
* ICMP "Fragmentation needed" messages (Type 3, Code 4)
;2. Verify packet completeness:
In Wireshark, examine large SIP INVITE packets. If the SIP headers or SDP appear cut off or incomplete, packets are likely being lost in transit due to MTU issues.
;3. Identify the MTU bottleneck:
The issue is typically a network device with a lower MTU than the end devices. Common locations:
* VPN concentrators
* Firewalls
* Routers with tunnel interfaces
* Cloud provider gateways (typically 1500 bytes vs. standard 9000 jumbo frames)
To locate the problematic device, trace the MTU along the network path from the PBX to the VoIPmonitor sensor.
;4. Resolution options:
* Increase MTU on the bottleneck device to match the rest of the network (e.g., from 1500 to 9000 for jumbo frame environments)
* Enable Path MTU Discovery (PMTUD) on intermediate devices
* Ensure your switching infrastructure supports jumbo frames end-to-end if you are using them
For more information on the <code>snaplen</code> parameter, see [[Sniffer_configuration#Network_Interface_.26_Sniffing|Sniffer Configuration]].
=== Cause 3: External Source Packet Truncation (Traffic Mirroring/LBS Modules) ===
If packets are truncated or corrupted BEFORE they reach VoIPmonitor, changing <code>snaplen</code> will NOT fix the issue. This scenario occurs when using external SIP sources that have their own packet size limitations.
; Symptoms to identify this scenario:
* Large SIP packets (e.g., WebRTC INVITE with big Authorization headers ~4k) appear truncated
* Packets show as corrupted or malformatted in VoIPmonitor GUI
* Changing <code>snaplen</code> in <code>voipmonitor.conf</code> has no effect
* Using TCP instead of UDP in the external system does not resolve the issue
; Common external sources that may truncate packets:
# Kamailio <code>siptrace</code> module
# FreeSWITCH <code>sip_trace</code> module
# OpenSIPS tracing modules
# Custom HEP/HOMER agent implementations
# Load balancers or proxy servers with traffic mirroring


; Diagnose external source truncation:
Restart MySQL after changes:
Use <code>tcpdump</code> with <code>-s0</code> (snap infinite) on the VoIPmonitor sensor to compare packet sizes:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Capture traffic received by VoIPmonitor
systemctl restart mysql
sudo tcpdump -i eth0 -s0 -nn port 5060 -w /tmp/voipmonitor_input.pcap
# or
 
systemctl restart mariadb
# Analyze actual packet sizes received
tshark -r /tmp/voipmonitor_input.pcap -T fields -e frame.len -Y "sip.Method == INVITE" | sort -n | tail -10
</syntaxhighlight>
 
If:
* You see packets with truncated SIP headers or incomplete SDP
* The packet length is much smaller than expected (e.g., 1500 bytes instead of 4000+ bytes)
* Truncation is consistent across all calls
 
Then the external source is truncating packets before they reach VoIPmonitor.
 
; Solutions for Kamailio siptrace truncation:
If using Kamailio's <code>siptrace</code> module with traffic mirroring:
 
1. Configure Kamailio to use TCP transport for siptrace (may help in some cases):
<pre>
# In kamailio.cfg
modparam("siptrace", "duplicate_uri", "sip:voipmonitor_ip:port;transport=tcp")
</pre>
 
2. If Kamailio reports "Connection refused", VoIPmonitor does not open a TCP listener by default. Manually open one:
<syntaxhighlight lang="bash">
# Open TCP listener using socat
socat TCP-LISTEN:5888,fork,reuseaddr &
</syntaxhighlight>
</syntaxhighlight>
Then update kamailio.cfg to use the specified port instead of the standard SIP port.
=== SQL Queue Growth from Non-Call Data ===


3. Use HAProxy traffic 'tee' function (recommended):
If <code>sip-register</code>, <code>sip-options</code>, or <code>sip-subscribe</code> are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.
If your architecture includes HAProxy in front of Kamailio, use its traffic mirroring to send a copy of the WebSocket traffic directly to VoIPmonitor's standard SIP listening port. This bypasses the siptrace module entirely and preserves original packets:
<pre>
# In haproxy.cfg, within your frontend/backend configuration
# Send a copy of traffic to VoIPmonitor
option splice-response
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
use-server voipmonitor if { req_ssl_hello_type 1 }
listen voipmonitor_mirror
    bind :5888
    mode tcp
    server voipmonitor <voipmonitor_sensor_ip>:5060 send-proxy
</pre>


Note: The exact HAProxy configuration depends on your architecture and whether you are mirroring TCP (WebSocket) or UDP traffic.
{{Warning|1=Even with reduced <code>innodb_buffer_pool_size</code>, SQL queue will grow indefinitely without cleanup of non-call data.}}
 
; Solutions for other external sources:
# Check the external system's documentation for packet size limits or truncation settings
# Consider using standard network mirroring (SPAN/ERSPAN/GRE) instead of SIP tracing modules
# Ensure the external system captures full packet lengths (disable any internal packet size caps)
# Verify that the external system does not reassemble or modify SIP packets before forwarding
 
== Troubleshooting: CDR Shows 000 No Response Despite Valid SIP Response ==
 
If the CDR View displays "000 No Response" in the Last Response column for calls that actually have valid final SIP response codes (such as 403 Forbidden, 500 Server Error, etc.), this indicates that the sniffer is receiving response packets but failing to correlate them with their corresponding INVITE transactions before writing the CDR.
 
=== Diagnosis: Verify Response Packets Are Captured ===
 
;1. Locate the affected call in the CDR View:
:* Find a call showing "000 No Response" in the Last Response column.
 
;2. Check the SIP History:
:* Click the [+] icon to expand the call's detail view.
:* Open the "SIP History" tab.
:* Look for the actual SIP response (e.g., 403 Forbidden, 486 Busy Here, 500 Internal Server Error).
 
If the response packet IS present in SIP History, the issue is a correlation timing problem. Proceed to the solution below.
 
If the response packet is NOT present in SIP History, the issue is a network visibility problem (see [[#SPAN_Configuration_Troubleshooting|Step 3: Investigate Packet Encapsulation]] and other network troubleshooting sections).
 
=== Root Cause: libpcap Packet Queue Timeout ===
 
The issue is caused by VoIPmonitor's libpcap packet capture timing out before responses can be matched to their originating INVITEs. This typically occurs in high-traffic environments or when packet processing is temporarily delayed due to system load.
 
The sniffer creates CDR records based on SIP INVITE packets. It attempts to correlate subsequent SIP responses (403, 500, etc.) with the original INVITE. If the packet queue processing is too slow or the time window is too short, responses arrive after the CDR has already been written with "Last Response" set to 0.
 
=== Solution: Configure libpcap Nonblocking Mode ===
 
Edit the "/etc/voipmonitor.conf" file on the sniffer host and add the following parameters:


'''Solution: Enable automatic cleanup of old non-call data'''
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Enable libpcap nonblocking mode to prevent packet queue blocking
# /etc/voipmonitor.conf
libpcap_nonblock_mode = yes
# cleandatabase=2555 automatically deletes partitions older than 7 years
 
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
# Increase packet deque window length (in milliseconds) for response correlation
cleandatabase = 2555
# Default is often 2000ms, increasing to 5000ms gives more time for responses
pcap_queue_deque_window_length = 5000
</syntaxhighlight>
</syntaxhighlight>


Save the file and restart the voipmonitor service:
Restart the sniffer after changes:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


=== Additional Considerations ===
{{Note|See [[Data_Cleaning]] for detailed configuration options and other <code>cleandatabase_*</code> parameters.}}
 
== Service Startup Failures ==
;If the issue persists after applying the fix:
:* Try increasing <code>pcap_queue_deque_window_length</code> further (e.g., to 7000 or 10000 milliseconds)
:* Check system load to ensure the server is not under heavy CPU or I/O pressure
:* Verify adequate <code>ringbuffer</code> size is configured for your traffic volume (see [[Scaling|Scaling and Performance Tuning]])
 
;For distributed architectures:
:* Ensure all voipmonitor hosts have synchronized time (see [[#Verify_System_Time_Synchronization]])
:* Time mismatches between components can cause correlation failures
 
{{Note|The <code>pcap_queue_deque_window_length</code> parameter is also used in distributed mirroring scenarios to sort packets from multiple mirrors. Increasing this value improves packet correlation in both single-sensor and distributed setups.}}
 
For more information on packet capture configuration, see [[Sniffer_configuration|Sniffer Configuration]].
 
== Step 9: Probe Timeout Due to Virtualization Timing Issues ==
 
If remote probes are intermittently disconnecting from the central server with timeout errors, even on a high-performance network with low load, the issue may be related to virtualization host timing problems rather than network connectivity.


=== Diagnosis: Check System Log Timing Intervals ===
=== Interface No Longer Exists ===


The VoIPmonitor sensor generates status log messages approximately every 10 seconds during normal operation. If the timing system on the probe is inconsistent, the interval between these status messages can exceed 30 seconds, triggering a connection timeout.
After OS upgrade, interface names may change (eth0 → ensXXX):


;1. Monitor the system log on the affected probe:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
tail -f /var/log/syslog | grep voipmonitor
# Find current interface names
</syntaxhighlight>
ip a
 
;2. Examine the timestamps of voipmonitor status messages:
Look for repeating log entries that should appear approximately every 10 seconds during normal operations.
 
;3. Identify timing irregularities:
Calculate the time interval between successive status log entries. '''If the interval exceeds 30 seconds''', this indicates a timing system problem that will cause connection timeouts with the central server.
 
=== Root Cause: Virtualization Host RDTSC Issues ===
 
This problem is '''not''' network-related. It is a host-level timing issue that impacts the application's internal timers.
 
The issue typically occurs on virtualized probes where the host's CPU timekeeping is inconsistent. Specifically, problems with the RDTSC (Read Time-Stamp Counter) CPU instruction on the virtualization host can cause:
 
* Irregular system clock behavior on the guest VM
* Application timers that do not fire consistently
* Sporadic timeouts in client-server connections
 
=== Resolution ===


;1. Investigate the virtualization host configuration:
# Update all config locations
Check the host's hypervisor or virtualization platform documentation for known timekeeping issues related to RDTSC.
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/


Common virtualization platforms with known timing considerations:
# Also check GUI: Settings → Sensors → Configuration
* KVM/QEMU: Check CPU passthrough and TSC mode settings
* VMware: Verify time synchronization between guest and host
* Hyper-V: Review Integration Services time sync configuration
* Xen: Check TSC emulation settings
 
;2. Apply host-level fixes:
These are host-level fixes, not changes to the guest VM configuration. Consult your virtualization platform's documentation for specific steps to address RDTSC timing issues.
 
Typical solutions include:
* Enabling appropriate TSC modes on the host
* Configuring CPU features passthrough correctly
* Adjusting hypervisor timekeeping parameters
 
;3. Verify the fix:
After applying the host-level configuration changes, monitor the probe's status logs again to confirm that the timing intervals are now consistently around 10 seconds (never exceeding 30 seconds).
 
<syntaxhighlight lang="bash">
# Monitor for regular status messages
tail -f /var/log/syslog | grep voipmonitor
</syntaxhighlight>
</syntaxhighlight>


Once the timing is corrected, probe connections to the central server should remain stable without intermittent timeouts.
=== Missing Dependencies ===
 
== Troubleshooting: Audio Missing on One Call Leg ==
 
If the sniffer captures full audio on one call leg (e.g., carrier/outside) but only partial or no audio on the other leg (e.g., PBX/inside), use this diagnostic workflow to identify the root cause BEFORE applying any configuration fixes.
 
The key question to answer is: '''Are the RTP packets for the silent leg present on the wire?'''
 
=== Step 1: Use tcpdump to Capture Traffic During a Test Call ===
 
Initiate a new test call that reproduces the issue. During the call, use tcpdump or tshark directly on the sensor's sniffing interface to capture all traffic:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Capture traffic to a file during the test call
# Install common missing package
# Replace eth0 with your sniffing interface
apt install libpcap0.8  # Debian/Ubuntu
tcpdump -i eth0 -s0 -w /tmp/direct_capture.pcap
yum install libpcap    # RHEL/CentOS
 
# OR: Display live traffic for specific IPs (useful for real-time diagnostics)
tcpdump -i eth0 -s0 -nn "host <pbx_ip> or host <carrier_ip>"
</syntaxhighlight>
</syntaxhighlight>


Let the call run for 10-30 seconds, then stop tcpdump with Ctrl+C.
== Network Interface Issues ==
 
=== Step 2: Retrieve VoIPmonitor GUI's PCAP for the Same Call ===
 
After the call completes:
1. Navigate to the '''CDR View''' in the VoIPmonitor GUI
2. Find the test call you just made
3. Download the PCAP file for that call (click the PCAP icon/button)
4. Save it as: <code>/tmp/gui_capture.pcap</code>
 
=== Step 3: Compare the Two Captures ===


Analyze both captures to determine if RTP packets for the silent leg are present on the wire:
=== Promiscuous Mode ===


Required for SPAN port monitoring:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Count RTP packets in the direct capture
# Enable
tshark -r /tmp/direct_capture.pcap -Y "rtp" | wc -l
ip link set eth0 promisc on


# Count RTP packets in the GUI capture
# Verify
tshark -r /tmp/gui_capture.pcap -Y "rtp" | wc -l
ip link show eth0 | grep PROMISC
 
# Check for RTP from specific source IPs in the direct capture
tshark -r /tmp/direct_capture.pcap -Y "rtp" -T fields -e rtp.ssrc -e ip.src -e ip.dst
 
# Check Call-ID in both captures to verify they're the same call
tshark -r /tmp/direct_capture.pcap -Y "sip" -T fields -e sip.Call-ID | head -1
tshark -r /tmp/gui_capture.pcap -Y "sip" -T fields -e sip.Call-ID | head -1
</syntaxhighlight>
</syntaxhighlight>


=== Step 4: Interpret the Results ===
{{Note|Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.}}
 
{| class="wikitable" style="background:#e7f3ff; border:1px solid #3366cc;"
|-
! colspan="2" style="background:#3366cc; color: white;" | Diagnostic Decision Matrix
|-
! Observation
! Root Cause & Next Steps
|-
| '''RTP packets for silent leg are NOT present in direct capture'''
| '''Network/PBX Issue:''' The PBX or network is not sending the packets. This is not a VoIPmonitor problem. Troubleshoot the PBX (check NAT, RTP port configuration) or network (SPAN/mirror configuration, firewall rules).
|-
| '''RTP packets for silent leg ARE present in direct capture but missing in GUI capture'''
| '''Sniffer Configuration Issue:''' Packets are on the wire but VoIPmonitor is failing to capture or correlate them. Likely causes: NAT IP mismatch (natalias configuration incorrect), SIP signaling advertises different IP than RTP source, or restrictive filter rules. Proceed with configuration fixes.
|-
| '''RTP packets present in both captures but audio still silent'''
| '''Codec/Transcoding Issue:''' Packets are captured correctly but may not be decoded properly. Check codec compatibility, unsupported codecs, or transcoding issues on the PBX.
|}
 
=== Step 5: Apply the Correct Fix Based on Diagnosis ===
 
;If RTP is NOT on the wire (Network/PBX issue):
* Check PBX RTP port configuration and firewall rules
* Verify network SPAN/mirror is capturing bidirectional traffic (see [[#SPAN_Configuration_Troubleshooting|Section 3]])
* Check PBX NAT settings - RTP packets may be blocked or routed incorrectly
 
;If RTP is on the wire but not captured (Sniffer configuration issue):
 
==== Check rtp_check_both_sides_by_sdp Setting (Primary Cause) ====


This is the '''most common cause''' of one-way RTP capture when packets are present on the wire. The <code>rtp_check_both_sides_by_sdp</code> parameter controls how strictly RTP streams are correlated with SDP (Session Description Protocol) signaling.
=== Interface Drops ===


Check the current setting in <code>/etc/voipmonitor.conf</code>:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
grep "^rtp_check_both_sides_by_sdp" /etc/voipmonitor.conf
# Check for drops
</syntaxhighlight>
ip -s link show eth0 | grep -i drop
 
If the setting is <code>yes</code> or <code>strict</code> or <code>very_strict</code>, this requires '''BOTH sides of RTP to exactly match SDP (SIP signaling)''':
* <code>strict</code>: Only allows verified packets after first match (blocks unverified)
* <code>very_strict</code>: Blocks all unverified packets (most restrictive)
* <code>keep_rtp_packets</code>: Same as <code>yes</code> but stores unverified packets for debugging


Symptoms of restrictive <code>rtp_check_both_sides_by_sdp</code> settings:
# If drops present, increase ring buffer
* Only one call leg appears in CDR (caller OR called, not both)
ethtool -G eth0 rx 4096
* Received packets column shows 0 or very low on one leg
* tcpdump shows both RTP streams present, but GUI captures only one
* Affects many calls, not just specific ones
 
'''Solution:''' Change to <code>no</code> or comment out the line:
 
<syntaxhighlight lang="ini">
; /etc/voipmonitor.conf
rtp_check_both_sides_by_sdp = no
</syntaxhighlight>
</syntaxhighlight>


Restart the sniffer to apply:
=== Bonded/EtherChannel Interfaces ===


<syntaxhighlight lang="bash">
'''Symptom''': False packet loss when monitoring bond0 or br0.
systemctl restart voipmonitor
</syntaxhighlight>


If you previously set <code>rtp_check_both_sides_by_sdp = yes</code> to solve audio mixing issues in multi-call environments where multiple calls share the same IP:port, consider using alternative approaches like <code>sdp_multiplication</code> instead, as enabling strict checking breaks one-way RTP capture.
'''Solution''': Monitor physical interfaces, not logical:
 
==== Other Configuration Checks ====
 
If checking <code>rtp_check_both_sides_by_sdp</code> does not resolve the issue, proceed with these additional diagnostic steps:
 
* Configure '''natalias''' in <code>/etc/voipmonitor.conf</code> to map the IP advertised in SIP signaling to the actual RTP source IP (NAT scenarios only):
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
; /etc/voipmonitor.conf
# voipmonitor.conf - use physical interfaces
natalias = <Public_IP_Signaled> <Private_IP_Actual>
interface = eth0,eth1
</syntaxhighlight>
</syntaxhighlight>
: When using <code>natalias</code>, ensure <code>rtp_check_both_sides_by_sdp</code> is set to <code>no</code> (the default).
* Check for restrictive <code>filter</code> directives in <code>voipmonitor.conf</code>
* Verify <code>sipport</code> includes all necessary SIP ports


;If packets are captured but audio silent (Codec issue):
=== Network Offloading Issues ===
* Check CDR view for codec information on both legs
* Verify VoIPmonitor GUI has the necessary codec decoders installed
* Check for codec mismatches between call legs (transcoding may be missing)


=== Step 6: Verify the Fix After Configuration Changes ===
'''Symptom''': Kernel errors like <code>bad gso: type: 1, size: 1448</code>
 
After making changes in <code>/etc/voipmonitor.conf</code>:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Restart the sniffer
# Disable offloading on capture interface
systemctl restart voipmonitor
ethtool -K eth0 gso off tso off gro off lro off
 
# Make another test call and repeat the diagnostic workflow
# Compare direct vs GUI capture again
</syntaxhighlight>
</syntaxhighlight>


Confirm that RTP packets for the problematic leg now appear in both the direct tcpdump capture AND the GUI's PCAP file.
== Packet Ordering Issues ==


'''Note:''' This diagnostic methodology helps you identify whether the issue is in the network infrastructure (PBX, SPAN, firewall) or in VoIPmonitor configuration (natalias, filters). Applying VoIPmonitor configuration fixes when the root cause is a network issue will not resolve the problem.
If SIP messages appear out of sequence:


== Troubleshooting: Server Coredumps and SQL Queue Overload ==
'''First''': Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See [[FAQ]].


If the VoIPmonitor server is experiencing regular coredumps, the cause may be an SQL queue bottleneck that exceeds system limits. The SQL queue grows when the database cannot keep up with the rate of data being inserted from VoIPmonitor.
'''If genuine reordering''': Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see [[Napatech]]).
{{Note|For out-of-order packets in '''client/server mode''' (multiple sniffers), see [[Sniffer_distributed_architecture]] for <code>pcap_queue_dequeu_window_length</code> configuration.}}


=== Symptoms ===
=== Solutions for SPAN/Mirroring Reordering ===


* Server crashes or coredumps regularly, often during peak traffic hours
If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):
* Syslog messages showing a growing <code>SQLq</code> counter (SQL queries waiting)
* Crashes occur when OPTIONS, SUBSCRIBE, and NOTIFY messages are being processed at high volume


=== Identify the Root Cause ===
1. '''Configure switch to preserve packet order''': Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.


;1. Check the SQL queue metric in syslog:
2. '''Replace SPAN with TAP or packet broker''': Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.
<syntaxhighlight lang="bash">
== Database Issues ==
# Debian/Ubuntu
tail -f /var/log/syslog | grep "SQLq"


# CentOS/RHEL
=== SQL Queue Overload ===
tail -f /var/log/messages | grep "SQLq"
</syntaxhighlight>


Look for the <code>SQLq[XXX]</code> value where XXX is the number of queued SQL commands. If this number is consistently growing or reaching high values (thousands or more), the database is a bottleneck.
'''Symptom''': Growing <code>SQLq</code> metric, potential coredumps.
 
;2. Check if SIP message processing is enabled:
<syntaxhighlight lang="bash">
grep -E "sip-options=|sip-subscribe=|sip-notify=" /etc/voipmonitor.conf
</syntaxhighlight>
 
If these are set to <code>yes</code> and you have a high volume of these messages (OPTIONS pings sent frequently by SIP devices), this can overwhelm the database insert thread queue.
 
=== Solutions ===
 
There are three approaches to resolve SQL queue overload coredumps:
 
==== Solution 1: Increase MySQL Insert Threads ====
 
Increase the number of threads dedicated to inserting SIP messages into the database. This allows more parallel database operations.
 
Edit <code>/etc/voipmonitor.conf</code> and add or modify:


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Increase insert threads for SIP messages (default is 4, increase to 8 or higher for high traffic)
# voipmonitor.conf - increase threads
mysqlstore_max_threads_sip_msg = 8
mysqlstore_concat_limit_cdr = 1000
</syntaxhighlight>
cdr_check_exists_callid = 0
 
Restart VoIPmonitor for the change to take effect:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


{{Tip|For very high traffic environments, you may need to increase this value further (e.g., 12 or 16).}}
=== Error 1062 - Lookup Table Limit ===
 
==== Solution 2: Disable High-Volume SIP Message Types ====
 
Reduce the load on the SQL queue by disabling processing of specific high-volume SIP message types that are not needed for your analysis.


Edit <code>/etc/voipmonitor.conf</code>:
'''Symptom''': <code>Duplicate entry '16777215' for key 'PRIMARY'</code>


'''Quick fix''':
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Disable processing and database storage for specific message types
# voipmonitor.conf
sip-options = no
cdr_reason_string_enable = no
sip-subscribe = no
sip-notify = no
</syntaxhighlight>
 
Restart VoIPmonitor:
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


{{Note|See [[SIP_OPTIONS/SUBSCRIBE/NOTIFY]] for detailed information on these options and when to use <code>nodb</code> mode instead of disabling entirely.}}
See [[Database_troubleshooting#Database_Error_1062_-_Lookup_Table_Auto-Increment_Limit|Database Troubleshooting]] for complete solution.


==== Solution 3: Optimize MySQL Performance ====
== Bad Packet Errors ==


Tune the MySQL/MariaDB server for better write performance to handle the high insert rate from VoIPmonitor.
'''Symptom''': <code>bad packet with ether_type 0xFFFF detected on interface</code>


Edit your MySQL configuration file (typically <code>/etc/mysql/my.cnf</code> or <code>/etc/mysql/mariadb.conf.d/50-server.cnf</code>):
'''Diagnosis''':
 
<syntaxhighlight lang="ini">
[mysqld]
# InnoDB buffer pool size - set to approximately 50-70% of available RAM on a dedicated database server
# On servers running VoIPmonitor and MySQL together, use approximately 30-50% of RAM
innodb_buffer_pool_size = 8G
 
# Reduce transaction durability for faster writes (may lose up to 1 second of data on crash)
innodb_flush_log_at_trx_commit = 2
</syntaxhighlight>
 
Restart MySQL and VoIPmonitor:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart mysql
# Run diagnostic (let run 30-60 seconds, then kill)
systemctl restart voipmonitor
voipmonitor --check_bad_ether_type=eth0
</syntaxhighlight>
 
{{Warning|Setting <code>innodb_flush_log_at_trx_commit</code> to <code>2</code> trades some data safety for performance. In the event of a power loss or crash, up to 1 second of the most recent transactions may be lost.}}
 
==== Solution 4: Immediate SQL Queue Clearing ====
 
If CDRs have stopped appearing in the GUI for several days and the SQL queue (SQLq) is backed up, you can immediately clear the backlog by deleting queue files and restarting services.
 
=== Symptoms for SQL Queue Backup ===


* No new CDRs appearing in the GUI for an extended period (days)
# Find and kill the diagnostic process
* Sensor status shows a non-zero <code>SQLq</code> (SQL queue) count in '''Settings > Sensors'''
ps ax | grep voipmonitor
* The delay in CDRs equals the age of the oldest query file in the queue
kill -9 <PID>
 
=== Diagnose SQL Queue Status ===
 
;1. Check sensor status in the GUI:
:* Navigate to '''Settings > Sensors'''
:* Expand the status for the affected sensor
:* Look for a non-zero <code>SQLq</code> value
 
;2. Alternatively check via command line:
<syntaxhighlight lang="bash">
# Check for SQL queue messages in syslog
grep "SQLq" /var/log/syslog | tail -20
 
# Check for qoq* queue files in voipmonitor directory
# The default directory is typically /voipmonitor or /var/lib/voipmonitor
ls -la /voipmonitor/qoq*
</syntaxhighlight>
</syntaxhighlight>


=== Clear SQL Queue and Restart Processing ===
Causes: corrupted packets, driver issues, VLAN tagging problems. Check <code>ethtool -S eth0</code> for interface errors.
 
If the SQL queue is backed up and you want to resume processing immediately:


;1. Restart the voipmonitor and rsyslog services to begin processing the queue naturally:
== Useful Diagnostic Commands ==
<syntaxhighlight lang="bash">
# Restart both services
systemctl restart voipmonitor
systemctl restart rsyslog
</syntaxhighlight>


;2. Monitor the SQLq count until it reaches zero. The delay in CDRs equals the age of the oldest query file in the queue.
=== tshark Filters for SIP ===


;3. If the queue is too large or you want to skip old data and resume immediately, clear the queue files:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Stop the voipmonitor service first to prevent new data writes
# All SIP INVITEs
systemctl stop voipmonitor
 
# Delete all qoq* queue files from the voipmonitor directory
# Replace /voipmonitor with your actual directory path
rm -f /voipmonitor/qoq*
 
# Restart the voipmonitor service
systemctl start voipmonitor
</syntaxhighlight>
 
;4. After clearing the queue, monitor the <code>SQLf</code> (failed queries) stat in the sensor status:
:* Navigate to '''Settings > Sensors'''
:* Expand the status for the affected sensor
:* Watch for the <code>SQLf</code> counter
:* Ensure it remains low (e.g., below 10) and does not grow, indicating new queries are processing successfully
 
{{Warning|Deleting queue files permanently discards all pending CDR data. Only do this if you are willing to lose the queued data or if the backlog is so old that the data is no longer useful.}}
 
{{Tip|The voipmonitor directory location varies by installation. Common locations include <code>/voipmonitor</code>, <code>/var/lib/voipmonitor</code>, or the directory specified by the <code>pcap_dir</code> configuration option in <code>voipmonitor.conf</code>.}}
 
=== Additional Troubleshooting ===
 
* If increasing threads and disabling SIP message types do not resolve the issue, check if the database server itself has performance bottlenecks (CPU, disk I/O, memory)
* For systems with extremely high call volumes, consider moving the database to a separate dedicated server
* Monitor the <code>SQLq</code> metric after making changes to verify the queue is not growing unchecked
 
== Appendix: tshark Display Filter Syntax for SIP ==
When using <code>tshark</code> to analyze SIP traffic, it is important to use the '''correct Wireshark display filter syntax'''. Below are common filter examples:
 
=== Basic SIP Filters ===
<syntaxhighlight lang="bash">
# Show all SIP INVITE messages
tshark -r capture.pcap -Y "sip.Method == INVITE"
tshark -r capture.pcap -Y "sip.Method == INVITE"


# Show all SIP messages (any method)
# Find specific phone number
tshark -r capture.pcap -Y "sip"
 
# Show SIP and RTP traffic
tshark -r capture.pcap -Y "sip || rtp"
</syntaxhighlight>
 
=== Search for Specific Phone Number or Text ===
<syntaxhighlight lang="bash">
# Find calls containing a specific phone number (e.g., 5551234567)
tshark -r capture.pcap -Y 'sip contains "5551234567"'
tshark -r capture.pcap -Y 'sip contains "5551234567"'


# Find INVITE messages for a specific number
# Get Call-IDs
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"'
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID
</syntaxhighlight>


=== Extract Call-ID from Matching Calls ===
# SIP errors (4xx, 5xx)
<syntaxhighlight lang="bash">
# Get Call-ID for calls matching a phone number
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"' -T fields -e sip.Call-ID
 
# Get Call-ID along with From and To headers
tshark -r capture.pcap -Y 'sip.Method == INVITE' -T fields -e sip.Call-ID -e sip.from.user -e sip.to.user
</syntaxhighlight>
 
=== Filter by IP Address ===
<syntaxhighlight lang="bash">
# SIP traffic from a specific source IP
tshark -r capture.pcap -Y "sip && ip.src == 192.168.1.100"
 
# SIP traffic between two hosts
tshark -r capture.pcap -Y "sip && ip.addr == 192.168.1.100 && ip.addr == 10.0.0.50"
</syntaxhighlight>
 
=== Filter by SIP Response Code ===
<syntaxhighlight lang="bash">
# Show all 200 OK responses
tshark -r capture.pcap -Y "sip.Status-Code == 200"
 
# Show all 4xx and 5xx error responses
tshark -r capture.pcap -Y "sip.Status-Code >= 400"
tshark -r capture.pcap -Y "sip.Status-Code >= 400"
# Show 486 Busy Here responses
tshark -r capture.pcap -Y "sip.Status-Code == 486"
</syntaxhighlight>
=== Important Syntax Notes ===
* '''Field names are case-sensitive:''' Use <code>sip.Method</code>, <code>sip.Call-ID</code>, <code>sip.Status-Code</code> (not <code>sip.method</code> or <code>sip.call-id</code>)
* '''String matching uses <code>contains</code>:''' Use <code>sip contains "text"</code> (not <code>sip.contains()</code>)
* '''Use double quotes for strings:''' <code>sip contains "number"</code> (not single quotes)
* '''Boolean operators:''' Use <code>&&</code> (and), <code>||</code> (or), <code>!</code> (not)
For a complete reference, see the [https://www.wireshark.org/docs/dfref/s/sip.html Wireshark SIP Display Filter Reference].
== Troubleshooting: Database Error 1062 - Lookup Table Auto-Increment Limit ==
If the sniffer logs show a database error `1062 - Duplicate entry '16777215' for key 'PRIMARY'` and new CDRs stop being stored, this is caused by a lookup table reaching its maximum auto-increment limit.
=== Symptoms ===
* CDRs stop being inserted into the database
* Sniffer logs show: `query error in [call __insert_10_0S1();]: 1062 - Duplicate entry '16777215' for key 'PRIMARY'`
* The error affects a lookup table (such as `cdr_sip_response` or `cdr_reason`)
* The value 16777215 (16,777,215) indicates the table is using `MEDIUMINT UNSIGNED` for the ID column
=== Root Cause ===
VoIPmonitor uses lookup tables (like `cdr_sip_response` or `cdr_reason`) to store unique values such as SIP response reason strings or custom response text. These are used to normalize data and reduce storage in the main `cdr` table.
When the system receives many unique SIP response strings or reason messages (e.g., different error messages from various carriers, devices with custom SIP header formats, or PBX-specific responses), the lookup table's auto-increment ID can reach the `MEDIUMINT` limit of 16,777,215. Once this limit is hit, new unique values cannot be inserted, causing all subsequent CDRs to fail with error 1062.
=== Identifying the Affected Table ===
Check which lookup table is hitting the limit:
<syntaxhighlight lang="sql">
-- Check the current AUTO_INCREMENT value for lookup tables
SELECT
    TABLE_NAME,
    COLUMN_TYPE,
    AUTO_INCREMENT
FROM
    INFORMATION_SCHEMA.TABLES
JOIN
    INFORMATION_SCHEMA.COLUMNS
USING (TABLE_SCHEMA, TABLE_NAME)
WHERE
    TABLE_SCHEMA = 'voipmonitor' AND
    (TABLE_NAME LIKE 'cdr_sip%' OR TABLE_NAME LIKE 'cdr_reason%') AND
    COLUMN_KEY = 'PRI' AND
    EXTRA LIKE '%auto_increment%'
ORDER BY AUTO_INCREMENT DESC;
</syntaxhighlight>
</syntaxhighlight>


Look for AUTO_INCREMENT values approaching or exceeding 16,000,000 in tables using `MEDIUMINT`.
=== Interface Statistics ===
 
=== Solution: Prevent New Unique Entries ===
 
The most effective solution is to configure VoIPmonitor to stop storing or normalize the unique SIP response text that is causing the rapid growth of the lookup table.
 
==== Option 1: Disable SIP Response Text Storage ====
 
Edit `/etc/voipmonitor.conf` on the sniffer to disable storing SIP response reason text:
 
<syntaxhighlight lang="ini">
# Disable storing SIP response reason strings in lookup tables
cdr_reason_string_enable = no
</syntaxhighlight>
 
This prevents the system from creating new unique entries for SIP response reason strings. Restart the sniffer:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
# Detailed NIC stats
</syntaxhighlight>
ethtool -S eth0
 
==== Option 2: Normalize Response Text ====
 
If you need to keep some response text but reduce the number of unique entries, enable normalization in `/etc/voipmonitor.conf`:


<syntaxhighlight lang="ini">
# Watch packet rates
# Normalize SIP response text to reduce unique entries
watch -n 1 'cat /proc/net/dev | grep eth0'
cdr_reason_normalisation = yes
cdr_sip_response_normalisation = yes
</syntaxhighlight>
</syntaxhighlight>


Normalization transforms similar response strings into a single canonical form, significantly reducing the number of unique rows created.
== See Also ==
 
==== Option 3: Clean Existing Data (Optional) ====
 
After disabling or normalizing new entries, you may want to clear the lookup table to free space. The data in lookup tables is only used for display purposes and is not critical for historical analysis.


<syntaxhighlight lang="sql">
* [[Sniffer_configuration]] - Configuration parameter reference
-- Clear the cdr_sip_response table (adjust table name as needed)
* [[Sniffer_distributed_architecture]] - Client/server deployment
TRUNCATE TABLE cdr_sip_response;
* [[Capture_rules]] - GUI-based recording rules
</syntaxhighlight>
* [[Sniffing_modes]] - SPAN, ERSPAN, GRE, TZSP setup
* [[Scaling]] - Performance optimization
* [[Database_troubleshooting]] - Database issues
* [[FAQ]] - Common questions and Wireshark display issues


{{Warning|TRUNCATE permanently deletes all data. This will remove the exact SIP response text display in the GUI for historical CDRs, but will not affect the main CDR records or call data. Only do this if you are certain you no longer need the original response text.}}


=== Verification ===


After applying the fix:


1. Check that CDRs are being stored again by monitoring the sniffer logs
2. Verify the lookup table AUTO_INCREMENT is no longer increasing rapidly:
<syntaxhighlight lang="sql">
SELECT AUTO_INCREMENT FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME = 'cdr_sip_response' AND TABLE_SCHEMA = 'voipmonitor';
</syntaxhighlight>
3. Monitor the error logs to confirm the 1062 error has stopped appearing


=== Important Note: NOT a Database Schema Issue ===


This error is typically NOT solved by changing the database schema (e.g., migrating to BIGINT). The root cause is storing too many unique SIP response strings, which will continue to grow regardless of the ID column size. The correct solution is to configure VoIPmonitor to stop creating these unique entries via the `cdr_reason_string_enable` configuration option.


{{Warning|Do NOT confuse this with the unrelated `cdr` table integer overflow problem. The main `cdr` table may encounter limits around 4 billion rows (32-bit INT), which is addressed in the [[Upgrade_to_bigint]] guide. Lookup table issues at 16.7 million (MEDIUMINT) are solved by configuration, not schema migration.}}


=== Routing Loops ===
== AI Summary for RAG ==


Routing loops occur when SIP INVITE requests continuously circulate between SIP servers without completing, causing excessive traffic and call failures. Common symptoms include:
<!-- This section is for AI/RAG systems. Do not edit manually. -->


* High volume of calls to a single destination number in a short time period
=== Summary ===
* Many INVITE requests with no SIP response (response code 0)
Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).
* Very long Post Dial Delay (PDD) values
* Rapid retransmission of INVITE to the same called number
 
{{Note|Routing loops can be caused by misconfigured dial plans, incorrect SIP URI formats, or circular forwarding rules.}}
 
==== Detection Methods ====
 
Use alerts to detect routing loops:
 
* '''SIP Response Alert (Response code 0)''': Configure an alert to detect unreplied INVITE requests. [[Alerts|Configure this in GUI > Alerts]] by setting Response code to 0. This catches calls in a loop that never receive any SIP response.
 
* '''PDD (Post Dial Delay) Alert''': Configure a PDD alert with a threshold (e.g., <code>PDD > 30</code> seconds) to detect calls taking excessively long to complete. Routing loops often have very high PDD values as INVITEs continue retransmitting. [[Alerts|See Alerts documentation for PDD configuration]].
 
* '''Fraud: Sequential Alert''': Monitor for excessive calls to any single destination number within a short time window. Configure [[Anti-fraud|Fraud: Sequential]] with an appropriate Interval and Limit (e.g., 50 calls in 1 hour to the same number). Leave the called number field empty to monitor all destinations.
 
==== Troubleshooting Steps ====
 
1. Identify the looping destination number from alert logs or CDR search
2. Check the SIP dialog to trace the call path (use PCAP analysis in GUI)
3. Verify dial plan configuration on all involved SIP servers
4. Look for forwarding rules or translation patterns that may create circular routing
5. Fix the misconfiguration and verify the loop no longer occurs
 
== See Also ==
* [[Sniffer_configuration]] - Complete configuration reference for voipmonitor.conf
* [[Sniffer_distributed_architecture]] - Client/server deployment and troubleshooting
* [[Capture_rules]] - GUI-based selective recording configuration
* [[Sniffing_modes]] - Traffic forwarding methods (SPAN, ERSPAN, GRE, TZSP)
* [[Scaling]] - Performance tuning and optimization
* [[Upgrade_to_bigint]] - Migrating CDR table to BIGINT (unrelated to lookup table issues)
 
== AI Summary for RAG ==
'''Summary:''' Step-by-step troubleshooting guide for VoIPmonitor sensor not capturing calls. Key diagnostic flow: (1) Check service status with systemctl, (2) Verify traffic reaches interface with tshark -i eth0 -Y "sip || rtp", (3) Check network/SPAN configuration including interface drops and asymmetric mirroring, (4) Verify voipmonitor.conf settings (interface, sipport, filter), (5) Check GUI capture rules for Skip option, (6) Review logs for OOM or SQL queue issues. Common causes: missing rrdtool package, SPAN not configured for both directions, packet truncation from external sources (Kamailio siptrace), interface packet drops (check with ip -s -s l l), asymmetric traffic mirroring across multiple interfaces. Post-reboot: verify firewall rules and NTP time sync (critical for packetbuffer_sender mode).


'''Keywords:''' troubleshooting, no calls, no CDRs, tshark, tcpdump, SPAN, RSPAN, ERSPAN, GRE, TZSP, voipmonitor.conf, interface, sipport, filter, capture rules, OOM, snaplen, packet truncation, Kamailio siptrace, HAProxy tee, interface packet drops, asymmetric mirroring, bidirectional capture, rrdtool, missing package, NTP, time synchronization, packetbuffer_sender, SQL queue, coredump, error 1062, 16777215, cdr_sip_response, rtp_check_both_sides_by_sdp, one-way RTP, promiscuous mode
=== Keywords ===
troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark


'''Key Questions:'''
=== Key Questions ===
* Why is VoIPmonitor not recording any calls?
* Why are no calls being recorded in VoIPmonitor?
* How do I verify SIP/RTP traffic is reaching the sensor? (tshark -i eth0 -Y "sip || rtp" -n)
* How to diagnose PACKETBUFFER FULL or memory is FULL error?
* What are symptoms of interface packet drops? (Missing calls, 000 responses, silent audio - check with ip -s -s l l)
* How to determine if bottleneck is I/O or CPU?
* How do I check for asymmetric traffic mirroring? (Capture on each interface, compare Call-IDs for INVITEs vs responses)
* What do heap values in syslog mean?
* What causes one-way RTP capture? (rtp_check_both_sides_by_sdp set too strict - change to no)
* What does t0CPU percentage indicate?
* How do I fix SPAN configured for one direction only? (Use "both" in monitor session command)
* How to use sniffer_threads manager command?
* What causes error 1062 duplicate entry 16777215? (Lookup table MEDIUMINT limit - set cdr_reason_string_enable=no)
* How to connect to manager socket or port 5029?
* How do I fix Kamailio siptrace truncating packets? (Use TCP transport or HAProxy tee)
* What to do when t0 thread is at 100%?
* What should I verify after server reboot? (Firewall rules and NTP time sync)
* How to fix one-way audio or missing RTP?
* Why is time sync critical for packetbuffer_sender? (Max 2 second difference allowed for call correlation)
* How to configure natalias for NAT?
* How do I fix SQL queue overload causing coredumps? (Increase mysqlstore_max_threads_sip_msg or disable sip-options/notify)
* How to increase max_buffer_mem for high traffic?
* Which package is commonly missing on new sensors? (rrdtool - apt-get install rrdtool)
* How to disable jitterbuffer to save CPU?
* Do I need promiscuous mode for ERSPAN/GRE tunnels? (No - only for SPAN/RSPAN)
* What causes OOM kills of voipmonitor or MySQL?
* How to check disk I/O performance with iostat?
* How to enable promiscuous mode on interface?
* How to fix packet ordering issues with SPAN?
* What is Error 1062 duplicate entry?
* How to verify traffic reaches capture interface?

Latest revision as of 19:08, 22 January 2026

Sniffer Troubleshooting

This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see Sniffer_configuration. For performance tuning, see Scaling.

Critical First Step: Is Traffic Reaching the Interface?

⚠️ Warning: Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help. 

# Check for SIP traffic on the capture interface
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10

# If no packets: Network/SPAN issue - contact network admin
# If packets visible: Proceed with sensor troubleshooting below

Quick Diagnostic Checklist

Check Command Expected Result
Service running systemctl status voipmonitor Active (running)
Traffic on interface tshark -i eth0 -c 5 -Y "sip" SIP packets displayed
Interface errors ip -s link show eth0 No RX errors/drops
Promiscuous mode ip link show eth0 PROMISC flag present
Logs grep voip No critical errors
GUI rules Settings → Capture Rules No unexpected "Skip" rules

No Calls Being Recorded

Service Not Running

# Check status
systemctl status voipmonitor

# View recent logs
journalctl -u voipmonitor --since "10 minutes ago"

# Start/restart
systemctl restart voipmonitor

Common startup failures:

  • Interface not found: Check interface in voipmonitor.conf matches ip a output
  • Port already in use: Another process using the management port
  • License issue: Check License for activation problems

Wrong Interface or Port Configuration

# Check current config
grep -E "^interface|^sipport" /etc/voipmonitor.conf

# Example correct config:
# interface = eth0
# sipport = 5060

💡 Tip:

GUI Capture Rules Blocking

Navigate to Settings → Capture Rules and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.

See Capture_rules for detailed configuration.

SPAN/Mirror Not Configured

If tcpdump shows no traffic:

  1. Verify switch SPAN/mirror port configuration
  2. Check that both directions (ingress + egress) are mirrored
  3. Confirm VLAN tagging is preserved if needed
  4. Test physical connectivity (cable, port status)

See Sniffing_modes for SPAN, RSPAN, and ERSPAN configuration.

Filter Parameter Too Restrictive

If filter is set in voipmonitor.conf, it may exclude traffic: 

# Check filter
grep "^filter" /etc/voipmonitor.conf

# Temporarily disable to test
# Comment out the filter line and restart


Missing id_sensor Parameter

Symptom: SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.

Cause: The id_sensor parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.

Solution: 

# Check if id_sensor is set
grep "^id_sensor" /etc/voipmonitor.conf

# Add or correct the parameter
echo "id_sensor = 1" >> /etc/voipmonitor.conf

# Restart the service
systemctl restart voipmonitor

💡 Tip: Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See id_sensor documentation.

Missing Audio / RTP Issues

One-Way Audio (Asymmetric Mirroring)

Symptom: SIP recorded but only one RTP direction captured.

Cause: SPAN port configured for only one direction.

Diagnosis: 

# Count RTP packets per direction
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c

If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.

RTP Not Associated with Call

Symptom: Audio plays in sniffer but not in GUI, or RTP listed under wrong call.

Possible causes:

1. SIP and RTP on different interfaces/VLANs: 

# voipmonitor.conf - enable automatic RTP association
auto_enable_use_blocks = yes

2. NAT not configured: 

# voipmonitor.conf - for NAT scenarios
natalias = <public_ip> <private_ip>

# If not working, try reversed order:
natalias = <private_ip> <public_ip>

3. External device modifying media ports:

If SDP advertises one port but RTP arrives on different port (SBC/media server issue): 

# Compare SDP ports vs actual RTP
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u

If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.

RTP Incorrectly Associated with Wrong Call (PBX Port Reuse)

Symptom: RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.

Cause: When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.

Solution: Enable rtp_check_both_sides_by_sdp to require verification of both source and destination IP:port against SDP: 

# voipmonitor.conf - require both source and destination to match SDP
rtp_check_both_sides_by_sdp = yes

# Alternative (strict) mode - allows initial unverified packets
rtp_check_both_sides_by_sdp = strict

⚠️ Warning: Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use natalias mappings or the strict setting to mitigate this.

Snaplen Truncation

Symptom: Large SIP messages truncated, incomplete headers.

Solution: 

# voipmonitor.conf - increase packet capture size
snaplen = 8192

For Kamailio siptrace, also check trace_msg_fragment_size in Kamailio config. See snaplen documentation.

PACKETBUFFER Saturation

Symptom: Log shows PACKETBUFFER: memory is FULL, truncated RTP recordings.

⚠️ Warning: This alert refers to VoIPmonitor's internal packet buffer (max_buffer_mem), NOT system RAM. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.

Before testing solutions, gather diagnostic data:

  • Check sensor logs: /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS)
  • Generate debug log via GUI: Tools → Generate debug log

Diagnose: I/O vs CPU Bottleneck

⚠️ Warning: Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to confirm findings, not as the primary diagnostic method.

Step 1: Check IO[] Metrics (v2026.01.3+)

Starting with version 2026.01.3, VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status: 

[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]

Quick interpretation:

Metric Meaning Problem Indicator
C (Capacity) % of disk's sustainable throughput used C ≥ 80% = Warning, C ≥ 95% = Saturated
L (Latency) Current write latency in ms L ≥ 3× B (baseline) = Saturated
U (Utilization) % time disk is busy U > 90% = Disk at limit

If you see DISK_SAT or WARN after IO[]: 

IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT

→ This confirms I/O bottleneck. Skip to I/O Bottleneck Solutions.

For older versions or additional confirmation, continue with the steps below.

ℹ️ Note: See Syslog Status Line - IO[] section for detailed field descriptions.

Step 2: Read the Full Syslog Status Line

VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool: 

# Monitor in real-time
journalctl -u voipmonitor -f
# or
tail -f /var/log/syslog | grep voipmonitor

Example status line: 

calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB

Key metrics for bottleneck identification:

Metric What It Indicates I/O Bottleneck Sign CPU Bottleneck Sign
heap[A|B|C] Buffer fill % (primary / secondary / processing) High A with low t0CPU High A with high t0CPU
t0CPU[X%] Packet capture thread (single-core, cannot parallelize) Low (<50%) High (>80%)
comp[X] Active compression threads Very high (maxed out) Normal
SQLq[C:X M:Y] Pending SQL queries Growing = database bottleneck Stable
tacCPU[...] TAR compression threads All near 100% = compression bottleneck Normal

Interpretation flowchart:

Step 3: Linux I/O Diagnostics

Use these standard Linux tools to confirm I/O bottleneck:

Install required tools: 

# Debian/Ubuntu
apt install sysstat iotop ioping

# CentOS/RHEL
yum install sysstat iotop ioping

2a) iostat - Disk utilization and wait times 

# Run for 10 intervals of 2 seconds
iostat -xz 2 10

Key output columns: 

Device   r/s     w/s   rkB/s   wkB/s  await  %util
sda     12.50  245.30  50.00  1962.40  45.23  98.50
Column Description Problem Indicator
%util Device utilization percentage > 90% = disk saturated
await Average I/O wait time (ms) > 20ms for SSD, > 50ms for HDD = high latency
w/s Writes per second Compare with disk's rated IOPS

2b) iotop - Per-process I/O usage 

# Show I/O by process (run as root)
iotop -o

Look for voipmonitor or mysqld dominating I/O. If voipmonitor shows high DISK WRITE but system %util is 100%, disk cannot keep up.

2c) ioping - Quick latency check 

# Test latency on VoIPmonitor spool directory
cd /var/spool/voipmonitor
ioping -c 20 .

Expected results:

Storage Type Healthy Latency Problem Indicator
NVMe SSD < 0.5 ms > 2 ms
SATA SSD < 1 ms > 5 ms
HDD (7200 RPM) < 10 ms > 30 ms

Step 4: Linux CPU Diagnostics

3a) top - Overall CPU usage 

# Press '1' to show per-core CPU
top

Look for:

  • Individual CPU core at 100% (t0 thread is single-threaded)
  • High %wa (I/O wait) vs high %us/%sy (CPU-bound)

3b) Verify voipmonitor threads 

# Show voipmonitor threads with CPU usage
top -H -p $(pgrep voipmonitor)

If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).

Step 5: Decision Matrix

Observation Likely Cause Go To
heap high, t0CPU > 80%, iostat %util low CPU Bottleneck CPU Solution
heap high, t0CPU < 50%, iostat %util > 90% I/O Bottleneck I/O Solution
heap high, t0CPU < 50%, iostat %util < 50%, SQLq growing Database Bottleneck Database Solution
heap normal, comp maxed, tacCPU all ~100% Compression Bottleneck (type of I/O) I/O Solution

Step 6: Confirmation Test (Optional)

After identifying the likely cause with the tools above, you can confirm with a storage disable test: 

# /etc/voipmonitor.conf - temporarily disable all storage
savesip = no
savertp = no
savertcp = no
savegraph = no

systemctl restart voipmonitor
# Monitor for 5-10 minutes during peak traffic
journalctl -u voipmonitor -f | grep heap
  • If heap values drop to near zero → confirms I/O bottleneck
  • If heap values remain high → confirms CPU bottleneck

⚠️ Warning: Remember to re-enable storage after testing! This test causes call recordings to be lost.

Solution: I/O Bottleneck

ℹ️ Note: If you see IO[...] DISK_SAT or WARN in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See IO[] Metrics for details.

Quick confirmation (for older versions):

Temporarily save only RTP headers to reduce disk write load: 

# /etc/voipmonitor.conf
savertp = header

Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.

Check storage health before upgrading: 

# Check drive health
smartctl -a /dev/sda

# Check for I/O errors in system logs
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"

Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.

Storage controller cache settings:

Storage Type Recommended Cache Mode
HDD / NAS WriteBack (requires battery-backed cache)
SSD WriteThrough (or WriteBack with power loss protection)

Use vendor-specific tools to configure cache policy (megacli, ssacli, perccli).

Storage upgrades (in order of effectiveness):

Solution IOPS Improvement Notes
NVMe SSD 50-100x vs HDD Best option, handles 10,000+ concurrent calls
SATA SSD 20-50x vs HDD Good option, handles 5,000+ concurrent calls
RAID 10 with BBU 5-10x vs single disk Enable WriteBack cache (requires battery backup)
Separate storage server Variable Use client/server mode

Filesystem tuning (ext4): 

# Check current mount options
mount | grep voipmonitor

# Recommended mount options for /var/spool/voipmonitor
# Add to /etc/fstab: noatime,data=writeback,barrier=0
# WARNING: barrier=0 requires battery-backed RAID

Verify improvement: 

# After changes, monitor iostat
iostat -xz 2 10
# %util should drop below 70%, await should decrease

Solution: CPU Bottleneck

Identify CPU Bottleneck Using Manager Commands

VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.

Connect to manager interface: 

# Via Unix socket (local, recommended)
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket

# Via TCP port 5029 (remote or local)
echo 'sniffer_threads' | nc 127.0.0.1 5029

# Monitor continuously (every 2 seconds)
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"

ℹ️ Note: TCP port 5029 is encrypted by default. For unencrypted access, set manager_enable_unencrypted = yes in voipmonitor.conf (security risk on public networks).

Example output: 

t0 - binlog1 fifo pcap read          (  12345) :  78.5  FIFO  99     1234
t2 - binlog1 pb write                (  12346) :  12.3               456
rtp thread binlog1 binlog1 0         (  12347) :   8.1               234
rtp thread binlog1 binlog1 1         (  12348) :   6.2               198
t1 - binlog1 call processing         (  12349) :   4.5               567
tar binlog1 compression 0            (  12350) :   3.2                89

Column interpretation:

Column Description
Thread name Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
(TID) Linux thread ID (useful for top -H -p TID)
CPU % Current CPU usage percentage - key metric
Sched Scheduler type (FIFO = real-time, empty = normal)
Priority Thread priority
CS/s Context switches per second

Critical threads to watch:

Thread Role If at 90-100%
t0 (pcap read) Packet capture from NIC Single-core limit reached! Cannot parallelize. Need DPDK/Napatech.
t2 (pb write) Packet buffer processing Processing bottleneck. Check t2CPU breakdown.
rtp thread RTP packet processing Threads auto-scale. If still saturated, consider DPDK/Napatech.
tar compression PCAP archiving I/O bottleneck (compression waiting for disk)
mysql store Database writes Database bottleneck. Check SQLq metric.

⚠️ Warning: If t0 thread is at 90-100%, you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and cannot be parallelized. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:

  • Reduce captured traffic using interface_ip_filter or BPF filter
  • Use kernel bypass (DPDK or Napatech) which eliminates kernel overhead entirely

Interpreting t2CPU Detailed Breakdown

The syslog status line shows t2CPU with detailed sub-metrics: 

t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
Code Function High Value Indicates
pb Packet buffer output Buffer management overhead
d Dispatch Structure creation bottleneck
s SIP parsing Complex/large SIP messages
e Entity lookup Call table lookup overhead
c Call processing Call state machine processing
g Register processing High REGISTER volume
r, rm, rh, rd RTP processing stages High RTP volume (threads auto-scale)

Thread auto-scaling: VoIPmonitor automatically spawns additional threads when load increases:

  • If d > 50% → SIP parsing thread (s) starts
  • If s > 50% → Entity lookup thread (e) starts
  • If e > 50% → Call/register/RTP threads start

Configuration for High Traffic (>10,000 calls/sec)

# /etc/voipmonitor.conf

# Increase buffer to handle processing spikes (value in MB)
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
# Larger buffer absorbs I/O and CPU spikes without packet loss
max_buffer_mem = 10000

# Use IP filter instead of BPF (more efficient)
interface_ip_filter = 10.0.0.0/8
interface_ip_filter = 192.168.0.0/16
# Comment out any 'filter' parameter

CPU Optimizations

# /etc/voipmonitor.conf

# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
jitterbuffer_f1 = no
jitterbuffer_f2 = yes
jitterbuffer_adapt = no

# If MOS metrics are not needed at all, disable everything:
# jitterbuffer_f1 = no
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no

Kernel Bypass Solutions (Extreme Loads)

When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
Napatech Hardware SmartNIC >97% (< 3% at 10Gbit) Extreme performance requirements

Verify Improvement

# Monitor thread CPU after changes
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"

# Or monitor syslog
journalctl -u voipmonitor -f
# t0CPU should drop, heap values should stay < 20%

ℹ️ Note: After changes, monitor syslog heap[A|B|C] values - should stay below 20% during peak traffic. See Syslog_Status_Line for detailed metric explanations.

Storage Hardware Failure

Symptom: Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.

Diagnosis: 

# Check disk health
smartctl -a /dev/sda

# Check RAID status (if applicable)
cat /proc/mdstat
mdadm --detail /dev/md0

Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.

OOM (Out of Memory)

Identify OOM Victim

# Check for OOM kills
dmesg | grep -i "out of memory\|oom\|killed process"
journalctl --since "1 hour ago" | grep -i oom

MySQL Killed by OOM

Reduce InnoDB buffer pool: 

# /etc/mysql/my.cnf
innodb_buffer_pool_size = 2G  # Reduce from default

Voipmonitor Killed by OOM

Reduce buffer sizes in voipmonitor.conf: 

max_buffer_mem = 2000  # Reduce from default
ringbuffer = 50        # Reduce from default

Runaway External Process

# Find memory-hungry processes
ps aux --sort=-%mem | head -20

# Kill orphaned/runaway process
kill -9 <PID>

For servers limited to 16GB RAM or when experiencing repeated MySQL OOM kills: 

# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G

# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
innodb_flush_log_at_trx_commit = 2

Restart MySQL after changes: 

systemctl restart mysql
# or
systemctl restart mariadb

SQL Queue Growth from Non-Call Data

If sip-register, sip-options, or sip-subscribe are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.

⚠️ Warning: Even with reduced innodb_buffer_pool_size, SQL queue will grow indefinitely without cleanup of non-call data.

Solution: Enable automatic cleanup of old non-call data 

# /etc/voipmonitor.conf
# cleandatabase=2555 automatically deletes partitions older than 7 years
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase = 2555

Restart the sniffer after changes: 

systemctl restart voipmonitor

ℹ️ Note: See Data_Cleaning for detailed configuration options and other cleandatabase_* parameters.

Service Startup Failures

Interface No Longer Exists

After OS upgrade, interface names may change (eth0 → ensXXX): 

# Find current interface names
ip a

# Update all config locations
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/

# Also check GUI: Settings → Sensors → Configuration

Missing Dependencies

# Install common missing package
apt install libpcap0.8  # Debian/Ubuntu
yum install libpcap     # RHEL/CentOS

Network Interface Issues

Promiscuous Mode

Required for SPAN port monitoring: 

# Enable
ip link set eth0 promisc on

# Verify
ip link show eth0 | grep PROMISC

ℹ️ Note: Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.

Interface Drops

# Check for drops
ip -s link show eth0 | grep -i drop

# If drops present, increase ring buffer
ethtool -G eth0 rx 4096

Bonded/EtherChannel Interfaces

Symptom: False packet loss when monitoring bond0 or br0.

Solution: Monitor physical interfaces, not logical: 

# voipmonitor.conf - use physical interfaces
interface = eth0,eth1

Network Offloading Issues

Symptom: Kernel errors like bad gso: type: 1, size: 1448 

# Disable offloading on capture interface
ethtool -K eth0 gso off tso off gro off lro off

Packet Ordering Issues

If SIP messages appear out of sequence:

First: Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See FAQ.

If genuine reordering: Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see Napatech).

ℹ️ Note: For out-of-order packets in client/server mode (multiple sniffers), see Sniffer_distributed_architecture for pcap_queue_dequeu_window_length configuration.

Solutions for SPAN/Mirroring Reordering

If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):

1. Configure switch to preserve packet order: Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.

2. Replace SPAN with TAP or packet broker: Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.

Database Issues

SQL Queue Overload

Symptom: Growing SQLq metric, potential coredumps. 

# voipmonitor.conf - increase threads
mysqlstore_concat_limit_cdr = 1000
cdr_check_exists_callid = 0

Error 1062 - Lookup Table Limit

Symptom: Duplicate entry '16777215' for key 'PRIMARY'

Quick fix: 

# voipmonitor.conf
cdr_reason_string_enable = no

See Database Troubleshooting for complete solution.

Bad Packet Errors

Symptom: bad packet with ether_type 0xFFFF detected on interface

Diagnosis: 

# Run diagnostic (let run 30-60 seconds, then kill)
voipmonitor --check_bad_ether_type=eth0

# Find and kill the diagnostic process
ps ax | grep voipmonitor
kill -9 <PID>

Causes: corrupted packets, driver issues, VLAN tagging problems. Check ethtool -S eth0 for interface errors.

Useful Diagnostic Commands

tshark Filters for SIP

# All SIP INVITEs
tshark -r capture.pcap -Y "sip.Method == INVITE"

# Find specific phone number
tshark -r capture.pcap -Y 'sip contains "5551234567"'

# Get Call-IDs
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID

# SIP errors (4xx, 5xx)
tshark -r capture.pcap -Y "sip.Status-Code >= 400"

Interface Statistics

# Detailed NIC stats
ethtool -S eth0

# Watch packet rates
watch -n 1 'cat /proc/net/dev | grep eth0'

See Also





AI Summary for RAG

Summary

Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).

Keywords

troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark

Key Questions

  • Why are no calls being recorded in VoIPmonitor?
  • How to diagnose PACKETBUFFER FULL or memory is FULL error?
  • How to determine if bottleneck is I/O or CPU?
  • What do heap values in syslog mean?
  • What does t0CPU percentage indicate?
  • How to use sniffer_threads manager command?
  • How to connect to manager socket or port 5029?
  • What to do when t0 thread is at 100%?
  • How to fix one-way audio or missing RTP?
  • How to configure natalias for NAT?
  • How to increase max_buffer_mem for high traffic?
  • How to disable jitterbuffer to save CPU?
  • What causes OOM kills of voipmonitor or MySQL?
  • How to check disk I/O performance with iostat?
  • How to enable promiscuous mode on interface?
  • How to fix packet ordering issues with SPAN?
  • What is Error 1062 duplicate entry?
  • How to verify traffic reaches capture interface?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Sniffer_troubleshooting&oldid=11050"