Sniffer troubleshooting: Difference between revisions

From VoIPmonitor.org
(Add troubleshooting section for random missing calls in distributed NFS setup)
(Patch: replace '=== Solution: I/O Bottleneck =...')
 
(95 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Troubleshooting: No Calls Being Sniffed}}
= Sniffer Troubleshooting =
[[Category:Troubleshooting]]
[[Category:Sensor]]


'''This guide provides a systematic process to diagnose why the VoIPmonitor sensor might not be capturing any calls. Use it to quickly identify and resolve the most common issues.'''
This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see [[Sniffer_configuration]]. For performance tuning, see [[Scaling]].


__TOC__
== Critical First Step: Is Traffic Reaching the Interface? ==


== Quick Diagnostic Flowchart ==
{{Warning|Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help.}}


Use this flowchart to quickly identify where your problem lies:
<syntaxhighlight lang="bash">
# Check for SIP traffic on the capture interface
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10
 
# If no packets: Network/SPAN issue - contact network admin
# If packets visible: Proceed with sensor troubleshooting below
</syntaxhighlight>


<kroki lang="mermaid">
<kroki lang="mermaid">
flowchart TD
graph TD
     A[No Calls in GUI] --> B{Is voipmonitor<br/>service running?}
     A[No Calls Recorded] --> B{Packets on interface?<br/>tcpdump -i eth0 port 5060}
     B -->|No| C[Start service:<br/>systemctl start voipmonitor]
     B -->|No packets| C[Network Issue]
     B -->|Yes| D{Does tshark see<br/>SIP traffic?}
     C --> C1[Check SPAN/mirror config]
     D -->|No| E[Network/Mirror<br/>Configuration Issue]
     C --> C2[Verify VLAN tagging]
     D -->|Yes| F{Check voipmonitor.conf:<br/>interface, sipport, filter}
     C --> C3[Check cable/port]
     F -->|Config OK| G{Check GUI<br/>Capture Rules}
     B -->|Packets visible| D[Sensor Issue]
     G -->|Rules OK| H{Database<br/>connection errors?}
     D --> D1[Check voipmonitor.conf]
     H -->|Yes| I[Fix MySQL connection<br/>in voipmonitor.conf]
     D --> D2[Check GUI Capture Rules]
     H -->|No| J[Check logs for<br/>other errors]
     D --> D3[Check logs for errors]
</kroki>


    E --> E1[Verify SPAN/TAP config]
== Quick Diagnostic Checklist ==
    E --> E2[Check promiscuous mode<br/>for Layer 2 mirrors]


    C --> B
{| class="wikitable"
|-
! Check !! Command !! Expected Result
|-
| Service running || <code>systemctl status voipmonitor</code> || Active (running)
|-
| Traffic on interface || <code>tshark -i eth0 -c 5 -Y "sip"</code> || SIP packets displayed
|-
| Interface errors || <code>ip -s link show eth0</code> || No RX errors/drops
|-
| Promiscuous mode || <code>ip link show eth0</code> || PROMISC flag present
|-
| Logs || <code>tail -100 /var/log/syslog \| grep voip</code> || No critical errors
|-
| GUI rules || Settings → Capture Rules || No unexpected "Skip" rules
|}


    style A fill:#ff6b6b
== No Calls Being Recorded ==
    style C fill:#4ecdc4
    style I fill:#4ecdc4
    style E1 fill:#ffe66d
    style E2 fill:#ffe66d
</kroki>


== Is the VoIPmonitor Service Running Correctly? ==
=== Service Not Running ===
First, confirm the sensor process is active and loaded the correct configuration file.


;1. Check the service status (for modern systemd systems):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">systemctl status voipmonitor</syntaxhighlight>
# Check status
Look for a line that says <code>Active: active (running)</code>. If it is inactive or failed, try restarting it with <code>systemctl restart voipmonitor</code> and check the status again.
systemctl status voipmonitor


;2. Service Fails to Start with "Binary Not Found" After Crash:
# View recent logs
If the VoIPmonitor service fails to start after a crash or watchdog restart with an error message indicating the binary cannot be found (e.g., "No such file or directory" for <code>/usr/local/sbin/voipmonitor</code>), the binary may have been renamed with an underscore suffix during the crash recovery process.
journalctl -u voipmonitor --since "10 minutes ago"


Check for a renamed binary:
# Start/restart
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
# Check if the standard binary path exists
ls -l /usr/local/sbin/voipmonitor
 
# If not found, look for a renamed version with underscore suffix
ls -l /usr/local/sbin/voipmonitor_*
</syntaxhighlight>
</syntaxhighlight>


If you find a renamed binary (e.g., <code>voipmonitor_</code>, <code>voipmonitor_20250104</code>, etc.), rename it back to the standard name:
Common startup failures:
<syntaxhighlight lang="bash">
* '''Interface not found''': Check <code>interface</code> in voipmonitor.conf matches <code>ip a</code> output
mv /usr/local/sbin/voipmonitor_ /usr/local/sbin/voipmonitor
* '''Port already in use''': Another process using the management port
</syntaxhighlight>
* '''License issue''': Check [[License]] for activation problems


Then restart the service:
=== Wrong Interface or Port Configuration ===
<syntaxhighlight lang="bash">
systemctl start voipmonitor
</syntaxhighlight>


Verify the service starts correctly:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl status voipmonitor
# Check current config
</syntaxhighlight>
grep -E "^interface|^sipport" /etc/voipmonitor.conf


;3. Sensor Becomes Unresponsive After GUI Update:
# Example correct config:
If the sensor service fails to start or becomes unresponsive after updating a sensor through the Web GUI, the update process may have left the service in a stuck state. The solution is to forcefully stop the service and restart it using these commands:
# interface = eth0
<syntaxhighlight lang="bash">
# sipport = 5060
# SSH into the sensor host and execute:
killall voipmonitor
systemctl stop voipmonitor
systemctl start voipmonitor
</syntaxhighlight>
</syntaxhighlight>
After running these commands, verify the sensor status in the GUI to confirm it is responding correctly. This sequence ensures: (1) Any zombie or hung processes are terminated with <code>killall</code>, (2) systemd is fully stopped, and (3) a clean start of the service.


;4. Verify the running process:
{{Tip|For multiple SIP ports: <code>sipport = 5060,5061,5080</code>}}
<syntaxhighlight lang="bash">ps aux | grep voipmonitor</syntaxhighlight>
This command will show the running process and the exact command line arguments it was started with. Critically, ensure it is using the correct configuration file, for example: <code>--config-file /etc/voipmonitor.conf</code>. If it is not, there may be an issue with your startup script.


== Is Network Traffic Reaching the Server? ==
=== GUI Capture Rules Blocking ===
If the service is running, verify if the VoIP packets (SIP/RTP) are actually arriving at the server's network interface. The best tool for this is <code>tshark</code> (the command-line version of Wireshark).


;1. Install tshark:
Navigate to '''Settings → Capture Rules''' and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
apt-get update && apt-get install tshark


# For CentOS/RHEL/AlmaLinux
See [[Capture_rules]] for detailed configuration.
yum install wireshark
</syntaxhighlight>
 
;2. Listen for SIP traffic on the correct interface:
Replace <code>eth0</code> with the interface name you have configured in <code>voipmonitor.conf</code>.
<syntaxhighlight lang="bash">
tshark -i eth0 -Y "sip || rtp" -n
</syntaxhighlight>
* '''If you see a continuous stream of SIP and RTP packets''', it means traffic is reaching the server, and the problem is likely in VoIPmonitor's configuration (see [[#Check the VoIPmonitor Configuration|Check the VoIPmonitor Configuration]]).
* '''If you see NO packets''', the problem lies with your network configuration. See [[#Troubleshoot Network and Interface Configuration|Troubleshoot Network and Interface Configuration]].


;3. Advanced: Capture to PCAP File for Definitive Testing
=== SPAN/Mirror Not Configured ===
Live monitoring with tshark is useful for observation, but capturing traffic to a .pcap file during a test call provides definitive evidence for troubleshooting intermittent issues or specific call legs.


'''Method 1: Using tcpdump (Recommended)'''
If <code>tcpdump</code> shows no traffic:
<syntaxhighlight lang="bash">
# Verify switch SPAN/mirror port configuration
# Start capture on the correct interface (replace eth0)
# Check that both directions (ingress + egress) are mirrored
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap port 5060
# Confirm VLAN tagging is preserved if needed
# Test physical connectivity (cable, port status)


# Or capture both SIP and RTP traffic:
See [[Sniffing_modes]] for SPAN, RSPAN, and ERSPAN configuration.
tcpdump -i eth0 -s 0 -w /tmp/test_capture.pcap "(port 5060 or udp)"


# Let it run while you make a test call with the missing call leg
=== Filter Parameter Too Restrictive ===
# Press Ctrl+C to stop the capture


# Analyze the capture file:
If <code>filter</code> is set in voipmonitor.conf, it may exclude traffic:
tshark -r /tmp/test_capture.pcap -Y "sip"
</syntaxhighlight>


'''Method 2: Using tshark to capture to file'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Start capture:
# Check filter
tshark -i eth0 -w /tmp/test_capture.pcap -f "tcp port 5060 or udp"
grep "^filter" /etc/voipmonitor.conf


# Make your test call, then press Ctrl+C to stop
# Temporarily disable to test
# Comment out the filter line and restart
</syntaxhighlight>


# Analyze the capture:
tshark -r /tmp/test_capture.pcap -Y "sip" -V
</syntaxhighlight>


'''Decision Tree for PCAP Analysis:'''


After capturing a test call known to have a missing leg:
==== Missing id_sensor Parameter ====


* '''If SIP packets are missing from the .pcap file:'''
'''Symptom''': SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.
** The problem is with your network mirroring configuration (SPAN/TAP port, AWS Traffic Mirroring, etc.)
** The packets never reached the VoIPmonitor sensor's network interface
** Fix the switch mirroring setup or infrastructure configuration first


* '''If SIP packets ARE present in the .pcap file but missing in the VoIPmonitor GUI:'''
'''Cause''': The <code>id_sensor</code> parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.
** The problem is with VoIPmonitor's configuration or processing
** Packets reached the NIC but were not processed correctly
** Review [[#Check the VoIPmonitor Configuration|VoIPmonitor Configuration]] and [[#Check GUI Capture Rules (Causing Call Stops)|Capture Rules]]


'''Example Test Call Workflow:'''
'''Solution''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# 1. Start capture
# Check if id_sensor is set
tcpdump -i eth0 -s 0 -w /tmp/test.pcap "port 5060 and host 10.0.1.100"
grep "^id_sensor" /etc/voipmonitor.conf


# 2. Make a test call from phone at 10.0.1.100 to 10.0.2.200
# Add or correct the parameter
#    (a call that you know should have recordings but is missing)
echo "id_sensor = 1" >> /etc/voipmonitor.conf


# 3. Stop capture (Ctrl+C)
# Restart the service
systemctl restart voipmonitor
</syntaxhighlight>


# 4. Check for the specific call's Call-ID
{{Tip|Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See [[Sniffer_configuration#id_sensor|id_sensor documentation]].}}
tshark -r /tmp/test.pcap -Y "sip" -T fields -e sip.Call-ID
== Missing Audio / RTP Issues ==


# 5. Verify if packets for both A-leg and B-leg exist
=== One-Way Audio (Asymmetric Mirroring) ===
tshark -r /tmp/test.pcap -Y "sip && ip.addr == 10.0.1.100"


# 6. Compare results with VoIPmonitor GUI
'''Symptom''': SIP recorded but only one RTP direction captured.
#    - If packets found in .pcap: VoIPmonitor software issue
#    - If packets missing from .pcap: Network mirroring issue
</syntaxhighlight>


=== Decision Tree - When SIP Packets Are Present but Not Captured ===
'''Cause''': SPAN port configured for only one direction.


If <code>tshark</code> confirms SIP packets ARE present in the .pcap file but VoIPmonitor GUI shows no calls, follow this systematic troubleshooting approach:
'''Diagnosis''':
<syntaxhighlight lang="bash">
# Count RTP packets per direction
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c
</syntaxhighlight>


* '''Check VoIPmonitor Configuration:''' Verify <code>interface</code>, <code>sipport</code>, and <code>filter</code> settings in <code>/etc/voipmonitor.conf</code>
If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.
* '''Check Capture Rules:''' GUI "Capture Rules" with Skip:ON can silently block traffic
* '''Check for Encapsulation:''' Traffic may be wrapped in ERSPAN, GRE, VXLAN, or VLAN tags (see [[#Check for Network Encapsulation Issues|Check for Encapsulation]] below)
* '''Check for Packet Drops:''' Review GUI → Settings → Sensors for drop counters
* '''Check Database Connection:''' Sensor may be processing calls but unable to write to MySQL


== Check for Network Encapsulation Issues ==
=== RTP Not Associated with Call ===
If <code>tshark</code> confirms traffic is reaching the interface but VoIPmonitor is not capturing it, network encapsulation may be the cause. Common encapsulation protocols like ERSPAN, GRE, VXLAN, or TZSP wrap the original VoIP packets in an additional protocol layer.


=== Symptoms of Encapsulation Issues ===
'''Symptom''': Audio plays in sniffer but not in GUI, or RTP listed under wrong call.
* <code>tcpdump</code> shows SIP/RTP packets arriving at the interface
* Packets appear with unexpected outer headers (tunnel protocol, VLAN tags)
* VoIPmonitor CDRs are empty despite confirmed traffic presence
* Traffic works on the network but is invisible to VoIPmonitor


=== Diagnostic Step 1: Use tshark to Detect Encapsulation ===
'''Possible causes''':
Analyze your capture file to identify the protocol layers:


<syntaxhighlight lang="bash">
'''1. SIP and RTP on different interfaces/VLANs''':
# Show protocol hierarchy (all protocols present in the capture)
<syntaxhighlight lang="ini">
tshark -r /tmp/test_capture.pcap -q -z io,phs
# voipmonitor.conf - enable automatic RTP association
 
auto_enable_use_blocks = yes
# Look for these encapsulation protocols in the output:
# - vlan (802.1Q VLAN tags)
# - gre (Generic Routing Encapsulation)
# - er_span (Cisco ERSPAN)
# - vxlan (Virtual Extensible LAN)
# - tzsp (TaZmen Sniffer Protocol)
</syntaxhighlight>
</syntaxhighlight>


=== Diagnostic Step 2: Visual Inspection with tshark ===
'''2. NAT not configured''':
Check individual packets to see the full protocol stack:
<syntaxhighlight lang="ini">
 
# voipmonitor.conf - for NAT scenarios
<syntaxhighlight lang="bash">
natalias = <public_ip> <private_ip>
# Show the first few packets with protocol details
tshark -r /tmp/test_capture.pcap -V -c 3 | head -50


# Look for nested protocol chains like:
# If not working, try reversed order:
# Ethernet -> VLAN -> IP -> UDP -> SIP
natalias = <private_ip> <public_ip>
# Ethernet -> GRE -> IP -> UDP -> SIP
# Ethernet -> ERSPAN -> IP -> UDP -> SIP
</syntaxhighlight>
</syntaxhighlight>


=== Diagnostic Step 3: Check for VLAN Tags Specifically ===
'''3. External device modifying media ports''':
VLAN tags are the most common form of encapsulation:


If SDP advertises one port but RTP arrives on different port (SBC/media server issue):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Look for VLAN tagging in packet headers
# Compare SDP ports vs actual RTP
tshark -r /tmp/test_capture.pcap -Y "vlan" -T fields -e vlan.id | head -20
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
 
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u
# Show which VLAN IDs are carrying your SIP traffic
tshark -r /tmp/test_capture.pcap -Y "sip" -T fields -e frame.protocols | head -20
</syntaxhighlight>
</syntaxhighlight>


=== Encapsulation Types and VoIPmonitor Handling ===
If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.
=== RTP Incorrectly Associated with Wrong Call (PBX Port Reuse) ===
'''Symptom''': RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.


{| class="wikitable"
'''Cause''': When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.
|-
! Encapsulation Type !! VoIPmonitor Support !! Configuration Notes
|-
| VLAN (802.1Q tags) || '''Automatic''' || Configure mirror port in "trunk" mode to allow tagged traffic. No special VoIPmonitor configuration needed.
|-
| ERSPAN || '''Automatic''' || VoIPmonitor decapsulates ERSPAN packets automatically. No configuration needed.
|-
| GRE || '''Automatic''' || VoIPmonitor decapsulates GRE packets automatically. No configuration needed.
|-
| VXLAN || '''Automatic''' || VoIPmonitor decapsulates VXLAN packets automatically. No configuration needed.
|-
| TZSP || '''Automatic''' || VoIPmonitor decapsulates TZSP packets automatically. No configuration needed.
|}


=== Additional Configuration for Encapsulated Traffic ===
'''Solution''': Enable <code>rtp_check_both_sides_by_sdp</code> to require verification of both source and destination IP:port against SDP:
<syntaxhighlight lang="ini">
# voipmonitor.conf - require both source and destination to match SDP
rtp_check_both_sides_by_sdp = yes


While VoIPmonitor handles most encapsulation automatically, verify these settings:
# Alternative (strict) mode - allows initial unverified packets
rtp_check_both_sides_by_sdp = strict
</syntaxhighlight>


;1. For VLAN-tagged traffic with BPF filters:
{{Warning|Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use <code>natalias</code> mappings or the <code>strict</code> setting to mitigate this.}}
If using a custom <code>filter</code> in <code>voipmonitor.conf</code>, be aware that some BPF expressions may drop VLAN-tagged packets. Test by temporarily disabling the filter:
=== Snaplen Truncation ===
<pre>
# Temporarily comment out the filter line in voipmonitor.conf
# filter = udp port 5060  <-- comment this out


# Restart the sensor
'''Symptom''': Large SIP messages truncated, incomplete headers.
systemctl restart voipmonitor


# If traffic appears, the filter was the problem
'''Solution''':
# Update filter to be VLAN-aware or remove it entirely
<syntaxhighlight lang="ini">
</pre>
# voipmonitor.conf - increase packet capture size
snaplen = 8192
</syntaxhighlight>


;2. Mirror port configuration for VLANs:
For Kamailio siptrace, also check <code>trace_msg_fragment_size</code> in Kamailio config. See [[Sniffer_configuration#snaplen|snaplen documentation]].
Ensure your switch mirror port is configured in trunk mode to allow all VLAN tags:
<pre>
# Check switch configuration (command varies by vendor/switch OS)
# Example (Cisco):
show monitor session 1
# Verify: "Allowed VLANs: 1-4094" or specific VLAN range containing your SIP traffic
</pre>


;3. Verify interface configuration:
== PACKETBUFFER Saturation ==
If traffic uses encapsulation, confirm your <code>voipmonitor.conf</code> <code>interface</code> setting matches the configured mirror port:
<pre>
interface = eth0  # This should match your switch mirror destination port
</pre>


=== Troubleshooting Decision Tree for Encapsulation ===
'''Symptom''': Log shows <code>PACKETBUFFER: memory is FULL</code>, truncated RTP recordings.


If encapsulation is detected in the .pcap file:
{{Warning|This alert refers to VoIPmonitor's '''internal packet buffer''' (<code>max_buffer_mem</code>), '''NOT system RAM'''. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.}}


* '''If VLAN tags are present:'''
'''Before testing solutions''', gather diagnostic data:
** Check if your mirror port is in trunk mode (not access mode)
* Check sensor logs: <code>/var/log/syslog</code> (Debian/Ubuntu) or <code>/var/log/messages</code> (RHEL/CentOS)
** Verify VLAN IDs match expected SIP traffic VLANs
* Generate debug log via GUI: '''Tools → Generate debug log'''
** Check BPF filter - some filters like "udp" drop VLAN-tagged packets


* '''If tunnel protocols (GRE/ERSPAN/VXLAN) are present:'''
=== Diagnose: I/O vs CPU Bottleneck ===
** This is expected - VoIPmonitor should handle these automatically
** Verify the sensor is running on an interface where tunnel traffic is delivered
** Check for packet drops in GUI → Settings → Sensors


* '''If packets show NO encapsulation:'''
{{Warning|Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to '''confirm''' findings, not as the primary diagnostic method.}}
** The problem is not encapsulation-related
** Review other troubleshooting sections: voipmonitor.conf, filter settings, capture rules, database connection


=== When to Contact Support ===
==== Step 1: Check IO[] Metrics (v2026.01.3+) ====
If encapsulation is detected but VoIPmonitor still does not capture the traffic after verifying:


1. Your VoIPmonitor version is up to date
'''Starting with version 2026.01.3''', VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status:
2. Packets are confirmed visible with <code>tshark</code>
3. The <code>interface</code> parameter matches your configuration
4. No capture rules are blocking the traffic
5. Encapsulation is one of the supported types (VLAN, ERSPAN, GRE, VXLAN, TZSP)


Then gather and provide this information to VoIPmonitor support:
<syntaxhighlight lang="text">
* The <code>/etc/voipmonitor.conf</code> file
[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]
* A sample <code>.pcap</code> capture file created with:
</syntaxhighlight>
  <code>tcpdump -i eth0 -s 0 -w traffic.pcap port 5060</code>
* Output of protocol analysis:
  <code>tshark -r traffic.pcap -q -z io,phs</code>
* VoIPmonitor version
* Any relevant errors from sensor logs


For more information on deployment with traffic forwarding, see [[Sniffer_distributed_architecture|Distributed Architecture Guide]].
'''Quick interpretation:'''
 
== Check Sensor Statistics in GUI ==
If <code>tshark</code> confirms traffic is reaching the interface, use the VoIPmonitor GUI to verify the sensor is processing packets without drops.
 
;1. Navigate to '''Settings → Sensors'''
:Expand the sensor details to view real-time capture statistics.
 
;2. Check the '''# packet drops''' counter:
:This counter should ideally be '''0'''. If it shows a value other than zero, the sensor is dropping packets due to processing bottlenecks, insufficient buffer memory, or hardware limitations.
 
;3. Common causes of packet drops:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Symptom !! Likely Cause !! Solution
! Metric !! Meaning !! Problem Indicator
|-
| Drops increase with high traffic | Insufficient buffer memory | Increase <code>ringbuffer</code> or <code>max_buffer_mem</code> in <code>voipmonitor.conf</code><br/>See [[Scaling]] for tuning guidance
|-
|-
| Consistent drops at moderate traffic | CPU bottleneck | Check sensor CPU utilization; consider dedicating cores with <code>cpu_affinity</code>
| '''C''' (Capacity) || % of disk's sustainable throughput used || '''C ≥ 80% = Warning''', '''C ≥ 95% = Saturated'''
|-
|-
| Drops only on specific interfaces | Hardware/driver issue | Verify interface driver; check for errors with <code>ethtool -S eth0</code>
| '''L''' (Latency) || Current write latency in ms || '''L ≥ 3× B''' (baseline) = Saturated
|-
|-
| Drops after configuration change | New filter or feature overload | Remove or simplify BPF filters, disable unnecessary features
| '''U''' (Utilization) || % time disk is busy || '''U > 90%''' = Disk at limit
|}
|}


;4. Other useful sensor statistics:
'''If you see <code>DISK_SAT</code> or <code>WARN</code> after IO[]:'''
{| class="wikitable"
<syntaxhighlight lang="text">
|-
IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT
! Metric !! Description
</syntaxhighlight>
|-
 
| '''Packets/sec''' | Current capture rate
→ This confirms I/O bottleneck. Skip to [[#Solution:_I.2FO_Bottleneck|I/O Bottleneck Solutions]].
|-
| '''Bytes/sec''' | Current bandwidth utilization
|-
| '''Calls/sec''' | Call processing rate
|-
| '''Graph''' | Real-time graph of capture rates over time
|}


For detailed performance metrics beyond basic statistics, see [[Understanding_the_Sniffer's_Performance_Log]].
'''For older versions or additional confirmation''', continue with the steps below.


== Troubleshoot Random Missing Calls in Distributed NFS Setup ==
{{Note|See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|Syslog Status Line - IO[] section]] for detailed field descriptions.}}
If you are experiencing random call recordings missing in a distributed setup with a central instance and remote sensors using NFS for the spool directory, and you see errors like "No such file or directory" for .qoq or other files, follow this diagnostic workflow.


=== Step 1: Collect System Logs ===
==== Step 2: Read the Full Syslog Status Line ====


First, gather the relevant logs to understand what is happening when files are missing.
VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Collect full syslog or VoIPMonitor service logs from the affected sensor
# Monitor in real-time
# For the day when the issue occurred
journalctl -u voipmonitor -f
# or
tail -f /var/log/syslog | grep voipmonitor
</syntaxhighlight>


# For Debian/Ubuntu (systemd journal)
'''Example status line:'''
journalctl -u voipmonitor --since "yesterday" --until "today" > /tmp/voipmonitor_logs_yesterday.txt
<syntaxhighlight lang="text">
 
calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB
# For CentOS/RHEL/AlmaLinux
tail -n 10000 /var/log/messages | grep voipmonitor > /tmp/voipmonitor_recent.txt
 
# Or capture specific time window
journalctl -u voipmonitor --since "2025-01-05 08:00:00" --until "2025-01-05 18:00:00" > /tmp/voipmonitor_issue_window.txt
</syntaxhighlight>
</syntaxhighlight>


Look for error messages like:
'''Key metrics for bottleneck identification:'''
* <code>No such file or directory</code> for .qoq, .pcap, or other files
* NFS timeout or disconnection errors
* File write failure messages
* I/O errors


=== Step 2: Check Packet Drops in GUI ===
{| class="wikitable"
|-
! Metric !! What It Indicates !! I/O Bottleneck Sign !! CPU Bottleneck Sign
|-
| <code>heap[A&#124;B&#124;C]</code> || Buffer fill % (primary / secondary / processing) || High A with low t0CPU || High A with high t0CPU
|-
| <code>t0CPU[X%]</code> || Packet capture thread (single-core, cannot parallelize) || Low (<50%) || High (>80%)
|-
| <code>comp[X]</code> || Active compression threads || Very high (maxed out) || Normal
|-
| <code>SQLq[C:X M:Y]</code> || Pending SQL queries || Growing = database bottleneck || Stable
|-
| <code>tacCPU[...]</code> || TAR compression threads || All near 100% = compression bottleneck || Normal
|}


Verify that the sensor is not experiencing packet drops, which could indicate performance issues that lead to incomplete file writes.
'''Interpretation flowchart:'''


;1. Navigate to '''Settings → Sensors'''
<kroki lang="mermaid">
:Expand the sensor details to view real-time capture statistics.
graph TD
    A[heap values rising] --> B{Check t0CPU}
    B -->|t0CPU > 80%| C[CPU Bottleneck]
    B -->|t0CPU < 50%| D{Check comp and tacCPU}
    D -->|comp maxed, tacCPU high| E[I/O Bottleneck<br/>Disk cannot keep up with writes]
    D -->|comp normal| F{Check SQLq}
    F -->|SQLq growing| G[Database Bottleneck]
    F -->|SQLq stable| H[Mixed/Other Issue]


;2. Check the '''# packet drops''' counter:
    C --> C1[Solution: CPU optimization]
:If this counter is increasing, the sensor may be dropping packets, which can lead to incomplete call recordings that manifest as "missing calls."
    E --> E1[Solution: Faster storage]
    G --> G1[Solution: MySQL tuning]
</kroki>


If packet drops are occurring, see the [[#Check Sensor Statistics in GUI|Sensor Statistics section]] above for causes and solutions.
==== Step 3: Linux I/O Diagnostics ====


=== Step 3: Verify Spool Directory Configuration and Disk Space ===
Use these standard Linux tools to confirm I/O bottleneck:
 
Check if the spool directory is correctly configured and has sufficient space. Issues here can cause file write failures that appear as missing calls.


'''Install required tools:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Verify spool directory configuration
# Debian/Ubuntu
cat /etc/voipmonitor.conf | grep -E "^spooldir|^maxpool"
apt install sysstat iotop ioping


# Check available disk space on the spool directory
# CentOS/RHEL
df -h /var/spool/voipmonitor
yum install sysstat iotop ioping
</syntaxhighlight>


# Check actual disk usage
'''2a) iostat - Disk utilization and wait times'''
du -hs /var/spool/voipmonitor
<syntaxhighlight lang="bash">
# Run for 10 intervals of 2 seconds
iostat -xz 2 10
</syntaxhighlight>


# Check available inodes (can be exhausted even with free space)
'''Key output columns:'''
df -i /var/spool/voipmonitor
<syntaxhighlight lang="text">
Device  r/s    w/s  rkB/s  wkB/s  await  %util
sda    12.50  245.30  50.00  1962.40  45.23  98.50
</syntaxhighlight>
</syntaxhighlight>
;Common issues to check:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Issue !! Symptom !! Solution
! Column !! Description !! Problem Indicator
|-
| Disk full or near capacity | Use% shows 90-100% in df -h | Clean up old data, adjust maxpool settings
|-
|-
| No inodes available | IUse% shows 100% in df -i | Clean up many small files, consider enabling TAR archiving
| <code>%util</code> || Device utilization percentage || '''> 90%''' = disk saturated
|-
|-
| NFS mount disconnected | df shows stale mount, mount shows transport endpoint not connected | Check NFS server connectivity, remount
| <code>await</code> || Average I/O wait time (ms) || '''> 20ms''' for SSD, '''> 50ms''' for HDD = high latency
|-
|-
| Incorrect spooldir path | grep shows wrong path or no output | Edit voipmonitor.conf and restart sensor
| <code>w/s</code> || Writes per second || Compare with disk's rated IOPS
|}
|}


=== Step 4: Check NFS Mount Stability ===
'''2b) iotop - Per-process I/O usage'''
 
If the spool is on NFS, network issues or NFS server problems can cause file operations to fail.
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check if NFS mount is still accessible
# Show I/O by process (run as root)
touch /var/spool/voipmonitor/testfile.$$ && rm /var/spool/voipmonitor/testfile.$$
iotop -o
# If this fails, NFS mount is likely not functioning
 
# Check mount options for the NFS spool
mount | grep nfs
# Look for: soft/intr (can cause failures), hard defaults
# Recommended: hard,nfsvers=3,timeo=600
 
# Test NFS write latency
time dd if=/dev/zero of=/var/spool/voipmonitor/test_write bs=1M count=10
rm /var/spool/voipmonitor/test_write
 
# Check NFS server logs for errors (on NFS server)
tail -f /var/log/syslog | grep -i nfs
</syntaxhighlight>
</syntaxhighlight>


;Recommended NFS mount options for VoIPmonitor spooldir:
Look for <code>voipmonitor</code> or <code>mysqld</code> dominating I/O. If voipmonitor shows high DISK WRITE but system <code>%util</code> is 100%, disk cannot keep up.
 
<pre>
mount -t nfs -o hard,nfsvers=3,timeo=600,retrans=2,rsize=1048576,wsize=1048576 192.168.1.10:/export/voipmonitor /var/spool/voipmonitor
</pre>
 
These options provide:
* <code>hard</code> - Guarantees that write operations complete (do not silently fail)
* <code>nfsvers=3</code> - Use stable NFSv3 instead of v4 (fewer edge cases)
* <code>timeo=600</code> - Increase timeout to 60 seconds (handles slow storage)
* <code>large rsize/wsize</code> - Better throughput for large file operations
 
=== Step 5: Check for I/O Bottlenecks ===
 
Slow I/O to the NFS spool can cause the sensor's buffers to fill up, leading to incomplete files that later appear as "missing calls."


'''2c) ioping - Quick latency check'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check current disk I/O latency
# Test latency on VoIPmonitor spool directory
ioping -c 10 /var/spool/voipmonitor
cd /var/spool/voipmonitor
 
ioping -c 20 .
# Monitor I/O wait during heavy call traffic
iostat -x 2 10
 
# Check for NFS-specific latency
time cat /var/spool/voipmonitor/test_10mb.file > /dev/null
</syntaxhighlight>
</syntaxhighlight>


If I/O latency is high (>50ms average for ioping):
'''Expected results:'''
* Check network bandwidth to NFS server
{| class="wikitable"
* Verify NFS server is not overloaded
|-
* Consider local disk for spooldir instead of NFS
! Storage Type !! Healthy Latency !! Problem Indicator
* See [[IO_Measurement]] for detailed benchmarking
|-
| NVMe SSD || < 0.5 ms || > 2 ms
|-
| SATA SSD || < 1 ms || > 5 ms
|-
| HDD (7200 RPM) || < 10 ms || > 30 ms
|}


=== Step 6: Verify TAR Archiving Configuration ===
==== Step 4: Linux CPU Diagnostics ====
 
TAR archiving reduces the number of small file operations on NFS, which is critical for reducing IOPS and preventing file write failures.


'''3a) top - Overall CPU usage'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check if TAR archiving is enabled
# Press '1' to show per-core CPU
grep -E "^tar\s*=" /etc/voipmonitor.conf
top
 
# Recommended configuration:
# tar = yes
# tar_sip_size = 100 (100M per archive, balanced)
# tar_graph_size = 100
</syntaxhighlight>
</syntaxhighlight>


If TAR archiving is not enabled, strongly consider enabling it in distributed NFS setups to reduce I/O load. See [[Sniffer_configuration#PCAP/TAR_Storage_Strategy]] for details.
Look for:
 
* Individual CPU core at 100% (t0 thread is single-threaded)
=== Step 7: Check cachedir and Cross-Filesystem Issues ===
* High <code>%wa</code> (I/O wait) vs high <code>%us/%sy</code> (CPU-bound)
 
If <code>cachedir</code> points to a different filesystem (e.g., <code>/dev/shm</code> for RAM) while <code>spooldir</code> is on NFS, the cross-filesystem move operation can fail under load.


'''3b) Verify voipmonitor threads'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check cachedir configuration
# Show voipmonitor threads with CPU usage
grep -E "^cachedir\s*=" /etc/voipmonitor.conf
top -H -p $(pgrep voipmonitor)
 
# Check if cachedir and spooldir are on different filesystems
df /var/spool/voipmonitor
df /dev/shm  # if cachedir = /dev/shm
</syntaxhighlight>
</syntaxhighlight>


;If cachedir is on a different filesystem than spooldir:
If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).
* '''Option A (Recommended):''' Disable cachedir for NFS spooldir - set <code>cachedir = </code> (empty)
* '''Option B:** Move cachedir to the same NFS mount point - e.g., <code>cachedir = /var/spool/voipmonitor/cache</code>


This ensures atomic rename operations instead of cross-filesystem copies.
==== Step 5: Decision Matrix ====
 
== Troubleshoot Network and Interface Configuration ==
If <code>tshark</code> shows no traffic, it means the packets are not being delivered to the operating system correctly.
 
;1. Check if the interface is UP:
Ensure the network interface is active.
<syntaxhighlight lang="bash">ip link show eth0</syntaxhighlight>
The output should contain the word <code>UP</code>. If it doesn't, bring it up with:
<syntaxhighlight lang="bash">ip link set dev eth0 up</syntaxhighlight>
 
;2. Check for Promiscuous Mode (for SPAN/RSPAN Mirrored Traffic):
'''Important:''' Promiscuous mode requirements depend on your traffic mirroring method:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Mirroring Method !! Promiscuous Mode Required? !! Reason
! Observation !! Likely Cause !! Go To
|-
|-
| SPAN/RSPAN (Layer 2) || '''Yes''' || Mirrored packets retain original MAC addresses; interface must accept all packets
| <code>heap</code> high, <code>t0CPU</code> > 80%, iostat <code>%util</code> low || '''CPU Bottleneck''' || [[#Solution: CPU Bottleneck|CPU Solution]]
|-
|-
| ERSPAN/GRE/TZSP/VXLAN (Layer 3) || '''No''' || Tunneled traffic is addressed directly to sensor's IP; VoIPmonitor decapsulates automatically
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> > 90% || '''I/O Bottleneck''' || [[#Solution: I/O Bottleneck|I/O Solution]]
|-
| <code>heap</code> high, <code>t0CPU</code> < 50%, iostat <code>%util</code> < 50%, <code>SQLq</code> growing || '''Database Bottleneck''' || [[#SQL Queue Overload|Database Solution]]
|-
| <code>heap</code> normal, <code>comp</code> maxed, <code>tacCPU</code> all ~100% || '''Compression Bottleneck''' (type of I/O) || [[#Solution: I/O Bottleneck|I/O Solution]]
|}
|}


For SPAN/RSPAN deployments, check the current promiscuous mode status:
==== Step 6: Confirmation Test (Optional) ====
<syntaxhighlight lang="bash">ip link show eth0</syntaxhighlight>
Look for the <code>PROMISC</code> flag.


Enable promiscuous mode manually if needed:
After identifying the likely cause with the tools above, you can confirm with a storage disable test:
<syntaxhighlight lang="bash">ip link set eth0 promisc on</syntaxhighlight>
If this solves the problem, you should make the change permanent. The <code>install-script.sh</code> for the sensor usually attempts to do this, but it can fail.


;3. Verify Your SPAN/Mirror/TAP Configuration:
<syntaxhighlight lang="ini">
This is the most common cause of no traffic. Double-check your network switch or hardware tap configuration to ensure:
# /etc/voipmonitor.conf - temporarily disable all storage
* The correct source ports (where your PBX/SBC is connected) are being monitored.
savesip = no
* The correct destination port (where your VoIPmonitor sensor is connected) is configured.
savertp = no
* If you are monitoring traffic across different VLANs, ensure your mirror port is configured to carry all necessary VLAN tags (often called "trunk" mode).
savertcp = no
savegraph = no
</syntaxhighlight>


== Check the VoIPmonitor Configuration ==
<syntaxhighlight lang="bash">
If <code>tshark</code> sees traffic but VoIPmonitor does not, the problem is almost certainly in <code>voipmonitor.conf</code>.
systemctl restart voipmonitor
 
# Monitor for 5-10 minutes during peak traffic
;1. Check the <code>interface</code> directive:
journalctl -u voipmonitor -f | grep heap
:Make sure the <code>interface</code> parameter in <code>/etc/voipmonitor.conf</code> exactly matches the interface where you see traffic with <code>tshark</code>. For example: <code>interface = eth0</code>.
</syntaxhighlight>
 
;2. Check the <code>sipport</code> directive:
:By default, VoIPmonitor only listens on port 5060. If your PBX uses a different port for SIP, you must add it. For example:
:<code>sipport = 5060,5080</code>
 
;3. Check for a restrictive <code>filter</code>:
:If you have a BPF <code>filter</code> configured, ensure it is not accidentally excluding the traffic you want to see. For debugging, try commenting out the <code>filter</code> line entirely and restarting the sensor.
 
== Check GUI Capture Rules (Causing Call Stops) ==
If <code>tshark</code> sees SIP traffic and the sniffer configuration appears correct, but the probe stops processing calls or shows traffic only on the network interface, GUI capture rules may be the culprit.
 
Capture rules configured in the GUI can instruct the sniffer to ignore ("skip") all processing for matched calls. This includes calls matching specific IP addresses or telephone number prefixes.
 
;1. Review existing capture rules:
:Navigate to '''GUI → Capture rules''' and examine all rules for any that might be blocking your traffic.
:Look specifically for rules with the '''Skip''' option set to '''ON''' (displayed as "Skip: ON"). The Skip option instructs the sniffer to completely ignore matching calls (no files, RTP analysis, or CDR creation).
 
;2. Test by temporarily removing all capture rules:
:To isolate the issue, first create a backup of your GUI configuration:
:* Navigate to '''Tools → Backup & Restore → Backup GUI → Configuration tables'''
:* This saves your current settings including capture rules
:* Delete all capture rules from the GUI
:* Click the '''Apply''' button to save changes
:* Reload the sniffer by clicking the green '''"reload sniffer"''' button in the control panel
:* Test if calls are now being processed correctly
:* If resolved, restore the configuration from the backup and systematically investigate the rules to identify the problematic one
 
;3. Identify the problematic rule:
:* After restoring your configuration, remove rules one at a time and reload the sniffer after each removal
:* When calls start being processed again, you have identified the problematic rule
:* Review the rule's match criteria (IP addresses, prefixes, direction) against your actual traffic pattern
:* Adjust the rule's conditions or Skip setting as needed
 
;4. Verify rules are reloaded:
:After making changes to capture rules, remember that changes are '''not automatically applied''' to the running sniffer. You must click the '''"reload sniffer"''' button in the control panel, or the rules will continue using the previous configuration.
 
For more information on capture rules, see [[Capture_rules]].
 
== Troubleshoot Missing or One-Way Audio ==
 
If calls are being captured (you see them in the GUI CDR list), but audio is missing or only present in one direction, this indicates an RTP correlation issue. This is different from "no calls being captured" - the problem is specifically that VoIPmonitor cannot link RTP packets to the SIP call.


'''Symptom:''' CDR shows "No audio" or audio in only one column (Caller/Called), but tshark confirms RTP packets are present on the network.
* If <code>heap</code> values drop to near zero → confirms '''I/O bottleneck'''
* If <code>heap</code> values remain high → confirms '''CPU bottleneck'''


=== Step 1: Verify RTP Capture is Enabled ===
{{Warning|Remember to re-enable storage after testing! This test causes call recordings to be lost.}}


First, ensure the sensor is configured to save RTP packets.
=== Solution: I/O Bottleneck ===


;1. Check <code>savertp</code> in <code>voipmonitor.conf</code>:
{{Note|If you see <code>IO[...] DISK_SAT</code> or <code>WARN</code> in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See [[Syslog_Status_Line#IO.5B....5D_-_Disk_I.2FO_Monitoring_.28v2026.01.3.2B.29|IO[] Metrics]] for details.}}
<syntaxhighlight lang="ini">
# Verify RTP saving is enabled
savertp = yes


# Common mistake: savertp=no or savertp=header (saves only headers, not audio)
'''Quick confirmation (for older versions):'''
# For audio recording, savertp must be 'yes'
</syntaxhighlight>


;2. Check <code>savesip</code> is also enabled:
Temporarily save only RTP headers to reduce disk write load:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# SIP data is required to link RTP to calls
# /etc/voipmonitor.conf
savesip = yes
savertp = header
</syntaxhighlight>
</syntaxhighlight>


;3. Restart the sniffer after changes:
Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.
<syntaxhighlight lang="bash">systemctl restart voipmonitor</syntaxhighlight>
 
=== Step 2: Verify RTP Traffic is Present ===
 
If <code>savertp = yes</code> but you still have no audio, verify RTP packets are reaching the interface and are actually being processed, not just present in tshark.


'''Check storage health before upgrading:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Capture SIP and RTP for a test call
# Check drive health
tcpdump -i eth0 -n -w /tmp/test_audio.pcap "sip or udp"
smartctl -a /dev/sda
 
# Make a test call, then analyze:
tshark -r /tmp/test_audio.pcap -Y "rtp" -c 10


# Check what ports RTP is using
# Check for I/O errors in system logs
tshark -r /tmp/test_audio.pcap -Y "rtp" -T fields -e rtp.ssrc -e udp.srcport -e udp.dstport | head -20
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"
</syntaxhighlight>
</syntaxhighlight>


* '''If you see NO RTP packets in the capture:''' The network mirroring is not configured to capture UDP/RTP traffic (common in VLAN-based deployments).
Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.
* '''If you see RTP packets:''' Proceed to Step 3 for correlation troubleshooting.


=== Step 3: Diagnose RTP Correlation Failure ===
'''Storage controller cache settings:'''
{| class="wikitable"
|-
! Storage Type !! Recommended Cache Mode
|-
| HDD / NAS || WriteBack (requires battery-backed cache)
|-
| SSD || WriteThrough (or WriteBack with power loss protection)
|}


VoIPmonitor uses SDP (Session Description Protocol) information in SIP messages to associate RTP packets with calls. If this correlation fails, audio will not be displayed even though RTP packets are captured.
Use vendor-specific tools to configure cache policy (<code>megacli</code>, <code>ssacli</code>, <code>perccli</code>).


The most common causes are:
'''Storage upgrades (in order of effectiveness):'''
* NAT devices mismatching IP addresses (SDP shows private IP, RTP arrives from public IP)
{| class="wikitable"
* SBCs/media servers modifying RTP ports (SDP advertises port X, RTP arrives on port Y)
|-
* Proxies separating SIP signaling from RTP media path
! Solution !! IOPS Improvement !! Notes
|-
| '''NVMe SSD''' || 50-100x vs HDD || Best option, handles 10,000+ concurrent calls
|-
| '''SATA SSD''' || 20-50x vs HDD || Good option, handles 5,000+ concurrent calls
|-
| '''RAID 10 with BBU''' || 5-10x vs single disk || Enable WriteBack cache (requires battery backup)
|-
| '''Separate storage server''' || Variable || Use [[Sniffer_distributed_architecture|client/server mode]]
|}


=== Step 4: Configure NAT Aliases (natalias) ===
'''Filesystem tuning (ext4):'''
<syntaxhighlight lang="bash">
# Check current mount options
mount | grep voipmonitor


If endpoints are behind NAT, the SDP messages may contain private IPs (e.g., `10.x.x.x`, `192.168.x.x`) but RTP arrives from the firewall's public IP. You must explicitly map these IPs.
# Recommended mount options for /var/spool/voipmonitor
 
# Add to /etc/fstab: noatime,data=writeback,barrier=0
;1. Verify correct <code>natalias</code> syntax:
# WARNING: barrier=0 requires battery-backed RAID
<syntaxhighlight lang="ini">
# Syntax: natalias = <public_ip> <private_ip>
# OR: natalias = <private_ip> <public_ip>
# Use only TWO parameters (IP addresses, not CIDR notation for simple mappings)
 
# Example: Map firewall public IP to internal subnet
natalias = 203.0.113.5 10.0.0.50
</syntaxhighlight>
</syntaxhighlight>


Multiple <code>natalias</code> lines can be used for multiple mappings.
'''Verify improvement:'''
 
<syntaxhighlight lang="bash">
;2. Try reversing IP order if initial configuration does not work:
# After changes, monitor iostat
<syntaxhighlight lang="ini">
iostat -xz 2 10
# If configuration above doesn't fix issue, try the reverse order:
# %util should drop below 70%, await should decrease
natalias = 10.0.0.50 203.0.113.5
</syntaxhighlight>
</syntaxhighlight>


This is a common troubleshooting step - the correct order depends on whether RTP is being received from the public IP (typical) or the private IP (after NAT traversal).
=== Solution: CPU Bottleneck ===


;3. Enable automatic RTP endpoint detection:
==== Identify CPU Bottleneck Using Manager Commands ====
<syntaxhighlight lang="ini">
# Helps identify actual RTP endpoints behind proxies/NAT
rtpip_find_endpoints = yes
</syntaxhighlight>


;4. Relax strict RTP source checking:
VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.
<syntaxhighlight lang="ini">
# Accept RTP even if source IP differs from SIP header IP
rtpfromsdp_onlysip = no
</syntaxhighlight>


;5. Restart the sniffer after changes:
'''Connect to manager interface:'''
<syntaxhighlight lang="bash">systemctl restart voipmonitor</syntaxhighlight>
 
For more configuration options, see [[Sniffer_configuration#NAT_Handling]].
 
=== Step 5: Perform Port Mismatch Analysis ===
 
If <code>natalias</code> fixes do not resolve the issue, an external device (SBC, media server) may be modifying the RTP ports. This is a common scenario in carrier environments.
 
;1. Capture a test call with both SIP and RTP:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Start capture
# Via Unix socket (local, recommended)
tcpdump -i eth0 -n -w /tmp/port_mismatch.pcap "sip or udp"
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket


# Make a test call with missing audio
# Via TCP port 5029 (remote or local)
# Press Ctrl+C to stop
echo 'sniffer_threads' | nc 127.0.0.1 5029
</syntaxhighlight>


;2. Extract SDP-advertised RTP ports from SIP:
# Monitor continuously (every 2 seconds)
<syntaxhighlight lang="bash">
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"
# Find INVITE and extract Call-ID
tshark -r /tmp/port_mismatch.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID -e sip.to.user
 
# For a specific Call-ID, show SDP port information
tshark -r /tmp/port_mismatch.pcap -Y "sip and sip.Call-ID == YOUR_CALL_ID" -T fields -e sdp.media.port | head -20
</syntaxhighlight>
</syntaxhighlight>


This shows what ports the SIP SDP says RTP should use.
{{Note|1=TCP port 5029 is encrypted by default. For unencrypted access, set <code>manager_enable_unencrypted = yes</code> in voipmonitor.conf (security risk on public networks).}}


;3. Extract actual RTP ports from captured packets:
'''Example output:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="text">
# Show all RTP packets and their ports
t0 - binlog1 fifo pcap read          (  12345) :  78.5  FIFO  99    1234
tshark -r /tmp/port_mismatch.pcap -Y "rtp" -T fields -e rtp.ssrc -e udp.srcport -e udp.dstport -e ip.src -e ip.dst | sort -u | head -20
t2 - binlog1 pb write                (  12346) :  12.3              456
rtp thread binlog1 binlog1 0        (  12347) :  8.1              234
rtp thread binlog1 binlog1 1        (  12348) :  6.2              198
t1 - binlog1 call processing        (  12349) :  4.5              567
tar binlog1 compression 0            (  12350) :  3.2                89
</syntaxhighlight>
</syntaxhighlight>


This shows what ports RTP is actually using in the capture.
'''Column interpretation:'''
 
{| class="wikitable"
;4. Compare the results:
|-
 
! Column !! Description
{| class="wikitable"
|-
| Thread name || Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
|-
| (TID) || Linux thread ID (useful for <code>top -H -p TID</code>)
|-
| CPU % || Current CPU usage percentage - '''key metric'''
|-
| Sched || Scheduler type (FIFO = real-time, empty = normal)
|-
|-
! Source !! SDP Port (from SIP) !! Actual RTP Port !! Match?
| Priority || Thread priority
|-
|-
| Example | 50100 | 32456 | '''NO - Port mismatch'''
| CS/s || Context switches per second
|}
|}


* '''If ports MATCH''': The issue is not port modification. Check NAT configuration again or verify <code>savertp</code> is enabled.
'''Critical threads to watch:'''
* '''If ports DO NOT MATCH''': An external device is modifying the media ports.
 
=== Step 6: External Device Root Cause ===
 
If a port mismatch is detected (Step 5), the root cause is an external device (SBC, media server, SIP proxy) modifying the RTP ports.
 
'''Important:''' VoIPmonitor requires the SDP-advertised ports to match the actual RTP packet ports for proper correlation. If these do not match, fixing the issue requires changing the configuration of the external device, not VoIPmonitor.
 
Common external devices that modify RTP ports:
* Session Border Controllers (SBC)
* Media servers
* SIP proxies with media handling
* IP-PBX systems with built-in NAT traversal
 
'''Solutions:'''
1. Check the external device's configuration for media port handling
2. If possible, disable port modification on the external device
3. Use media mirroring features (e.g., Ribbon SBC Monitoring Profile) if available
4. Consider deploying a dedicated sensor on the internal network side where SDP and RTP ports match
 
For example documentation on SBC mirroring, see [[Ribbon7k_monitoring_profiles]].
 
=== Step 7: Distributed Architecture Check ===
 
VoIPmonitor cannot associate SIP and RTP if they are captured by different sensors.
 
;Scenario:*
* Sensor A captures SIP traffic (VLAN 10)
* Sensor B captures RTP traffic (VLAN 20)
* Both sensors send data to the same GUI
 
;Result:* The GUI cannot reconstruct the call because SIP and RTP are not on the same sensor.
 
;Solution:*
* Ensure one single sensor instance sees both SIP signaling and RTP media
* Use a trunk port in the switch mirror configuration
* Or use a combined mode where all traffic goes to one interface
 
For more details, see [[Sniffer_distributed_architecture]].
 
=== Quick Decision Matrix ===
 
{| class="wikitable"
{| class="wikitable"
|-
|-
! Symptom !! Likely Cause !! Primary Solution
! Thread !! Role !! If at 90-100%
|-
|-
| tshark shows NO RTP traffic | Network Mirroring/VLANs | Fix SPAN/TAP or allow VLAN trunk
| '''t0''' (pcap read) || Packet capture from NIC || '''Single-core limit reached!''' Cannot parallelize. Need DPDK/Napatech.
|-
|-
| tshark shows RTP, GUI shows "No audio" | <code>savertp ≠ yes</code> | Set <code>savertp = yes</code>
| '''t2''' (pb write) || Packet buffer processing || Processing bottleneck. Check t2CPU breakdown.
|-
|-
| tshark shows RTP, GUI shows one-way audio | NAT/Proxy correlation failure | Enable <code>rtpip_find_endpoints</code> and <code>natalias</code>
| '''rtp thread''' || RTP packet processing || Threads auto-scale. If still saturated, consider DPDK/Napatech.
|-
|-
| SDP and RTP ports do not match | External device modifying ports | Fix SBC/media server configuration
| '''tar compression''' || PCAP archiving || I/O bottleneck (compression waiting for disk)
|-
|-
| SIP on sensor A, RTP on sensor B | Distributed architecture issue | Capture both on single sensor
| '''mysql store''' || Database writes || Database bottleneck. Check SQLq metric.
|}
|}


== Troubleshoot MySQL/MariaDB Database Connection Errors ==
{{Warning|If '''t0 thread is at 90-100%''', you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and '''cannot be parallelized'''. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:
If you see "Connection refused (111)" errors or the sensor cannot connect to your database server, the issue is with the MySQL/MariaDB database connection configuration in <code>/etc/voipmonitor.conf</code>.
* '''Reduce captured traffic''' using <code>interface_ip_filter</code> or BPF <code>filter</code>
* '''Use kernel bypass''' ([[DPDK]] or [[Napatech]]) which eliminates kernel overhead entirely}}


Error 111 (Connection refused) indicates that the database server is reachable on the network, but no MySQL/MariaDB service is listening on the specified port, or the connection is being blocked by a firewall. This commonly happens after migrations when the database server IP address has changed.
==== Interpreting t2CPU Detailed Breakdown ====


=== Symptoms and Common Errors ===
The syslog status line shows <code>t2CPU</code> with detailed sub-metrics:
<syntaxhighlight lang="text">
t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
</syntaxhighlight>


{| class="wikitable"
{| class="wikitable"
|-
|-
! Error Message !! Likely Cause
! Code !! Function !! High Value Indicates
|-
|-
| <code>Can't connect to MySQL server on 'IP' (111)</code> || Wrong host/port or service not running
| '''pb''' || Packet buffer output || Buffer management overhead
|-
|-
| <code>Access denied for user 'user'@'host'</code> || Wrong username or password
| '''d''' || Dispatch || Structure creation bottleneck
|-
|-
| <code>Unknown database 'voipmonitor'</code> || Wrong database name
| '''s''' || SIP parsing || Complex/large SIP messages
|-
| '''e''' || Entity lookup || Call table lookup overhead
|-
| '''c''' || Call processing || Call state machine processing
|-
| '''g''' || Register processing || High REGISTER volume
|-
| '''r, rm, rh, rd''' || RTP processing stages || High RTP volume (threads auto-scale)
|}
|}


=== Diagnostic Steps ===
'''Thread auto-scaling:''' VoIPmonitor automatically spawns additional threads when load increases:
* If '''d''' > 50% → SIP parsing thread ('''s''') starts
* If '''s''' > 50% → Entity lookup thread ('''e''') starts
* If '''e''' > 50% → Call/register/RTP threads start


;1. Check for database connection errors in sensor logs:
==== Configuration for High Traffic (>10,000 calls/sec) ====
<syntaxhighlight lang="bash">
# For Debian/Ubuntu (systemd journal)
journalctl -u voipmonitor --since "1 hour ago" | grep -iE "mysql|database|connection|can.t connect"


# For systems using traditional syslog
<syntaxhighlight lang="ini">
tail -f /var/log/syslog | grep voipmonitor | grep -iE "mysql|database|connection"
# /etc/voipmonitor.conf


# For CentOS/RHEL/AlmaLinux
# Increase buffer to handle processing spikes (value in MB)
tail -f /var/log/messages | grep voipmonitor | grep -iE "mysql|database|connection"
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
</syntaxhighlight>
# Larger buffer absorbs I/O and CPU spikes without packet loss
max_buffer_mem = 10000


;2. Verify database connection parameters in <code>voipmonitor.conf</code>:
# Use IP filter instead of BPF (more efficient)
<syntaxhighlight lang="ini">
interface_ip_filter = 10.0.0.0/8
# Database Connection Parameters
interface_ip_filter = 192.168.0.0/16
mysqlhost = 192.168.1.10      # IP address or hostname of MySQL/MariaDB server
# Comment out any 'filter' parameter
mysqlport = 3306              # TCP port of the database server (default: 3306)
mysqlusername = root          # Database username
mysqlpassword = your_password  # Database password
mysqldatabase = voipmonitor    # Database name
</syntaxhighlight>
</syntaxhighlight>


;3. Test MySQL connectivity from the sensor host:
==== CPU Optimizations ====
<syntaxhighlight lang="bash">
# Test basic TCP connectivity (replace IP and port as needed)
nc -zv 192.168.1.10 3306


# Or using telnet
<syntaxhighlight lang="ini">
telnet 192.168.1.10 3306
# /etc/voipmonitor.conf
</syntaxhighlight>


If you see "Connection refused", the database service is not running or not listening on that port.
# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
jitterbuffer_f1 = no
jitterbuffer_f2 = yes
jitterbuffer_adapt = no


;4. Test MySQL authentication using credentials from <code>voipmonitor.conf</code>:
# If MOS metrics are not needed at all, disable everything:
<syntaxhighlight lang="bash">
# jitterbuffer_f1 = no
mysql -h 192.168.1.10 -P 3306 -u root -p'your_password' voipmonitor
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no
</syntaxhighlight>
</syntaxhighlight>


Commands to run inside mysql client to verify:
==== Kernel Bypass Solutions (Extreme Loads) ====
<syntaxhighlight lang="sql">
-- Check if connected correctly
SELECT USER(), CURRENT_USER();


-- Check database exists
When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:
SHOW DATABASES LIKE 'voipmonitor';


-- Test write access
{| class="wikitable"
USE voipmonitor;
|-
SHOW TABLES;
! Solution !! Type !! CPU Reduction !! Use Case
EXIT;
|-
</syntaxhighlight>
| '''[[DPDK]]''' || Open-source || ~70% || Multi-gigabit on commodity hardware
|-
| '''[[Napatech]]''' || Hardware SmartNIC || >97% (< 3% at 10Gbit) || Extreme performance requirements
|}


;5. Compare with a working sensor's configuration:
==== Verify Improvement ====
If you have other sensors that successfully connect to the database, compare their configuration files:
<syntaxhighlight lang="bash">
diff <(grep -E "^mysql" /etc/voipmonitor.conf) <(grep -E "^mysql" /path/to/working/sensor/voipmonitor.conf)
</syntaxhighlight>


;6. Check firewall and network connectivity:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test network reachability
# Monitor thread CPU after changes
ping -c 4 192.168.1.10
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"
 
# Check if MySQL port is reachable
nc -zv 192.168.1.10 3306


# Check firewall rules (if using firewalld)
# Or monitor syslog
firewall-cmd --list-ports
journalctl -u voipmonitor -f
 
# t0CPU should drop, heap values should stay < 20%
# Check firewall rules (if using iptables)
iptables -L -n | grep 3306
</syntaxhighlight>
</syntaxhighlight>


;7. Verify MySQL/MariaDB service is running:
{{Note|1=After changes, monitor syslog <code>heap[A&#124;B&#124;C]</code> values - should stay below 20% during peak traffic. See [[Syslog_Status_Line]] for detailed metric explanations.}}
On the database server, check if the service is active:
<syntaxhighlight lang="bash">
# Check MySQL/MariaDB service status
systemctl status mariadb    # or systemctl status mysql


# Restart service if needed
== Storage Hardware Failure ==
systemctl restart mariadb


# Check which port MySQL is listening on
'''Symptom''': Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.
ss -tulpn | grep mysql
</syntaxhighlight>


;8. Apply configuration changes and restart the sensor:
'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Restart the VoIPmonitor service to apply changes
# Check disk health
systemctl restart voipmonitor
smartctl -a /dev/sda


# Alternatively, reload without full restart (if supported in your version)
# Check RAID status (if applicable)
echo 'reload' | nc 127.0.0.1 5029
cat /proc/mdstat
 
mdadm --detail /dev/md0
# Verify the service started successfully
systemctl status voipmonitor
 
# Check logs for database connection confirmation
journalctl -u voipmonitor -n 20
</syntaxhighlight>
</syntaxhighlight>


=== Common Troubleshooting Scenarios ===
Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.
 
{| class="wikitable"
|-
! Scenario !! Symptom !! Solution
|-
| Database server IP changed || "Can't connect to MySQL server on '10.1.1.10' (111)" || Update <code>mysqlhost</code> in <code>voipmonitor.conf</code>
|-
| Wrong credentials || "Access denied for user" || Verify and update <code>mysqlusername</code> and <code>mysqlpassword</code>
|-
| Database service not running || "Connection refused (111)" || Start service: <code>systemctl start mariadb</code>
|-
| Firewall blocking port || <code>nc</code> shows "refused" but MySQL is running || Open port 3306 in firewall
|-
| Localhost vs remote confusion || Works locally but fails from sensor || Use actual IP address instead of <code>localhost</code>
|}


For more detailed information about all <code>mysql*</code> configuration parameters, see [[Sniffer_configuration#Database_Configuration]].
== OOM (Out of Memory) ==


== Check for Storage Hardware Errors (HEAP FULL / packetbuffer Issues) ==
=== Identify OOM Victim ===
If the sensor is crashing with "HEAP FULL" errors or showing "packetbuffer: MEMORY IS FULL" messages, you must distinguish between '''actual storage hardware failures''' (requires disk replacement) and '''performance bottlenecks''' (requires tuning).


;1. Check kernel message buffer for storage errors:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
dmesg -T | grep -iE "ext4-fs error|i/o error|nvram warning|ata.*failed|sda.*error|disk failure|smart error" | tail -50
# Check for OOM kills
dmesg | grep -i "out of memory\|oom\|killed process"
journalctl --since "1 hour ago" | grep -i oom
</syntaxhighlight>
</syntaxhighlight>


Look for these hardware error indicators:
=== MySQL Killed by OOM ===
* <code>ext4-fs error</code> - Filesystem corruption or disk failure
* <code>I/O error</code> or <code>BUG: soft lockup</code> - Disk read/write failures
* <code>NVRAM WARNING: nvram_check: failed</code> - RAID controller battery/capacitor issues
* <code>ata.*: FAILED</code> - Hard drive SMART failure
* <code>Buffer I/O error</code> - Disk unable to complete operations


'''If you see ANY of these errors:'''
Reduce InnoDB buffer pool:
* The storage subsystem is failing and likely needs hardware replacement
<syntaxhighlight lang="ini">
* Do not attempt performance tuning - replace the failed disk/RAID first
# /etc/mysql/my.cnf
* Check SMART status: <code>smartctl -a /dev/sda</code>
innodb_buffer_pool_size = 2G  # Reduce from default
* Check RAID health: <code>cat /proc/mdstat</code> or RAID controller tools
</syntaxhighlight>


;2. If dmesg is clean of errors → Performance Bottleneck:
=== Voipmonitor Killed by OOM ===
If the kernel logs show no storage errors, the issue is a performance bottleneck (disk too slow, network latency, etc.).


'''Check disk I/O performance:'''
Reduce buffer sizes in voipmonitor.conf:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
# Current I/O wait (should be < 10% normally)
max_buffer_mem = 2000  # Reduce from default
iostat -x 5
ringbuffer = 50        # Reduce from default
 
</syntaxhighlight>
# Detailed disk stats
dstat -d


# Real-time disk latency
=== Runaway External Process ===
ioping -c 10 .
</syntaxhighlight>


'''Check NFS latency (if using NFS storage):'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Test NFS read/write latency
# Find memory-hungry processes
time dd if=/dev/zero of=/var/spool/voipmonitor/testfile bs=1M count=100
ps aux --sort=-%mem | head -20
time cat /var/spool/voipmonitor/testfile > /dev/null
rm /var/spool/voipmonitor/testfile


# Check NFS mount options
# Kill orphaned/runaway process
mount | grep nfs
kill -9 <PID>
</syntaxhighlight>
</syntaxhighlight>
For servers limited to '''16GB RAM''' or when experiencing repeated MySQL OOM kills:


'''Common performance solutions:'''
<syntaxhighlight lang="ini">
* Use SSD/NVMe for VoIPmonitor spool directory
# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
* Ensure proper NIC queue settings for high-throughput NFS
[mysqld]
* Check network switch port configuration for NFS
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
* Review [[Scaling]] guide for detailed optimization
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G


See also [[IO_Measurement]] for comprehensive disk benchmarking tools.
# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
innodb_flush_log_at_trx_commit = 2
</syntaxhighlight>


== Check for OOM (Out of Memory) Issues ==
Restart MySQL after changes:
If VoIPmonitor suddenly stops processing CDRs and a service restart temporarily restores functionality, the system may be experiencing OOM (Out of Memory) killer events. The Linux OOM killer terminates processes when available RAM is exhausted, and MySQL (<code>mysqld</code>) is a common target due to its memory-intensive nature.
 
;1. Check for OOM killer events in kernel logs:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
systemctl restart mysql
grep -i "out of memory\|killed process" /var/log/syslog | tail -20
# or
systemctl restart mariadb
</syntaxhighlight>
=== SQL Queue Growth from Non-Call Data ===


# For CentOS/RHEL/AlmaLinux
If <code>sip-register</code>, <code>sip-options</code>, or <code>sip-subscribe</code> are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.
grep -i "out of memory\|killed process" /var/log/messages | tail -20


# Also check dmesg:
{{Warning|1=Even with reduced <code>innodb_buffer_pool_size</code>, SQL queue will grow indefinitely without cleanup of non-call data.}}
dmesg | grep -i "killed process" | tail -10
</syntaxhighlight>


Typical OOM killer messages look like:
'''Solution: Enable automatic cleanup of old non-call data'''
<syntaxhighlight lang="text">
<syntaxhighlight lang="ini">
Out of memory: Kill process 1234 (mysqld) score 123 or sacrifice child
# /etc/voipmonitor.conf
Killed process 1234 (mysqld) total-vm: 12345678kB, anon-rss: 1234567kB
# cleandatabase=2555 automatically deletes partitions older than 7 years
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase = 2555
</syntaxhighlight>
</syntaxhighlight>


;2. Monitor current memory usage:
Restart the sniffer after changes:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Check available memory (look for low 'available' or 'free' values)
systemctl restart voipmonitor
free -h
</syntaxhighlight>


# Check per-process memory usage (sorted by RSS)
{{Note|See [[Data_Cleaning]] for detailed configuration options and other <code>cleandatabase_*</code> parameters.}}
ps aux --sort=-%mem | head -15
== Service Startup Failures ==


# Check MySQL memory usage in bytes
=== Interface No Longer Exists ===
cat /proc/$(pgrep mysqld)/status | grep -E "VmSize|VmRSS"
</syntaxhighlight>


'''Warning signs:'''
After OS upgrade, interface names may change (eth0 → ensXXX):
* Available memory consistently below 500MB during operation
* MySQL consuming most of the available RAM
* Swap usage near 100% (if swap is enabled)
* Frequent process restarts without clear error messages


;3. First Fix: Check and correct innodb_buffer_pool_size:
<syntaxhighlight lang="bash">
Before upgrading hardware, verify that <code>innodb_buffer_pool_size</code> is not set too high. This is a common cause of OOM incidents.
# Find current interface names
ip a


'''Calculate the correct buffer pool size:'''
# Update all config locations
For a server running both VoIPmonitor and MySQL on the same host:
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/
<syntaxhighlight lang="text">
Formula: innodb_buffer_pool_size = (Total RAM - VoIPmonitor memory - OS overhead) / 2


Example for a 32GB server:
# Also check GUI: Settings → Sensors → Configuration
- Total RAM: 32GB
- VoIPmonitor process memory (check with ps aux): ~2GB
- OS + other services overhead: ~2GB
- Available for buffer pool: 28GB
- Recommended innodb_buffer_pool_size = 14G
</syntaxhighlight>
</syntaxhighlight>


'''Edit the MariaDB configuration file:'''
=== Missing Dependencies ===
<syntaxhighlight lang="ini">
# Common locations: /etc/mysql/my.cnf, /etc/mysql/mariadb.conf.d/50-server.cnf


innodb_buffer_pool_size = 14G  # Adjust based on your calculation
</syntaxhighlight>
'''Restart MariaDB to apply:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
systemctl restart mariadb # or systemctl restart mysql
# Install common missing package
apt install libpcap0.8 # Debian/Ubuntu
yum install libpcap    # RHEL/CentOS
</syntaxhighlight>
</syntaxhighlight>


;4. Second Fix: Reduce VoIPmonitor buffer memory usage:
== Network Interface Issues ==
VoIPmonitor allocates significant memory for packet buffers. The total buffer memory is calculated based on:


{| class="wikitable"
=== Promiscuous Mode ===
|-
! Parameter !! Default !! Description
|-
| <code>ringbuffer</code> || 50MB || Ring buffer size per interface (recommended ≥500MB for >100 Mbit traffic)
|-
| <code>max_buffer_mem</code> || 2000MB || Maximum buffer memory limit
|}


'''Total formula:''' Approximate total = (ringbuffer × number of interfaces) + max_buffer_mem
Required for SPAN port monitoring:
<syntaxhighlight lang="bash">
# Enable
ip link set eth0 promisc on


'''To reduce VoIPmonitor memory usage:'''
# Verify
<syntaxhighlight lang="ini">
ip link show eth0 | grep PROMISC
# Edit /etc/voipmonitor.conf
</syntaxhighlight>


# Reduce ringbuffer for each interface (e.g., from 50 to 20)
{{Note|Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.}}
ringbuffer = 20


# Reduce maximum buffer memory (e.g., from 2000 to 1000)
=== Interface Drops ===
max_buffer_mem = 1000


# Alternatively, reduce the number of sniffing interfaces if not all are needed
<syntaxhighlight lang="bash">
interface = eth0,eth1  # Instead of eth0,eth1,eth2,eth3
# Check for drops
</syntaxhighlight>
ip -s link show eth0 | grep -i drop


'''After making changes:'''
# If drops present, increase ring buffer
<syntaxhighlight lang="bash">
ethtool -G eth0 rx 4096
systemctl restart voipmonitor
</syntaxhighlight>
</syntaxhighlight>


'''Important notes:'''
=== Bonded/EtherChannel Interfaces ===
* Reducing <code>ringbuffer</code> may increase packet loss during traffic spikes
* Reducing <code>max_buffer_mem</code> affects how many packets can be buffered before being written to disk
* Monitor packet loss statistics in the GUI after reducing buffers to ensure acceptable performance


;5. Solution: Increase physical memory (if buffer tuning is insufficient):
'''Symptom''': False packet loss when monitoring bond0 or br0.
If correcting both MySQL and VoIPmonitor buffer settings does not resolve the OOM issues, upgrade the server's physical RAM. After upgrading:
* Verify memory improvements with <code>free -h</code>
* Recalculate and adjust <code>innodb_buffer_pool_size</code>
* Re-tune <code>ringbuffer</code> and <code>max_buffer_mem</code>
* Monitor for several days to ensure OOM events stop


== Sensor Upgrade Fails with "Permission denied" from /tmp ==
'''Solution''': Monitor physical interfaces, not logical:
If the sensor upgrade process fails with "Permission denied" errors when executing scripts from the <code>/tmp</code> directory, or the service fails to restart after upgrade, the <code>/tmp</code> partition may be mounted with the <code>noexec</code> flag.
<syntaxhighlight lang="ini">
# voipmonitor.conf - use physical interfaces
interface = eth0,eth1
</syntaxhighlight>


The <code>noexec</code> mount option prevents execution of any script or binary from the <code>/tmp</code> directory for security reasons. However, the VoIPmonitor sensor upgrade process uses <code>/tmp</code> for temporary script execution.
=== Network Offloading Issues ===


;1. Check the mount options for /tmp:
'''Symptom''': Kernel errors like <code>bad gso: type: 1, size: 1448</code>
<syntaxhighlight lang="bash">
mount | grep /tmp
</syntaxhighlight>
Look for the <code>noexec</code> flag in the mount options:
<syntaxhighlight lang="text">
/dev/sda2 on /tmp type ext4 rw,relatime,noexec,nosuid,nodev
</syntaxhighlight>


;2. Remount /tmp without noexec (temporary fix):
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
mount -o remount,exec /tmp
# Disable offloading on capture interface
 
ethtool -K eth0 gso off tso off gro off lro off
# Verify the change:
mount | grep /tmp
</syntaxhighlight>
</syntaxhighlight>
The output should no longer contain <code>noexec</code>.


;3. Make the change permanent (edit /etc/fstab):
== Packet Ordering Issues ==
<syntaxhighlight lang="bash">
nano /etc/fstab
</syntaxhighlight>


Remove the <code>noexec</code> option from the /tmp line:
If SIP messages appear out of sequence:
<syntaxhighlight lang="text">
# Before:
/dev/sda2  /tmp  ext4  rw,relatime,noexec,nosuid,nodev  0 0


# After (remove noexec):
'''First''': Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See [[FAQ]].
/dev/sda2  /tmp  ext4  rw,relatime,nosuid,nodev  0 0
</syntaxhighlight>


If <code>/tmp</code> is a separate partition, remount for changes to take effect:
'''If genuine reordering''': Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see [[Napatech]]).
<syntaxhighlight lang="bash">
{{Note|For out-of-order packets in '''client/server mode''' (multiple sniffers), see [[Sniffer_distributed_architecture]] for <code>pcap_queue_dequeu_window_length</code> configuration.}}
mount -o remount /tmp
</syntaxhighlight>


;4. Re-run the sensor upgrade:
=== Solutions for SPAN/Mirroring Reordering ===
After fixing the mount options, retry the sensor upgrade process.


== "No space left on device" Despite Disks Having Free Space ==
If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):
If system services (like php-fpm, voipmonitor, or commands like <code>screen</code>) fail with a "No space left on device" error even though <code>df -h</code> shows sufficient disk space, the issue is likely with '''temporary filesystems''' (<code>/tmp</code>, <code>/run</code>) filling up, not with main disk storage.


;1. Check usage of temporary filesystems:
1. '''Configure switch to preserve packet order''': Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.
<syntaxhighlight lang="bash">
# Check /tmp usage
df -h /tmp


# Check /run usage
2. '''Replace SPAN with TAP or packet broker''': Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.
df -h /run
== Database Issues ==
</syntaxhighlight>


If <code>/tmp</code> or <code>/run</code> show 100% usage despite main filesystems having free space, these temporary filesystems need to be cleaned.
=== SQL Queue Overload ===


;2. Check what is consuming space:
'''Symptom''': Growing <code>SQLq</code> metric, potential coredumps.
<syntaxhighlight lang="bash">
# Find large files in /tmp
du -sh /tmp/* 2>/dev/null | sort -hr | head -20


# Check journal disk usage
<syntaxhighlight lang="ini">
journalctl --disk-usage
# voipmonitor.conf - increase threads
mysqlstore_concat_limit_cdr = 1000
cdr_check_exists_callid = 0
</syntaxhighlight>
</syntaxhighlight>


;3. Immediate cleanup of journal logs:
=== Error 1062 - Lookup Table Limit ===
System journal logs stored in <code>/run/log/journal/</code> can fill up the <code>/run</code> filesystem.
<syntaxhighlight lang="bash">
# Limit journal to 100MB total size
sudo journalctl --vacuum-size=100M


# Or limit by time (keep only last 2 days)
'''Symptom''': <code>Duplicate entry '16777215' for key 'PRIMARY'</code>
sudo journalctl --vacuum-time=2d
</syntaxhighlight>


;4. Permanent solution - Configure journal rotation:
'''Quick fix''':
Edit <code>/etc/systemd/journald.conf</code>:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
[Journal]
# voipmonitor.conf
SystemMaxUse=100M
cdr_reason_string_enable = no
MaxRetentionSec=1month
</syntaxhighlight>
</syntaxhighlight>


Apply changes:
See [[Database_troubleshooting#Database_Error_1062_-_Lookup_Table_Auto-Increment_Limit|Database Troubleshooting]] for complete solution.
<syntaxhighlight lang="bash">
sudo systemctl restart systemd-journald
</syntaxhighlight>


;5. Quick fix - System reboot:
== Bad Packet Errors ==
The quickest way to free space in <code>/tmp</code> and <code>/run</code> is a system reboot, as these filesystems are cleared on each boot.


== Check VoIPmonitor Logs for General Errors ==
'''Symptom''': <code>bad packet with ether_type 0xFFFF detected on interface</code>
After addressing the specific issues above, check the system logs for other error messages from the sensor process that may reveal additional problems.


'''Diagnosis''':
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# For Debian/Ubuntu
# Run diagnostic (let run 30-60 seconds, then kill)
tail -f /var/log/syslog | grep voipmonitor
voipmonitor --check_bad_ether_type=eth0


# For CentOS/RHEL/AlmaLinux
# Find and kill the diagnostic process
tail -f /var/log/messages | grep voipmonitor
ps ax | grep voipmonitor
kill -9 <PID>
</syntaxhighlight>
</syntaxhighlight>


'''Common errors to look for:'''
Causes: corrupted packets, driver issues, VLAN tagging problems. Check <code>ethtool -S eth0</code> for interface errors.
* <code>"pcap_open_live(eth0) error: eth0: No such device"</code> - Wrong interface name
* <code>"Permission denied"</code> - Sensor not running with sufficient privileges
* Messages about connection issues - See [[#Troubleshoot MySQL/MariaDB Database Connection Errors|database troubleshooting]]
* Messages about dropping packets - See [[Scaling]] guide


== Benign Database Errors When Features Are Disabled ==
== Useful Diagnostic Commands ==
Some VoIPmonitor features may generate harmless database errors when those features are not enabled in your configuration. These errors are '''benign''' and can be safely ignored.


=== Common Benign Error: Missing Tables ===
=== tshark Filters for SIP ===
If you see MySQL errors stating that a table does not exist (e.g., "Table 'voipmonitor.ss7' doesn't exist") even though the corresponding feature is disabled, this is expected behavior.


'''Common examples:'''
<syntaxhighlight lang="bash">
* Errors about the <code>ss7</code> table when <code>ss7 = no</code> in <code>voipmonitor.conf</code>
# All SIP INVITEs
* Errors about the <code>register_failed</code>, <code>register_state</code>, or <code>sip_msg</code> tables when those features are disabled
tshark -r capture.pcap -Y "sip.Method == INVITE"


=== Solution: Ignore or Suppress in Monitoring ===
# Find specific phone number
Since these errors indicate that a feature is simply not active, they do not impact system functionality:
tshark -r capture.pcap -Y 'sip contains "5551234567"'


# '''Do not change the configuration''' to fix these errors
# Get Call-IDs
# '''Add monitoring exceptions''' to suppress warnings for table-not-found errors (MySQL error code 1146)
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID
# Configure alerting systems to exclude these specific SQL errors from notifications


=== When to Take Action ===
# SIP errors (4xx, 5xx)
You only need to take action if:
tshark -r capture.pcap -Y "sip.Status-Code >= 400"
* You actually want to use the feature (enable the corresponding configuration option)
</syntaxhighlight>
* Errors persist about tables for features that '''are''' explicitly enabled in <code>voipmonitor.conf</code>


== Appendix: tshark Display Filter Syntax for SIP ==
=== Interface Statistics ===
When using <code>tshark</code> to analyze SIP traffic, it is important to use the correct Wireshark display filter syntax.


=== Basic SIP Filters ===
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Show all SIP INVITE messages
# Detailed NIC stats
tshark -r capture.pcap -Y "sip.Method == INVITE"
ethtool -S eth0
 
# Show all SIP messages (any method)
tshark -r capture.pcap -Y "sip"


# Show SIP and RTP traffic
# Watch packet rates
tshark -r capture.pcap -Y "sip || rtp"
watch -n 1 'cat /proc/net/dev | grep eth0'
</syntaxhighlight>
</syntaxhighlight>


=== Search for Specific Phone Number or Text ===
== See Also ==
<syntaxhighlight lang="bash">
# Find calls containing a specific phone number (e.g., 5551234567)
tshark -r capture.pcap -Y 'sip contains "5551234567"'


# Find INVITE messages for a specific number
* [[Sniffer_configuration]] - Configuration parameter reference
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"'
* [[Sniffer_distributed_architecture]] - Client/server deployment
</syntaxhighlight>
* [[Capture_rules]] - GUI-based recording rules
* [[Sniffing_modes]] - SPAN, ERSPAN, GRE, TZSP setup
* [[Scaling]] - Performance optimization
* [[Database_troubleshooting]] - Database issues
* [[FAQ]] - Common questions and Wireshark display issues


=== Extract Call-ID from Matching Calls ===
<syntaxhighlight lang="bash">
# Get Call-ID for calls matching a phone number
tshark -r capture.pcap -Y 'sip.Method == INVITE && sip contains "5551234567"' -T fields -e sip.Call-ID


# Get Call-ID along with From and To headers
tshark -r capture.pcap -Y 'sip.Method == INVITE' -T fields -e sip.Call-ID -e sip.from.user -e sip.to.user
</syntaxhighlight>


=== Filter by IP Address ===
<syntaxhighlight lang="bash">
# SIP traffic from a specific source IP
tshark -r capture.pcap -Y "sip && ip.src == 192.168.1.100"


# SIP traffic between two hosts
tshark -r capture.pcap -Y "sip && ip.addr == 192.168.1.100 && ip.addr == 10.0.0.50"
</syntaxhighlight>


=== Filter by SIP Response Code ===
<syntaxhighlight lang="bash">
# Show all 200 OK responses
tshark -r capture.pcap -Y "sip.Status-Code == 200"


# Show all 4xx and 5xx error responses
tshark -r capture.pcap -Y "sip.Status-Code >= 400"


# Show 486 Busy Here responses
tshark -r capture.pcap -Y "sip.Status-Code == 486"
</syntaxhighlight>


=== Important Syntax Notes ===
== AI Summary for RAG ==
{| class="wikitable"
|-
! Syntax Element !! Correct Usage !! Notes
|-
| Field names || <code>sip.Method</code>, <code>sip.Call-ID</code> || Case-sensitive
|-
| String matching || <code>sip contains "text"</code> || Use <code>contains</code> keyword
|-
| String quotes || Double quotes <code>"..."</code> || Not single quotes
|-
| Boolean operators || <code>&&</code>, <code>||</code>, <code>!</code> || AND, OR, NOT
|}


For a complete reference, see the [https://www.wireshark.org/docs/dfref/s/sip.html Wireshark SIP Display Filter Reference].
<!-- This section is for AI/RAG systems. Do not edit manually. -->


== AI Summary for RAG ==
=== Summary ===
'''Summary:''' Systematic troubleshooting guide for VoIPmonitor sensor issues. Part 1 covers "no calls being captured": service startup problems (binary renamed after crash, unresponsive after GUI update), network traffic verification using tshark (pcap capture and tshark analysis), decision tree when SIP packets present but not captured, GUI sensor statistics (packet drops counter), network encapsulation troubleshooting (detecting VLAN, ERSPAN, GRE, VXLAN, TZSP with tshark protocol analysis -Z io,phs, visual inspection -V, VLAN tag detection), promiscuous mode requirements (needed for SPAN/RSPAN but not for ERSPAN/GRE/TZSP - VoIPmonitor decapsulates automatically), voipmonitor.conf configuration checks (interface, sipport, filter), GUI capture rules with Skip option, database connection errors (Error 111 after migration), HEAP FULL errors (hardware vs performance issues), OOM killer problems (innodb_buffer_pool_size and ringbuffer/max_buffer_mem tuning), upgrade failures due to /tmp noexec flag, and "no space left" errors caused by full /tmp or /run filesystems. Part 2 covers "missing or one-way audio": verifying savertp=yes, RTP traffic presence, NAT alias configuration (natalias with correct two-parameter syntax, try reversing IP order if needed), rtpip_find_endpoints, rtpfromsdp_onlysip, SDP vs actual RTP port mismatch analysis using tshark/tcpdump, external device (SBC/media server) port modification issues, and distributed architecture constraints (SIP and RTP must be captured by same sensor).
Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).


'''Keywords:''' troubleshooting, no calls, tshark, promiscuous mode, SPAN, ERSPAN, GRE, TZSP, VXLAN, VLAN, encapsulation, vlan tags, io,phs protocol analysis, voipmonitor.conf, interface, sipport, capture rules, Skip, packet drops, sensor statistics, Settings → Sensors, ringbuffer, max_buffer_mem, OOM killer, innodb_buffer_pool_size, HEAP FULL, Connection refused 111, noexec, /tmp, journal logs, no space left on device, missing audio, one-way audio, no audio, RTP correlation, natalias, rtpip_find_endpoints, rtpfromsdp_onlysip, savertp, SDP port mismatch, SBC, media server, distributed architecture
=== Keywords ===
troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark


'''Key Questions:'''
=== Key Questions ===
* Why is VoIPmonitor not recording any calls?
* Why are no calls being recorded in VoIPmonitor?
* How do I check if VoIP traffic is reaching my sensor?
* How to diagnose PACKETBUFFER FULL or memory is FULL error?
* How do I capture traffic to a .pcap file for analysis?
* How to determine if bottleneck is I/O or CPU?
* How do I use tshark to detect network encapsulation?
* What do heap values in syslog mean?
* How do I check if my traffic is encapsulated in ERSPAN, GRE, VXLAN, or VLAN?
* What does t0CPU percentage indicate?
* How do I detect VLAN tags in my capture file?
* How to use sniffer_threads manager command?
* Does VoIPmonitor handle ERSPAN, GRE, VXLAN, and TZSP automatically?
* How to connect to manager socket or port 5029?
* Do I need promiscuous mode for ERSPAN or GRE tunnels?
* What to do when t0 thread is at 100%?
* Why does tcpdump show traffic but VoIPmonitor doesn't capture it?
* How to fix one-way audio or missing RTP?
* How do I diagnose packet drops in VoIPmonitor?
* How to configure natalias for NAT?
* How do I check for packet drops in the GUI sensor statistics?
* How to increase max_buffer_mem for high traffic?
* What is the acceptable value for # packet drops in Settings → Sensors?
* How to disable jitterbuffer to save CPU?
* How do I fix "Connection refused (111)" database errors?
* What causes OOM kills of voipmonitor or MySQL?
* VoIPmonitor crashes with HEAP FULL error, what should I check?
* How to check disk I/O performance with iostat?
* How do I fix OOM killer issues on VoIPmonitor server?
* How to enable promiscuous mode on interface?
* Why does sensor upgrade fail with permission denied from /tmp?
* How to fix packet ordering issues with SPAN?
* "No space left on device" but disk has free space, what to check?
* What is Error 1062 duplicate entry?
* Why is audio missing or one-way in the GUI CDR view?
* How to verify traffic reaches capture interface?
* How do I configure natalias for NAT scenarios?
* How do I diagnose SDP vs RTP port mismatches?
* What should I do if external SBC is modifying RTP ports?

Latest revision as of 19:08, 22 January 2026

Sniffer Troubleshooting

This page covers common VoIPmonitor sniffer/sensor problems organized by symptom. For configuration reference, see Sniffer_configuration. For performance tuning, see Scaling.

Critical First Step: Is Traffic Reaching the Interface?

⚠️ Warning: Before any sensor tuning, verify packets are reaching the network interface. If packets aren't there, no amount of sensor configuration will help. 

# Check for SIP traffic on the capture interface
tcpdump -i eth0 -nn "host <PROBLEMATIC_IP> and port 5060" -c 10

# If no packets: Network/SPAN issue - contact network admin
# If packets visible: Proceed with sensor troubleshooting below

Quick Diagnostic Checklist

Check Command Expected Result
Service running systemctl status voipmonitor Active (running)
Traffic on interface tshark -i eth0 -c 5 -Y "sip" SIP packets displayed
Interface errors ip -s link show eth0 No RX errors/drops
Promiscuous mode ip link show eth0 PROMISC flag present
Logs grep voip No critical errors
GUI rules Settings → Capture Rules No unexpected "Skip" rules

No Calls Being Recorded

Service Not Running

# Check status
systemctl status voipmonitor

# View recent logs
journalctl -u voipmonitor --since "10 minutes ago"

# Start/restart
systemctl restart voipmonitor

Common startup failures:

  • Interface not found: Check interface in voipmonitor.conf matches ip a output
  • Port already in use: Another process using the management port
  • License issue: Check License for activation problems

Wrong Interface or Port Configuration

# Check current config
grep -E "^interface|^sipport" /etc/voipmonitor.conf

# Example correct config:
# interface = eth0
# sipport = 5060

💡 Tip:

GUI Capture Rules Blocking

Navigate to Settings → Capture Rules and check for rules with action "Skip" that may be blocking calls. Rules are processed in order - a Skip rule early in the list will block matching calls.

See Capture_rules for detailed configuration.

SPAN/Mirror Not Configured

If tcpdump shows no traffic:

  1. Verify switch SPAN/mirror port configuration
  2. Check that both directions (ingress + egress) are mirrored
  3. Confirm VLAN tagging is preserved if needed
  4. Test physical connectivity (cable, port status)

See Sniffing_modes for SPAN, RSPAN, and ERSPAN configuration.

Filter Parameter Too Restrictive

If filter is set in voipmonitor.conf, it may exclude traffic: 

# Check filter
grep "^filter" /etc/voipmonitor.conf

# Temporarily disable to test
# Comment out the filter line and restart


Missing id_sensor Parameter

Symptom: SIP packets visible in Capture/PCAP section but missing from CDR, SIP messages, and Call flow.

Cause: The id_sensor parameter is not configured or is missing. This parameter is required to associate captured packets with the CDR database.

Solution: 

# Check if id_sensor is set
grep "^id_sensor" /etc/voipmonitor.conf

# Add or correct the parameter
echo "id_sensor = 1" >> /etc/voipmonitor.conf

# Restart the service
systemctl restart voipmonitor

💡 Tip: Use a unique numeric identifier (1-65535) for each sensor. Essential for multi-sensor deployments. See id_sensor documentation.

Missing Audio / RTP Issues

One-Way Audio (Asymmetric Mirroring)

Symptom: SIP recorded but only one RTP direction captured.

Cause: SPAN port configured for only one direction.

Diagnosis: 

# Count RTP packets per direction
tshark -i eth0 -Y "rtp" -T fields -e ip.src -e ip.dst | sort | uniq -c

If one direction shows 0 or very few packets, configure the switch to mirror both ingress and egress traffic.

RTP Not Associated with Call

Symptom: Audio plays in sniffer but not in GUI, or RTP listed under wrong call.

Possible causes:

1. SIP and RTP on different interfaces/VLANs: 

# voipmonitor.conf - enable automatic RTP association
auto_enable_use_blocks = yes

2. NAT not configured: 

# voipmonitor.conf - for NAT scenarios
natalias = <public_ip> <private_ip>

# If not working, try reversed order:
natalias = <private_ip> <public_ip>

3. External device modifying media ports:

If SDP advertises one port but RTP arrives on different port (SBC/media server issue): 

# Compare SDP ports vs actual RTP
tshark -r call.pcap -Y "sip.Method == INVITE" -V | grep "m=audio"
tshark -r call.pcap -Y "rtp" -T fields -e udp.dstport | sort -u

If ports don't match, the external device must be configured to preserve SDP ports - VoIPmonitor cannot compensate.

RTP Incorrectly Associated with Wrong Call (PBX Port Reuse)

Symptom: RTP streams from one call appear associated with a different CDR when your PBX aggressively reuses the same IP:port across multiple calls.

Cause: When PBX reuses media ports, VoIPmonitor may incorrectly correlate RTP packets to the wrong call based on weaker correlation methods.

Solution: Enable rtp_check_both_sides_by_sdp to require verification of both source and destination IP:port against SDP: 

# voipmonitor.conf - require both source and destination to match SDP
rtp_check_both_sides_by_sdp = yes

# Alternative (strict) mode - allows initial unverified packets
rtp_check_both_sides_by_sdp = strict

⚠️ Warning: Enabling this may prevent RTP association for calls using NAT, as the source IP:port will not match the SDP. Use natalias mappings or the strict setting to mitigate this.

Snaplen Truncation

Symptom: Large SIP messages truncated, incomplete headers.

Solution: 

# voipmonitor.conf - increase packet capture size
snaplen = 8192

For Kamailio siptrace, also check trace_msg_fragment_size in Kamailio config. See snaplen documentation.

PACKETBUFFER Saturation

Symptom: Log shows PACKETBUFFER: memory is FULL, truncated RTP recordings.

⚠️ Warning: This alert refers to VoIPmonitor's internal packet buffer (max_buffer_mem), NOT system RAM. High system memory availability does not prevent this error. The root cause is always a downstream bottleneck (disk I/O or CPU) preventing packets from being processed fast enough.

Before testing solutions, gather diagnostic data:

  • Check sensor logs: /var/log/syslog (Debian/Ubuntu) or /var/log/messages (RHEL/CentOS)
  • Generate debug log via GUI: Tools → Generate debug log

Diagnose: I/O vs CPU Bottleneck

⚠️ Warning: Do not guess the bottleneck source. Use proper diagnostics first to identify whether the issue is disk I/O, CPU, or database-related. Disabling storage as a test is valid but should be used to confirm findings, not as the primary diagnostic method.

Step 1: Check IO[] Metrics (v2026.01.3+)

Starting with version 2026.01.3, VoIPmonitor includes built-in disk I/O monitoring that directly shows disk saturation status: 

[283.4/283.4Mb/s] IO[B1.1|L0.7|U45|C75|W125|R10|WI1.2k|RI0.5k]

Quick interpretation:

Metric Meaning Problem Indicator
C (Capacity) % of disk's sustainable throughput used C ≥ 80% = Warning, C ≥ 95% = Saturated
L (Latency) Current write latency in ms L ≥ 3× B (baseline) = Saturated
U (Utilization) % time disk is busy U > 90% = Disk at limit

If you see DISK_SAT or WARN after IO[]: 

IO[B1.1|L8.5|U98|C97|W890|R5|WI12.5k|RI0.1k] DISK_SAT

→ This confirms I/O bottleneck. Skip to I/O Bottleneck Solutions.

For older versions or additional confirmation, continue with the steps below.

ℹ️ Note: See Syslog Status Line - IO[] section for detailed field descriptions.

Step 2: Read the Full Syslog Status Line

VoIPmonitor outputs a status line every 10 seconds. This is your first diagnostic tool: 

# Monitor in real-time
journalctl -u voipmonitor -f
# or
tail -f /var/log/syslog | grep voipmonitor

Example status line: 

calls[424] PS[C:4 S:41 R:13540] SQLq[C:0 M:0] heap[45|30|20] comp[48] [25.6Mb/s] t0CPU[85%] t1CPU[12%] t2CPU[8%] tacCPU[8|8|7|7%] RSS/VSZ[365|1640]MB

Key metrics for bottleneck identification:

Metric What It Indicates I/O Bottleneck Sign CPU Bottleneck Sign
heap[A|B|C] Buffer fill % (primary / secondary / processing) High A with low t0CPU High A with high t0CPU
t0CPU[X%] Packet capture thread (single-core, cannot parallelize) Low (<50%) High (>80%)
comp[X] Active compression threads Very high (maxed out) Normal
SQLq[C:X M:Y] Pending SQL queries Growing = database bottleneck Stable
tacCPU[...] TAR compression threads All near 100% = compression bottleneck Normal

Interpretation flowchart:

Step 3: Linux I/O Diagnostics

Use these standard Linux tools to confirm I/O bottleneck:

Install required tools: 

# Debian/Ubuntu
apt install sysstat iotop ioping

# CentOS/RHEL
yum install sysstat iotop ioping

2a) iostat - Disk utilization and wait times 

# Run for 10 intervals of 2 seconds
iostat -xz 2 10

Key output columns: 

Device   r/s     w/s   rkB/s   wkB/s  await  %util
sda     12.50  245.30  50.00  1962.40  45.23  98.50
Column Description Problem Indicator
%util Device utilization percentage > 90% = disk saturated
await Average I/O wait time (ms) > 20ms for SSD, > 50ms for HDD = high latency
w/s Writes per second Compare with disk's rated IOPS

2b) iotop - Per-process I/O usage 

# Show I/O by process (run as root)
iotop -o

Look for voipmonitor or mysqld dominating I/O. If voipmonitor shows high DISK WRITE but system %util is 100%, disk cannot keep up.

2c) ioping - Quick latency check 

# Test latency on VoIPmonitor spool directory
cd /var/spool/voipmonitor
ioping -c 20 .

Expected results:

Storage Type Healthy Latency Problem Indicator
NVMe SSD < 0.5 ms > 2 ms
SATA SSD < 1 ms > 5 ms
HDD (7200 RPM) < 10 ms > 30 ms

Step 4: Linux CPU Diagnostics

3a) top - Overall CPU usage 

# Press '1' to show per-core CPU
top

Look for:

  • Individual CPU core at 100% (t0 thread is single-threaded)
  • High %wa (I/O wait) vs high %us/%sy (CPU-bound)

3b) Verify voipmonitor threads 

# Show voipmonitor threads with CPU usage
top -H -p $(pgrep voipmonitor)

If one thread shows ~100% CPU while others are low, you have a CPU bottleneck on the capture thread (t0).

Step 5: Decision Matrix

Observation Likely Cause Go To
heap high, t0CPU > 80%, iostat %util low CPU Bottleneck CPU Solution
heap high, t0CPU < 50%, iostat %util > 90% I/O Bottleneck I/O Solution
heap high, t0CPU < 50%, iostat %util < 50%, SQLq growing Database Bottleneck Database Solution
heap normal, comp maxed, tacCPU all ~100% Compression Bottleneck (type of I/O) I/O Solution

Step 6: Confirmation Test (Optional)

After identifying the likely cause with the tools above, you can confirm with a storage disable test: 

# /etc/voipmonitor.conf - temporarily disable all storage
savesip = no
savertp = no
savertcp = no
savegraph = no

systemctl restart voipmonitor
# Monitor for 5-10 minutes during peak traffic
journalctl -u voipmonitor -f | grep heap
  • If heap values drop to near zero → confirms I/O bottleneck
  • If heap values remain high → confirms CPU bottleneck

⚠️ Warning: Remember to re-enable storage after testing! This test causes call recordings to be lost.

Solution: I/O Bottleneck

ℹ️ Note: If you see IO[...] DISK_SAT or WARN in the syslog status line (v2026.01.3+), disk saturation is already confirmed. See IO[] Metrics for details.

Quick confirmation (for older versions):

Temporarily save only RTP headers to reduce disk write load: 

# /etc/voipmonitor.conf
savertp = header

Restart the sniffer and monitor. If heap usage stabilizes and "MEMORY IS FULL" errors stop, the issue is confirmed to be storage I/O.

Check storage health before upgrading: 

# Check drive health
smartctl -a /dev/sda

# Check for I/O errors in system logs
dmesg | grep -i "i/o error\|sd.*error\|ata.*error"

Look for reallocated sectors, pending sectors, or I/O errors. Replace failing drives before considering upgrades.

Storage controller cache settings:

Storage Type Recommended Cache Mode
HDD / NAS WriteBack (requires battery-backed cache)
SSD WriteThrough (or WriteBack with power loss protection)

Use vendor-specific tools to configure cache policy (megacli, ssacli, perccli).

Storage upgrades (in order of effectiveness):

Solution IOPS Improvement Notes
NVMe SSD 50-100x vs HDD Best option, handles 10,000+ concurrent calls
SATA SSD 20-50x vs HDD Good option, handles 5,000+ concurrent calls
RAID 10 with BBU 5-10x vs single disk Enable WriteBack cache (requires battery backup)
Separate storage server Variable Use client/server mode

Filesystem tuning (ext4): 

# Check current mount options
mount | grep voipmonitor

# Recommended mount options for /var/spool/voipmonitor
# Add to /etc/fstab: noatime,data=writeback,barrier=0
# WARNING: barrier=0 requires battery-backed RAID

Verify improvement: 

# After changes, monitor iostat
iostat -xz 2 10
# %util should drop below 70%, await should decrease

Solution: CPU Bottleneck

Identify CPU Bottleneck Using Manager Commands

VoIPmonitor provides manager commands to monitor thread CPU usage in real-time. This is essential for identifying which thread is saturated.

Connect to manager interface: 

# Via Unix socket (local, recommended)
echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket

# Via TCP port 5029 (remote or local)
echo 'sniffer_threads' | nc 127.0.0.1 5029

# Monitor continuously (every 2 seconds)
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket"

ℹ️ Note: TCP port 5029 is encrypted by default. For unencrypted access, set manager_enable_unencrypted = yes in voipmonitor.conf (security risk on public networks).

Example output: 

t0 - binlog1 fifo pcap read          (  12345) :  78.5  FIFO  99     1234
t2 - binlog1 pb write                (  12346) :  12.3               456
rtp thread binlog1 binlog1 0         (  12347) :   8.1               234
rtp thread binlog1 binlog1 1         (  12348) :   6.2               198
t1 - binlog1 call processing         (  12349) :   4.5               567
tar binlog1 compression 0            (  12350) :   3.2                89

Column interpretation:

Column Description
Thread name Descriptive name (t0=capture, t1=call processing, t2=packet buffer write)
(TID) Linux thread ID (useful for top -H -p TID)
CPU % Current CPU usage percentage - key metric
Sched Scheduler type (FIFO = real-time, empty = normal)
Priority Thread priority
CS/s Context switches per second

Critical threads to watch:

Thread Role If at 90-100%
t0 (pcap read) Packet capture from NIC Single-core limit reached! Cannot parallelize. Need DPDK/Napatech.
t2 (pb write) Packet buffer processing Processing bottleneck. Check t2CPU breakdown.
rtp thread RTP packet processing Threads auto-scale. If still saturated, consider DPDK/Napatech.
tar compression PCAP archiving I/O bottleneck (compression waiting for disk)
mysql store Database writes Database bottleneck. Check SQLq metric.

⚠️ Warning: If t0 thread is at 90-100%, you have hit the fundamental single-core capture limit. The t0 thread reads packets from the kernel and cannot be parallelized. Disabling features like jitterbuffer will NOT help - those run on different threads. The only solutions are:

  • Reduce captured traffic using interface_ip_filter or BPF filter
  • Use kernel bypass (DPDK or Napatech) which eliminates kernel overhead entirely

Interpreting t2CPU Detailed Breakdown

The syslog status line shows t2CPU with detailed sub-metrics: 

t2CPU[pb:10/ d:39/ s:24/ e:17/ c:6/ g:6/ r:7/ rm:24/ rh:16/ rd:19/]
Code Function High Value Indicates
pb Packet buffer output Buffer management overhead
d Dispatch Structure creation bottleneck
s SIP parsing Complex/large SIP messages
e Entity lookup Call table lookup overhead
c Call processing Call state machine processing
g Register processing High REGISTER volume
r, rm, rh, rd RTP processing stages High RTP volume (threads auto-scale)

Thread auto-scaling: VoIPmonitor automatically spawns additional threads when load increases:

  • If d > 50% → SIP parsing thread (s) starts
  • If s > 50% → Entity lookup thread (e) starts
  • If e > 50% → Call/register/RTP threads start

Configuration for High Traffic (>10,000 calls/sec)

# /etc/voipmonitor.conf

# Increase buffer to handle processing spikes (value in MB)
# 10000 = 10 GB - can go higher (20000, 30000+) if RAM allows
# Larger buffer absorbs I/O and CPU spikes without packet loss
max_buffer_mem = 10000

# Use IP filter instead of BPF (more efficient)
interface_ip_filter = 10.0.0.0/8
interface_ip_filter = 192.168.0.0/16
# Comment out any 'filter' parameter

CPU Optimizations

# /etc/voipmonitor.conf

# Reduce jitterbuffer calculations to save CPU (keeps MOS-F2 metric)
jitterbuffer_f1 = no
jitterbuffer_f2 = yes
jitterbuffer_adapt = no

# If MOS metrics are not needed at all, disable everything:
# jitterbuffer_f1 = no
# jitterbuffer_f2 = no
# jitterbuffer_adapt = no

Kernel Bypass Solutions (Extreme Loads)

When t0 thread hits 100% on standard NIC, kernel bypass is the only solution:

Solution Type CPU Reduction Use Case
DPDK Open-source ~70% Multi-gigabit on commodity hardware
Napatech Hardware SmartNIC >97% (< 3% at 10Gbit) Extreme performance requirements

Verify Improvement

# Monitor thread CPU after changes
watch -n 2 "echo 'sniffer_threads' | nc -U /tmp/vm_manager_socket | head -10"

# Or monitor syslog
journalctl -u voipmonitor -f
# t0CPU should drop, heap values should stay < 20%

ℹ️ Note: After changes, monitor syslog heap[A|B|C] values - should stay below 20% during peak traffic. See Syslog_Status_Line for detailed metric explanations.

Storage Hardware Failure

Symptom: Sensor shows disconnected (red X) with "DROPPED PACKETS" at low traffic volumes.

Diagnosis: 

# Check disk health
smartctl -a /dev/sda

# Check RAID status (if applicable)
cat /proc/mdstat
mdadm --detail /dev/md0

Look for reallocated sectors, pending sectors, or RAID degraded state. Replace failing disk.

OOM (Out of Memory)

Identify OOM Victim

# Check for OOM kills
dmesg | grep -i "out of memory\|oom\|killed process"
journalctl --since "1 hour ago" | grep -i oom

MySQL Killed by OOM

Reduce InnoDB buffer pool: 

# /etc/mysql/my.cnf
innodb_buffer_pool_size = 2G  # Reduce from default

Voipmonitor Killed by OOM

Reduce buffer sizes in voipmonitor.conf: 

max_buffer_mem = 2000  # Reduce from default
ringbuffer = 50        # Reduce from default

Runaway External Process

# Find memory-hungry processes
ps aux --sort=-%mem | head -20

# Kill orphaned/runaway process
kill -9 <PID>

For servers limited to 16GB RAM or when experiencing repeated MySQL OOM kills: 

# /etc/my.cnf or /etc/mysql/mariadb.conf.d/50-server.cnf
[mysqld]
# On 16GB server: 6GB buffer pool + 6GB MySQL overhead = 12GB total
# Leaves 4GB for OS + GUI, preventing OOM
innodb_buffer_pool_size = 6G

# Enable write buffering (may lose up to 1s of data on crash but reduces memory pressure)
innodb_flush_log_at_trx_commit = 2

Restart MySQL after changes: 

systemctl restart mysql
# or
systemctl restart mariadb

SQL Queue Growth from Non-Call Data

If sip-register, sip-options, or sip-subscribe are enabled, non-call SIP-messages (OPTIONS, REGISTER, SUBSCRIBE, NOTIFY) can accumulate in the database and cause the SQL queue to grow unbounded. This increases MySQL memory usage and leads to OOM kills of mysqld.

⚠️ Warning: Even with reduced innodb_buffer_pool_size, SQL queue will grow indefinitely without cleanup of non-call data.

Solution: Enable automatic cleanup of old non-call data 

# /etc/voipmonitor.conf
# cleandatabase=2555 automatically deletes partitions older than 7 years
# Covers: CDR, register_state, register_failed, and sip_msg (OPTIONS/SUBSCRIBE/NOTIFY)
cleandatabase = 2555

Restart the sniffer after changes: 

systemctl restart voipmonitor

ℹ️ Note: See Data_Cleaning for detailed configuration options and other cleandatabase_* parameters.

Service Startup Failures

Interface No Longer Exists

After OS upgrade, interface names may change (eth0 → ensXXX): 

# Find current interface names
ip a

# Update all config locations
grep -r "interface" /etc/voipmonitor.conf /etc/voipmonitor.conf.d/

# Also check GUI: Settings → Sensors → Configuration

Missing Dependencies

# Install common missing package
apt install libpcap0.8  # Debian/Ubuntu
yum install libpcap     # RHEL/CentOS

Network Interface Issues

Promiscuous Mode

Required for SPAN port monitoring: 

# Enable
ip link set eth0 promisc on

# Verify
ip link show eth0 | grep PROMISC

ℹ️ Note: Promiscuous mode is NOT required for ERSPAN/GRE tunnels where traffic is addressed to the sensor.

Interface Drops

# Check for drops
ip -s link show eth0 | grep -i drop

# If drops present, increase ring buffer
ethtool -G eth0 rx 4096

Bonded/EtherChannel Interfaces

Symptom: False packet loss when monitoring bond0 or br0.

Solution: Monitor physical interfaces, not logical: 

# voipmonitor.conf - use physical interfaces
interface = eth0,eth1

Network Offloading Issues

Symptom: Kernel errors like bad gso: type: 1, size: 1448 

# Disable offloading on capture interface
ethtool -K eth0 gso off tso off gro off lro off

Packet Ordering Issues

If SIP messages appear out of sequence:

First: Rule out Wireshark display artifact - disable "Analyze TCP sequence numbers" in Wireshark. See FAQ.

If genuine reordering: Usually caused by packet bursts in network infrastructure. Use tcpdump to verify packets arrive out of order at the interface. Work with network admin to implement QoS or traffic shaping. For persistent issues, consider dedicated capture card with hardware timestamping (see Napatech).

ℹ️ Note: For out-of-order packets in client/server mode (multiple sniffers), see Sniffer_distributed_architecture for pcap_queue_dequeu_window_length configuration.

Solutions for SPAN/Mirroring Reordering

If packets arrive out of order at the SPAN/mirror port (e.g., 302 responses before INVITE causing "000 no response" errors):

1. Configure switch to preserve packet order: Many switches allow configuring SPAN/mirror ports to maintain packet ordering. Consult your switch documentation for packet ordering guarantees in mirroring configuration.

2. Replace SPAN with TAP or packet broker: Unlike software-based SPAN mirroring, hardware TAPs and packet brokers guarantee packet order. Consider upgrading to a dedicated TAP or packet broker device for mission-critical monitoring.

Database Issues

SQL Queue Overload

Symptom: Growing SQLq metric, potential coredumps. 

# voipmonitor.conf - increase threads
mysqlstore_concat_limit_cdr = 1000
cdr_check_exists_callid = 0

Error 1062 - Lookup Table Limit

Symptom: Duplicate entry '16777215' for key 'PRIMARY'

Quick fix: 

# voipmonitor.conf
cdr_reason_string_enable = no

See Database Troubleshooting for complete solution.

Bad Packet Errors

Symptom: bad packet with ether_type 0xFFFF detected on interface

Diagnosis: 

# Run diagnostic (let run 30-60 seconds, then kill)
voipmonitor --check_bad_ether_type=eth0

# Find and kill the diagnostic process
ps ax | grep voipmonitor
kill -9 <PID>

Causes: corrupted packets, driver issues, VLAN tagging problems. Check ethtool -S eth0 for interface errors.

Useful Diagnostic Commands

tshark Filters for SIP

# All SIP INVITEs
tshark -r capture.pcap -Y "sip.Method == INVITE"

# Find specific phone number
tshark -r capture.pcap -Y 'sip contains "5551234567"'

# Get Call-IDs
tshark -r capture.pcap -Y "sip.Method == INVITE" -T fields -e sip.Call-ID

# SIP errors (4xx, 5xx)
tshark -r capture.pcap -Y "sip.Status-Code >= 400"

Interface Statistics

# Detailed NIC stats
ethtool -S eth0

# Watch packet rates
watch -n 1 'cat /proc/net/dev | grep eth0'

See Also





AI Summary for RAG

Summary

Comprehensive troubleshooting guide for VoIPmonitor sniffer/sensor problems. Covers: verifying traffic reaches interface (tcpdump/tshark), diagnosing no calls recorded (service, config, capture rules, SPAN), missing audio/RTP issues (one-way audio, NAT, natalias, rtp_check_both_sides_by_sdp), PACKETBUFFER FULL errors (I/O vs CPU bottleneck diagnosis using syslog metrics heap/t0CPU/SQLq and Linux tools iostat/iotop/ioping), manager commands for thread monitoring (sniffer_threads via socket or port 5029), t0 single-core capture limit and solutions (DPDK/Napatech kernel bypass), I/O solutions (NVMe/SSD, async writes, pcap_dump_writethreads), CPU solutions (max_buffer_mem 10GB+, jitterbuffer tuning), OOM issues (MySQL buffer pool, voipmonitor buffers), network interface problems (promiscuous mode, drops, offloading), packet ordering, database issues (SQL queue, Error 1062).

Keywords

troubleshooting, sniffer, sensor, no calls, missing audio, one-way audio, RTP, PACKETBUFFER FULL, memory is FULL, buffer saturation, I/O bottleneck, CPU bottleneck, heap, t0CPU, t1CPU, t2CPU, SQLq, comp, tacCPU, iostat, iotop, ioping, sniffer_threads, manager socket, port 5029, thread CPU, t0 thread, single-core limit, DPDK, Napatech, kernel bypass, NVMe, SSD, async write, pcap_dump_writethreads, tar_maxthreads, max_buffer_mem, jitterbuffer, interface_ip_filter, OOM, out of memory, innodb_buffer_pool_size, promiscuous mode, interface drops, ethtool, packet ordering, SPAN, mirror, SQL queue, Error 1062, natalias, NAT, id_sensor, snaplen, capture rules, tcpdump, tshark

Key Questions

  • Why are no calls being recorded in VoIPmonitor?
  • How to diagnose PACKETBUFFER FULL or memory is FULL error?
  • How to determine if bottleneck is I/O or CPU?
  • What do heap values in syslog mean?
  • What does t0CPU percentage indicate?
  • How to use sniffer_threads manager command?
  • How to connect to manager socket or port 5029?
  • What to do when t0 thread is at 100%?
  • How to fix one-way audio or missing RTP?
  • How to configure natalias for NAT?
  • How to increase max_buffer_mem for high traffic?
  • How to disable jitterbuffer to save CPU?
  • What causes OOM kills of voipmonitor or MySQL?
  • How to check disk I/O performance with iostat?
  • How to enable promiscuous mode on interface?
  • How to fix packet ordering issues with SPAN?
  • What is Error 1062 duplicate entry?
  • How to verify traffic reaches capture interface?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Sniffer_troubleshooting&oldid=11050"