Sniffing modes: Difference between revisions

From VoIPmonitor.org
(Review: oprava pre na syntaxhighlight, oprava úrovně nadpisu, konverze markdown tabulky na wiki formát)
(Rewrite: consolidated structure, removed redundancy, improved tables, kept all essential info)
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:VoIPmonitor Deployment & Topology Guide}}
{{DISPLAYTITLE:VoIPmonitor Deployment & Topology Guide}}


'''This guide provides a comprehensive overview of VoIPmonitor's deployment models. It covers the fundamental choice between on-host and dedicated sensors, methods for capturing traffic, and detailed configurations for scalable, multi-site architectures.'''
This guide covers VoIPmonitor deployment options: where to install the sensor, how to forward traffic, and distributed architectures for multi-site monitoring.


== Core Concept: Where to Capture Traffic ==
<kroki lang="mermaid">
The first decision in any deployment is where the VoIPmonitor sensor (sniffer) will run.
%%{init: {'flowchart': {'nodeSpacing': 15, 'rankSpacing': 30}}}%%
flowchart TB
    START[Where to deploy sensor?] --> Q1{PBX runs on Linux?}
    Q1 -->|Yes| Q2{Spare resources?}
    Q1 -->|No - Windows| DED[Dedicated Sensor]
    Q2 -->|Yes| ONHOST[On-Host Capture]
    Q2 -->|No| DED


=== 1. On-Host Capture (on the PBX/SBC) ===
    DED --> Q3{Traffic forwarding method?}
The sensor can be installed directly on the same Linux server that runs your PBX or SBC.
    Q3 --> SPAN[SPAN/RSPAN]
* '''Pros:''' Requires no extra hardware, network changes, or port mirroring. It is the simplest setup.
    Q3 --> TUNNEL[Software Tunnel]
* '''Cons:''' Adds CPU, memory, and disk I/O load to your production voice server. If these resources are critical, a dedicated sensor is the recommended approach.
    Q3 --> CLOUD[Cloud Mirroring]


'''Platform Note:''' The VoIPmonitor sensor runs exclusively on Linux. While some PBXs are available on Windows (e.g., 3CX Windows edition, certain legacy systems), the sensor cannot be installed on Windows. For Windows-based PBXs, you must use a dedicated Linux sensor with traffic mirroring (see below).
    TUNNEL --> T1[GRE/ERSPAN]
    TUNNEL --> T2[TZSP/VXLAN]
    TUNNEL --> T3[HEP/AudioCodes]
</kroki>


=== 2. Dedicated Sensor ===
= Sensor Deployment Options =
A dedicated Linux server runs only the VoIPmonitor sensor. This is the recommended approach for production environments as it isolates monitoring resources from your voice platform. To use a dedicated sensor, you must forward a copy of the network traffic to it using one of the methods below.


'''When a Dedicated Sensor is Required:'''
== On-Host Capture ==
* Windows-based PBXs (e.g., 3CX Windows edition) - VoIPmonitor sensor is Linux-only
* When your PBX/SBC server has limited CPU, RAM, or disk I/O resources
* When you want zero monitoring impact on your production voice platform
* When capturing from multiple sites with a centralized collector


== Methods for Forwarding Traffic to a Dedicated Sensor ==
Install the sensor directly on the same Linux server as your PBX/SBC.


=== A. Hardware Port Mirroring (SPAN/RSPAN) ===
{| class="wikitable"
This is the most common and reliable method. You configure your physical network switch to copy all traffic from the switch ports connected to your PBX/SBC to the switch port connected to the VoIPmonitor sensor. This feature is commonly called '''Port Mirroring''', '''SPAN''', or '''RSPAN'''. Consult your switch's documentation for configuration details.
! Pros !! Cons
 
The VoIPmonitor sensor interface will be put into promiscuous mode automatically. To capture from multiple interfaces, set <code>interface = any</code> in <code>voipmonitor.conf</code> and enable promiscuous mode manually on each NIC (e.g., <code>ip link set dev eth1 promisc on</code>).
 
{| class="wikitable" style="background:#d4edda; border:1px solid #28a745;"
|-
|-
! colspan="2" style="background:#28a745; color: white;" | Critical for Multiple Mirrored Interfaces
| No extra hardware, network changes, or port mirroring required || Adds CPU, memory, and disk I/O load to production voice server
|-
|-
| style="vertical-align: top;" | '''Issue:'''
| Simplest setup || Not suitable if resources are critical
| When sniffing from multiple mirrored interfaces, VLANs, or switch ports, packets may arrive as duplicates (same traffic from multiple SPAN sources). This can cause incomplete calls, missing audio, or incorrect SIP/RTP session reassembly.
|-
| style="vertical-align: top;" | '''Solution:'''
| Add <code>auto_enable_use_blocks = yes</code> to <code>voipmonitor.conf</code>. This enables automatic packet deduplication and defragmentation, ensuring packets from multiple sources are correctly identified and merged. See [[Sniffer_configuration#auto_enable_use_blocks|Sniffer_configuration]] for details.
|}
|}


=== B. Software-based Tunnelling ===
{{Note|1=VoIPmonitor sensor runs '''exclusively on Linux'''. For Windows-based PBXs (e.g., 3CX Windows edition), you must use a dedicated Linux sensor with traffic mirroring.}}
When hardware mirroring is not an option, many network devices and PBXs can encapsulate VoIP packets and send them to the sensor's IP address using a tunnel. VoIPmonitor natively supports a wide range of protocols.
 
* '''Built-in Support:''' IP-in-IP, GRE, ERSPAN
== Dedicated Sensor ==
* '''UDP-based Tunnels:''' Configure the corresponding port in <code>voipmonitor.conf</code>:
 
** <code>udp_port_tzsp = 37008</code> (for MikroTik's TZSP)
A separate Linux server runs only VoIPmonitor. '''Recommended for production environments''' as it isolates monitoring from voice platform resources.
** <code>udp_port_l2tp = 1701</code>
** <code>udp_port_vxlan = 4789</code> (common in cloud environments)
* '''Proprietary & Other Protocols:'''
** [[audiocodes tunneling|AudioCodes Tunneling]] (uses <code>udp_port_audiocodes</code> or <code>tcp_port_audiocodes</code>)
** HEP (Homer Encapsulation Protocol)
** IPFIX (for Oracle SBCs) (enable <code>ipfix*</code> options)


==== HEP (Homer Encapsulation Protocol) ====
'''When Required:'''
* Windows-based PBXs
* Limited CPU/RAM/disk I/O on PBX server
* Zero monitoring impact needed
* Centralized capture from multiple sites


HEP is a lightweight protocol for capturing and mirroring VoIP packets. Many SBCs and sip proxies (such as Kamailio, OpenSIPS, FreeSWITCH) support HEP to send a copy of traffic to a monitoring server.
= Traffic Forwarding Methods =


'''Configuration in voipmonitor.conf:'''
When using a dedicated sensor, you must forward traffic to it using one of these methods.


<syntaxhighlight lang="ini">
== Hardware Port Mirroring (SPAN/RSPAN) ==
# Enable HEP support
 
hep = yes
Physical or virtual switches copy traffic from source port(s) to a monitoring port.


# Port to listen for HEP packets (default: 9060)
=== Physical Switch ===
hep_bind_port = 9060


# Optional: Bind to specific IP address
Configure your switch to mirror traffic from PBX/SBC ports to the sensor's port. Consult your switch documentation for specific commands.
# hep_bind_ip = 0.0.0.0


# Optional: Enable UDP binding (default: yes)
<syntaxhighlight lang="ini">
hep_bind_udp = yes
# /etc/voipmonitor.conf
interface = eth0
sipport = 5060
savertp = yes
</syntaxhighlight>
</syntaxhighlight>


When <code>hep = yes</code>, VoIPmonitor listens for HEPv3 (and compatible HEPv2) packets and extracts the original VoIP traffic from the encapsulation.
{{Tip|To capture from multiple interfaces, set <code>interface = any</code> and enable promiscuous mode on each NIC: <code>ip link set dev eth1 promisc on</code>}}


'''Use Cases:'''
=== VMware/ESXi Virtual Switch ===
* Remote SBCs or PBXs export traffic to a centralized VoIPmonitor server
* Kamailio/FreeSWITCH <code>siptrace</code> module integration
* Environments where standard tunnels (GRE/ERSPAN) are not available


'''Note:''' There is also <code>hep_kamailio_protocol_id_fix = yes</code> for Kamailio-specific protocol ID issues.
For virtualized environments, VMware provides port mirroring at the virtual switch level.


'''Known Limitations:'''
'''Standard vSwitch:'''
# In vSphere Client, navigate to ESXi host
# Select virtual switch → Properties/Edit Settings → Enable Port Mirroring
# Set source (SBC VM) and destination (VoIPmonitor VM) ports
 
'''Distributed vSwitch:'''
# In vSphere Web Client → Networking → Select distributed switch
# Configure tab → Port Mirroring → Create mirroring session
# Specify source/destination ports and enable


===== HEP Timestamp Precision =====
{{Note|1=Distributed switch mirroring can span multiple ESXi hosts within a cluster.}}


HEP3 packets include a timestamp field that represents when the packet was captured at the source. VoIPmonitor uses this HEP timestamp for the call record. If the source HEP server has an unreliable or unsynchronized time source, this can cause incorrect timestamps in the captured calls.
=== Multiple VoIP Platforms ===


Currently, there is no built-in configuration option to ignore the HEP timestamp and instead use the time when VoIPmonitor receives the packet. If you need this functionality, please:
Monitor multiple platforms (e.g., Mitel + FreeSWITCH) with a single sensor by mirroring multiple source ports to one destination port.


* Request the feature on the product roadmap (no guaranteed ETA)
'''GUI differentiation:'''
* Consider a custom development project for a fee
* Filter by IP address ranges
* Filter by number prefixes
* Use separate sensors with unique <code>id_sensor</code> values


===== No HEP Correlation ID Support =====
{{Warning|1='''Critical:''' When sniffing from multiple mirrored sources, packets may arrive as duplicates. Add <code>auto_enable_use_blocks = yes</code> to voipmonitor.conf to enable automatic deduplication. See [[Sniffer_configuration#auto_enable_use_blocks|Sniffer_configuration]] for details.}}


**VoIPmonitor does not use HEP correlation ID (captureNodeID) to correlate SIP and RTP packets.**
== Software-based Tunneling ==


When SIP signaling and RTP media are encapsulated in HEP and arrive from different HEP sources (different capture nodes or sensors), VoIPmonitor cannot correlate them into a single CDR using the HEP protocol metadata.
When hardware mirroring is unavailable, use software tunneling to encapsulate and forward packets.


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
{| class="wikitable"
! Protocol !! Configuration Parameter !! Notes
|-
|-
! colspan="2" style="background:#ffc107;" | HEP Correlation Limitation
| IP-in-IP, GRE, ERSPAN || Built-in (auto-detected) || No additional config needed
|-
|-
| style="vertical-align: top;" | '''What does not work:'''
| TZSP (MikroTik) || <code>udp_port_tzsp = 37008</code> ||
| * HEP Source A sends SIP packets
* HEP Source B sends RTP packets for the same call
* VoIPmonitor tries to use HEP captureNodeID/correlation ID to merge them
* Result: SIP and RTP are NOT correlated; the call appears incomplete or missing
|-
|-
| style="vertical-align: top;" | '''Why:'''
| L2TP || <code>udp_port_l2tp = 1701</code> ||
| VoIPmonitor extracts the payload from HEP encapsulation and correlates using standard SIP Call-ID, To/From tags, and RTP SSRC fields. It does not utilize the HEP envelope metadata fields (correlation ID, capture node ID, etc.) for cross-sensor correlation.
|-
|-
| style="vertical-align: top;" | '''Workaround (Feature Request VS-1703):'''
| VXLAN || <code>udp_port_vxlan = 4789</code> || Common in cloud environments
| Currently, there is no available workaround. The only options are:
|-
* Wait for a future release that adds HEP correlation ID support (feature request VS-1703 has been logged)
| AudioCodes || <code>udp_port_audiocodes = 925</code> || See [[Audiocodes_tunneling|AudioCodes Tunneling]]
* Pursue a custom paid implementation to add this functionality
|-
 
| IPFIX (Oracle SBCs) || <code>ipfix*</code> options || Enable ipfix options in config
This is fundamentally different from VoIPmonitor's standard "single sniffer requirement" for raw packet capture. The HEP protocol limitation is specific to how HEP encapsulation is processed.
|}
|}


=== Pre-Deployment Compatibility Verification ===
=== HEP (Homer Encapsulation Protocol) ===


Before full production deployment, especially when integrating VoIPmonitor with network hardware (Cisco/Juniper routers, SBCs), or complex mirroring setups (RSPAN, ERSPAN, tunnels), it is highly recommended to verify that VoIPmonitor can correctly capture and process packets from your specific environment.
Lightweight protocol for mirroring VoIP packets. Supported by Kamailio, OpenSIPS, FreeSWITCH, and many SBCs.


This approach allows you to identify compatibility issues early, without committing to a full deployment that may need adjustments.
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf
hep = yes
hep_bind_port = 9060
hep_bind_udp = yes
# Optional: hep_kamailio_protocol_id_fix = yes
</syntaxhighlight>


'''Typical Use Cases:'''
'''Known Limitations:'''
* Deploying a dedicated sensor with SPAN/RSPAN from a Cisco router or switch
* Using ERSPAN to forward VoIP traffic across routers
* Capturing from proprietary SBCs or VoIP gateways (Cisco C2951, AudioCodes, etc.)
* Implementing newer or complex tunneling protocols (VXLAN, GRE with specific configurations)


'''Verification Workflow:'''
{{Warning|1='''HEP Correlation ID Not Supported:''' VoIPmonitor does NOT use HEP correlation ID (captureNodeID) to correlate SIP and RTP packets. If SIP and RTP arrive from different HEP sources, they will NOT be correlated into a single CDR.


1. **Configure Mirroring in Test Mode:** Set up the SPAN, RSPAN, ERSPAN, or tunnel configuration to forward a small subset of VoIP traffic to a test sensor or VM.
VoIPmonitor correlates using standard SIP Call-ID, To/From tags, and RTP SSRC fields only. Feature request VS-1703 has been logged but there is no workaround currently.}}


2. **Capture Test Calls:**
'''HEP Timestamp:''' VoIPmonitor uses the HEP timestamp field. If the source has an unsynchronized clock, call timestamps will be incorrect. There is no option to ignore HEP timestamps.
  * Make a few test calls through your VoIP system.
  * Using <code>tcpdump</code> or <code>tshark</code>, capture the mirrored traffic into a pcap file:
    <syntaxhighlight lang="bash">
# Example: Capture SIP and RTP from the mirrored interface
sudo tcpdump -i eth0 -s0 port 5060 -w /tmp/compatibility_test.pcap
    </syntaxhighlight>


3. **Verify Packet Capture:**
'''HEP3 with Port 0:''' Not captured by default. Add port 0 to sipport:
  * Confirm the pcap contains both SIP signaling and RTP audio:
    <syntaxhighlight lang="bash">
tshark -r /tmp/compatibility_test.pcap -Y "sip || rtp"
    </syntaxhighlight>
  * Check for expected packet sizes, codecs, and call flow.


4. **Submit for Analysis:** Send the pcap file to VoIPmonitor support along with details about:
<syntaxhighlight lang="ini">
  * Your network hardware (Cisco router model, switch model, SBC model)
sipport = 0,5060
  * Mirroring method (SPAN, RSPAN, ERSPAN, GRE, VXLAN, etc.)
</syntaxhighlight>
  * Any special configurations (VLAN tags, MPLS labels, encapsulation)
  * Your planned deployment (on-host vs. dedicated sensor, client/server vs. standalone)
 
5. **Feedback and Adjustment:** Support will analyze the pcap and confirm if VoIPmonitor can process your specific traffic structure. They may recommend configuration changes (e.g., adjusting <code>sipport</code>, enabling tunnel decapsulation, modifying TCP/UDP port settings) or identify incompatible traffic patterns.
 
'''Benefits of Pre-Deployment Testing:'''
* Confirms VoIPmonitor compatibility with your specific hardware and network setup
* Identifies configuration needs before full production deployment
* Saves time by avoiding trial-and-error during go-live
* Provides documented proof of concept for stakeholders
* Allows tuning of sensor resources (CPU/RAM/disk) based on actual traffic characteristics
 
If verification fails or reveals incompatibilities, support can often suggest alternative approaches or configuration adjustments before you proceed.
 
==== Cloud Packet Mirroring (GCP, AWS, Azure) ====
 
Cloud providers offer native packet mirroring services that can forward traffic to a dedicated VoIPmonitor sensor. These services typically use '''VXLAN''' or '''GRE''' encapsulation.
 
'''Supported Cloud Services:'''
 
* Google Cloud Platform (GCP): Packet Mirroring
* Amazon Web Services (AWS): Traffic Mirroring
* Microsoft Azure: Virtual Network TAP
 
'''Configuration Steps:'''
 
1. **Create a Dedicated Sensor VM:** Deploy a VoIPmonitor sensor instance in your cloud environment. This VM should be sized appropriately for your expected traffic volume.
 
2. **Configure Cloud Mirroring Policy:** In your cloud provider's console, create a mirroring policy:
  * Select source VMs or subnets where your VoIP traffic (PBX/SBC) originates.
  * Set the destination to the internal IP of your VoIPmonitor sensor VM.
  * Ensure the encapsulation protocol is compatible with VoIPmonitor (VXLAN is recommended and most common).
 
3. **Critical: Bidirectional Capture:** Configure the mirroring policy to capture traffic '''in BOTH directions''':
  * <code>INGRESS</code> (incoming traffic to sources)
  * <code>EGRESS</code> (outgoing traffic from sources)
  * <code>BOTH</code> or <code>EITHER</code> is recommended
 
  <code>WARNING: Capturing only ingress or only egress will result in incomplete call data and broken CDRs.</code>
 
4. **Configure VoIPmonitor Sensor:**
 
  <syntaxhighlight lang="ini">
  # Enable VXLAN support for cloud packet mirroring
  udp_port_vxlan = 4789
 
  # Interface configuration
  interface = eth0
 
  # SIP ports
  sipport = 5060


  # Optional: Filter at source to save bandwidth
== Cloud Packet Mirroring ==
  # Configure cloud mirroring filters to forward only SIP/RTP traffic
  </syntaxhighlight>


5. **VM Sizing for Cloud Sensor:** Properly size the sensor VM instance:
Cloud providers offer native mirroring services using VXLAN or GRE encapsulation.
  * <code>vCPU:</code> Allow 1-2 cores per 100 concurrent calls (adjusted for codec complexity and packet rate).
  * <code>RAM:</code> 4GB minimum for production; more if using on-disk compression or high PCAP retention.
  * <code>Storage:</code> Use SSD or high-throughput block storage for the <code>spooldir</code>. VoIPmonitor is I/O intensive — persistent disk performance is critical to avoid packet loss.
  * <code>Network:</code> Ensure sufficient NIC bandwidth; mirroring multiple high-traffic sources can saturate the sensor's interface.


6. **NTP Synchronization:** Accurate timekeeping is critical. Ensure all VMs (sources, sensor, and related infrastructure) use the cloud provider's internal NTP servers or a reliable external NTP source.
'''Best Practices for Cloud Mirroring:'''
* <code>Filter at the Source:</code> Use cloud mirroring filters to forward only SIP signaling and RTP audio ports. Sending all network traffic (HTTP, SSH, etc.) wastes CPU and bandwidth.
* <code>Monitor Network Limits:</code> Cloud NICs have bandwidth limits (e.g., 10 Gbps). Mirroring multiple high-traffic sources may saturate the sensor VM's interface.
* <code>MTU Considerations:</code> VXLAN adds ~50 bytes of overhead. If original packets are near 1500 bytes MTU, encapsulated packets may exceed it, causing fragmentation or drops. Ensure network path supports jumbo frames or proper fragmentation handling.
* <code>Test Load:</code> Start with filtered ports and a subset of traffic, monitor performance, then expand to full production volume.
'''Alternative: Client/Server Architecture with On-Host Sensors'''
Instead of cloud packet mirroring, consider installing VoIPmonitor sensors directly on each PBX/SBC VM using the [[Sniffer_distributed_architecture|Client/Server architecture]]:
* Install sensor on each Asterisk/SBC VM (on-host capture)
* Sensors process calls locally or forward packets via <code>packetbuffer_sender</code> to a central collector
* Eliminates mirroring overhead and potential incomplete capture issues
* May have better performance for high-traffic environments
== Distributed Deployment Models ==
For monitoring multiple remote offices or a large infrastructure, a distributed model is essential. This involves a central GUI/Database server collecting data from multiple remote sensors.
=== Classic Mode: Standalone Remote Sensors ===
In this traditional model, each remote sensor is a fully independent entity.
* '''How it works:''' The remote sensor processes packets and stores PCAPs locally. It connects directly to the central MySQL/MariaDB database to write CDRs. For PCAP retrieval, the GUI typically needs network access to each sensor's management port (default <code>TCP/5029</code>).
* '''Pros:''' Simple conceptual model.
* '''Cons:''' Requires opening firewall ports to each sensor and managing database credentials on every remote machine.
==== Alternative PCAP Access: NFS/SSHFS Mounting ====
For environments where direct TCP/5029 access to remote sensors is impractical (e.g., firewalls, VPN limitations), you can mount remote spool directories on the central GUI server using NFS or SSHFS.
'''Use Cases:'''
* Firewall policies block TCP/5029 but allow SSH or NFS traffic
* Remote sensors have local databases that need to be queried separately
* You want the GUI to access PCAPs directly from mounted filesystems instead of proxying through TCP/5029
'''Configuration Steps:'''
1. **Mount remote spools on GUI server:**
  Using NFS:
  <syntaxhighlight lang="bash">
  # On GUI server, mount remote spool directory
  sudo mount -t nfs 10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1
  sudo mount -t nfs 10.224.0.102:/var/spool/voipmonitor /mnt/voipmonitor/sensor2
  # Add to /etc/fstab for persistent mounts
  10.224.0.101:/var/spool/voipmonitor  /mnt/voipmonitor/sensor1  nfs  defaults  0  0
  10.224.0.102:/var/spool/voipmonitor  /mnt/voipmonitor/sensor2  nfs  defaults  0  0
  </syntaxhighlight>
  Using SSHFS:
  <syntaxhighlight lang="bash">
  # On GUI server, mount remote spool via SSHFS
  sshfs voipmonitor@10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1
  sshfs voipmonitor@10.224.0.102:/var/spool/voipmonitor /mnt/voipmonitor/sensor2
  # Add to /etc/fstab for persistent mounts (with key-based auth)
  voipmonitor@10.224.0.101:/var/spool/voipmonitor  /mnt/voipmonitor/sensor1  fuse.sshfs  defaults,IdentityFile=/home/voipmonitor/.ssh/id_rsa  0  0
  </syntaxhighlight>
2. **Configure PCAP spooldir path in GUI:**
  In the GUI, go to '''Settings > System Configuration > Sniffer data path''' and set it to search multiple spool directories. Each directory is separated by a colon (<code>:</code>).
  <syntaxhighlight lang="text">
  Sniffer data path: /var/spool/voipmonitor:/mnt/voipmonitor/sensor1:/mnt/voipmonitor/sensor2
  </syntaxhighlight>
  The GUI will search these paths in order when looking for PCAP files.
3. **Register remote sensors in GUI:**
  Go to '''Settings > Sensors''' and register each remote sensor:
  * '''Sensor ID:''' Must match <code>id_sensor</code> in each remote's <code>voipmonitor.conf</code>
  * '''Name:''' Descriptive name (e.g., "Site 1 - London")
  * '''Manager IP, Port:''' Optional with NFS/SSHFS mount (leave empty if mounting spools directly)
'''Important Notes:'''
* Each remote sensor must have a unique <code>id_sensor</code> configured in <code>voipmonitor.conf</code>
* Remote sensors write directly to their local MySQL database (or possibly to a central database)
* Filter calls by site using the <code>id_sensor</code> column in the CDR view
* Ensure mounted directories are writable by the GUI user for PCAP uploads
* For better performance, use NFS with async or SSHFS with caching options
'''Filtering and Site Identification:'''
* In the CDR view, use the '''Sensor''' dropdown filter to select specific sites
* Alternatively, filter by IP address ranges using CDR columns
* The <code>id_sensor</code> column in the database uniquely identifies which sensor captured each call
* Sensor names can be customized in '''Settings > Sensors''' for easier identification
'''Comparison: TCP/5029 vs NFS/SSHFS'''
{| class="wikitable"
{| class="wikitable"
! Approach
! Provider !! Service Name
! Network Traffic
! Firewall Requirements
! Performance
! Use Case
|-
|-
| TCP/5029 Proxy (Standard)
| Google Cloud || Packet Mirroring
| On-demand fetch per request
| TCP/5029 outbound from GUI to sensors
| Better (no continuous mount overhead)
| Most deployments
|-
|-
| NFS Mount
| AWS || Traffic Mirroring
| Continuous (filesystem access)
| NFS ports (usually 2049) bidirectional
| Excellent (local filesystem speed)
| Local networks, high-throughput
|-
|-
| SSHFS Mount
| Azure || Virtual Network TAP
| Continuous (encrypted filesystem)
| SSH (TCP/22) outbound from GUI
| Good (some encryption overhead)
| Remote sites, cloud/VPN
|}
|}


=== Troubleshooting NFS/SSHFS Mounts ===
'''Configuration Steps:'''


If you experience missing CDRs or PCAP files for a specific time period, or if the GUI reports files not found despite sensors receiving traffic, the issue is often NFS/SSHFS connectivity between the probe and storage server.
# Create a VoIPmonitor sensor VM in your cloud environment
# Create mirroring policy: select source VMs/subnets, set destination to sensor VM
# '''Critical:''' Capture traffic in '''BOTH directions''' (INGRESS and EGRESS)
# Configure sensor:


==== Check for NFS/SSHFS Connectivity Issues ====
<syntaxhighlight lang="ini">
udp_port_vxlan = 4789
interface = eth0
sipport = 5060
</syntaxhighlight>
 
{{Warning|1=Capturing only ingress or only egress results in incomplete CDRs and broken call data.}}


Missing data (both CDRs and PCAPs) for a specific time period is typically caused by network unavailability between the VoIPmonitor probe and the NFS/SSHFS storage server.
'''Best Practices:'''
* Filter at source to forward only SIP/RTP ports
* Monitor NIC bandwidth limits
* Account for VXLAN overhead (~50 bytes) - may need jumbo frames
* Ensure NTP sync across all VMs


* **1. Check system logs for NFS or SSHFS errors:**
'''Alternative:''' Consider [[Sniffer_distributed_architecture|Client/Server architecture]] with on-host sensors instead of cloud mirroring for better performance.


  <syntaxhighlight lang="bash">
== Pre-Deployment Verification ==
  # Check for NFS-specific errors
  journalctl -u voipmonitor --since "2024-01-01" --until "2024-01-02"


  # Look for specific patterns in syslog
For complex setups (RSPAN, ERSPAN, proprietary SBCs), verify compatibility before production deployment:
  grep "nfs: server.*not responding" /var/log/syslog
  grep "nfs.*timed out" /var/log/syslog
  grep "I/O error" /var/log/syslog


  # For SSHFS issues
# Configure test mirroring with a subset of traffic
  grep "sshfs.*Connection reset" /var/log/syslog
# Capture test calls with tcpdump: <code>sudo tcpdump -i eth0 -s0 port 5060 -w /tmp/test.pcap</code>
  grep "sshfs.*Transport endpoint is not connected" /var/log/syslog
# Verify pcap contains SIP and RTP: <code>tshark -r /tmp/test.pcap -Y "sip || rtp"</code>
  </syntaxhighlight>
# Submit pcap to VoIPmonitor support with hardware/configuration details


  Key error messages to look for:
= Distributed Architectures =
  * <code>nfs: server 192.168.1.100 not responding, timed out</code> - NFS server unreachable
  * <code>nfs: server 192.168.1.100 OK</code> - Connection restored after interruption
  * <code>Stale file handle</code> - NFS mount needs remounting
  * <code>Transport endpoint is not connected</code> - SSHFS mount disconnected


* **2. Verify network connectivity to the storage server:**
For multi-site monitoring, sensors can be deployed in various configurations.


  <syntaxhighlight lang="bash">
== Classic Mode: Standalone Sensors ==
  # Ping test to the NFS/SSHFS server
  ping 192.168.1.100


  # Trace the network path to identify bottlenecks
Each sensor operates independently:
  traceroute 192.168.1.100
* Processes packets and stores PCAPs locally
* Connects directly to central MySQL to write CDRs
* GUI needs network access to each sensor's <code>TCP/5029</code> for PCAP retrieval


  # Test DNS resolution if using hostnames
'''Alternative: NFS/SSHFS Mounting'''
  nslookup storage-server.domain.com
  </syntaxhighlight>


* **3. Ensure the NFS/SSHFS server is running and accessible:**
If TCP/5029 access is blocked, mount remote spool directories on the GUI server:


  <syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
  # On the probe/sensor side - check if mount is active
# NFS mount
  mount | grep nfs
sudo mount -t nfs 10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1
  mount | grep fuse.sshfs


  # Check mount status for all mounted spool directories
# SSHFS mount
  stat /mnt/voipmonitor/sensor1
sshfs voipmonitor@10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1
</syntaxhighlight>


  # On the NFS server side - verify services are running
Configure GUI: '''Settings > System Configuration > Sniffer data path:'''
  systemctl status nfs-server
<code>/var/spool/voipmonitor:/mnt/voipmonitor/sensor1:/mnt/voipmonitor/sensor2</code>
  systemctl status sshd
  </syntaxhighlight>


* **4. Check for mount-specific issues:**
{{Tip|For NFS, use <code>hard,nofail,tcp</code> mount options for reliability.}}


  <syntaxhighlight lang="bash">
== Modern Mode: Client/Server (v20+) — Recommended ==
  # Test NFS mount manually (unmount and remount)
  sudo umount /mnt/voipmonitor/sensor1
  sudo mount -t nfs 10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1


  # Check /etc/fstab for mount errors
Secure encrypted TCP channel between remote sensors and central server. GUI communicates only with central server.
  sudo mount -a  # Test all mounts in /etc/fstab


  # Verify mount permissions
<kroki lang="mermaid">
  ls -la /mnt/voipmonitor/sensor1
%%{init: {'flowchart': {'nodeSpacing': 10, 'rankSpacing': 25}}}%%
  </syntaxhighlight>
flowchart LR
    subgraph "Local Processing"
        R1[Remote Sensor] -->|CDRs only| C1[Central Server]
        R1 -.->|PCAP on demand| C1
    end


==== Common Causes of Missing Data ====
    subgraph "Packet Mirroring"
        R2[Remote Sensor] -->|Raw packets| C2[Central Server]
    end
</kroki>


{| class="wikitable"
{| class="wikitable"
! Symptom
! Mode !! Processing !! PCAP Storage !! WAN Traffic !! Best For
! Most Likely Cause
! Troubleshooting Step
|-
| Gap in data during a specific time period
| '''NFS/SSHFS server unreachable'''
| Check logs for "not responding, timed out"
|-
|-
| Stale file handle errors
| '''Local Processing''' (<code>packetbuffer_sender=no</code>) || Remote || Remote || Low (CDRs only) || Limited WAN bandwidth
| NFS server rebooted or export changed
| Remount NFS share
|-
|-
| Connection resets
| '''Packet Mirroring''' (<code>packetbuffer_sender=yes</code>) || Central || Central || High (full packets) || Low-resource remote sites
| Network interruption or unstable connection
| Check network stability and ping times
|-
| Very slow file access
| Network latency or bandwidth saturation
| Monitor network throughput
|-
| GUI shows "File not found"
| Mount point dismounted
| Check mount status and remount if needed
|}
|}


==== Preventative Measures ====
For detailed configuration, see [[Sniffer_distributed_architecture|Distributed Architecture: Client-Server Mode]].
 
To minimize data loss from NFS/SSHFS connectivity issues:
 
* **Use TCP for NFS** (more reliable than UDP):
  <syntaxhighlight lang="bash">
  # Mount NFS with TCP explicitly
  sudo mount -t nfs -o tcp 10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1
  </syntaxhighlight>
 
* **Use the <code>hard,nofail</code> mount options:**
  <syntaxhighlight lang="bash">
  # In /etc/fstab
  10.224.0.101:/var/spool/voipmonitor  /mnt/voipmonitor/sensor1  nfs  hard,nofail,tcp  0  0
  </syntaxhighlight>
  * <code>hard</code>: Make NFS operations wait indefinitely instead of timing out
  * <code>nofail</code>: Do not fail if the mount is unavailable at boot time


* **Monitor mount status:**
'''Quick Start - Remote Sensor (Local Processing):'''
  Set up automated monitoring to alert when NFS/SSHFS mounts become unresponsive or disconnected.


* **Consider Client/Server mode as alternative:**
  If NFS/SSHFS connectivity is unreliable, consider using the modern [[Sniffer_distributed_architecture|Client/Server architecture]] instead, which uses encrypted TCP channels and is more resilient to network interruptions.
=== Modern Mode: Client/Server Architecture (v20+) — Recommended ===
This model uses a secure, encrypted TCP channel between remote sensors (clients) and a central sensor instance (server). The GUI communicates with the central server only, which significantly simplifies networking and security.
This architecture supports two primary modes:
# '''Local Processing:''' Remote sensors process packets locally and send only lightweight CDR data over the encrypted channel. PCAPs remain on the remote sensor. On-demand PCAP fetch is proxied via the central server (to the sensor's <code>TCP/5029</code>).
# '''Packet Mirroring:''' Remote sensors forward the entire raw packet stream to the central server, which performs all processing and storage. Ideal for low-resource remote sites.
==== Architecture Diagrams (PlantUML) ====
<kroki lang="plantuml">
  @startuml
  skinparam shadowing false
  skinparam defaultFontName Arial
  skinparam rectangle {
    BorderColor #4A90E2
    BackgroundColor #FFFFFF
    stereotypeFontColor #333333
  }
  skinparam packageBorderColor #B0BEC5
  skinparam packageBackgroundColor #F7F9FC
  title Client/Server Architecture — Local Processing Mode
  package "Remote Site" {
    [Remote Probe/Sensor] as Remote
    database "Local Storage (PCAP)" as RemotePCAP
  }
  package "Central Site" {
    [Central VoIPmonitor Server] as Central
    database "Central MySQL/MariaDB" as CentralDB
    [Web GUI] as GUI
  }
  Remote -[#2F6CB0]-> Central : Encrypted TCP/60024\nCDRs only
  Remote --> RemotePCAP : Stores PCAP locally
  Central --> CentralDB : Writes CDRs
  GUI -[#2F6CB0]-> Central : Queries data & requests PCAPs
  Central -[#2F6CB0]-> RemotePCAP : Fetches PCAPs on demand (TCP/5029)
  @enduml
  </kroki>
<kroki lang="plantuml">
  @startuml
  skinparam shadowing false
  skinparam defaultFontName Arial
  skinparam rectangle {
    BorderColor #4A90E2
    BackgroundColor #FFFFFF
    stereotypeFontColor #333333
  }
  skinparam packageBorderColor #B0BEC5
  skinparam packageBackgroundColor #F7F9FC
  title Client/Server Architecture — Packet Mirroring Mode
  package "Remote Site" {
    [Remote Probe/Sensor\n(Low Resource)] as Remote
  }
  package "Central Site" {
    [Central VoIPmonitor Server] as Central
    database "Central MySQL/MariaDB" as CentralDB
    database "Central Storage (PCAP)" as CentralPCAP
    [Web GUI] as GUI
  }
  Remote -[#2F6CB0]-> Central : Encrypted TCP/60024\nRaw packet stream
  Central --> CentralDB : Writes CDRs
  Central --> CentralPCAP : Processes & stores PCAPs
  GUI -[#2F6CB0]-> Central : Queries data & downloads PCAPs
  @enduml
  </kroki>
==== Step-by-Step Configuration Guide ====
; Prerequisites
* VoIPmonitor v20+ on all sensors.
* Central database reachable from the central server instance.
* Unique <code>id_sensor</code> per sensor (< 65536).
* NTP running everywhere (see '''Time Synchronization''' below).
; Scenario A — Local Processing (default, low WAN usage)
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the REMOTE sensor (LOCAL PROCESSING)
id_sensor              = 2
 
id_sensor              = 2         # unique per sensor (< 65536)
server_destination      = 10.224.0.250
server_destination      = 10.224.0.250
server_destination_port = 60024
server_destination_port = 60024
server_password        = your_strong_password
server_password        = your_strong_password
packetbuffer_sender    = no
interface              = eth0
sipport                = 5060
# No MySQL credentials needed - central server writes to DB
</syntaxhighlight>


packetbuffer_sender    = no        # local analysis; sends only CDRs
'''Quick Start - Central Server:'''
interface              = eth0      # or: interface = any
sipport                = 5060      # example; add your usual sniffer options
 
# No MySQL credentials here — remote sensor does NOT write to DB directly.
</syntaxhighlight>


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the CENTRAL server (LOCAL PROCESSING network)
server_bind            = 0.0.0.0
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_bind_port        = 60024
server_password        = your_strong_password
server_password        = your_strong_password
mysqlhost              = 10.224.0.201
mysqlhost              = 10.224.0.201
mysqldb                = voipmonitor
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
mysqlpassword          = db_password
 
cdr_partition          = yes
cdr_partition          = yes       # partitions for CDR tables
interface              =   # Leave empty - don't sniff locally
mysqlloadconfig        = yes        # allows DB-driven config if used
 
interface              =           # leave empty to avoid local sniffing
# The central server will proxy on-demand PCAP fetches to sensors (TCP/5029).
</syntaxhighlight>
</syntaxhighlight>


; Scenario B — Packet Mirroring (centralized processing/storage)
== Firewall Requirements ==
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the REMOTE sensor (PACKET MIRRORING)


id_sensor              = 3
{| class="wikitable"
server_destination      = 10.224.0.250
! Deployment !! Port !! Direction !! Purpose
server_destination_port = 60024
|-
server_password        = your_strong_password
| Client/Server || TCP/60024 || Remote → Central || Encrypted CDR/packet channel
|-
| Client/Server || TCP/5029 || Central → Remote || On-demand PCAP fetch (Local Processing mode)
|-
| GUI Access || TCP/5029 || GUI → Central || Management/API
|-
| Cloud Mode || TCP/60023 || Sensor → cloud.voipmonitor.org || Cloud service connection
|}


packetbuffer_sender    = yes        # send RAW packet stream to central
= Configuration Notes =
interface              = eth0      # capture source; no DB settings needed
</syntaxhighlight>


<syntaxhighlight lang="ini">
== Critical Parameters ==
# /etc/voipmonitor.conf on the CENTRAL server (PACKET MIRRORING)


server_bind            = 0.0.0.0
{| class="wikitable"
server_bind_port        = 60024
! Parameter !! Description !! Notes
server_password        = your_strong_password
|-
| <code>id_sensor</code> || Unique sensor identifier (1-65535) || '''Mandatory''' in distributed deployments
|-
| <code>cdr_partition</code> || Enable daily CDR table partitions || Enable on server writing to DB
|-
| <code>mysqlloadconfig</code> || Load config from database || Enable on central server only
|-
| <code>interface</code> || Capture interface || Use specific NIC or <code>any</code>
|}


mysqlhost              = 10.224.0.201
== Time Synchronization ==
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password


cdr_partition          = yes
{{Warning|1=Accurate NTP sync is '''critical''' for correlating call legs across sensors. All servers (GUI, DB, sensors) must run NTP client (chrony or ntpd).}}
mysqlloadconfig        = yes


# As this server does all analysis, configure as if sniffing locally:
== First Startup ==
sipport                = 5060
# ... add your usual sniffer/storage options (pcap directories, limits, etc.)
</syntaxhighlight>


==== Firewall Checklist (Quick Reference) ====
On first start against empty database:
* '''Modern Client/Server (v20+):'''
# Start service: <code>systemctl start voipmonitor</code>
** '''Central Server:''' Allow inbound <code>TCP/60024</code> from remote sensors. Allow inbound <code>TCP/5029</code> from GUI (management/API to central sensor).
# Monitor logs: <code>journalctl -u voipmonitor -f</code>
** '''Remote Sensors (Local Processing only):''' Allow inbound <code>TCP/5029</code> from the central server (for on-demand PCAP fetch via proxy). Outbound <code>TCP/60024</code> to the central server.
# Wait for schema/partition creation to complete
* '''Cloud Mode:'''
** '''Remote Sensors:''' Allow outbound <code>TCP/60023</code> to <code>cloud.voipmonitor.org</code>.


== Configuration & Checklists ==
If you see <code>Table 'cdr_next_1' doesn't exist</code> errors, check DB connectivity and privileges.


=== Parameter Notes (clarifications) ===
= Deployment Comparison =
* '''<code>id_sensor</code>''' — Mandatory in any distributed deployment (Classic or Client/Server). Must be unique per sensor (< 65536). The value is written to the database and used by the GUI to identify where a call was captured.
* '''<code>cdr_partition</code>''' — In Client/Server, enable on the central server instance that writes to the database. It can be disabled on remote "client" sensors that only mirror packets.
* '''<code>mysqlloadconfig</code>''' — When enabled, the sensor can load additional parameters dynamically from the <code>sensor_config</code> table in the database. Typically enabled on the central server sensor that writes to DB; keep disabled on remote clients which do not access DB directly.
* '''<code>interface</code>''' — Use a specific NIC (e.g., <code>eth0</code>) or <code>any</code> to capture from multiple NICs. For <code>any</code> ensure promiscuous mode on each NIC.


=== Initial Service Start & Database Initialization ===
After installation, the '''first startup''' against a new/empty database is critical.
# Start the service: <code>systemctl start voipmonitor</code>
# Follow logs to ensure schema/partition creation completes:
#* <code>journalctl -u voipmonitor -f</code>
#* or <code>tail -f /var/log/syslog | grep voipmonitor</code>
You should see creation of functions and partitions shortly after start. If you see errors like <code>Table 'cdr_next_1' doesn't exist</code>, the sensor is failing to initialize the schema — usually due to insufficient DB privileges or connectivity. Fix DB access and restart the sensor so it can finish initialization.
=== Time Synchronization ===
Accurate and synchronized time is '''critical''' for correlating call legs from different sensors. All servers (GUI, DB, and all Sensors) must run an NTP client (e.g., <code>chrony</code> or <code>ntpdate</code>) to keep clocks in sync.
== Comparison of Remote Deployment Modes ==
{| class="wikitable"
{| class="wikitable"
! Deployment Model
! Model !! Processing !! PCAP Storage !! WAN Traffic !! GUI Connectivity
! Packet Processing Location
! PCAP Storage Location
! Network Traffic to Central Server
! GUI Connectivity
|-
|-
| Classic Standalone
| Classic Standalone || Remote || Remote || Minimal (MySQL CDRs) || GUI ↔ each Sensor
| Remote
| Remote
| Minimal (MySQL CDRs)
| GUI ↔ each Sensor (management port)
|-
|-
| '''Modern Client/Server (Local Processing)'''
| '''Client/Server (Local Processing)''' || Remote || Remote || Minimal (encrypted CDRs) || '''GUI ↔ Central only'''
| Remote
| Remote
| Minimal (Encrypted CDRs)
| '''GUI ↔ Central Server only''' (central proxies PCAP fetch)
|-
|-
| '''Modern Client/Server (Packet Mirroring)'''
| '''Client/Server (Packet Mirroring)''' || Central || Central || High (encrypted packets) || '''GUI ↔ Central only'''
| '''Central'''
| '''Central'''
| High (Encrypted full packets)
| '''GUI ↔ Central Server only'''
|}
|}


== FAQ & Common Pitfalls ==
= Troubleshooting =
* '''Do remote sensors need DB credentials in Client/Server?''' No. Only the central server instance writes to DB.
* '''Why is <code>id_sensor</code> required everywhere?''' The GUI uses it to tag and filter calls by capture source.
* '''Local Processing still fetches PCAPs from remote — who connects to whom?''' The GUI requests via the central server; the central server then connects to the remote sensor's <code>TCP/5029</code> to retrieve the PCAP.
== AI Summary for RAG ==
'''Summary:''' This guide covers the deployment topologies for VoIPmonitor. It contrasts running the sensor on the same host as a PBX versus on a dedicated server. For dedicated sensors, it details methods for forwarding traffic, including hardware-based port mirroring (SPAN) and various software-based tunneling protocols (IP-in-IP, GRE, TZSP, VXLAN, HEP, AudioCodes, IPFIX). HEP (Homer Encapsulation Protocol) is a lightweight protocol for capturing and mirroring VoIP packets. When `hep = yes`, VoIPmonitor listens for HEPv3 (and compatible HEPv2) packets and extracts the original VoIP traffic from the encapsulation. CRITICAL HEP LIMITATION: VoIPmonitor does NOT use HEP correlation ID (captureNodeID) to correlate SIP and RTP packets. When SIP signaling and RTP media are encapsulated in HEP and arrive from different HEP sources (different capture nodes or sensors), VoIPmonitor cannot correlate them into a single CDR using HEP protocol metadata. This is feature request VS-1703 and there is currently no available workaround. The article covers cloud service packet mirroring options (GCP Packet Mirroring, AWS Traffic Mirroring, Azure Virtual Network TAP) with critical requirements: bidirectional capture (ingress and egress) and proper VM sizing (vCPU, RAM, storage I/O). The core of the article explains distributed architectures for multi-site monitoring, comparing the "classic" standalone remote sensor model with the modern, recommended "client/server" model. It details the two operational modes of the client/server architecture: local processing (sending only CDRs, PCAPs remain remote with central-proxied fetch) and packet mirroring (sending full, raw packets for central processing), which is ideal for low-resource endpoints. The article also explains an alternative approach for classic remote sensors: mounting PCAP spools via NFS or SSHFS when TCP/5029 access to sensors is blocked by firewalls, including troubleshooting steps for missing data due to NFS/SSHFS connectivity issues (checking logs for "not responding, timed out" errors, verifying network connectivity with ping/traceroute, and ensuring NFS/SSHFS server is running and accessible). The guide concludes with step-by-step configuration, firewall rules, critical parameter notes, and the importance of NTP plus first-start DB initialization.


'''Keywords:''' deployment, architecture, topology, on-host, dedicated sensor, port mirroring, SPAN, RSPAN, traffic mirroring, tunneling, GRE, TZSP, VXLAN, HEP, HEP correlation ID, captureNodeID, HEP limitation, HEP SIP RTP correlation, AudioCodes, IPFIX, cloud mirroring, GCP, AWS, Azure, Packet Mirroring, Traffic Mirroring, Virtual Network TAP, ingress, egress, bidirectional, VM sizing, remote sensor, multi-site, client server mode, packet mirroring, local processing, firewall rules, NTP, time synchronization, cloud mode, NFS, SSHFS, spooldir mounting, NFS troubleshooting, SSHFS troubleshooting, missing data, network connectivity
== NFS/SSHFS Connectivity ==


'''Key Questions:'''
Missing data for specific time periods usually indicates storage server connectivity issues.
* Can I use cloud packet mirroring (GCP/AWS/Azure) with VoIPmonitor?
* How should I configure cloud packet mirroring for ingress and egress traffic?
* What is the difference between the classic remote sensor and the modern client/server mode?
* When should I use packet mirroring (<code>packetbuffer_sender</code>) instead of local processing?
* What are the firewall requirements for the client/server deployment model?
* How can I access PCAP files from remote sensors if TCP/5029 is blocked?
* How do I configure NFS or SSHFS to mount remote PCAP spools?
* How do I configure the GUI sniffer data path for multiple mounted spools?
* How do I troubleshoot missing CDRs or PCAPs when using NFS or SSHFS mounts?
* What should I look for in logs to diagnose NFS connectivity issues?
* Can I run the sensor on the same machine as my Asterisk/FreeSWITCH server?
* What is a SPAN port and how is it used with VoIPmonitor?
* Why is NTP important for a distributed VoIPmonitor setup?
* What is HEP and how do I configure VoIPmonitor to receive HEP packets?
* Does VoIPmonitor use HEP correlation ID (captureNodeID) to correlate SIP and RTP packets?
* Can VoIPmonitor correlate SIP and RTP packets that arrive from different HEP sources?
* Is there a workaround for HEP SIP/RTP correlation across multiple HEP capture nodes?
* How do I configure GRE, ERSPAN, and VXLAN tunneling for VoIPmonitor?mory issues:


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
{| class="wikitable"
! Symptom !! Likely Cause !! Solution
|-
| Data gap in time period || NFS/SSHFS server unreachable || Check logs for "not responding, timed out"
|-
|-
! colspan="2" style="background:#ffc107;" | max_buffer_mem Guidance by Bottleneck Type
| Stale file handle || Server rebooted or export changed || Remount NFS share
|-
|-
| style="vertical-align: top;" | '''RAM-Based Memory Issue (Step 3)'''
| Connection resets || Network interruption || Check network stability
| Local server cannot process traffic fast enough.<br>'''Solution:''' INCREASE <code>max_buffer_mem</code> (e.g., 4000-8000 MB) to give more headroom
|-
|-
| style="vertical-align: top;" | '''Network Bottleneck (distributed mode)'''
| GUI shows "File not found" || Mount point dismounted || Verify mount with <code>mount | grep nfs</code>
| Probe cannot send packets to central server fast enough.<br>'''Solution:''' DECREASE <code>max_buffer_mem</code> (e.g., from 8000 to 2000 MB) so the buffer fails faster without exhausting RAM, enabling quicker recovery from network congestion.
|}
|}


Edit <code>/etc/voipmonitor.conf</code> on the probe:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="ini">
# Check for NFS errors
[general]
grep "nfs: server.*not responding" /var/log/syslog
# REDUCE max_buffer_mem for network throughput bottlenecks
grep "nfs.*timed out" /var/log/syslog
# This causes the buffer to fail sooner (releasing memory) instead of
# continuing to occupy RAM waiting for a network connection that cannot keep up
max_buffer_mem = 2000


# Enable compression to reduce network traffic
# Verify mount status
packetbuffer_compress = yes
mount | grep nfs
stat /mnt/voipmonitor/sensor1
</syntaxhighlight>
</syntaxhighlight>


Restart the probe service:
= See Also =
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
 
For detailed documentation on distributed architecture configuration, see [[Sniffer_distributed_architecture|Distributed Architecture: Client-Server Mode]].
 
=== Related Issues ===
 
For performance tuning and scaling guidance, see:
* [[Scaling|Scaling and Performance Tuning Guide]]
* [[IO_Measurement|I/O Performance Measurement]]
* [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual|High-Performance Setup]]
 
== Related Documentation ==


* [[Scaling|Scaling and Performance Tuning Guide]] - For performance optimization
* [[Sniffer_distributed_architecture|Distributed Architecture: Client-Server Mode]] - Detailed client/server configuration
* [[Anti-fraud|Anti-Fraud Rules]] - For attack detection and mitigation
* [[Sniffer_troubleshooting|Sniffer Troubleshooting]] - Diagnostic procedures
* [[Sniffer_troubleshooting|Sniffer Troubleshooting]] - For systematic diagnostic procedures
* [[Audiocodes_tunneling|AudioCodes Tunneling]] - AudioCodes SBC integration
* [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual|High-Performance Setup]] - For optimizing high-traffic deployments
* [[Tls|TLS/SRTP Decryption]] - Encrypted traffic monitoring
* [[Systemd_for_voipmonitor_service_management|Systemd Service Management]] - For service management best practices
* [[Cloud|Cloud Service Configuration]] - Cloud deployment specifics
* [[Scaling|Scaling and Performance Tuning]] - Performance optimization


== AI Summary for RAG ==
= AI Summary for RAG =


'''Summary:''' Emergency procedures for recovering VoIPmonitor from critical failures. Topics covered: (1) "Too high load" error - intentional protection feature, not a crash; assess CPU with htop per-core, compare Load Average to core count (LA < cores = acceptable). (2) Excessive CPU / runaway process - use kill -9 PID to terminate, root causes include SIP REGISTER floods, pcapcommand directive, RTP thread overload, audio features. (3) OOM memory exhaustion - check dmesg -T for killed processes, reduce innodb_buffer_pool_size and max_buffer_mem. (4) Service restart loop with "MEMORY IS FULL" and "Cannot bind to port 5029" - has 5 root causes: (a) Kernel storage errors (Step 0: dmesg -T), (b) Disk I/O bottleneck (Step 1: iostat -x 1, dd write test), (c) RAM exhaustion (Step 3: increase max_buffer_mem, packetbuffer_compress), (d) Adaptive jitterbuffer overload (Step 4: disable jitterbuffer_adapt=no), (e) Network throughput bottleneck in distributed mode (Step 6: switch packetbuffer_sender=no for Local Processing). Port 5029 binding error (Step 5): clear zombie processes with killall -9 voipmonitor. (5) Packet loss on sensors due to CPU overload - check journalctl for Load Average/heap, use sniffer_stat API for packets_dropped, add CPU cores or use temporary mitigation (callslimit, ringbuffer, silencedetect=no). (6) System hangs - collect core dump with gdb/gcore before restart for developer analysis.
'''Summary:''' VoIPmonitor deployment guide covering sensor placement (on-host vs dedicated), traffic forwarding methods (SPAN/RSPAN, software tunneling, cloud mirroring), and distributed architectures. Key traffic forwarding options: hardware port mirroring (physical/VMware switches), software tunnels (GRE, ERSPAN, TZSP, VXLAN, HEP, AudioCodes, IPFIX), and cloud provider services (GCP Packet Mirroring, AWS Traffic Mirroring, Azure Virtual Network TAP). CRITICAL HEP LIMITATION: VoIPmonitor does NOT use HEP correlation ID (captureNodeID) - SIP and RTP from different HEP sources will NOT be correlated (feature request VS-1703, no workaround). HEP3 packets with port 0 require adding port 0 to sipport directive. Cloud mirroring requires BIDIRECTIONAL capture (ingress+egress) or CDRs will be incomplete. Distributed architectures: Classic standalone (each sensor writes to central DB, GUI connects to each sensor) vs Modern Client/Server (recommended, encrypted TCP/60024 channel, GUI connects only to central server). Client/Server modes: Local Processing (packetbuffer_sender=no, CDRs only, PCAPs remain remote) vs Packet Mirroring (packetbuffer_sender=yes, full packets sent to central). Alternative for blocked TCP/5029: mount remote spools via NFS/SSHFS, configure multiple paths in GUI Sniffer data path setting. NFS troubleshooting: check for "not responding, timed out" in logs, verify mount status, use hard,nofail,tcp mount options. Critical requirement: NTP sync across all servers.


'''Keywords:''' emergency recovery, high CPU, system unresponsive, runaway process, kill -9, too high load, Live Sniffer, htop, load average, CPU cores, SIP REGISTER flood, pcapcommand, OOM, out of memory, dmesg, MEMORY IS FULL, HEAP FULL, packetbuffer, disk I/O, iostat, %util, dd oflag=direct, max_buffer_mem, innodb_buffer_pool_size, Cannot bind to port 5029, zombie process, restart loop, jitterbuffer_adapt, MOS_adaptive, NFS, nfsiostat, packetbuffer_sender, Local Processing, Packet Mirroring, distributed mode, network bottleneck, packets_dropped, sniffer_stat, ringbuffer, callslimit, core dump, gdb, gcore, iDRAC, iLO, IPMI
'''Keywords:''' deployment, topology, on-host, dedicated sensor, SPAN, RSPAN, port mirroring, VMware, vSwitch, dvSwitch, tunneling, GRE, ERSPAN, TZSP, VXLAN, HEP, HEP correlation ID, captureNodeID, VS-1703, HEP port 0, sipport, AudioCodes, IPFIX, cloud mirroring, GCP, AWS, Azure, Packet Mirroring, Traffic Mirroring, ingress, egress, bidirectional, client server, packetbuffer_sender, local processing, packet mirroring, TCP 60024, TCP 5029, NFS, SSHFS, sniffer data path, NTP, time synchronization, id_sensor, cdr_partition


'''Key Questions:'''
'''Key Questions:'''
* What does "too high load" error mean in Live Sniffer?
* Should I install VoIPmonitor on my PBX or use a dedicated sensor?
* How to interpret load average against CPU cores?
* How do I configure port mirroring (SPAN) for VoIPmonitor?
* What to do when VoIPmonitor consumes excessive CPU or becomes unresponsive?
* How do I configure VMware/ESXi virtual switch mirroring?
* How to forcefully terminate a runaway VoIPmonitor process?
* What software tunneling protocols does VoIPmonitor support?
* What causes SIP REGISTER flood CPU spikes and how to mitigate?
* How do I configure HEP (Homer Encapsulation Protocol)?
* How to diagnose OOM events with dmesg?
* Does VoIPmonitor use HEP correlation ID to correlate SIP and RTP?
* What causes "packetbuffer: MEMORY IS FULL" error?
* Why are SIP and RTP from different HEP sources not correlated?
* How to distinguish between RAM exhaustion, disk I/O bottleneck, and network bottleneck?
* How do I capture HEP3 packets with port 0?
* What is the first diagnostic step for MEMORY IS FULL errors (dmesg -T)?
* How do I configure cloud packet mirroring (GCP/AWS/Azure)?
* How to use iostat to diagnose disk I/O bottleneck?
* Why do I get incomplete CDRs with cloud mirroring?
* How to perform write speed test to spool directory?
* What is the difference between classic and client/server deployment?
* What does "Cannot bind to port 5029" error mean and how to fix?
* What is the difference between local processing and packet mirroring mode?
* How is jitterbuffer_adapt related to MEMORY IS FULL + HEAP FULL errors?
* How do I access PCAPs if TCP/5029 is blocked?
* What are the trade-offs when disabling jitterbuffer_adapt?
* How do I configure NFS/SSHFS for remote spool access?
* How to fix MEMORY IS FULL in distributed mode (packetbuffer_sender)?
* How do I troubleshoot missing data with NFS mounts?
* How to switch from Packet Mirroring to Local Processing mode?
* What firewall ports are required for client/server mode?
* How to check packet loss on sensors using sniffer_stat API?
* Why is NTP important for distributed VoIPmonitor?
* What temporary configurations reduce CPU load (callslimit, ringbuffer, silencedetect)?
* How to collect core dump for developer analysis during system hang?
* How to recover when binary is not found after crash?

Latest revision as of 16:49, 8 January 2026


This guide covers VoIPmonitor deployment options: where to install the sensor, how to forward traffic, and distributed architectures for multi-site monitoring.

Sensor Deployment Options

On-Host Capture

Install the sensor directly on the same Linux server as your PBX/SBC.

Pros Cons
No extra hardware, network changes, or port mirroring required Adds CPU, memory, and disk I/O load to production voice server
Simplest setup Not suitable if resources are critical

ℹ️ Note: VoIPmonitor sensor runs exclusively on Linux. For Windows-based PBXs (e.g., 3CX Windows edition), you must use a dedicated Linux sensor with traffic mirroring.

Dedicated Sensor

A separate Linux server runs only VoIPmonitor. Recommended for production environments as it isolates monitoring from voice platform resources.

When Required:

  • Windows-based PBXs
  • Limited CPU/RAM/disk I/O on PBX server
  • Zero monitoring impact needed
  • Centralized capture from multiple sites

Traffic Forwarding Methods

When using a dedicated sensor, you must forward traffic to it using one of these methods.

Hardware Port Mirroring (SPAN/RSPAN)

Physical or virtual switches copy traffic from source port(s) to a monitoring port.

Physical Switch

Configure your switch to mirror traffic from PBX/SBC ports to the sensor's port. Consult your switch documentation for specific commands. 

# /etc/voipmonitor.conf
interface = eth0
sipport = 5060
savertp = yes

💡 Tip:

VMware/ESXi Virtual Switch

For virtualized environments, VMware provides port mirroring at the virtual switch level.

Standard vSwitch:

  1. In vSphere Client, navigate to ESXi host
  2. Select virtual switch → Properties/Edit Settings → Enable Port Mirroring
  3. Set source (SBC VM) and destination (VoIPmonitor VM) ports

Distributed vSwitch:

  1. In vSphere Web Client → Networking → Select distributed switch
  2. Configure tab → Port Mirroring → Create mirroring session
  3. Specify source/destination ports and enable

ℹ️ Note: Distributed switch mirroring can span multiple ESXi hosts within a cluster.

Multiple VoIP Platforms

Monitor multiple platforms (e.g., Mitel + FreeSWITCH) with a single sensor by mirroring multiple source ports to one destination port.

GUI differentiation:

  • Filter by IP address ranges
  • Filter by number prefixes
  • Use separate sensors with unique id_sensor values

⚠️ Warning: Critical: When sniffing from multiple mirrored sources, packets may arrive as duplicates. Add auto_enable_use_blocks = yes to voipmonitor.conf to enable automatic deduplication. See Sniffer_configuration for details.

Software-based Tunneling

When hardware mirroring is unavailable, use software tunneling to encapsulate and forward packets.

Protocol Configuration Parameter Notes
IP-in-IP, GRE, ERSPAN Built-in (auto-detected) No additional config needed
TZSP (MikroTik) udp_port_tzsp = 37008
L2TP udp_port_l2tp = 1701
VXLAN udp_port_vxlan = 4789 Common in cloud environments
AudioCodes udp_port_audiocodes = 925 See AudioCodes Tunneling
IPFIX (Oracle SBCs) ipfix* options Enable ipfix options in config

HEP (Homer Encapsulation Protocol)

Lightweight protocol for mirroring VoIP packets. Supported by Kamailio, OpenSIPS, FreeSWITCH, and many SBCs. 

# /etc/voipmonitor.conf
hep = yes
hep_bind_port = 9060
hep_bind_udp = yes
# Optional: hep_kamailio_protocol_id_fix = yes

Known Limitations:

⚠️ Warning: HEP Correlation ID Not Supported: VoIPmonitor does NOT use HEP correlation ID (captureNodeID) to correlate SIP and RTP packets. If SIP and RTP arrive from different HEP sources, they will NOT be correlated into a single CDR.

VoIPmonitor correlates using standard SIP Call-ID, To/From tags, and RTP SSRC fields only. Feature request VS-1703 has been logged but there is no workaround currently.

HEP Timestamp: VoIPmonitor uses the HEP timestamp field. If the source has an unsynchronized clock, call timestamps will be incorrect. There is no option to ignore HEP timestamps.

HEP3 with Port 0: Not captured by default. Add port 0 to sipport: 

sipport = 0,5060

Cloud Packet Mirroring

Cloud providers offer native mirroring services using VXLAN or GRE encapsulation.

Provider Service Name
Google Cloud Packet Mirroring
AWS Traffic Mirroring
Azure Virtual Network TAP

Configuration Steps:

  1. Create a VoIPmonitor sensor VM in your cloud environment
  2. Create mirroring policy: select source VMs/subnets, set destination to sensor VM
  3. Critical: Capture traffic in BOTH directions (INGRESS and EGRESS)
  4. Configure sensor:
udp_port_vxlan = 4789
interface = eth0
sipport = 5060

⚠️ Warning: Capturing only ingress or only egress results in incomplete CDRs and broken call data.

Best Practices:

  • Filter at source to forward only SIP/RTP ports
  • Monitor NIC bandwidth limits
  • Account for VXLAN overhead (~50 bytes) - may need jumbo frames
  • Ensure NTP sync across all VMs

Alternative: Consider Client/Server architecture with on-host sensors instead of cloud mirroring for better performance.

Pre-Deployment Verification

For complex setups (RSPAN, ERSPAN, proprietary SBCs), verify compatibility before production deployment:

  1. Configure test mirroring with a subset of traffic
  2. Capture test calls with tcpdump: sudo tcpdump -i eth0 -s0 port 5060 -w /tmp/test.pcap
  3. Verify pcap contains SIP and RTP: tshark -r /tmp/test.pcap -Y "sip || rtp"
  4. Submit pcap to VoIPmonitor support with hardware/configuration details

Distributed Architectures

For multi-site monitoring, sensors can be deployed in various configurations.

Classic Mode: Standalone Sensors

Each sensor operates independently:

  • Processes packets and stores PCAPs locally
  • Connects directly to central MySQL to write CDRs
  • GUI needs network access to each sensor's TCP/5029 for PCAP retrieval

Alternative: NFS/SSHFS Mounting

If TCP/5029 access is blocked, mount remote spool directories on the GUI server: 

# NFS mount
sudo mount -t nfs 10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1

# SSHFS mount
sshfs voipmonitor@10.224.0.101:/var/spool/voipmonitor /mnt/voipmonitor/sensor1

Configure GUI: Settings > System Configuration > Sniffer data path: /var/spool/voipmonitor:/mnt/voipmonitor/sensor1:/mnt/voipmonitor/sensor2

💡 Tip: For NFS, use hard,nofail,tcp mount options for reliability.

Modern Mode: Client/Server (v20+) — Recommended

Secure encrypted TCP channel between remote sensors and central server. GUI communicates only with central server.

Mode Processing PCAP Storage WAN Traffic Best For
Local Processing (packetbuffer_sender=no) Remote Remote Low (CDRs only) Limited WAN bandwidth
Packet Mirroring (packetbuffer_sender=yes) Central Central High (full packets) Low-resource remote sites

For detailed configuration, see Distributed Architecture: Client-Server Mode.

Quick Start - Remote Sensor (Local Processing): 

id_sensor               = 2
server_destination      = 10.224.0.250
server_destination_port = 60024
server_password         = your_strong_password
packetbuffer_sender     = no
interface               = eth0
sipport                 = 5060
# No MySQL credentials needed - central server writes to DB

Quick Start - Central Server: 

server_bind             = 0.0.0.0
server_bind_port        = 60024
server_password         = your_strong_password
mysqlhost               = 10.224.0.201
mysqldb                 = voipmonitor
mysqluser               = voipmonitor
mysqlpassword           = db_password
cdr_partition           = yes
interface               =   # Leave empty - don't sniff locally

Firewall Requirements

Deployment Port Direction Purpose
Client/Server TCP/60024 Remote → Central Encrypted CDR/packet channel
Client/Server TCP/5029 Central → Remote On-demand PCAP fetch (Local Processing mode)
GUI Access TCP/5029 GUI → Central Management/API
Cloud Mode TCP/60023 Sensor → cloud.voipmonitor.org Cloud service connection

Configuration Notes

Critical Parameters

Parameter Description Notes
id_sensor Unique sensor identifier (1-65535) Mandatory in distributed deployments
cdr_partition Enable daily CDR table partitions Enable on server writing to DB
mysqlloadconfig Load config from database Enable on central server only
interface Capture interface Use specific NIC or any

Time Synchronization

⚠️ Warning: Accurate NTP sync is critical for correlating call legs across sensors. All servers (GUI, DB, sensors) must run NTP client (chrony or ntpd).

First Startup

On first start against empty database:

  1. Start service: systemctl start voipmonitor
  2. Monitor logs: journalctl -u voipmonitor -f
  3. Wait for schema/partition creation to complete

If you see Table 'cdr_next_1' doesn't exist errors, check DB connectivity and privileges.

Deployment Comparison

Model Processing PCAP Storage WAN Traffic GUI Connectivity
Classic Standalone Remote Remote Minimal (MySQL CDRs) GUI ↔ each Sensor
Client/Server (Local Processing) Remote Remote Minimal (encrypted CDRs) GUI ↔ Central only
Client/Server (Packet Mirroring) Central Central High (encrypted packets) GUI ↔ Central only

Troubleshooting

NFS/SSHFS Connectivity

Missing data for specific time periods usually indicates storage server connectivity issues.

Symptom Likely Cause Solution
Data gap in time period NFS/SSHFS server unreachable Check logs for "not responding, timed out"
Stale file handle Server rebooted or export changed Remount NFS share
Connection resets Network interruption Check network stability
GUI shows "File not found" Mount point dismounted grep nfs 
# Check for NFS errors
grep "nfs: server.*not responding" /var/log/syslog
grep "nfs.*timed out" /var/log/syslog

# Verify mount status
mount | grep nfs
stat /mnt/voipmonitor/sensor1

See Also

AI Summary for RAG

Summary: VoIPmonitor deployment guide covering sensor placement (on-host vs dedicated), traffic forwarding methods (SPAN/RSPAN, software tunneling, cloud mirroring), and distributed architectures. Key traffic forwarding options: hardware port mirroring (physical/VMware switches), software tunnels (GRE, ERSPAN, TZSP, VXLAN, HEP, AudioCodes, IPFIX), and cloud provider services (GCP Packet Mirroring, AWS Traffic Mirroring, Azure Virtual Network TAP). CRITICAL HEP LIMITATION: VoIPmonitor does NOT use HEP correlation ID (captureNodeID) - SIP and RTP from different HEP sources will NOT be correlated (feature request VS-1703, no workaround). HEP3 packets with port 0 require adding port 0 to sipport directive. Cloud mirroring requires BIDIRECTIONAL capture (ingress+egress) or CDRs will be incomplete. Distributed architectures: Classic standalone (each sensor writes to central DB, GUI connects to each sensor) vs Modern Client/Server (recommended, encrypted TCP/60024 channel, GUI connects only to central server). Client/Server modes: Local Processing (packetbuffer_sender=no, CDRs only, PCAPs remain remote) vs Packet Mirroring (packetbuffer_sender=yes, full packets sent to central). Alternative for blocked TCP/5029: mount remote spools via NFS/SSHFS, configure multiple paths in GUI Sniffer data path setting. NFS troubleshooting: check for "not responding, timed out" in logs, verify mount status, use hard,nofail,tcp mount options. Critical requirement: NTP sync across all servers.

Keywords: deployment, topology, on-host, dedicated sensor, SPAN, RSPAN, port mirroring, VMware, vSwitch, dvSwitch, tunneling, GRE, ERSPAN, TZSP, VXLAN, HEP, HEP correlation ID, captureNodeID, VS-1703, HEP port 0, sipport, AudioCodes, IPFIX, cloud mirroring, GCP, AWS, Azure, Packet Mirroring, Traffic Mirroring, ingress, egress, bidirectional, client server, packetbuffer_sender, local processing, packet mirroring, TCP 60024, TCP 5029, NFS, SSHFS, sniffer data path, NTP, time synchronization, id_sensor, cdr_partition

Key Questions:

  • Should I install VoIPmonitor on my PBX or use a dedicated sensor?
  • How do I configure port mirroring (SPAN) for VoIPmonitor?
  • How do I configure VMware/ESXi virtual switch mirroring?
  • What software tunneling protocols does VoIPmonitor support?
  • How do I configure HEP (Homer Encapsulation Protocol)?
  • Does VoIPmonitor use HEP correlation ID to correlate SIP and RTP?
  • Why are SIP and RTP from different HEP sources not correlated?
  • How do I capture HEP3 packets with port 0?
  • How do I configure cloud packet mirroring (GCP/AWS/Azure)?
  • Why do I get incomplete CDRs with cloud mirroring?
  • What is the difference between classic and client/server deployment?
  • What is the difference between local processing and packet mirroring mode?
  • How do I access PCAPs if TCP/5029 is blocked?
  • How do I configure NFS/SSHFS for remote spool access?
  • How do I troubleshoot missing data with NFS mounts?
  • What firewall ports are required for client/server mode?
  • Why is NTP important for distributed VoIPmonitor?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Sniffing_modes&oldid=10156"