Sniffer distributed architecture: Difference between revisions

From VoIPmonitor.org
(Add mirror_bind_sensor_id_by_sender configuration and separate CDRs documentation)
(Add HEP Protocol in Client/Server Mode documentation)
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Distributed Architecture: Client-Server Mode}}
{{DISPLAYTITLE:Distributed Architecture: Client-Server Mode}}


This guide explains how to deploy multiple VoIPmonitor sensors in a distributed architecture using the modern Client-Server mode.
This guide covers deploying multiple VoIPmonitor sensors in a distributed architecture using Client-Server mode (v20+).


== Overview ==
For deployment options including on-host vs dedicated sensors and traffic forwarding methods (SPAN, GRE, TZSP, VXLAN), see [[Sniffing_modes|VoIPmonitor Deployment & Topology Guide]].


VoIPmonitor v20+ uses a '''Client-Server architecture''' for distributed deployments. Remote sensors connect to a central server via encrypted TCP channel.
= Overview =
 
VoIPmonitor v20+ uses '''Client-Server architecture''' for distributed deployments. Remote sensors connect to a central server via encrypted TCP (default port 60024, zstd compression).


{| class="wikitable"
{| class="wikitable"
|-
|-
! Mode !! What is sent !! Processing location !! Use case
! Mode !! <code>packetbuffer_sender</code> !! What is Sent !! Processing Location !! Use Case
|-
| '''Local Processing''' || CDRs only || Remote sensor || Multiple sites, low bandwidth
|-
| '''Packet Mirroring''' || Raw packets || Central server || Centralized analysis, low-resource remotes
|}
 
The mode is controlled by a single option: <code>packetbuffer_sender</code>
 
For comprehensive deployment options including on-host vs dedicated sensors, traffic forwarding methods (SPAN, GRE, TZSP, VXLAN), and NFS/SSHFS alternatives, see [[Sniffing_modes|VoIPmonitor Deployment & Topology Guide]].
 
=== When to Use Client-Server Mode (Use Cases) ===
 
'''AWS VPC Traffic Mirroring Packet Loss'''
If you are experiencing packet loss when using AWS VPC Traffic Mirroring to capture traffic from source servers to a central VoIPmonitor instance, consider using client-server mode as an alternative architecture. AWS VPC Traffic Mirroring encapsulates packets using VXLAN (which adds ~50 bytes overhead), can saturate network links when mirroring from multiple sources, and may cause MTU/fragmentation issues.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | AWS VPC Traffic Mirroring vs Client-Server Mode
|-
|-
| style="vertical-align: top;" | '''AWS Traffic Mirroring Problems:'''
| '''Local Processing''' || <code>no</code> (default) || CDRs only || Remote sensor || Multi-site, low bandwidth
| * Packet loss due to VXLAN encapsulation overhead (MTU issues)<br>* Network link saturation when mirroring from multiple sources<br>* High sensor CPU from processing fragmented packets<br>* Packet loss during network interruptions (UDP-based mirroring)
|-
|-
| style="vertical-align: top;" | '''Client-Server Mode Solution:'''
| '''Packet Mirroring''' || <code>yes</code> || Raw packets || Central server || Centralized analysis, low-resource remotes
| * Install VoIPmonitor sniffer on each source server (client mode)<br>* Send captured packets via encrypted, zstd-compressed TCP to central server<br>* TCP ensures reliable packet delivery and buffering during network issues<br>* No VXLAN overhead, no MTU fragmentation issues<br>* Better control with ringbuffer on client sensors
|}
|}
'''Configuration for AWS VPC Traffic Mirroring Replacement:'''
On each source EC2 instance (e.g., Kamailio server):
<syntaxhighlight lang="ini">
# Install VoIPmonitor on each source server (client mode)
id_sensor              = 2                      # ID matches the one shown in GUI later
server_destination      = <central_voipmonitor_ip>
server_destination_port = 60024
server_password        = your_strong_password
packetbuffer_sender    = yes                    # Send raw packets to central server
# Capture settings (same as regular sensor)
interface              = eth0
sipport                = 5060
udp_port_vxlan          = 4789                  # Only needed if still receiving VXLAN traffic
</syntaxhighlight>
On the central server:
<syntaxhighlight lang="ini">
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = your_strong_password
packetbuffer_sender    = no                    # Receiving packets from clients
</syntaxhighlight>
This approach eliminates the AWS VPC Traffic Mirroring layer entirely, providing direct capture on each source server with reliable TCP delivery to the central instance.
=== Adding a New Sensor: Diagnostic Workflow ===
Before configuring a new sensor/VPS for your distributed deployment, follow this diagnostic workflow to ensure proper configuration:
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Diagnostic Workflow: Before Adding a New Sensor
|-
| style="vertical-align: top;" | '''Step 1: Generate Debug Log'''
| 1. Log in to the VoIPmonitor GUI on your central server.<br>2. Navigate to '''Tools → Generate debug log'''.<br>3. Provide an email address to receive the log link.<br>4. The debug log contains comprehensive system information including current configuration, sensor connections, and network settings.
|-
| style="vertical-align: top;" | '''Step 2: Compare with Working Sensor Configuration'''
| 1. SSH into a '''currently working sensor''' (one that is successfully sending data to the GUI).<br>2. View its configuration file: <code>cat /etc/voipmonitor.conf</code>.<br>3. Note the key parameters: <code>id_sensor</code>, <code>server_destination</code>, <code>server_destination_port</code>, <code>server_password</code>, <code>packetbuffer_sender</code>.<br>4. This ensures your new sensor uses the same deployment pattern (client-server mode, mirroring mode, or direct database connection).
|-
| style="vertical-align: top;" | '''Step 3: Verify Central Server is Accepting Connections'''
| 1. On the central server, check if the server is listening: <code>ss -tulpn | grep voipmonitor</code>.<br>2. Verify the port matches your expected configuration (default: 60024 for client-server mode).<br>3. Test connectivity from the new sensor location: <code>nc -zv <central_server_ip> <port></code>.
|}
This diagnostic approach prevents configuration mismatches and helps identify the correct deployment mode before making changes to your new sensor.
== Client-Server Mode ==
=== Architecture ===


<kroki lang="plantuml">
<kroki lang="plantuml">
Line 112: Line 43:
</kroki>
</kroki>


=== Configuration ===
== Use Cases ==
 
'''AWS VPC Traffic Mirroring Alternative:'''
If experiencing packet loss with AWS VPC Traffic Mirroring (VXLAN overhead, MTU fragmentation), use client-server mode instead:
* Install VoIPmonitor on each source EC2 instance
* Send via encrypted TCP to central server
* Eliminates VXLAN encapsulation and MTU issues
 
= Configuration =
 
== Remote Sensor (Client) ==


'''Remote Sensor (client):'''
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
id_sensor              = 2                    # unique per sensor
id_sensor              = 2                    # Unique per sensor (1-65535)
server_destination      = central.server.ip
server_destination      = central.server.ip
server_destination_port = 60024
server_destination_port = 60024
server_password        = your_strong_password
server_password        = your_strong_password


# Choose one:
# Choose mode:
packetbuffer_sender    = no    # Local Processing: analyze locally, send CDRs
packetbuffer_sender    = no    # Local Processing: analyze locally, send CDRs
# packetbuffer_sender  = yes    # Packet Mirroring: send raw packets
# packetbuffer_sender  = yes    # Packet Mirroring: send raw packets
Line 130: Line 70:
</syntaxhighlight>
</syntaxhighlight>


'''Important: Source IP Binding with <code>manager_ip</code>'''
{{Tip|1=For HA setups with floating IPs, use <code>manager_ip = 10.0.0.5</code> to bind outgoing connections to a static IP address.}}


For remote sensors with multiple IP addresses (e.g., in High Availability setups with a floating/virtual IP), use the <code>manager_ip</code> parameter to bind the outgoing connection to a specific static IP address. This ensures the central server sees a consistent source IP from each sensor, preventing connection issues during failover.
== Central Server ==


<syntaxhighlight lang="ini">
# On sensor with multiple interfaces (e.g., static IP + floating HA IP)
manager_ip              = 10.0.0.5    # Bind to the static IP address
server_destination      = 192.168.1.100
# The outgoing connection will use 10.0.0.5 as the source IP instead of the floating IP
</syntaxhighlight>
Useful scenarios:
* HA pairs: Sensors use static IPs while floating IP is only for failover management
* Multiple VNICs: Explicit source IP selection on systems with multiple virtual interfaces
* Network ACLs: Ensure connections originate from whitelisted IP addresses
'''Central Server:'''
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
server_bind            = 0.0.0.0
server_bind            = 0.0.0.0
Line 159: Line 86:
# If receiving raw packets (packetbuffer_sender=yes on clients):
# If receiving raw packets (packetbuffer_sender=yes on clients):
sipport                = 5060
sipport                = 5060
# ... other sniffer options
savertp                = yes
</syntaxhighlight>
savesip                = yes
 
=== Custom Port Configuration ===
 
'''Critical:''' The <code>server_bind_port</code> on the central server must match the <code>server_destination_port</code> on each remote sensor. If these ports do not match, sensors cannot connect.
 
<syntaxhighlight lang="ini">
# Central Server (listening on custom port 50291)
server_bind            = 0.0.0.0
server_bind_port        = 50291      # Custom port (default is 60024)
server_password        = your_strong_password
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="ini">
{{Warning|1='''Critical:''' Exclude <code>server_bind_port</code> from <code>sipport</code> on the central server. Including it causes continuously increasing memory usage.
# Remote Sensor (must match the server's custom port)
server_destination      = 45.249.9.2
server_destination_port = 50291      # MUST match server_bind_port
server_password        = your_strong_password
</syntaxhighlight>
 
'''Common reasons to use a custom port:'''
 
* Firewall restrictions that block the default port 60024
* Running multiple VoIPmonitor instances on the same server (each with a different port)
* Compliance requirements for non-standard ports
* Avoiding port conflicts with other services
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Critical: Exclude server_bind_port from sipport
|-
| style="vertical-align: top;" | '''The Issue:'''
| In client/server deployments, the <code>server_bind_port</code> (default 60024) used for sensor communication must be <strong>excluded</strong> from the <code>sipport</code> directive on the central server. If the sensor communication port is included in <code>sipport</code>, the sensor-to-server traffic itself is captured as SIP packets, causing high and continuously increasing memory utilization.
|-
| style="vertical-align: top;" | '''Symptoms:'''
| VoIPmonitor service exhibits high and increasing memory usage on the central server even when call volume is normal or low. The issue persists even after increasing <code>max_buffer_mem</code>.
|-
| style="vertical-align: top;" | '''The Fix:'''
| Configure <code>sipport</code> to exclude the <code>server_bind_port</code>. For example, if using the default <code>server_bind_port = 60024</code>:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# WRONG - includes sensor communication port:
# WRONG - includes sensor communication port:
sipport = 1-65535
sipport = 1-65535


# CORRECT - excludes sensor communication port:
# CORRECT - excludes port 60024:
sipport = 1-60023
sipport = 1-60023,60025-65535
sipport = 60025-65535</syntaxhighlight>
</syntaxhighlight>}}
|-
| style="vertical-align: top;" | '''Applies To:'''
| Central servers in both Local Processing (<code>packetbuffer_sender=no</code>) and Packet Mirroring (<code>packetbuffer_sender=yes</code>) modes. Remote sensors typically do not need this exclusion as they are clients (not servers).
|}


'''Troubleshooting Connection Failures:'''
== Key Configuration Rules ==


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
{| class="wikitable"
|-
|-
! colspan="2" style="background:#ffc107;" | Critical First Step: Check Traffic Rate Indicator
! Rule !! Applies To !! Why
|-
|-
| style="vertical-align: top;" | '''IMPORTANT:'''
| <code>server_bind_port</code> must match <code>server_destination_port</code> || Both || Connection fails if mismatched
| Before troubleshooting communication issues, check if the probe is receiving traffic. The traffic rate indicator in the sensor logs shows the current packet capture rate in the format <code>[x.xMb/s]</code> (e.g., <code>[12.5Mb/s]</code> or <code>[0.0Mb/s]</code>).
|-
|-
| style="vertical-align: top;" | '''How to check:'''
| <code>sipport</code> must match on probe and central server || Packet Mirroring || Missing ports = missing calls
| Run <code>journalctl -u voipmonitor -n 100</code> on the probe and look for the traffic rate indicator printed in the status logs.
|-
|-
| style="vertical-align: top;" | '''If showing <code>[0.0Mb/s]</code>:'''
| <code>natalias</code> only on central server || Packet Mirroring || Prevents RTP correlation issues
| The issue is NOT communication or authentication. The problem is network configuration on the probe side. Common causes: incorrect SPAN/mirror port setup on the switch, wrong network interface selected in <code>voipmonitor.conf</code>, or the probe is not receiving any traffic at all. Fix the network configuration first.
|-
|-
| style="vertical-align: top;" | '''If showing traffic (non-zero rate):'''
| Each sensor needs unique <code>id_sensor</code> || All || Required for identification
| The probe IS receiving traffic from the network, so the handshake issue is with communication/authentication. Proceed with the steps below.
|}
|}


If probes cannot connect to the server and the traffic rate indicator shows non-zero traffic:
= Local Processing vs Packet Mirroring =
 
1. '''Verify ports match on both sides:'''
  <syntaxhighlight lang="bash">
  # On central server - check which port it is listening on
  ss -tulpn | grep voipmonitor
  # Should show: voipmonitor LISTEN 0.0.0.0:50291
  </syntaxhighlight>
 
2. '''Test connectivity from remote sensor:'''
  <syntaxhighlight lang="bash">
  # Test TCP connection to the server's custom port
  nc -zv 45.249.9.2 50291
  # Success: "Connection to 45.249.9.2 50291 port [tcp/*] succeeded!"
  # Timeout/Refused: Check firewall or misconfigured port
  </syntaxhighlight>
 
3. '''Ensure firewall allows the custom port:'''
  <syntaxhighlight lang="bash">
  # Allow inbound TCP on custom port (example for firewalld)
  firewall-cmd --permanent --add-port=50291/tcp
  firewall-cmd --reload
  </syntaxhighlight>
 
4. '''Check logs on both sides:'''
  <syntaxhighlight lang="bash">
  journalctl -u voipmonitor -f
  # Look for: "connecting to server", "connection refused", or "timeout"
  </syntaxhighlight>
 
5. '''Verify MySQL database is accessible (if web GUI works but sensors cannot connect):'''
  If the web portal is accessible but sensors cannot connect, verify that the MySQL/MariaDB database service on the primary server is running and responsive. The central VoIPmonitor service requires a functioning database connection to accept sensor data.
  <syntaxhighlight lang="bash">
  # Check if MySQL service is running
  systemctl status mariadb
  # or
  systemctl status mysqld
 
  # Check for database errors in MySQL error log
  # Common locations:
  tail -50 /var/log/mariadb/mariadb.log
  tail -50 /var/log/mysql/error.log
  # Look for critical errors that might prevent database connections
  </syntaxhighlight>
 
If MySQL is down or experiencing critical errors, the central VoIPmonitor server may not be able to accept sensor connections even though the web interface (PHP) remains accessible. Restart the database service if needed and monitor the logs for recurring errors.
 
After changing port configuration, restart the service:
 
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
 
=== Checking Sensor Health Status via Management API ===
 
Each VoIPmonitor sensor exposes a TCP management API (default port 5029) that can be used to query its operational status and health. This is useful for monitoring multiple sensors, especially in distributed deployments.
 
'''Important Notes:'''
* There is NO single command to check all sensors simultaneously
* Each sensor must be queried individually
* The `sniffer_stat` command returns JSON with sensor status information
* In newer VoIPmonitor versions, the sensor's management API communication may be encrypted
 
==== Basic Health Check Command ====
 
To check the status of a single sensor:
 
<syntaxhighlight lang="bash">
# Query sensor status via management port
echo 'sniffer_stat' | nc <sensor_ip> <sensor_port>
</syntaxhighlight>
 
Replace:
* <code>&lt;sensor_ip&gt;</code> with the IP address of the sensor
* <code>&lt;sensor_port&gt;</code> with the management port (default: 5029)
 
==== Example Response ====
 
The command returns a JSON object with sensor status information:
 
<syntaxhighlight lang="json">
{
  "status": "running",
  "version": "30.3-SVN.123",
  "uptime": 86400,
  "calls_active": 42,
  "calls_total": 12345,
  "packets_per_second": 1250.5,
  "packets_dropped": 0
}
</syntaxhighlight>
 
==== Scripting Multiple Sensors ====
 
To check multiple sensors and get a consolidated result, create a script that queries each sensor individually:
 
<syntaxhighlight lang="bash">
#!/bin/bash
# Check health of multiple sensors
 
SENSORS=("192.168.1.10:5029" "192.168.1.11:5029" "192.168.1.12:5029")
ALL_OK=true
 
for SENSOR in "${SENSORS[@]}"; do
    IP=$(echo $SENSOR | cut -d: -f1)
    PORT=$(echo $SENSOR | cut -d: -f2)
 
    echo -n "Checking $IP:$PORT ... "
 
    # Query sensor and check for running status
    STATUS=$(echo 'sniffer_stat' | nc -w 2 $IP $PORT 2>/dev/null | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
 
    if [ "$STATUS" = "running" ]; then
        echo "OK"
    else
        echo "FAILED (status: $STATUS)"
        ALL_OK=false
    fi
done
 
if [ "$ALL_OK" = true ]; then
    echo "All sensors healthy"
    exit 0
else
    echo "One or more sensors unhealthy"
    exit 1
fi
</syntaxhighlight>
 
==== Troubleshooting Management API Access ====
 
If you cannot connect to the sensor management API:
 
1. '''Verify the management port is listening:'''
  <syntaxhighlight lang="bash">
  # On the sensor host
  netstat -tlnp | grep 5029
  # or
  ss -tlnp | grep voipmonitor
  </syntaxhighlight>
 
2. '''Check firewall rules:'''
  Ensure TCP port 5029 is allowed from the monitoring host to the sensor.
 
3. '''Test connectivity with netcat:'''
  <syntaxhighlight lang="bash">
  nc -zv <sensor_ip> 5029
  </syntaxhighlight>
 
4. '''Encrypted Communication (Newer Versions):'''
  In newer VoIPmonitor versions, the sensor's API communication may be encrypted. If management API access fails with encryption errors:
  * Check VoIPmonitor documentation for your version
  * Encryption may need to be disabled for management API access
  * Consult support for encrypted CLI tools if available
 
==== Encryption Considerations ====
 
If your sensors use encrypted management API (newer versions):
 
* The standard netcat command may not work with encrypted connections
* Check if `manager_bind` (default port 5029) has encryption enabled
* For encrypted connections, you may need VoIPmonitor-specific CLI tools
* Refer to your VoIPmonitor version documentation or contact support for encrypted API access
 
=== Connection Compression ===
 
The client-server channel supports compression to reduce bandwidth usage:
 
<syntaxhighlight lang="ini">
# On both client and server (default: zstd)
server_type_compress = zstd
</syntaxhighlight>
 
Available options: <code>zstd</code> (default, recommended), <code>gzip</code>, <code>lzo</code>, <code>none</code>
 
=== High Availability (Failover) ===
 
Remote sensors can specify multiple central server IPs for automatic failover:
 
<syntaxhighlight lang="ini">
# Remote sensor configuration with failover
server_destination = 192.168.0.1, 192.168.0.2
</syntaxhighlight>
 
If the primary server becomes unavailable, the sensor automatically connects to the next server in the list.
 
== Local Processing vs Packet Mirroring ==


{| class="wikitable"
{| class="wikitable"
Line 424: Line 122:
| '''<code>packetbuffer_sender</code>''' || <code>no</code> (default) || <code>yes</code>
| '''<code>packetbuffer_sender</code>''' || <code>no</code> (default) || <code>yes</code>
|-
|-
| '''Packet analysis''' || On remote sensor || On central server
| '''Processing location''' || Remote sensor || Central server
|-
|-
| '''PCAP storage''' || On remote sensor || On central server
| '''PCAP storage''' || Remote sensor || Central server
|-
|-
| '''WAN bandwidth''' || Low (CDRs only) || High (full packets)
| '''WAN bandwidth''' || Low (CDRs only, 1Gb sufficient) || High (full packets)
|-
|-
| '''Remote CPU load''' || Higher || Minimal
| '''Remote CPU load''' || Higher || Minimal
|-
|-
| '''Use case''' || Standard multi-site || Low-resource remotes
| '''Capture rules applied''' || On sensor || On central server only
|}
|}


=== Network Bandwidth Requirements ===
== PCAP Access in Local Processing Mode ==


The network bandwidth requirements between remote sensors and the central server depend on the selected operational mode:
PCAPs are stored on remote sensors. The GUI retrieves them through the central server, which proxies the request to the sensor '''over the existing TCP/60024 connection''' - the same persistent encrypted channel the sensor uses for sending CDRs. This connection is bidirectional; the central server does not open any separate connection back to the sensor.
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | Bandwidth Guidelines
|-
| style="vertical-align: top;" | '''Local Processing Mode (<code>packetbuffer_sender=no</code>):'''
| PCAP files are stored locally on sensors. Network traffic consists mainly of CDR data (SQL queries). <br><br>
'''A 1Gb network connection between sensors and the central GUI/Database server is generally sufficient for most deployments.'''
|-
| style="vertical-align: top;" | '''Packet Mirroring Mode (<code>packetbuffer_sender=yes</code>):'''
| Raw packet stream is forwarded to central server. Bandwidth consumption is roughly equivalent to the VoIP traffic volume itself (minus Ethernet headers, plus compression overhead). <br><br>
Consider your expected VoIP traffic volume and network capacity. Use <code>server_type_compress=zstd</code> to reduce bandwidth usage.
|}
 
For optimal throughput in high-latency environments, see the server concatenation limit configuration in [[Sniffer_configuration#SQL_Concatenation_Throughput_Tuning|Sniffer Configuration: SQL Concatenation Throughput]].
 
=== PCAP Access in Local Processing Mode ===
 
When using Local Processing, PCAPs are stored on remote sensors. The GUI retrieves them via the central server, which proxies requests to each sensor's management port (TCP/5029).


'''Firewall requirements:'''
'''Firewall requirements:'''
* Central server must reach remote sensors on TCP/5029
* Remote sensors must reach central server on TCP/60024
== Dashboard Statistics ==
Dashboard widgets (SIP/RTP/REGISTER counts) depend on where packet processing occurs:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Configuration !! Where statistics appear
! Direction !! Port !! Purpose
|-
|-
| '''<code>packetbuffer_sender = yes</code>''' (Packet Mirroring) || Central server only
| Remote sensors → Central server || TCP/60024 || Persistent encrypted channel (CDRs from sensor, PCAP requests from server - bidirectional)
|-
|-
| '''<code>packetbuffer_sender = no</code>''' (Local Processing) || Both sensor and central server
| GUI → Central server || TCP/5029 || Manager API (sensor status, active calls, configuration)
|-
| GUI → Central server || TCP/60024 || Server API (list connected sensors, proxy PCAP retrieval)
|}
|}


'''Note:''' If you are using Packet Mirroring mode (<code>packetbuffer_sender=yes</code>) and see empty dashboard widgets for the forwarding sensor, this is expected behavior. The sender sensor only captures and forwards raw packets - it does not create database records or statistics. The central server performs all processing.
{{Note|1=The central server does '''not''' initiate connections to remote sensors. All server↔sensor communication happens over the single TCP/60024 connection that the sensor established.}}


=== Enabling Local Statistics on Forwarding Sensors ===
{{Tip|1=Packet Mirroring (<code>packetbuffer_sender=yes</code>) '''automatically deduplicates calls''' - the central server merges packets from all probes for the same Call-ID into a single unified CDR. This also ensures one logical call only consumes one license channel.}}
= Advanced Topics =


If you need local statistics on a sensor that was previously configured to forward packets:
== High Availability (Failover) ==
 
Remote sensors can specify multiple central servers:


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# On the forwarding sensor
server_destination = 192.168.0.1, 192.168.0.2
packetbuffer_sender = no
</syntaxhighlight>
</syntaxhighlight>


This disables packet forwarding and enables full local processing. Note that this increases CPU and RAM usage on the sensor since it must perform full SIP/RTP analysis.
If primary is unavailable, the sensor automatically connects to the next server.
 
== Controlling Packet Storage in Packet Mirroring Mode ==


When using Packet Mirroring (<code>packetbuffer_sender=yes</code>), the central server processes raw packets received from sensors. The <code>save*</code> options on the '''central server''' control which packets are saved to disk.
== Connection Compression ==


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Central Server Configuration (receiving raw packets from sensors)
# On both client and server (default: zstd)
server_bind            = 0.0.0.0
server_type_compress = zstd  # Options: zstd, gzip, lzo, none
server_bind_port        = 60024
server_password        = your_strong_password
 
# Database Configuration
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
 
# Sniffer options needed when receiving raw packets:
sipport                = 5060
 
# CONTROL PACKET STORAGE HERE:
# These settings on the central server determine what gets saved:
savertp                = yes          # Save RTP packets
savesip                = yes          # Save SIP packets
saveaudio              = wav          # Export audio recordings (optional)
</syntaxhighlight>
</syntaxhighlight>


{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
== Intermediate Server (Hub-and-Spoke) ==
|-
! colspan="2" style="background:#4A90E2; color: white;" | Important: Central Server Controls Storage
|-
| style="vertical-align: top;" | '''Key Point:'''
| When sensors send raw packets to a central server, the storage is controlled by the <code>savertp</code>, <code>savesip</code>, and <code>saveaudio</code> options configured on the '''central server''', not on the individual sensors. The sensors are only forwarding raw packets - they do not make decisions about what to save unless you are using Local Processing mode.
|}


This centralized control allows you to:
An intermediate server can receive from multiple sensors and forward to a central server:
* Enable/disable packet types (RTP, SIP, audio) from one location
* Adjust storage settings without touching each sensor
* Apply capture rules from the central server to filter traffic


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
<kroki lang="plantuml">
|-
@startuml
! colspan="2" style="background:#ffc107;" | Critical: Capture Rules are Applied on Central Server Only
skinparam shadowing false
|-
skinparam defaultFontName Arial
| style="vertical-align: top;" | '''Packet Mirroring Mode (<code>packetbuffer_sender=yes</code>):'''
| In this mode, the remote sensor operates in "re-sending mode only" - it forwards raw packets without analyzing them. '''Capture rules MUST be configured on the Central Server only.''' Capture rules configured on the remote sensor are ignored because the sensor does not perform packet analysis. All packet processing, including capture rule evaluation, happens on the destination server.
|-
| style="vertical-align: top;" | '''Local Processing Mode (<code>packetbuffer_sender=no</code>):'''
| The remote sensor analyzes SIP/RTP locally and applies capture rules configured on the central server database (assuming <code>mysqlloadconfig = yes</code> is set on the sensor).
|}


== Data Storage Summary ==
rectangle "Remote Sensors" as RS
rectangle "Intermediate Server" as INT
rectangle "Central Server" as CS
database "MySQL" as DB


* '''CDRs''': Always stored in MySQL on central server
RS --> INT : TCP/60024
* '''PCAPs''':
INT --> CS : TCP/60024
*: Local Processing → stored on each remote sensor
CS --> DB
*: Packet Mirroring → stored on central server
@enduml
</kroki>


== Handling Same Call-ID from Multiple Sensors ==
<syntaxhighlight lang="ini">
# On INTERMEDIATE SERVER
id_sensor              = 100


When a call passes through multiple sensors that see the same SIP Call-ID, VoIPmonitor automatically merges the SIP packets into a single CDR on the central server. This is expected behavior when using Packet Mirroring mode.
# Receive from remote sensors
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = sensor_password


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
# Forward to central server
|-
server_destination      = central.server.ip
! colspan="2" style="background:#ffc107;" | Call-ID Merging Behavior
server_destination_port = 60024
|-
| style="vertical-align: top;" | '''What happens:'''
| If Sensor A and Sensor B both forward packets for a call with the same Call-ID to the central server, VoIPmonitor creates a single CDR containing SIP packets from both sensors. The RTP packets are captured from whichever sensor processed the media.
|-
| style="vertical-align: top;" | '''Why:'''
| VoIPmonitor uses the SIP Call-ID as the primary unique identifier. When multiple sensors forward packets with the same Call-ID to a central server, they are automatically treated as one call.
|-
| style="vertical-align: top;" | '''Is it a problem?'''
| Usually not. For most deployments, combining records from multiple sensors for the same call (different call legs passing through different points in the network) is the desired behavior.
|}


=== Preventing Duplicate CDRs in Local Processing Mode ===
packetbuffer_sender    = no    # or yes, depending on desired mode
</syntaxhighlight>


When using '''Local Processing mode''' (<code>packetbuffer_sender=no</code>), each remote probe processes its own packets and writes CDRs directly to a central database. If multiple probes capture the same call (e.g., redundant taps or overlapping SPAN ports), this creates '''duplicate CDR entries''' in the database.
{{Note|1=This works because the intermediate server does NOT do local packet capture - it only relays. Original remote sensors must be manually added to GUI Settings for visibility.}}


To prevent duplicates in this scenario, use the <code>cdr_check_exists_callid</code> option on '''all probes''':
== Multiple Receivers for Packet Mirroring ==


{| class="wikitable" style="background:#f8f9fa; border:1px solid #dee2e6;"
{{Warning|1=Multiple sensors with <code>packetbuffer_sender=yes</code> sending to a '''single receiver instance''' can cause call processing conflicts (calls appear in Active Calls but missing from CDRs).}}
|-
! rowspan="2" | Setting
! colspan="2" | Result
|-
|<code>cdr_check_exists_callid = no</code> (default)
|Each probe creates its own CDR row. Multiple probes capturing the same call result in duplicate entries with the same Call-ID but different id_sensor values.
|-
|<code>cdr_check_exists_callid = yes</code>
|Probes check for an existing CDR with the same Call-ID before inserting. If found, they update the existing row instead of creating a new one. The final CDR will be associated with the id_sensor of the probe that last processed the call.
|}


'''Prerequisites:'''
'''Solution:''' Run separate receiver instances on different hosts, each dedicated to specific sensors:
* MySQL user must have <code>UPDATE</code> privileges on the <code>cdr</code> table
* All probes must be configured with this setting


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Add to voipmonitor.conf on each probe (Local Processing mode only)
# Receiver Instance 1 (Host 1, for Sensor A)
cdr_check_exists_callid = yes
server_bind_port        = 60024
</syntaxhighlight>
id_sensor              = 1
 
'''Note:''' This setting is only useful in Local Processing mode. In Packet Mirroring mode (<code>packetbuffer_sender=yes</code>), the central server automatically merges packets with the same Call-ID, so this option is not needed.
 
=== Keeping Records Separate Per Sensor ===


If you need to keep records completely separate when multiple sensors see the same Call-ID (e.g., each sensor should create its own independent CDR even for calls with overlapping Call-IDs), you must run '''multiple receiver instances on the central server'''.
# Receiver Instance 2 (Host 2, for Sensor B)
 
<syntaxhighlight lang="ini">
# Receiver Instance 1 (for Sensor A)
[receiver_sensor_a]
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_bind_port        = 60024
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = <password>
mysqltableprefix        = sensor_a_  # Separate CDR tables
id_sensor              = 2
id_sensor              = 2
# ... other options
# Receiver Instance 2 (for Sensor B)
[receiver_sensor_b]
server_bind            = 0.0.0.0
server_bind_port        = 60025  # Different port
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = <password>
mysqltableprefix        = sensor_b_  # Separate CDR tables
id_sensor              = 3
# ... other options
</syntaxhighlight>
</syntaxhighlight>


Each receiver instance runs as a separate process, listens on a different port, and can write to separate database tables (using <code>mysqltableprefix</code>). Configure each sensor to connect to its dedicated receiver port.
Alternative: Use '''Local Processing mode''' (<code>packetbuffer_sender=no</code>) which processes calls independently on each sensor.
 
For more details on correlating multiple call legs from the same call, see [[Merging_or_correlating_multiple_call_legs]].
 
== GUI Visibility ==
 
Remote sensors appear automatically when connected. To customize names or configure additional settings:
# Go to '''GUI → Settings → Sensors'''
# Sensors are identified by their <code>id_sensor</code> value
 
== Troubleshooting Distributed Deployments ==
 
=== Probe Not Detecting All Calls on Expected Ports ===
 
If a remote sensor (probe) configured for packet mirroring is not detecting all calls on expected ports, check configuration on '''both''' the probe and the central analysis host.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Critical: sipport Must Match in Distributed Deployments
|-
| style="vertical-align: top;" | '''The Issue:'''
| In distributed/probe setups using Packet Mirroring (<code>packetbuffer_sender=yes</code>), calls will be missing if the <code>sipport</code> configuration is not aligned between the probe and central server. Common symptom: Probe sees traffic via <code>tcpdump</code> but central server records incomplete CDRs.
|-
| style="vertical-align: top;" | '''Configuration Requirement:'''
| The probe and central host must have consistent <code>sipport</code> values. If your network uses SIP on multiple ports (e.g., 5060, 5061, 5080, 6060), ALL ports must be listed on both systems.
|}


The solution involves three steps:
== Preventing Duplicate CDRs (Local Processing) ==


;1. Verify traffic reachability on the probe:
When multiple probes capture the same call in Local Processing mode:
Use <code>tcpdump</code> on the probe VM to confirm SIP packets for the missing calls are arriving on the expected ports.
<syntaxhighlight lang="bash">
# On the probe VM
tcpdump -i eth0 -n port 5060
</syntaxhighlight>


;2. Check the probe's ''voipmonitor.conf'':
Ensure the <code>sipport</code> directive on the probe includes all necessary SIP ports used in your network.
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the PROBE
# On each probe
sipport = 5060,5061,5080,6060
cdr_check_exists_callid = yes
</syntaxhighlight>
</syntaxhighlight>


;3. Check the central analysis host's ''voipmonitor.conf'':
This checks for existing CDRs before inserting. Requires MySQL UPDATE privileges.
'''This is the most common cause of missing calls in distributed setups.''' The central analysis host (specified by <code>server_bind</code> on the central server, or by <code>server_destination</code> configured on the probe) must also have the <code>sipport</code> directive configured with the same list of ports used by all probes.
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the CENTRAL HOST
sipport = 5060,5061,5080,6060
</syntaxhighlight>


;4. Restart both services:
== Critical: SIP and RTP Must Be Captured Together ==
Apply the configuration changes:
<syntaxhighlight lang="bash">
# On both probe and central host
systemctl restart voipmonitor
</syntaxhighlight>


{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
VoIPmonitor cannot correlate SIP and RTP from different sniffer instances. A '''single sniffer must process both SIP and RTP''' for each call. Parameters like <code>cdr_check_exists_callid</code> do NOT enable split SIP/RTP correlation.
|-
! colspan="2" style="background:#4A90E2; color: white;" | Why Both Systems Must Match
|-
| style="vertical-align: top;" | '''Probe side:'''
| The probe captures packets from the network interface. Its <code>sipport</code> setting determines which UDP ports it considers as SIP traffic to capture and forward.
|-
| style="vertical-align: top;" | '''Central server side:'''
| When receiving raw packets in Packet Mirroring mode, the central server analyzes the packets locally. Its <code>sipport</code> setting determines which ports it interprets as SIP during analysis. If a port is missing here, packets are captured but not recognized as SIP, resulting in missing CDRs.
|}


=== Quick Diagnosis Commands ===


On the probe:
<syntaxhighlight lang="bash">
# Check which sipport values are configured
grep -E "^sipport" /etc/voipmonitor.conf


# Verify traffic is arriving on expected ports
==== Split SIP/RTP with Packet Mirroring Mode ====
tcpdump -i eth0 -nn -c 10 port 5061
</syntaxhighlight>


On the central server:
{{Note|1='''Exception for Packet Mirroring Mode:''': The above limitation applies to '''Local Processing mode''' (<code>packetbuffer_sender=no</code>) where each sensor processes calls independently. In '''Packet Mirroring mode''' (<code>packetbuffer_sender=yes</code>), the central server receives raw packets from multiple remote sensors and processes them together. This allows scenarios where SIP and RTP are captured on separate nodes - configure both as packet senders and let the central server correlate them into single unified CDRs.}}
<syntaxhighlight lang="bash">
# Check which sipport values are configured
grep -E "^sipport" /etc/voipmonitor.conf


# Check syslog for analysis activity (should see processing packets)
Example scenario: Separate SIP signaling node and RTP handling node:
tail -f /var/log/syslog | grep voipmonitor
<syntaxhighlight lang="ini">
</syntaxhighlight>
# SIP Signaling Node (packet sender)
id_sensor              = 1
packetbuffer_sender    = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password        = your_password


If probes still miss calls after ensuring <code>sipport</code> matches on both sides, check the [[Sniffer_troubleshooting|full troubleshooting guide]] for other potential issues such as network connectivity, firewall rules, or interface misconfiguration.
# RTP Handling Node (packet sender)
 
id_sensor              = 2
=== Post-Migration Sensor Connection Issues ===
packetbuffer_sender    = yes
 
server_destination      = central.server.ip
If a sensor becomes unreachable with a '''"Connection refused (111)"''' error after migrating the VoIPmonitor system (e.g., after moving to new hardware, changing IP addresses, or relocating servers), the issue is almost always that the sensor configuration still points to the '''old server address'''.
server_destination_port = 60024
 
server_password        = your_password
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Common Migration Issue: Connection Refused (111) After Server Move
|-
| style="vertical-align: top;" | '''The Issue:'''
| After migrating your VoIPmonitor infrastructure (moving central server to new hardware or IP address), remote sensors fail to connect with <code>Connection refused (111)</code>. The POSIX error <code>ECONNREFUSED</code> means the sensor successfully reached an IP address and port, but no VoIPmonitor service was listening there - usually because the sensor is trying to connect to the '''old server IP address'''.
|-
| style="vertical-align: top;" | '''Root Cause:'''
| The <code>server_destination</code> parameter in the sensor's <code>voipmonitor.conf</code> file contains the old IP address/hostname of the central server that no longer exists or is not running the VoIPmonitor service.
|-
| style="vertical-align: top;" | '''Resolution:'''
| Update the <code>server_destination</code> configuration on each affected sensor to point to the new central server address.
|}
 
'''Diagnosing the Problem'''
 
;1. Check sensor logs for connection errors:
<syntaxhighlight lang="bash">
# On the sensor machine
journalctl -u voipmonitor -n 100 | grep -i "connection refused\|failed to connect"
</syntaxhighlight>
</syntaxhighlight>


Look for messages attempting to connect to the old server IP.
The central server merges packets from both senders by Call-ID, creating unified CDRs with complete SIP and RTP data.


;2. Compare configurations between working and failing sensors:
<syntaxhighlight lang="bash">
# On a WORKING sensor
grep "server_destination" /etc/voipmonitor.conf


# On the FAILING sensor
==== HEP Protocol in Client/Server Mode ====
grep "server_destination" /etc/voipmonitor.conf
</syntaxhighlight>


The failing sensor will likely show an old IP address that is no longer valid.
VoIPmonitor supports receiving HEP-encapsulated traffic on sniffer clients and forwarding it to a central server. This enables distributed capture from HEP sources (Kamailio, OpenSIPS, rtpproxy, FreeSWITCH) in a client/server architecture.


;3. Verify the central server is reachable from the sensor:
'''Scenario:''' SIP proxy and RTP proxy at different locations sending HEP to remote sniffer clients:
<syntaxhighlight lang="bash">
# Replace <new_server_ip> with the actual server IP
nc -zv <new_server_ip> 60024
# Should show: "Connection to <new_server_ip> 60024 port [tcp/*] succeeded!"
</syntaxhighlight>


'''Fixing the Configuration'''
;1. Update <code>server_destination</code> on the failing sensor:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on the Failing sensor
# Remote Sniffer Client A (receives HEP from Kamailio)
id_sensor              = 1
hep                    = yes
hep_bind_port          = 9060
packetbuffer_sender    = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password        = your_password


# OLD - points to old server that no longer exists:
# Remote Sniffer Client B (receives HEP from rtpproxy)
# server_destination = 192.168.1.50
id_sensor              = 2
 
hep                    = yes
# NEW - correct server address:
hep_bind_port          = 9060
server_destination = 192.168.1.100
packetbuffer_sender    = yes
server_destination     = central.server.ip
server_destination_port = 60024
server_destination_port = 60024
server_password = your_strong_password
server_password         = your_password
</syntaxhighlight>
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | Critical: server_destination vs Database Settings
|-
| style="vertical-align: top;" | '''<code>server_destination</code>'''
| Controls client-server communication between sensor and central server via the proprietary protocol (default TCP port 60024). This is for sensors in distributed deployments using <code>packetbuffer_sender</code>.
|-
| style="vertical-align: top;" | '''<code>mysqlhost</code> / <code>mysql_host</code>'''
| Controls direct MySQL database connection for storing CDRs. Used in Local Processing mode or standalone sensor configurations.
|-
| style="vertical-align: top;" | '''Which to check:'''
| If your sensor uses <code>server_destination</code> (client/server mode), update this value after migration. If your sensor uses <code>mysqlhost</code> (direct DB mode), update the database host value instead.
|}
 
;2. Restart the sensor service:
<syntaxhighlight lang="bash">
# On the sensor machine
systemctl restart voipmonitor
</syntaxhighlight>
 
;3. Verify connection established:
<syntaxhighlight lang="bash">
# Check sensor logs for successful connection
journalctl -u voipmonitor -n 20 | grep -i "connected"
 
# Check GUI Settings → Sensors
# The sensor should now appear as "Connected" or "Online"
</syntaxhighlight>
</syntaxhighlight>


'''Preventing Migration Issues'''
The central server receives packets from both clients and correlates them into unified CDRs using standard SIP Call-ID and IP:port from SDP.
 
When planning a VoIPmonitor infrastructure migration involving server IP changes:
 
* Maintain an inventory of all <code>server_destination</code> values across your sensor fleet
* Test connectivity from sensors to the new server before updating configurations
* Update sensors in batches to avoid simultaneous outages across your entire deployment
* Verify each sensor appears as "Connected" in the GUI Settings → Sensors before moving to the next batch
* Document the new central server address(es) in your configuration management system


For sensors using direct database connections (<code>mysqlhost</code>), ensure MySQL user privileges allow connections from the new sensor IP addresses and update firewall rules accordingly.
{{Note|1=This also works for IPFIX (Oracle SBCs) and RibbonSBC protocols forwarded via client/server mode.}}


=== Sensor Registration Errors: Stale Database Records ===
'''Alternative: Direct HEP to single sniffer'''


If a new sensor fails to connect to the main server with the error '''"failed response from server - bad password"''' even when the <code>server_password</code> is correctly configured in both files, the issue may be a stale sensor record in the GUI database.
If both HEP sources can reach the same sniffer directly, no client/server setup is needed:
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Important: Stale Sensor Records Cause Authentication Failures
|-
| style="vertical-align: top;" | '''The Issue:'''
| The GUI database retains a record of a previously-deployed sensor. This stale record can prevent a new sensor with the same <code>id_sensor</code> from authenticating correctly, even when the password matches exactly. The error is misleading; the problem is not a password mismatch but a database conflict.
|-
| style="vertical-align: top;" | '''Common Scenarios:'''
| * Replacing a failed sensor with new hardware<br>* Reinstalling or reconfiguring a sensor<br>* Changing sensor IDs and back<br>* Restoring from backups where sensor records are out of sync
|}
 
'''Resolution: Delete the Stale Sensor Record and Re-register'''
 
;1. Delete the problematic sensor record from the GUI:
Navigate to '''GUI → Settings → Sensors''' and delete the sensor entry that is causing issues. Click the delete icon (typically a trash can or × icon) next to the sensor record.
 
;2. Restart the voipmonitor service on the '''sensor/probe machine''':
<syntaxhighlight lang="bash">
# On the agent/probe machine only
systemctl restart voipmonitor
</syntaxhighlight>
 
;3. Verify network connectivity from the sensor to the server:
Test that the server is reachable on the configured port:
<syntaxhighlight lang="bash">
# From the sensor, test connectivity to the central server
# Replace <server_ip> with the actual server IP
# Replace 60024 with your server_bind_port if not using the default
telnet <server_ip> 60024
</syntaxhighlight>
If <code>telnet</code> shows "Connected to <server_ip>", the server is reachable. If it shows "Connection refused" or times out, check firewall rules and ensure the server service is running:
<syntaxhighlight lang="bash">
# On the central server, check if voipmonitor is listening
ss -tulpn | grep 60024
# or
netstat -tlnp | grep voipmonitor
</syntaxhighlight>
 
;4. Verify automatic re-registration:
After the service restart, the sensor automatically registers with the central server and appears as a new entry in '''GUI &rarr; Settings &rarr; Sensors'''. The connection status should show as "Connected" or "Online".
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | How Sensor Registration Works
|-
| style="vertical-align: top;" | '''Automatic Process:'''
| Sensors automatically register with the central server when they start. The sensor sends a handshake packet with its <code>id_sensor</code> and authentication credentials. The central server checks:
* If a sensor with this <code>id_sensor</code> exists in the database, it validates the credentials
* If no such sensor exists, the server creates a database record automatically
|-
| style="vertical-align: top;" | '''Why Stale Records Matter:'''
| When a stale sensor record exists, the server attempts to validate the new sensor against the existing record. If the records are out of sync (e.g., different passwords, different manager IPs, corrupted state), authentication fails. Deleting the stale record allows the registration process to start fresh.
|}
 
'''Prevention: Keep Sensor Database Clean'''
 
* When decommissioning a sensor, delete its record from '''GUI → Settings → Sensors'''
* When replacing sensor hardware, delete the old sensor record before bringing up the new one
* Verify sensor IDs are unique across your deployment (duplicate <code>id_sensor</code> values will cause conflicts)
* Use the [[Backing_Up_GUI_Configuration|GUI backup feature]] to maintain a clean baseline, then restore sensors selectively as needed
 
=== RTP Streams End Prematurely in Distributed Deployments ===
 
If RTP streams end prematurely in call recordings when using a remote sniffer with a central GUI, this is often caused by the <code>natalias</code> configuration being set on the wrong system.
 
'''The Problem:'''
 
When packets are forwarded from a remote sniffer to a central server (Packet Mirroring mode), the central server sees the packets with their original IP addresses as captured by the sniffer. If <code>natalias</code> is configured on the remote sniffer, the IP address substitution happens at capture time. This can cause the central server's RTP correlation logic to fail because the substituted addresses do not match what the central server sees in the SIP signaling.
 
'''The Solution:'''
 
Configure <code>natalias</code> only on the central server that receives and processes the packets, not on the remote sniffer that captures and forwards them.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Critical: natalias Configuration Placement
|-
| style="vertical-align: top;" | '''Remote Sniffer (packet forwarding):'''
| Do NOT set <code>natalias</code> on the remote sensor. Let it forward packets with their original IP addresses.
|-
| style="vertical-align: top;" | '''Central Server (packet processing):'''
| Configure <code>natalias</code> on the central server that performs the analysis. The address substitution happens during correlation, at the point where SIP and RTP are matched.
|}
 
'''Configuration Example:'''


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# WRONG: Do NOT configure natalias on remote sniffer
# Single sniffer receiving HEP from multiple sources
# /etc/voipmonitor.conf on REMOTE SENSOR
hep                    = yes
# natalias = 1.2.3.4 10.0.0.5  # DON'T DO THIS
hep_bind_port          = 9060
 
interface              = eth0  # Can also sniff locally if needed
# CORRECT: Configure natalias on central server
# /etc/voipmonitor.conf on CENTRAL SERVER
natalias = 1.2.3.4 10.0.0.5
server_bind = 0.0.0.0
server_bind_port = 60024
# ... other central server settings
</syntaxhighlight>
</syntaxhighlight>


'''After Changing Configuration:'''
Both Kamailio (SIP) and rtpproxy (RTP) send HEP to this sniffer on port 9060. The sniffer correlates them automatically based on Call-ID and SDP IP:port.
= Sensor Health Monitoring =


<syntaxhighlight lang="bash">
== Management API ==
# Restart voipmonitor on BOTH systems
systemctl restart voipmonitor
</syntaxhighlight>
 
This ensures that RTP packets are correctly associated with their SIP dialogs on the central server, even when the network traverses NAT devices.
 
=== Measuring Network Throughput Between Probe and Server ===
 
If you experience memory buffer issues ("packetbuffer: MEMORY IS FULL"), high packet loss, or slow CDR delivery in distributed deployments, the bottleneck may be insufficient network bandwidth between the probe and central server. Before adding hardware, verify network performance using iperf.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Test Network Throughput Before Hardware Upgrades
|-
| style="vertical-align: top;" | '''When to test:'''
| * "packetbuffer: MEMORY IS FULL" errors on probes<br>* System load consistently high (e.g., 70-80% on 8-core system)<br>* Slow CDR display<br>* Packet loss during peak traffic
|-
| style="vertical-align: top;" | '''Objective:'''
| Identify whether the bottleneck is network bandwidth or local resources (CPU/RAM/Disk)
|}
 
==== Step 1: Install iperf3 on Both Systems ====


Install iperf3 on both the probe and the central server:
Query sensor status via TCP port 5029:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# On Debian/Ubuntu systems
echo 'sniffer_stat' | nc <sensor_ip> 5029
sudo apt-get install iperf3
 
# On RHEL/CentOS systems
sudo yum install iperf3
</syntaxhighlight>
</syntaxhighlight>


==== Step 2: Start iperf3 Server on Central Server ====
Returns JSON with status, version, active calls, packets per second, etc.


On the central server (listening side), run:
== Multi-Sensor Health Check Script ==


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Start iperf3 server (listening on all interfaces)
#!/bin/bash
iperf3 -s
SENSORS=("192.168.1.10:5029" "192.168.1.11:5029")
 
for SENSOR in "${SENSORS[@]}"; do
# Or specify a specific port if needed
    IP=$(echo $SENSOR | cut -d: -f1)
iperf3 -s -p 5201
    PORT=$(echo $SENSOR | cut -d: -f2)
</syntaxhighlight>
    STATUS=$(echo 'sniffer_stat' | nc -w 2 $IP $PORT 2>/dev/null | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
 
    echo "$IP: ${STATUS:-FAILED}"
Leave the server running.
done
 
==== Step 3: Run iperf3 Client on Probe ====
 
On the remote probe (sending side), test the connection to the central server:
 
<syntaxhighlight lang="bash">
# Test TCP throughput to central server
iperf3 -c <central_server_ip>
 
# Example:
iperf3 -c 192.168.1.100
 
# Test bidirectional throughput (useful for symmetric traffic)
iperf3 -c <central_server_ip> -R
 
# Run for 60 seconds with multiple parallel streams
iperf3 -c <central_server_ip> -t 60 -P 4
</syntaxhighlight>
</syntaxhighlight>


Replace <code>&lt;central_server_ip&gt;</code> with the IP address or hostname of your central server.
= Version Compatibility =
 
==== Step 4: Interpret Results ====
 
Analyze the iperf3 output to determine if your network is a bottleneck:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Throughput !! Interpretation !! Recommended Action
! Scenario !! Compatibility !! Notes
|-
| '''Expected bandwidth''' (e.g., >900 Mbps on 1Gb link)|| Network is NOT the bottleneck || Check local resources: CPU load, RAM, disk I/O
|-
|-
| '''Significantly lower''' (e.g., 200-500 Mbps on 1Gb link)|| Network is a bottleneck || Investigate network infrastructure: switch capacity, link quality, congestion
| '''GUI ≥ Sniffer''' || ✅ Compatible || Recommended
|-
|-
| '''Very low''' (e.g., <50 Mbps on 1Gb link)|| Severe network issue || Check for duplex mismatches, faulty cabling, switch configuration, ISP limitations
| '''GUI < Sniffer''' || ⚠️ Risk || Sensor may write to non-existent columns
|}
|}


==== Step 5: Decision Matrix After iperf3 Test ====
'''Best practice:''' Upgrade GUI first (applies schema changes), then upgrade sensors.


Based on the network test results:
For mixed versions temporarily, add to central server:
<syntaxhighlight lang="ini">
server_cp_store_simple_connect_response = yes  # Sniffer 2024.11.0+
</syntaxhighlight>


{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
= Troubleshooting =
|-
! colspan="2" style="background:#4A90E2; color: white;" | When Network is NOT the Bottleneck (High Throughput)
|-
| style="vertical-align: top;" | '''Symptom:'''
| iperf3 shows maximum bandwidth (e.g., >900 Mbps on 1Gb), but probe still shows MEMORY IS FULL or high CPU load
|-
| style="vertical-align: top;" | '''Root Cause:'''
| Local resource constraints on the probe machine
|-
| style="vertical-align: top;" | '''Solution:'''
| Check CPU usage (htop, uptime). If system load is consistently high (e.g., 70-80% on an 8-core system) and cannot process packets fast enough, '''increase CPU cores on the probe machine''' by upgrading hardware or adding more vCPUs in virtualized environments. Configuring tuning options (rtpthreads, ringbuffer) is a temporary workaround; adding CPU cores addresses the root cause.
|-
| colspan="2" | See [[Hardware|Hardware Sizing Examples]] for CPU requirements based on concurrent call volume, or [[Scaling|Scaling and Performance Tuning]] for optimization guidance.
|}


{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
== Quick Diagnosis ==
|-
! colspan="2" style="background:#ffc107;" | When Network IS the Bottleneck (Low Throughput)
|-
| style="vertical-align: top;" | '''Symptom:'''
| iperf3 shows significantly lower bandwidth than link capacity
|-
| style="vertical-align: top;" | '''Root Cause:'''
| Network infrastructure cannot handle VoIP traffic volume between probe and server
|-
| style="vertical-align: top;" | '''Solution:'''
| * Inspect network path: Check switches, routers, VLANs for congestion or misconfiguration<br>* Verify link speed and duplex settings (ethtool)<br>* Check for packet loss (ping with statistics) and latency (traceroute)<br>* Upgrade network: 1GbE to 10GbE, add dedicated links, or improve network routing<br>* Consider switching to Local Processing mode (<code>packetbuffer_sender=no</code>) to reduce network traffic
|}
 
==== Step 6: Network Configuration Checks ====
 
If iperf3 shows low throughput, check these network configuration issues:


{| class="wikitable"
{| class="wikitable"
|-
|-
! Check !! Command !! What to Look For
! Symptom !! First Check !! Likely Cause
|-
|-
| Link speed and duplex || <code>ethtool eth0</code> || "Speed: 1000Mb/s", "Duplex: Full"
| Sensor not connecting || <code>journalctl -u voipmonitor -f</code> on sensor || Check <code>server_destination</code>, password, firewall
|-
|-
| Packet loss on the path || <code>ping -c 100 <central_server_ip></code> || "0% packet loss" is ideal. Loss >1% indicates network issues
| Traffic rate <code>[0.0Mb/s]</code> || tcpdump on sensor interface || Network/SPAN issue, not communication
|-
|-
| Network latency || <code>traceroute <central_server_ip></code> || Consistent sub-millisecond hops are ideal. High variance indicates congestion
| High memory on central server || Check if <code>sipport</code> includes 60024 || Exclude server port from sipport
|-
|-
| Interface errors || <code>ethtool -S eth0| grep -i error</code> || Should be zero or very low. High errors indicate hardware issues
| Missing calls || Compare <code>sipport</code> on probe vs central || Must match on both sides
|}
 
Example commands:
 
<syntaxhighlight lang="bash">
# Check network interface speed and duplex
ethtool eth0
# Look for: Speed: 1000Mb/s, Duplex: Full
 
# Test for packet loss
ping -c 100 192.168.1.100
# Look for: 0% packet loss
 
# Check network path latency
traceroute 192.168.1.100
 
# Check interface error counters
ethtool -S eth0 | grep -i error
</syntaxhighlight>
 
==== Considerations for Packet Mirroring vs Local Processing ====
 
When network bandwidth is constrained:
 
* '''Local Processing Mode (<code>packetbuffer_sender=no</code>):''' Probes analyze packets locally and send only CDRs (SQL queries) to the central server. Network traffic is minimal (typically <1 Mbps even during high traffic).
 
* '''Packet Mirroring Mode (<code>packetbuffer_sender=yes</code>):''' Probes forward raw packets to the central server. Network bandwidth requirement roughly equals VoIP traffic volume (e.g., 300 Mbps for 3 Gbit/s traffic).
 
For deployments with limited network bandwidth between probes and central server, Local Processing mode is generally preferred.
 
For more details on switching between modes, see [[Sniffer_distributed_architecture#Local_Processing_vs_Packet_Mirroring|Local Processing vs Packet Mirroring]].
 
=== Call Legs Visible in Active Calls but Missing from CDRs ===
 
If you are using multiple sensors configured with '''Packet Mirroring mode''' (<code>packetbuffer_sender=yes</code>) and observe that:
 
* Some call legs appear in the '''Active calls''' view (real-time monitoring works)
* The same legs are '''missing from the CDR''' view (no database records created)
* Sensors are capturing traffic correctly (tcpdump shows packets)
 
The root cause is that '''multiple sensors sending raw packets to the same receiver instance''' can interfere with each other's call processing logic.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Critical: packetbuffer_sender Requires Unique Receiver
| "Bad password" error || GUI → Settings → Sensors || Delete stale sensor record, restart sensor
|-
|-
| style="vertical-align: top;" | '''The Problem:'''
| "Connection refused (111)" after migration || Check <code>server_destination</code> in config || Points to old server IP
| When multiple sensors with <code>packetbuffer_sender=yes</code> send raw packets to a single central receiver instance, packets from different sensors can conflict in the receiver's call table structure. This causes some call legs to display in Active calls (which reads from live memory) but never complete as CDRs because the call processing logic becomes tangled.
|-
|-
| style="vertical-align: top;" | '''Why It Happens:'''
| RTP streams end prematurely || Check <code>natalias</code> location || Configure only on central server
| Packet Mirroring forwards the complete raw packet stream to the receiver for processing. When multiple sensors send packets to the same receiver, the single call table must track all streams simultaneously. In this architecture, packets from different sensors sharing similar Call-IDs, timestamps, or IP addresses can interfere with each other's call finalization logic.
|-
|-
| style="vertical-align: top;" | '''The Solution:'''
| Time sync errors || <code>timedatectl status</code> || Fix NTP or increase tolerance
| Configure each sensor with <code>packetbuffer_sender=yes</code> to connect to a '''unique separate receiver instance'''. Each receiver should run on a dedicated server host (or separate VM/container) to ensure complete isolation of packet processing.
|}
|}


==== Correct topology for packetbuffer_sender=yes ====
== Connection Testing ==


<kroki lang="plantuml">
<syntaxhighlight lang="bash">
@startuml
# Test connectivity from sensor to server
skinparam shadowing false
nc -zv <server_ip> 60024
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #dc3545
  BackgroundColor #FFFFFF
}


rectangle "Sensor A\n(packetbuffer_sender=yes)" as SA
# Verify server is listening
rectangle "Sensor B\n(packetbuffer_sender=yes)" as SB
ss -tulpn | grep voipmonitor
rectangle "Receiver Instance 1\n(server_bind for A)" as RA
rectangle "Receiver Instance 2\n(server_bind for B)" as RB
database "MySQL\n(shared database)" as DB
 
SA --> RA : encrypted TCP
SB --> RB : encrypted TCP
RA --> DB : CDRs from A
RB --> DB : CDRs from B
 
note right of RA
  Each sensor connects
  to its OWN receiver instance
  on a dedicated server host
end note
@enduml
</kroki>
 
==== Configuration Example ====


Run separate VoIPmonitor receiver instances on different hosts:
# Check sensor logs
 
journalctl -u voipmonitor -n 100 | grep -i "connect"
'''Receiver Instance 1 (dedicated to Sensor A):'''
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on Host 1
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = password_A
 
# Database configuration
mysqlhost              = 192.168.1.100  # Shared MySQL server
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
 
id_sensor              = 1
sipport                = 5060
# ... other sniffer options
</syntaxhighlight>
</syntaxhighlight>


'''Receiver Instance 2 (dedicated to Sensor B):'''
== Time Synchronization Errors ==
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on Host 2
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = password_B
 
# Database configuration
mysqlhost              = 192.168.1.100  # Shared MySQL server
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
 
id_sensor              = 2
sipport                = 5060
# ... other sniffer options
</syntaxhighlight>


Configure each sensor to connect to its dedicated receiver:
If seeing "different time between server and client" errors:


'''Sensor A configuration:'''
'''Immediate workaround:''' Increase tolerance on both sides:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# /etc/voipmonitor.conf on Sensor A
client_server_connect_maximum_time_diff_s = 30
id_sensor              = 1
receive_packetbuffer_maximum_time_diff_s = 30
server_destination      = 192.168.1.1    # IP of Receiver Instance 1
server_destination_port = 60024
server_password        = password_A
 
packetbuffer_sender    = yes            # Packet Mirroring mode
interface              = eth0
sipport                = 5060
</syntaxhighlight>
</syntaxhighlight>


'''Sensor B configuration:'''
'''Root cause fix:''' Ensure NTP is working:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="bash">
# /etc/voipmonitor.conf on Sensor B
timedatectl status          # Check sync status
id_sensor              = 2
chronyc tracking            # Check offset (Chrony)
server_destination      = 192.168.1.2    # IP of Receiver Instance 2
ntpq -p                      # Check offset (NTP)
server_destination_port = 60024
server_password        = password_B
 
packetbuffer_sender    = yes            # Packet Mirroring mode
interface              = eth0
sipport                = 5060
</syntaxhighlight>
</syntaxhighlight>


==== Alternative: Local Processing Mode ====
== Network Throughput Testing ==
 
If running multiple receiver instances is not feasible, consider switching to '''Local Processing mode''' (<code>packetbuffer_sender=no</code>). In this mode:
 
* Each sensor processes packets locally and generates CDRs independently
* Only CDR data (not raw packets) is sent to the central server
* No receiver instance conflicts because each sensor handles its own call processing
 
{| class="wikitable"
|-
! Mode !! Configuration !! Call Processing Location !! CDR Generation
|-
| '''Packet Mirroring''' || <code>packetbuffer_sender=yes</code> || Central receiver || Vulnerable to multi-sensor conflicts
|-
| '''Local Processing''' || <code>packetbuffer_sender=no</code> || Each sensor independently || Isolated per sensor
|}
 
==== Why Active Calls Works but CDRs Do Not ====


* '''Active calls view''' reads directly from the live call table in sensor memory. If packets are being captured, they appear here regardless of whether CDR generation will complete.
If experiencing "packetbuffer: MEMORY IS FULL" errors, test network with iperf3:
* '''CDR generation''' requires proper call termination logic (BYE processing, timeout handling, MySQL insertion). When multiple sensors send to a single receiver, the call table can become confused about which packets belong to which call leg, preventing proper CDR finalization.
 
==== Diagnosis Commands ====
 
To verify if sensors are sharing a receiver instance:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# On the central receiver server
# On central server
# Check all listening VoIPmonitor instances
iperf3 -s
ss -tulpn | grep voipmonitor
 
# Expected output for one receiver:
# voipmonitor LISTEN 0.0.0.0:60024 (single instance)
 
# Expected output for independent receivers:
# voipmonitor1 LISTEN 0.0.0.0:60024 (on host 192.168.1.1)
# voipmonitor2 LISTEN 0.0.0.0:60024 (on host 192.168.1.2)


# Check how many sensors are connected
# On probe
journalctl -u voipmonitor | grep "connected from"
iperf3 -c <server_ip>
</syntaxhighlight>
</syntaxhighlight>
== Legacy: Mirror Mode ==
'''Note:''' The older <code>mirror_destination</code>/<code>mirror_bind</code> options still exist but the modern Client-Server approach with <code>packetbuffer_sender=yes</code> is preferred as it provides encryption and simpler management.
If you are using mirror mode (remote probes mirroring traffic to a central receiver using legacy <code>mirror_bind</code>), the following configuration option is available:
=== mirror_bind_sensor_id_by_sender ===
By default, when a remote probe sends mirrored traffic to a central receiver, all packets are recorded using the central receiver's <code>id_sensor</code> value. This can make it difficult to distinguish which probe captured each call in the GUI.
The <code>mirror_bind_sensor_id_by_sender</code> option allows the central receiver to use the sending probe's <code>id_sensor</code> instead of its own.


{| class="wikitable"
{| class="wikitable"
|-
|-
! Setting !! Behavior
! Result !! Interpretation !! Action
|-
|-
| <code>mirror_bind_sensor_id_by_sender no</code> (default) || All packets are recorded with the central receiver's <code>id_sensor</code>. All calls appear to come from one sensor.
| Expected bandwidth (>900 Mbps on 1Gb) || Network OK || Check local CPU/RAM
|-
|-
| <code>mirror_bind_sensor_id_by_sender yes</code> || Packets are recorded using the sending probe's <code>id_sensor</code>. Each call is tagged with the probe that captured it.
| Low throughput || Network bottleneck || Check switches, cabling, consider Local Processing mode
|}
|}


'''Usage Example:'''
== Debugging SIP Traffic ==


<syntaxhighlight lang="ini">
<code>sngrep</code> does not work on the central server because traffic is encapsulated in the TCP tunnel.
# Central receiver configuration (/etc/voipmonitor.conf)
mirror_bind        = 0.0.0.0:9000
id_sensor          = 1  # Default, overridden when option is enabled


# Use the sending probe's id_sensor for each call
'''Options:'''
mirror_bind_sensor_id_by_sender = yes
* '''Live Sniffer:''' Use GUI → Live Sniffer to view SIP from remote sensors
# After enabling, each call will show its originating probe's id_sensor (e.g., 2, 3, 4...)
* '''sngrep on sensor:''' Run <code>sngrep -i eth0</code> directly on the remote sensor
</syntaxhighlight>


'''When to Use This Option:'''
== Stale Sensor Records ==


* You have multiple remote probes mirroring to one central receiver
If a new sensor fails with "bad password" despite correct credentials:
* You want to distinguish which probe captured each call in the GUI
* You want to filter calls by probe in the CDR view
* You need accurate per-probe statistics and reporting


'''Important Notes:'''
# Delete the sensor record from '''GUI → Settings → Sensors'''
# Restart voipmonitor on the sensor: <code>systemctl restart voipmonitor</code>
# The sensor will re-register automatically


* Each remote probe must have a unique <code>id_sensor</code> configured
= Legacy: Mirror Mode =
* This option only works with legacy mirror mode (using <code>mirror_bind</code>)
* For new deployments, use the modern Client/Server mode with <code>packetbuffer_sender</code> instead


=== Creating Separate CDRs Per Probe ===
The older <code>mirror_destination</code>/<code>mirror_bind</code> options still work but Client-Server mode is preferred (encryption, simpler management).


If you need completely separate CDR records for each probe (calls with the same Call-ID appearing as separate records rather than being merged), you must run '''multiple receiver instances''' on the central server, each listening on a different port.
To migrate from mirror mode:
# Stop sensors, comment out <code>mirror_*</code> parameters
# Configure <code>server_bind</code> on central, <code>server_destination</code> on sensors
# Restart all services


For mirror mode <code>id_sensor</code> attribution, use:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Receiver Instance 1 configuration (/etc/voipmonitor1.conf)
# On central receiver
mirror_bind        = 0.0.0.0:9000
mirror_bind_sensor_id_by_sender = yes
id_sensor          = 2
# ... other options
 
# Receiver Instance 2 configuration (/etc/voipmonitor2.conf)
mirror_bind        = 0.0.0.0:9001
id_sensor          = 3
# ... other options
</syntaxhighlight>
</syntaxhighlight>


Configure each probe to send to its dedicated receiver port. This ensures that calls from different probes are kept completely separate.
= See Also =


For modern Client/Server deployments with Packet Mirroring, see the section on [[#Multiple_receivers_for_Packet_Mirroring_mode|Multiple Receivers for Packet Mirroring]].
* [[Sniffing_modes|Deployment & Topology Guide]] - Traffic forwarding methods
* [[Sniffer_configuration|Sniffer Configuration]] - All parameters reference
* [[Merging_or_correlating_multiple_call_legs|Call Correlation]] - Multi-leg call handling
* [[FAQ#One_GUI_for_multiple_sniffers|FAQ: One GUI for Multiple Sniffers]]


=== Debugging SIP Traffic in Distributed Architecture ===
== Filtering Options in Packet Mirroring Mode ==


When using the Client-Server architecture (Packet Mirroring or Local Processing), standard packet capture tools like <code>sngrep</code> cannot see SIP packets on the central server because the traffic is encapsulated inside the encrypted TCP tunnel between the sensor and the central server.
{{Note|1='''Important distinction:''' In Packet Mirroring mode (<code>packetbuffer_sender=yes</code>):


{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
* '''Capture rules (GUI-based):''' Applied ONLY on the central server
|-
* '''BPF filters / IP filters:''' CAN be applied on the remote sensor to reduce bandwidth
! colspan="2" style="background:#4A90E2; color: white;" | Why sngrep Does Not Work on Central Server
|-
| style="vertical-align: top;" | '''The Issue:'''
| <code>sngrep</code> relies on capturing raw SIP packets directly from the network interface. In Client-Server mode, remote sensors forward traffic to the central server inside an encrypted TCP channel (default port 60024 with zstd compression). The SIP packets are wrapped inside this tunnel and cannot be inspected by standard tools.
|-
| style="vertical-align: top;" | '''What You See:'''
| <code>tcpdump</code> on the central server shows encrypted TCP packets on port 60024 (Packet Mirroring) or SQL traffic on port 3306 (Local Processing). No raw SIP packets (UDP 5060) are visible on the wire.
|}


To inspect SIP packets in distributed deployments, use one of these methods:
Use the following options on the '''remote sensor''' to filter traffic BEFORE sending to the central server:


==== Live Sniffer (Recommended) ====
The '''Live Sniffer''' feature in the VoIPmonitor GUI provides real-time SIP packet display from remote sensors. This is the preferred method for debugging call flows and network issues in distributed architectures.
<syntaxhighlight lang="bash">
# To use Live Sniffer:
# 1. Open the VoIPmonitor GUI
# 2. Navigate to "Live Sniffer"
# 3. Select the remote sensor from the dropdown
# 4. Click "Start" to view live SIP packets from that sensor
</syntaxhighlight>
The Live Sniffer streams SIP packets from the sensor to the GUI in real-time via the Manager API (TCP port 5029). Features include:
* Call coloring by Call-ID
* Packet details inspection
* Call flow visualization
* Multi-user support
For setup and troubleshooting of Live Sniffer, see [[Live_sniffer|Live Sniffer documentation]].
==== sngrep on Remote Sensors ====
If you need to use <code>sngrep</code> directly, run it on the '''remote sensor machine''' where the traffic first arrives from the network interface before VoIPmonitor encapsulation occurs.
<syntaxhighlight lang="bash">
# SSH into the remote sensor
# Run sngrep on the interface connected to the SPAN/mirror port
sngrep -i eth0
</syntaxhighlight>
Replace <code>eth0</code> with the actual network interface on the sensor that receives the mirrored traffic from the switch.
==== Verify Connectivity on Central Server ====
If you need to verify that data is flowing from sensors to the central server:
<syntaxhighlight lang="bash">
# Check for encrypted tunnel traffic (Packet Mirroring mode)
tcpdump -i any port 60024
# Check for SQL traffic (Local Processing mode)
tcpdump -i any port 3306
# Check sensor statistics via Management API
echo 'sniffer_stat' | nc <sensor_ip> 5029
</syntaxhighlight>
=== Migrating from Mirror Mode to Client-Server Mode ===
If your system uses the legacy mirror mode (<code>mirror_destination</code> on probes, <code>mirror_bind</code> on server), you should migrate to the modern client/server mode. Common symptoms of mirror mode issues include all CDRs being incorrectly associated with a single sensor after system updates.
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Why Migration is Recommended
|-
| style="vertical-align: top;" | '''Mirror Mode Limitations:'''
| * No encryption (raw UDP traffic)
* Complex firewall configuration (must open mirroring port)
* Less robust connection handling
* Configuration can be lost during OS upgrades
|-
| style="vertical-align: top;" | '''Client-Server Advantages:'''
| * Encrypted TCP connections
* Automatic reconnection with failover
* Centralized port configuration
* Better troubleshooting capabilities
|}
==== Prerequisites ====
* Central server hostname or IP address
* Port for client-server communication (default: 60024)
* Strong shared password for authentication
==== Migration Steps ====
;1. Stop the voipmonitor sniffer service on all probe machines:
<syntaxhighlight lang="bash">
# On each probe
systemctl stop voipmonitor
</syntaxhighlight>
;2. Update GUI Sensors list:
# Log in to the VoIPmonitor GUI
# Navigate to '''Settings → Sensors'''
# Remove all old probe records, keeping only the server instance (e.g., localhost or the central server IP)
;3. Configure the Central Server:
Edit <code>/etc/voipmonitor.conf</code> on the central server:
<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# COMMENT OUT or remove mirror mode parameters:
# On REMOTE SENSOR (client)
# mirror_bind_ip = 1.2.3.4
# mirror_bind_port = 9000


# ADD client-server mode parameters:
# Option 1: BPF filter (tcpdump syntax) - most flexible
server_bind              = <server_ip>    # Use 0.0.0.0 to listen on all interfaces
filter = not net 192.168.0.0/16 and not net 10.0.0.0/8
server_bind_port        = <port>          # Default is 60024
server_password          = <a_strong_password>
 
# MySQL configuration remains unchanged
mysqlhost                = localhost
mysqldb                  = voipmonitor
mysqluser                = voipmonitor
mysqlpassword            = <your_db_password>
</syntaxhighlight>
 
Restart the service on the central server:
<syntaxhighlight lang="bash">
# On central server
systemctl restart voipmonitor
</syntaxhighlight>


Verify the server is listening:
# Option 2: IP allow-list filter - CPU-efficient, no negation support
<syntaxhighlight lang="bash">
interface_ip_filter = 192.168.1.0/24
# Check that voipmonitor is listening on the configured port
interface_ip_filter = 10.0.0.0/8
ss -tulpn | grep voipmonitor
# Should show: voipmonitor LISTEN 0.0.0.0:60024 (or your custom port)
</syntaxhighlight>
</syntaxhighlight>


;4. Configure each Probe:
<b>Benefits of filtering on remote sensor:</b>
Edit <code>/etc/voipmonitor.conf</code> on each remote probe:
* Reduces WAN bandwidth usage between sensor and central server
<syntaxhighlight lang="ini">
* Reduces processing load on central server
# COMMENT OUT or remove mirror mode parameters:
* Use <code>filter</code> for complex conditions (tcpdump/BPF syntax)
# mirror_destination_ip = 1.2.3.4
* Use <code>interface_ip_filter</code> for simple IP allow-lists (more efficient)
# mirror_destination_port = 9000


# ADD client-server mode parameters:
<b>Filtering approaches:</b>
id_sensor                = <unique_id>     # Must be unique per sensor
* For <b>SIP header-based filtering</b>: Apply capture rules on the '''central server''' only
server_destination      = <server_ip>
* For <b>IP/subnet filtering</b>: Use <code>filter</code> or <code>interface_ip_filter</code> on '''remote sensor'''}}
server_destination_port  = <port>         # Must match server_bind_port
server_password          = <a_strong_password> # Same password used on server


# IMPORTANT: Set packet handling mode
== Supported Configuration Options in Packet Mirroring Mode ==
packetbuffer_sender      = no    # Local Processing: analyze locally, send CDRs only
# OR
# packetbuffer_sender    = yes  # Packet Mirroring: send raw packets to server


# Capture settings remain unchanged
In Packet Mirroring mode (<code>packetbuffer_sender = yes</code>), the remote sensor forwards raw packets without processing them. This means many configuration options that manipulate packet behavior are '''unsupported''' on the remote sensor.
interface                = eth0
sipport                  = 5060
# No MySQL credentials needed on remote sensors for Local Processing mode
</syntaxhighlight>


Restart the service on each probe:
== Supported Options on Remote Sensor (packetbuffer_sender) ==
<syntaxhighlight lang="bash">
# On each probe
systemctl restart voipmonitor
</syntaxhighlight>


;5. Verify Connection in GUI:
The following options work correctly on the remote sensor in packet mirroring mode:
# Log in to the VoIPmonitor GUI
# Navigate to '''Settings → Sensors'''
# Verify that probes appear automatically with their configured <code>id_sensor</code> values
# Check the connection status (online/offline)


;6. Test Data Flow:
{| class="wikitable"
# Generate test traffic on a probe network (make a test call)
! Parameter !! Description
# Check CDR view in GUI
# Verify that new records show the correct <code>id_sensor</code> for that probe
# Confirm PCAP files are accessible (click play button in CDR view)
 
==== Common Issues During Migration ====
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
|-
! colspan="2" style="background:#4A90E2; color: white;" | Troubleshooting Connection Problems
| <code>id_sensor</code> || Unique sensor identifier
|-
|-
| style="vertical-align: top;" | '''Probes cannot connect:'''
| <code>server_destination</code> || Central server address
| * Verify <code>server_password</code> is identical on server and all probes
* Check firewall: allow incoming TCP on <code>server_bind_port</code> (default 60024) on central server
* Verify network connectivity: <code>nc -zv <server_ip> <server_bind_port></code> from probe
|-
|-
| style="vertical-align: top;" | '''All CDRs show same sensor:'''
| <code>server_destination_port</code> || Central server port (default 60024)
| This typically indicates the old mirror mode configuration is still active or the <code>id_sensor</code> is not set on probes. Double-check that:
* Mirror parameters are commented out on both sides
* Each probe has a unique <code>id_sensor</code> value
* Services were restarted after configuration changes
|-
| style="vertical-align: top;" | '''PCAP files not accessible:'''
| In Local Processing mode (<code>packetbuffer_sender=no</code>), PCAPs are stored on probes and retrieved via TCP port 5029. Ensure:
* Central server can reach each probe on TCP/5029
* Firewall allows TCP/5029 from central server to probes
|}
 
== Critical Requirement: SIP and RTP must be captured by the same sniffer instance ==
 
'''VoIPmonitor cannot reconstruct a complete call record if SIP signaling and RTP media are captured by different sniffer instances.'''
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Important: Single sniffer requirement
| <code>server_password</code> || Authentication password
|-
|-
| style="vertical-align: top;" | '''What does not work:'''
| <code>server_destination_timeout</code> || Connection timeout settings
| * Sniffer A in Availability Zone 1 captures SIP signaling
* Sniffer B in Availability Zone 2 captures RTP media
* Result: Incomplete call record, GUI cannot reconstruct the call
|-
|-
| style="vertical-align: top;" | '''Why:'''
| <code>server_destination_reconnect</code> || Auto-reconnect behavior
| Call correlation requires a '''single sniffer instance to process both SIP and RTP packets from the same call'''. The sniffer correlates SIP signaling (INVITE, BYE, etc.) with RTP media in real-time during packet processing. If packets are split across multiple sniffers, the correlation cannot occur.
|-
| style="vertical-align: top;" | '''Solution:'''
| Forward traffic so that '''one sniffer processes both SIP and RTP for each call'''. Options:
* Route both SIP and RTP through the same Availability Zone for capture
* Use Packet Mirroring mode to forward complete traffic (SIP+RTP) to a central server that processes everything
* Configure network routers/firewalls to forward the required stream to the correct zone
|}
 
Configuration parameters like <code>receiver_check_id_sensor</code> and <code>cdr_check_exists_callid</code> are for other scenarios (multipath routing, duplicate Call-ID handling) and '''do NOT enable split SIP/RTP correlation'''. Setting these parameters does not allow SIP from one sniffer to be merged with RTP from another sniffer.
 
== Intermediate Server: Multi-Sensor Aggregation ==
 
An intermediate server can receive traffic from multiple remote sensors and forward it to a central server. This is useful for aggregating traffic from many locations before sending to a central data center.
 
=== Architecture ===
 
<kroki lang="plantuml">
@startuml
skinparam shadowing false
skinparam defaultFontName Arial
skinparam rectangle {
  BorderColor #4A90E2
  BackgroundColor #FFFFFF
}
 
rectangle "Remote Sensor A" as RA
rectangle "Remote Sensor B" as RB
rectangle "Remote Sensor C" as RC
rectangle "Intermediate Server\n(server_bind + server_destination)" as INT
rectangle "Central Server\n(server_bind)" as CS
database "MySQL" as DB
 
RA --> INT : encrypted TCP
RB --> INT : encrypted TCP
RC --> INT : encrypted TCP
 
INT --> CS : encrypted TCP
CS --> DB : CDRs
 
note right of INT
  Behavior controlled by
  packetbuffer_sender on
  intermediate server:
 
  * packetbuffer_sender=no:
    Process traffic locally,
    send CDRs to central
 
  * packetbuffer_sender=yes:
    Forward raw packets to
    central server
end note
@enduml
</kroki>
 
This is supported because the intermediate server does NOT do local packet capture - it only acts as a relay.
 
=== Intermediate Server Configuration ===
 
The intermediate server has both <code>server_bind</code> (to receive from remote sensors) and <code>server_destination</code> (to send to central server).
 
<syntaxhighlight lang="ini">
# On INTERMEDIATE SERVER
# Acts as server for remote sensors, client to central server
id_sensor              = 100    # Unique ID for this intermediate server
 
# Receive from remote sensors (server role)
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = sensor_password
 
# Send to central server (client role)
server_destination      = central.server.ip
server_destination_port = 60024
server_password        = central_password
 
# CRITICAL: packetbuffer_sender controls what happens to forwarded traffic
 
# Option 1: Local Processing on intermediate server
packetbuffer_sender    = no      # Process locally, send CDRs to central
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
 
# OR Option 2: Forward raw packets to central server
# packetbuffer_sender  = yes    # Forward raw packets (no database needed here)
</syntaxhighlight>
 
=== <code>packetbuffer_sender</code> on Intermediate Server ===
 
The <code>packetbuffer_sender</code> setting on the intermediate server determines how it handles traffic from remote sensors:
 
{| class="wikitable"
|-
|-
! Setting !! What Happens !! Storage Location
| <code>filter</code> || BPF filter to limit capture (use this to capture only SIP)
|-
|-
| <code>packetbuffer_sender=no</code> || Intermediate server processes traffic (SIP/RTP analysis) and sends CDRs to central server || PCAPs on intermediate server
| <code>interface_ip_filter</code> || IP-based packet filtering
|-
|-
| <code>packetbuffer_sender=yes</code> || Intermediate server forwards raw packets to central server, which processes them || PCAPs on central server
| <code>interface</code> || Capture interface
|}
 
In both cases, the '''original remote sensors must still be manually added to the GUI for visibility'''
 
=== Original vs Intermediate Sensor Visibility ===
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
|-
! colspan="2" style="background:#4A90E2; color: white;" | Important: Manual Sensor Registration
| <code>sipport</code> || SIP ports to monitor
|-
|-
| style="vertical-align: top;" | '''Behavior:'''
| <code>promisc</code> || Promiscuous mode
| When using an intermediate server, the original remote sensors (A, B, C) are not automatically visible in the GUI Settings. Only the intermediate server itself appears.
|-
|-
| style="vertical-align: top;" | '''Solution:'''
| <code>rrd</code> || RRD statistics
| To view statistics and status for the original sensors, they must be manually added to the GUI Settings list with their <code>id_sensor</code> values, even though they connect to the intermediate server rather than directly to the central server.
|}
 
=== Example: Local Processing Mode ===
 
Remote sensors forward CDRs to intermediate server, which forwards them to central server:
 
<syntaxhighlight lang="ini">
# Remote Sensors (A, B, C)
id_sensor              = 2        # Unique values: 2, 3, 4...
server_destination      = intermediate.server.ip
server_destination_port = 60024
server_password        = sensor_password
 
packetbuffer_sender    = no      # Local Processing: process here, send CDRs
interface              = eth0
sipport                = 5060
</syntaxhighlight>
 
<syntaxhighlight lang="ini">
# Intermediate Server
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = sensor_password
 
server_destination      = central.server.ip
server_destination_port = 60024
server_password        = central_password
 
packetbuffer_sender    = no      # Process locally, send CDRs onward
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
</syntaxhighlight>
 
<syntaxhighlight lang="ini">
# Central Server
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = central_password
 
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
</syntaxhighlight>
 
=== Example: Packet Mirroring Mode ===
 
Remote sensors forward raw packets to intermediate server, which forwards them to central server:
 
<syntaxhighlight lang="ini">
# Remote Sensors (A, B, C)
id_sensor              = 2        # Unique values: 2, 3, 4...
server_destination      = intermediate.server.ip
server_destination_port = 60024
server_password        = sensor_password
 
packetbuffer_sender    = yes    # Packet Mirroring: send raw packets
interface              = eth0
sipport                = 5060
</syntaxhighlight>
 
<syntaxhighlight lang="ini">
# Intermediate Server
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = sensor_password
 
server_destination      = central.server.ip
server_destination_port = 60024
server_password        = central_password
 
packetbuffer_sender    = yes    # Forward raw packets onward
# No database configuration needed on intermediate server
</syntaxhighlight>
 
<syntaxhighlight lang="ini">
# Central Server
server_bind            = 0.0.0.0
server_bind_port        = 60024
server_password        = central_password
 
mysqlhost              = localhost
mysqldb                = voipmonitor
mysqluser              = voipmonitor
mysqlpassword          = db_password
 
# Processing and storage options (configured on central server)
sipport                = 5060
savertp                = yes
savesip                = yes
</syntaxhighlight>
 
== Version Compatibility ==
 
=== General Compatibility Rules ===
 
In general, there is '''no strict version locking''' between the VoIPmonitor GUI (web interface) and the Sniffer (sensor components). The primary compatibility constraint is the '''Database Schema''', which is managed primarily by the GUI.
 
{| class="wikitable"
|-
|-
! Scenario !! Compatibility !! Risk Level !! Details
| <code>spooldir</code> || Temporary packet buffer directory
|-
|-
| '''GUI ≥ Sniffer''' || '''Compatible''' || ✅ Low || GUI is newer or same version. It can visualize data from older sensors, and the database schema supports all data the sensors write.
| <code>ringbuffer</code> || Ring buffer size for packet mirroring
|-
|-
| '''GUI < Sniffer''' || '''Potentially Incompatible''' || ⚠️ High || Sensor is newer. It may try to write to database columns or tables that do not exist in the older GUI's schema, leading to SQL insert errors.
| <code>max_buffer_mem</code> || Maximum buffer memory
|}
 
'''Best Practice:''' Maintain the GUI version equal to or higher than the sniffer version (GUI ≥ Sniffer).
 
When upgrading components, always upgrade the GUI first so it applies new database schemas, then upgrade the sniffers.
 
=== Client-Server Mode Version Matching ===
 
For client-server mode deployments (remote sensors connecting to a central server), it is '''strongly recommended that clients and receivers use the same version'''.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
|-
! colspan="2" style="background:#ffc107;" | Version Matching Recommendation
| <code>packetbuffer_enable</code> || Enable packet buffering
|-
|-
| style="vertical-align: top;" | '''For client-server mode:'''
| <code>packetbuffer_compress</code> || Enable compression for forwarded packets
| Keep all sensors and their central receivers on the same version whenever possible. While different versions can work together, matching versions ensure full compatibility and access to the latest features and fixes.
|-
|-
| style="vertical-align: top;" | '''For gradual upgrades:'''
| <code>packetbuffer_compress_ratio</code> || Compression ratio
| If you cannot upgrade all sensors simultaneously, upgrade the central receiver/server first, then upgrade sensors one by one. The receiver should be at least as new as the newest connecting sensor.
|}
|}


=== Mixed Version Compatibility ===
== Unsupported Options on Remote Sensor ==


If you need to run mixed versions in your deployment temporarily, a compatibility option is available in sensor version 2024.11.0 and newer.
The following options '''do NOT work''' on the remote sensor in packet mirroring mode because the sensor does not parse packets:
 
<strong>server_cp_store_simple_connect_response</strong>
 
This configuration option enables the client-server communication protocol to work with mixed-version environments. When enabled, the server uses a simpler protocol variant that is compatible with older sensor versions.
 
<syntaxhighlight lang="ini">
# On the central receiver/server (sniffer 2024.11.0+)
server_cp_store_simple_connect_response = yes
</syntaxhighlight>


{| class="wikitable"
{| class="wikitable"
! Parameter !! Reason
|-
|-
! Condition !! Default Value !! Recommended Setting
| <code>natalias</code> || NAT alias handling (configure on central server instead)
|-
| Matching versions (normal) || no || no (default)
|-
|-
| Mixed versions (temporary) || no || yes (enable only if needed)
| <code>rtp_check_both_sides_by_sdp</code> || RTP correlation requires packet parsing
|}
 
{| class="wikitable" style="background:#f8f9fa; border:1px solid #dee2e6;"
|-
|-
! colspan="2" style="background:#dee2e6;" | Important Notes on Mixed Versions
| <code>disable_process_sdp</code> || SDP processing happens on central server
|-
|-
| This is a '''temporary compatibility option''' for migration periods. Using matching versions for all components is the preferred long-term configuration.
| <code>save_sdp_ipport</code> || SDP extraction happens on central server
|-
|-
| The option should be set on the '''central receiver/server''' instance that receives connections from sensors.
| <code>rtpfromsdp_onlysip</code> || RTP mapping requires packet parsing
|-
|-
| Once all sensors are upgraded to their target versions, disable this option (<code>server_cp_store_simple_connect_response = no</code>) for normal operation.
| <code>rtpip_find_endpoints</code> || Endpoint discovery requires packet parsing
|}
|}


=== Version Verification ===
{{Warning|1='''Critical: Storage options''' (<code>savesip</code>, <code>savertp</code>, <code>saveaudio</code>) '''must be configured on the CENTRAL SERVER''' in packet mirroring mode. The remote sensor only forwards packets and does not perform any storage operations.}}


To check the running version of any component:
== SIP-Only Capture Example ==


<syntaxhighlight lang="bash">
To capture and forward only SIP packets (excluding RTP/RTCP) for security or compliance:
# On the sensor or server host
/usr/local/sbin/voipmonitor --version
 
# Or via management API (default port 5029)
echo 'sniffer_version' | nc 127.0.0.1 5029
</syntaxhighlight>
 
In the GUI, navigate to '''Settings → Sensors''' to see the version of each connected sensor.
 
== Limitations ==
 
* All sensors must use the same <code>server_password</code> at each connection level (sensors→intermediate and intermediate→central)
* '''A single sniffer cannot do local packet capture AND act as both server and client simultaneously.''' The intermediate server configuration works because it does NOT capture from its own network interface - it only receives from sensors and forwards to central server.
* Each sensor requires a unique <code>id_sensor</code> (< 65536)
* Time synchronization (NTP) is critical for correlating calls across sensors
* Maximum allowed time difference between client and server: 2 seconds (configurable via <code>client_server_connect_maximum_time_diff_s</code>)
 
For a complete reference of all client-server parameters, see [[Sniffer_configuration#Distributed_Operation:_Client/Server_&_Mirroring|Sniffer Configuration: Distributed Operation]].
 
=== Troubleshooting: Time Synchronization Errors ===
 
If sensors repeatedly log errors such as <code>send packetbuffer block error: failed response from server - different time between server and client</code> or <code>client_server_connect_maximum_time_diff</code>, this indicates that the clock offset between the client and server has exceeded the permitted limit.
 
{| class="wikitable" style="background:#fff3cd; border:1px solid #ffc107;"
|-
! colspan="2" style="background:#ffc107;" | Time Synchronization Error
|-
| style="vertical-align: top;" | '''Error Message:'''
| <code>send packetbuffer block error: failed response from server - different time between server and client</code>
|-
| style="vertical-align: top;" | '''Meaning:'''
| The actual time difference between client and server exceeds the configured threshold. Even if both systems are configured for UTC and use the same NTP servers, clock drift, NTP polling intervals, network latency to NTP servers, or firewall restrictions on UDP port 123 can cause the offset to exceed the limit.
|}
 
==== Solution 1: Increase Time Tolerance (Immediate Workaround) ====
 
If you cannot immediately resolve the NTP synchronization precision issue, increase the allowed time difference in the configuration.
 
Add or modify the following parameters in <code>/etc/voipmonitor.conf</code> on '''both the client and the server''':


<syntaxhighlight lang="ini">
<syntaxhighlight lang="ini">
# Increase time difference tolerance (in seconds)
# /etc/voipmonitor.conf - Remote Sensor
# For persistent clock offset issues, consider values like 30 or higher
id_sensor              = 2
client_server_connect_maximum_time_diff_s = 30
server_destination      = central.server.ip
receive_packetbuffer_maximum_time_diff_s = 30
server_destination_port = 60024
onewaytimeout = 120
server_password        = your_strong_password
</syntaxhighlight>
packetbuffer_sender    = yes
 
interface              = eth0
After changing the configuration, restart the VoIPmonitor service on both nodes:
sipport                = 5060,5061
 
<syntaxhighlight lang="bash">
systemctl restart voipmonitor
</syntaxhighlight>
 
{| class="wikitable" style="background:#e8f4f8; border:1px solid #4A90E2;"
|-
! colspan="2" style="background:#4A90E2; color: white;" | Parameter Function
|-
| style="vertical-align: top;" | <code>client_server_connect_maximum_time_diff_s</code>
| (Default: 2) Controls the maximum time difference allowed during the initial client-server connection handshake.
|-
| style="vertical-align: top;" | <code>receive_packetbuffer_maximum_time_diff_s</code>
| (Default: 30) Controls the maximum time difference allowed when clients send packet buffer data (CDRs or raw packets) to the server.
|-
| style="vertical-align: top;" | <code>onewaytimeout</code>
| (Default: 15) Controls the maximum time tolerance for one-way SIP call correlation. If no reply is received from the other side within this time, the call is terminated with <code>cdr.bye = 101</code>. Increase this value (e.g., to 120 seconds) if you experience "Time different errors" with one-way SIP flows.
|}
 
==== Solution 2: Verify and Fix NTP Synchronization (Root Cause Fix) ====
 
The proper solution is to ensure NTP is synchronized with minimal clock drift.
 
'''Check NTP status on both Client and Server:'''
 
<syntaxhighlight lang="bash">
# Check system time synchronization status
timedatectl status
 
# Ensure "System clock synchronized: yes" is shown
</syntaxhighlight>
 
'''Check clock offset (Chrony):'''
 
<syntaxhighlight lang="bash">
chronyc tracking


# Look at "Last offset" and "RMS offset" values
# Filter to capture ONLY SIP packets (exclude RTP/RTCP)
# If values fluctuate near or above 2000ms (2 seconds), connections will fail
filter = port 5060 or port 5061
</syntaxhighlight>
</syntaxhighlight>


'''Check clock offset (NTP):'''
{{Note|1=The <code>filter</code> parameter using BPF syntax (tcpdump-compatible) is the recommended way to filter packets at the source in packet mirroring mode. This reduces bandwidth by forwarding only SIP packets to the central server.}}


<syntaxhighlight lang="bash">
ntpq -p


# Look at delay, offset, and jitter columns
# High jitter or offset will trigger sporadic errors
</syntaxhighlight>


'''Common NTP issues:'''
* Firewall silently dropping UDP port 123 (NTP)
* High network latency to NTP servers
* NTP service not running or misconfigured
* Different NTP server pools with divergent time


'''Verify firewall allows NTP:'''


<syntaxhighlight lang="bash">
# Check firewalld (CentOS/RHEL)
firewall-cmd --list-ports


# Check iptables/ufw (Debian/Ubuntu)
iptables -L -n -v | grep 123
# or
ufw status verbose | grep 123
</syntaxhighlight>


If needed, allow NTP traffic:


<syntaxhighlight lang="bash">
= AI Summary for RAG =
# firewalld
firewall-cmd --permanent --add-service=ntp
firewall-cmd --reload
 
# ufw
ufw allow ntp
</syntaxhighlight>


== AI Summary for RAG ==
'''Summary:''' VoIPmonitor v20+ Client-Server architecture for distributed deployments using encrypted TCP (default port 60024, zstd compression). Two modes: '''Local Processing''' (<code>packetbuffer_sender=no</code>) analyzes locally and sends CDRs only (1Gb sufficient); '''Packet Mirroring''' (<code>packetbuffer_sender=yes</code>) forwards raw packets to central server. Critical requirements: (1) exclude server_bind_port from sipport on central server (prevents memory issues); (2) sipport must match on probe and central server; (3) single sniffer must process both SIP and RTP for same call; (4) natalias only on central server. Intermediate servers supported for hub-and-spoke topology. Use <code>manager_ip</code> to bind outgoing connections to specific IP on HA setups. Sensor health via management API port 5029: <code>echo 'sniffer_stat' | nc <ip> 5029</code>. Debug SIP using Live Sniffer in GUI or sngrep on remote sensor. Stale sensor records cause "bad password" errors - delete from GUI Settings → Sensors and restart. Time sync errors: fix NTP or increase <code>client_server_connect_maximum_time_diff_s</code>.
'''Summary:''' VoIPmonitor v20+ Client-Server architecture for distributed deployments using encrypted TCP (default port 60024, zstd compression). Two modes: '''Local Processing''' (<code>packetbuffer_sender=no</code>) analyzes locally and sends CDRs only (1Gb connection sufficient); '''Packet Mirroring''' (<code>packetbuffer_sender=yes</code>) forwards raw packets to central server (bandwidth equals VoIP traffic volume). Recommended as AWS VPC Traffic Mirroring alternative to avoid VXLAN overhead and MTU fragmentation. Critical requirements: (1) server_bind_port MUST be excluded from sipport on central server (causes memory issues); (2) sipport must match on both probe and central server; (3) single sniffer instance must process both SIP and RTP for same call; (4) natalias configured only on central server, not remote sensors; (5) use <code>manager_ip</code> to bind outgoing connection to static IP on sensors with multiple interfaces (e.g., HA with floating IP). Intermediate servers supported (both server_bind and server_destination) for hub-and-spoke topology. Sensor health check via management API (port 5029): <code>echo 'sniffer_stat' | nc <sensor_ip> 5029</code>. Debug SIP traffic using Live Sniffer in GUI or sngrep on remote sensor (not central server). Version compatibility: GUI >= Sniffer recommended; use <code>server_cp_store_simple_connect_response=yes</code> for mixed versions. Time sync errors: increase <code>client_server_connect_maximum_time_diff_s</code> or fix NTP. Network bottleneck testing with iperf3. Stale sensor records cause "bad password" errors - delete from GUI Settings → Sensors and restart sensor.


'''Keywords:''' distributed architecture, client-server, packetbuffer_sender, local processing, packet mirroring, server_destination, server_bind, server_bind_port, sipport exclusion, high memory utilization, AWS VPC Traffic Mirroring, VXLAN packet loss, intermediate server, sensor health check, sniffer_stat, management API, Live Sniffer, sngrep, natalias, RTP correlation, version compatibility, time synchronization, NTP, iperf3, network throughput, stale sensor record, connection refused, mirror mode migration, manager_ip, source IP binding, static IP, floating IP, high availability, HA failover
'''Keywords:''' distributed architecture, client-server, packetbuffer_sender, local processing, packet mirroring, server_destination, server_bind, sipport exclusion, AWS VPC Traffic Mirroring alternative, intermediate server, sensor health, sniffer_stat, Live Sniffer, natalias, version compatibility, time synchronization, NTP, stale sensor record, mirror mode migration, manager_ip, high availability


'''Key Questions:'''
'''Key Questions:'''
* How do I connect multiple VoIPmonitor sensors to a central server?
* How do I connect multiple VoIPmonitor sensors to a central server?
* What is the difference between Local Processing and Packet Mirroring mode?
* What is the difference between Local Processing and Packet Mirroring mode?
* Why is VoIPmonitor using high memory on the central server in client-server mode?
* Why is VoIPmonitor using high memory on the central server?
* Why is sngrep not showing SIP packets on the central server?
* Why is a remote probe not detecting all calls on expected ports?
* How do I check VoIPmonitor sensor health status?
* How do I check VoIPmonitor sensor health status?
* Why does a new sensor fail to connect with "bad password" error despite correct credentials?
* Why does a new sensor fail with "bad password" error?
* How do I migrate from mirror mode to client-server mode?
* How do I migrate from mirror mode to client-server mode?
* What causes time synchronization errors between client and server?
* What causes time synchronization errors between client and server?
* How do I test network throughput between probe and central server?
* Where should natalias be configured in distributed deployments?
* Where should natalias be configured in distributed deployments?
* Can VoIPmonitor act as an intermediate server forwarding to a central server?
* Can VoIPmonitor act as an intermediate server?
* What is an alternative to AWS VPC Traffic Mirroring for packet capture?
* What is an alternative to AWS VPC Traffic Mirroring?
* How do I configure a remote sensor to use a specific source IP when connecting to central server?
* How do I prevent source IP confusion in HA sensor pairs with floating IPs?

Latest revision as of 20:48, 19 January 2026


This guide covers deploying multiple VoIPmonitor sensors in a distributed architecture using Client-Server mode (v20+).

For deployment options including on-host vs dedicated sensors and traffic forwarding methods (SPAN, GRE, TZSP, VXLAN), see VoIPmonitor Deployment & Topology Guide.

Overview

VoIPmonitor v20+ uses Client-Server architecture for distributed deployments. Remote sensors connect to a central server via encrypted TCP (default port 60024, zstd compression).

Mode packetbuffer_sender What is Sent Processing Location Use Case
Local Processing no (default) CDRs only Remote sensor Multi-site, low bandwidth
Packet Mirroring yes Raw packets Central server Centralized analysis, low-resource remotes

Use Cases

AWS VPC Traffic Mirroring Alternative: If experiencing packet loss with AWS VPC Traffic Mirroring (VXLAN overhead, MTU fragmentation), use client-server mode instead:

  • Install VoIPmonitor on each source EC2 instance
  • Send via encrypted TCP to central server
  • Eliminates VXLAN encapsulation and MTU issues

Configuration

Remote Sensor (Client)

id_sensor               = 2                    # Unique per sensor (1-65535)
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_strong_password

# Choose mode:
packetbuffer_sender     = no     # Local Processing: analyze locally, send CDRs
# packetbuffer_sender   = yes    # Packet Mirroring: send raw packets

interface               = eth0
sipport                 = 5060
# No MySQL credentials needed on remote sensors

💡 Tip: For HA setups with floating IPs, use manager_ip = 10.0.0.5 to bind outgoing connections to a static IP address.

Central Server

server_bind             = 0.0.0.0
server_bind_port        = 60024
server_password         = your_strong_password

mysqlhost               = localhost
mysqldb                 = voipmonitor
mysqluser               = voipmonitor
mysqlpassword           = db_password

# If receiving raw packets (packetbuffer_sender=yes on clients):
sipport                 = 5060
savertp                 = yes
savesip                 = yes

⚠️ Warning: Critical: Exclude server_bind_port from sipport on the central server. Including it causes continuously increasing memory usage. 

# WRONG - includes sensor communication port:
sipport = 1-65535

# CORRECT - excludes port 60024:
sipport = 1-60023,60025-65535

Key Configuration Rules

Rule Applies To Why
server_bind_port must match server_destination_port Both Connection fails if mismatched
sipport must match on probe and central server Packet Mirroring Missing ports = missing calls
natalias only on central server Packet Mirroring Prevents RTP correlation issues
Each sensor needs unique id_sensor All Required for identification

Local Processing vs Packet Mirroring

Local Processing Packet Mirroring
packetbuffer_sender no (default) yes
Processing location Remote sensor Central server
PCAP storage Remote sensor Central server
WAN bandwidth Low (CDRs only, 1Gb sufficient) High (full packets)
Remote CPU load Higher Minimal
Capture rules applied On sensor On central server only

PCAP Access in Local Processing Mode

PCAPs are stored on remote sensors. The GUI retrieves them through the central server, which proxies the request to the sensor over the existing TCP/60024 connection - the same persistent encrypted channel the sensor uses for sending CDRs. This connection is bidirectional; the central server does not open any separate connection back to the sensor.

Firewall requirements:

Direction Port Purpose
Remote sensors → Central server TCP/60024 Persistent encrypted channel (CDRs from sensor, PCAP requests from server - bidirectional)
GUI → Central server TCP/5029 Manager API (sensor status, active calls, configuration)
GUI → Central server TCP/60024 Server API (list connected sensors, proxy PCAP retrieval)

ℹ️ Note: The central server does not initiate connections to remote sensors. All server↔sensor communication happens over the single TCP/60024 connection that the sensor established.

💡 Tip: Packet Mirroring (packetbuffer_sender=yes) automatically deduplicates calls - the central server merges packets from all probes for the same Call-ID into a single unified CDR. This also ensures one logical call only consumes one license channel.

Advanced Topics

High Availability (Failover)

Remote sensors can specify multiple central servers: 

server_destination = 192.168.0.1, 192.168.0.2

If primary is unavailable, the sensor automatically connects to the next server.

Connection Compression

# On both client and server (default: zstd)
server_type_compress = zstd   # Options: zstd, gzip, lzo, none

Intermediate Server (Hub-and-Spoke)

An intermediate server can receive from multiple sensors and forward to a central server: 

# On INTERMEDIATE SERVER
id_sensor               = 100

# Receive from remote sensors
server_bind             = 0.0.0.0
server_bind_port        = 60024
server_password         = sensor_password

# Forward to central server
server_destination      = central.server.ip
server_destination_port = 60024

packetbuffer_sender     = no    # or yes, depending on desired mode

ℹ️ Note: This works because the intermediate server does NOT do local packet capture - it only relays. Original remote sensors must be manually added to GUI Settings for visibility.

Multiple Receivers for Packet Mirroring

⚠️ Warning: Multiple sensors with packetbuffer_sender=yes sending to a single receiver instance can cause call processing conflicts (calls appear in Active Calls but missing from CDRs).

Solution: Run separate receiver instances on different hosts, each dedicated to specific sensors: 

# Receiver Instance 1 (Host 1, for Sensor A)
server_bind_port        = 60024
id_sensor               = 1

# Receiver Instance 2 (Host 2, for Sensor B)
server_bind_port        = 60024
id_sensor               = 2

Alternative: Use Local Processing mode (packetbuffer_sender=no) which processes calls independently on each sensor.

Preventing Duplicate CDRs (Local Processing)

When multiple probes capture the same call in Local Processing mode: 

# On each probe
cdr_check_exists_callid = yes

This checks for existing CDRs before inserting. Requires MySQL UPDATE privileges.

Critical: SIP and RTP Must Be Captured Together

VoIPmonitor cannot correlate SIP and RTP from different sniffer instances. A single sniffer must process both SIP and RTP for each call. Parameters like cdr_check_exists_callid do NOT enable split SIP/RTP correlation.


Split SIP/RTP with Packet Mirroring Mode

ℹ️ Note: Exception for Packet Mirroring Mode:: The above limitation applies to Local Processing mode (packetbuffer_sender=no) where each sensor processes calls independently. In Packet Mirroring mode (packetbuffer_sender=yes), the central server receives raw packets from multiple remote sensors and processes them together. This allows scenarios where SIP and RTP are captured on separate nodes - configure both as packet senders and let the central server correlate them into single unified CDRs.

Example scenario: Separate SIP signaling node and RTP handling node: 

# SIP Signaling Node (packet sender)
id_sensor               = 1
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

# RTP Handling Node (packet sender)
id_sensor               = 2
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

The central server merges packets from both senders by Call-ID, creating unified CDRs with complete SIP and RTP data.


HEP Protocol in Client/Server Mode

VoIPmonitor supports receiving HEP-encapsulated traffic on sniffer clients and forwarding it to a central server. This enables distributed capture from HEP sources (Kamailio, OpenSIPS, rtpproxy, FreeSWITCH) in a client/server architecture.

Scenario: SIP proxy and RTP proxy at different locations sending HEP to remote sniffer clients: 

# Remote Sniffer Client A (receives HEP from Kamailio)
id_sensor               = 1
hep                     = yes
hep_bind_port           = 9060
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

# Remote Sniffer Client B (receives HEP from rtpproxy)
id_sensor               = 2
hep                     = yes
hep_bind_port           = 9060
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

The central server receives packets from both clients and correlates them into unified CDRs using standard SIP Call-ID and IP:port from SDP.

ℹ️ Note: This also works for IPFIX (Oracle SBCs) and RibbonSBC protocols forwarded via client/server mode.

Alternative: Direct HEP to single sniffer

If both HEP sources can reach the same sniffer directly, no client/server setup is needed: 

# Single sniffer receiving HEP from multiple sources
hep                     = yes
hep_bind_port           = 9060
interface               = eth0   # Can also sniff locally if needed

Both Kamailio (SIP) and rtpproxy (RTP) send HEP to this sniffer on port 9060. The sniffer correlates them automatically based on Call-ID and SDP IP:port.

Sensor Health Monitoring

Management API

Query sensor status via TCP port 5029: 

echo 'sniffer_stat' | nc <sensor_ip> 5029

Returns JSON with status, version, active calls, packets per second, etc.

Multi-Sensor Health Check Script

#!/bin/bash
SENSORS=("192.168.1.10:5029" "192.168.1.11:5029")
for SENSOR in "${SENSORS[@]}"; do
    IP=$(echo $SENSOR | cut -d: -f1)
    PORT=$(echo $SENSOR | cut -d: -f2)
    STATUS=$(echo 'sniffer_stat' | nc -w 2 $IP $PORT 2>/dev/null | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
    echo "$IP: ${STATUS:-FAILED}"
done

Version Compatibility

Scenario Compatibility Notes
GUI ≥ Sniffer ✅ Compatible Recommended
GUI < Sniffer ⚠️ Risk Sensor may write to non-existent columns

Best practice: Upgrade GUI first (applies schema changes), then upgrade sensors.

For mixed versions temporarily, add to central server: 

server_cp_store_simple_connect_response = yes   # Sniffer 2024.11.0+

Troubleshooting

Quick Diagnosis

Symptom First Check Likely Cause
Sensor not connecting journalctl -u voipmonitor -f on sensor Check server_destination, password, firewall
Traffic rate [0.0Mb/s] tcpdump on sensor interface Network/SPAN issue, not communication
High memory on central server Check if sipport includes 60024 Exclude server port from sipport
Missing calls Compare sipport on probe vs central Must match on both sides
"Bad password" error GUI → Settings → Sensors Delete stale sensor record, restart sensor
"Connection refused (111)" after migration Check server_destination in config Points to old server IP
RTP streams end prematurely Check natalias location Configure only on central server
Time sync errors timedatectl status Fix NTP or increase tolerance

Connection Testing

# Test connectivity from sensor to server
nc -zv <server_ip> 60024

# Verify server is listening
ss -tulpn | grep voipmonitor

# Check sensor logs
journalctl -u voipmonitor -n 100 | grep -i "connect"

Time Synchronization Errors

If seeing "different time between server and client" errors:

Immediate workaround: Increase tolerance on both sides: 

client_server_connect_maximum_time_diff_s = 30
receive_packetbuffer_maximum_time_diff_s = 30

Root cause fix: Ensure NTP is working: 

timedatectl status           # Check sync status
chronyc tracking             # Check offset (Chrony)
ntpq -p                      # Check offset (NTP)

Network Throughput Testing

If experiencing "packetbuffer: MEMORY IS FULL" errors, test network with iperf3: 

# On central server
iperf3 -s

# On probe
iperf3 -c <server_ip>
Result Interpretation Action
Expected bandwidth (>900 Mbps on 1Gb) Network OK Check local CPU/RAM
Low throughput Network bottleneck Check switches, cabling, consider Local Processing mode

Debugging SIP Traffic

sngrep does not work on the central server because traffic is encapsulated in the TCP tunnel.

Options:

  • Live Sniffer: Use GUI → Live Sniffer to view SIP from remote sensors
  • sngrep on sensor: Run sngrep -i eth0 directly on the remote sensor

Stale Sensor Records

If a new sensor fails with "bad password" despite correct credentials:

  1. Delete the sensor record from GUI → Settings → Sensors
  2. Restart voipmonitor on the sensor: systemctl restart voipmonitor
  3. The sensor will re-register automatically

Legacy: Mirror Mode

The older mirror_destination/mirror_bind options still work but Client-Server mode is preferred (encryption, simpler management).

To migrate from mirror mode:

  1. Stop sensors, comment out mirror_* parameters
  2. Configure server_bind on central, server_destination on sensors
  3. Restart all services

For mirror mode id_sensor attribution, use: 

# On central receiver
mirror_bind_sensor_id_by_sender = yes

See Also

Filtering Options in Packet Mirroring Mode

ℹ️ Note: Important distinction: In Packet Mirroring mode (packetbuffer_sender=yes):

  • Capture rules (GUI-based): Applied ONLY on the central server
  • BPF filters / IP filters: CAN be applied on the remote sensor to reduce bandwidth

Use the following options on the remote sensor to filter traffic BEFORE sending to the central server: 

# On REMOTE SENSOR (client)

# Option 1: BPF filter (tcpdump syntax) - most flexible
filter = not net 192.168.0.0/16 and not net 10.0.0.0/8

# Option 2: IP allow-list filter - CPU-efficient, no negation support
interface_ip_filter = 192.168.1.0/24
interface_ip_filter = 10.0.0.0/8

Benefits of filtering on remote sensor:

  • Reduces WAN bandwidth usage between sensor and central server
  • Reduces processing load on central server
  • Use filter for complex conditions (tcpdump/BPF syntax)
  • Use interface_ip_filter for simple IP allow-lists (more efficient)

Filtering approaches:

  • For SIP header-based filtering: Apply capture rules on the central server only
  • For IP/subnet filtering: Use filter or interface_ip_filter on remote sensor

Supported Configuration Options in Packet Mirroring Mode

In Packet Mirroring mode (packetbuffer_sender = yes), the remote sensor forwards raw packets without processing them. This means many configuration options that manipulate packet behavior are unsupported on the remote sensor.

Supported Options on Remote Sensor (packetbuffer_sender)

The following options work correctly on the remote sensor in packet mirroring mode:

Parameter Description
id_sensor Unique sensor identifier
server_destination Central server address
server_destination_port Central server port (default 60024)
server_password Authentication password
server_destination_timeout Connection timeout settings
server_destination_reconnect Auto-reconnect behavior
filter BPF filter to limit capture (use this to capture only SIP)
interface_ip_filter IP-based packet filtering
interface Capture interface
sipport SIP ports to monitor
promisc Promiscuous mode
rrd RRD statistics
spooldir Temporary packet buffer directory
ringbuffer Ring buffer size for packet mirroring
max_buffer_mem Maximum buffer memory
packetbuffer_enable Enable packet buffering
packetbuffer_compress Enable compression for forwarded packets
packetbuffer_compress_ratio Compression ratio

Unsupported Options on Remote Sensor

The following options do NOT work on the remote sensor in packet mirroring mode because the sensor does not parse packets:

Parameter Reason
natalias NAT alias handling (configure on central server instead)
rtp_check_both_sides_by_sdp RTP correlation requires packet parsing
disable_process_sdp SDP processing happens on central server
save_sdp_ipport SDP extraction happens on central server
rtpfromsdp_onlysip RTP mapping requires packet parsing
rtpip_find_endpoints Endpoint discovery requires packet parsing

⚠️ Warning: Critical: Storage options (savesip, savertp, saveaudio) must be configured on the CENTRAL SERVER in packet mirroring mode. The remote sensor only forwards packets and does not perform any storage operations.

SIP-Only Capture Example

To capture and forward only SIP packets (excluding RTP/RTCP) for security or compliance: 

# /etc/voipmonitor.conf - Remote Sensor
id_sensor               = 2
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_strong_password
packetbuffer_sender     = yes
interface               = eth0
sipport                 = 5060,5061

# Filter to capture ONLY SIP packets (exclude RTP/RTCP)
filter = port 5060 or port 5061

ℹ️ Note: The filter parameter using BPF syntax (tcpdump-compatible) is the recommended way to filter packets at the source in packet mirroring mode. This reduces bandwidth by forwarding only SIP packets to the central server.





AI Summary for RAG

Summary: VoIPmonitor v20+ Client-Server architecture for distributed deployments using encrypted TCP (default port 60024, zstd compression). Two modes: Local Processing (packetbuffer_sender=no) analyzes locally and sends CDRs only (1Gb sufficient); Packet Mirroring (packetbuffer_sender=yes) forwards raw packets to central server. Critical requirements: (1) exclude server_bind_port from sipport on central server (prevents memory issues); (2) sipport must match on probe and central server; (3) single sniffer must process both SIP and RTP for same call; (4) natalias only on central server. Intermediate servers supported for hub-and-spoke topology. Use manager_ip to bind outgoing connections to specific IP on HA setups. Sensor health via management API port 5029: echo 'sniffer_stat' | nc <ip> 5029. Debug SIP using Live Sniffer in GUI or sngrep on remote sensor. Stale sensor records cause "bad password" errors - delete from GUI Settings → Sensors and restart. Time sync errors: fix NTP or increase client_server_connect_maximum_time_diff_s.

Keywords: distributed architecture, client-server, packetbuffer_sender, local processing, packet mirroring, server_destination, server_bind, sipport exclusion, AWS VPC Traffic Mirroring alternative, intermediate server, sensor health, sniffer_stat, Live Sniffer, natalias, version compatibility, time synchronization, NTP, stale sensor record, mirror mode migration, manager_ip, high availability

Key Questions:

  • How do I connect multiple VoIPmonitor sensors to a central server?
  • What is the difference between Local Processing and Packet Mirroring mode?
  • Why is VoIPmonitor using high memory on the central server?
  • Why is a remote probe not detecting all calls on expected ports?
  • How do I check VoIPmonitor sensor health status?
  • Why does a new sensor fail with "bad password" error?
  • How do I migrate from mirror mode to client-server mode?
  • What causes time synchronization errors between client and server?
  • Where should natalias be configured in distributed deployments?
  • Can VoIPmonitor act as an intermediate server?
  • What is an alternative to AWS VPC Traffic Mirroring?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Sniffer_distributed_architecture&oldid=10813"