Sniffer distributed architecture

This guide covers deploying multiple VoIPmonitor sensors in a distributed architecture using Client-Server mode (v20+).

For deployment options including on-host vs dedicated sensors and traffic forwarding methods (SPAN, GRE, TZSP, VXLAN), see VoIPmonitor Deployment & Topology Guide.

Overview

VoIPmonitor v20+ uses Client-Server architecture for distributed deployments. Remote sensors connect to a central server via encrypted TCP (default port 60024, zstd compression).

Mode	packetbuffer_sender	What is Sent	Processing Location	Use Case
Local Processing	no (default)	CDRs only	Remote sensor	Multi-site, low bandwidth
Packet Mirroring	yes	Raw packets	Central server	Centralized analysis, low-resource remotes

Use Cases

AWS VPC Traffic Mirroring Alternative: If experiencing packet loss with AWS VPC Traffic Mirroring (VXLAN overhead, MTU fragmentation), use client-server mode instead:

• Install VoIPmonitor on each source EC2 instance
• Send via encrypted TCP to central server
• Eliminates VXLAN encapsulation and MTU issues

Configuration

Remote Sensor (Client)

id_sensor               = 2                    # Unique per sensor (1-65535)
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_strong_password

# Choose mode:
packetbuffer_sender     = no     # Local Processing: analyze locally, send CDRs
# packetbuffer_sender   = yes    # Packet Mirroring: send raw packets

interface               = eth0
sipport                 = 5060
# No MySQL credentials needed on remote sensors

💡 Tip: For HA setups with floating IPs, use manager_ip = 10.0.0.5 to bind outgoing connections to a static IP address.

Central Server

server_bind             = 0.0.0.0
server_bind_port        = 60024
server_password         = your_strong_password

mysqlhost               = localhost
mysqldb                 = voipmonitor
mysqluser               = voipmonitor
mysqlpassword           = db_password

# If receiving raw packets (packetbuffer_sender=yes on clients):
sipport                 = 5060
savertp                 = yes
savesip                 = yes

⚠️ Warning: Critical: Exclude server_bind_port from sipport on the central server. Including it causes continuously increasing memory usage.

# WRONG - includes sensor communication port:
sipport = 1-65535

# CORRECT - excludes port 60024:
sipport = 1-60023,60025-65535

Key Configuration Rules

Rule	Applies To	Why
server_bind_port must match server_destination_port	Both	Connection fails if mismatched
sipport must match on probe and central server	Packet Mirroring	Missing ports = missing calls
natalias only on central server	Packet Mirroring	Prevents RTP correlation issues
Each sensor needs unique id_sensor	All	Required for identification

Local Processing vs Packet Mirroring

	Local Processing	Packet Mirroring
packetbuffer_sender	no (default)	yes
Processing location	Remote sensor	Central server
PCAP storage	Remote sensor	Central server
WAN bandwidth	Low (CDRs only, 1Gb sufficient)	High (full packets)
Remote CPU load	Higher	Minimal
Capture rules applied	On sensor	On central server only

PCAP Access in Local Processing Mode

PCAPs are stored on remote sensors. The GUI retrieves them through the central server, which proxies the request to the sensor over the existing TCP/60024 connection - the same persistent encrypted channel the sensor uses for sending CDRs. This connection is bidirectional; the central server does not open any separate connection back to the sensor.

Firewall requirements:

Direction	Port	Purpose
Remote sensors → Central server	TCP/60024	Persistent encrypted channel (CDRs from sensor, PCAP requests from server - bidirectional)
GUI → Central server	TCP/5029	Manager API (sensor status, active calls, configuration)
GUI → Central server	TCP/60024	Server API (list connected sensors, proxy PCAP retrieval)

ℹ️ Note: The central server does not initiate connections to remote sensors. All server↔sensor communication happens over the single TCP/60024 connection that the sensor established.

💡 Tip: Packet Mirroring (packetbuffer_sender=yes) automatically deduplicates calls - the central server merges packets from all probes for the same Call-ID into a single unified CDR. This also ensures one logical call only consumes one license channel.

Advanced Topics

High Availability (Failover)

Remote sensors can specify multiple central servers:

server_destination = 192.168.0.1, 192.168.0.2

If primary is unavailable, the sensor automatically connects to the next server.

Connection Compression

# On both client and server (default: zstd)
server_type_compress = zstd   # Options: zstd, gzip, lzo, none

Intermediate Server (Hub-and-Spoke)

An intermediate server can receive from multiple sensors and forward to a central server:

# On INTERMEDIATE SERVER
id_sensor               = 100

# Receive from remote sensors
server_bind             = 0.0.0.0
server_bind_port        = 60024
server_password         = sensor_password

# Forward to central server
server_destination      = central.server.ip
server_destination_port = 60024

packetbuffer_sender     = no    # or yes, depending on desired mode

ℹ️ Note: This works because the intermediate server does NOT do local packet capture - it only relays. Original remote sensors must be manually added to GUI Settings for visibility.

Multiple Receivers for Packet Mirroring

⚠️ Warning: Multiple sensors with packetbuffer_sender=yes sending to a single receiver instance can cause call processing conflicts (calls appear in Active Calls but missing from CDRs).

Solution: Run separate receiver instances on different hosts, each dedicated to specific sensors:

# Receiver Instance 1 (Host 1, for Sensor A)
server_bind_port        = 60024
id_sensor               = 1

# Receiver Instance 2 (Host 2, for Sensor B)
server_bind_port        = 60024
id_sensor               = 2

Alternative: Use Local Processing mode (packetbuffer_sender=no) which processes calls independently on each sensor.

Preventing Duplicate CDRs (Local Processing)

When multiple probes capture the same call in Local Processing mode:

# On each probe
cdr_check_exists_callid = yes

This checks for existing CDRs before inserting. Requires MySQL UPDATE privileges.

Critical: SIP and RTP Must Be Captured Together

VoIPmonitor cannot correlate SIP and RTP from different sniffer instances. A single sniffer must process both SIP and RTP for each call. Parameters like cdr_check_exists_callid do NOT enable split SIP/RTP correlation.

Split SIP/RTP with Packet Mirroring Mode

ℹ️ Note: Exception for Packet Mirroring Mode:: The above limitation applies to Local Processing mode (packetbuffer_sender=no) where each sensor processes calls independently. In Packet Mirroring mode (packetbuffer_sender=yes), the central server receives raw packets from multiple remote sensors and processes them together. This allows scenarios where SIP and RTP are captured on separate nodes - configure both as packet senders and let the central server correlate them into single unified CDRs.

Example scenario: Separate SIP signaling node and RTP handling node:

# SIP Signaling Node (packet sender)
id_sensor               = 1
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

# RTP Handling Node (packet sender)
id_sensor               = 2
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

The central server merges packets from both senders by Call-ID, creating unified CDRs with complete SIP and RTP data.

HEP Protocol in Client/Server Mode

VoIPmonitor supports receiving HEP-encapsulated traffic on sniffer clients and forwarding it to a central server. This enables distributed capture from HEP sources (Kamailio, OpenSIPS, rtpproxy, FreeSWITCH) in a client/server architecture.

Scenario: SIP proxy and RTP proxy at different locations sending HEP to remote sniffer clients:

# Remote Sniffer Client A (receives HEP from Kamailio)
id_sensor               = 1
hep                     = yes
hep_bind_port           = 9060
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

# Remote Sniffer Client B (receives HEP from rtpproxy)
id_sensor               = 2
hep                     = yes
hep_bind_port           = 9060
packetbuffer_sender     = yes
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_password

The central server receives packets from both clients and correlates them into unified CDRs using standard SIP Call-ID and IP:port from SDP.

ℹ️ Note: This also works for IPFIX (Oracle SBCs) and RibbonSBC protocols forwarded via client/server mode.

Alternative: Direct HEP to single sniffer

If both HEP sources can reach the same sniffer directly, no client/server setup is needed:

# Single sniffer receiving HEP from multiple sources
hep                     = yes
hep_bind_port           = 9060
interface               = eth0   # Can also sniff locally if needed

Both Kamailio (SIP) and rtpproxy (RTP) send HEP to this sniffer on port 9060. The sniffer correlates them automatically based on Call-ID and SDP IP:port.

Sensor Health Monitoring

Management API

Query sensor status via TCP port 5029:

echo 'sniffer_stat' | nc <sensor_ip> 5029

Returns JSON with status, version, active calls, packets per second, etc.

Multi-Sensor Health Check Script

#!/bin/bash
SENSORS=("192.168.1.10:5029" "192.168.1.11:5029")
for SENSOR in "${SENSORS[@]}"; do
    IP=$(echo $SENSOR | cut -d: -f1)
    PORT=$(echo $SENSOR | cut -d: -f2)
    STATUS=$(echo 'sniffer_stat' | nc -w 2 $IP $PORT 2>/dev/null | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
    echo "$IP: ${STATUS:-FAILED}"
done

Version Compatibility

Scenario	Compatibility	Notes
GUI ≥ Sniffer	✅ Compatible	Recommended
GUI < Sniffer	⚠️ Risk	Sensor may write to non-existent columns

Best practice: Upgrade GUI first (applies schema changes), then upgrade sensors.

For mixed versions temporarily, add to central server:

server_cp_store_simple_connect_response = yes   # Sniffer 2024.11.0+

Troubleshooting

Quick Diagnosis

Symptom	First Check	Likely Cause
Sensor not connecting	journalctl -u voipmonitor -f on sensor	Check server_destination, password, firewall
Traffic rate [0.0Mb/s]	tcpdump on sensor interface	Network/SPAN issue, not communication
High memory on central server	Check if sipport includes 60024	Exclude server port from sipport
Missing calls	Compare sipport on probe vs central	Must match on both sides
"Bad password" error	GUI → Settings → Sensors	Delete stale sensor record, restart sensor
"Connection refused (111)" after migration	Check server_destination in config	Points to old server IP
RTP streams end prematurely	Check natalias location	Configure only on central server
Time sync errors	timedatectl status	Fix NTP or increase tolerance

Connection Testing

# Test connectivity from sensor to server
nc -zv <server_ip> 60024

# Verify server is listening
ss -tulpn | grep voipmonitor

# Check sensor logs
journalctl -u voipmonitor -n 100 | grep -i "connect"

Time Synchronization Errors

If seeing "different time between server and client" errors:

Immediate workaround: Increase tolerance on both sides:

client_server_connect_maximum_time_diff_s = 30
receive_packetbuffer_maximum_time_diff_s = 30

Root cause fix: Ensure NTP is working:

timedatectl status           # Check sync status
chronyc tracking             # Check offset (Chrony)
ntpq -p                      # Check offset (NTP)

Network Throughput Testing

If experiencing "packetbuffer: MEMORY IS FULL" errors, test network with iperf3:

# On central server
iperf3 -s

# On probe
iperf3 -c <server_ip>

Result	Interpretation	Action
Expected bandwidth (>900 Mbps on 1Gb)	Network OK	Check local CPU/RAM
Low throughput	Network bottleneck	Check switches, cabling, consider Local Processing mode

Debugging SIP Traffic

sngrep does not work on the central server because traffic is encapsulated in the TCP tunnel.

Options:

• Live Sniffer: Use GUI → Live Sniffer to view SIP from remote sensors
• sngrep on sensor: Run sngrep -i eth0 directly on the remote sensor

Stale Sensor Records

If a new sensor fails with "bad password" despite correct credentials:

1. Delete the sensor record from GUI → Settings → Sensors
2. Restart voipmonitor on the sensor: systemctl restart voipmonitor
3. The sensor will re-register automatically

Legacy: Mirror Mode

The older mirror_destination/mirror_bind options still work but Client-Server mode is preferred (encryption, simpler management).

To migrate from mirror mode:

1. Stop sensors, comment out mirror_* parameters
2. Configure server_bind on central, server_destination on sensors
3. Restart all services

For mirror mode id_sensor attribution, use:

# On central receiver
mirror_bind_sensor_id_by_sender = yes

See Also

• Deployment & Topology Guide - Traffic forwarding methods
• Sniffer Configuration - All parameters reference
• Call Correlation - Multi-leg call handling
• FAQ: One GUI for Multiple Sniffers

Filtering Options in Packet Mirroring Mode

ℹ️ Note: Important distinction: In Packet Mirroring mode (packetbuffer_sender=yes):

• Capture rules (GUI-based): Applied ONLY on the central server
• BPF filters / IP filters: CAN be applied on the remote sensor to reduce bandwidth

Use the following options on the remote sensor to filter traffic BEFORE sending to the central server:

# On REMOTE SENSOR (client)

# Option 1: BPF filter (tcpdump syntax) - most flexible
filter = not net 192.168.0.0/16 and not net 10.0.0.0/8

# Option 2: IP allow-list filter - CPU-efficient, no negation support
interface_ip_filter = 192.168.1.0/24
interface_ip_filter = 10.0.0.0/8

Benefits of filtering on remote sensor:

• Reduces WAN bandwidth usage between sensor and central server
• Reduces processing load on central server
• Use filter for complex conditions (tcpdump/BPF syntax)
• Use interface_ip_filter for simple IP allow-lists (more efficient)

Filtering approaches:

• For SIP header-based filtering: Apply capture rules on the central server only
• For IP/subnet filtering: Use filter or interface_ip_filter on remote sensor

Supported Configuration Options in Packet Mirroring Mode

In Packet Mirroring mode (packetbuffer_sender = yes), the remote sensor forwards raw packets without processing them. This means many configuration options that manipulate packet behavior are unsupported on the remote sensor.

Supported Options on Remote Sensor (packetbuffer_sender)

The following options work correctly on the remote sensor in packet mirroring mode:

Parameter	Description
id_sensor	Unique sensor identifier
server_destination	Central server address
server_destination_port	Central server port (default 60024)
server_password	Authentication password
server_destination_timeout	Connection timeout settings
server_destination_reconnect	Auto-reconnect behavior
filter	BPF filter to limit capture (use this to capture only SIP)
interface_ip_filter	IP-based packet filtering
interface	Capture interface
sipport	SIP ports to monitor
promisc	Promiscuous mode
rrd	RRD statistics
spooldir	Temporary packet buffer directory
ringbuffer	Ring buffer size for packet mirroring
max_buffer_mem	Maximum buffer memory
packetbuffer_enable	Enable packet buffering
packetbuffer_compress	Enable compression for forwarded packets
packetbuffer_compress_ratio	Compression ratio

Unsupported Options on Remote Sensor

The following options do NOT work on the remote sensor in packet mirroring mode because the sensor does not parse packets:

Parameter	Reason
natalias	NAT alias handling (configure on central server instead)
rtp_check_both_sides_by_sdp	RTP correlation requires packet parsing
disable_process_sdp	SDP processing happens on central server
save_sdp_ipport	SDP extraction happens on central server
rtpfromsdp_onlysip	RTP mapping requires packet parsing
rtpip_find_endpoints	Endpoint discovery requires packet parsing

⚠️ Warning: Critical: Storage options (savesip, savertp, saveaudio) must be configured on the CENTRAL SERVER in packet mirroring mode. The remote sensor only forwards packets and does not perform any storage operations.

SIP-Only Capture Example

To capture and forward only SIP packets (excluding RTP/RTCP) for security or compliance:

# /etc/voipmonitor.conf - Remote Sensor
id_sensor               = 2
server_destination      = central.server.ip
server_destination_port = 60024
server_password         = your_strong_password
packetbuffer_sender     = yes
interface               = eth0
sipport                 = 5060,5061

# Filter to capture ONLY SIP packets (exclude RTP/RTCP)
filter = port 5060 or port 5061

ℹ️ Note: The filter parameter using BPF syntax (tcpdump-compatible) is the recommended way to filter packets at the source in packet mirroring mode. This reduces bandwidth by forwarding only SIP packets to the central server.

AI Summary for RAG

Summary: VoIPmonitor v20+ Client-Server architecture for distributed deployments using encrypted TCP (default port 60024, zstd compression). Two modes: Local Processing (packetbuffer_sender=no) analyzes locally and sends CDRs only (1Gb sufficient); Packet Mirroring (packetbuffer_sender=yes) forwards raw packets to central server. Critical requirements: (1) exclude server_bind_port from sipport on central server (prevents memory issues); (2) sipport must match on probe and central server; (3) single sniffer must process both SIP and RTP for same call; (4) natalias only on central server. Intermediate servers supported for hub-and-spoke topology. Use manager_ip to bind outgoing connections to specific IP on HA setups. Sensor health via management API port 5029: echo 'sniffer_stat' | nc <ip> 5029. Debug SIP using Live Sniffer in GUI or sngrep on remote sensor. Stale sensor records cause "bad password" errors - delete from GUI Settings → Sensors and restart. Time sync errors: fix NTP or increase client_server_connect_maximum_time_diff_s.

Keywords: distributed architecture, client-server, packetbuffer_sender, local processing, packet mirroring, server_destination, server_bind, sipport exclusion, AWS VPC Traffic Mirroring alternative, intermediate server, sensor health, sniffer_stat, Live Sniffer, natalias, version compatibility, time synchronization, NTP, stale sensor record, mirror mode migration, manager_ip, high availability

Key Questions:

• How do I connect multiple VoIPmonitor sensors to a central server?
• What is the difference between Local Processing and Packet Mirroring mode?
• Why is VoIPmonitor using high memory on the central server?
• Why is a remote probe not detecting all calls on expected ports?
• How do I check VoIPmonitor sensor health status?
• Why does a new sensor fail with "bad password" error?
• How do I migrate from mirror mode to client-server mode?
• What causes time synchronization errors between client and server?
• Where should natalias be configured in distributed deployments?
• Can VoIPmonitor act as an intermediate server?
• What is an alternative to AWS VPC Traffic Mirroring?