Alerts: Difference between revisions

From VoIPmonitor.org
(Add troubleshooting section for concurrent calls alerts not triggering (SQL queue, CDR timing, Check interval, fraud vs regular concurrent calls differences))
(Fix template syntax - remove curly braces from 'from' and 'to' field names)
 
(28 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{DISPLAYTITLE:Alerts & Reports}}
{{DISPLAYTITLE:Alerts & Reports}}
Category:GUI manual
[[Category:GUI manual]]


== Alerts & Reports ==
= Alerts & Reports =


Alerts & Reports generate email notifications based on QoS parameters or SIP error conditions. The system includes daily reports, ad hoc reports, and stores all generated items in history.
Email notifications triggered by QoS thresholds, SIP errors, or sensor health conditions. The system stores all alerts in history for review.
 
=== Overview ===
 
The alert system monitors call quality and SIP signaling in real-time, triggering notifications when configured thresholds are exceeded.


<kroki lang="plantuml">
<kroki lang="plantuml">
Line 14: Line 10:
skinparam shadowing false
skinparam shadowing false
skinparam defaultFontName Arial
skinparam defaultFontName Arial
 
rectangle "Sensor" as sensor
rectangle "VoIPmonitor\nSensor" as sensor
database "MySQL" as db
database "MySQL\nDatabase" as db
rectangle "Cron\n(1min)" as cron
rectangle "Cron Job\n(every minute)" as cron
rectangle "Alert\nProcessor" as processor
rectangle "Alert\nProcessor" as processor
rectangle "MTA\n(Postfix/Exim)" as mta
rectangle "MTA" as mta
actor "Admin" as admin
actor "Admin" as admin
 
sensor --> db : CDRs
sensor --> db : CDRs with\nQoS metrics
cron --> processor : Trigger
cron --> processor : Trigger
processor --> db : Query alerts\n& CDRs
processor --> db : Query
processor --> mta : Send email
processor --> mta : Email
mta --> admin : Alert notification
mta --> admin : Alert
@enduml
@enduml
</kroki>
</kroki>


=== Email Configuration Prerequisites ===
== Prerequisites ==


Emails are sent using PHP's <code>mail()</code> function, which relies on the server's Mail Transfer Agent (MTA) such as Exim, Postfix, or Sendmail. Configure your MTA according to your Linux distribution documentation.
=== Email Configuration ===


==== Setting Up the Cron Job ====
Alerts use PHP's <code>mail()</code> function via the server's MTA (Postfix/Exim/Sendmail).


Alert processing requires a cron job that runs every minute:
{| class="wikitable"
|-
! Setting !! Location !! Description
|-
| From Address || GUI > Settings > System Configuration > Email || <code>DEFAULT_EMAIL_FROM</code> - sender address for all alerts
|-
| Cron Job || <code>/etc/crontab</code> || Required for alert processing
|}


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Add to /etc/crontab (adjust path based on your GUI installation)
# Add cron job (required)
echo "* * * * * root php /var/www/html/php/run.php cron" >> /etc/crontab
echo "* * * * * root php /var/www/html/php/run.php cron" >> /etc/crontab
# Reload crontab
killall -HUP cron  # Debian/Ubuntu
killall -HUP cron  # Debian/Ubuntu
# or
# or: killall -HUP crond  # RHEL/CentOS
killall -HUP crond  # CentOS/RHEL
</syntaxhighlight>
</syntaxhighlight>


=== Configure Alerts ===
== Alert Types ==


Email alerts can trigger on SIP protocol events or RTP QoS metrics. Access alerts configuration via '''GUI > Alerts'''.
Access via '''GUI > Alerts'''.


[[File:alertgrid.png|frame|center|Alert configuration grid]]
=== RTP Alerts ===


==== Alert Types ====
Trigger on voice quality metrics:
 
* '''MOS''' - below threshold
===== RTP Alerts =====
 
RTP alerts trigger based on voice quality metrics:
* '''MOS''' (Mean Opinion Score) - below threshold
* '''Packet loss''' - percentage exceeded
* '''Packet loss''' - percentage exceeded
* '''Jitter''' - variation exceeded
* '''Jitter''' - variation exceeded
* '''Delay''' (PDV) - latency exceeded
* '''Delay''' (PDV) - latency exceeded
* '''One-way calls''' - answered but one RTP stream missing
* '''One-way calls''' - one RTP stream missing
* '''Missing RTP''' - answered but both RTP streams missing
* '''Missing RTP''' - both RTP streams missing


Configure alerts to trigger when:
Configure alerts to trigger when number of incidents OR percentage of CDRs exceeds threshold.
* Number of incidents exceeds a set value, OR
* Percentage of CDRs exceeds a threshold


[[File:alertrtpform.png|frame|center|RTP alert configuration form]]
=== RTP&CDR Alerts ===


===== SIP Response Alerts =====
Combine RTP metrics with CDR conditions including '''PDD (Post Dial Delay)'''.


SIP response alerts trigger based on SIP response codes:
'''Using Filter Templates:'''
* '''Empty response field''': Matches all call attempts per configured filters
# Create CDR filter in '''GUI > CDR'''
* '''Response code 0''': Matches unreplied INVITE requests (no response received)
# Save as template
* '''Specific codes''': Match exact codes like 404, 503, etc.
# In alert config, select from '''Filter template''' dropdown


[[File:alertsipform.png|frame|center|SIP response alert configuration form]]
{{Tip|1=Use filter templates for complex conditions like <code>duration > 14400</code> (calls over 4 hours) or <code>absolute_timeout</code> (truncated recordings).}}


===== Detecting 408 Request Timeout Failures =====
=== SIP Response Alerts ===


A '''408 Request Timeout''' response occurs when the caller sends multiple INVITE retransmissions and receives no final response. This is useful for alerting on calls that timeout after the UAS (User Agent Server) sends a provisional response like '''100 Trying''' but then fails to send any further responses.
{| class="wikitable"
|-
! Response Code !! Meaning
|-
| Empty || All call attempts per filters
|-
| '''0''' || No response received (routing loops)
|-
| '''408''' || Timeout after provisional response (server unresponsive)
|-
| Specific || Exact codes (404, 503, etc.)
|}


'''Use Cases:'''
==== "from all" Checkbox (Percentage Thresholds) ====
* Detect failing PBX or SBC (Session Border Controller) instances that accept calls but stop processing
* Monitor network failures where SIP messages stop flowing after initial dialog establishment
* Identify servers that become unresponsive mid-call setup


'''Configuration:'''
{{Warning|1=This setting is critical for IP group monitoring.}}
1. Navigate to '''GUI > Alerts'''
2. Create new alert with type '''SIP Response'''
3. Set '''Response code''' to <code>408</code>
4. Optionally add Common Filters (IP addresses, numbers) to narrow scope
5. Save the alert


'''Understanding the Difference Between Response Code 0 and 408:'''
* '''CHECKED''': % calculated from ALL CDRs in database
* '''Response code 0'': Matches calls that received absolutely no response (not even a 100 Trying). These are network or reachability issues.
* '''UNCHECKED''': % calculated only from filtered CDRs (correct for specific IP groups)
* '''Response code 408''': Matches calls that received at least one provisional response (like 100 Trying) but eventually timed out. These indicate a server or application layer problem where the UAS stopped responding after initial acknowledgment.


Note: When a call times out with a 408 response, the CDR stores <code>408</code> as the Last SIP Response. Alerting on 408 will catch all call setup timeouts, including those where a 100 Trying was initially received.


===== Sensors Alerts =====


Sensors alerts monitor the health of VoIPmonitor probes and sniffer instances. This is the most reliable method to check if remote sensors are online and actively monitoring traffic.
==== SIP Response vs Last SIP Response ====


Unlike simple network port monitoring (which may show a port as open even if the process is frozen or unresponsive), sensors alerts verify that the sensor instance is actively communicating with the VoIPmonitor GUI server.
There are two different fields for matching SIP responses:


;Setup:
{| class="wikitable"
:# Configure sensors in '''Settings > Sensors'''
|-
:# Create a sensors alert to be notified when a probe goes offline or becomes unresponsive
! Field !! Location !! Supports % Threshold !! Use Case
|-
| '''SIP response''' || GUI > Alerts > SIP Response Alerts || {{Yes}} || Match by numeric code (e.g., 487, 503)
|-
| '''Last sip response''' || GUI > Alerts > Filter common || {{No}} || Match by full text (e.g., "487 Request Terminated")
|}


===== SIP REGISTER RRD Beta Alerts =====
{{Warning|1=The GUI '''cannot trigger alerts based on percentage of full textual response strings'''. If you need percentage-based triggering for SIP response codes, use the '''SIP response''' numeric field instead.}}


The '''SIP REGISTER RRD beta''' alert type monitors SIP REGISTER response times and alerts when REGISTER packets do not receive a response within a specified threshold (in milliseconds). This is useful for detecting network latency issues, packet loss, or failing switches that cause SIP retransmissions.
The '''Last sip response''' field supports wildcard patterns (%, %Request Terminated%, %487%) but only triggers based on count thresholds, not percentages.
=== International Call Alerts (Called Number Prefixes) ===


This alert serves as an effective proxy to monitor for registration issues, as REGISTER retransmissions often indicate problems with network connectivity or unresponsive SIP servers.
Monitor calls to international destinations using '''prefix-based matching''' (dialing patterns like 00, +).


;Configuration:
{{Note|1=This uses phone number prefix detection, NOT IP geolocation. For GeoIP-based detection, see [[Anti-fraud|Anti-Fraud Rules]].}}
:# Navigate to '''GUI > Alerts'''
:# Create a new alert with type '''SIP REGISTER RRD beta'''
:# Set the response time threshold in milliseconds (e.g., alert if REGISTER does not receive a response within 2000ms)
:# Configure recipient email addresses
:# Save the alert configuration


The system monitors REGISTER packets and triggers an alert when responses exceed the configured threshold, indicating potential SIP registration failures or network issues.
'''Configuration:'''
# '''GUI > Settings > Country prefixes''' - Define international prefixes (00, +), local country, minimum digits
# '''GUI > Alerts > Filter common''' - Configure:


===== Multiple Register Beta Alerts =====
{| class="wikitable"
 
|-
The '''multiple register (beta)''' alert type detects SIP accounts that are registered from multiple different IP addresses. This is useful for identifying potential security issues, configuration errors, or unauthorized use of SIP credentials.
! Setting !! Description
 
|-
This alert specifically finds phone numbers or SIP accounts that have registered from more than one distinct IP address within the monitored timeframe.
| Called number prefixes || Which prefixes trigger alert (uncheck ALL for all international)
 
|-
;Use Cases:
| Exclude called number || Country codes to exclude (e.g., +44, 0044 for UK)
:* Detect SIP account compromise (credential theft leading to registrations from unauthorized IPs)
|-
:* Identify configuration issues where phones are registering from multiple networks unexpectedly
| Strict for prefixes || Require international prefix (00/+)
:* Monitor for roaming behavior when multi-IP registration is not expected
|-
:* Audit SIP account usage across distributed environments
| NANPA || North American Numbering Plan
 
|}
;Configuration:
:# Navigate to '''GUI > Alerts'''
:# Create a new alert with type '''multiple register (beta)'''
:# Configure recipient email addresses
:# Optionally add Common Filters (IP addresses, numbers) to narrow scope - '''leave filters empty to check ALL SIP numbers/accounts across all customers'''
:# Save the alert configuration
 
;Alert Scope:
:* '''With filters''': Monitors only the specific IP addresses, numbers, or groups defined in the Common Filters section
:* '''Without filters''': Monitors all SIP numbers/accounts across all customers in your system
 
The alert will trigger whenever it detects a SIP account that has registered from multiple distinct IP addresses, providing details about the affected account(s) and the IP addresses observed.


[[File:alertgrid.png|frame|center|Alert configuration grid]]
=== Sensors Alerts ===


==== Common Filters ====
Monitor sensor health and status:
* '''Offline detection''' - Sensor not communicating
* '''Old CDR''' - No recent CDRs written (capture or DB issue)
* '''Big SQL queue stat''' - Growing queue indicates DB bottleneck (warning: >20 files, critical: >100)


All alert types support the following filters:
=== SIP REGISTER Alerts ===


{| class="wikitable"
{| class="wikitable"
|-
|-
! Filter !! Description
! Alert Type !! Purpose !! Use Case
|-
|-
| IP/Number Group || Apply alert to predefined groups (from '''Groups''' menu)
| '''SIP REGISTER RRD beta''' || Response time monitoring || Network latency, packet loss
|-
|-
| IP Addresses || Individual IPs or ranges (one per line)
| '''SIP failed Register (beta)''' || Failed registrations by IP || Brute-force, credential stuffing
|-
|-
| Numbers || Individual phone numbers or prefixes (one per line)
| '''multiple register (beta)''' || Same account from multiple IPs || Credential compromise detection
|-
| Email Group || Send alerts to group-defined email addresses
|-
| Emails || Individual recipient emails (one per line)
|-
| External script || Path to custom script to execute when alert triggers (see below)
|}
|}


[[File:alertgroup.png|frame|center|Alert filter configuration]]
{{Warning|1='''multiple register (beta)''' detects SIMULTANEOUS registrations from multiple IPs (security). For detecting IP changes when device moves networks, use CDR&RTP alert with external script.}}
 
=== Using External Scripts for Alert Actions ===
 
Beyond email notifications, alerts can execute custom scripts when triggered. This enables integration with third-party systems (webhooks, Datadog, Slack, custom monitoring tools) without sending emails.


==== Configuration ====


1. Navigate to '''GUI > Alerts'''
2. Create or edit an alert (RTP, SIP Response, Sensors, etc.)
3. In the configuration form, locate the '''External script''' field
4. Enter the full path to your custom script (e.g., <code>/usr/local/bin/alert-webhook.sh</code>)
5. Save the alert configuration


The script will execute immediately when the alert triggers.
==== Alert Output Fields ====


==== Script Arguments ====
The '''multiple register (beta)''' and other SIP REGISTER alerts output the following fields in email notifications and GUI:
 
The custom script receives alert data as command-line arguments. The format is identical to anti-fraud scripts (see [[Anti-fraud|Anti-Fraud Rules]]):


{| class="wikitable"
{| class="wikitable"
|-
|-
! Argument !! Description
! Field !! Source !! Description
|-
|-
| <code>$1</code> || Alert ID (numeric identifier)
| '''username''' || SIP Contact header || The registered user identity
|-
|-
| <code>$2</code> || Alert name/type
| '''from''' fields || SIP From header || From-number, From-domain extracted from From header
|-
|-
| <code>$3</code> || Unix timestamp of alert trigger
| '''to''' fields || SIP To header || To-number, To-domain extracted from To header
|-
|-
| <code>$4</code> || JSON-encoded alert data
| '''lookup name''' || Tools > Prefix Lookup || Custom label if phone number matches a configured prefix entry
|}
|}


==== Alert Data Structure ====
{{Note|1=The '''lookup name''' column displays custom labels from [[Tools#Prefix_Lookup|Prefix Lookup]] when a phone number matches a configured prefix. If no match exists, the field remains empty or shows the raw number.}}
 
=== CDR Trends Alerts ===
The JSON in the fourth argument contains CDR IDs affected by the alert:
 
<syntaxhighlight lang="json">
{
  "cdr": [12345, 12346, 12347],
  "alert_type": "MOS below threshold",
  "threshold": 3.5,
  "actual_value": 2.8
}
</syntaxhighlight>
 
Use the <code>cdr</code> array to query additional information from the database if needed.
 
==== Example: Send Webhook to Datadog ====
 
This bash script sends an alert notification to a Datadog webhook API:
 
<syntaxhighlight lang="bash">
#!/bin/bash
# /usr/local/bin/datadog-alert.sh
 
# Configuration
WEBHOOK_URL="https://webhook.site/your-custom-url"
DATADOG_API_KEY="your-datadog-api-key"
 
# Parse arguments
ALERT_ID="$1"
ALERT_NAME="$2"
TIMESTAMP="$3"
ALERT_DATA="$4"
 
# Convert Unix timestamp to readable date
DATE=$(date -d "@$TIMESTAMP" '+%Y-%m-%d %H:%M:%S')
 
# Extract relevant data from JSON
cdrCount=$(echo "$ALERT_DATA" | jq -r '.cdr | length')
threshold=$(echo "$ALERT_DATA" | jq -r '.threshold // empty')
actualValue=$(echo "$ALERT_DATA" | jq -r '.actual_value // empty')
 
# Build webhook payload
PAYLOAD=$(cat <<EOF
{
  "alert_id": "$ALERT_ID",
  "alert_name": "$ALERT_NAME",
  "triggered_at": "$DATE",
  "cdr_count": $cdrCount,
  "threshold": $threshold,
  "actual_value": $actualValue,
  "source": "voipmonitor"
}
EOF
)
 
# Send webhook
curl -X POST "$WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DATADOG_API_KEY" \
  -d "$PAYLOAD"
</syntaxhighlight>
 
Make the script executable:


<syntaxhighlight lang="bash">
Monitor metric deviations from historical baselines (e.g., ASR drops).
chmod +x /usr/local/bin/datadog-alert.sh
</syntaxhighlight>
 
==== Example: Send Slack Notification ====
 
<syntaxhighlight lang="bash">
#!/bin/bash
# /usr/local/bin/slack-alert.sh
 
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
 
ALERT_NAME="$2"
ALERT_DATA="$4"
cdrCount=$(echo "$ALERT_DATA" | jq -r '.cdr | length')
 
curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "VoIPmonitor Alert: '"$ALERT_NAME"'",
    "attachments": [{
      "color": "danger",
      "fields": [
        {"title": "CDRs affected", "value": "'"$cdrCount"'"}
      ]
    }]
  }'
</syntaxhighlight>
 
==== Example: Store Alert Details in File ====
 
<syntaxhighlight lang="bash">
#!/bin/bash
# /usr/local/bin/log-alert.sh
 
LOG_DIR="/var/log/voipmonitor-alerts"
mkdir -p "$LOG_DIR"
 
# Log all arguments for debugging
echo "=== Alert triggered at $(date) ===" >> "$LOG_DIR/alerts.log"
echo "Alert ID: $1" >> "$LOG_DIR/alerts.log"
echo "Alert name: $2" >> "$LOG_DIR/alerts.log"
echo "Timestamp: $3" >> "$LOG_DIR/alerts.log"
echo "Data: $4" >> "$LOG_DIR/alerts.log"
echo "" >> "$LOG_DIR/alerts.log"
</syntaxhighlight>
 
==== Example: Access Source IP Addresses ====
 
When querying the CDR database from an alert script, IP addresses are stored as decimal integers in the <code>cdr</code> table. To convert them to human-readable dotted-decimal format (e.g., <code>185.107.80.4</code>), use either PHP's <code>long2ip()</code> function or MySQL's <code>INET_NTOA()</code> function.
 
===== Using PHP's long2ip() (Recommended for Post-Processing) =====
 
If you fetch the raw integer value from the database and convert it in your script:
 
<syntaxhighlight lang="php">
#!/usr/bin/php
<?php
// Parse alert data
$alert = json_decode($argv[4]);
$cdrIds = implode(',', $alert->cdr);
 
// Query the CDR table - note: sipcallerip is a decimal integer
$query = "SELECT id, sipcallerip, sipcalledip
          FROM voipmonitor.cdr
          WHERE id IN ($cdrIds)";
$command = "mysql -h MYSQLHOST -u MYSQLUSER -pMYSQLPASS -N -e \"$query\"";
exec($command, $results);
 
// Process results and convert IP addresses
foreach ($results as $line) {
    list($id, $callerIP, $calledIP) = preg_split('/\t/', trim($line));
 
    // Convert decimal integer to dotted-decimal format
    $callerIPFormatted = long2ip($callerIP);
    $calledIPFormatted = long2ip($calledIP);
 
    echo "CDR ID $id: Caller IP $callerIPFormatted, Called IP $calledIPFormatted\n";
 
    // Example: long2ip(3110817796) returns "185.107.80.4"
}
?>
</syntaxhighlight>
 
===== Using MySQL's INET_NTOA() (Recommended for Database Queries) =====
 
If you prefer to handle conversion in the SQL query itself:
 
<syntaxhighlight lang="php">
#!/usr/bin/php
<?php
// Parse alert data
$alert = json_decode($argv[4]);
$cdrIds = implode(',', $alert->cdr);
 
// Query with IP conversion done in MySQL
$query = "SELECT id, INET_NTOA(sipcallerip) as caller_ip, INET_NTOA(sipcalledip) as called_ip
          FROM voipmonitor.cdr
          WHERE id IN ($cdrIds)";
$command = "mysql -h MYSQLHOST -u MYSQLUSER -pMYSQLPASS -N -e \"$query\"";
exec($command, $results);
 
// Process results - IPs are already formatted
foreach ($results as $line) {
    list($id, $callerIP, $calledIP) = preg_split('/\t/', trim($line));
    echo "CDR ID $id: Caller IP $callerIP, Called IP $calledIP\n";
}
?>
</syntaxhighlight>
 
===== Common IP Columns in CDR Table =====
 
The following columns contain IP addresses (all stored as decimal integers):


{| class="wikitable"
{| class="wikitable"
|-
|-
! Column !! Description
! Parameter !! Description
|-
|-
| <code>sipcallerip</code> || SIP signaling source IP
| Type || Metric to monitor (ASR, ACD, etc.)
|-
|-
| <code>sipcalledip</code> || SIP signaling destination IP
| Offset || Historical baseline (1 week, 1 day)
|-
|-
| <code>rtpsrcipX</code> || RTP source IP for stream X (where X = 0-9)
| Range || Current evaluation window (1 hour)
|-
|-
| <code>rtpdstipX</code> || RTP destination IP for stream X (where X = 0-9)
| Method || Deviation (%) or Threshold (absolute)
|-
| Limit Inc./Dec. || Trigger threshold percentage
|}
|}


===== Troubleshooting IP Format Issues =====
== Common Filters ==
 
If your alert script receives IP addresses as large numbers (e.g., <code>3110817796</code> instead of <code>185.107.80.4</code>):
 
1. Verify you are querying the <code>cdr</code> table directly (not using formatted variables)
2. Use <code>long2ip()</code> in PHP or <code>INET_NTOA()</code> in MySQL to convert the value
3. Check that the column is not already being converted by another layer of the application
 
For reference:
* <code>long2ip(3110817796)</code> returns <code>185.107.80.4</code>
* <code>long2ip(3232255785)</code> returns <code>192.168.1.101</code>
* <code>long2ip(2130706433)</code> returns <code>127.0.0.1</code>
 
==== Important Notes ====
 
* '''IP Address Format''': IP addresses in the <code>cdr</code> table are stored as decimal integers. Use <code>long2ip()</code> (PHP) or <code>INET_NTOA()</code> (MySQL) to convert to dotted-decimal format.
* '''Script execution time''': The alert processor waits for the script to complete. Keep scripts fast (under 5 seconds) or run them in the background if processing takes longer.
* '''Script permissions''': Ensure the script is executable by the web server user (typically <code>www-data</code> or <code>apache</code>).
* '''Error handling''': Script failures are logged but do not prevent email alerts from being sent.
* '''Querying CDRs''': The script receives CDR IDs in the JSON data. Query the <code>cdr</code> table to retrieve detailed information like caller numbers, call duration, etc.
* '''Security''': Validate input before using it in commands or database queries to prevent injection attacks.
 
=== Sent Alerts ===
 
All triggered alerts are saved in history and can be viewed via '''GUI > Alerts > Sent Alerts'''. The content matches what was sent via email.
 
[[File:alert-sentalerts.png|frame|center|Sent alerts history]]
 
==== Parameters Table ====
 
The parameters table shows QoS metrics with problematic values highlighted for quick identification.
 
[[File:alert-perameters.png|frame|center|Alert parameters with highlighted bad values]]
 
==== CDR Records Table ====
 
The CDR records table lists all calls that triggered the alert. Each row includes alert flags indicating which thresholds were exceeded:
* '''(M)''' - MOS below threshold
* '''(J)''' - Jitter exceeded
* '''(P)''' - Packet loss exceeded
* '''(D)''' - Delay exceeded
 
=== Anti-Fraud Alerts ===
 
VoIPmonitor includes specialized anti-fraud alert rules for detecting attacks and fraudulent activity. These include:
* Realtime concurrent calls monitoring
* SIP REGISTER flood/attack detection
* SIP PACKETS flood detection
* Country/Continent destination alerts
* CDR/REGISTER country change detection
 
For detailed configuration of anti-fraud rules and custom action scripts, see [[Anti-fraud|Anti-Fraud Rules]].
 
=== Alerts Based on Custom Reports ===
 
In addition to native alert types (RTP, SIP response, Sensors), VoIPmonitor supports generating alerts from custom reports. This workflow enables alerts based on criteria not available in native alert types, such as SIP header values captured via CDR custom headers.
 
==== Workflow Overview =====
 
1. [[Settings#CDR_Custom_Headers|Capture custom SIP headers]] in the database
2. Create a custom report filtered by the custom header values
3. Generate an alert from that report
4. Configure alert email options (e.g., limit email size)
 
==== Example: Alert on SIP Max-Forwards Header Value =====
 
This example shows how to receive an alert when the SIP <code>Max-Forwards</code> header value drops below 15.
 
'''Step 1: Configure Sniffer Capture'''
 
Add the header to your <code>/etc/voipmonitor.conf</code> configuration file:
 
<syntaxhighlight lang="ini">
# Capture Max-Forwards header
custom_headers = Max-Forwards
</syntaxhighlight>
 
Restart the sniffer to apply changes:
 
<syntaxhighlight lang="bash">
service voipmonitor restart
</syntaxhighlight>
 
'''Step 2: Configure Custom Header in GUI'''
 
1. Navigate to '''GUI > Settings > CDR Custom Headers'''
2. Select <code>Max-Forwards</code> from the available headers
3. Enable '''Show as Column''' to display it in CDR views
4. Save configuration
 
'''Step 3: Create Custom Report'''
 
1. Navigate to '''GUI > CDR Custom Headers''' or use the Report Generator
2. Create a filter for calls where <code>Max-Forwards</code> is less than 15
3. Since custom headers store string values, use a filter expression that matches the desired values:
<syntaxhighlight lang="text">
15 14 13 12 11 10 0_ _
</syntaxhighlight>
  Include additional space-separated values or use NULL to match other ranges as needed.
 
4. Run the report to verify it captures the expected calls
 
'''Step 4: Generate Alert from Report'''
 
You can create an alert based on this custom report using the Daily Reports feature:
 
1. Navigate to '''GUI > Reports > Configure Daily Reports'''
2. Click '''Add Daily Report'''
3. Configure the filter to target the custom header criteria (e.g., Max-Forwards < 15)
4. Set the schedule (e.g., run every hour)
5. Save the daily report configuration
 
'''Step 5: Limit Alert Email Size (Optional)'''
 
If the custom report generates many matching calls, the alert email can become large. To limit the email size:
 
1. Edit the daily report
2. Go to the '''Basic Data''' tab
3. Set the '''max-lines in body''' option to the desired limit (e.g., 100 lines)
 
==== Additional Use Cases =====
 
This workflow can be used for various custom monitoring scenarios:
 
* '''SIP headers beyond standard SIP response codes''' - Monitor any custom SIP header
* '''Complex filtering logic''' - Create reports based on multiple custom header filters
* '''Threshold monitoring for string fields''' - When numeric comparison is not available, use string matching
 
For more information on configuring custom headers, see [[Settings#CDR_Custom_Headers|CDR Custom Headers]].
 
=== Troubleshooting Email Alerts ===
 
If email alerts are not being sent, the issue is typically with the Mail Transfer Agent (MTA) rather than VoIPmonitor.
 
==== Step 1: Test Email Delivery from Command Line ====
 
Before investigating complex issues, verify your server can send emails:
 
<syntaxhighlight lang="bash">
# Test using the 'mail' command
echo "Test email body" | mail -s "Test Subject" your.email@example.com
</syntaxhighlight>
 
If this fails, the issue is with your MTA configuration, not VoIPmonitor.


==== Step 2: Check MTA Service Status ====
All alert types support:
* '''IP/Number Group''' - Predefined groups from '''Groups''' menu
* '''IP Addresses''' / '''Numbers''' - Individual values (one per line)
* '''Email Group''' / '''Emails''' - Recipients
* '''Last sip response''' - Filter by response text (requires <code>save_sip_history = responses</code>)
* '''External script''' - Custom script path for integrations


Ensure the MTA service is running:
{{Warning|1=Alerts use '''OR logic''' between conditions. AND logic is NOT supported. Workaround: create separate alerts and correlate manually.}}


<syntaxhighlight lang="bash">
=== Caller vs Called Filtering ===
# For Postfix (most common)
sudo systemctl status postfix


# For Exim (Debian default)
The Numbers filter matches against '''both caller and called fields'''. You cannot create alerts that trigger only when a number is the caller or only the called. Use IP Groups with Trunk/Server checkboxes for direction-based filtering. See [[Groups]].
sudo systemctl status exim4


# For Sendmail
== External Scripts ==
sudo systemctl status sendmail
</syntaxhighlight>


If the service is not running or not installed, install and configure it according to your Linux distribution's documentation.
Enable webhook integration (Datadog, Slack, custom systems).


==== Step 3: Check Mail Logs ====
'''Configuration:''' Enter full absolute path in '''External script''' field (e.g., <code>/usr/local/bin/alert-webhook.sh</code>).


Examine the MTA logs for specific error messages:
'''Arguments passed to script:'''
 
<syntaxhighlight lang="bash">
# Debian/Ubuntu
tail -f /var/log/mail.log
 
# RHEL/CentOS/AlmaLinux/Rocky
tail -f /var/log/maillog
</syntaxhighlight>
 
Common errors and their meanings:
{| class="wikitable"
{| class="wikitable"
|-
|-
! Error Message !! Cause !! Solution
! Arg !! Description
|-
|-
| Connection refused || MTA not running or firewall blocking || Start MTA service, check firewall rules
| <code>$1</code> || Alert ID
|-
|-
| Relay access denied || SMTP relay misconfiguration || See "Configuring SMTP Relay" below
| <code>$2</code> || Alert name
|-
|-
| Authentication failed || Incorrect credentials || Verify credentials in sasl_passwd
| <code>$3</code> || Unix timestamp
|-
|-
| Host or domain name lookup failed || DNS issues || Check /etc/resolv.conf
| <code>$4</code> || JSON data with CDR IDs
|-
| Greylisted || Temporary rejection || Wait and retry, or whitelist sender
|}
|}


==== Step 4: Check Mail Queue ====
'''Example - Slack notification:'''
 
Emails may be stuck in the queue if delivery is failing:
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# View the mail queue
#!/bin/bash
mailq
# /usr/local/bin/slack-alert.sh
 
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
# Force immediate delivery attempt
curl -X POST "$SLACK_WEBHOOK" -H "Content-Type: application/json" \
postqueue -f
  -d '{"text": "VoIPmonitor Alert: '"$2"'"}'
</syntaxhighlight>
</syntaxhighlight>


Deferred or failed messages in the queue contain error details explaining why delivery failed.
{{Note|1=IP addresses in CDR table are decimal integers. Use <code>long2ip()</code> (PHP) or <code>INET_NTOA()</code> (MySQL) for conversion.}}


==== Configuring SMTP Relay ===
== Sent Alerts ==


If you encounter "Relay access denied" errors, your Postfix server cannot send emails through your external SMTP server. There are two solutions:
View triggered alerts via '''GUI > Alerts > Sent Alerts'''. Shows:
* '''Parameters table''' - QoS metrics with highlighted bad values
* '''CDR records''' - Calls that triggered alert with flags: (M)OS, (J)itter, (P)acket loss, (D)elay


'''Solution 1: Configure External SMTP to Permit Relaying (Recommended for Trusted Networks)'''
== Custom Report Alerts ==


If the VoIPmonitor server is in a trusted network, configure your external SMTP server to permit relaying from the VoIPmonitor server's IP address:
Alert on criteria not in native types (e.g., custom SIP headers).


1. Access your external SMTP server configuration
'''Workflow:'''
2. Add the VoIPmonitor server's IP address to the allowed relay hosts (mynetworks)
# Capture header in <code>/etc/voipmonitor.conf</code>: <code>custom_headers = Max-Forwards</code>
3. Save configuration and reload: <code>postfix reload</code>
# Enable in '''GUI > Settings > CDR Custom Headers'''
# Create filter in CDR view, save as template
# Create Daily Report with filter in '''GUI > Reports > Configure Daily Reports'''


'''Solution 2: Configure Postfix SMTP Authentication (Recommended for Remote SMTP)'''
{{Note|1=Custom report alerts cannot group by caller/called for threshold detection (e.g., "alert if same caller has >X failures"). Use CDR Summary reports for aggregated data.}}


If using an external SMTP server that requires authentication, configure Postfix to authenticate using SASL:
== Troubleshooting ==


1. Install SASL authentication packages:
=== Email Not Sent ===
<syntaxhighlight lang="bash">
# Debian/Ubuntu
sudo apt-get install libsasl2-modules


# RHEL/CentOS/AlmaLinux/Rocky
'''Diagnosis:'''
sudo yum install cyrus-sasl-plain
* Entries in "Sent Alerts" but no email → MTA issue
</syntaxhighlight>
* No entries in "Sent Alerts" → Alert conditions or cron issue
 
2. Configure Postfix to use the external SMTP relay:
 
Edit <code>/etc/postfix/main.cf</code>:
<syntaxhighlight lang="ini">
# Use external SMTP as relay host
relayhost = smtp.yourprovider.com:587
 
# Enable SASL authentication
smtp_sasl_auth_enable = yes
 
# Use SASL password file
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
 
# Disable anonymous authentication (use only SASL)
smtp_sasl_security_options = noanonymous
 
# Enable TLS (recommended)
smtp_tls_security_level = encrypt
</syntaxhighlight>
 
3. Create the SASL password file with your SMTP credentials:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
# Create the file (your SMTP username and password)
# Test MTA
echo "[smtp.yourprovider.com]:587 username:password" | sudo tee /etc/postfix/sasl_passwd
echo "Test" | mail -s "Test" your@email.com


# Secure the file (rw root only)
# Check MTA status
sudo chmod 600 /etc/postfix/sasl_passwd
systemctl status postfix # or exim4/sendmail


# Create the Postfix hash database
# Check logs
sudo postmap /etc/postfix/sasl_passwd
tail -f /var/log/mail.log  # Debian/Ubuntu
tail -f /var/log/maillog  # RHEL/CentOS


# Reload Postfix
# Check mail queue
sudo systemctl reload postfix
mailq
</syntaxhighlight>
</syntaxhighlight>


4. Test email delivery:
'''Status 250 or "Queued mail for delivery"''' = Your server delivered successfully. If recipient didn't receive, issue is on their side (spam folder, quarantine, blacklisting).
'''Mail Queue Not Delivering:'''
If emails accumulate in the queue but are not being sent:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
echo "Test email" | mail -s "SMTP Relay Test" your.email@example.com
# Verify queue manager is running
</syntaxhighlight>
ps aux | grep qmgr


If successful, emails should be delivered through the authenticated SMTP relay.
# Restart Postfix
systemctl restart postfix


==== Step 5: Verify Cronjob ====
# Force immediate delivery of queued emails
 
postfix flush
Ensure the alert processing script runs every minute:
 
<syntaxhighlight lang="bash">
# Check current crontab
crontab -l
</syntaxhighlight>
</syntaxhighlight>
=== Alerts Not Triggering ===


You should see:
'''Enable debug logging:'''
<syntaxhighlight lang="bash">
<syntaxhighlight lang="php">
* * * * * root php /var/www/html/php/run.php cron
// Add to ./config/system_configuration.php
define('CRON_LOG_FILE', '/tmp/alert.log');
</syntaxhighlight>
</syntaxhighlight>


If missing, add it:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
crontab -e
# Monitor processing
# Add the line above, then reload cron
tail -f /tmp/alert.log
killall -HUP cron
</syntaxhighlight>
</syntaxhighlight>


==== Step 6: Verify Alert Configuration in GUI ====
'''Common causes:'''
 
* Cron not running - verify with <code>crontab -l</code>
After confirming the MTA works:
* PHP CLI version mismatch - use <code>update-alternatives --set php /usr/bin/php8.x</code>
# Navigate to '''GUI > Alerts'''
* SQL queue growing - DB can't keep up (see [[Scaling]])
# Verify alert conditions are enabled
* Alert disabled or filter mismatch
# Check that recipient email addresses are valid
# Go to '''GUI > Alerts > Sent Alerts''' to see if alerts were triggered
 
'''Diagnosis:'''
* Entries in "Sent Alerts" but no emails received → MTA issue
* No entries in "Sent Alerts" → Check alert conditions or cronjob
 
==== Step 7: Test PHP mail() Function ====
 
Isolate the issue by testing PHP directly:
 
<syntaxhighlight lang="bash">
php -r "mail('your.email@example.com', 'Test from PHP', 'This is a test email');"
</syntaxhighlight>
 
* If this works but VoIPmonitor alerts don't → Check GUI cronjob and alert configuration
* If this fails → MTA or PHP configuration issue
 
=== Troubleshooting Concurrent Calls Alerts Not Triggering ===
 
CDR-based concurrent calls alerts may not trigger as expected due to database queue delays or alert timing configuration. Unlike realtime concurrent calls alerts (see [[Anti-fraud|Anti-Fraud Rules]]), CDR-based alerts require CDRs to be written to the database before evaluation.
 
==== Check SQL Cache Files Queue ====
 
A growing SQL cache queue can prevent CDR-based alerts from triggering because the alert processor evaluates CDRs that have already been stored in the database, not calls still waiting in the queue.
 
* Navigate to '''GUI > Settings > Sensors'''
* Check the RRD chart for '''SQL cache files''' (SQLq/SQLf metric)
* '''If the queue is growing during peak times:'''
** Database cannot keep up with CDR insertion rates
** Alerts evaluate outdated data because recent CDRs have not been written yet
** See [[SQL_queue_is_growing_in_a_peaktime|Delay between active call and cdr view]] for solutions
 
==== CDR Timing vs "CDR not older than" Setting ====
 
CDR-based alerts include a ''CDR not older than'' parameter that filters which CDRs are considered for alert evaluation.
 
* '''Parameter location:''' In the concurrent calls alert configuration form
* '''Function:''' Only CDRs newer than this time window are evaluated
* '''Diagnosis:'''
** Verify that the time difference between ''Last CDR in database'' and ''Last CDR in processing queue'' (in Sensors status) is smaller than your ''CDR not older than'' value
** If the delay is larger, CDRs are being excluded from alert evaluation
** Common causes: Database overload, slow storage, insufficient MySQL configuration
* '''Solution:'''
** Increase the ''CDR not older than'' value to match your database performance
** See [[SQL_queue_is_growing_in_a_peaktime]] and [[Scaling]] for database tuning
** Check SQLq value should remain low (under 1000) during peak load
 
==== "Check Interval" Parameter and Low Thresholds ====
 
When testing concurrent calls alerts with very low thresholds (e.g., greater than 0 calls or 1 call), consider the ''Check interval'' parameter.
 
* '''Parameter location:''' In the concurrent calls alert configuration form
* '''Function:''' How often the alert condition is evaluated (time window for concurrent call calculation)
* '''Issue with low thresholds:**
** A call lasting 300 seconds (5 minutes) will show as concurrent for the entire duration
** If ''Check interval'' is shorter than typical call durations, you may see temporary concurrent counts that disappear between evaluations
* '''Recommendation for testing:**
** Increase the ''Check interval'' to a longer duration (e.g., 60 minutes) when testing with very low thresholds
** This ensures concurrent calls are counted over a longer time window, avoiding false negatives from short interval checks
 
==== Fraud Concurrent Calls vs Regular Concurrent Calls Alerts ====


VoIPmonitor provides two different alert types for concurrent calls monitoring, which operate on different data sources and have different capabilities:
=== Concurrent Calls Alerts ===


{| class="wikitable"
{| class="wikitable"
|-
|-
! Feature !! Fraud Concurrent Calls !! Regular Concurrent Calls
! Type !! Data Source !! Aggregation !! Timing
|-
| '''Data source''' || SIP INVITEs (realtime) || CDRs (after call ends)
|-
|-
| '''Processing type''' || Realtime (packet inspection) || CDR-based (database query)
| '''Fraud concurrent calls''' || SIP INVITEs (realtime) || Source IP only || Immediate
|-
|-
| '''Domain filtering''' || '''Not available''' (major limitation) || Available via SQL filter
| '''Regular concurrent calls''' || CDRs (database) || Source/Dest IP, Domain, Custom || Delayed
|-
| '''Timing''' || Immediate (no database delay) || Delayed (requires CDR insertion)
|-
| '''Where configured''' || '''GUI > Alerts > Anti Fraud''' || '''GUI > Alerts'''
|-
| '''Table name''' || `list_concurrent_calls` in anti-fraud section | Standard alerts table
|}
|}


'''Key Differences:'''
Use regular concurrent calls for destination IP monitoring (trunk capacity).
'''Investigating Fraud: Realtime Concurrent Calls Alerts'''


* '''Fraud concurrent calls''' detect concurrent INVITEs in realtime but '''cannot filter by domain''' in SIP headers. Use this for broad concurrent channel monitoring where domain filtering is not required.
Since this alert type triggers before CDRs are written, use the following procedure to investigate the calls that triggered the alert:
* '''Regular concurrent calls alerts''' use stored CDRs, support SQL filters including domain matching, but are affected by database queue delays. Use this when you need domain-specific alerts or detailed CDR-based conditions.
# Navigate to '''GUI → CDR'''
# Use the filter form to add the '''is international''' filter
# Set the '''from''' and '''to''' date range to match the time the alert was sent
# Go to the bottom of the CDR view and enable grouping by '''country'''
# Analyze the traffic by country to identify the source of the fraudulent activity
=== External Script Not Running ===


If you have configured a concurrent calls alert in the standard '''GUI > Alerts''' section and added a domain filter, verify that you are using the correct alert type. The domain filter will work with regular CDR-based alerts but not with fraud concurrent calls (which lack this capability).
# Use '''preview button''' to test alert triggers
# Verify absolute path (not relative)
# Check permissions: <code>chmod 755 /path/to/script.sh</code>
# Include shebang: <code>#!/bin/bash</code>
# Use full command paths (e.g., <code>/usr/bin/curl</code>)
# For URLs, create script with curl/wget - cannot put URL directly in field


=== Troubleshooting Alerts Not Triggering (General) ===
=== "Crontab Log Too Old" Warning ===


If alerts are not appearing in the '''Sent Alerts''' history at all, the problem is typically with the alert processor not evaluating alerts. This is different from MTA issues where alerts appear in history but emails are not sent.
'''Causes:'''
# Cron not running → Add cron entry
# PHP CLI version mismatch → <code>update-alternatives --set php /usr/bin/php8.x</code>
# Database overload → Check SQLq in '''GUI > Settings > Sensors''', see [[Scaling]]


==== Enable Detailed Alert Processing Logs ====
== See Also ==


To debug why alerts are not being evaluated, enable detailed logging for the alert processor by adding the following line to your GUI configuration file:
* [[Anti-fraud|Anti-Fraud Rules]] - Realtime fraud detection
* [[Reports]] - Daily reports and report generator
* [[Groups]] - IP and number groups for filtering


<syntaxhighlight lang="bash">
# Edit the config file (adjust path based on your GUI installation)
nano ./config/system_configuration.php
</syntaxhighlight>


Add this line at the end of the file:
<syntaxhighlight lang="php">
<?php
define('CRON_LOG_FILE', '/tmp/alert.log');
?>
</syntaxhighlight>


This enables logging that shows which alerts are being processed during each cron job run.
==== Increase Parallel Processing Threads ====
If you have many alerts or reports, the default number of parallel threads may cause timeout issues. Increase the parallel task limit:
<syntaxhighlight lang="bash">
# Edit the configuration file
nano ./config/configuration.php
</syntaxhighlight>
Add this line at the end of the file:
<syntaxhighlight lang="php">
<?php
define('CRON_PARALLEL_TASKS', 8);
?>
</syntaxhighlight>
The value of 8 is recommended for high-load environments. Adjust based on your alert/report volume and server capacity.
==== Monitor Alert Processing Logs ====
After enabling logging, monitor the alert log file to see which alerts are being processed:
<syntaxhighlight lang="bash">
# Watch the log in real-time
tail -f /tmp/alert.log
</syntaxhighlight>
The log shows entries like:
<syntaxhighlight lang="text">
begin alert [alert_name]
end alert [alert_name]
</syntaxhighlight>
'''Interpreting the logs:'''
* If you '''do not see''' your alert name in the logs → The alert processor is not evaluating it. Check your alert configuration, filters, and data availability.
* If you '''see''' the alert in logs but it does not trigger → The alert conditions are not being met. Check your thresholds, filter logic, and verify the CDR data matches your expectations.
* If logs are completely empty → The cron job may not be running or the GUI configuration files are not being loaded. Verify the cron job and file paths.
==== Alert Not Appearing in Logs ====
If your alert does not appear in `/tmp/alert.log`:
1. '''Verify the cron job is running:'''
<syntaxhighlight lang="bash">
# Check the cron job exists
crontab -l
# Manually test the cron script to see errors
php ./php/run.php cron
</syntaxhighlight>
2. '''Verify data exists in CDR:'''
<syntaxhighlight lang="bash">
# Check if the calls that should trigger the alert exist
# Navigate to GUI > CDR > Browse and filter for the timeframe
</syntaxhighlight>
3. '''Check alert configuration:'''
* Verify alert is enabled
* Verify filter logic matches your data (IP addresses, numbers, groups)
* Verify thresholds are reasonable for the actual QoS metrics
* Verify GUI license is not locked (Check '''GUI > Settings > License''')
=== "Crontab Log is Too Old" Warning - Database Performance Issues ===
The VoIPmonitor GUI displays a warning message "Crontab log is too old" when the last successful cron run timestamp exceeds the expected interval. While this often indicates a missing or misconfigured cron job, it can also occur when the database is overloaded and the cron script runs slowly.
==== Common Causes ====
# '''Missing or broken cron entry''' - The cron job does not exist in /etc/crontab or the command fails when executed
# '''Database overload''' - The cron job runs but completes slowly due to database performance bottlenecks, causing the "last run" timestamp to drift outside the expected window
==== Distinguishing the Causes ====
Use the following diagnostic workflow to determine if the issue is cron configuration vs. database performance:
'''Step 1: Verify the cron job is actually running'''
Check if the cron execution timestamp is updating (even if slowly):
<syntaxhighlight lang="bash"># Check the current cron timestamp from the database
mysql -u voipmonitor -p voipmonitor -e "SELECT name, last_run FROM scheduler LIMIT 1"
# The last_run timestamp should update at least every few minutes
# If it never updates, the cron is not running (see Step 2)
# If it updates but lags by more than 5-10 minutes, it's a performance issue (see Step 3)
</syntaxhighlight>
'''Step 2: If cron is not running at all'''
Follow the standard cron setup instructions in the "Setting Up the Cron Job" section above. Common issues:
* Cron entry missing from /etc/crontab
* Incorrect PHP path (use full path like /usr/bin/php instead of php)
* PHP CLI missing IonCube loader (check with `php -r 'echo extension_loaded("ionCube Loader")?"yes":"no";'`)
* Wrong file permissions or incorrect web directory path
'''Step 3: If cron runs but slowly (database performance issue)'''
When the cron job runs but takes a long time to complete, the issue is database overload. Diagnose using the Sensors statistics:
# Navigate to '''GUI > Settings > Sensors'''
# Click on the sensor status to view detailed statistics
# Compare the following timestamps:
**'''Last CDR in database'''** - The timestamp of the most recently completed call stored in MySQL
**'''Last CDR in processing queue'''** - The timestamp of the most recent call reached by the sniffer
If there is a significant delay (minutes or more) between these two timestamps during peak traffic, the database cannot keep up with CDR insertion. This causes alert/reports processing (run.php cron) to also run slowly.
=== Solutions for Database Performance Issues ===
**1. Check MySQL configuration**
Ensure your MySQL/MariaDB configuration follows the recommended settings for your call volume. Key parameters:
* <code>innodb_flush_log_at_trx_commit</code> - Set to 2 for better performance (or 0 in extreme high-CPS environments)
* <code>innodb_buffer_pool_size</code> - Allocate 70-80% of available RAM for high-volume deployments
* <code>innodb_io_capacity</code> - Match your storage system capabilities (e.g., 1000000 for NVMe SSDs)
See [[Scaling]] and [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for detailed tuning guides.
**2. Increase database write threads**
In <code>/etc/voipmonitor.conf</code>, increase the number of threads used for writing CDRs:
<syntaxhighlight lang="ini">
mysqlstore_max_threads_cdr = 8  # Default is 4, increase based on workload
</syntaxhighlight>
**3. Monitor SQL queue statistics**
In the expanded status view (GUI > Settings > Sensors > status), check the SQLq value:
* '''SQLq (SQL queue) growing steadily''' - Database is a bottleneck, calls are waiting in memory
* '''SQLq remains low (under 1000)''' - Database is keeping up, may need other tuning
See [[SQL_queue_is_growing_in_a_peaktime]] for more information.
**4. Reduce alert/report processing load**
Too many alert rules or complex reports can exacerbate the problem:
* Review and disable unnecessary alerts in '''GUI > Alerts'''
* Reduce the frequency of daily reports (edit in '''GUI > Reports''')
* Increase parallel processing tasks: In <code>/var/www/html/configuration.php</code>, set <code>define('CRON_PARALLEL_TASKS', 8);</code> (requires increasing PHP memory limits)
**5. Check database query performance**
Identify slow queries:
<syntaxhighlight lang="bash"># Enable slow query logging in my.cnf
slow_query_log = 1
long_query_time = 2
# After waiting for a cron cycle, check the slow query log
tail -f /var/log/mysql/slow.log
</syntaxhighlight>
Look for queries taking more than a few seconds. Common culprits:
* Missing indexes on frequently filtered columns (caller, callee, sipcallerip, etc.)
* Complex alert conditions joining large tables
* Daily reports scanning millions of rows without date range limitations
**6. Scale database architecture**
For very high call volumes (4000+ concurrent calls), consider:
* Separate database server from sensor hosts
* Use MariaDB with LZ4 page compression
* Implement database replication for read queries
* Use hourly table partitioning for improved write performance
See [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for architecture recommendations.
=== Verification ===
After applying fixes:
1. '''Monitor the "Crontab log is too old" timestamp in the GUI'''
  * The timestamp should update every 1-3 minutes during normal operation
  * If it still lags by 10+ minutes, further tuning is required
2. '''Check sensor statistics (GUI > Settings > Sensors)'''
  * The delay between "Last CDR in database" and "Last CDR in processing queue" should be under 1-2 minutes during peak load
  * SQLq should remain below 1000 and not grow continuously
3. '''Test alert processing manually'''
<syntaxhighlight lang="bash">
# Run the cron script manually and measure execution time
time php /var/www/html/php/run.php cron
# Should complete within 10-30 seconds in most environments
# If it takes longer than 60-120 seconds, database tuning is needed
</syntaxhighlight>


=== See Also ===


* [[Anti-fraud|Anti-Fraud Rules]] - Detailed fraud detection configuration
* [[Reports|Reports]] - Daily reports and report generator
* [[Sniffer_troubleshooting|Sniffer Troubleshooting]] - General troubleshooting


== AI Summary for RAG ==
== AI Summary for RAG ==


'''Summary:''' VoIPmonitor Alerts & Reports system for email notifications on QoS and SIP issues. Covers RTP alerts (MOS, jitter, packet loss), SIP response alerts (including detecting 408 Request Timeout from 100 Trying scenarios), sensors health monitoring, SIP REGISTER RRD beta alerts for monitoring registration response times, multiple register (beta) alerts for detecting SIP accounts registered from multiple IP addresses (useful for detecting credential theft or unauthorized access), and creating alerts from custom reports based on CDR custom headers. Includes email troubleshooting for MTA configuration, detailed debugging for alerts not triggering using CRON_LOG_FILE and CRON_PARALLEL_TASKS, external scripts for webhook integration (Datadog, Slack, third-party monitoring), and accessing source IP addresses from alert scripts with proper decimal-to-dotted-decimal conversion using PHP's long2ip() or MySQL's INET_NTOA(). TROUBLESHOOTING concurrent calls alerts: CDR-based alerts may not trigger due to SQL queue delays (check SQL cache files RRD chart in GUI > Settings > Sensors), verify "CDR not older than" parameter is larger than delay between "Last CDR in database" and "Last CDR in processing queue", increase "Check interval" for low threshold testing (e.g., >0 calls use 60 minutes), and understand fraud concurrent calls vs regular concurrent calls: fraud uses INVITEs realtime but lacks domain filtering, regular uses CDRs with domain support but has database delay.
'''Summary:''' VoIPmonitor Alerts system provides email notifications for QoS thresholds (RTP: MOS, jitter, packet loss), SIP response codes (0=no response, 408=timeout), sensor health, and registration monitoring. Alert types include RTP, RTP&CDR (with filter templates for duration/absolute_timeout), SIP Response (use "from all" unchecked for IP group percentages), International Calls (prefix-based, NOT GeoIP), Sensors, SIP REGISTER alerts (RRD beta for latency, failed Register beta for brute-force, multiple register beta for credential compromise), and CDR Trends (ASR deviation monitoring). External scripts enable webhook integrations. CRITICAL: Alerts use OR logic only - AND not supported. IP addresses stored as integers - use long2ip()/INET_NTOA() for conversion.


'''Keywords:''' alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 Request Timeout, 100 Trying, response code 0, INVITE retransmissions, sensors monitoring, SIP REGISTER RRD beta, REGISTER retransmissions, registration monitoring, multiple register beta, multiple IP addresses, SIP account compromise, credential theft, security alerts, unauthorized SIP usage, crontab, MTA, Postfix, Exim, troubleshooting, custom headers, custom reports, Max-Forwards, daily reports, CRON_LOG_FILE, CRON_PARALLEL_TASKS, alert not triggering, /tmp/alert.log, begin alert, end alert, configuration.php, system_configuration.php, external scripts, webhooks, Datadog, Slack, command-line arguments, JSON data, CDR array, IP address conversion, long2ip, INET_NTOA, decimal IP, dotted-decimal, sipcallerip, sipcalledip, rtpsrcip, rtpdstip, human-readable IP, IP format, 3110817796, concurrent calls not triggering, CDR not older than, Check interval, SQL queue, SQL cache files, SQLq, SQLf, growing queue, database delay, CDR timing, fraud concurrent calls, realtime concurrent calls, SIP INVITE concurrent alerts, domain filtering, domain-based alerts, CDR-based alerts, anti-fraud concurrent calls, low threshold testing, concurrent calls window, Last CDR in database, Last CDR in processing queue
'''Keywords:''' alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 timeout, sensors monitoring, SIP REGISTER, brute force, credential stuffing, international calls, called number prefixes, CDR trends, ASR, external scripts, webhooks, from all checkbox, OR logic, crontab, MTA, Postfix, CRON_LOG_FILE, concurrent calls, SQL queue


'''Key Questions:'''
'''Key Questions:'''
* How do I set up email alerts in VoIPmonitor?
* How do I configure email alerts in VoIPmonitor?
* What types of alerts are available (RTP, SIP, Sensors, REGISTER RRD beta, multiple register beta)?
* What alert types are available (RTP, SIP, Sensors)?
* How do I detect calls with 408 Request Timeout after 100 Trying?
* How do I configure international call alerts with prefix filtering?
* How do I create an alert for calls where 100 Trying is sent but no further response?
* What does the "from all" checkbox do in percentage alerts?
* What is the difference between response code 0 and 408?
* How do I integrate alerts with webhooks (Slack, Datadog)?
* How do I monitor and alert on SIP REGISTER retransmissions?
* Why are my alerts not triggering?
* How do I detect registration response time issues?
* How do I troubleshoot email delivery issues?
* How can I use SIP REGISTER RRD beta alert for detecting switch problems?
* What is the difference between fraud and regular concurrent calls alerts?
* How do I detect SIP accounts registered from multiple IP addresses?
* How do I detect SIP registration attacks (brute-force)?
* How do I use multiple register (beta) alert to find accounts with multiple IP registrations?
* Do alerts support AND logic between conditions?
* How do I configure multiple register (beta) alert to check all SIP numbers across all customers?
* How do I detect SIP account compromise or credential theft using alerts?
* How do I monitor for unauthorized SIP account usage?
* How do I configure crontab for alert processing?
* How do I monitor remote sensor health?
* Why are email alerts not being sent?
* How do I troubleshoot MTA email issues?
* How can I create alerts based on SIP headers like Max-Forwards?
* How do I use CDR custom headers for custom reports?
* How do I limit the size of alert emails from custom reports?
* Why are alerts not triggering or appearing in Sent Alerts?
* How do I enable detailed alert processing logs using CRON_LOG_FILE?
* How do I increase parallel alert processing threads with CRON_PARALLEL_TASKS?
* How do I monitor /tmp/alert.log for alert processing debug information?
* How do I configure external scripts for alerts?
* How do I send webhooks from VoIPmonitor alerts to Datadog?
* How do I send alerts to Slack from VoIPmonitor?
* What command-line arguments are passed to alert scripts?
* What JSON data structure is provided to external scripts?
* How do I access source IP addresses from alert scripts?
* Why are IP addresses in alert scripts showing as decimal numbers?
* How do I convert decimal IP addresses to human-readable format?
* What is long2ip() in PHP and how do I use it for IP addresses?
* What is INET_NTOA() in MySQL and how do I use it for IP conversion?
* How do I access sipcallerip, sipcalledip, rtpsrcip from alert scripts?
* How do I convert 3110817796 to 185.107.80.4 in an alert script?
* Why are concurrent calls alerts not triggering?
* How do I check SQL queue for alert timing issues?
* What is the "CDR not older than" parameter in concurrent calls alerts?
* How do I verify if database delays are preventing alerts from triggering?
* How do I monitor SQL cache files queue in GUI > Settings > Sensors?
* What is the "Check interval" parameter in concurrent calls alerts?
* How should I configure "Check interval" for low threshold testing (e.g., >0 calls)?
* What is the difference between fraud concurrent calls and regular concurrent calls alerts?
* Can fraud concurrent calls filter by domain in SIP headers?
* Do regular concurrent calls alerts support domain filtering?
* Why is my domain-based concurrent calls alert not working?
* What is SQLq/SQLf in the Sensors status?
* How does "Last CDR in database" vs "Last CDR in processing queue" affect alerts?
* What should the SQL queue value be during peak load?

Latest revision as of 10:34, 17 January 2026


Alerts & Reports

Email notifications triggered by QoS thresholds, SIP errors, or sensor health conditions. The system stores all alerts in history for review.

Prerequisites

Email Configuration

Alerts use PHP's mail() function via the server's MTA (Postfix/Exim/Sendmail).

Setting Location Description
From Address GUI > Settings > System Configuration > Email DEFAULT_EMAIL_FROM - sender address for all alerts
Cron Job /etc/crontab Required for alert processing 
# Add cron job (required)
echo "* * * * * root php /var/www/html/php/run.php cron" >> /etc/crontab
killall -HUP cron   # Debian/Ubuntu
# or: killall -HUP crond  # RHEL/CentOS

Alert Types

Access via GUI > Alerts.

RTP Alerts

Trigger on voice quality metrics:

  • MOS - below threshold
  • Packet loss - percentage exceeded
  • Jitter - variation exceeded
  • Delay (PDV) - latency exceeded
  • One-way calls - one RTP stream missing
  • Missing RTP - both RTP streams missing

Configure alerts to trigger when number of incidents OR percentage of CDRs exceeds threshold.

RTP&CDR Alerts

Combine RTP metrics with CDR conditions including PDD (Post Dial Delay).

Using Filter Templates:

  1. Create CDR filter in GUI > CDR
  2. Save as template
  3. In alert config, select from Filter template dropdown

💡 Tip: Use filter templates for complex conditions like duration > 14400 (calls over 4 hours) or absolute_timeout (truncated recordings).

SIP Response Alerts

Response Code Meaning
Empty All call attempts per filters
0 No response received (routing loops)
408 Timeout after provisional response (server unresponsive)
Specific Exact codes (404, 503, etc.)

"from all" Checkbox (Percentage Thresholds)

⚠️ Warning: This setting is critical for IP group monitoring.

  • CHECKED: % calculated from ALL CDRs in database
  • UNCHECKED: % calculated only from filtered CDRs (correct for specific IP groups)


SIP Response vs Last SIP Response

There are two different fields for matching SIP responses:

Field Location Supports % Threshold Use Case
SIP response GUI > Alerts > SIP Response Alerts ✓ Yes Match by numeric code (e.g., 487, 503)
Last sip response GUI > Alerts > Filter common ✗ No Match by full text (e.g., "487 Request Terminated")

⚠️ Warning: The GUI cannot trigger alerts based on percentage of full textual response strings. If you need percentage-based triggering for SIP response codes, use the SIP response numeric field instead.

The Last sip response field supports wildcard patterns (%, %Request Terminated%, %487%) but only triggers based on count thresholds, not percentages.

International Call Alerts (Called Number Prefixes)

Monitor calls to international destinations using prefix-based matching (dialing patterns like 00, +).

ℹ️ Note: This uses phone number prefix detection, NOT IP geolocation. For GeoIP-based detection, see Anti-Fraud Rules.

Configuration:

  1. GUI > Settings > Country prefixes - Define international prefixes (00, +), local country, minimum digits
  2. GUI > Alerts > Filter common - Configure:
Setting Description
Called number prefixes Which prefixes trigger alert (uncheck ALL for all international)
Exclude called number Country codes to exclude (e.g., +44, 0044 for UK)
Strict for prefixes Require international prefix (00/+)
NANPA North American Numbering Plan

Sensors Alerts

Monitor sensor health and status:

  • Offline detection - Sensor not communicating
  • Old CDR - No recent CDRs written (capture or DB issue)
  • Big SQL queue stat - Growing queue indicates DB bottleneck (warning: >20 files, critical: >100)

SIP REGISTER Alerts

Alert Type Purpose Use Case
SIP REGISTER RRD beta Response time monitoring Network latency, packet loss
SIP failed Register (beta) Failed registrations by IP Brute-force, credential stuffing
multiple register (beta) Same account from multiple IPs Credential compromise detection

⚠️ Warning: multiple register (beta) detects SIMULTANEOUS registrations from multiple IPs (security). For detecting IP changes when device moves networks, use CDR&RTP alert with external script.


Alert Output Fields

The multiple register (beta) and other SIP REGISTER alerts output the following fields in email notifications and GUI:

Field Source Description
username SIP Contact header The registered user identity
from fields SIP From header From-number, From-domain extracted from From header
to fields SIP To header To-number, To-domain extracted from To header
lookup name Tools > Prefix Lookup Custom label if phone number matches a configured prefix entry

ℹ️ Note: The lookup name column displays custom labels from Prefix Lookup when a phone number matches a configured prefix. If no match exists, the field remains empty or shows the raw number.

CDR Trends Alerts

Monitor metric deviations from historical baselines (e.g., ASR drops).

Parameter Description
Type Metric to monitor (ASR, ACD, etc.)
Offset Historical baseline (1 week, 1 day)
Range Current evaluation window (1 hour)
Method Deviation (%) or Threshold (absolute)
Limit Inc./Dec. Trigger threshold percentage

Common Filters

All alert types support:

  • IP/Number Group - Predefined groups from Groups menu
  • IP Addresses / Numbers - Individual values (one per line)
  • Email Group / Emails - Recipients
  • Last sip response - Filter by response text (requires save_sip_history = responses)
  • External script - Custom script path for integrations

⚠️ Warning: Alerts use OR logic between conditions. AND logic is NOT supported. Workaround: create separate alerts and correlate manually.

Caller vs Called Filtering

The Numbers filter matches against both caller and called fields. You cannot create alerts that trigger only when a number is the caller or only the called. Use IP Groups with Trunk/Server checkboxes for direction-based filtering. See Groups.

External Scripts

Enable webhook integration (Datadog, Slack, custom systems).

Configuration: Enter full absolute path in External script field (e.g., /usr/local/bin/alert-webhook.sh).

Arguments passed to script:

Arg Description
$1 Alert ID
$2 Alert name
$3 Unix timestamp
$4 JSON data with CDR IDs

Example - Slack notification: 

#!/bin/bash
# /usr/local/bin/slack-alert.sh
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
curl -X POST "$SLACK_WEBHOOK" -H "Content-Type: application/json" \
  -d '{"text": "VoIPmonitor Alert: '"$2"'"}'

ℹ️ Note: IP addresses in CDR table are decimal integers. Use long2ip() (PHP) or INET_NTOA() (MySQL) for conversion.

Sent Alerts

View triggered alerts via GUI > Alerts > Sent Alerts. Shows:

  • Parameters table - QoS metrics with highlighted bad values
  • CDR records - Calls that triggered alert with flags: (M)OS, (J)itter, (P)acket loss, (D)elay

Custom Report Alerts

Alert on criteria not in native types (e.g., custom SIP headers).

Workflow:

  1. Capture header in /etc/voipmonitor.conf: custom_headers = Max-Forwards
  2. Enable in GUI > Settings > CDR Custom Headers
  3. Create filter in CDR view, save as template
  4. Create Daily Report with filter in GUI > Reports > Configure Daily Reports

ℹ️ Note: Custom report alerts cannot group by caller/called for threshold detection (e.g., "alert if same caller has >X failures"). Use CDR Summary reports for aggregated data.

Troubleshooting

Email Not Sent

Diagnosis:

  • Entries in "Sent Alerts" but no email → MTA issue
  • No entries in "Sent Alerts" → Alert conditions or cron issue
# Test MTA
echo "Test" | mail -s "Test" your@email.com

# Check MTA status
systemctl status postfix  # or exim4/sendmail

# Check logs
tail -f /var/log/mail.log  # Debian/Ubuntu
tail -f /var/log/maillog   # RHEL/CentOS

# Check mail queue
mailq

Status 250 or "Queued mail for delivery" = Your server delivered successfully. If recipient didn't receive, issue is on their side (spam folder, quarantine, blacklisting). Mail Queue Not Delivering: If emails accumulate in the queue but are not being sent: 

# Verify queue manager is running
ps aux | grep qmgr

# Restart Postfix
systemctl restart postfix

# Force immediate delivery of queued emails
postfix flush

Alerts Not Triggering

Enable debug logging: 

// Add to ./config/system_configuration.php
define('CRON_LOG_FILE', '/tmp/alert.log');

# Monitor processing
tail -f /tmp/alert.log

Common causes:

  • Cron not running - verify with crontab -l
  • PHP CLI version mismatch - use update-alternatives --set php /usr/bin/php8.x
  • SQL queue growing - DB can't keep up (see Scaling)
  • Alert disabled or filter mismatch

Concurrent Calls Alerts

Type Data Source Aggregation Timing
Fraud concurrent calls SIP INVITEs (realtime) Source IP only Immediate
Regular concurrent calls CDRs (database) Source/Dest IP, Domain, Custom Delayed

Use regular concurrent calls for destination IP monitoring (trunk capacity). Investigating Fraud: Realtime Concurrent Calls Alerts

Since this alert type triggers before CDRs are written, use the following procedure to investigate the calls that triggered the alert:

  1. Navigate to GUI → CDR
  2. Use the filter form to add the is international filter
  3. Set the from and to date range to match the time the alert was sent
  4. Go to the bottom of the CDR view and enable grouping by country
  5. Analyze the traffic by country to identify the source of the fraudulent activity

External Script Not Running

  1. Use preview button to test alert triggers
  2. Verify absolute path (not relative)
  3. Check permissions: chmod 755 /path/to/script.sh
  4. Include shebang: #!/bin/bash
  5. Use full command paths (e.g., /usr/bin/curl)
  6. For URLs, create script with curl/wget - cannot put URL directly in field

"Crontab Log Too Old" Warning

Causes:

  1. Cron not running → Add cron entry
  2. PHP CLI version mismatch → update-alternatives --set php /usr/bin/php8.x
  3. Database overload → Check SQLq in GUI > Settings > Sensors, see Scaling

See Also




AI Summary for RAG

Summary: VoIPmonitor Alerts system provides email notifications for QoS thresholds (RTP: MOS, jitter, packet loss), SIP response codes (0=no response, 408=timeout), sensor health, and registration monitoring. Alert types include RTP, RTP&CDR (with filter templates for duration/absolute_timeout), SIP Response (use "from all" unchecked for IP group percentages), International Calls (prefix-based, NOT GeoIP), Sensors, SIP REGISTER alerts (RRD beta for latency, failed Register beta for brute-force, multiple register beta for credential compromise), and CDR Trends (ASR deviation monitoring). External scripts enable webhook integrations. CRITICAL: Alerts use OR logic only - AND not supported. IP addresses stored as integers - use long2ip()/INET_NTOA() for conversion.

Keywords: alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 timeout, sensors monitoring, SIP REGISTER, brute force, credential stuffing, international calls, called number prefixes, CDR trends, ASR, external scripts, webhooks, from all checkbox, OR logic, crontab, MTA, Postfix, CRON_LOG_FILE, concurrent calls, SQL queue

Key Questions:

  • How do I configure email alerts in VoIPmonitor?
  • What alert types are available (RTP, SIP, Sensors)?
  • How do I configure international call alerts with prefix filtering?
  • What does the "from all" checkbox do in percentage alerts?
  • How do I integrate alerts with webhooks (Slack, Datadog)?
  • Why are my alerts not triggering?
  • How do I troubleshoot email delivery issues?
  • What is the difference between fraud and regular concurrent calls alerts?
  • How do I detect SIP registration attacks (brute-force)?
  • Do alerts support AND logic between conditions?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Alerts&oldid=10738"