Alerts: Difference between revisions

From VoIPmonitor.org
(Add documentation for multiple register (beta) alert type)
(Add troubleshooting for 'Crontab log is too old' when caused by database performance (ticket #82237))
Line 707: Line 707:
* Verify thresholds are reasonable for the actual QoS metrics
* Verify thresholds are reasonable for the actual QoS metrics
* Verify GUI license is not locked (Check '''GUI > Settings > License''')
* Verify GUI license is not locked (Check '''GUI > Settings > License''')
=== "Crontab Log is Too Old" Warning - Database Performance Issues ===
The VoIPmonitor GUI displays a warning message "Crontab log is too old" when the last successful cron run timestamp exceeds the expected interval. While this often indicates a missing or misconfigured cron job, it can also occur when the database is overloaded and the cron script runs slowly.
==== Common Causes ====
# '''Missing or broken cron entry''' - The cron job does not exist in /etc/crontab or the command fails when executed
# '''Database overload''' - The cron job runs but completes slowly due to database performance bottlenecks, causing the "last run" timestamp to drift outside the expected window
==== Distinguishing the Causes ====
Use the following diagnostic workflow to determine if the issue is cron configuration vs. database performance:
'''Step 1: Verify the cron job is actually running'''
Check if the cron execution timestamp is updating (even if slowly):
<syntaxhighlight lang="bash"># Check the current cron timestamp from the database
mysql -u voipmonitor -p voipmonitor -e "SELECT name, last_run FROM scheduler LIMIT 1"
# The last_run timestamp should update at least every few minutes
# If it never updates, the cron is not running (see Step 2)
# If it updates but lags by more than 5-10 minutes, it's a performance issue (see Step 3)
</syntaxhighlight>
'''Step 2: If cron is not running at all'''
Follow the standard cron setup instructions in the "Setting Up the Cron Job" section above. Common issues:
* Cron entry missing from /etc/crontab
* Incorrect PHP path (use full path like /usr/bin/php instead of php)
* PHP CLI missing IonCube loader (check with `php -r 'echo extension_loaded("ionCube Loader")?"yes":"no";'`)
* Wrong file permissions or incorrect web directory path
'''Step 3: If cron runs but slowly (database performance issue)'''
When the cron job runs but takes a long time to complete, the issue is database overload. Diagnose using the Sensors statistics:
# Navigate to '''GUI > Settings > Sensors'''
# Click on the sensor status to view detailed statistics
# Compare the following timestamps:
**'''Last CDR in database'''** - The timestamp of the most recently completed call stored in MySQL
**'''Last CDR in processing queue'''** - The timestamp of the most recent call reached by the sniffer
If there is a significant delay (minutes or more) between these two timestamps during peak traffic, the database cannot keep up with CDR insertion. This causes alert/reports processing (run.php cron) to also run slowly.
=== Solutions for Database Performance Issues ===
**1. Check MySQL configuration**
Ensure your MySQL/MariaDB configuration follows the recommended settings for your call volume. Key parameters:
* <code>innodb_flush_log_at_trx_commit</code> - Set to 2 for better performance (or 0 in extreme high-CPS environments)
* <code>innodb_buffer_pool_size</code> - Allocate 70-80% of available RAM for high-volume deployments
* <code>innodb_io_capacity</code> - Match your storage system capabilities (e.g., 1000000 for NVMe SSDs)
See [[Scaling]] and [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for detailed tuning guides.
**2. Increase database write threads**
In <code>/etc/voipmonitor.conf</code>, increase the number of threads used for writing CDRs:
<syntaxhighlight lang="ini">
mysqlstore_max_threads_cdr = 8  # Default is 4, increase based on workload
</syntaxhighlight>
**3. Monitor SQL queue statistics**
In the expanded status view (GUI > Settings > Sensors > status), check the SQLq value:
* '''SQLq (SQL queue) growing steadily''' - Database is a bottleneck, calls are waiting in memory
* '''SQLq remains low (under 1000)''' - Database is keeping up, may need other tuning
See [[SQL_queue_is_growing_in_a_peaktime]] for more information.
**4. Reduce alert/report processing load**
Too many alert rules or complex reports can exacerbate the problem:
* Review and disable unnecessary alerts in '''GUI > Alerts'''
* Reduce the frequency of daily reports (edit in '''GUI > Reports''')
* Increase parallel processing tasks: In <code>/var/www/html/configuration.php</code>, set <code>define('CRON_PARALLEL_TASKS', 8);</code> (requires increasing PHP memory limits)
**5. Check database query performance**
Identify slow queries:
<syntaxhighlight lang="bash"># Enable slow query logging in my.cnf
slow_query_log = 1
long_query_time = 2
# After waiting for a cron cycle, check the slow query log
tail -f /var/log/mysql/slow.log
</syntaxhighlight>
Look for queries taking more than a few seconds. Common culprits:
* Missing indexes on frequently filtered columns (caller, callee, sipcallerip, etc.)
* Complex alert conditions joining large tables
* Daily reports scanning millions of rows without date range limitations
**6. Scale database architecture**
For very high call volumes (4000+ concurrent calls), consider:
* Separate database server from sensor hosts
* Use MariaDB with LZ4 page compression
* Implement database replication for read queries
* Use hourly table partitioning for improved write performance
See [[High-Performance_VoIPmonitor_and_MySQL_Setup_Manual]] for architecture recommendations.
=== Verification ===
After applying fixes:
1. '''Monitor the "Crontab log is too old" timestamp in the GUI'''
  * The timestamp should update every 1-3 minutes during normal operation
  * If it still lags by 10+ minutes, further tuning is required
2. '''Check sensor statistics (GUI > Settings > Sensors)'''
  * The delay between "Last CDR in database" and "Last CDR in processing queue" should be under 1-2 minutes during peak load
  * SQLq should remain below 1000 and not grow continuously
3. '''Test alert processing manually'''
<syntaxhighlight lang="bash">
# Run the cron script manually and measure execution time
time php /var/www/html/php/run.php cron
# Should complete within 10-30 seconds in most environments
# If it takes longer than 60-120 seconds, database tuning is needed
</syntaxhighlight>


=== See Also ===
=== See Also ===
Line 716: Line 848:
== AI Summary for RAG ==
== AI Summary for RAG ==


'''Summary:''' VoIPmonitor Alerts & Reports system for email notifications on QoS and SIP issues. Covers RTP alerts (MOS, jitter, packet loss), SIP response alerts (including detecting 408 Request Timeout from 100 Trying scenarios), sensors health monitoring, SIP REGISTER RRD beta alerts for monitoring registration response times, multiple register (beta) alerts for detecting SIP accounts registered from multiple IP addresses (useful for detecting credential theft or unauthorized access), and creating alerts from custom reports based on CDR custom headers. Includes email troubleshooting for MTA configuration, detailed debugging for alerts not triggering using CRON_LOG_FILE and CRON_PARALLEL_TASKS, and external scripts for webhook integration (Datadog, Slack, third-party monitoring).
'''Summary:''' VoIPmonitor Alerts & Reports system for email notifications on QoS and SIP issues. Covers RTP alerts (MOS, jitter, packet loss), SIP response alerts (including detecting 408 Request Timeout from 100 Trying scenarios), sensors health monitoring, SIP REGISTER RRD beta alerts for monitoring registration response times, multiple register (beta) alerts for detecting SIP accounts registered from multiple IP addresses (useful for detecting credential theft or unauthorized access), and creating alerts from custom reports based on CDR custom headers. Includes email troubleshooting for MTA configuration, detailed debugging for alerts not triggering using CRON_LOG_FILE and CRON_PARALLEL_TASKS, and external scripts for webhook integration (Datadog, Slack, third-party monitoring). Also provides troubleshooting for "Crontab log is too old" warning which can indicate database performance issues when the cron job runs slowly due to database overload - diagnostic workflow includes checking scheduler.last_run timestamp and comparing "Last CDR in database" vs "Last CDR in processing queue" in GUI > Settings > Sensors; solutions include MySQL/MariaDB tuning (innodb_flush_log_at_trx_commit, innodb_buffer_pool_size, innodb_io_capacity), increasing mysqlstore_max_threads_cdr, monitoring SQLq statistics, reducing alert/report processing load, checking slow query logs, and scaling database architecture for high call volumes (separate database host, MariaDB LZ4 compression, hourly table partitioning).


'''Keywords:''' alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 Request Timeout, 100 Trying, response code 0, INVITE retransmissions, sensors monitoring, SIP REGISTER RRD beta, REGISTER retransmissions, registration monitoring, multiple register beta, multiple IP addresses, SIP account compromise, credential theft, security alerts, unauthorized SIP usage, crontab, MTA, Postfix, Exim, troubleshooting, custom headers, custom reports, Max-Forwards, daily reports, CRON_LOG_FILE, CRON_PARALLEL_TASKS, alert not triggering, /tmp/alert.log, begin alert, end alert, configuration.php, system_configuration.php, external scripts, webhooks, Datadog, Slack, command-line arguments, JSON data, CDR array
'''Keywords:''' alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 Request Timeout, 100 Trying, response code 0, INVITE retransmissions, sensors monitoring, SIP REGISTER RRD beta, REGISTER retransmissions, registration monitoring, multiple register beta, multiple IP addresses, SIP account compromise, credential theft, security alerts, unauthorized SIP usage, crontab, MTA, Postfix, Exim, troubleshooting, custom headers, custom reports, Max-Forwards, daily reports, CRON_LOG_FILE, CRON_PARALLEL_TASKS, alert not triggering, /tmp/alert.log, begin alert, end alert, configuration.php, system_configuration.php, external scripts, webhooks, Datadog, Slack, command-line arguments, JSON data, CDR array, Crontab log is too old, database performance, database overload, cron running slowly, scheduler.last_run, last CDR in database, last CDR in processing queue, Settings Sensors, SQLq, SQL queue growing, MySQL tuning, innodb_flush_log_at_trx_commit, innodb_buffer_pool_size, innodb_io_capacity, mysqlstore_max_threads_cdr, slow query log, database scaling, hourly table partitioning, MariaDB LZ4 compression


'''Key Questions:'''
'''Key Questions:'''
Line 750: Line 882:
* What command-line arguments are passed to alert scripts?
* What command-line arguments are passed to alert scripts?
* What JSON data structure is provided to external scripts?
* What JSON data structure is provided to external scripts?
* What does "Crontab log is too old" warning mean in VoIPmonitor?
* How do I troubleshoot "Crontab log is too old" error when cron job exists?
* How do I check if cron job is actually running in VoIPmonitor?
* How do I distinguish between cron configuration issues and database performance issues?
* How do I check scheduler.last_run timestamp in VoIPmonitor database?
* How do I compare "Last CDR in database" vs "Last CDR in processing queue" in GUI?
* Where do I find processing delay statistics in VoIPmonitor GUI?
* What causes cron job to run slowly in VoIPmonitor?
* How do I fix database overload causing slow cron execution?
* What MySQL settings should I tune for high call volumes in VoIPmonitor?
* How do I increase mysqlstore_max_threads_cdr for better database performance?
* What does SQLq value mean in VoIPmonitor sensor status?
* How do I monitor SQL queue growth in VoIPmonitor?
* How do I reduce alert and report processing load on VoIPmonitor database?
* How do I use CRON_PARALLEL_TASKS to improve cron processing performance?
* How do I check for slow MySQL queries in VoIPmonitor?
* What should I do if "Crontab log is too old" persists after adding cron job?

Revision as of 04:58, 6 January 2026

Category:GUI manual

Alerts & Reports

Alerts & Reports generate email notifications based on QoS parameters or SIP error conditions. The system includes daily reports, ad hoc reports, and stores all generated items in history.

Overview

The alert system monitors call quality and SIP signaling in real-time, triggering notifications when configured thresholds are exceeded.

Email Configuration Prerequisites

Emails are sent using PHP's mail() function, which relies on the server's Mail Transfer Agent (MTA) such as Exim, Postfix, or Sendmail. Configure your MTA according to your Linux distribution documentation.

Setting Up the Cron Job

Alert processing requires a cron job that runs every minute:

# Add to /etc/crontab (adjust path based on your GUI installation)
echo "* * * * * root php /var/www/html/php/run.php cron" >> /etc/crontab

# Reload crontab
killall -HUP cron   # Debian/Ubuntu
# or
killall -HUP crond  # CentOS/RHEL

Configure Alerts

Email alerts can trigger on SIP protocol events or RTP QoS metrics. Access alerts configuration via GUI > Alerts.

Alert configuration grid

Alert Types

RTP Alerts

RTP alerts trigger based on voice quality metrics:

  • MOS (Mean Opinion Score) - below threshold
  • Packet loss - percentage exceeded
  • Jitter - variation exceeded
  • Delay (PDV) - latency exceeded
  • One-way calls - answered but one RTP stream missing
  • Missing RTP - answered but both RTP streams missing

Configure alerts to trigger when:

  • Number of incidents exceeds a set value, OR
  • Percentage of CDRs exceeds a threshold
RTP alert configuration form
SIP Response Alerts

SIP response alerts trigger based on SIP response codes:

  • Empty response field: Matches all call attempts per configured filters
  • Response code 0: Matches unreplied INVITE requests (no response received)
  • Specific codes: Match exact codes like 404, 503, etc.
SIP response alert configuration form
Detecting 408 Request Timeout Failures

A 408 Request Timeout response occurs when the caller sends multiple INVITE retransmissions and receives no final response. This is useful for alerting on calls that timeout after the UAS (User Agent Server) sends a provisional response like 100 Trying but then fails to send any further responses.

Use Cases:

  • Detect failing PBX or SBC (Session Border Controller) instances that accept calls but stop processing
  • Monitor network failures where SIP messages stop flowing after initial dialog establishment
  • Identify servers that become unresponsive mid-call setup

Configuration: 1. Navigate to GUI > Alerts 2. Create new alert with type SIP Response 3. Set Response code to 408 4. Optionally add Common Filters (IP addresses, numbers) to narrow scope 5. Save the alert

Understanding the Difference Between Response Code 0 and 408:

  • 'Response code 0: Matches calls that received absolutely no response (not even a 100 Trying). These are network or reachability issues.
  • Response code 408: Matches calls that received at least one provisional response (like 100 Trying) but eventually timed out. These indicate a server or application layer problem where the UAS stopped responding after initial acknowledgment.

Note: When a call times out with a 408 response, the CDR stores 408 as the Last SIP Response. Alerting on 408 will catch all call setup timeouts, including those where a 100 Trying was initially received.

Sensors Alerts

Sensors alerts monitor the health of VoIPmonitor probes and sniffer instances. This is the most reliable method to check if remote sensors are online and actively monitoring traffic.

Unlike simple network port monitoring (which may show a port as open even if the process is frozen or unresponsive), sensors alerts verify that the sensor instance is actively communicating with the VoIPmonitor GUI server.

Setup
  1. Configure sensors in Settings > Sensors
  2. Create a sensors alert to be notified when a probe goes offline or becomes unresponsive
SIP REGISTER RRD Beta Alerts

The SIP REGISTER RRD beta alert type monitors SIP REGISTER response times and alerts when REGISTER packets do not receive a response within a specified threshold (in milliseconds). This is useful for detecting network latency issues, packet loss, or failing switches that cause SIP retransmissions.

This alert serves as an effective proxy to monitor for registration issues, as REGISTER retransmissions often indicate problems with network connectivity or unresponsive SIP servers.

Configuration
  1. Navigate to GUI > Alerts
  2. Create a new alert with type SIP REGISTER RRD beta
  3. Set the response time threshold in milliseconds (e.g., alert if REGISTER does not receive a response within 2000ms)
  4. Configure recipient email addresses
  5. Save the alert configuration

The system monitors REGISTER packets and triggers an alert when responses exceed the configured threshold, indicating potential SIP registration failures or network issues.

Multiple Register Beta Alerts

The multiple register (beta) alert type detects SIP accounts that are registered from multiple different IP addresses. This is useful for identifying potential security issues, configuration errors, or unauthorized use of SIP credentials.

This alert specifically finds phone numbers or SIP accounts that have registered from more than one distinct IP address within the monitored timeframe.

Use Cases
  • Detect SIP account compromise (credential theft leading to registrations from unauthorized IPs)
  • Identify configuration issues where phones are registering from multiple networks unexpectedly
  • Monitor for roaming behavior when multi-IP registration is not expected
  • Audit SIP account usage across distributed environments
Configuration
  1. Navigate to GUI > Alerts
  2. Create a new alert with type multiple register (beta)
  3. Configure recipient email addresses
  4. Optionally add Common Filters (IP addresses, numbers) to narrow scope - leave filters empty to check ALL SIP numbers/accounts across all customers
  5. Save the alert configuration
Alert Scope
  • With filters: Monitors only the specific IP addresses, numbers, or groups defined in the Common Filters section
  • Without filters: Monitors all SIP numbers/accounts across all customers in your system

The alert will trigger whenever it detects a SIP account that has registered from multiple distinct IP addresses, providing details about the affected account(s) and the IP addresses observed.

Alert configuration grid

Common Filters

All alert types support the following filters:

Filter Description
IP/Number Group Apply alert to predefined groups (from Groups menu)
IP Addresses Individual IPs or ranges (one per line)
Numbers Individual phone numbers or prefixes (one per line)
Email Group Send alerts to group-defined email addresses
Emails Individual recipient emails (one per line)
External script Path to custom script to execute when alert triggers (see below)
Alert filter configuration

Using External Scripts for Alert Actions

Beyond email notifications, alerts can execute custom scripts when triggered. This enables integration with third-party systems (webhooks, Datadog, Slack, custom monitoring tools) without sending emails.

Configuration

1. Navigate to GUI > Alerts 2. Create or edit an alert (RTP, SIP Response, Sensors, etc.) 3. In the configuration form, locate the External script field 4. Enter the full path to your custom script (e.g., /usr/local/bin/alert-webhook.sh) 5. Save the alert configuration

The script will execute immediately when the alert triggers.

Script Arguments

The custom script receives alert data as command-line arguments. The format is identical to anti-fraud scripts (see Anti-Fraud Rules):

Argument Description
$1 Alert ID (numeric identifier)
$2 Alert name/type
$3 Unix timestamp of alert trigger
$4 JSON-encoded alert data

Alert Data Structure

The JSON in the fourth argument contains CDR IDs affected by the alert:

{
  "cdr": [12345, 12346, 12347],
  "alert_type": "MOS below threshold",
  "threshold": 3.5,
  "actual_value": 2.8
}

Use the cdr array to query additional information from the database if needed.

Example: Send Webhook to Datadog

This bash script sends an alert notification to a Datadog webhook API:

#!/bin/bash
# /usr/local/bin/datadog-alert.sh

# Configuration
WEBHOOK_URL="https://webhook.site/your-custom-url"
DATADOG_API_KEY="your-datadog-api-key"

# Parse arguments
ALERT_ID="$1"
ALERT_NAME="$2"
TIMESTAMP="$3"
ALERT_DATA="$4"

# Convert Unix timestamp to readable date
DATE=$(date -d "@$TIMESTAMP" '+%Y-%m-%d %H:%M:%S')

# Extract relevant data from JSON
cdrCount=$(echo "$ALERT_DATA" | jq -r '.cdr | length')
threshold=$(echo "$ALERT_DATA" | jq -r '.threshold // empty')
actualValue=$(echo "$ALERT_DATA" | jq -r '.actual_value // empty')

# Build webhook payload
PAYLOAD=$(cat <<EOF
{
  "alert_id": "$ALERT_ID",
  "alert_name": "$ALERT_NAME",
  "triggered_at": "$DATE",
  "cdr_count": $cdrCount,
  "threshold": $threshold,
  "actual_value": $actualValue,
  "source": "voipmonitor"
}
EOF
)

# Send webhook
curl -X POST "$WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DATADOG_API_KEY" \
  -d "$PAYLOAD"

Make the script executable:

chmod +x /usr/local/bin/datadog-alert.sh

Example: Send Slack Notification

#!/bin/bash
# /usr/local/bin/slack-alert.sh

SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

ALERT_NAME="$2"
ALERT_DATA="$4"
cdrCount=$(echo "$ALERT_DATA" | jq -r '.cdr | length')

curl -X POST "$SLACK_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "VoIPmonitor Alert: '"$ALERT_NAME"'",
    "attachments": [{
      "color": "danger",
      "fields": [
        {"title": "CDRs affected", "value": "'"$cdrCount"'"}
      ]
    }]
  }'

Example: Store Alert Details in File

#!/bin/bash
# /usr/local/bin/log-alert.sh

LOG_DIR="/var/log/voipmonitor-alerts"
mkdir -p "$LOG_DIR"

# Log all arguments for debugging
echo "=== Alert triggered at $(date) ===" >> "$LOG_DIR/alerts.log"
echo "Alert ID: $1" >> "$LOG_DIR/alerts.log"
echo "Alert name: $2" >> "$LOG_DIR/alerts.log"
echo "Timestamp: $3" >> "$LOG_DIR/alerts.log"
echo "Data: $4" >> "$LOG_DIR/alerts.log"
echo "" >> "$LOG_DIR/alerts.log"

Important Notes

  • Script execution time: The alert processor waits for the script to complete. Keep scripts fast (under 5 seconds) or run them in the background if processing takes longer.
  • Script permissions: Ensure the script is executable by the web server user (typically www-data or apache).
  • Error handling: Script failures are logged but do not prevent email alerts from being sent.
  • Querying CDRs: The script receives CDR IDs in the JSON data. Query the cdr table to retrieve detailed information like caller numbers, call duration, etc.
  • Security: Validate input before using it in commands or database queries to prevent injection attacks.

Sent Alerts

All triggered alerts are saved in history and can be viewed via GUI > Alerts > Sent Alerts. The content matches what was sent via email.

Sent alerts history

Parameters Table

The parameters table shows QoS metrics with problematic values highlighted for quick identification.

Alert parameters with highlighted bad values

CDR Records Table

The CDR records table lists all calls that triggered the alert. Each row includes alert flags indicating which thresholds were exceeded:

  • (M) - MOS below threshold
  • (J) - Jitter exceeded
  • (P) - Packet loss exceeded
  • (D) - Delay exceeded

Anti-Fraud Alerts

VoIPmonitor includes specialized anti-fraud alert rules for detecting attacks and fraudulent activity. These include:

  • Realtime concurrent calls monitoring
  • SIP REGISTER flood/attack detection
  • SIP PACKETS flood detection
  • Country/Continent destination alerts
  • CDR/REGISTER country change detection

For detailed configuration of anti-fraud rules and custom action scripts, see Anti-Fraud Rules.

Alerts Based on Custom Reports

In addition to native alert types (RTP, SIP response, Sensors), VoIPmonitor supports generating alerts from custom reports. This workflow enables alerts based on criteria not available in native alert types, such as SIP header values captured via CDR custom headers.

Workflow Overview =

1. Capture custom SIP headers in the database 2. Create a custom report filtered by the custom header values 3. Generate an alert from that report 4. Configure alert email options (e.g., limit email size)

Example: Alert on SIP Max-Forwards Header Value =

This example shows how to receive an alert when the SIP Max-Forwards header value drops below 15.

Step 1: Configure Sniffer Capture

Add the header to your /etc/voipmonitor.conf configuration file:

# Capture Max-Forwards header
custom_headers = Max-Forwards

Restart the sniffer to apply changes:

service voipmonitor restart

Step 2: Configure Custom Header in GUI

1. Navigate to GUI > Settings > CDR Custom Headers 2. Select Max-Forwards from the available headers 3. Enable Show as Column to display it in CDR views 4. Save configuration

Step 3: Create Custom Report

1. Navigate to GUI > CDR Custom Headers or use the Report Generator 2. Create a filter for calls where Max-Forwards is less than 15 3. Since custom headers store string values, use a filter expression that matches the desired values:

15 14 13 12 11 10 0_ _
  Include additional space-separated values or use NULL to match other ranges as needed.

4. Run the report to verify it captures the expected calls

Step 4: Generate Alert from Report

You can create an alert based on this custom report using the Daily Reports feature:

1. Navigate to GUI > Reports > Configure Daily Reports 2. Click Add Daily Report 3. Configure the filter to target the custom header criteria (e.g., Max-Forwards < 15) 4. Set the schedule (e.g., run every hour) 5. Save the daily report configuration

Step 5: Limit Alert Email Size (Optional)

If the custom report generates many matching calls, the alert email can become large. To limit the email size:

1. Edit the daily report 2. Go to the Basic Data tab 3. Set the max-lines in body option to the desired limit (e.g., 100 lines)

Additional Use Cases =

This workflow can be used for various custom monitoring scenarios:

  • SIP headers beyond standard SIP response codes - Monitor any custom SIP header
  • Complex filtering logic - Create reports based on multiple custom header filters
  • Threshold monitoring for string fields - When numeric comparison is not available, use string matching

For more information on configuring custom headers, see CDR Custom Headers.

Troubleshooting Email Alerts

If email alerts are not being sent, the issue is typically with the Mail Transfer Agent (MTA) rather than VoIPmonitor.

Step 1: Test Email Delivery from Command Line

Before investigating complex issues, verify your server can send emails:

# Test using the 'mail' command
echo "Test email body" | mail -s "Test Subject" your.email@example.com

If this fails, the issue is with your MTA configuration, not VoIPmonitor.

Step 2: Check MTA Service Status

Ensure the MTA service is running:

# For Postfix (most common)
sudo systemctl status postfix

# For Exim (Debian default)
sudo systemctl status exim4

# For Sendmail
sudo systemctl status sendmail

If the service is not running or not installed, install and configure it according to your Linux distribution's documentation.

Step 3: Check Mail Logs

Examine the MTA logs for specific error messages:

# Debian/Ubuntu
tail -f /var/log/mail.log

# RHEL/CentOS/AlmaLinux/Rocky
tail -f /var/log/maillog

Common errors and their meanings:

Error Message Cause Solution
Connection refused MTA not running or firewall blocking Start MTA service, check firewall rules
Relay access denied SMTP relay misconfiguration See "Configuring SMTP Relay" below
Authentication failed Incorrect credentials Verify credentials in sasl_passwd
Host or domain name lookup failed DNS issues Check /etc/resolv.conf
Greylisted Temporary rejection Wait and retry, or whitelist sender

Step 4: Check Mail Queue

Emails may be stuck in the queue if delivery is failing:

# View the mail queue
mailq

# Force immediate delivery attempt
postqueue -f

Deferred or failed messages in the queue contain error details explaining why delivery failed.

= Configuring SMTP Relay

If you encounter "Relay access denied" errors, your Postfix server cannot send emails through your external SMTP server. There are two solutions:

Solution 1: Configure External SMTP to Permit Relaying (Recommended for Trusted Networks)

If the VoIPmonitor server is in a trusted network, configure your external SMTP server to permit relaying from the VoIPmonitor server's IP address:

1. Access your external SMTP server configuration 2. Add the VoIPmonitor server's IP address to the allowed relay hosts (mynetworks) 3. Save configuration and reload: postfix reload

Solution 2: Configure Postfix SMTP Authentication (Recommended for Remote SMTP)

If using an external SMTP server that requires authentication, configure Postfix to authenticate using SASL:

1. Install SASL authentication packages:

# Debian/Ubuntu
sudo apt-get install libsasl2-modules

# RHEL/CentOS/AlmaLinux/Rocky
sudo yum install cyrus-sasl-plain

2. Configure Postfix to use the external SMTP relay:

Edit /etc/postfix/main.cf:

# Use external SMTP as relay host
relayhost = smtp.yourprovider.com:587

# Enable SASL authentication
smtp_sasl_auth_enable = yes

# Use SASL password file
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

# Disable anonymous authentication (use only SASL)
smtp_sasl_security_options = noanonymous

# Enable TLS (recommended)
smtp_tls_security_level = encrypt

3. Create the SASL password file with your SMTP credentials:

# Create the file (your SMTP username and password)
echo "[smtp.yourprovider.com]:587 username:password" | sudo tee /etc/postfix/sasl_passwd

# Secure the file (rw root only)
sudo chmod 600 /etc/postfix/sasl_passwd

# Create the Postfix hash database
sudo postmap /etc/postfix/sasl_passwd

# Reload Postfix
sudo systemctl reload postfix

4. Test email delivery:

echo "Test email" | mail -s "SMTP Relay Test" your.email@example.com

If successful, emails should be delivered through the authenticated SMTP relay.

Step 5: Verify Cronjob

Ensure the alert processing script runs every minute:

# Check current crontab
crontab -l

You should see:

* * * * * root php /var/www/html/php/run.php cron

If missing, add it:

crontab -e
# Add the line above, then reload cron
killall -HUP cron

Step 6: Verify Alert Configuration in GUI

After confirming the MTA works:

  1. Navigate to GUI > Alerts
  2. Verify alert conditions are enabled
  3. Check that recipient email addresses are valid
  4. Go to GUI > Alerts > Sent Alerts to see if alerts were triggered

Diagnosis:

  • Entries in "Sent Alerts" but no emails received → MTA issue
  • No entries in "Sent Alerts" → Check alert conditions or cronjob

Step 7: Test PHP mail() Function

Isolate the issue by testing PHP directly:

php -r "mail('your.email@example.com', 'Test from PHP', 'This is a test email');"
  • If this works but VoIPmonitor alerts don't → Check GUI cronjob and alert configuration
  • If this fails → MTA or PHP configuration issue

Troubleshooting Alerts Not Triggering

If alerts are not appearing in the Sent Alerts history at all, the problem is typically with the alert processor not evaluating alerts. This is different from MTA issues where alerts appear in history but emails are not sent.

Enable Detailed Alert Processing Logs

To debug why alerts are not being evaluated, enable detailed logging for the alert processor by adding the following line to your GUI configuration file:

# Edit the config file (adjust path based on your GUI installation)
nano ./config/system_configuration.php

Add this line at the end of the file:

<?php
define('CRON_LOG_FILE', '/tmp/alert.log');
?>

This enables logging that shows which alerts are being processed during each cron job run.

Increase Parallel Processing Threads

If you have many alerts or reports, the default number of parallel threads may cause timeout issues. Increase the parallel task limit:

# Edit the configuration file
nano ./config/configuration.php

Add this line at the end of the file:

<?php
define('CRON_PARALLEL_TASKS', 8);
?>

The value of 8 is recommended for high-load environments. Adjust based on your alert/report volume and server capacity.

Monitor Alert Processing Logs

After enabling logging, monitor the alert log file to see which alerts are being processed:

# Watch the log in real-time
tail -f /tmp/alert.log

The log shows entries like:

begin alert [alert_name]
end alert [alert_name]

Interpreting the logs:

  • If you do not see your alert name in the logs → The alert processor is not evaluating it. Check your alert configuration, filters, and data availability.
  • If you see the alert in logs but it does not trigger → The alert conditions are not being met. Check your thresholds, filter logic, and verify the CDR data matches your expectations.
  • If logs are completely empty → The cron job may not be running or the GUI configuration files are not being loaded. Verify the cron job and file paths.

Alert Not Appearing in Logs

If your alert does not appear in `/tmp/alert.log`:

1. Verify the cron job is running:

# Check the cron job exists
crontab -l

# Manually test the cron script to see errors
php ./php/run.php cron

2. Verify data exists in CDR:

# Check if the calls that should trigger the alert exist
# Navigate to GUI > CDR > Browse and filter for the timeframe

3. Check alert configuration:

  • Verify alert is enabled
  • Verify filter logic matches your data (IP addresses, numbers, groups)
  • Verify thresholds are reasonable for the actual QoS metrics
  • Verify GUI license is not locked (Check GUI > Settings > License)

"Crontab Log is Too Old" Warning - Database Performance Issues

The VoIPmonitor GUI displays a warning message "Crontab log is too old" when the last successful cron run timestamp exceeds the expected interval. While this often indicates a missing or misconfigured cron job, it can also occur when the database is overloaded and the cron script runs slowly.

Common Causes

  1. Missing or broken cron entry - The cron job does not exist in /etc/crontab or the command fails when executed
  2. Database overload - The cron job runs but completes slowly due to database performance bottlenecks, causing the "last run" timestamp to drift outside the expected window

Distinguishing the Causes

Use the following diagnostic workflow to determine if the issue is cron configuration vs. database performance:

Step 1: Verify the cron job is actually running

Check if the cron execution timestamp is updating (even if slowly):

# Check the current cron timestamp from the database
mysql -u voipmonitor -p voipmonitor -e "SELECT name, last_run FROM scheduler LIMIT 1"

# The last_run timestamp should update at least every few minutes
# If it never updates, the cron is not running (see Step 2)
# If it updates but lags by more than 5-10 minutes, it's a performance issue (see Step 3)

Step 2: If cron is not running at all

Follow the standard cron setup instructions in the "Setting Up the Cron Job" section above. Common issues:

  • Cron entry missing from /etc/crontab
  • Incorrect PHP path (use full path like /usr/bin/php instead of php)
  • PHP CLI missing IonCube loader (check with `php -r 'echo extension_loaded("ionCube Loader")?"yes":"no";'`)
  • Wrong file permissions or incorrect web directory path

Step 3: If cron runs but slowly (database performance issue)

When the cron job runs but takes a long time to complete, the issue is database overload. Diagnose using the Sensors statistics:

  1. Navigate to GUI > Settings > Sensors
  2. Click on the sensor status to view detailed statistics
  3. Compare the following timestamps:
    • Last CDR in database** - The timestamp of the most recently completed call stored in MySQL
    • Last CDR in processing queue** - The timestamp of the most recent call reached by the sniffer

If there is a significant delay (minutes or more) between these two timestamps during peak traffic, the database cannot keep up with CDR insertion. This causes alert/reports processing (run.php cron) to also run slowly.

Solutions for Database Performance Issues

    • 1. Check MySQL configuration**

Ensure your MySQL/MariaDB configuration follows the recommended settings for your call volume. Key parameters:

  • innodb_flush_log_at_trx_commit - Set to 2 for better performance (or 0 in extreme high-CPS environments)
  • innodb_buffer_pool_size - Allocate 70-80% of available RAM for high-volume deployments
  • innodb_io_capacity - Match your storage system capabilities (e.g., 1000000 for NVMe SSDs)

See Scaling and High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for detailed tuning guides.

    • 2. Increase database write threads**

In /etc/voipmonitor.conf, increase the number of threads used for writing CDRs:

mysqlstore_max_threads_cdr = 8   # Default is 4, increase based on workload
    • 3. Monitor SQL queue statistics**

In the expanded status view (GUI > Settings > Sensors > status), check the SQLq value:

  • SQLq (SQL queue) growing steadily - Database is a bottleneck, calls are waiting in memory
  • SQLq remains low (under 1000) - Database is keeping up, may need other tuning

See SQL_queue_is_growing_in_a_peaktime for more information.

    • 4. Reduce alert/report processing load**

Too many alert rules or complex reports can exacerbate the problem:

  • Review and disable unnecessary alerts in GUI > Alerts
  • Reduce the frequency of daily reports (edit in GUI > Reports)
  • Increase parallel processing tasks: In /var/www/html/configuration.php, set define('CRON_PARALLEL_TASKS', 8); (requires increasing PHP memory limits)
    • 5. Check database query performance**

Identify slow queries:

# Enable slow query logging in my.cnf
slow_query_log = 1
long_query_time = 2

# After waiting for a cron cycle, check the slow query log
tail -f /var/log/mysql/slow.log

Look for queries taking more than a few seconds. Common culprits:

  • Missing indexes on frequently filtered columns (caller, callee, sipcallerip, etc.)
  • Complex alert conditions joining large tables
  • Daily reports scanning millions of rows without date range limitations
    • 6. Scale database architecture**

For very high call volumes (4000+ concurrent calls), consider:

  • Separate database server from sensor hosts
  • Use MariaDB with LZ4 page compression
  • Implement database replication for read queries
  • Use hourly table partitioning for improved write performance

See High-Performance_VoIPmonitor_and_MySQL_Setup_Manual for architecture recommendations.

Verification

After applying fixes:

1. Monitor the "Crontab log is too old" timestamp in the GUI

  * The timestamp should update every 1-3 minutes during normal operation
  * If it still lags by 10+ minutes, further tuning is required

2. Check sensor statistics (GUI > Settings > Sensors)

  * The delay between "Last CDR in database" and "Last CDR in processing queue" should be under 1-2 minutes during peak load
  * SQLq should remain below 1000 and not grow continuously

3. Test alert processing manually

# Run the cron script manually and measure execution time
time php /var/www/html/php/run.php cron

# Should complete within 10-30 seconds in most environments
# If it takes longer than 60-120 seconds, database tuning is needed

See Also

AI Summary for RAG

Summary: VoIPmonitor Alerts & Reports system for email notifications on QoS and SIP issues. Covers RTP alerts (MOS, jitter, packet loss), SIP response alerts (including detecting 408 Request Timeout from 100 Trying scenarios), sensors health monitoring, SIP REGISTER RRD beta alerts for monitoring registration response times, multiple register (beta) alerts for detecting SIP accounts registered from multiple IP addresses (useful for detecting credential theft or unauthorized access), and creating alerts from custom reports based on CDR custom headers. Includes email troubleshooting for MTA configuration, detailed debugging for alerts not triggering using CRON_LOG_FILE and CRON_PARALLEL_TASKS, and external scripts for webhook integration (Datadog, Slack, third-party monitoring). Also provides troubleshooting for "Crontab log is too old" warning which can indicate database performance issues when the cron job runs slowly due to database overload - diagnostic workflow includes checking scheduler.last_run timestamp and comparing "Last CDR in database" vs "Last CDR in processing queue" in GUI > Settings > Sensors; solutions include MySQL/MariaDB tuning (innodb_flush_log_at_trx_commit, innodb_buffer_pool_size, innodb_io_capacity), increasing mysqlstore_max_threads_cdr, monitoring SQLq statistics, reducing alert/report processing load, checking slow query logs, and scaling database architecture for high call volumes (separate database host, MariaDB LZ4 compression, hourly table partitioning).

Keywords: alerts, email notifications, QoS, MOS, jitter, packet loss, SIP response, 408 Request Timeout, 100 Trying, response code 0, INVITE retransmissions, sensors monitoring, SIP REGISTER RRD beta, REGISTER retransmissions, registration monitoring, multiple register beta, multiple IP addresses, SIP account compromise, credential theft, security alerts, unauthorized SIP usage, crontab, MTA, Postfix, Exim, troubleshooting, custom headers, custom reports, Max-Forwards, daily reports, CRON_LOG_FILE, CRON_PARALLEL_TASKS, alert not triggering, /tmp/alert.log, begin alert, end alert, configuration.php, system_configuration.php, external scripts, webhooks, Datadog, Slack, command-line arguments, JSON data, CDR array, Crontab log is too old, database performance, database overload, cron running slowly, scheduler.last_run, last CDR in database, last CDR in processing queue, Settings Sensors, SQLq, SQL queue growing, MySQL tuning, innodb_flush_log_at_trx_commit, innodb_buffer_pool_size, innodb_io_capacity, mysqlstore_max_threads_cdr, slow query log, database scaling, hourly table partitioning, MariaDB LZ4 compression

Key Questions:

  • How do I set up email alerts in VoIPmonitor?
  • What types of alerts are available (RTP, SIP, Sensors, REGISTER RRD beta, multiple register beta)?
  • How do I detect calls with 408 Request Timeout after 100 Trying?
  • How do I create an alert for calls where 100 Trying is sent but no further response?
  • What is the difference between response code 0 and 408?
  • How do I monitor and alert on SIP REGISTER retransmissions?
  • How do I detect registration response time issues?
  • How can I use SIP REGISTER RRD beta alert for detecting switch problems?
  • How do I detect SIP accounts registered from multiple IP addresses?
  • How do I use multiple register (beta) alert to find accounts with multiple IP registrations?
  • How do I configure multiple register (beta) alert to check all SIP numbers across all customers?
  • How do I detect SIP account compromise or credential theft using alerts?
  • How do I monitor for unauthorized SIP account usage?
  • How do I configure crontab for alert processing?
  • How do I monitor remote sensor health?
  • Why are email alerts not being sent?
  • How do I troubleshoot MTA email issues?
  • How can I create alerts based on SIP headers like Max-Forwards?
  • How do I use CDR custom headers for custom reports?
  • How do I limit the size of alert emails from custom reports?
  • Why are alerts not triggering or appearing in Sent Alerts?
  • How do I enable detailed alert processing logs using CRON_LOG_FILE?
  • How do I increase parallel alert processing threads with CRON_PARALLEL_TASKS?
  • How do I monitor /tmp/alert.log for alert processing debug information?
  • How do I configure external scripts for alerts?
  • How do I send webhooks from VoIPmonitor alerts to Datadog?
  • How do I send alerts to Slack from VoIPmonitor?
  • What command-line arguments are passed to alert scripts?
  • What JSON data structure is provided to external scripts?
  • What does "Crontab log is too old" warning mean in VoIPmonitor?
  • How do I troubleshoot "Crontab log is too old" error when cron job exists?
  • How do I check if cron job is actually running in VoIPmonitor?
  • How do I distinguish between cron configuration issues and database performance issues?
  • How do I check scheduler.last_run timestamp in VoIPmonitor database?
  • How do I compare "Last CDR in database" vs "Last CDR in processing queue" in GUI?
  • Where do I find processing delay statistics in VoIPmonitor GUI?
  • What causes cron job to run slowly in VoIPmonitor?
  • How do I fix database overload causing slow cron execution?
  • What MySQL settings should I tune for high call volumes in VoIPmonitor?
  • How do I increase mysqlstore_max_threads_cdr for better database performance?
  • What does SQLq value mean in VoIPmonitor sensor status?
  • How do I monitor SQL queue growth in VoIPmonitor?
  • How do I reduce alert and report processing load on VoIPmonitor database?
  • How do I use CRON_PARALLEL_TASKS to improve cron processing performance?
  • How do I check for slow MySQL queries in VoIPmonitor?
  • What should I do if "Crontab log is too old" persists after adding cron job?