Disaster Recovery

From VoIPmonitor.org
Revision as of 20:51, 7 January 2026 by Admin (talk | contribs) (Add disaster recovery guide for disk failure scenarios)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


This guide describes the recovery process when a VoIPmonitor server experiences a disk failure requiring full operating system and application reinstallation. This applies to sniffer servers, GUI servers, and all-in-one deployments.

Overview

When a disk fails and requires replacement, the recovery process involves coordination between you (the customer) and VoIPmonitor support staff. This is not a self-service procedure - VoIPmonitor support will handle the actual VoIPmonitor installation and configuration after you prepare the system.

Key Concept: Support-Coordinated Recovery
Customer Tasks Replace failed disks, install operating system, provide remote access
Support Tasks Install VoIPmonitor, configure database connection, restore configuration

This approach ensures the system is properly configured and minimizes downtime. However, you can perform some components yourself if you have the necessary expertise.

Recovery Workflow

The recovery process consists of these phases:

Step 1: Hardware Replacement

Replace the failed physical disks on the server with new disks of adequate capacity.

ℹ️ Note: Disk capacity should be at least equal to or greater than the original disks, especially if you plan to restore PCAP data from backups.

Step 2: Operating System Installation

Install a supported operating system on the new disks. Common choices include:

  • Ubuntu 24.04 LTS (recommended for new deployments)
  • Debian 12 (Bookworm)
  • Rocky Linux 9**
  • CentOS 7.2** (older systems, see migration notes)

⚠️ Warning: Do not attempt to copy binaries from the old disk if the OS version differs significantly. Always install VoIPmonitor from scratch using the official installer for the new OS.

Step 3: Network Configuration

Configure the network interfaces to match the original server's configuration:

  • Set the same IP address (if required for fixed network configuration)
  • Configure DNS servers
  • Ensure proper routing

This is critical for：

  • Remote access for support staff
  • Connection to central database/GUI servers (for distributed deployments)
  • SPAN/TAP port configuration (for sniffer servers)

Step 4: Timezone Configuration (CRITICAL)

Ensure the new server's timezone matches the old server exactly.

Why Is This Critical?
Problem PCAP file paths in VoIPmonitor are based on the local OS time. A timezone mismatch will prevent the GUI from finding historical recordings. The spooldir follows the format /var/spool/voipmonitor/YYYY/MM/DD based on local time.
Solution Match the original timezone exactly before starting capture or restore operations. 
# Check current timezone
timedatectl

# Set timezone (example: Europe/Prague)
sudo timedatectl set-timezone Europe/Prague

# Verify timezone is set correctly
timedatectl

Step 5: Provide Remote Access to Support Staff

Arrange remote access for VoIPmonitor support staff to perform the VoIPmonitor installation. Choose one of the following methods:

ℹ️ Note: If you have internal IT staff with VoIPmonitor expertise, you can perform the installation yourself using the standard Sniffer Installation Guide or GUI Installation Guide.

= Method 1: SSH Access

Provide direct SSH access to the server:

  • Create a dedicated support user account (optional but recommended)
  • Alternatively, provide temporary root access
  • Disable password authentication and use SSH keys for security
# Create support user (optional)
adduser supportuser
usermod -aG sudo supportuser

# Ensure SSH is running
systemctl status sshd  # or ssh service name depending on OS

= Method 2: VPN Access

If your network uses a VPN, add VoIPmonitor support staff to the VPN configuration and provide:

  • VPN connection details
  • Server internal IP address
  • Any required authentication

= Method 3: Reverse SSH Tunnel

If the server is behind a firewall and does not have direct SSH access from the internet, set up a reverse SSH tunnel:

`syntaxhighlight lang="bash">`

  1. On a server that has inbound SSH access (e.g., your bastion host)
  2. This command allows support to connect via the bastion to your server
  1. From your internal server (run this command):

ssh -R 2222:localhost:22 -N support@bastion.yourdomain.com

  1. VoIPmonitor support can now connect by SSHing to bastion.yourdomain.com:2222

`</syntaxhighlight>`


Step 6: VoIPmonitor Installation by Support Staff

VoIPmonitor support staff will perform the following tasks:

= For Sniffer Servers (Distributed Mode)

1. Install VoIPmonitor sniffer binary 2. Configure network interface for packet captureSPAN, TAP, or mirror port) 3. Connect sniffer to the central database/GUI server 4. Restore `voipmonitor.conf` configuration or recreate from documentation 5. Capture rules and sensor settings 6. Start the service and verify connectivity

= For GUI/Database Servers

1. Install MariaDB/MySQL database 2. Install VoIPmonitor web GUI 3. Install PHP dependencies 4. Restore GUI configuration from backups if available 5. Restore database if backups exist 6. Start services and verify web interface

= For All-in-One Servers

1. Install sniffer, GUI, and database components 2. Restore full configuration from backups 3. Reconstruct database from PCAP files if backups are unavailable (see below) 4. Verify all components are functioning

Step 7: Restore Data (If Available)

Data recovery depends on what backups or recoverable PCAP files you have.

= Case 1: Database Backups Available

If you have database backups from a backup procedure or mysqldump:

1. Restore database using mysql command-line client 2. Verify restored data integrity 3. Check that GUI connects to database correctly

`syntaxhighlight lang="bash">`

  1. Restore from mysqldump backup

mysql -u root -p voipmonitor < /path/to/backup.sql `</syntaxhighlight>`


= Case 2: PCAP Files Survived, Database Lost

If the disk failed but PCAP files can be recovered (e.g., from a separate storage volume or mounted old disk), you can reconstruct the database by processing the PCAP files.

{syntaxhighlight lang="bash">`

  1. 1. Restore PCAP files to the spooldir

rsync -avz /mnt/old_disk/var/spool/voipmonitor/ /var/spool/voipmonitor/

  1. 2. Ensure database server is running with a fresh database

mysql -e "CREATE DATABASE voipmonitor;"

  1. 3. Restore GUI configuration first (if available)
  2. This ensures users, sensors, and settings are in place before processing PCAPs
  1. 4. Run sniffer in restore mode to rebuild database from PCAPs

voipmonitor --config-file=/etc/voipmonitor.conf --readpcapdir /var/spool/voipmonitor `</syntaxhighlight>`

ℹ️ Note: Processing PCAP files in restore mode can take significant time depending on data volume. The duration scales with number of calls and PCAP file sizes.

If the disk is completely destroyed without backups:

  • Historical PCAP data is PERMANENTLY LOST
  • CDRs and call recordings cannot be recovered
  • Only configuration can be restored from backups

= Case 3: No Backups, No Recoverable Data

If all data was on the failed disk and no backups exist:

  • All historical data is permanently lost
  • System must be reconfigured from scratch
  • Ensure proper backups are implemented after recovery (see Prevention section below)

Step 8: Service Verification

After installation and restore operations, verify all services are running correctly.

`syntaxhighlight lang="bash">`

  1. Check VoIPmonitor service status

systemctl status voipmonitor

  1. Check web GUI service (if applicable)

systemctl status apache2 # or nginx, depending on configuration

  1. View service logs for errors

journalctl -u voipmonitor -f tail -f /var/log/voipmonitor.log `</syntaxhighlight>`

Verify in the GUI:

  • Sensors appear as connected (for distributed mode)
  • New calls are being captured
  • Historical recordings appear (if restored)
  • Reports and dashboards work correctly

Self-Service Installation (Advanced) =

If you have VoIPmonitor expertise and prefer to perform the installation yourself, follow these guides:

⚠️ Warning: Incorrect configuration can cause data loss or monitoring gaps. If unsure, contact VoIPmonitor support for assistance.

Recovery by Deployment Type

Sniffer-Only (Client Mode) Recovery

In distributed deployments where the sniffer sends data to a remote central server:

Recovery Complexity Key Tasks
Data Loss LOW (typically) PCAPs usually stored separately or on central server
Configuration MEDIUM Restore `voipmonitor.conf`, capture rules, sensor name
Network HIGH Must reconnect to central database/GUI server

{syntaxhighlight lang="bash">`

  1. Critical parameters for sniffer-only recovery

interface = eth0 # Network interface for packet capture name_sensor = SensorName # Must match GUI sensor configuration

  1. Connection to central server

server_destination = central.server.ip server_destination_port = 60024 `</syntaxhighlight>`


All-in-One Server Recovery

When sniffer, GUI, and database were all on the failed disk:

! Recovery Complexity Key Tasks
Data Loss Potentially SEVERE All data (CDRs, PCAPs, config) on same disk
Configuration HIGH Must restore/recreate all components
Reconstruction POSSIBLE (if PCAPs survive) Regenerate database from PCAP files

For all-in-one scenarios, prioritize data recovery:

1. Attempt to mount old disk and extract PCAP files 2. If successful, reconstruct database (see case 2 above) 3. If unsuccessful, start fresh and monitor going forward

Prevention: Future Disaster Recovery =

To minimize impact of future disk failures, implement the following:

1. **Automated GUI Backups** 

  Follow the  GUI Automated Backup Guide to schedule daily backups of configuration, users, sensors, alerts, and dashboards.

2. **Separate PCAP Storage** 

  Move `/var/spool/voipmonitor` to a NAS, SAN, or separate physical disk that survives OS disk failures.

3. **Database Replication** 

  Set up redundant using database replication for high availability.

4. **Database Backups** 

  Schedule regular mysqldump backups and store them off-site.

{syntaxhighlight lang="bash">`

  1. Example cron job for daily database backup

0 2 * * * mysqldump -u root -pPASSWORD voipmonitor | gzip > /backup/db_backup_$(date +\%Y\%m\%d).sql.gz `</syntaxhighlight>`

5. **Monitor Disk Health** 

  Use SMART tools to monitor disk health and predict failures before they occur.

{syntaxhighlight lang="bash">`

  1. Check disk SMART status

smartctl -a /dev/sda

  1. Install smartmontools if not available

apt-get install smartmontools `</syntaxhighlight>`

Summary for Self-Service vs Support-Managed Recovery

Self-Service Support-Managed
Advantages Expert installation, proper configuration, reduced errors
Disadvantages Requires coordination, scheduling
Recommended For Production environments, complex setups, data recovery

AI Summary for RAG

Summary: Disaster recovery workflow after disk failure. Customer replaces hardware, installs OS (Ubuntu 24.04, Debian 12, Rocky 8/9, CentOS 7.2), configures network, sets timezone (critical for PCAP path matching), provides remote access to VoIPmonitor support (SSH, VPN, or reverse SSH tunnel). Support staff installs VoIPmonitor components, connects to database/GUI, restores configuration. Recovery scenarios: sniffer-only (low data loss, reconnect to central), all-in-one (potentially severe data loss, reconstruct database from PCAPs if available). Data recovery options: (1) Database backups, restore via mysql/mysqldump. (2) PCAP files survive: use voipmonitor --readpcapdir /var/spool/voipmonitor to reconstruct CDRs from PCAPs, can take significant time based on data volume. (3) No backups: data permanently lost, reconfigure from scratch. Prevention: automated GUI backups via Backing_Up_GUI_Configuration, separate PCAP storage, database replication via Redundant_database, regular mysqldump, use smartctl for disk health monitoring. Self-service possible if you have VoIPmonitor expertise, use Sniffer Installation Guide.

Keywords: disaster recovery, disk failure, data recovery, OS reinstall, remote access, SSH, VPN, reverse SSH tunnel, timezone settings, PCAP restoration, database reconstruction, spooldir, voipmonitor.conf, all-in-one, distributed mode, database backup, mysqldump, smartctl, prevention, NAS, SAN, high availability

Key Questions:

  • What do I do after a disk failure on my VoIPmonitor server?
  • How do I recover VoIPmonitor after a complete disk failure?
  • Do I need VoIPmonitor support for disaster recovery?
  • What information do I need to provide to support staff after a disk failure?
  • What are the steps for providing remote access to support staff (SSH, VPN, reverse tunnel)?
  • Why is timezone configuration critical after disk failure?
  • Can I recover my PCAP files if the disk failed?
  • How do I reconstruct the VoIPmonitor database from PCAP files?
  • What is the difference between sniffer-only and all-in-one recovery?
  • How can I prevent data loss from future disk failures?
  • What backups should I have in place for disaster recovery?
  • Should I attempt self-service recovery or wait for support?
Retrieved from "https://www.voipmonitor.org/doc/index.php?title=Disaster_Recovery&oldid=9922"