High availability redundancy failover

VoIPmonitor supports full redundancy and seamlessly failover switch. VoIPmonitor is installed on two servers which are both connected to mirroring switch where each server receives the same SIP/RTP traffic. MySQL is configured to Master-Master replication. Active node activly writes CDR to database while the passive is running but CDR are turned off but still writing pcap files on disk. Once active node dies or switches to maintainance mode the secondary nodes takes a shared IP and activates writing to CDR database.

This guide is for Debian 6

configure voipmonitor sniffer with redundancy

topology

Node1 IP:10.0.0.1 
Node2 IP:10.0.0.2
Shared IP 10.0.0.128

Installing hearbeat

Installing HA

apt-get install heartbeat 

edit /etc/hosts

127.0.0.1       localhost
10.0.0.1   voipmonitor1
10.0.0.2   voipmonitor2

edit /etc/hostname to match voipmonitor1 and voipmonitor2

allow binding of shared ip adress by editing /etc/sysctl.conf adding the following line (lb1&lb2)

net.ipv4.ip_nonlocal_bind=1

run sysctl to activate changes

sysctl -p

generate file /etc/ha.d/authkeys on both nodes following content:

auth 3
3 md5 somerandomstring

set permissions

chmod 600 /etc/ha.d/authkeys

create file /etc/ha.d/ha.cf on both nodes

#
#       keepalive: how many seconds between heartbeats
#
keepalive 2
#
#       deadtime: seconds-to-declare-host-dead
#
deadtime 10
#
#       What UDP port to use for udp or ppp-udp communication?
#
udpport        694
bcast  eth0
# mcast eth0 225.0.0.1 694 1 0
ucast eth0 10.0.0.2
#       What interfaces to heartbeat over?
udp     eth0
#
#       Facility to use for syslog()/logger (alternative to log/debugfile)
#
#logfacility     local0
#
#       Tell what machines are in the cluster
#       node    nodename ...    -- must match uname -n
node    voipmonitor1
node    voipmonitor2

on node2 change ucast eth0 10.0.0.2 to ucast eth0 10.0.0.1

on both nodes create file /etc/ha.d/haresources with the same exact content (do not be confused that the voipmonitor1 is also on node2 this in fact tells that voipmonitor1 is primary node)

voipmonitor1 10.0.0.128 IPsrcaddr::10.0.0.128 voipmonitor

this harseources will activate IP 10.0.0.128 and runs voipmonitor script (it is run from left to right once the node becomes master). When the node becomes slave it will run the script from right to left with parametr stop)

Create file in /etc/ha.d/resource.d/voipmonitor

#!/bin/bash
#
# This script is inteded to be used as resource script by heartbeat
#
# May 2012 by Martin Vit
#
###

. /etc/ha.d/shellfuncs
case "$1" in
    start)
        echo "enablecdr" | nc localhost 5029 >/dev/null 2>/dev/null
        ;;
    stop)
        echo "disablecdr" | nc localhost 5029 >/dev/null 2>/dev/null
        ;;
    status)
        ;;
    *)
        echo "Usage: {start|stop|status}"
        exit 1
        ;;
esac
exit 0

configuring db's redundancy

topology

Node1 IP:10.0.0.1 
Node2 IP:10.0.0.2
Shared IP 10.0.0.128

MySQL master master replication

On both nodes edit /etc/mysql/my.cf change bind-adress from 127.0.0.1 to 0.0.0.0

bind-address            = 0.0.0.0

on voipmonitor1 edit file /etc/mysql/my.cf and add this to [mysqld] section

server-id = 1
replicate-same-server-id = 0
auto-increment-increment = 2
auto-increment-offset = 1
replicate-do-db = voipmonitor
log_bin = /var/log/mysql/mysql-bin.log

on voipmonitor2 edit file /etc/mysql/my.cf and add this to [mysqld] section

server-id = 2
replicate-same-server-id = 0
auto-increment-increment = 2
auto-increment-offset = 2
replicate-do-db = voipmonitor
log_bin = /var/log/mysql/mysql-bin.log

Run this mysql commands on both nodes:

GRANT REPLICATION SLAVE ON *.* to 'replication'@'%' identified by 'fMHiMFIQ'; 
FLUSH PRIVILEGES;

mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 |     3172 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

note the File and Position values

On voipmonitor1 run

mysql> CHANGE MASTER TO
      MASTER_HOST='10.0.0.2',
      MASTER_PORT=3306,
      MASTER_USER='replication',
      MASTER_PASSWORD='fMHiMFIQ',
      MASTER_LOG_FILE='mysql-bin.000002',
      MASTER_LOG_POS=3172;

START SLAVE;

where MASTER_LOG_FILE and MASTER_LOG_POS is the one you get by running SHOW MASTER STATUS on voipmonitor2 node.

On voipmonitor2 run

mysql> CHANGE MASTER TO
      MASTER_HOST='10.0.0.1',
      MASTER_PORT=3306,
      MASTER_USER='replication',
      MASTER_PASSWORD='fMHiMFIQ',
      MASTER_LOG_FILE='mysql-bin.000002',
      MASTER_LOG_POS=3172;

START SLAVE;

where MASTER_LOG_FILE and MASTER_LOG_POS is the one you get by running SHOW MASTER STATUS on voipmonitor1 node. (do not be confused that in our example the LOG_FILE and LOG_POS are same for both nodes. It can be different)

Now you can check status on both nodes with

mysql> show slave status;
+----------------------------------+-------------+-------------+-------------+---------------+------------------+---------------------+-------------------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+
| Slave_IO_State                   | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File  | Read_Master_Log_Pos | Relay_Log_File          | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |
+----------------------------------+-------------+-------------+-------------+---------------+------------------+---------------------+-------------------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+
| Waiting for master to send event | 10.0.0.2    | replication |        3306 |            60 | mysql-bin.000002 |                2492 | mysqld-relay-bin.000005 |           251 | mysql-bin.000002      | Yes              | Yes               | voipmonitor     |                     |                    |                        |                         |                             |          0 |            |            0 |                2492 |             552 | None            |                |             0 | No                 |                    |                    |                 |                   |                |                     0 | No                            |             0 |               |              0 |                |
+----------------------------------+-------------+-------------+-------------+---------------+------------------+---------------------+-------------------------+---------------+-----------------------+------------------+-------------------+-----------------+---------------------+--------------------+------------------------+-------------------------+-----------------------------+------------+------------+--------------+---------------------+-----------------+-----------------+----------------+---------------+--------------------+--------------------+--------------------+-----------------+-------------------+----------------+-----------------------+-------------------------------+---------------+---------------+----------------+----------------+
1 row in set (0.00 sec)