mysql-mmm——實現mysql互為主從復制HA功能

                   
每個mysql服務器節點都需要運行mmmd_agent,同時在另外一臺機器【可以是獨立的一臺服務器也可以是和AppServer共同享一個服務器】上運行mmmd_mon
mmm利用瞭虛擬IP技術,1個網卡可以使用多個IP
所以使用mmm時,需要2*n+1個IP,n為mysql節點數,包括slave和master
當數據庫節點fail時,mmmd_mon檢測不到mmmd_agent的心跳或者對應的mysql服務器的狀態時
mmmd_mon將作出決定並下令給某個正常的數據庫節點的mmmd_agent,使得該mmmd_agent“篡位”
【即 使用剛才fail的那個結點的虛擬IP,使得虛擬IP從fail結點指向此時的正常機器】

 

 

 

MMMM需要5個IP,兩個節點使用固定IP,兩個程式讀IP(隻讀),1個 程式寫IP(用來更新)
後面這三個虛擬IP是根據節點的可用性在節點之間實現跳轉的

 

一。IP分配
IP分配如下:
A :mysql master 246
固定IP:211.100.97.246
程式讀IP:211.100.97.244  (虛擬)

B:mysql master 250
固定IP:211.100.97.250
程式讀IP:211.100.97.243  (虛擬)

monitor 245
程式寫IP:211.100.97.248  (虛擬)


給246添加虛擬IP 211.100.97.244
 ifconfig eth1:1 211.100.97.244 netmask 255.255.255.224 up
[root@XKWB5510 software]# ifconfig -a|grep "inet addr"|head -3|tail -2|awk -F "[ :]+" '{print $4"/"$NF}'
211.100.97.246/255.255.255.224
211.100.97.244/255.255.255.224

給250添加虛擬IP 211.100.97.243
 ifconfig eth0:1 211.100.97.243 netmask 255.255.255.224 up
 [root@XKWB5705 software]# ifconfig -a|grep "inet addr"|head -2|awk -F "[ :]+" '{print $4"/"$NF}'
211.100.97.250/255.255.255.224
211.100.97.243/255.255.255.224

給245添加虛擬IP:211.100.97.248
 ifconfig eth1:1 211.100.97.248 netmask 255.255.255.224 up
[root@CentOS mysql-5.1.56]# ifconfig -a|grep "inet addr"|head -3|tail -2|awk -F "[ :]+" '{print $4"/"$NF}'
211.100.97.245/255.255.255.224
211.100.97.248/255.255.255.224

二 授權
在AB機器添加代理賬號 useradd rep_agent
在monitor機器上添加監控賬號 useradd rep_monitor

A上授權
mysql> grant all privileges on *.* to 'rep_monitor'@'%' identified by '123456';
mysql> grant all privileges on *.* to 'rep_agent'@'%' identified by '123456';

查看授權情況
mysql> select host,user from mysql.user where user like 'rep%';
+—————-+————-+
| host           | user        |
+—————-+————-+
| %              | rep_agent   |
| %              | rep_monitor |
| 211.100.97.250 | replication |
| localhost      | replication |
+—————-+————-+
4 rows in set (0.01 sec)


B上授權
mysql> grant all privileges on *.* to 'rep_monitor'@'%' identified by '123456';
mysql> grant all privileges on *.* to 'rep_agent'@'%' identified by '123456';

mysql> select host,user from mysql.user where user like 'rep%';
+—————-+————-+
| host           | user        |
+—————-+————-+
| %              | rep_agent   |
| %              | rep_monitor |
| 211.100.97.246 | replication |
| localhost      | replication |
+—————-+————-+
4 rows in set (0.00 sec)


三 mmm安裝
CentOS軟件倉庫默認是不含這些軟件的,必須要有epel這個包的支持。故我們必須先安裝epel:
wget  http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm

源碼包安裝mysql-mmm
wget http://mysql-master-master.googlecode.com/files/mysql-master-master-1.2.3.tar.gz
tar zxf mysql-master-master-1.2.3.tar.gz
cd mysql-master-master-1.2.3
 ./install.pl 
 
 
 
另外安裝mmm之前需要安裝以下幾個必須的perl模塊
Data::Dumper
POSIX
Cwd
threads
threads::shared
Thread::Queue
Thread::Semaphore
IO::Socket
Proc::Daemon
Time::HiRes
DBI
DBD::mysql
Algorithm::Diff

否則在安裝mmm執行install.pl命令的時候,會出現如下報錯:
1)
Checking required module 'Proc::Daemon'…Error!
 Can't locate Proc/Daemon.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at (eval 13) line 2.
BEGIN failed–compilation aborted at (eval 13) line 2.

————————————————————
Required module 'Proc::Daemon' is not found on this system!
Install it (e.g. run command 'cpan Proc::Daemon') and try again.

根據以上報錯提示運行
cpan Proc::Daemon    基本上都是回車

2)
Checking required module 'DBI'…Error!
 Can't locate DBI.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at (eval 16) line 2.
BEGIN failed–compilation aborted at (eval 16) line 2.

————————————————————
Required module 'DBI' is not found on this system!
Install it (e.g. run command 'cpan DBI') and try again.
++++++++++++++++++++++++++++
根據報錯提示運行命令
cpan DBI

3)
Checking required module 'DBD::mysql'…Error!
 Can't locate DBD/mysql.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at (eval 19) line 2.
BEGIN failed–compilation aborted at (eval 19) line 2.

————————————————————
Required module 'DBD::mysql' is not found on this system!
Install it (e.g. run command 'cpan DBD::mysql') and try again.


運行這個命令cpan DBD::mysql
報錯如下:
CPAN.pm: Going to build C/CA/CAPTTOFU/DBD-mysql-4.020.tar.gz

Can't exec "mysql_config": No such file or directory at Makefile.PL line 83.

Cannot find the file 'mysql_config'! Your execution PATH doesn't seem 
not contain the path to mysql_config. Resorting to guessed values!
Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located


PLEASE NOTE:

For 'make test' to run properly, you must ensure that the 
database user 'root' can connect to your MySQL server 
and has the proper privileges that these tests require such 
as 'drop table', 'create table', 'drop procedure', 'create procedure'
as well as others.

mysql> grant all privileges on test.* to 'root'@'localhost' identified by 's3kr1t';

You can also optionally set the user to run 'make test' with:

perl Makefile.PL –testuser=username

Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located
Can't exec "mysql_config": No such file or directory at Makefile.PL line 478.
Can't find mysql_config. Use –mysql_config option to specify where mysql_config is located
Failed to determine directory of mysql.h. Use

  perl Makefile.PL –cflags=-I<dir>

to set this directory. For details see the INSTALL.html file,
section "C Compiler flags" or type

  perl Makefile.PL –help
Running make test
  Make had some problems, maybe interrupted? Won't test
Running make install
  Make had some problems, maybe interrupted? Won't install


根據以上腦挫提示查看是否有mysql_config
[root@XKWB5510 mysql-master-master-1.2.3]# find /usr/local/mysql/  -name "mysql_config*"
/usr/local/mysql/bin/mysql_config
/usr/local/mysql/share/man/man1/mysql_config.1

看一下mysql_config的權限
[root@XKWB5510 mysql-master-master-1.2.3]# ls -l /usr/local/mysql/bin/mysql_config
-rwxr-xr-x 1 root root 6105 Sep 21 22:43 /usr/local/mysql/bin/mysql_config

解決辦法
URL:http://search.cpan.org/dist/DBD-mysql/
DBD-mysql-4.020.tar.gz 
tar zxf DBD-mysql-4.020.tar.gz
cd DBD-mysql-4.020
perl Makefile.PL –mysql_config=/usr/local/mysql/bin/mysql_config
make
make install


4)
Checking required module 'Algorithm::Diff'…Error!
 Can't locate Algorithm/Diff.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at (eval 20) line 2.
BEGIN failed–compilation aborted at (eval 20) line 2.

————————————————————
Required module 'Algorithm::Diff' is not found on this system!
Install it (e.g. run command 'cpan Algorithm::Diff') and try again.
+++++++++++++++++
運行
cpan Algorithm::Diff

一直跟著錯誤提示走就行,直到沒有Error

5)
最後安裝成功的提示是這樣的:
[root@XKWB5705 mysql-master-master-1.2.3]# ./install.pl 
Checking platform support… linux Ok!
Checking required module 'Data::Dumper'…Ok!
Checking required module 'POSIX'…Ok!
Checking required module 'Cwd'…Ok!
Checking required module 'threads'…Ok!
Checking required module 'threads::shared'…Ok!
Checking required module 'Thread::Queue'…Ok!
Checking required module 'Thread::Semaphore'…Ok!
Checking required module 'IO::Socket'…Ok!
Checking required module 'Proc::Daemon'…Ok!
Checking required module 'Time::HiRes'…Ok!
Checking required module 'DBI'…Ok!
Checking required module 'DBD::mysql'…Ok!
Checking required module 'Algorithm::Diff'…Ok!
Checking iproute installation…Ok!
Installing mmm files…
Confgiuration:
  – installation directory: '/usr/local/mmm'
  – create symlinks: on
  – symlinks directory: '/usr/local/sbin'

Copying files to '/usr/local/mmm' directory…Ok!

Creating symlink: '/usr/local/mmm/sbin/mmm_control' -> '/usr/local/sbin/mmm_control'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmmd_agent' -> '/usr/local/sbin/mmmd_agent'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmm_restore' -> '/usr/local/sbin/mmm_restore'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmmd_angel' -> '/usr/local/sbin/mmmd_angel'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmm_get_dump' -> '/usr/local/sbin/mmm_get_dump'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmm_backup' -> '/usr/local/sbin/mmm_backup'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmm_clone' -> '/usr/local/sbin/mmm_clone'…Ok!
Creating symlink: '/usr/local/mmm/sbin/mmmd_mon' -> '/usr/local/sbin/mmmd_mon'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmmd_mon.1' -> '/usr/local/man/man1/mmmd_mon.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmm_restore.1' -> '/usr/local/man/man1/mmm_restore.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmmd_agent.1' -> '/usr/local/man/man1/mmmd_agent.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmm_clone.1' -> '/usr/local/man/man1/mmm_clone.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmm_get_dump.1' -> '/usr/local/man/man1/mmm_get_dump.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmm_control.1' -> '/usr/local/man/man1/mmm_control.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmmd_angel.1' -> '/usr/local/man/man1/mmmd_angel.1'…Ok!
Creating symlink: '/usr/local/mmm/man/man1/mmm_backup.1' -> '/usr/local/man/man1/mmm_backup.1'…Ok!

Installation is done!

++++++++++++++++++++++++++++++++++++++

四  配置

 

 cp /home/sysadmin/zhaoyj/software/mysql-master-master-1.2.3/etc/examples/mmm_agent.conf.example  /usr/local/mmm/etc/mmm_agent.conf
——————————–
這個配置選項是必須的嗎?【不是】
# Cluster interface 
cluster_interface eth0 
———————-
db1的配置文件 /usr/local/mmm/etc/mmm_agent.conf
#
# Master-Master Manager config (agent)
#

include mmm_common.conf

# Paths
pid_path /usr/local/mmm/var/mmmd_agent.pid
bin_path /usr/local/mmm/bin

# MMMD command socket tcp-port and ip
bind_port 9989

# Logging setup 
log mydebug 
    file /usr/local/mmm/var/mmm-debug.log 
    level debug 
log mytraps 
    file /usr/local/mmm/var/mmm-traps.log 
    level trap

# Define current server id
this db1
mode master

# For masters 
peer db2

# Cluster hosts addresses and access params
host db1
    ip 211.100.97.246
    port 3306
    user rep_agent
    password 123456

host db2
    ip 211.100.97.250
    port 3306
    user rep_agent
    password 123456

————————
db2的配置文件  /usr/local/mmm/etc/mmm_agent.conf
# Master-Master Manager config (agent)
#

include mmm_common.conf

# Paths
pid_path /usr/local/mmm/var/mmmd_agent.pid
bin_path /usr/local/mmm/bin

# MMMD command socket tcp-port and ip
bind_port 9989

# Logging setup 
log mydebug 
    file /usr/local/mmm/var/mmm-debug.log 
    level debug 
log mytraps 
    file /usr/local/mmm/var/mmm-traps.log 
    level trap


# Define current server id
this db2
mode master

# For masters 
peer db1


# Cluster hosts addresses and access params
host db1
    ip 211.100.97.246
    port 3306
    user rep_agent
    password 123456

host db2
    ip 211.100.97.250
    port 3306
    user rep_agent
    password 123456
 
 
————————–
 db1 db2以及monitor共同的配置文件/usr/local/mmm/etc/mmm_common.conf  
# Cluster interface
#cluster_interface eth0

# Debug mode
debug no

# Paths
bin_path /usr/local/mmm/bin
pid_path /usr/local/mmm/var/mmmd.pid 
status_path /usr/local/mmm/var/mmmd.status

# Choose the default failover method [manual|wait|auto] 
failover_method auto

# How many seconds to wait for both masters to become ONLINE 
# before switching from WAIT to AUTO failover method, 0 = wait indefinitely 
wait_for_other_master 2

# How many seconds to wait before switching node status from AWAITING_RECOVERY to ONLINE
# 0 = disabled 
auto_set_online 1


# Logging setup
log mydebug 
    file /usr/local/mmm/var/mmm-debug.log
    level debug

log mytraps 
    file /usr/local/mmm/var/mmm-traps.log
    level trap
    email yanjing.zhao@pcstars.com
    email 15810370266@139.com

# Email notification settings
email notify
    from_address mmm_control@example.com
    from_name MMM Control

# Define roles
active_master_role reader

# MMMD command socket tcp-port
agent_port 9989
monitor_ip 127.0.0.1


# Cluster hosts addresses and access params

host db1
    ip 211.100.97.246
    port 3306
    user rep_agent
    password 123456 
    mode master
    pear db2

host db2
    ip 211.100.97.250
    port 3306
    user rep_agent
    password 123456
    mode master
    pear db1


# Define roles that are assigned to the above hosts
# Mysql Reader role
role reader
    mode balanced
    servers db1, db2
    ip 211.100.97.243,211.100.97.244

# Mysql Writer role
role writer
    mode exclusive
    servers db1, db2
    ip 211.100.97.248

# Replication credentials used by slaves to connect to the master
replication_user replication
replication_password 123456

# Checks parameters

# Ping checker
check ping
    check_period 1
    trap_period 5
    timeout 2

# Mysql checker 
# (restarts after 10000 checks to prevent memory leaks)
check mysql
    check_period 1
    trap_period  2
    timeout 2
    restart_after 10000

# Mysql replication backlog checker 
# (restarts after 10000 checks to prevent memory leaks)
check rep_backlog
    check_period 5
    trap_period 10
    max_backlog 60
    timeout 2
    restart_after 10000

# Mysql replication threads checker 
# (restarts after 10000 checks to prevent memory leaks)
check rep_threads
    check_period 1
    trap_period 5
    timeout 2
    restart_after 10000
 
—————————-
 
monit機上 mmm_mon.conf 的配置文件

# Cluster interface
#cluster_interface eth0

# Debug mode
debug no

# Paths
bin_path /usr/local/mmm/bin
pid_path /usr/local/mmm/var/mmmd.pid 
status_path /usr/local/mmm/var/mmmd.status

# Choose the default failover method [manual|wait|auto] 
failover_method auto

# How many seconds to wait for both masters to become ONLINE 
# before switching from WAIT to AUTO failover method, 0 = wait indefinitely 
wait_for_other_master 2

# How many seconds to wait before switching node status from AWAITING_RECOVERY to ONLINE
# 0 = disabled 
auto_set_online 1


# Logging setup
log mydebug 
    file /usr/local/mmm/var/mmm-debug.log
    level debug

log mytraps 
    file /usr/local/mmm/var/mmm-traps.log
    level trap
    email yanjing.zhao@pcstars.com
    email 15810370266@139.com

# Email notification settings
email notify
    from_address mmm_control@example.com
    from_name MMM Control

# Define roles
active_master_role writer

# MMMD command socket tcp-port
agent_port 9989
monitor_ip 127.0.0.1


# Cluster hosts addresses and access params

host db1
    ip 211.100.97.246
    port 3306
    user rep_agent
    password 123456 
    mode master
    pear db2

host db2
    ip 211.100.97.250
    port 3306
    user rep_agent
    password 123456
    mode master
    pear db1


# Define roles that are assigned to the above hosts
# Mysql Reader role
role reader
    mode balanced
    servers db1, db2
    ip 211.100.97.243,211.100.97.244

# Mysql Writer role
role writer
    mode exclusive
    servers db1, db2
    ip 211.100.97.248

# Replication credentials used by slaves to connect to the master
replication_user replication
replication_password 123456

# Checks parameters

# Ping checker
check ping
    check_period 1
    trap_period 5
    timeout 2

# Mysql checker 
# (restarts after 10000 checks to prevent memory leaks)
check mysql
    check_period 1
    trap_period  2
    timeout 2
    restart_after 10000

# Mysql replication backlog checker 
# (restarts after 10000 checks to prevent memory leaks)
check rep_backlog
    check_period 5
    trap_period 10
    max_backlog 60
    timeout 2
    restart_after 10000

# Mysql replication threads checker 
# (restarts after 10000 checks to prevent memory leaks)
check rep_threads
    check_period 1
    trap_period 5
    timeout 2
    restart_after 10000
 ————————–


五 啟動進程
 
在db1和db2上配置完mmm_agent.conf和mmm_common.conf之後才能啟動agent進程
啟動 mmmd_agent進程
[root@XKWB5705 etc]# /usr/local/mmm/scripts/init.d/mmm_agent start
Starting MMM Agent daemon: MySQL Multi-Master Replication Manager
Version: 1.2.3
Ok

在monit上配置完mmm_mon.conf之後啟動mon進程
[root@CentOS etc]# /usr/local/mmm/scripts/init.d/mmm_mon start
Daemon bin: '/usr/local/mmm/sbin/mmmd_mon'
Daemon pid: '/usr/local/mmm/var/mmmd.pid'
Starting MMM Monitor daemon: MySQL Multi-Master Replication Manager
Version: 1.2.3
Reading config file: 'mmm_mon.conf'
$VAR1 = {};
Ok

————————
db1和db2查看進程
[root@XKWB5510 etc]# ps aux |grep mmm
root     13702  0.4  0.4 107260  8444 ?        S    15:21   0:03 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent

monit上查看進程
[root@CentOS etc]# ps aux |grep mmm
root     24608  0.6  1.9 258556 39352 ?        Sl   15:17   0:04 perl /usr/local/mmm/sbin/mmmd_mon
root     24611  0.0  0.4 107392  8280 ?        S    15:17   0:00 perl /usr/local/mmm/bin/check/checker rep_backlog
root     24613  0.1  0.4 107392  8252 ?        S    15:17   0:00 perl /usr/local/mmm/bin/check/checker mysql
root     24615  0.2  0.2  91668  5368 ?        S    15:17   0:01 perl /usr/local/mmm/bin/check/checker ping
root     24617  0.2  0.4 107392  8280 ?        S    15:17   0:01 perl /usr/local/mmm/bin/check/checker rep_threads

連續觀察瞭幾次monit上的進程變化情況
從變化情況可以看出monitor用fping檢測兩個節點的存活狀況
[root@CentOS etc]# ps aux |grep mmm
root     24608  0.6  1.9 258556 39356 ?        Sl   15:17   0:07 perl /usr/local/mmm/sbin/mmmd_mon
root     24611  0.0  0.4 107392  8288 ?        S    15:17   0:00 perl /usr/local/mmm/bin/check/checker rep_backlog
root     24613  0.1  0.4 107392  8264 ?        S    15:17   0:01 perl /usr/local/mmm/bin/check/checker mysql
root     24615  0.2  0.2  91668  5368 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker ping
root     24617  0.1  0.4 107392  8280 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker rep_threads

[root@CentOS etc]# ps aux |grep mmm
root     24608  0.6  1.9 258556 39356 ?        Sl   15:17   0:07 perl /usr/local/mmm/sbin/mmmd_mon
root     24611  0.0  0.4 107392  8288 ?        S    15:17   0:00 perl /usr/local/mmm/bin/check/checker rep_backlog
root     24613  0.1  0.4 107392  8264 ?        S    15:17   0:01 perl /usr/local/mmm/bin/check/checker mysql
root     24615  0.2  0.2  91668  5368 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker ping
root     24617  0.1  0.4 107392  8280 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker rep_threads
root     25887  0.0  0.0   1804   504 ?        S    15:35   0:00 /usr/local/mmm/bin/sys/fping -q -u -t 500 -C 1 211.100.97.246

[root@CentOS etc]# ps aux |grep mmm
root     24608  0.6  1.9 258556 39356 ?        Sl   15:17   0:07 perl /usr/local/mmm/sbin/mmmd_mon
root     24611  0.0  0.4 107392  8288 ?        S    15:17   0:00 perl /usr/local/mmm/bin/check/checker rep_backlog
root     24613  0.1  0.4 107392  8264 ?        S    15:17   0:01 perl /usr/local/mmm/bin/check/checker mysql
root     24615  0.2  0.2  91668  5368 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker ping
root     24617  0.1  0.4 107392  8280 ?        S    15:17   0:02 perl /usr/local/mmm/bin/check/checker rep_threads
root     25890  0.0  0.0   1804   504 ?        S    15:35   0:00 /usr/local/mmm/bin/sys/fping -q -u -t 500 -C 1 211.100.97.250

連續觀察db2上進程的變化情況
可以看到db2不斷檢測db1的讀進程以及monit的寫進程
[root@XKWB5705 etc]# ps aux |grep mmm
root      1613  0.3  0.2 107272  8440 ?        S    15:18   0:04 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent
root     12026  0.0  0.1  99564  7056 ?        S    15:38   0:00 perl /usr/local/mmm/bin/agent/check_role writer(211.100.97.248;)
root     12027  0.0  0.1 102200  7572 ?        R    15:38   0:00 perl /usr/local/mmm/bin/mysql_allow_write


[root@XKWB5705 etc]# ps aux |grep mmm
root      1613  0.3  0.2 107272  8440 ?        S    15:18   0:04 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent
root     12121  0.0  0.1  92552  6176 ?        R    15:38   0:00 perl /usr/local/mmm/bin/agent/check_role writer(211.100.97.248;)

root      1613  0.3  0.2 107272  8440 ?        S    15:18   0:04 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent
root     12202  0.0  0.1  90072  5744 ?        R    15:38   0:00 perl /usr/local/mmm/bin/agent/check_role reader(211.100.97.244;)


[root@XKWB5705 etc]# ps aux |grep mmm
root      1613  0.3  0.2 107272  8440 ?        S    15:18   0:04 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent

連續觀察db1上進程的變化情況
可以看到db1不斷檢測db2的讀進程
[root@XKWB5510 etc]# ps aux |grep mmm |grep -v grep
root     13702  0.4  0.4 107260  8444 ?        S    15:21   0:06 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent
root     26820  0.0  0.3  97212  6712 ?        R    15:44   0:00 perl /usr/local/mmm/bin/agent/check_role reader(211.100.97.243;)
[root@XKWB5510 etc]# ps aux |grep mmm |grep -v grep
root     13702  0.4  0.4 107260  8444 ?        S    15:21   0:06 /usr/bin/perl /usr/local/mmm/sbin/mmmd_agent
—————————-

六 添加開機自動啟動

db1, db2 開機自啟動
cp -r /usr/local/mmm/scripts/init.d/mmm_agent /etc/init.d/mmmd
chkconfig –add mmmd
chkconfig –level 345 mmmd on
查看一下添加結果:
[root@XKWB5705 etc]# chkconfig –list mmmd
mmmd            0:off 1:off 2:off 3:on 4:on 5:on 6:off

Mon 開機自啟動
[root@CentOS etc]# cp -r /usr/local/mmm/scripts/init.d/mmm_agent /etc/init.d/mmmd
[root@CentOS etc]# chkconfig –add mmmd
[root@CentOS etc]# chkconfig –level 345 mmmd on
[root@CentOS etc]# chkconfig –list mmmd
mmmd            0:off 1:off 2:off 3:on 4:on 5:on 6:off
———————————-

七 測試

先介紹一下幾種狀態:
online  正常運行
admin_offline  主機被手動設置成offline
hard_offline   主機處於offline狀態,可能是檢測ping或者mysql失敗
awaiting_recovery  主機正在等待恢復
replication_delay   replication backlog太大瞭(檢測rep_backlog線程失敗)
replication_fail    replication線程沒有運行(檢測rep_threads線程失敗)


最初始狀態:

mmm_control set_online db1    讓db1上線

mmm_control set_online db2    讓db2上線

[root@CentOS etc]# mmm_control show  
Servers status:
  db1(211.100.97.246): master/ONLINE. Roles: reader(211.100.97.243;), writer(211.100.97.248;)
  db2(211.100.97.250): master/ONLINE. Roles: reader(211.100.97.244;)

  從以上可以看到db1是主寫服務器
————————————-


當我停止db1【246】的mysql進程時,日志信息
[2011-09-28 16:00:00]: 24608: Check: CHECK_FAIL('db2', 'rep_threads')  Returned message: ERROR: Replication is broken
[2011-09-28 16:00:07]: 24608: Check: CHECK_OK('db1', 'mysql')
[2011-09-28 16:00:08]: 24608: Daemon: State change(db1): HARD_OFFLINE -> AWAITING_RECOVERY
[2011-09-28 16:00:10]: 24608: Daemon: State change(db1): AWAITING_RECOVERY -> ONLINE. Uptime diff = 12.109999999986 seconds; Status change diff = 1317196810

[root@CentOS var]# mmm_control show
Servers status:
  db1(211.100.97.246): master/HARD_OFFLINE. Roles: None
  db2(211.100.97.250): master/ONLINE. Roles: reader(211.100.97.243;), reader(211.100.97.244;), writer(211.100.97.248;)
從以上可以看到主寫服務器已經從db1切換到db2,而且db1是offline狀態


當我重新啟動db1【246】的mysql進程時
[2011-09-28 16:01:54]: 24608: Check: CHECK_OK('db1', 'mysql')
[2011-09-28 16:01:55]: 24608: Daemon: State change(db1): HARD_OFFLINE -> AWAITING_RECOVERY
[2011-09-28 16:01:56]: 24608: Check: CHECK_OK('db2', 'rep_threads')
[2011-09-28 16:01:56]: 24608: Daemon: State change(db1): AWAITING_RECOVERY -> ONLINE. Uptime diff = 2.94999999995343 seconds; Status change diff = 1317196916
——————————-

當我停止db2【250】的mysql進程時
[2011-09-28 16:29:22]: 24608: Check: CHECK_FAIL('db2', 'mysql')  Returned message: ERROR: Connect error (host = 211.100.97.250:3306, user = rep_agent, pass = 'xxxxxx')! Lost connection to MySQL server at 'reading initial communication packet', system error: 111
[2011-09-28 16:29:23]: 24608: Daemon: State change(db2): ONLINE -> HARD_OFFLINE
[2011-09-28 16:29:26]: 24608: Check: CHECK_FAIL('db1', 'rep_threads')  Returned message: ERROR: Replication is broken

[root@CentOS var]# mmm_control show
Servers status:
  db1(211.100.97.246): master/ONLINE. Roles: reader(211.100.97.243;), reader(211.100.97.244;), writer(211.100.97.248;)
  db2(211.100.97.250): master/HARD_OFFLINE. Roles: None
從以上可以看到db2處於offline狀態

當我重新啟動db2【250】的mysql進程時,日志裡面的狀態提示已經發生瞭變化,變成瞭online狀態
[2011-09-28 16:34:26]: 24608: Check: CHECK_OK('db2', 'mysql')
[2011-09-28 16:34:28]: 24608: Daemon: State change(db2): HARD_OFFLINE -> AWAITING_RECOVERY
[2011-09-28 16:34:29]: 24608: Daemon: State change(db2): AWAITING_RECOVERY -> ONLINE. Uptime diff = 306.320000000065 seconds; Status change diff = 1317198869

[root@CentOS var]# mmm_control show
Servers status:
  db1(211.100.97.246): master/ONLINE. Roles: reader(211.100.97.244;), writer(211.100.97.248;)
  db2(211.100.97.250): master/ONLINE. Roles: reader(211.100.97.243;)

通過以上測試證明整個搭建成功,已經實現瞭高可用,實現失敗節點的自動切換


八  附—MMM簡介
 
MMM即Master-Master Replication Manager for MySQL(mysql主主復制管理器)關於mysql主主復制配置的監控、故障轉移和管理的一套可伸縮的腳本套件(在任何時候隻有一個節點可以被寫入),這個套件也能對居於標準的主從配置的任意數量的從服務器進行讀負載均衡,所以你可以用它來在一組居於復制的服務器啟動虛擬ip,除此之外,它還有實現數據備份、節點之間重新同步功能的腳本。
 
MySQL本身沒有提供replication failover的解決方案,通過MMM方案能實現服務器的故障轉移,從而實現mysql的高可用。
 
官方網站為:http://mysql-mmm.org
 
Mmm主要功能由下面三個腳本提供
 
l         mmm_mond  負責所有的監控工作的監控守護進程,決定節點的移除等等
 
l         mmm_agentd  運行在mysql服務器上的代理守護進程,通過簡單遠程服務集提供給監控節點
 
l         mmm_control  通過命令行管理mmm_mond進程

作者“ANLJF的專欄”
 

發佈留言