DRBD 部署

主机规划

主机名 主机IP 系统 存储
GitLabCESvr01 10.37.21.21 centos 6.5 200G
GitLabCESvr02 10.37.21.22 centos 6.5 200G

将centos 6.5的内核进行了升级,并重启系统

1
2
3
yum update kernel
yum install kernel-devel
reboot

安装drbd

1
2
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum -y install drbd83-utils kmod-drbd83

加载drbd模块

1
modprobe drbd

对共享的磁盘分区

1
[root@GitLabCESvr01 ~] fdisk /dev/xvde

加载磁盘信息:

1
partprobe /dev/xvde

配置drbd.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
vi /etc/drbd.conf
#include "drbd.d/global_common.conf";
#include "drbd.d/*.res";
global {
usage-count no;
}
common {
syncer { rate 200M; }
}
resource r0 {
protocol C;
startup {
wfc-timeout 120;
degr-wfc-timeout 120;
}
disk {
on-io-error detach;
}
net{
timeout 60;
connect-int 10;
ping-int 10;
max-buffers 2048;
max-epoch-size 2048;
cram-hmac-alg "sha1";
shared-secret "1234-abcD";
}
on GitLabCESvr01 {
device /dev/drbd0;
disk /dev/xvde5;
address 10.37.21.21:6666;
meta-disk internal;
}
on GitLabCESvr02 {
device /dev/drbd0;
disk /dev/xvde5;
address 10.37.21.22:6666;
meta-disk internal;
}
}

在两台机器上创建drbd元数据信息:

1
drbdadm create-md all

启动drbd 服务

1
service drbd start

看状态:

1
2
3
4
5
[root@GitLabCESvr01 ~]#  drbd-overview
0:r0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----

[root@GitLabCESvr02 ~]# drbd-overview
0:r0 Connected Secondary/Secondary Inconsistent/Inconsistent C r-----

在主节点GitLabCESvr01上执行下面的命令让其成为主节点

1
drbdadm -- --overwrite-data-of-peer primary all

看同步状态:

1
2
3
[root@GitLabCESvr01 ~]# drbd-overview
0:r0 SyncSource Primary/Secondary UpToDate/Inconsistent C r-----
[>....................] sync'ed: 0.4% (204132/204788)M

加速同步状态

1
drbdsetup /dev/drbd/by-res/r0 syncer -r 500M

将/dev/drbd0格式化并挂载

1
2
mkfs.ext4 /dev/drbd0
mount /dev/drbd0 /var/opt/gitlab

验证数据同步

1
2
3
1.在主节点:
cd /var/opt/gitlab
echo "test123" > testfile
1
2
3
4
2.在备份节点GitLabCESvr02上执行下面的命令
mkdir /data
mount /dev/drbd0 /var/opt/gitlab
#mount 会出错,因为mount只能在Primary一端使用
1
2
3
3.在主节点GitLabCESvr01上执行下面的命令变成备份节点
umount /var/opt/gitlab
drbdadm secondary all
1
2
3
4
5
4.在备份节点GitLabCESvr02上执行下面的命令变为主节点,可以看到GitLabCESvr01同步过来的内容
drbdadm primary all
mount /dev/drbd0 /var/opt/gitlab
less /data/testfile
可以查看到文件的内容为"test123"

RBD故障处理

1
2
场景:
GitLabCESvr01主,GitLabCESvr02备,主设备故障重启,然后查看drbd状态异常
1
2
3
4
5
6
7
8
9
10
11
12
GitLabCESvr01故障后
[root@GitLabCESvr01 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
ns:4 nr:102664 dw:102668 dr:157 al:1 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
GitLabCESvr02的状态:
[root@GitLabCESvr02 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----
ns:0 nr:2141684 dw:2141684 dr:0 al:0 bm:130 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
1
2
3
4
5
6
7
8
9
10
11
12
解决方法:
1.将现在的GitLabCESvr01角色修改为secondary
[root@GitLabCESvr01 ~]# drbdadm secondary r0
[root@GitLabCESvr01 ~]# drbdadm -- --discard-my-data connect r0 #该命令告诉drbdsecondary上的数据不正确,以primary上的数据为准。
2.在GitLabCESvr01上执行下面操作
[root@GitLabCESvr02 /]# drbdadm connect r0
这样GitLabCESvr01就能和GitLabCESvr02开始连接上了,并且保证数据不会丢失:
[root@GitLabCESvr01 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
srcversion: 299AFE04D7AFD98B3CA0AF9
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:20592 dw:20592 dr:0 al:0 bm:4 lo:0 pe:0 ua:0