GlusterFS部署和测试

下载软件:wget -r -np https://download.gluster.org/pub/gluster/glusterfs/5/5.1/Fedora/fedora-28/x86_64/
GlusterFS部署
设置yum库
[glusterfs]
name=glusterfs
url=file:///tmp/glusterfs
gpgcheck=0
enabled=1
centos7
yum install centos-release-gluster310.noarch
yum install glusterfs-server
systemctl restart glusterd
设置主机名解析
node6.robin.com 1.1.1.6
node7.robin.com 1.1.1.7
node8.robin.com 1.1.1.8
node9.robin.com 1.1.1.9

四台服务器安装glusterfs分布式文件系统软件
yum install glusterfs-*
/etc/init.d/glusterd restart

GlusterFS节点管理
添加节点

[root@node1 ～]# gluster peer probe 1.1.1.7
[root@node1 ～]# gluster peer probe 1.1.1.8
[root@node1 ～]# gluster peer probe 1.1.1.9

查询节点
gluster peer status 查看状态
Number of Peers: 3

Hostname: 1.1.1.7
Port: 24007
Uuid: d7c594f5-70ec-4967-b5fd-1eae8f354d8f
State: Peer in Cluster (Connected)

Hostname: 1.1.1.8
Port: 24007
Uuid: 1999c883-32b0-4bbc-b118-6d7d5ec958c8
State: Peer in Cluster (Connected)

Hostname: 1.1.1.9
Port: 24007
Uuid: d2230cac-9d55-4336-8e09-5c283fab3f56
State: Peer in Cluster (Connected)


删除节点
gluster peer detach 1.1.1.9
移除节点时,需要确保该节点上没有brick ,需要提前将brick移除

查看卷信息:
[root@node1 ～]# gluster volume info
Volume Name: datav1
Type: Distributed-Replicate
Volume ID: 9f07dd7f-9283-4324-87ba-1e54262ab6fc
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 1.1.1.6:/data1
Brick2: 1.1.1.7:/data1
Brick3: 1.1.1.8:/data1
Brick4: 1.1.1.9:/data1

查看卷状态:
[root@node1 ～]# gluster volume status

启动卷：
        gluster volume start volumename
停止卷：
        gluster volume stop volumename
删除卷
gluster volume delete volumename

客户端测试
启动起来后我们可以mount测试一下：
    mount -t glusterfs 1.1.1.6:/datav1 /media
上面这种方法是基于fuse实现的，glusterfs当然不会只能这么玩，还可以支持NFS，NFS又分两种原生NFS和系统NFS。


使用原生NFS挂载：mount -t nfs -o vers=3 1.1.1.6:/datav1 /media
        使用此种方法要确保服务端rpcbind要打开并且要注意系统NFS不能启动

使用系统NFS挂载：
      首先把卷使用glusterfs方法挂载到某个目录
      mount -t glusterfs 1.1.1.6:/datav1 /media
      然后修改/etc/exports文件
      例子：/media  *(rw,fsid=1235)
      然后停掉原生NFS：gluster volume set jerryafr nfs.disable on
      最后再启动系统NFS服务即可


 

Brick管理

添加Brick

:
若是副本卷，则一次添加的Bricks数是replica的整数倍；stripe 具有同样的要求。
gluster peer probe 1.1.1.10
gluster volume add-brick datav1 1.1.1.10:/data1 1.1.1.11:/data1

添加数据节点完成后，平衡数据
gluster volume rebalance datav1 start force

 

移除Brick

:
若是副本卷，则移除的Bricks数是replica的整数倍；stripe具有同样的要求。
gluster volume remove-brick datav1 1.1.1.8:/data1 1.1.1.9:/data1 start
volume remove-brick start: success
ID: 8b11f231-e3dc-45cf-a09d-1e4088fec73f
GlusterFS_3.4.1版本在执行移除Brick的时候会将数据迁移到其他可用的Brick上，当数据迁移结束之后才将Brick移除。执行start命令，开始迁移数据，正常移除Brick。

gluster volume remove-brick datav1 status
//在执行开始移除start之后，可以使用status命令进行start状态查看。

gluster volume remove-brick datav1 commit
//使用commit命令执行Brick移除，则不会进行数据迁移而直接删除Brick，符合不需要数据迁移的用户需求。


PS：系统的扩容及缩容可以通过如上节点管理、Brick 管理组合达到目的。
(1)扩容时:可以先增加系统节点,然后添加新增节点上的Brick即可。
(2)缩容时:先移除Brick,然后再进行节点删除则达到缩容的目的,且可以保证数据不丢失

 

替换Brick

:
gluster volume replace-brick datav1 1.1.1.6:/data1 1.1.1.100:/data1 start
//如上，执行replcace-brick卷替换启动命令，使用start启动命令后，开始将原始Brick的数据迁移到即将需要替换的Brick上。

gluster volume replace-brick datav1 1.1.1.6:/data1 1.1.1.100:/data1 status
//在数据迁移的过程中，可以查看替换任务是否完成

gluster volume replace-brick datav1 1.1.1.6:/data1 1.1.1.100:/data1 abort
//在数据迁移的过程中，可以执行abort命令终止Brick替换。

gluster volume replace-brick datav1 1.1.1.6:/data1 1.1.1.100:/data1 commit
//在数据迁移结束之后，执行commit命令结束任务，则进行Brick替换。使用volume info命令可以查看到Brick已经被替换。

 

系统扩展维护

系统配额

开启/关闭系统配额

gluster volume quota VOLNAME enable/disable
//在使用系统配额功能时，需要使用enable将其开启；disable为关闭配额功能命令。
 

设置(重置)目录配额

gluster volume quota VOLNAME limit-usage /directory limit-value
gluster volume quota datav1 limit-usage /quota 10GB
//如上，设置datav1卷下的quota子目录的限额为10GB。
PS：这个目录是以系统挂载目录为根目录”/”,所以/quota即客户端挂载目录下的子目录quota

 

配额查看

# gluster volume quota VOLNAME list
查看目的卷的所有配额设置

# gluster volume quota VOLNAME list /directory name
查看执行目录
/可以显示配额大小及当前使用容量，若无使用容量(最小0KB)则说明设置的目录可能是错误的(不存在)。

分布式逻辑卷

[root@master ~]# gluster volume create datav1 transport tcp 192.168.122.101:/data1 192.168.122.102:/data1 192.168.122.103:/data1 force
volume create: datav1: success: please start the volume to access data
[root@master ~]#
[root@master ~]# gluster volume info

Volume Name: datav1
Type: Distribute
Volume ID: f725823e-0cf3-4a5c-aea2-34eb4537fabf
Status: Created
Snapshot Count: 0
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 192.168.122.101:/data1
Brick2: 192.168.122.102:/data1
Brick3: 192.168.122.103:/data1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@master ~]#
开启
[root@master ~]# gluster volume start datav1
volume start: datav1: success
[root@master ~]#
挂载测试
[root@master ~]# mount -t glusterfs 192.168.122.101:/datav1 /media
[root@master ~]#

向逻辑卷里面添加新的节点
查看帮助
[root@master media]# gluster volume help|grep add
volume add-brick <VOLNAME> [<stripe|replica> <COUNT> [arbiter <COUNT>]] <NEW-BRICK> ... [force] - add brick to volume <VOLNAME>
volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick <brick>] [list-cnt <value>] |
[root@master media]#

[root@master media]# gluster volume add-brick datav1 192.168.122.100:/data1 force
volume add-brick: success
[root@master media]#
查看逻辑卷信息
[root@master media]# gluster volume info

Volume Name: datav1
Type: Distribute
Volume ID: f725823e-0cf3-4a5c-aea2-34eb4537fabf
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 192.168.122.101:/data1
Brick2: 192.168.122.102:/data1
Brick3: 192.168.122.103:/data1
Brick4: 192.168.122.100:/data1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@master media]#

[root@master media]# gluster volume help|grep balance
volume rebalance <VOLNAME> {{fix-layout start} | {start [force]|stop|status}} - rebalance operations
[root@master media]#
[root@master media]# gluster volume rebalance datav1 start
volume rebalance: datav1: success: Rebalance on datav1 has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 89dc2246-d115-4968-930a-8e132ca72ec8
[root@master media]#
停止、删除逻辑卷
[root@master ~]# gluster volume stop datav1
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: datav1: success
[root@master ~]#

[root@master ~]# gluster volume delete datav1
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: datav1: success
[root@master ~]#

[root@master ~]# gluster volume create help
Usage: volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
[root@master ~]#


[root@master ~]# gluster volume create datav2 replica 2 transport tcp 192.168.122.100:/data2 192.168.122.101:/data2 192.168.122.102:/data2 192.168.122.103:/data2 force
volume create: datav2: success: please start the volume to access data
[root@master ~]#


[root@master ~]# gluster volume info

Volume Name: datav2
Type: Distributed-Replicate
Volume ID: 38622b67-d973-482c-9540-74092226eb3a
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.122.100:/data2
Brick2: 192.168.122.101:/data2
Brick3: 192.168.122.102:/data2
Brick4: 192.168.122.103:/data2
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
挂载使用测试
前两个节点
[root@master media]# vim a.txt
[root@master media]# touch {1..100}.txt
[root@master media]# ls
100.txt 19.txt 28.txt 37.txt 46.txt 55.txt 64.txt 73.txt 82.txt 91.txt a.txt
10.txt 1.txt 29.txt 38.txt 47.txt 56.txt 65.txt 74.txt 83.txt 92.txt
11.txt 20.txt 2.txt 39.txt 48.txt 57.txt 66.txt 75.txt 84.txt 93.txt
12.txt 21.txt 30.txt 3.txt 49.txt 58.txt 67.txt 76.txt 85.txt 94.txt
13.txt 22.txt 31.txt 40.txt 4.txt 59.txt 68.txt 77.txt 86.txt 95.txt
14.txt 23.txt 32.txt 41.txt 50.txt 5.txt 69.txt 78.txt 87.txt 96.txt
15.txt 24.txt 33.txt 42.txt 51.txt 60.txt 6.txt 79.txt 88.txt 97.txt
16.txt 25.txt 34.txt 43.txt 52.txt 61.txt 70.txt 7.txt 89.txt 98.txt
17.txt 26.txt 35.txt 44.txt 53.txt 62.txt 71.txt 80.txt 8.txt 99.txt
18.txt 27.txt 36.txt 45.txt 54.txt 63.txt 72.txt 81.txt 90.txt 9.txt
[root@master media]# ls /data2
13.txt 27.txt 33.txt 40.txt 48.txt 52.txt 62.txt 76.txt 82.txt 8.txt 96.txt
17.txt 28.txt 34.txt 41.txt 49.txt 53.txt 63.txt 78.txt 83.txt 90.txt 97.txt
19.txt 29.txt 35.txt 42.txt 4.txt 56.txt 66.txt 79.txt 85.txt 91.txt 99.txt
23.txt 31.txt 37.txt 45.txt 50.txt 58.txt 68.txt 80.txt 86.txt 93.txt 9.txt
26.txt 32.txt 39.txt 46.txt 51.txt 59.txt 73.txt 81.txt 89.txt 94.txt a.txt
[root@master media]# ls /data2|wc -l
55
[root@master media]#
[root@ ~]# ls /data2|wc -l
55
[root@ ~]#

后两个节点的文件数量相同
[root@data1 ~]# ls /data2|wc -l
46
[root@data1 ~]#
[root@data2 ~]# ls /data2|wc -l
46
[root@data2 ~]#
分成了两组，两组中的数据是数据的总和

[root@master ~]# umount /media
[root@master ~]# gluster volume stop datav2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: datav2: success
[root@master ~]#

[root@master ~]# gluster volume delete datav2
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: datav2: success
[root@master ~]#
创建分布式和条带逻辑卷

[root@master ~]# gluster volume create datav3 stripe 2 transport tcp 192.168.122.100:/data3 192.168.122.101:/data3 192.168.122.102:/data3 192.168.122.103:/data3 force
volume create: datav3: success: please start the volume to access data
[root@master ~]#

[root@master ~]# gluster volume info

Volume Name: datav3
Type: Distributed-Stripe
Volume ID: 95441e83-4998-420c-ab0c-91fb4d0c6e7f
Status: Created
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.122.100:/data3
Brick2: 192.168.122.101:/data3
Brick3: 192.168.122.102:/data3
Brick4: 192.168.122.103:/data3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@master ~]#
开启了逻辑娟组
[root@master ~]# gluster volume start datav3
volume start: datav3: success
[root@master ~]#

将逻辑卷组挂载测试
[root@master ~]# mount -t glusterfs 192.168.122.101:/datav3 /media
[root@master ~]#
[root@master media]# cp /data2/a.txt ./
[root@master media]# ls
a.txt
[root@master media]# vim a.txt
[root@master media]# cp a.txt b.txt
前两个节点的的文件内容的字节之和等于文件的总大小
[root@master media]# ll a.txt
-rw-r--r-- 1 root root 159711 11月 7 15:20 a.txt
[root@master media]#

[root@master media]# ll /data3
总用量 132
-rw-r--r-- 2 root root 131072 11月 7 15:20 a.txt
[root@master media]#
[root@ ~]# ll /data3
总用量 32
-rw-r--r-- 2 root root 28639 11月 7 15:09 a.txt
[root@ ~]#

后两个节点的的文件内容的字节之和等于文件的总大小
[root@master media]# ll b.txt
-rw-r--r-- 1 root root 159711 11月 7 2018 b.txt
[root@master media]#

[root@data1 ~]# ll /data3
总用量 132
-rw-r--r-- 2 root root 131072 11月 7 17:56 b.txt
[root@data1 ~]#

[root@data2 ~]# ll /data3
总用量 32
-rw-r--r-- 2 root root 28639 11月 7 15:30 b.txt
[root@data2 ~]#
[root@master ~]# gluster volume stop datav3
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: datav3: success
[root@master ~]# gluster volume delete datav3
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: datav3: success
[root@master ~]# stripe 2

gluster volume create datav4 replica 2 stripe 2 transport tcp 192.168.122.100:/data4 192.168.122.101:/data4 192.168.122.102:/data4 192.168.122.103:/data4 force


[root@master ~]# gluster volume create datav4 replica 2 stripe 2 transport tcp 192.168.122.100:/data4 192.168.122.101:/data4 192.168.122.102:/data4 192.168.122.103:/data4 force
volume create: datav4: success: please start the volume to access data
查看信息
[root@master ~]# gluster volume info

Volume Name: datav4
Type: Striped-Replicate
Volume ID: 406c027e-528c-4942-acbf-f1925d32708b
Status: Created
Snapshot Count: 0
Number of Bricks: 1 x 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 192.168.122.100:/data4
Brick2: 192.168.122.101:/data4
Brick3: 192.168.122.102:/data4
Brick4: 192.168.122.103:/data4
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@master ~]#
开启路基卷进行测试
[root@master ~]# gluster volume start datav4
volume start: datav4: success
[root@master ~]# mount -t glusterfs 192.168.122.101:/datav4 /media
[root@master ~]#
前两个的文件大小相同
[root@master media]# vim a.txt
[root@master media]# ll
总用量 166
-rw-r--r-- 1 root root 169485 11月 7 2018 a.txt
[root@master media]# ll /data4
总用量 132
-rw-r--r-- 2 root root 131072 11月 7 15:22 a.txt
[root@master media]#
[root@ ~]# ll /data4
总用量 132
-rw-r--r-- 2 root root 131072 11月 7 15:12 a.txt
[root@ ~]#

后两个文件的大小相同
[root@data1 ~]# ll /data4
总用量 44
-rw-r--r-- 2 root root 38413 11月 7 18:38 a.txt
[root@data1 ~]#
[root@data2 ~]# ll /data4
总用量 44
-rw-r--r-- 2 root root 38413 11月 7 15:32 a.txt
[root@data2 ~]#

两组的数据大小等于总共文件的大小