0 块设备介绍
Ceph 的块设备（Ceph Block Device, RBD）是其存储服务的一种实现形式，通过 librbd 库或 Linux 内核模块提供块设备支持，所以可以与主流云平台（如 OpenStack）、虚拟化平台（如 KVM）和容器平台（如 K8S）无缝集成
分布式存储架构特性：
- 数据被切分为多个对象，并分布在多个 OSD（对象存储守护进程）上
- 使用 CRUSH 算法动态管理数据分布，确保高性能和均衡性
高可靠性与弹性：
-
提供多副本或 Erasure Coding（纠删码）以保障数据可靠性
-
故障自愈：当某个 OSD 出现故障时，系统会自动修复并重新分布数据
动态扩展性：
- 快照支持创建时间点一致的只读快照，可用于备份或恢复；快照还可以克隆
1 块设备的基本管理
1.创建存储池zhiyong-rbd
[root@ceph141~]# ceph osd pool create zhiyong-rbd 16 16
pool 'zhiyong-rbd' created[root@ceph141~]# ceph osd pool ls
.mgr
wzy
zhiyong-rbd
2.使用rbd工具初始化池以供RBD块设备使用
[root@ceph141~]# rbd pool init zhiyong-rbd
3.创建一个2G的rbd设备，这个块设备名是wordpress
[root@ceph141~]# rbd create -s 2048 zhiyong-rbd/wordpress
4.查看镜像的详细信息
-l表示长格式信息输出
-p指定镜像名称
[root@ceph141~]# rbd ls -p zhiyong-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wordpress 2 GiB 2[root@ceph141~]# rbd info zhiyong-rbd/wordpress
rbd image 'wordpress':size 2 GiB in 512 objectsorder 22 (4 MiB objects)snapshot_count: 0id: fd4d9658c0d2block_name_prefix: rbd_data.fd4d9658c0d2format: 2features: layering, exclusive-lock, object-map, fast-diff, deep-flattenop_features: flags: create_timestamp: Tue Nov 5 01:15:13 2024access_timestamp: Tue Nov 5 01:15:13 2024modify_timestamp: Tue Nov 5 01:15:13 2024
这个镜像为2G，512个对象，每个对象4MB，
5.修改rbd的名称
[root@ceph141~]# rbd -p zhiyong-rbd rename wordpress wp
6.修改rbd的大小
[root@ceph141~]# rbd -p zhiyong-rbd resize -s 5G wp
Resizing image: 100% complete...done.
[root@ceph141~]# rbd ls -p zhiyong-rbd -l
NAME SIZE PARENT FMT PROT LOCK
wp 5 GiB 2
7.删除镜像
[root@ceph141~]# rbd rm zhiyong-rbd/wp
Removing image: 100% complete...done.
2 Ubuntu使用ceph块设备案例
01 ceph141节点挂载并使用
1.先准备存储池和初始化，（如果没有的情况下）
[root@ceph141~]# ceph osd pool create zhiyong18-rbd 128 128
pool 'zhiyong18-rbd' created
[root@ceph141~]# rbd pool init zhiyong18-rbd
创建一个块设备叫wordpress和mysqld
[root@ceph141~]# rbd create -s 2G zhiyong18-rbd/wordpress
[root@ceph141~]# rbd create -s 5G -p zhiyong18-rbd --image mysqld --image-feature layering,exclusive-lock
2.查看镜像的详细信息
[root@ceph141~]# rbd info -p zhiyong18-rbd wordpress
rbd image 'wordpress':size 2 GiB in 512 objectsorder 22 (4 MiB objects)snapshot_count: 0id: 149e2d0fa83eeblock_name_prefix: rbd_data.149e2d0fa83eeformat: 2features: layering, exclusive-lock, object-map, fast-diff, deep-flattenop_features: flags: create_timestamp: Tue Nov 5 18:15:24 2024access_timestamp: Tue Nov 5 18:15:24 2024modify_timestamp: Tue Nov 5 18:15:24 2024[root@ceph141~]# rbd info -p zhiyong18-rbd mysqld
rbd image 'mysqld':size 5 GiB in 1280 objectsorder 22 (4 MiB objects)snapshot_count: 0id: 149e89b3c43d6block_name_prefix: rbd_data.149e89b3c43d6format: 2features: layering, exclusive-lockop_features: flags: create_timestamp: Tue Nov 5 18:15:30 2024access_timestamp: Tue Nov 5 18:15:30 2024modify_timestamp: Tue Nov 5 18:15:30 2024
3.客户端映射镜像(块设备)
rbd -p zhiyong18-rbd map mysqld
rbd -p zhiyong18-rbd map wordpress
查看镜像块
[root@ceph141~]# tree /dev/rbd/
/dev/rbd/
└── zhiyong18-rbd├── mysqld -> ../../rbd0└── wordpress -> ../../rbd1
4.对块设备进行格式化
[root@ceph141~]# mkfs.xfs /dev/rbd0
[root@ceph141~]# mkfs.ext4 /dev/rbd1
5.对设备进行挂载
[root@ceph141~]# mkdir -pv /wenzhiyong/data/{wordpress,mysql}[root@ceph141~]# mount /dev/rbd0 /wenzhiyong/data/mysql
[root@ceph141~]# mount /dev/rbd1 /wenzhiyong/data/wordpress[root@ceph141~]# df -h | grep wen
/dev/rbd0 5.0G 69M 5.0G 2% /wenzhiyong/data/mysql
/dev/rbd1 2.0G 24K 1.8G 1% /wenzhiyong/data/wordpress
6.测试可用性，可以写入文件
[root@ceph141~]# echo mysql_data >> /wenzhiyong/data/mysql/mysql.txt
[root@ceph141~]# echo wordpress_data >> /wenzhiyong/data/mysql/wordpress.txt
7.卸载设备
[root@ceph141~]# umount /wenzhiyong/data/mysql
[root@ceph141~]# umount /wenzhiyong/data/wordpress
8.取消映射
[root@ceph141~]# rbd unmap zhiyong18-rbd/mysqld
[root@ceph141~]# rbd unmap zhiyong18-rbd/wordpress[root@ceph141~]# ll /dev/rbd*
ls: cannot access '/dev/rbd*': No such file or directory
02 ceph143节点使用
1.如果没有拷贝授权文件到ceph143节点，这一步是做不通的
[root@ceph143~]# rbd -p zhiyong18-rbd map mysqld
/dev/rbd0
[root@ceph143~]# rbd -p zhiyong18-rbd map wordpress
/dev/rbd1
2.挂载
[root@ceph143~]# mkdir -pv /wenzhiyong/data/{wordpress,mysql}[root@ceph143~]# mount /dev/rbd0 /wenzhiyong/data/mysql
[root@ceph143~]# mount /dev/rbd1 /wenzhiyong/data/wordpress
[root@ceph143~]# df -h | grep wenzhiyong
/dev/rbd0 5.0G 69M 5.0G 2% /wenzhiyong/data/mysql
/dev/rbd1 2.0G 24K 1.8G 1% /wenzhiyong/data/wordpress
3.查看数据
[root@ceph143~]# tree /wenzhiyong/data/
/wenzhiyong/data/
├── mysql
│ ├── mysql.txt
│ └── wordpress.txt
└── wordpress└── lost+found
03 多个节点一起写入块设备的结果
1.这时让ceph142节点也使用这2个rbd设备会发生什么情况呢
[root@ceph142~]# mkdir -pv /wenzhiyong/data/{wordpress,mysql}
[root@ceph142~]# rbd -p zhiyong18-rbd map mysqld
/dev/rbd0
[root@ceph142~]# rbd -p zhiyong18-rbd map wordpress
/dev/rbd1
[root@ceph142~]# mount /dev/rbd0 /wenzhiyong/data/mysql
[root@ceph142~]# mount /dev/rbd1 /wenzhiyong/data/wordpress
[root@ceph142~]# df -h | grep wenzhiyong
/dev/rbd0 5.0G 69M 5.0G 2% /wenzhiyong/data/mysql
/dev/rbd1 2.0G 24K 1.8G 1% /wenzhiyong/data/wordpress
[root@ceph142~]# tree /wenzhiyong/data/
/wenzhiyong/data/
├── mysql
│ ├── mysql.txt
│ └── wordpress.txt
└── wordpress└── lost+found
2.ceph142节点执行挂载后，ceph143节点的数据也是有的，也没有丢失。而且目前还算一致。
[root@ceph143~]# tree /wenzhiyong/data/
/wenzhiyong/data/
├── mysql
│ ├── mysql.txt
│ └── wordpress.txt
└── wordpress└── lost+found
3.在ceph142节点执行写入会发生什么呢，写入一个文件ceph142_write.txt试试
[root@ceph142~]# echo ceph_write_data-mysql > /wenzhiyong/data/mysql/ceph142_write.txt
[root@ceph142~]# tree /wenzhiyong/data/
/wenzhiyong/data/
├── mysql
│ ├── ceph142_write.txt
│ ├── mysql.txt
│ └── wordpress.txt
└── wordpress└── lost+found
4.立马去ceph143节点查看这个文件ceph142_write.txt，是看不到的
[root@ceph143~]# tree /wenzhiyong/data/
/wenzhiyong/data/
├── mysql
│ ├── mysql.txt
│ └── wordpress.txt
└── wordpress└── lost+found
3 块设备的快照管理
01 创建快照
1.ceph141创建测试的镜像
[root@ceph141 ~]# rbd create -s 2G zhiyong18-rbd/rbd-snap
[root@ceph141 ~]# rbd -p zhiyong18-rbd info rbd-snap | grep "\sfeatures"features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
2.ceph142客户端添加映射并挂载设备写入测试数据
[root@ceph142 ~]# rbd map zhiyong18-rbd/rbd-snap
/dev/rbd2
[root@ceph142~]# mkfs.xfs /dev/rbd2
meta-data=/dev/rbd2 isize=512 agcount=8, agsize=65536 blks= sectsz=512 attr=2, projid32bit=1= crc=1 finobt=1, sparse=1, rmapbt=0= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=524288, imaxpct=25= sunit=16 swidth=16 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
[root@ceph142~]# ll /mnt/
total 0
3.客户端往里面写入文件进行测试
[root@ceph142~]# ll /dev/rbd*
brw-rw---- 1 root disk 252, 0 Nov 5 19:06 /dev/rbd0
brw-rw---- 1 root disk 252, 16 Nov 5 19:06 /dev/rbd1
brw-rw---- 1 root disk 252, 32 Nov 5 20:21 /dev/rbd2[root@ceph142~]# df -h | grep mnt
/dev/rbd2 2.0G 47M 2.0G 3% /mnt[root@ceph142~]# cp /etc/os-release /etc/fstab /etc/hosts /mnt/[root@ceph142~]# ll /mnt/
-rw-r--r-- 1 root root 604 Nov 5 20:23 fstab
-rw-r--r-- 1 root root 279 Nov 5 20:23 hosts
-rw-r--r-- 1 root root 386 Nov 5 20:23 os-release
4.在ceph服务端创建快照，查看快照信息。
[root@ceph141~]# rbd snap create zhiyong18-rbd/rbd-snap --snap zhiyong-`date +%F_%T`-v1
Creating snap: 100% complete...done.
[root@ceph141~]# rbd info zhiyong18-rbd/rbd-snap | grep snapshot_countsnapshot_count: 1
[root@ceph141~]# rbd snap list zhiyong18-rbd/rbd-snap
SNAPID NAME SIZE PROTECTED TIMESTAMP 4 zhiyong-2024-11-05_20:24:51-v1 2 GiB Tue Nov 5 20:24:52 2024
5.ceph142接着删除2个文件做测试，看最后能否找回
[root@ceph142~]# rm -f /mnt/{fstab,os-release}
[root@ceph142~]# ll /mnt/
-rw-r--r-- 1 root root 279 Nov 5 20:23 hosts
02 客户端取消块设备的锁
1.客户端删除映射前在服务端查看块设备镜像是否被锁住，注意LOCK字段是否有"excl"属性；如果客户端不取消块设备映射就无法释放锁
[root@ceph141~]# rbd -p zhiyong18-rbd ls -l
NAME SIZE PARENT FMT PROT LOCK
mysqld 5 GiB 2 excl
rbd-snap 2 GiB 2 excl
rbd-snap@zhiyong-2024-11-05_20:24:51-v1 2 GiB 2
wordpress 2 GiB 2 excl
在客户端ceph142节点取消挂载，最后取消映射
[root@ceph142~]# df -h | grep mnt
/dev/rbd2 2.0G 47M 2.0G 3% /mnt
[root@ceph142~]# umount /mnt
[root@ceph142~]# df -h | grep mnt
[root@ceph142~]# ls /dev/rbd2
/dev/rbd2
[root@ceph142~]# rbd unmap zhiyong18-rbd/rbd-snap
[root@ceph142~]# ls /dev/rbd2
ls: cannot access '/dev/rbd2': No such file or directory
03 执行快照回滚
1.客户端删除映射后在服务端查看镜像是否被锁住，已经没有excl属性了，成功释放锁
[root@ceph141~]# rbd -p zhiyong18-rbd ls -l | grep -i rbd-snap
rbd-snap 2 GiB 2
rbd-snap@zhiyong-2024-11-05_20:24:51-v1 2 GiB 2
2.基于之前的快照信息执行回滚操作以恢复数据；回滚前先查看确认快照名称
[root@ceph141~]# rbd snap list zhiyong18-rbd/rbd-snap
SNAPID NAME SIZE PROTECTED TIMESTAMP 4 zhiyong-2024-11-05_20:24:51-v1 2 GiB Tue Nov 5 20:24:52 2024
[root@ceph141~]# rbd snap rollback zhiyong18-rbd/rbd-snap --snap zhiyong-2024-11-05_20:24:51-v1
Rolling back to snapshot: 100% complete...done.
04 验证找回的数据
随机找一个客户端重新挂载块设备文件，验证数据是否找回
1.由于在之前的Ubuntu使用ceph块设备案例中，ceph143使用了块设备并写入了文件；为排除干扰，然ceph143取消使用。（如果没有挂载的话，就不用取消了）
[root@ceph143~]# rm -rf /wenzhiyong/data/mysql/* /wenzhiyong/data/wordpress/*[root@ceph143~]# umount /wenzhiyong/data/mysql /wenzhiyong/data/wordpress [root@ceph143~]# df -h | grep wen[root@ceph143~]# rbd unmap zhiyong18-rbd/mysqld
[root@ceph143~]# rbd unmap zhiyong18-rbd/wordpress
2.挂载前确认设备编号，挂载后数据找回成功
[root@ceph143~]# ls /dev/rbd*
/dev/rbd0/dev/rbd:
zhiyong18-rbd
[root@ceph143~]# mount /dev/rbd0 /mnt/
[root@ceph143~]# ll /mnt/
-rw-r--r-- 1 root root 604 Nov 5 20:23 fstab
-rw-r--r-- 1 root root 279 Nov 5 20:23 hosts
-rw-r--r-- 1 root root 386 Nov 5 20:23 os-release
05 快照的删除
前面提到了快照的创建，还没说快照的删除。现在就演示一下。还是那句话，这些操作在web页面可以轻松执行，也不用记难记的命令
[root@ceph141~]# rbd snap list zhiyong18-rbd/rbd-snap
SNAPID NAME SIZE PROTECTED TIMESTAMP 4 zhiyong-2024-11-05_20:24:51-v1 2 GiB Tue Nov 5 20:24:52 2024
[root@ceph141~]# rbd snap rm zhiyong18-rbd/rbd-snap --snap zhiyong-2024-11-05_20:24:51-v1
Removing snap: 100% complete...done.
06 快照数量限制
1.创建rbd设备，再对快照数量限制为3。
[root@ceph141~]# rbd create -s 5G zhiyong18-rbd/zhiyong
[root@ceph141~]# rbd info zhiyong18-rbd/zhiyong | egrep "snapshot_count|snapshot_limit"snapshot_count: 0
[root@ceph141~]# rbd snap limit set zhiyong18-rbd/zhiyong --limit 3
[root@ceph141~]# rbd info zhiyong18-rbd/zhiyong | egrep "snapshot_count|snapshot_limit"snapshot_count: 0snapshot_limit: 3
2.连续创建3个快照，创建第4个时提示数量限制了
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v1
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v2
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v3
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v4
Creating snap: 10% complete...failed.
rbd: failed to create snapshot: (122) Disk quota exceeded
3.清除创建快照的数量限制
[root@ceph141~]# rbd info zhiyong18-rbd/zhiyong | egrep "snapshot_count|snapshot_limit"snapshot_count: 3snapshot_limit: 3
[root@ceph141~]# rbd snap limit clear zhiyong18-rbd/zhiyong
[root@ceph141~]# rbd info zhiyong18-rbd/zhiyong | egrep "snapshot_count|snapshot_limit"snapshot_count: 3
4.限制快照数量限制解除成功，可以超过限制时的3了
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v4
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v5
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap create zhiyong18-rbd/zhiyong --snap v6
Creating snap: 100% complete...done.
[root@ceph141~]# rbd snap list zhiyong18-rbd/zhiyong
SNAPID NAME SIZE PROTECTED TIMESTAMP 6 v1 5 GiB Tue Nov 5 21:31:44 20247 v2 5 GiB Tue Nov 5 21:31:47 20248 v3 5 GiB Tue Nov 5 21:31:49 202411 v4 5 GiB Tue Nov 5 21:34:21 202412 v5 5 GiB Tue Nov 5 21:34:24 202413 v6 5 GiB Tue Nov 5 21:34:27 2024
![[C++] STL数据结构小结](http://pic.xiahunao.cn/nshx/[C++] STL数据结构小结)
