这篇学习笔记是基于杜宽老师在51CTO上发布的视频课程制作的。在此，感谢杜宽老师的分享和教学。如有侵权，请及时联系我。版权归原作者所有，未经允许不得转载或使用。
一、什么是StorageClass和CSI
StorageClass：存储类，由K8s管理员创建，用于动态PV的管理，可以链接至不同的后端存储，比如Ceph、GlusterFS等。之后对存储的请求可以指向 StorageClass，然后StorageClass会自动的创建、删除PV。
实现方式：in-tree: 内置于K8s核心代码，对于存储的管理，都需要编写相应的代码。
out-of-tree：由存储厂商提供一个驱动（CSI或Flex Volume），安装到K8s集群，然后StorageClass只需要配置该驱动即可，驱动器会代替StorageClass管理存储。
官方文档：https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/
二、Rook
Rook是一个自我管理的分布式存储编排系统，它本身并不是存储系统，在存储和k8s之前搭建了一个桥梁，使存储系统的搭建或者维护变得特别简单，Rook将分布式存储系统转变为自我管理、自我扩展、自我修复的存储服务。它让一些存储的操作，比如部署、配置、扩容、升级、迁移、灾难恢复、监视和资源管理变得自动化，无需人工处理。并且Rook支持CSI，可以利用CSI做一些PVC的快照、扩容、克隆等操作。
1、Rook架构
Rook由Operator和Cluster两部分组成：
Operator：由一些CRD和一个All in one镜像构成，包含包含启动和监控存储系统的所有功能。主要用于有状态的服务，或者用于比较复杂应用的管理。
Cluster：负责创建CRD对象，指定相关参数，包括ceph镜像、元数据持久化位置、磁盘位置、dashboard等等…
Rook：
-
Agent: 在每个存储节点上运行，用于配置一个FlexVolume插件，和k8s的存储卷进行集成。挂载网络存储、加载存储卷、格式化文件系统。
-
Discover: 用于检测连接到存储节点上的设备。
Ceph：
-
OSD: 直接连接每个集群节点的物理磁盘或者是目录。集群的副本数，高可用性和容错性。
-
Mon: 集群监控，所有集群的节点都会向Mon汇报，他记录了集群的拓扑以及数据存储位置的信息。
-
MDS: 元数据服务器，负责跟踪文件层次结构并存储Ceph元数据。
-
RGW: restful API 接口
-
MGR: 提供额外的监控和界面。
2、部署前提条件
- K8s集群至少五个节点，每个节点的内存不低于5G，CPU不低于2核
- 至少有三个存储节点，并且每个节点至少有一个裸盘
- K8s集群所有的节点时间必须一致
- k8s 1.21以上版本，快照功能需要单独安装snapshot控制器
- rook的版本大于1.3，不要使用目录创建集群，要使用单独的裸盘进行创建，也就是创建一个新的磁盘，挂载到宿主机，不进行格式化，直接使用即可。
- 如果从 Ceph 共享文件系统 (CephFS) 创建 RWX 卷，建议的最低内核版本为4.17。如果内核版本低于 4.17，则不会强制执行请求的 PVC 大小。存储配额只会在较新的内核上实施。
3、部署Rook
1、获取安装源码
git clone --single-branch --branch v1.11.8 https://github.com/rook/rook.git
2、配置更改
# cd rook/deploy/examples
修改Rook CSI镜像地址，原本的地址可能是k8s.io的镜像，但是k8s.io的镜像无法被国内访问，所以需要同步k8s.io的镜像到阿里云镜像仓库或者内网镜像仓库
# vim operator.yaml
110 ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.8.0"
111 ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.7.0"
112 ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.7.0"
113 ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.4.0"
114 ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1"
115 ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.1.0"
还是operator文件，新版本rook默认关闭了自动发现容器的部署，可以找到ROOK_ENABLE_DISCOVERY_DAEMON改成true即可
495 ROOK_ENABLE_DISCOVERY_DAEMON: "true"
如何需要指定部署位置，需要开启污点配置
185 # CSI_PROVISIONER_TOLERATIONS: |
186 # - effect: NoSchedule
187 # key: node-role.kubernetes.io/control-plane
188 # operator: Exists
189 # - effect: NoExecute
190 # key: node-role.kubernetes.io/etcd
191 # operator: Exists
192 # (Optional) CephCSI plugin NodeAffinity (applied to both CephFS and RBD plugin).
193 # CSI_PLUGIN_NODE_AFFINITY: "role=storage-node; storage=rook, ceph"
194 # (Optional) CephCSI plugin tolerations list(applied to both CephFS and RBD plugin).
195 # Put here list of taints you want to tolerate in YAML format.
196 # CSI plugins need to be started on all the nodes where the clients need to mount the storage.
197 # CSI_PLUGIN_TOLERATIONS: |
198 # - effect: NoSchedule
199 # key: node-role.kubernetes.io/control-plane
200 # operator: Exists
201 # - effect: NoExecute
202 # key: node-role.kubernetes.io/etcd
203 # operator: Exists
4、部署rook
kubectl create ns rook-ceph
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
等待operator容器和discover容器启动
# kubectl -n rook-ceph get pod
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-7c7d8846f4-fsv9f 1/1 Running 0 25h
rook-discover-qw2ln 1/1 Running 0 28h
rook-discover-wf8t7 1/1 Running 0 28h
rook-discover-z6dhq 1/1 Running 0 28h
三、创建ceph集群
1、配置更改
主要更改的是osd节点所在的位置
# vim cluster.yaml
24 image: quay.io/ceph/ceph:v17.2.6 # 修改为内网仓库镜像
37 skipUpgradeChecks: false # 跳过升级（可忽略不改）
72 ssl: false # dashboard不通过ssl访问
76 enabled: false # 打开prometheus监控指标
239 storage:
240 useAllNodes: false # 不使用宿主机所有节点
241 useAllDevices: false # 不使用宿主机所有磁盘
243 config:
252 nodes: # 手动指定存储节点及磁盘，注意config和nodes对齐
253 - name: "k8s-master03"
254 devices:
255 - name: "sdb"
- name: "k8s-node01"
devices:
- name: "sdb"
- name: "k8s-node02"
devices:
- name: "sdb"
###
- name: "k8s-master03"
devices:
- name: "sdb"
- name: "k8s-node01"
devices:
- name: "sdb"
- name: "k8s-node02"
devices:
- name: "sdb"
###
注意：新版必须采用裸盘，即未格式化的磁盘。其中k8s-master03 k8s-node01 node02有新加的一个磁盘，可以通过lsblk -f查看新添加的磁盘名称。建议最少三个节点，否则后面的试验可能会出现问题
2、创建Ceph集群
kubectl create -f cluster.yaml
创建完成后，可以查看pod的状态
# kubectl -n rook-ceph get pod
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-6nwbp 3/3 Running 0 25h
csi-cephfsplugin-b7h6k 3/3 Running 0 25h
csi-cephfsplugin-provisioner-785798bc8f-78tvr 6/6 Running 0 25h
csi-cephfsplugin-provisioner-785798bc8f-krsdx 6/6 Running 0 25h
csi-rbdplugin-2mmmj 3/3 Running 0 25h
csi-rbdplugin-85sbg 3/3 Running 0 25h
csi-rbdplugin-provisioner-75cdf8cd6d-ghl8f 6/6 Running 0 25h
csi-rbdplugin-provisioner-75cdf8cd6d-wf6h8 6/6 Running 0 25h
rook-ceph-crashcollector-k8s-master03-64c56d8d8b-9vqrk 1/1 Running 0 28h
rook-ceph-crashcollector-k8s-node01-7fc9b79798-6r2rn 1/1 Running 0 28h
rook-ceph-crashcollector-k8s-node02-6954497cb9-pqll7 1/1 Running 0 28h
rook-ceph-mgr-a-dd4bf8445-scsrt 1/1 Running 0 25h
rook-ceph-mon-a-856779ddfd-8v7s2 1/1 Running 0 28h
rook-ceph-mon-b-6c94bddf8c-wb69x 1/1 Running 0 28h
rook-ceph-mon-c-5659bcb5c9-pjn5f 1/1 Running 0 28h
rook-ceph-operator-7c7d8846f4-fsv9f 1/1 Running 0 25h
rook-ceph-osd-0-7c6cdb8546-kkcts 1/1 Running 0 28h
rook-ceph-osd-1-f8b598d47-qjnwl 1/1 Running 0 28h
rook-ceph-osd-2-55846dbcd9-jvm62 1/1 Running 0 28h
rook-ceph-osd-prepare-k8s-master03-h8lt2 0/1 Completed 0 5h1m
rook-ceph-osd-prepare-k8s-node01-jqz7x 0/1 Completed 0 5h1m
rook-ceph-osd-prepare-k8s-node02-hm8lc 0/1 Completed 0 5h1m
rook-discover-qw2ln 1/1 Running 0 29h
rook-discover-wf8t7 1/1 Running 0 29h
rook-discover-z6dhq 1/1 Running 0 29h
需要注意的是，osd-x的容器必须是存在的，且是正常的。如果上述Pod均正常，则认为集群安装成功。
更多配置：https://rook.io/docs/rook/v1.11/CRDs/Cluster/ceph-cluster-crd/
3、安装ceph snapshot控制器
k8s 1.19版本以上需要单独安装snapshot控制器，才能完成pvc的快照功能，所以在此提前安装下，如果是1.19以下版本，不需要单独安装。
snapshot控制器的部署在集群安装时的k8s-ha-install项目中，需要切换到1.20.x分支
cd /root/k8s-ha-install/
git checkout manual-installation-v1.26.x
创建snapshot controller
# kubectl create -f snapshotter/ -n kube-system
# kubectl get po -n kube-system -l app=snapshot-controller
NAME READY STATUS RESTARTS AGE
snapshot-controller-0 1/1 Running 0 27h
具体文档：https://rook.io/docs/rook/v1.11/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot/?h=snapshot
4、安装ceph客户端工具
# kubectl create -f deploy/examples/toolbox.yaml
deployment.apps/rook-ceph-tools created
待容器Running后，即可执行相关命令
# kubectl -n rook-ceph rollout status deploy/rook-ceph-tools
NAME READY STATUS RESTARTS AGE
rook-ceph-tools-6f7467bb4d-qzsvg 1/1 Running 0 <invalid>
# kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
# ceph status
cluster:
id: 356e0bac-b3e3-4abc-891b-ee6b8a0bc56f
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 29h)
mgr: a(active, since 5h)
osd: 3 osds: 3 up (since 29h), 3 in (since 29h)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 57 GiB / 60 GiB avail
pgs: 1 active+clean
# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 k8s-node01 1029M 18.9G 0 0 0 0 exists,up
1 k8s-master03 1029M 18.9G 0 0 0 0 exists,up
2 k8s-node02 1029M 18.9G 0 0 0 0 exists,up
# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 60 GiB 57 GiB 17 MiB 3.0 GiB 5.03
TOTAL 60 GiB 57 GiB 17 MiB 3.0 GiB 5.03
--- POOLS ---
POOL ID STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 1 0 B 0 0 B 0 18 GiB
# ceph config set mon auth_allow_insecure_global_id_reclaim false # 解决警告
警告解决：https://docs.ceph.com/en/octopus/rados/operations/health-checks/
5、Ceph dashboard
1、暴露服务
默认情况下，ceph dashboard是无法直接访问的，可以创建一个nodePort类型的Service暴露服务
# kubectl create -f deploy/examples/dashboard-external-https.yaml
保存退出后，会创建一个端口，然后通过https://192.168.99.110:31176即可访问该dashboard
# kubectl -n rook-ceph get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-mgr ClusterIP 10.108.111.192 <none> 9283/TCP 4h
rook-ceph-mgr-dashboard ClusterIP 10.110.113.240 <none> 8443/TCP 4h
rook-ceph-mgr-dashboard-external-https NodePort 10.101.209.6 <none> 8443:31176/TCP 4h
2、登录
账号为admin，查看密码
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
四、ceph块存储的使用
块存储一般用于一个Pod挂载一块存储使用，相当于一个服务器新挂了一个盘，只给一个应用使用。
参考文档：https://rook.io/docs/rook/v1.11/Storage-Configuration/Block-Storage-RBD/block-storage/
1、创建StorageClass和ceph的存储池
# pwd
/root/rook/deploy/examples/csi/rbd
# vim storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph # namespace:cluster
spec:
failureDomain: host
replicated:
size: 3 # 生产环境最少为3，且要小于等于osd的数量
requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-ceph # namespace:cluster
pool: replicapool
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph # namespace:cluster
csi.storage.k8s.io/fstype: ext4 # 创建文件类型
allowVolumeExpansion: true # 是否允许扩容
reclaimPolicy: Delete # PV回收策略
创建StorageClass和存储池
# kubectl create -f deploy/examples/csi/rbd/storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
查看创建的cephblockpool和storageClass（StorageClass没有namespace隔离性）
# kubectl get cephblockpool -n rook-ceph
NAME AGE
replicapool 39s
# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 107s
此时可以在ceph dashboard查看到该Pool，如果没有显示说明没有创建成功
2、挂载测试
创建一个MySQL服务
# pwd
/root/rook/deploy/examples
# kubectl create -f mysql.yaml
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created
该文件有一段pvc的配置
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: wordpress
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
pvc会连接刚才创建的storageClass，然后动态创建pv，然后连接到ceph创建对应的存储
之后创建pvc只需要指定storageClassName为刚才创建的StorageClass名称即可连接到rook的ceph。如果是statefulset，只需要将volumeTemplateClaim里面的Claim名称改为StorageClass名称即可动态创建Pod，具体请听视频。
其中MySQL deployment的volumes配置挂载了该pvc
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
value: changeme
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
claimName为pvc的名称
因为MySQL的数据不能多个MySQL实例连接同一个存储，所以一般只能用块存储。相当于新加了一块盘给MySQL使用。
创建完成后可以查看创建的pvc和pv
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-1843c13e-09cb-46c6-9dd8-5f54a834681b 20Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 65m
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-1843c13e-09cb-46c6-9dd8-5f54a834681b 20Gi RWO rook-ceph-block 66m
此时在ceph dashboard上面也可以查看到对应的image
3、StatefulSet volumeClaimTemplates动态存储
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
serviceName: "nginx"
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates: # 创建带有固定标识符的存储
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "rook-ceph-block"
resources:
requests:
storage: 1Gi
五、共享文件系统的使用
共享文件系统一般用于多个Pod共享一个存储
官方文档：https://rook.io/docs/rook/v1.11/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage/
1、创建共享类型的文件系统
通过为CephFilesystemCRD中的元数据池，数据池和元数据服务器指定所需的设置来创建文件系统
# pwd
/root/rook/deploy/examples
# kubectl create -f filesystem.yaml
cephfilesystem.ceph.rook.io/myfs created
# vim filesystem.yaml
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
name: myfs
namespace: rook-ceph
spec:
metadataPool:
replicated:
size: 3
dataPools:
- name: replicated
replicated:
size: 3
preserveFilesystemOnDelete: true
metadataServer:
activeCount: 1
activeStandby: true
创建完成后会启动mds容器，需要等待启动后才可进行创建pv
# kubectl -n rook-ceph get pod -l app=rook-ceph-mds
NAME READY STATUS RESTARTS AGE
rook-ceph-mds-myfs-7d59fdfcf4-h8kw9 1/1 Running 0 12s
rook-ceph-mds-myfs-7d59fdfcf4-kgkjp 1/1 Running 0 12s
也可以在ceph dashboard上面查看状态
2、创建共享类型文件系统的StorageClass
# pwd
/root/rook/deploy/examples/csi/cephfs
# kubectl create -f storageclass.yaml
storageclass.storage.k8s.io/rook-cephfs created
之后将pvc的storageClassName设置成rook-cephfs即可创建共享文件类型的存储，类似于NFS，可以给多个Pod共享数据
3、挂载测试
# pwd
/root/rook/deploy/examples/csi/cephfs
# ls
kube-registry.yaml pod-ephemeral.yaml pod.yaml pvc-clone.yaml pvc-restore.yaml pvc.yaml snapshotclass.yaml snapshot.yaml storageclass-ec.yaml storageclass.yaml
# kubectl create -f kube-registry.yaml
persistentvolumeclaim/cephfs-pvc created
deployment.apps/kube-registry created
查看创建的pvc和pod
# kubectl get po -n kube-system -l k8s-app=kube-registry
NAME READY STATUS RESTARTS AGE
kube-registry-66d4c7bf47-46bpq 1/1 Running 0 34s
kube-registry-66d4c7bf47-jp88x 1/1 Running 0 34s
kube-registry-66d4c7bf47-xdtpx 1/1 Running 0 34s
# kubectl get pvc -n kube-system
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-pvc Bound pvc-0149518a-7346-4d16-9030-97b2f9b8e9d2 1Gi RWX rook-cephfs 100s
该文件的pvc配置，用于连接ceph创建存储
apiVersion: v1
kind: Persistent VolumeClaim
metadata:
name: cephfs-pvc
namespace: kube-system
spec :
accessModes:
- ReadiriteMany
resources:
requests:
storage: 1G
storageClassName: rook-cephfs
然后在deployment的volumes配置添加了该pvc，并将其挂载到了/var/lib/registry
注意claimName为pvc的名称。
此时一共创建了三个Pod，这三个Pod共用了一个存储，挂载到了/var/lib/registry，该目录三个容器共享数据。
4、Nginx挂载测试
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
selector:
app: nginx
type: ClusterIP
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nginx-share-pvc
spec:
storageClassName: rook-cephfs
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx # has to match .spec.template.metadata.labels
replicas: 3 # by default is 1
template:
metadata:
labels:
app: nginx # has to match .spec.selector.matchLabels
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumes:
- name: www
persistentVolumeClaim:
claimName: nginx-share-pvc
六、PVC扩容
文件共享类型的PVC扩容需要k8s 1.15+
块存储类型的PVC扩容需要k8s 1.16+
PVC扩容需要开启ExpandCSIVolumes，新版本的k8s已经默认打开了这个功能，可以查看自己的k8s版本是否已经默认打开了该功能：
# kube-apiserver -h |grep ExpandCSIVolumes
ExpandCSIVolumes=true|false (BETA - default=true)
如果default为true就不需要打开此功能，如果default为false，需要开启该功能。
1、扩容文件共享型PVC
找到刚才创建的文件共享型StorageClass，将allowVolumeExpansion设置为true（新版rook默认为true，如果不为true更改后执行kubectl replace即可）
找到之前创建的pvc
# kubectl get pvc -n kube-system
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-pvc Bound pvc-0149518a-7346-4d16-9030-97b2f9b8e9d2 1Gi RWX rook-cephfs 44m
# kubectl edit pvc cephfs-pvc -n kube-system
将大小修改为2Gi，之前是1Gi
保存退出
# kubectl edit pvc cephfs-pvc -n kube-system
persistentvolumeclaim/cephfs-pvc edited
查看PV和PVC的大小
# kubectl get pvc -n kube-system
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cephfs-pvc Bound pvc-0149518a-7346-4d16-9030-97b2f9b8e9d2 2Gi RWX rook-cephfs 46m
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-0149518a-7346-4d16-9030-97b2f9b8e9d2 2Gi RWX Delete Bound kube-system/cephfs-pvc rook-cephfs 47m
pvc-fd8b9860-c0d4-4797-8e1d-1fab880bc9fc 2Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 64m
查看容器内是否已经完成扩容
# kubectl exec -ti kube-registry-66d4c7bf47-46bpq -n kube-system -- df -Th | grep "/var/lib/registry"
ceph 2.0G 0 2.0G 0% /var/lib/registry
同样的步骤可以扩容到3G
2、扩容块存储
扩容步骤类似，找到PVC，直接edit即可
# kubectl edit pvc mysql-pv-claim
persistentvolumeclaim/mysql-pv-claim edited
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-fd8b9860-c0d4-4797-8e1d-1fab880bc9fc 2Gi RWO rook-ceph-block 70m
可以看到此时pvc并没有扩容，但是pv已经扩容
# kubectl get pv | grep default/mysql-pv-claim
pvc-fd8b9860-c0d4-4797-8e1d-1fab880bc9fc 3Gi RWO Delete Bound default/mysql-pv-claim rook-ceph-block 75m
也可以看到ceph dashboard的image也完成了扩容，但是pvc和pod里面的状态会有延迟，大概等待5-10分钟后，即可完成扩容
七、PVC快照
注意：PVC快照功能需要k8s 1.17+
1、块存储快照
1、创建snapshotClass
# pwd
/root/rook/deploy/examples/csi/rbd
# kubectl create -f snapshotclass.yaml
volumesnapshotclass.snapshot.storage.k8s.io/csi-rbdplugin-snapclass created
2、创建快照
首先在之前创建的MySQL容器里创建一个文件夹，并创建一个文件
# kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-m5gj5 1/1 Running 1 3d4h
wordpress-mysql-6965fc8cc8-6wt6j 1/1 Running 0 86m
# kubectl exec -ti wordpress-mysql-6965fc8cc8-6wt6j -- bash
root@wordpress-mysql-6965fc8cc8-6wt6j:/# ls
bin boot dev docker-entrypoint-initdb.d entrypoint.sh etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
root@wordpress-mysql-6965fc8cc8-6wt6j:/# cd /var/lib/mysql
root@wordpress-mysql-6965fc8cc8-6wt6j:/var/lib/mysql# ls
auto.cnf ib_logfile0 ib_logfile1 ibdata1 lost+found mysql performance_schema
root@wordpress-mysql-6965fc8cc8-6wt6j:/var/lib/mysql# mkdir test_snapshot
root@wordpress-mysql-6965fc8cc8-6wt6j:/var/lib/mysql# ls
auto.cnf ib_logfile0 ib_logfile1 ibdata1 lost+found mysql performance_schema test_snapshot
root@wordpress-mysql-6965fc8cc8-6wt6j:/var/lib/mysql# echo "test for snapshot" > test_snapshot/1.txt
root@wordpress-mysql-6965fc8cc8-6wt6j:/var/lib/mysql# cat test_snapshot/1.txt
test for snapshot
修改snapshot.yaml文件的source pvc为创建的MySQL pvc
创建快照及查看状态
# kubectl create -f snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/rbd-pvc-snapshot created
# kubectl get volumesnapshotclass
NAME DRIVER DELETIONPOLICY AGE
csi-rbdplugin-snapclass rook-ceph.rbd.csi.ceph.com Delete 8m37s
# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
rbd-pvc-snapshot true mysql-pv-claim 3Gi csi-rbdplugin-snapclass snapcontent-715c2841-d1ba-493a-9eb9-52178df3c2e6 <invalid> <invalid>
3、指定快照创建PVC
如果想要创建一个具有某个数据的PVC，可以从某个快照恢复
# cat pvc-restore.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-restore
spec:
storageClassName: rook-ceph-block
dataSource:
name: rbd-pvc-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
# kubectl create -f pvc-restore.yaml
persistentvolumeclaim/rbd-pvc-restore created
注意：dataSource为快照名称，storageClassName为新建pvc的storageClass，storage的大小不能低于原pvc的大小
# kubectl create -f pvc-restore.yaml
persistentvolumeclaim/rbd-pvc-restore created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-fd8b9860-c0d4-4797-8e1d-1fab880bc9fc 3Gi RWO rook-ceph-block 95m
rbd-pvc-restore Bound pvc-d86a7535-2331-4fef-ae93-c570c8d3f9e7 3Gi RWO rook-ceph-block 2s
4、数据校验
创建一个容器，挂载该PVC，查看是否含有之前的文件
# cat restore-check-snapshot-rbd.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: check-snapshot-restore
spec:
selector:
matchLabels:
app: check
strategy:
type: Recreate
template:
metadata:
labels:
app: check
spec:
containers:
- image: alpine:3.8
name: check
command:
- sh
- -c
- sleep 36000
volumeMounts:
- name: check-mysql-persistent-storage
mountPath: /mnt
volumes:
- name: check-mysql-persistent-storage
persistentVolumeClaim:
claimName: rbd-pvc-restore
# kubectl create -f restore-check-snapshot-rbd.yaml
deployment.apps/check-snapshot-restore created
查看数据是否存在
# kubectl get po
NAME READY STATUS RESTARTS AGE
check-snapshot-restore-64b85c5f88-zvr62 1/1 Running 0 97s
nginx-6799fc88d8-m5gj5 1/1 Running 1 3d5h
wordpress-mysql-6965fc8cc8-6wt6j 1/1 Running 0 104m
You have new mail in /var/spool/mail/root
# kubectl exec -ti check-snapshot-restore-64b85c5f88-zvr62 -- ls
bin etc lib mnt root sbin sys usr
dev home media proc run srv tmp var
# kubectl exec -ti check-snapshot-restore-64b85c5f88-zvr62 -- ls /mnt
auto.cnf ibdata1 performance_schema
ib_logfile0 lost+found test_snapshot
ib_logfile1 mysql
# kubectl exec -ti check-snapshot-restore-64b85c5f88-zvr62 -- ls /mnt/test_snapshot
1.txt
# kubectl exec -ti check-snapshot-restore-64b85c5f88-zvr62 -- cat /mnt/test_snapshot/1.txt
test for snapshot
测试无误后清理数据（snapshotclass可以不删除，后期创建rbd快照直接用该snapshotclass即可）
# kubectl delete -f restore-check-snapshot-rbd.yaml -f pvc-restore.yaml -f snapshot.yaml
deployment.apps "check-snapshot-restore" deleted
persistentvolumeclaim "rbd-pvc-restore" deleted
volumesnapshot.snapshot.storage.k8s.io "rbd-pvc-snapshot" deleted
2、文件共享型快照
1、创建snapshotClass
# pwd
/root/rook/deploy/examples/csi/cephfs
# kubectl create -f snapshotclass.yaml
3、快照回滚
# pwd
/root/rook/deploy/examples/csi/cephfs
# kubectl create -f pvc-restore.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc-restore
spec:
storageClassName: rook-cephfs # 与pv的名称一致
dataSource:
name: cephfs-pvc-snapshot # 快照名称
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi # 不能小于需要恢复pv的大小
八、PVC克隆
# pwd
/root/rook/deploy/examples/csi/rbd
# cat pvc-clone.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-clone
spec:
storageClassName: rook-ceph-block
dataSource:
name: mysql-pv-claim
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
# kubectl create -f pvc-clone.yaml
persistentvolumeclaim/rbd-pvc-clone created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-fd8b9860-c0d4-4797-8e1d-1fab880bc9fc 3Gi RWO rook-ceph-block 110m
rbd-pvc-clone Bound pvc-6dda14c9-9d98-41e6-9d92-9d4ea382c614 3Gi RWO rook-ceph-block 4s
需要注意的是pvc-clone.yaml的dataSource的name是被克隆的pvc名称，在此是mysql-pv-claim，storageClassName为新建pvc的storageClass名称，storage不能小于之前pvc的大小
九、测试数据清理
如果Rook要继续使用，可以只清理创建的deploy、pod、pvc即可。之后可以直接投入使用
数据清理步骤：
1、首先清理挂载了PVC和Pod，可能需要清理单独创建的Pod和Deployment或者是其他的高级资源
kubectl get pod
kubectl delete pod pod名
2、之后清理PVC，清理掉所有通过ceph StorageClass创建的PVC后，最好检查下PV是否被清理
kubectl get pvc
kubectl delete pvc pvc名
3、之后清理快照
kubectl get volumesnapshot
kubectl delete volumesnapshot XXXXXXXX
4、之后清理创建的Pool，包括块存储和文件存储
kubectl delete -n rook-ceph cephblockpool replicapool
kubectl delete storageclass rook-ceph-block
kubectl delete -f csi/cephfs/kube-registry.yaml
kubectl delete storageclass csi-cephfs
5、清理StorageClass：
kubectl get sc
kubectl delete sc rook-ceph-block rook-cephfs
6、清理Ceph集群：
kubectl -n rook-ceph patch cephcluster rook-ceph --type merge -p '{"spec":{"cleanupPolicy":{"confirmation":"yes-really-destroy-data"}}}'
kubectl -n rook-ceph delete cephcluster rook-ceph
kubectl -n rook-ceph get cephcluster
7、删除Rook资源：
kubectl delete -f operator.yaml
kubectl delete -f common.yaml
kubectl delete -f crds.yaml
8、如果卡住需要参考Rook的troubleshooting
for CRD in $(kubectl get crd -n rook-ceph | awk '/ceph.rook.io/ {print $1}'); do
kubectl get -n rook-ceph "$CRD" -o name | \
xargs -I {} kubectl patch -n rook-ceph {} --type merge -p '{"metadata":{"finalizers": []}}'
done
9、清理数据目录和磁盘
参考链接：https://rook.io/docs/rook/v1.11/Storage-Configuration/ceph-teardown/#delete-the-data-on-hosts
十、K8S连接外部Ceph集群
1、使⽤Ceph-CSI为默认storageClass
1、cephCSI存储配置
ceph存储由外部提供,以下步骤在ceph上操作,⽬的是为kubenetes访问ceph使⽤
1、ceph新建cephpool
ceph osd pool create kubernetes 64 64
pool 'kubernetes' created
2、创建一个名字为client.k8s的用户，且只能使用指定的pool（kubernetes）
# ceph auth get-or-create client.k8s mon 'allow r' osd 'allow class-read object_prefix rbd_children,allow rwx pool=kubernetes'
[client.k8s]
key = AQD+pg9gjreqMRAAtwq4dQnwX0kX4Vx6TueAJQ==
查看ceph信息
记录fsid的信息(集群ID)a.
mon_host信息(Ceph监视器),暂时只⽀持V1,参考⽂档
# cat ceph.conf
[global]
fsid = 6553e7de-1b81-454c-a381-40fdsaFfas9 # 重点
mon_initial_members = uic_sh_gw, uic_sh_stack03, uic_sh_stack04, uic_sh_stack05,
uic_sh_stack06
mon_host = 10.10.160.2,10.10.160.3,10.10.160.4,10.10.160.5,10.10.160.6 #重点
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
2、部署Ceph-CSI
1、ceph-csi下载
git clone --depth 1 --branch v3.1.0 https://gitclone.com/github.com/ceph/ceph-csi
2、修改ConfigMap信息
修改访问Ceph集群配置，进⼊⽬录(/root/ceph-csi/deploy/rbd/kubernetes)
# pwd
/root/ceph-csi/deploy/rbd/kubernetes
# ll
total 44
-rw-r--r-- 1 root root 317 Jan 18 12:43 csi-config-map.yaml
-rw-r--r-- 1 root root 1689 Jan 18 12:53 csi-nodeplugin-psp.yaml
-rw-r--r-- 1 root root 881 Jan 18 12:53 csi-nodeplugin-rbac.yaml
-rw-r--r-- 1 root root 1315 Jan 18 12:53 csi-provisioner-psp.yaml
-rw-r--r-- 1 root root 3131 Jan 18 12:53 csi-provisioner-rbac.yaml
-rw-r--r-- 1 root root 5502 Jan 18 13:08 csi-rbdplugin-provisioner.yaml
-rw-r--r-- 1 root root 5857 Jan 18 18:21 csi-rbdplugin.yaml
-rw-r--r-- 1 root root 161 Jan 18 17:19 csi-rbd-secret.yaml #⾃⼰⽣成的secret
-rw-r--r-- 1 root root 706 Jan 18 13:21 storageclass.yaml #⾃⼰⽣成的动态存储
修改ConfigMap的信息,
使⽤configmap将服务的clusterID等信息录⼊到集群(namespace)中
# vim csi-config-map.yaml
apiVersion: v1
kind: ConfigMap
data:
config.json: |-
[
{
"clusterID": "081dc49f-2525-4aaa-a56d-89d641cef302", # ceph mon dump获取
"monitors": [
"192.168.1.129:6789",
"192.168.1.130:6789",
"192.168.1.131:6789"
]
}
]
metadata:
name: ceph-csi-config
# kubectl apply -f csi-config-map.yaml
configmap/ceph-csi-config created
# kubectl get cm ceph-csi-config
NAME DATA AGE
ceph-csi-config 1 19s
3、创建ceph-CSI的namespace空间
ceph-CSI单独使⽤的ns资源
kubectl create ns ceph-csi
kubectl -n ceph-csi apply -f csi-config-map.yaml
4、生成ceph用户（k8s）密钥
新建secret⽂件，将ceph创建的k8s的⽤⼾名和密码⽣成secret
# cat csi-rbd-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
namespace: ceph-csi
stringData:
userID: k8s # ceph集群创建的username # 这里用admin的要不然下面会报错！！！！！！！！！！！！！
userKey: AQD+pg9gjreqMRAAtwq4dQnwX0kX4Vx6TueAJQ== # 这个key不需要base64加密！！！！！！！！！！！！！！！！！
⽣成secret
# kubectl apply -f csi-rbd-secret.yaml
secret/csi-rbd-secret created
# kubectl get secret csi-rbd-secret
NAME TYPE DATA AGE
csi-rbd-secret Opaque 2 14s
3、RBAC权限配置
1、更改默认配置的namespace
将所有配置清单中的 namespace 改成 ceph-csi
sed -i "s/namespace: default/namespace: ceph-csi/g" $(grep -rl "namespace: default" ./)
sed -i -e "/^kind: ServiceAccount/{N;N;a\ namespace: ceph-csi # 输⼊到这⾥的时候
需要按⼀下回⻋键，在下⼀⾏继续输⼊
}" $(egrep -rl "^kind: ServiceAccount" ./)
2、创建 ServiceAccount 和RBACClusterRole/ClusterRoleBinding资源对象
kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml
3、创建PodSecurityPolicy
kubectl create -f csi-provisioner-psp.yaml
kubectl create -f csi-nodeplugin-psp.yaml
4、CSIsidecar和RBDCSIdriver
CSIsidecar：⽤来简化开发和部署，⽤来监控KubernetesAPI，针对CSI容器触发⼀些适当的操作，根据需求更新API，⼀般配合CSiDrive去使⽤
1、创建ceph csi供应器和节点插件
# vim csi-rbdplugin-provisioner.yaml
kind: Service
apiVersion: v1
metadata:
name: csi-rbdplugin-provisioner
labels:
app: csi-metrics
spec:
selector:
app: csi-rbdplugin-provisioner
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-rbdplugin-provisioner
spec:
replicas: 3
selector:
matchLabels:
app: csi-rbdplugin-provisioner
template:
metadata:
labels:
app: csi-rbdplugin-provisioner
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- csi-rbdplugin-provisioner
topologyKey: "kubernetes.io/hostname"
serviceAccount: rbd-csi-provisioner
containers:
- name: csi-provisioner
image: k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4
args:
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--timeout=150s"
- "--retry-interval-start=500ms"
- "--leader-election=true"
# set it to true to use topology based provisioning
- "--feature-gates=Topology=false"
# if fstype is not specified in storageclass, ext4 is default
- "--default-fstype=ext4"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-snapshotter
image: k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.2
args:
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--timeout=150s"
- "--leader-election=true"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
securityContext:
privileged: true
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-attacher
image: k8s.gcr.io/sig-storage/csi-attacher:v3.0.2
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election=true"
- "--retry-interval-start=500ms"
env:
- name: ADDRESS
value: /csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-resizer
image: k8s.gcr.io/sig-storage/csi-resizer:v1.0.1
args:
- "--csi-address=$(ADDRESS)"
- "--v=5"
- "--timeout=150s"
- "--leader-election"
- "--retry-interval-start=500ms"
- "--handle-volume-inuse-error=false"
env:
- name: ADDRESS
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: csi-rbdplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
# for stable functionality replace canary with latest release version
image: quay.io/cephcsi/cephcsi:canary
args:
- "--nodeid=$(NODE_ID)"
- "--type=rbd"
- "--controllerserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--pidlimit=-1"
- "--rbdhardmaxclonedepth=8"
- "--rbdsoftmaxclonedepth=4"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# - name: POD_NAMESPACE
# valueFrom:
# fieldRef:
# fieldPath: spec.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: ceph-csi-encryption-kms-config
mountPath: /etc/ceph-csi-encryption-kms-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: csi-rbdplugin-controller
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
# for stable functionality replace canary with latest release version
image: quay.io/cephcsi/cephcsi:canary
args:
- "--type=controller"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
- "--drivernamespace=$(DRIVER_NAMESPACE)"
env:
- name: DRIVER_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: liveness-prometheus
image: quay.io/cephcsi/cephcsi:canary
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi-provisioner.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: lib-modules
hostPath:
path: /lib/modules
- name: socket-dir
emptyDir: {
medium: "Memory"
}
- name: ceph-csi-config
configMap:
name: ceph-csi-config
- name: ceph-csi-encryption-kms-config
configMap:
name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
# vim csi-rbdplugin.yaml
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-rbdplugin
spec:
selector:
matchLabels:
app: csi-rbdplugin
template:
metadata:
labels:
app: csi-rbdplugin
spec:
serviceAccount: rbd-csi-nodeplugin
hostNetwork: true
hostPID: true
# to use e.g. Rook orchestrated cluster, and mons' FQDN is
# resolved through k8s service, set dns policy to cluster first
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: driver-registrar
# This is necessary only for systems with SELinux, where
# non-privileged sidecar containers cannot access unix domain socket
# created by privileged CSI driver container.
securityContext:
privileged: true
image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1
args:
- "--v=5"
- "--csi-address=/csi/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: csi-rbdplugin
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
# for stable functionality replace canary with latest release version
image: quay.io/cephcsi/cephcsi:canary
args:
- "--nodeid=$(NODE_ID)"
- "--type=rbd"
- "--nodeserver=true"
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=5"
- "--drivername=rbd.csi.ceph.com"
# If topology based provisioning is desired, configure required
# node labels representing the nodes topology domain
# and pass the label names below, for CSI to consume and advertise
# its equivalent topology domain
# - "--domainlabels=failure-domain/region,failure-domain/zone"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
# - name: POD_NAMESPACE
# valueFrom:
# fieldRef:
# fieldPath: spec.namespace
# - name: KMS_CONFIGMAP_NAME
# value: encryptionConfig
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- mountPath: /dev
name: host-dev
- mountPath: /sys
name: host-sys
- mountPath: /run/mount
name: host-mount
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- name: ceph-csi-config
mountPath: /etc/ceph-csi-config/
- name: ceph-csi-encryption-kms-config
mountPath: /etc/ceph-csi-encryption-kms-config/
- name: plugin-dir
mountPath: /var/lib/kubelet/plugins
mountPropagation: "Bidirectional"
- name: mountpoint-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- name: keys-tmp-dir
mountPath: /tmp/csi/keys
- name: liveness-prometheus
securityContext:
privileged: true
image: quay.io/cephcsi/cephcsi:canary
args:
- "--type=liveness"
- "--endpoint=$(CSI_ENDPOINT)"
- "--metricsport=8680"
- "--metricspath=/metrics"
- "--polltime=60s"
- "--timeout=3s"
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- name: socket-dir
mountPath: /csi
imagePullPolicy: "IfNotPresent"
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/rbd.csi.ceph.com
type: DirectoryOrCreate
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins
type: Directory
- name: mountpoint-dir
hostPath:
path: /var/lib/kubelet/pods
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/
type: Directory
- name: host-dev
hostPath:
path: /dev
- name: host-sys
hostPath:
path: /sys
- name: host-mount
hostPath:
path: /run/mount
- name: lib-modules
hostPath:
path: /lib/modules
- name: ceph-csi-config
configMap:
name: ceph-csi-config
- name: ceph-csi-encryption-kms-config
configMap:
name: ceph-csi-encryption-kms-config
- name: keys-tmp-dir
emptyDir: {
medium: "Memory"
}
---
# This is a service to expose the liveness metrics
apiVersion: v1
kind: Service
metadata:
name: csi-metrics-rbdplugin
labels:
app: csi-metrics
spec:
ports:
- name: http-metrics
port: 8080
protocol: TCP
targetPort: 8680
selector:
app: csi-rbdplugin
# kubectl apply -f csi-rbdplugin-provisioner.yaml -f csi-rbdplugin.yaml
2、部署Sidecar容器
包括 external-provisioner 、 external-attacher 、 csi-resizer 和 csi-rbdplugin
kubectl -n ceph-csi create -f csi-rbdplugin-provisioner.yaml
3、部署RBDCSIDriver容器
kubectl -n ceph-csi create -f csi-rbdplugin.yaml
查看部署信息
kubectl get pod -n ceph-csi
4、创建Storageclass
clusterID：对应的fsid(ceph)。
imageFeatures：这是磁盘映像⽂件的特征，需要 uname -r 查看集群系统内核所⽀持的特性，centos7可以⽤layering
# cat csi-rbd-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com # 驱动
parameters:
clusterID: b9127830-b0cc-4e34-aa47-9d1a2e9949a8 # ceph集群id（ceph -s 查看）
pool: kubernetes # 访问的pool name
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret # 默认是这些个名字
csi.storage.k8s.io/provisioner-secret-namespace: default # 默认在default namespace
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
mountOptions:
- discard
创建storageClass
# kubectl apply -f csi-rbd-sc.yaml
storageclass.storage.k8s.io/csi-rbd-sc created
# kubectl get storageclass.storage.k8s.io/csi-rbd-sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc rbd.csi.ceph.com Delete Immediate false 17s
设置成默认存储(可选)
kubectl patch storageclass csi-rbd-sc -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
# kubectl get StorageClass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-rbd-sc (default) rbd.csi.ceph.com Delete Immediate true 25h
验证：PVC
使用官方PVCyaml，创建pvc资源
cd /root/ceph-csi/examples/rbd
kubectl apply -f pvc.yaml
查看：STATUS显⽰Bound表示成功
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-7f79cfc7-46ee-4dc9-9bc3-c3f1ab98002b 1Gi RWO csi-rbd-sc 21h
评论区