目录

系统环境

Debian		version 	11.2
Docker		version 	20.10.10
Ceph 		version 	16.2.7
Kubesphere	version		v3.2.1
Kubernetes	version 	v1.21.5
Rook 		version		v1.8.7

前期准备

  • 尽量准备一个干净的系统
  • 安装 docker 20.10.10
  • 安装 lvm2 sudo apt-get install -y lvm2

安装流程总览

  • 下载kubekey 安装k8s
  • 使用k8s安装Rook
  • 使用Rook安装ceph集群
  • 替换k8s默认存储方式
  • 安装kubesphere

安装kubekey

export KKZONE=cn

curl -sfL https://get-kk.kubesphere.io | VERSION=v1.2.1 sh -

下载完成后

使用

./kk create cluster --with-kubernetes v1.21.5

生成k8s的安装配置文件，此时不需要生成kubesphere配置，否则会直接安装自带默认存储openEBS的kubesphere。

修改配置，此配置可以参照 kubesphere多节点安装

spec:
  hosts:
  - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, user: root, password: Pass@123}
  - {name: node1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: Pass@123}
  - {name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: Pass@123}
  roleGroups:
    etcd:
    - master1
    master:
    - master1
    worker:
    - node1
    - node2

安装k8s ./kk create cluster -f config-sample.yaml

安装Rook

准备

至少需要三块无分区或格式化文件系统的硬盘, 在下面例子中每一台服务挂载了一块名称为 vdb 的ssd
如果硬盘已经格式化了 可参考 此文章

root@master1:~# kubectl get node
NAME      STATUS   ROLES                         AGE   VERSION
master1   Ready    control-plane,master,worker   35h   v1.21.5
node1     Ready    worker                        35h   v1.21.5
node2     Ready    worker                        35h   v1.21.5
root@master1:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    254:0    0  100G  0 disk 
└─vda1 254:1    0  100G  0 part /
vdb    254:16   0  300G  0 disk

拉取rook代码
git clone --single-branch --branch v1.8.7 https://github.com/rook/rook.git

cd rook/deploy/examples

建议配置 docker 配置国内镜像，但是ceph几个镜像国内无法访问
直接拉取 代替镜像，之后配合修改ceph yaml 文件

docker pull willdockerhub/csi-node-driver-registrar:v2.5.0
docker pull willdockerhub/csi-snapshotter:v5.0.1
docker pull willdockerhub/csi-provisioner:v3.1.0
docker pull willdockerhub/csi-provisioner:v3.1.0
docker pull willdockerhub/csi-resizer:v1.4.0

安装operator

修改 operator.yaml 添加

  ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.5.1"
  ROOK_CSI_REGISTRAR_IMAGE: "willdockerhub/csi-node-driver-registrar:v2.5.0"
  ROOK_CSI_RESIZER_IMAGE: "willdockerhub/csi-resizer:v1.4.0"
  ROOK_CSI_PROVISIONER_IMAGE: "willdockerhub/csi-provisioner:v3.1.0"
  ROOK_CSI_SNAPSHOTTER_IMAGE: "willdockerhub/csi-snapshotter:v5.0.1"
  ROOK_CSI_ATTACHER_IMAGE: "willdockerhub/csi-attacher:v3.4.0"

kubectl create -f crds.yaml -f common.yaml operator.yaml

配置 cluster

修改 cluster.yaml 配置参考官网

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph # namespace:cluster
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v16.2.7
    allowUnsupported: false
  #修改挂载的目录，不需要预先挂载，ceph集群会自动分配管理
  dataDirHostPath: /data/rook
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  waitTimeoutForHealthyOSDInMinutes: 10
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
    modules:
      - name: pg_autoscaler
        enabled: true
  dashboard:
    enabled: true
    ssl: false
  monitoring:
    enabled: false
    rulesNamespace: rook-ceph
  network:
  crashCollector:
    disable: false
  cleanupPolicy:
    confirmation: ""
    sanitizeDisks:
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
  annotations:
  labels:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  storage: 
  #修改的重点一定要和k8s节点一致
    useAllNodes: false
    useAllDevices: false
    #deviceFilter:
    nodes:
    - name: "master1"
      devices:
      - name: "vdb"
    - name: "node1"
      devices:
      - name: "vdb"
    - name: "node2"
      devices:
      - name: "vdb"
    config:
    onlyApplyOSDPlacement: false
  disruptionManagement:
    managePodBudgets: true
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0
    manageMachineDisruptionBudgets: false
    machineDisruptionBudgetNamespace: openshift-machine-api

  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

为osd节点增加label

注意规则kubectl label nodes [节点名称] ceph-osd=enabled

kubectl label nodes master1 ceph-osd=enabled
kubectl label nodes node1 ceph-osd=enabled
kubectl label nodes node2 ceph-osd=enabled

执行安装

kubectl apply -f cluster.yaml

安装toolbox

Rook工具箱是一个容器，其中包含用于rook调试和测试的常用工具。

一旦 toolbox 的 Pod 运行成功后，我们就可以使用下面的命令进入到工具箱内部进行操作：

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

进入tool

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

使用的例子

  • ceph status
  • ceph osd status
  • ceph df
  • rados df

访问dashboard

root@node1:~# kubectl get svc -n rook-ceph
NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
csi-cephfsplugin-metrics                ClusterIP   10.233.57.122   <none>        8080/TCP,8081/TCP   36h
csi-rbdplugin-metrics                   ClusterIP   10.233.9.112    <none>        8080/TCP,8081/TCP   36h
rook-ceph-mgr                           ClusterIP   10.233.30.91    <none>        9283/TCP            36h
rook-ceph-mgr-dashboard                 ClusterIP   10.233.29.237   <none>        7000/TCP            36h
rook-ceph-mon-a                         ClusterIP   10.233.4.114    <none>        6789/TCP,3300/TCP   36h
rook-ceph-mon-b                         ClusterIP   10.233.9.74     <none>        6789/TCP,3300/TCP   36h
rook-ceph-mon-c                         ClusterIP   10.233.2.224    <none>        6789/TCP,3300/TCP   36h

如果需要外网访问可以在当前目录下创建一个NodePort访问服务
touch dashboard-nodeport.yaml

内容如下：

apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mgr-dashboard-nodeport-http
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-mgr-dashboard
    rook_cluster: rook-ceph # namespace:cluster
spec:
  ports:
    - name: dashboard
      port: 7000
      protocol: TCP
      targetPort: 7000
      nodePort: 30008
  selector:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort

获取密码
kubectl get secrets -n rook-ceph rook-ceph-dashboard-password -o jsonpath='{.data.password}' | base64 -d

使用ip:30008就能访问了

创建pool和StorageClass

kubectl apply -f csi/rbd/storageclass.yaml

此时能看到成功创建的storageClass了

root@node1:~# kubectl get sc
NAME                        PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block             rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   31h

修改默认storageClass
kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

验证：

root@node1:~# kubectl get sc
NAME                        PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block (default)   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   31h

安装kubesphere

重新生成config-sample.yaml

./kk create config --with-kubesphere v3.2.1 --with-kubernetes v1.21.5

编辑config-sample.yaml和最开始保持一致

spec:
  hosts:
  - {name: master, address: 192.168.0.2, internalAddress: 192.168.0.2, user: root, password: Pass@123}
  - {name: node1, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: Pass@123}
  - {name: node2, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: Pass@123}
  roleGroups:
    etcd:
    - master1
    master:
    - master1
    worker:
    - node1
    - node2

安装kubesphere

./kk create cluster -f config-sample.yaml

验证

访问kubesphere

9F71FABD-8117-48AA-B6F7-6E6B9F67A7B1

成功

Q.E.D.