新版使用cephadm安装ceph octopus

admin · 发表于 2021-7-19 10:56:03

马上注册，结交更多好友，享用更多功能，让你轻松玩转社区。

您需要登录才可以下载或查看，没有帐号？开始注册

x

Ceph集群基础库安装
步骤：
2-1. 安装配置其它基础包，pip、deltarpm、ceph-common
2-2. 防火墙规则添加

2-1 安装配置其它基础包，pip、deltarpm、ceph-common

1. sed -i 's/^SELINUX\=.*/SELINUX=disabled/g' /etc/selinux/config2.  yum install -y python3 epel-release ceph-mgr-dashboard ceph-common pip3 install --upgrade pip3. yum install -y snappy leveldb gdisk python3-ceph-argparse python3-flask gperftools-libs4. yum install -y ceph

2-2 防火墙规则添加

bash> firewall-cmd --zone=public --add-service=ceph-mon --permanentbash> firewall-cmd --zone=public --add-service=ceph --permanentbash> firewall-cmd --zone=public --add-service=ntp --permanentbash> firewall-cmd --reload
附：ceph相关使端口列表
　　CephMonitor(ceph-mon)：3300、6789(TCP)
　　CephManager(ceph-mgr)：6800、6801、以及一个自定义Web端口(TCP)
　　CephOSD(ceph-osd): 6800->7300(TCP)
以上为止，则整个ceph集群的基础依赖环境都安装好。

Cephadm使用podman容器和systemd安装和管理Ceph分布式集群，并与CLI和仪表板GUI紧密集成。
•cephadm仅支持octopus  v15.2.0或者更高版本。
•cephadm与新的业务流程API完全集成，并完全支持新的CLI和仪表板功能来管理集群部署。
•cephadm需要容器支持（podman或docker）和Python 3。
•时间同步

这里使用的centos8来安装的ceph

配置hosts解析cat >> /etc/hosts <<EOF192.168.8.65 node1192.168.8.66 node2192.168.8.67 node3EOF关闭selinux
setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config设置主机名hostnamectl set-hostname node1安装podmandnf install -y podman
安装cephadm
cephadm命令可以
1.引导新集群
2.使用有效的Ceph CLI启动容器化的Shell
3.帮助调试容器化的Ceph守护进程。

以下操作只在一台节点执行就可以
curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
安装cephadmn
./cephadm add-repo --release octopus
./cephadm install
要引导群集，需要先创建一个目录：/etc/ceph
mkdir -p /etc/ceph
然后运行该命令：ceph bootstrap
cephadm bootstrap --mon-ip 192.168.8.65

此命令将会进行以下操作：
为本地主机上的新群集创建monitor和manager守护程序。
为 Ceph 群集生成新的 SSH 密钥，并将其添加到root用户的文件/root/.ssh/authorized_keys
将与新群集通信所需的最小配置文件保存到 /etc/ceph/ceph.conf
将client.admin管理（特权！）密钥的副本写入/etc/ceph/ceph.client.admin.keyring
将公钥的副本写入/etc/ceph/ceph.pub
安装完成后会有一个dashboard界面

执行完成后我们可以查看ceph.conf已经写入了

启用 CEPH CLI
cephadm shell命令在安装了所有Ceph包的容器中启动bash shell。默认情况下，如果在主机上的/etc/ceph中找到配置和keyring文件，则会将它们传递到容器环境中，以便shell完全正常工作。
cephadm shell

可以在节点上安装包含所有 ceph 命令的包，包括、（用于安装 CephFS 文件系统）等
cephadm add-repo --release octopus
cephadm install ceph-common

安装过程很慢，可以手动将源改为本地镜像：

添加主机到集群
将公钥添加到新主机
ssh-copy-id -f -i /etc/ceph/ceph.pub node2
ssh-copy-id -f -i /etc/ceph/ceph.pub node3

告诉Ceph，新节点是集群的一部分
[root@localhost ~]# ceph orch host add node2
Added host 'node2'
[root@localhost ~]# ceph orch host add node3
Added host 'node3'

添加主机会自动扩展mon和mgr节点

部署其他监视器（可选）
典型的 Ceph 群集具有三个或五个分布在不同主机的mon守护程序。如果群集中有五个或更多节点，建议部署五个mon。
当Ceph知道mon应该使用什么IP子网时，它可以随着群集的增长（或收缩）自动部署和缩放mon。默认情况下，Ceph假定其他mon使用与第一个mon的IP相同的子网。
在单个子网的情况下，如果向集群中添加主机，默认最多只会添加5个mon 如果有特定的IP子网给mon使用，可以使用CIDR格式配置该子网：

ceph config set mon public_network 10.1.2.0/24
cephadm只会在配置了特定子网IP的主机上部署mon守护程序如果要调整特定子网mon的默认数量，可以执行以下命令：
ceph orch apply mon *<number-of-monitors>*
如果要在一组特定的主机上部署mon，可以执行以下命令：
ceph orch apply mon *<host1,host2,host3,...>*
如果要查看当前主机和标签，可以执行以下命令：

[root@node1 ~]# ceph orch host ls
HOST ADDR LABELS  STATUS
node1  node1
node2  node2
node3  node3
如果要禁用自动mon部署，执行以下命令：
ceph orch apply mon --unmanaged
要在不同网络中添加mon执行以下命令：
ceph orch apply mon --unmanaged
ceph orch daemon add mon node2:192.168.8.66
ceph orch daemon add mon node2:192.168.17.0/24

如果要添加mon到多个主机，也可以用以下命令：
ceph orch apply mon "host1,host2,host3"

部署OSD
可以用以下命令显示集群中的存储设备清单
ceph orch device ls

如果满足以下所有_条件_，则存储设备被视为可用：
设备必须没有分区。
设备不得具有任何 LVM 状态。
不得安装设备。
设备不能包含文件系统。
设备不得包含 Ceph BlueStore OSD。
设备必须大于 5 GB。
Ceph 拒绝在不可用的设备上预配 OSD。为保证能成功添加osd，我刚才在每个node上新加了一块磁盘创建新 OSD 的方法有几种方法：
在未使用的设备上自动创建osd
[root@node1 ~]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
可以看到已经在三块磁盘上创建了osd

从特定主机上的特定设备创建 OSD
ceph orch daemon add osd host1:/dev/sdb
部署MDS
使用 CephFS 文件系统需要一个或多个 MDS 守护程序。如果使用新的ceph fs卷接口来创建新文件系统，则会自动创建这些文件部署元数据服务器：
ceph orch apply mds *<fs-name>* --placement="*<num-daemons>* [*<host1>* ...]"
CephFS 需要两个 Pools，cephfs-data 和 cephfs-metadata,分别存储文件数据和文件元数据
[root@node1 ~]# ceph osd pool create cephfs_data 64 64
[root@node1 ~]# ceph osd pool create cephfs_metadata 64 64
创建一个 CephFS, 名字为 cephfs
[root@node1 ~]# ceph fs new cephfs cephfs_metadata cephfs_data
[root@node1 ~]# ceph orch apply mds cephfs --placement="3 node1 node2 node3"
Scheduled mds.cephfs update...

验证至少有一个MDS已经进入active状态，默认情况下，ceph只支持一个活跃的MDS，其他的作为备用MDS
ceph fs status cephfs

部署RGW
Cephadm将radosgw部署为管理特定领域和区域的守护程序的集合，RGW是Ceph对象存储网关服务RADOS Gateway的简称，是一套基于LIBRADOS接口封装而实现的FastCGI服务，对外提供RESTful风格的对象存储数据访问和管理接口。

使用cephadm时，radosgw守护程序是通过mon配置数据库而不是通过ceph.conf或命令行配置的。如果该配置尚未就绪，则radosgw守护进程将使用默认设置启动（默认绑定到端口80）。要在node1、node2和node3上部署3个服务于myorg领域和us-east-1区域的rgw守护进程，在部署rgw守护进程之前，如果它们不存在，则自动创建提供的域和区域：
ceph orch apply rgw myorg cn-east-1 --placement="3 node1 node2 node3"
或者可以使用radosgw-admin命令手动创建区域、区域组和区域：
radosgw-admin realm create --rgw-realm=myorg --default
radosgw-admin zonegroup create --rgw-zonegroup=default --master --default
radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=cn-east-1 --master --default
radosgw-admin period update --rgw-realm=myorg --c
可以看到RGW已经创建完成

并且cephadm会自动安装Prometheus和grafana等组件，grafana默认用户密码为admin/admin，并且已经导入了Prometheus监控ceph的仪表盘

admin · 发表于 2021-7-19 10:56:11

安装ceph源

[root@controller ~]# yum install centos-release-ceph-octopus.noarch -y
[root@computer ~]# yum install centos-release-ceph-octopus.noarch -y
安装ceph组件

[root@controller ~]# yum install cephadm -y
[root@computer ~]# yum install ceph -y
computer结点安装libvirt

[root@computer ~]# yum install libvirt -y
部署ceph集群
创建集群

[root@controller ~]# mkdir -p /etc/ceph
[root@controller ~]# cd /etc/ceph/
[root@controller ceph]# cephadm boostrap --mon-ip 192.168.29.148
[root@controller ceph]# ceph status
[root@controller ceph]# cephadm install ceph-common
[root@controller ceph]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@computer
修改配置

[root@controller ceph]# ceph config set mon public_network 192.168.29.0/24
添加主机

[root@controller ceph]# ceph orch host add computer
[root@controller ceph]# ceph orch host ls
初始化集群监控

[root@controller ceph]# ceph orch host label add controller mon
[root@controller ceph]# ceph orch host label add computer mon
[root@controller ceph]# ceph orch apply mon label:mon
[root@controller ceph]# ceph orch daemon add mon computer:192.168.29.149
创建OSD

[root@controller ceph]# ceph orch daemon add osd controller:/dev/nvme0n2
[root@controller ceph]# ceph orch daemon add osd computer:/dev/nvme0n3
查看集群状态

[root@controller ceph]# ceph -s
查看集群容量

[root@controller ceph]# ceph df
创建pool

[root@controller ceph]# ceph osd pool create volumes 64
[root@controller ceph]# ceph osd pool create vms 64

#设置自启动
[root@controller ceph]# ceph osd pool application enable vms mon
[root@controller ceph]# ceph osd pool application enable volumes mon
查看mon,osd,pool状态

[root@controller ceph]# ceph mon stat
[root@controller ceph]# ceph osd status
[root@controller ceph]# ceph osd lspools
查看pool情况

[root@controller ~]# rbd ls vms
[root@controller ~]# rbd ls volumes

admin · 发表于 2021-7-19 11:34:30

RUNNING THE BOOTSTRAP COMMAND
Run the ceph bootstrap command:

cephadm bootstrap --mon-ip *<mon-ip>*
This command will:

Create a monitor and manager daemon for the new cluster on the local host.

Generate a new SSH key for the Ceph cluster and add it to the root user’s /root/.ssh/authorized_keys file.

Write a copy of the public key to /etc/ceph/ceph.pub.

Write a minimal configuration file to /etc/ceph/ceph.conf. This file is needed to communicate with the new cluster.

Write a copy of the client.admin administrative (privileged!) secret key to /etc/ceph/ceph.client.admin.keyring.

Add the _admin label to the bootstrap host. By default, any host with this label will (also) get a copy of /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring.

FURTHER INFORMATION ABOUT CEPHADM BOOTSTRAP
The default bootstrap behavior will work for most users. But if you’d like immediately to know more about cephadm bootstrap, read the list below.

Also, you can run cephadm bootstrap -h to see all of cephadm’s available options.

By default, Ceph daemons send their log output to stdout/stderr, which is picked up by the container runtime (docker or podman) and (on most systems) sent to journald. If you want Ceph to write traditional log files to /var/log/ceph/$fsid, use --log-to-file option during bootstrap.

Larger Ceph clusters perform better when (external to the Ceph cluster) public network traffic is separated from (internal to the Ceph cluster) cluster traffic. The internal cluster traffic handles replication, recovery, and heartbeats between OSD daemons. You can define the cluster network by supplying the --cluster-network option to the bootstrap subcommand. This parameter must define a subnet in CIDR notation (for example 10.90.90.0/24 or fe80::/64).

cephadm bootstrap writes to /etc/ceph the files needed to access the new cluster. This central location makes it possible for Ceph packages installed on the host (e.g., packages that give access to the cephadm command line interface) to find these files.

Daemon containers deployed with cephadm, however, do not need /etc/ceph at all. Use the --output-dir *<directory>* option to put them in a different directory (for example, .). This may help avoid conflicts with an existing Ceph configuration (cephadm or otherwise) on the same host.

You can pass any initial Ceph configuration options to the new cluster by putting them in a standard ini-style configuration file and using the --config *<config-file>* option. For example:

cat <<EOF > initial-ceph.conf
[global]
osd crush chooseleaf type = 0
EOF
./cephadm bootstrap --config initial-ceph.conf ...
The --ssh-user *<user>* option makes it possible to choose which ssh user cephadm will use to connect to hosts. The associated ssh key will be added to /home/*<user>*/.ssh/authorized_keys. The user that you designate with this option must have passwordless sudo access.

If you are using a container on an authenticated registry that requires login, you may add the three arguments:

--registry-url <url of registry>

--registry-username <username of account on registry>

--registry-password <password of account on registry>

OR

--registry-json <json file with login info>

Cephadm will attempt to log in to this registry so it can pull your container and then store the login info in its config database. Other hosts added to the cluster will then also be able to make use of the authenticated registry.

ENABLE CEPH CLI
Cephadm does not require any Ceph packages to be installed on the host. However, we recommend enabling easy access to the ceph command. There are several ways to do this:

The cephadm shell command launches a bash shell in a container with all of the Ceph packages installed. By default, if configuration and keyring files are found in /etc/ceph on the host, they are passed into the container environment so that the shell is fully functional. Note that when executed on a MON host, cephadm shell will infer the config from the MON container instead of using the default configuration. If --mount <path> is given, then the host <path> (file or directory) will appear under /mnt inside the container:

cephadm shell
To execute ceph commands, you can also run commands like this:

cephadm shell -- ceph -s
You can install the ceph-common package, which contains all of the ceph commands, including ceph, rbd, mount.ceph (for mounting CephFS file systems), etc.:

cephadm add-repo --release pacific
cephadm install ceph-common
Confirm that the ceph command is accessible with:

ceph -v
Confirm that the ceph command can connect to the cluster and also its status with:

ceph status
ADDING HOSTS
Next, add all hosts to the cluster by following Adding Hosts.

By default, a ceph.conf file and a copy of the client.admin keyring are maintained in /etc/ceph on all hosts with the _admin label, which is initially applied only to the bootstrap host. We usually recommend that one or more other hosts be given the _admin label so that the Ceph CLI (e.g., via cephadm shell) is easily accessible on multiple hosts. To add the _admin label to additional host(s),

ceph orch host label add *<host>* _admin
ADDING ADDITIONAL MONS
A typical Ceph cluster has three or five monitor daemons spread across different hosts. We recommend deploying five monitors if there are five or more nodes in your cluster.

Please follow Deploying additional monitors to deploy additional MONs.

ADDING STORAGE
To add storage to the cluster, either tell Ceph to consume any available and unused device:

ceph orch apply osd --all-available-devices
Or See Deploy OSDs for more detailed instructions.

admin · 发表于 2021-7-19 11:37:35

cat <<EOF > initial-ceph.conf
[global]
osd crush chooseleaf type = 0
EOF
./cephadm bootstrap --config initial-ceph.conf --mon-ip *<mon-ip>*

cephadm shell

admin · 发表于 2021-7-19 11:38:48

To deploy an iSCSI gateway, create a yaml file containing a service specification for iscsi:

service_type: iscsi
service_id: iscsi
placement:
  hosts:
- host1
- host2
spec:
  pool: mypool  # RADOS pool where ceph-iscsi config data is stored.
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2"
  api_port: ... # optional
  api_user: ... # optional
  api_password: ... # optional
  api_secure: true/false # optional
  ssl_cert: | # optional
...
  ssl_key: | # optional
...
For example:

service_type: iscsi
service_id: iscsi
placement:
  hosts:
  - [...]
spec:
  pool: iscsi_pool
  trusted_ip_list: "IP_ADDRESS_1,IP_ADDRESS_2,IP_ADDRESS_3,..."
  api_user: API_USERNAME
  api_password: API_PASSWORD
  api_secure: true
  ssl_cert: |
-----BEGIN CERTIFICATE-----
MIIDtTCCAp2gAwIBAgIYMC4xNzc1NDQxNjEzMzc2MjMyXzxvQ7EcMA0GCSqGSIb3
DQEBCwUAMG0xCzAJBgNVBAYTAlVTMQ0wCwYDVQQIDARVdGFoMRcwFQYDVQQHDA5T
[...]
-----END CERTIFICATE-----
  ssl_key: |
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC5jdYbjtNTAKW4
/CwQr/7wOiLGzVxChn3mmCIF3DwbL/qvTFTX2d8bDf6LjGwLYloXHscRfxszX/4h
[...]
-----END PRIVATE KEY-----
The specification can then be applied using:

ceph orch apply -i iscsi.yaml

admin · 发表于 2021-7-19 11:44:03

ceph orch apply mon --unmanaged
ceph orch apply mon --unmanaged

ceph orch daemon add mon newhost1:10.1.2.123

ceph orch daemon add mon newhost2:10.1.2.0/24

ceph orch apply mon "host1,host2,host3"

admin · 发表于 2021-7-19 13:50:11

环境介绍
IP地址配置主机名 Ceph版本
10.15.253.161 c2m8h300 cephnode01 Octopus 15.2.4
10.15.253.193 c2m8h300 cephnode02 Octopus 15.2.4
10.15.253.225 c2m8h300 cephnode03 Octopus 15.2.4

#Linux系统版本
[root@cephnode01 ~]# cat /etc/redhat-release
CentOS Linux release 8.2.2004 (Core)
[root@cephnode01 ~]# uname -r
4.18.0-193.14.2.el8_2.x86_64

#网络设计:建议各网络单独分开
10.15.253.0/24 #Public Network  公共网络
172.31.253.0/24 #Cluster Network 集群网络
#每台ceph节点下除系统盘外，最少挂载两块相同的大容量硬盘，不需要进行分区
[root@cephnode01 ~]# lsblk
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda    8:0 0 20G  0 disk
├─sda1 8:1 0  200M  0 part /boot
├─sda2 8:2 0 1G  0 part [SWAP]
└─sda3 8:3 0 18.8G  0 part /
sdb    8:16 0 20G  0 disk
2.1.1 Ceph安装与版本选择
https://docs.ceph.com/docs/master/install/
ceph-deploy 是用于快速部署群集的工具；社区不再积极维护ceph-deploy。仅支持Nautilus版之前的Ceph版本上进行。它不支持RHEL8，CentOS 8或更新的操作系统。
这里的系统环境是centos8系统，所以需要使用cephadm部署工具部署octopus版的ceph
2.1.2 基础环境准备
全部Ceph节点上操作；以cephnode01节点为例；

#（1）关闭防火墙：
systemctl stop firewalld
systemctl disable firewalld
#（2）关闭selinux：
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
#（3）在cephnode01上配置免密登录到cephnode02、cephnode03
dnf install sshpass -y
ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''
for ip in 161 193 225 ;do sshpass -pZxzn@2020 ssh-copy-id -o StrictHostKeyChecking=no 10.15.253.$ip ;done
#（4）在cephnode01上添加主机名：已经配置过则不需要再次添加
cat >>/etc/hosts <<EOF
10.15.253.161 cephnode01
10.15.253.193 cephnode02
10.15.253.225 cephnode03
EOF
for ip in 193 225 ;do scp -rp /etc/hosts root@10.15.253.$ip:/etc/hosts ;done
#（5）设置文件连接数最大值
echo "ulimit -SHn 102400" >> /etc/rc.local
cat >> /etc/security/limits.conf << EOF
* soft nofile 65535
* hard nofile 65535
EOF
#（6）内核参数优化
echo 'net.ipv4.ip_forward = 1' >>/etc/sysctl.conf
echo 'kernel.pid_max = 4194303' >>/etc/sysctl.conf
#内存不足时低于此值，使用交换空间
echo "vm.swappiness = 0" >>/etc/sysctl.conf
sysctl -p
#（7）同步网络时间和修改时区；已经添加不需要配置
安装chrony时间同步同步cephnode01节点
yum install chrony -y
vim /etc/chrony.conf
server cephnode01 iburst
---
systemctl restart chronyd.service
systemctl enable chronyd.service
chronyc sources
#(8)read_ahead,通过数据预读并且记载到随机访问内存方式提高磁盘读操作
echo "8192" > /sys/block/sda/queue/read_ahead_kb
#(9) I/O Scheduler，SSD要用noop(电梯式调度程序)，SATA/SAS使用deadline(截止时间调度程序)
#https://blog.csdn.net/shipeng1022/article/details/78604910
echo "deadline" >/sys/block/sda/queue/scheduler
echo "deadline" >/sys/block/sdb/queue/scheduler
#echo "noop" >/sys/block/sd[x]/queue/scheduler
3. 添加Octopus版yum源

cat >>/etc/yum.repos.d/ceph.repo <<EOF
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/$basearch
enabled=1
gpgcheck=0
type=rpm-md
[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/noarch
enabled=1
gpgcheck=0
type=rpm-md
[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-octopus/el8/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
EOF
yum clean all && yum makecache
#安装基础软件
yum install net-tools wget vim bash-completion lrzsz unzip zip -y
4. cephadm工具部署
https://docs.ceph.com/docs/master/cephadm/install/
在15版本，支持使用cephadm工具部署，ceph-deploy在14版本前都支持
4.1 拉取最新的cephadm并赋权
在cephnode01节点配置；

[root@cephnode01 ~]# curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
[root@cephnode01 ~]# chmod +x cephadm
[root@cephnode01 ~]#ll
-rwxr-xr-x. 1 root root 184653 Sep 10 12:01 cephadm
4.2 使用cephadm获取octopus最新版本并安装
已手动配置为国内yum源，不需要按官方文档的步骤再进行添加yum源

#全部ceph节点安装
[root@cephnode01 ~]# dnf install python3 podman -y
[root@cephnode01 ~]# ./cephadm install
...
[root@cephnode01 ~]# which cephadm
/usr/sbin/cephadm
5. 创建ceph新集群
5.1 指定管理节点
创建一个可以被任何访问Ceph集群的主机访问的网络，指定mon-ip，并将生成的配置文件写进/etc/ceph目录里

[root@cephnode01 ~]# mkdir -p /etc/ceph
[root@cephnode01 ~]# cephadm bootstrap --mon-ip 10.15.253.161
...
      URL: https://cephnode01:8443/
      User: admin
Password: 6v7xazcbwk
...
可登陆URL: https://cephnode01:8443/，首次登陆要修改密码，进行验证

5.2 将ceph命令映射到本地
Cephadm不需要在主机上安装任何Ceph包。但是，建议启用对ceph命令的简单访问。
cephadm shell命令在安装了所有Ceph包的容器中启动一个bash shell。默认情况下，如果在主机上的/etc/ceph中找到配置和keyring文件，它们将被传递到容器环境中，这样就可以完全正常工作了。

[root@cephnode01 ~]# cephadm shell
[root@cephnode01 ~]# alias ceph='cephadm shell -- ceph'
[root@cephnode01 ~]# exit
#安装ceph-common包；包括ceph，rbd，mount.ceph的命令
[root@cephnode01 ~]# ephadm install ceph-common包；包括ceph，rbd，mount.ceph的命令
#查看版本
[root@cephnode01 ~]# ceph -v
ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)
查看状态

[root@cephnode01 ~]# ceph status
  cluster:
id:    8a4fdb4e-f31c-11ea-be33-000c29358c7a
health: HEALTH_OK
Reduced data availability: 1 pg inactive
OSD count 0 < osd_pool_default_size 3

  services:
mon: 1 daemons, quorum ceph135 (age 14m)
mgr: ceph03.oesega(active, since 10m)
osd: 0 osds: 0 up (since 31m), 0 in

  data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:    100.000% pgs unknown
         1 unknown
5.3 添加新节点进ceph集群

[root@cephnode01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@cephnode02
[root@cephnode01 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub root@cephnode03
[root@cephnode01 ~]# ceph orch host add cephnode02
Added host 'cephnode02'
[root@cephnode01 ~]# ceph orch host add cephnode03
Added host 'cephnode03'
5.4 部署添加 monitor
选择需要设置mon的节点，全选

[root@cephnode01 ~]# ceph orch host label add cephnode01 mon
Added label mon to host cephnode01
[root@cephnode01 ~]# ceph orch host label add cephnode02 mon
Added label mon to host cephnode02
[root@cephnode01 ~]# ceph orch host label add cephnode03 mon
Added label mon to host cephnode03
[root@cephnode01 ~]# ceph orch host ls
HOST       ADDR       LABELS  STATUS
cephnode01  cephnode01  mon
cephnode02  cephnode02  mon
cephnode03  cephnode03  mon
告诉cephadm根据标签部署mon，这步需要等待各节点拉取images并启动容器

[root@cephnode01 ~]# ceph orch apply mon label:mon
具体验证是否安装完成，其他两台节点可查看下

[root@cephnode02 ~]# podman ps -a
...
[root@cephnode02 ~]# podman images
REPOSITORY                   TAG    IMAGE ID    CREATED       SIZE
docker.io/ceph/ceph          v15    852b28cb10de 3 weeks ago    1 GB
docker.io/prom/node-exporter v0.18.1 e5a616e4b9cf 15 months ago 24.3 MB
6. 部署OSD
6.1 查看可使用的硬盘

[root@cephnode01 ~]# ceph orch device ls
HOST PATH    TYPE SIZE  DEVICE  AVAIL  REJECT REASONS
ceph01  /dev/sda  hdd 20.0G       False  locked, Insufficient space (<5GB) on vgs, LVM detected
ceph01  /dev/sdb  hdd 20.0G       True
ceph02  /dev/sda  hdd 20.0G       False  Insufficient space (<5GB) on vgs, LVM detected, locked
ceph02  /dev/sdb  hdd 20.0G       True
ceph03  /dev/sda  hdd 20.0G       False  locked, Insufficient space (<5GB) on vgs, LVM detected
ceph03  /dev/sdb  hdd 20.0G       True
6.2 使用所有可用硬盘

[root@cephnode01 ~]# ceph orch apply osd --all-available-devices
添加单块盘的方式

[root@cephnode01 ~]# ceph orch daemon add osd cephnode02:/dev/sdc
6.3 验证部署情况

[root@cephnode01 ~]# ceph osd df
ID  CLASS  WEIGHT REWEIGHT  SIZE RAW USE  DATA    OMAP    META    AVAIL %USE  VAR PGS  STATUS
0 hdd  0.01949 1.00000  20 GiB  1.0 GiB  3.8 MiB 1 KiB  1024 MiB  19 GiB  5.02  1.00 1    up
1 hdd  0.01949 1.00000  20 GiB  1.0 GiB  3.8 MiB 1 KiB  1024 MiB  19 GiB  5.02  1.00 1    up
2 hdd  0.01949 1.00000  20 GiB  1.0 GiB  3.8 MiB 1 KiB  1024 MiB  19 GiB  5.02  1.00 1    up
                     TOTAL  60 GiB  3.0 GiB 11 MiB  4.2 KiB 3.0 GiB  57 GiB  5.02
MIN/MAX VAR: 1.00/1.00  STDDEV: 0
7. 存储部署
7.1 CephFS部署
部署cephfs的mds服务，指定集群名及mds的数量

[root@cephnode01 ~]# ceph orch apply mds fs-cluster --placement=3

[root@cephnode01 ~]# ceph -s
  cluster:
id:    8a4fdb4e-f31c-11ea-be33-000c29358c7a
health: HEALTH_OK

  services:
mon: 3 daemons, quorum cephnode01,cephnode02,cephnode03 (age 1m)
mgr: cephnode01.oesega(active, since 49m), standbys: cephnode02.lphrtb, cephnode03.wkthtb
mds:  3 up:standby
osd: 3 osds: 3 up (since 51m), 3 in (since 30m)

  data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 57 GiB / 60 GiB avail
pgs:    1 active+clean
7.2 部署RGW
创建一个领域

[root@cephnode01 ~]# radosgw-admin realm create --rgw-realm=rgw-org --default
{
"id": "43dc34c0-6b5b-411c-9e23-687a29c8bd00",
"name": "rgw-org",
"current_period": "ea3dd54c-2dfe-4180-bf11-4415be6ccafd",
"epoch": 1
}
创建一个zonegroup区域组

[root@cephnode01 ~]# radosgw-admin zonegroup create --rgw-zonegroup=rgwgroup --master --default
{
"id": "1878ecaa-216b-4c99-ad4e-b72f4fa9193f",
"name": "rgwgroup",
"api_name": "rgwgroup",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [],
"placement_targets": [],
"default_placement": "",
"realm_id": "43dc34c0-6b5b-411c-9e23-687a29c8bd00",
"sync_policy": {
      "groups": []
}
}
创建一个区域

[root@cephnode01 ~]# radosgw-admin zone create --rgw-zonegroup=rgwgroup --rgw-zone=zone-dc1 --master --default
{
"id": "fbdc5f83-9022-4675-b98e-39738920bb57",
"name": "zone-dc1",
"domain_root": "zone-dc1.rgw.meta:root",
"control_pool": "zone-dc1.rgw.control",
"gc_pool": "zone-dc1.rgw.log:gc",
"lc_pool": "zone-dc1.rgw.log:lc",
"log_pool": "zone-dc1.rgw.log",
"intent_log_pool": "zone-dc1.rgw.log:intent",
"usage_log_pool": "zone-dc1.rgw.log:usage",
"roles_pool": "zone-dc1.rgw.meta:roles",
"reshard_pool": "zone-dc1.rgw.log:reshard",
"user_keys_pool": "zone-dc1.rgw.meta:users.keys",
"user_email_pool": "zone-dc1.rgw.meta:users.email",
"user_swift_pool": "zone-dc1.rgw.meta:users.swift",
"user_uid_pool": "zone-dc1.rgw.meta:users.uid",
"otp_pool": "zone-dc1.rgw.otp",
"system_key": {
      "access_key": "",
      "secret_key": ""
},
"placement_pools": [
      {
         "key": "default-placement",
         "val": {
            "index_pool": "zone-dc1.rgw.buckets.index",
            "storage_classes": {
                  "STANDARD": {
                     "data_pool": "zone-dc1.rgw.buckets.data"
                  }
            },
            "data_extra_pool": "zone-dc1.rgw.buckets.non-ec",
            "index_type": 0
         }
      }
],
"realm_id": "43dc34c0-6b5b-411c-9e23-687a29c8bd00"
}
为特定领域和区域部署一组radosgw守护进程，这里只指定了两个节点开启rgw
1
[root@cephnode01 ~]# ceph orch apply rgw rgw-org zone-dc1 --placement="2 cephnode02 cephnode03"
验证

[root@cephnode01 ~]# ceph -s
  cluster:
id:    8a4fdb4e-f31c-11ea-be33-000c29358c7a
health: HEALTH_OK

  services:
mon: 3 daemons, quorum cephnode01,cephnode02,cephnode03 (age 1m)
mgr: cephnode01.oesega(active, since 49m), standbys: cephnode02.lphrtb, cephnode03.wkthtb
mds:  3 up:standby
osd: 3 osds: 3 up (since 51m), 3 in (since 30m)
rgw: 2 daemons active (rgw-org.zone-dc1.cephnode02.cdgjsi, rgw-org.zone-dc1.cephnode03.nmbbsz)
  data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 57 GiB / 60 GiB avail
pgs:    1 active+clean
为RGW开启dashborad

#创建rgw的管理用户
[root@cephnode01 ~]# radosgw-admin user create --uid=admin --display-name=admin --system
{
"user_id": "admin",
"display_name": "admin",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [
      {
         "user": "admin",
         "access_key": "WG9W5O9O11TGGOLU6OD2",
         "secret_key": "h2DfrWvlS4NMkdgGin4g6OB6Z50F1VNmhRCRQo3W"
      }
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"system": "true",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"user_quota": {
      "enabled": false,
      "check_on_raw": false,
      "max_size": -1,
      "max_size_kb": 0,
      "max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}
设置dashboard凭证

[root@cephnode01 ~]# ceph dashboard set-rgw-api-access-key WG9W5O9O11TGGOLU6OD2
Option RGW_API_ACCESS_KEY updated
[root@cephnode01 ~]# ceph dashboard set-rgw-api-secret-key h2DfrWvlS4NMkdgGin4g6OB6Z50F1VNmhRCRQo3W
Option RGW_API_SECRET_KEY updated
设置禁用证书验证、http访问方式及使用admin账号

ceph dashboard set-rgw-api-ssl-verify False
ceph dashboard set-rgw-api-scheme http
ceph dashboard set-rgw-api-host 10.15.253.225
ceph dashboard set-rgw-api-port 80
ceph dashboard set-rgw-api-user-id admin
重启RGW

ceph orch restart rgw

帐号		自动登录	找回密码
密码			开始注册

新版使用cephadm安装ceph octopus

马上注册，结交更多好友，享用更多功能，让你轻松玩转社区。

站长推荐 /4