手动方式部署ceph集群以及添加osd

admin

一、准备

前一篇点击打开链接只部署了一个单机集群。在这一篇里，手动部署一个多机集群：mycluster。我们有三台机器nod1,node2和node3；其中node1可以免密ssh/scp任意其他两台机器。我们的所有工作都在node1上完成。

准备工作包括在各个机器上安装ceph rpm包（见前一篇第1节点击打开链接），并在各个机器上修改下列文件：

/usr/lib/systemd/system/ceph-mon@.service /usr/lib/systemd/system/ceph-osd@.service /usr/lib/systemd/system/ceph-mds@.service /usr/lib/systemd/system/ceph-mgr@.service /usr/lib/systemd/system/ceph-radosgw@.service

修改：

Environment=CLUSTER=ceph <--- 改成CLUSTER=mycluster ExecStart=/usr/bin/... --id %i --setuser ceph --setgroup ceph <--- 删掉--setuser ceph --setgroup ceph二、创建工作目录

在node1创建一个工作目录，后续所有工作都在node1上的这个工作目录中完成；

mkdir /tmp/mk-ceph-cluster cd /tmp/mk-ceph-cluster三、创建配置文件vim mycluster.conf [global] cluster = mycluster fsid = 116d4de8-fd14-491f-811f-c1bdd8fac141 public network = 192.168.100.0/24 cluster network = 192.168.73.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd pool default crush rule = 0 osd crush chooseleaf type = 1 admin socket = /var/run/ceph/$cluster-$name.asock pid file = /var/run/ceph/$cluster-$name.pid log file = /var/log/ceph/$cluster-$name.log log to syslog = false max open files = 131072 ms bind ipv6 = false [mon] mon initial members = node1,node2,node3 mon host = 192.168.100.131:6789,192.168.100.132:6789,192.168.100.133:6789 ;Yuanguo: the default value of {mon data} is /var/lib/ceph/mon/$cluster-$id, ; we overwrite it. mon data = /var/lib/ceph/mon/$cluster-$name mon clock drift allowed = 10 mon clock drift warn backoff = 30 mon osd full ratio = .95 mon osd nearfull ratio = .85 mon osd down out interval = 600 mon osd report timeout = 300 debug ms = 20 debug mon = 20 debug paxos = 20 debug auth = 20 [mon.node1] host = node1 mon addr = 192.168.100.131:6789 [mon.node2] host = node2 mon addr = 192.168.100.132:6789 [mon.node3] host = node3 mon addr = 192.168.100.133:6789 [mgr] ;Yuanguo: the default value of {mgr data} is /var/lib/ceph/mgr/$cluster-$id, ; we overwrite it. mgr data = /var/lib/ceph/mgr/$cluster-$name [osd] ;Yuanguo: we wish to overwrite {osd data}, but it seems that 'ceph-disk' forces ; to use the default value, so keep the default now; maybe in later versions ; of ceph the limitation will be eliminated. osd data = /var/lib/ceph/osd/$cluster-$id osd recovery max active = 3 osd max backfills = 5 osd max scrubs = 2 osd mkfs type = xfs osd mkfs options xfs = -f -i size=1024 osd mount options xfs = rw,noatime,inode64,logbsize=256k,delaylog filestore max sync interval = 5 osd op threads = 2 debug ms = 100 debug osd = 100

需要说明的是，在这个配置文件中，我们覆盖了一些默认值，比如：{mon data}和{mgr data}，但是没有覆盖{osd data}，因为ceph-disk貌似强制使用默认值。另外，pid, sock文件被放置在/var/run/ceph/中，以$cluster-$name命名；log文件放置在/var/log/ceph/中，也是以$cluster-$name命名。这些都可以覆盖。

四、生成keyring

在单机部署中点击打开链接，我们说过，有两种操作集群中user及其权限的方式，这里我们使用第一种：先生成keyring文件，然后在创建集群时带入使之生效。

ceph-authtool --create-keyring mycluster.keyring --gen-key -n mon. --cap mon 'allow *'
0 ^; d0 J% t; I2 x. uceph-authtool --create-keyring mycluster.client.admin.keyring --gen-key -n client.admin --set-uallow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
ceph-authtool --create-keyring mycluster.client.bootstrap-osd.keyring --gen-key -n client.bootstrap-osd --cap mon 'allow profile bootstrap-osd'
0 O: T; [5 e$ A0 D1 n9 Cceph-authtool --create-keyring mycluster.mgr.node1.keyring --gen-key -n mgr.node1 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
1 g  B' A% @+ _, oceph-authtool --create-keyring mycluster.mgr.node2.keyring --gen-key -n mgr.node2 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
7 w; j9 \  W" q, W! Aceph-authtool --create-keyring mycluster.mgr.node3.keyring --gen-key -n mgr.node3 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph-authtool mycluster.keyring --import-keyring mycluster.client.admin.keyring
ceph-authtool mycluster.keyring --import-keyring mycluster.client.bootstrap-osd.keyring
ceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node1.keyring
# u( M+ S" Q4 |: T  I) Bceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node2.keyring
6 j# [  b8 ~- S# f6 C% ]9 eceph-authtool mycluster.keyring --import-keyring mycluster.mgr.node3.keyring
# }8 p7 `2 o) M2 hcat mycluster.keyring [mon.] key = AQA525NZsY73ERAAIM1J6wSxglBNma3XAdEcVg== caps mon = "allow *" [client.admin] key = AQBJ25NZznIpEBAAlCdCy+OyUIvxtNq+1DSLqg== auid = 0 caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [client.bootstrap-osd] key = AQBW25NZtl/RBxAACGWafYy1gPWEmx9geCLi6w== caps mon = "allow profile bootstrap-osd" [mgr.node1] key = AQBb25NZ1mIeFhAA/PmRHFY6OgnAMXL1/8pSxw== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node2] key = AQBg25NZJ6jyHxAAf2GfBAG5tuNwf9YjkhhEWA== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *" [mgr.node3] key = AQBl25NZ7h6CJRAAaFiea7hiTrQNVoZysA7n/g== caps mds = "allow *" caps mon = "allow profile mgr" caps osd = "allow *"五、生成monmap

生成monmap并添加3个monitor

monmaptool --create --add node1 192.168.100.131:6789 --add node2 192.168.100.132:6789 --add node3 192.168.100.133:6789 --fsid 116d4de8-fd14-491f-811f-c1bdd8fac141
monmap [plain] view plain copy monmaptool --print monmap monmaptool: monmap file monmap epoch 0 fsid 116d4de8-fd14-491f-811f-c1bdd8fac141 last_changed 2017-08-16 05:45:37.851899 created 2017-08-16 05:45:37.851899 0: 192.168.100.131:6789/0 mon.node1 1: 192.168.100.132:6789/0 mon.node2 2: 192.168.100.133:6789/0 mon.node3六、分发配置文件

keyring和monmap

把第2、3和4步中生成的配置文件，keyring，monmap分发到各个机器。由于mycluster.mgr.nodeX.keyring暂时使用不到，先不分发它们(见第8节)。

cp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap /etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node2:/etc/ceph scp mycluster.client.admin.keyring mycluster.client.bootstrap-osd.keyring mycluster.keyring mycluster.conf monmap node3:/etc/ceph七、创建集群1、创建{mon data}目录mkdir /var/lib/ceph/mon/mycluster-mon.node1
ssh node2 mkdir /var/lib/ceph/mon/mycluster-mon.node2
  |+ t+ j' `: ^6 W# j) [ssh node3 mkdir /var/lib/ceph/mon/mycluster-mon.node3

注意，在配置文件mycluster.conf中，我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name，而不是默认的/var/lib/ceph/mon/$cluster-$id；
$cluster-$name展开为mycluster-mon.node1(23)；
; Q; Y/ Z2 W8 I4 s1 V% v2 D8 p# H默认的$cluster-$id展开为mycluster-node1(23)；

2、初始化monitorceph-mon --cluster mycluster --mkfs -i node1 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
ssh node2 ceph-mon --cluster mycluster --mkfs -i node2 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
ssh node3 ceph-mon --cluster mycluster --mkfs -i node3 --monmap /etc/ceph/monmap --keyring /etc/ceph/mycluster.keyring
- V+ c$ j# }1 J" a8 G

注意，在配置文件mycluster.conf，我们把{mon data}设置为/var/lib/ceph/mon/$cluster-$name，展开为/var/lib/ceph/mon/mycluster-mon.node1(23)。ceph-mon会
+ G+ K) q! e3 Z. |1 L% J根据–cluster mycluster找到配置文件mycluster.conf，并解析出{mon data}，然后在那个目录下进行初始化。

3、touch donetouch /var/lib/ceph/mon/mycluster-mon.node1/done
ssh node2 touch /var/lib/ceph/mon/mycluster-mon.node2/done
ssh node3 touch /var/lib/ceph/mon/mycluster-mon.node3/done4、启动monitorssystemctl start ceph-mon@node1
ssh node2 systemctl start [url=mailto:ceph-mon@node2]ceph-mon@node2
1 g: @& N$ g; {, B/ }3 y7 A& Cssh node3 systemctl start ceph-mon@node3[/url]
! m: M( ^' U4 p- O1 m; @5、检查机器状态ceph --cluster mycluster -s
/ B4 Q2 y- i7 Xcluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:八、添加osd

每台集群都有一个/dev/sdb，我们把它们作为osd。

1、删除它们的分区2、prepareceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node2 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 --bluestore --block.db /dev/sdb --block.wal /dev/sdb /dev/sdb ssh node3 ceph-disk prepare --cluster mycluster --cluster-uuid 116d4de8-fd14-491f-811f-c1bdd8fac141 /dev/sdb 注意：prepare node3:/dev/sdb时，我们没有加选项：--bluestore --block.db /dev/sdb --block.wal /dev/sdb；后面我们会看它和其他两个有什么不同。3、activateceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node2 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring ssh node3 ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring

注意：ceph-disk好像有两个问题：

• 前面说过，它不使用自定义的{osd data}，而强制使用默认值 /var/lib/ceph/osd/$cluster-$id
  ' J2 Q1 a% B+ D' K  s" Q0 ~/ U
• 好像不能为一个磁盘指定osd id，而只能依赖它自动生成。虽然ceph-disk prepare有一个选项–osd-id，但是ceph-disk activate并不使用它而是自己生成。当不匹配时，会出现 如下错误：
  
  ' p' m0 m9 [6 y& k+ C) ^5 [

# ceph-disk activate /dev/sdb1 --activate-key /etc/ceph/mycluster.client.bootstrap-osd.keyring command_with_stdin: Error EEXIST: entity osd.0 exists but key does not match mount_activate: Failed to activate '['ceph', '--cluster', 'mycluster', '--name', 'client.bootstrap-osd', '--keyring', '/etc/ceph/mycluster.client.bootstrap-osd.keyring', '-i', '-', 'osd', 'new', u'ca8aac6a-b442-4b07-8fa6-62ac93b7cd29']' failed with status code 17

从 ‘-i’, ‘-‘可以看出，它只能自动生成osd id；

4、检查osd

在ceph-disk prepare时，node1:/dev/sdb和node2:/dev/sdb一样，都有–bluestore –block.db /dev/sdb –block.wal选项；node3:/dev/sdb不同，没有加这些选项。我们看看有什么不同。

4.1 node1

mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-0 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-0/ activate.monmap block block.db_uuid block.wal bluefs fsid kv_backend mkfs_done systemd whoami active block.db block_uuid block.wal_uuid ceph_fsid keyring magic ready type ls -l /var/lib/ceph/osd/mycluster-0/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block -> /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/a12dd642-b64c-4fef-b9e6-0b45cff40fa9 -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-0/block_uuid a12dd642-b64c-4fef-b9e6-0b45cff40fa9 ls -l /var/lib/ceph/osd/mycluster-0/block.db lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.db -> /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/1c107775-45e6-4b79-8a2f-1592f5cb03f2 -> ../../sdb3 blkid /dev/sdb3 /dev/sdb3: PARTLABEL="ceph block.db" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.db_uuid 1c107775-45e6-4b79-8a2f-1592f5cb03f2 ls -l /var/lib/ceph/osd/mycluster-0/block.wal lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:52 /var/lib/ceph/osd/mycluster-0/block.wal -> /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f ls -l /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f lrwxrwxrwx. 1 root root 10 Aug 16 05:55 /dev/disk/by-partuuid/76055101-b892-4da9-b80a-c1920f24183f -> ../../sdb4 blkid /dev/sdb4 /dev/sdb4: PARTLABEL="ceph block.wal" PARTUU cat /var/lib/ceph/osd/mycluster-0/block.wal_uuid 76055101-b892-4da9-b80a-c1920f24183f

可见，node1(node2)上，/dev/sdb被分为4个分区：

• /dev/sdb1: metadata
• /dev/sdb2：the main block device
• /dev/sdb3: db
• /dev/sdb4: wal
  

具体见：ceph-disk prepare –help

4.2 node3

mount | grep sdb /dev/sdb1 on /var/lib/ceph/osd/mycluster-2 type xfs (rw,noatime,seclabel,attr2,inode64,noquota) ls /var/lib/ceph/osd/mycluster-2 activate.monmap active block block_uuid bluefs ceph_fsid fsid keyring kv_backend magic mkfs_done ready systemd type whoami ls -l /var/lib/ceph/osd/mycluster-2/block lrwxrwxrwx. 1 ceph ceph 58 Aug 16 05:54 /var/lib/ceph/osd/mycluster-2/block -> /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb ls -l /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb lrwxrwxrwx. 1 root root 10 Aug 16 05:56 /dev/disk/by-partuuid/0a70b661-43f5-4562-83e0-cbe6bdbd31fb -> ../../sdb2 blkid /dev/sdb2 /dev/sdb2: PARTLABEL="ceph block" PARTUU cat /var/lib/ceph/osd/mycluster-2/block_uuid 0a70b661-43f5-4562-83e0-cbe6bdbd31fb

可见，在node3上，/dev/sdb被分为2个分区：

• /dev/sdb1：metadata
• /dev/sdb2：the main block device；db和wal也在这个分区上。
  

具体见：ceph-disk prepare –help

5、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum node1,node2,node3 mgr: no daemons active osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 0 kB used, 0 kB / 0 kB avail pgs:

由于没有添加mgr，集群处于WARN状态。

九、添加mgr1、创建{mgr data}目录mkdir /var/lib/ceph/mgr/mycluster-mgr.node1 ssh node2 mkdir /var/lib/ceph/mgr/mycluster-mgr.node2 ssh node3 mkdir /var/lib/ceph/mgr/mycluster-mgr.node3

注意，和{mon data}类似，在配置文件mycluster.conf中，我们把{mgr data}设置为/var/lib/ceph/mgr/$cluster-$name，而不是默认的/var/lib/ceph/mgr/$cluster-$id。

2、分发mgr的keyringcp mycluster.mgr.node1.keyring /var/lib/ceph/mgr/mycluster-mgr.node1/keyring scp mycluster.mgr.node2.keyring node2:/var/lib/ceph/mgr/mycluster-mgr.node2/keyring scp mycluster.mgr.node3.keyring node3:/var/lib/ceph/mgr/mycluster-mgr.node3/keyring3、启动mgrsystemctl start ceph-mgr@node1 ssh node2 systemctl start ceph-mgr@node2 ssh node3 systemctl start ceph-mgr@node34、检查集群状态ceph --cluster mycluster -s cluster: id: 116d4de8-fd14-491f-811f-c1bdd8fac141 health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 mgr: node1(active), standbys: node3, node2 osd: 3 osds: 3 up, 3 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 bytes usage: 5158 MB used, 113 GB / 118 GB avail pgs:

可见，添加mgr之后，集群处于OK状态。

admin

部署Ceph mon服务
3 }8 O% p# i) |, a- d% Z安装Ceph-mon服务程序（所有设备执行）
yum install -y ceph-mon
1
初始化Mon服务（Ceph 01执行）
; P" U9 V5 A, X- i" Y- \) y0 X生成uuid
uuidgen
. V4 R# n1 |# s, T> 9bf24809-220b-4910-b384-c1f06ea80728
/ c% z  f, N5 x3 \: Y/ b1
1 N; U" @' @( Q. t6 \7 y8 }0 s2
8 M2 |5 T9 g' ?创建Ceph配置文件
cat >> /etc/ceph/ceph.conf <<EOF
[global]
fsid = 9bf24809-220b-4910-b384-c1f06ea80728
7 v) ^2 e3 h5 I% W/ u( Q: {mon_initial_members = ceph01,ceph02,ceph03
$ |9 F& o: `) W3 L, umon_host = 10.40.65.156,10.40.65.175,10.40.65.129
public_network = 10.40.65.0/24
auth_cluster_required = cephx
5 o4 a3 d' E0 i0 U) S9 Bauth_service_required = cephx
auth_client_required = cephx
2 h; T8 K- P# `9 J, x7 Fosd_journal_size = 1024
osd_pool_default_size = 3
0 q% x; `  V0 I3 s( |' i8 f2 uosd_pool_default_min_size = 2
# ^% [( @; q. g7 s2 {8 e( nosd_pool_default_pg_num = 64
osd_pool_default_pgp_num = 64
osd_crush_chooseleaf_type = 1
EOF
1
2
6 u5 h) G: @6 @# f# c3
" V) S8 l( k" V8 a8 A) `4
6 W8 ]: ^. Q- X6 h5
6
7
8
9
10
- B# `3 A0 u" M11
# @; G" _. |1 d5 y1 Y% W' S  }12
13
* z: H: l" S6 C" B14
! e& e4 T0 F, Q- ?15
16
创建集群Monitor密钥。
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
) Y" \$ W) u2 Z( i6 `1
创建client.admin用户、client.bootstrap-osd用户密钥，添加到集群密钥中。
9 ^5 R5 _/ q. _0 Vceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
) w$ u" A1 P3 ^; a+ Kceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
1
7 Y; w! E4 w- Y2
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
1
2
; i1 ~0 c& j3 m9 W: r使用主机名、主机IP地址、FSID生成monitor map。
monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap
1
初始化并启动monitor服务
  n# {; `. y9 W. ?sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
$ Z3 K) p6 X5 a8 jchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
& S3 f( V% g  Z1 ksudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph01/
, H" n$ n1 |1 Y4 o8 ~1
2
! ?; B2 {+ c" L2 g; `3
4
systemctl start ceph-mon@ceph01
2 g$ g8 d, m" W' r/ V$ \2 isystemctl enable ceph-mon@ceph01
( t) Y/ }  H# V- ~( Ksystemctl status ceph-mon@ceph01
1 f+ P/ ?2 w  U* \7 v1
2
- \1 i7 A& @2 i% Z$ a( T1 X( K( l3
同步配置文件、密钥、monmap到其他节点中（Ceph 01执行）
; k+ F* V* L$ R1 T" m+ F复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
scp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
1
% P* S1 g* B3 G. v' E2
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
scp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
3 ^% V, Z/ S. p6 {8 G  \4 Z: `" I1
6 ~% b& }9 L& G2
scp /tmp/ceph.mon.keyring root@ceph02:/tmp/
1 g3 S  x7 h1 e: U( iscp /tmp/ceph.mon.keyring root@ceph03:/tmp/
$ o  e+ @6 Q7 {0 R8 i, X, o  A1
3 T% Q' r2 U) F: j8 `8 X2
9 K0 z7 ~4 x2 B7 G4 sscp /tmp/monmap root@ceph02:/tmp/
scp /tmp/monmap root@ceph03:/tmp/
1
2
  {, s9 ?" H/ \+ L% a2 [$ A7 Gscp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
$ q( [# k; |3 s- ?scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
1
% N% {& f8 E2 J& J2
0 S0 M+ y2 j3 ^, v! _启动其他节点的monitor服务（Ceph 02执行）
2 x* F- Z8 }/ U# Csudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph02/
% ^* Y, C$ l1 t* `( Y5 N4 c1
2
* }9 F- V) V0 h/ h' W4 r3
: B8 R4 r, D9 u' H4
systemctl start ceph-mon@ceph02
: N5 S, G( m6 k6 R0 }* usystemctl enable ceph-mon@ceph02
4 m7 U# `: J& v  ?6 |0 nsystemctl status ceph-mon@ceph02
1
6 |1 ?( e: E9 k/ f2
3
启动其他节点的monitor服务（Ceph 03执行）
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
4 J  O  I2 t1 ^! b( Tsudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph03/
1
& ?8 T1 G3 ^  L! f7 u2
* S; x6 ?1 w- y& d3
4
8 F+ G  f* n' |2 m0 s7 J, Esystemctl start ceph-mon@ceph03
systemctl enable ceph-mon@ceph03
) l6 \- B( ~6 z. l1 `& f! E- Ssystemctl status ceph-mon@ceph03
1 J  ]2 c6 i8 _- z3 }, N6 |1
4 c1 _; I6 m7 `& W2
3
查看当前集群状态（任意节点执行）
通过ceph -s命令查询集群状态，可见集群services中3个mon服务已启动。
ceph -s
> cluster:
! Z- [& u+ `4 ]0 T2 p9 z>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  
  B4 I+ L6 _% v: v>   health: HEALTH_OK
>
$ B3 N  Q  q! v$ G. b0 R> services:  
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
>   mgr:   
>   osd:
/ h8 D4 F3 Q- e: S& p4 e5 ?>
> data:
>    pools:   
>   objects:
% I; ~! w  m: L# L. p, u2 u>   usage:   
>   pgs:
1
2
3
4
5
6 t, X" ?" b9 x! m0 k$ y( N6
7
3 b! T' X1 Y+ N8 N+ _8
9
10
11
12
  G+ R8 ^& c& ]0 q* k13
. w" ~+ V# m! L" V. C8 R14
15
部署Ceph mon服务（ceph-volume 自动化创建）
安装Ceph-osd服务程序（所有设备执行）
$ r  h% [5 q* O( B* c' Kyum install -y ceph-osd
1
9 {4 o  x' K8 ?, h! N初始化osd服务（所有设备执行）
6 t( c# x4 r5 ^" M; T' f通过fdisk等工具查看磁盘盘符，然后利用ceph-volume工具自动化创建osd服务。
( l8 j# p# r4 K" R9 h3 l5 `7 H6 _( ?4 Eceph-volume lvm create --data /dev/sda
ceph-volume lvm create --data /dev/sdb
ceph-volume lvm create --data /dev/sdc
5 A* ^  }+ i7 s* y1 ^+ g1
2
2 {  o% P9 U8 Y- e. u6 z9 |3
5 g1 }, a: _. v, y' ?7 X# W查看当前集群状态（任意节点执行）
) N: {, S$ y1 p: e, m& y& g通过ceph osd tree命令查询集群状态，可见集群services中所有osd服务已启动。
" R4 u3 o/ ?" f9 J1 Q) L# Gceph osd tree
+ `- G/ O  }5 c0 ]: R> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF
) ]  ]9 J# m/ d+ g> -1       16.36908 root default
3 ]7 L9 C( L) w1 g1 |> -3        5.45636     host ceph01
; \/ Q0 z9 m9 X8 G+ C/ Q! Q+ w8 Y>  0   hdd  1.81879         osd.0       up  1.00000 1.00000
- b" p/ @. X  k- L3 o+ A>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
>  2   hdd  1.81879         osd.2       up  1.00000 1.00000
+ e5 i$ @( S) \3 E3 S. Y> -5        5.45636     host ceph02
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
  m* }# f& B7 P3 L) \. ^& d$ c+ c>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
2 d  L3 ]/ L: U, \) y3 }! K>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
: w$ l' b  p) N$ B  S% k> -7        5.45636     host ceph03
% f. y; v# \' c>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
: o' t$ B- d5 ~% t+ g>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
1
2
2 a% y7 N9 a) v3
4 W% Z) z4 F# Y) Y: Z! Y4
5
6
7
) j8 ]1 X; ~1 |% I# E, q; f( ]8
/ W& X8 D$ [! T& k+ q1 o. o9
: k6 p4 a5 `7 n* W0 K/ J" c% W: g- n10
$ `4 v- {$ T# ~+ W' q1 e* n0 Q  B11
12
& T  _9 y- U- _) @  t4 a7 E# P) Q2 ?13
14
15
+ N. f  Z: i8 }: O  A6 N2 D部署Ceph mgr服务并开启Dashboard
安装Ceph-mgr服务程序（所有设备执行）
" P) J: T* ]) `$ ~$ f; j; Byum install -y ceph-mgr
9 [4 \9 K9 D3 N) R( s6 z- h! O% p1
; J  E2 C2 |& ?$ Z, ]- I' d初始化并启动主MGR服务（Ceph01执行）
; f9 [% R- J$ _mkdir -p /var/lib/ceph/mgr/ceph-ceph01
4 L: P5 T& g7 G1 l$ ]# T- c/ Kchown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
ceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
/ U( _* U; u6 {$ p0 e  \. O1
2
3
  y( a2 t4 r# b: ]$ _2 f+ Q4
5
systemctl start ceph-mgr@ceph01
$ @5 u$ K  J7 r, n8 _! ^$ Nsystemctl enable ceph-mgr@ceph01
! g% _, w6 F! J7 @systemctl status ceph-mgr@ceph01
1
2
8 v) }, u, l$ z5 Y9 _3
初始化并启动从MGR服务（Ceph02执行）
0 N  Y' o  v2 jmkdir -p /var/lib/ceph/mgr/ceph-ceph02
chown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
ceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring
$ X: I$ T4 m2 P1 P2 j' A, v" B1
- |& ?1 ], k: e* Q% }& m9 @2
3
, b4 O0 E9 V" \5 w4
5
systemctl start ceph-mgr@ceph02
systemctl enable ceph-mgr@ceph02
$ h" v9 P+ Y3 C0 y; l  j/ Xsystemctl status ceph-mgr@ceph02
1
! F$ F. a# M% @. f9 [1 ?5 ^2
3
初始化并启动从MGR服务（Ceph03执行）
. D# {9 n& H4 K% ~mkdir -p /var/lib/ceph/mgr/ceph-ceph03
chown ceph.ceph -R /var/lib/ceph
. N  ?7 z8 _# _  H+ _" T$ Pceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
0 ~# O0 k# W/ r( _ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring
9 ~8 h: J! w: |/ w% \ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring
4 B3 R' }. c4 S; Z5 y( F+ A9 A1
5 ~9 v9 E5 P/ [: @7 @) k2
& I7 e2 I; `9 T* @5 a. c3
4
5
8 P' i8 ^" |, M: @$ v: h* esystemctl start ceph-mgr@ceph03
systemctl enable ceph-mgr@ceph03
systemctl status ceph-mgr@ceph03
1
3 Z7 l+ w; K' a' Y$ d8 q9 C6 C2
3
' N( W" b, |4 _, y0 f1 |0 m查看当前集群状态（任意节点执行）
$ P& H: [% v6 i; A通过ceph -s命令查询集群状态，可见集群services中3个mgr服务已启动。
/ N7 [' H" b* w; kceph -s
> cluster:
>   id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a  
" M( C7 D% u; }. b6 A6 [4 J( g>   health: HEALTH_OK
>
' z/ y! n  E& I1 L' ^5 j> services:  
>   mon:  3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)  
! B/ [, n/ i! Q  R. i! k( n$ `>   mgr: ceph01(active, since 3w), standbys: ceph01、ceph02   
>   osd:
. Y! S& l" H4 K>
> data:
>    pools:   
>   objects:
>   usage:   
>   pgs:
1
* h1 S0 `6 s9 G; `) R' G& n4 ~# m2
3
0 W, q. _+ G" }0 K4
% N' a  Q4 i5 `0 a3 ]% Z) R9 i5
6
* W, Q! t# H7 u7
, A" t  k' H2 X5 D" w8
9
. ]( f9 }/ B4 w8 y+ L10
11
12
13
: w$ o( p1 y/ }1 p4 h14
15
使能Dashboard访问功能（任意节点执行）
开启mgr dashboard功能
ceph mgr module enable dashboard
1
4 b0 U/ n2 W9 ^3 Z; \; J生成并安装自签名的证书
ceph dashboard create-self-signed-cert
. Q; O7 v2 m! O0 D1
配置dashboard
ceph config set mgr mgr/dashboard/server_addr 10.40.65.148
ceph config set mgr mgr/dashboard/server_port 8080
: r  Y3 p5 d8 B' T1 L  L& Wceph config set mgr mgr/dashboard/ssl_server_port 8443
/ U/ S5 Y' U2 G/ H; b: `1
& s# b/ ^! |7 ]* p2
3
! }1 p# C. O; [% k创建一个dashboard登录用户名密码
echo '123456' > password.txt
9 }9 Z3 y& u6 O/ K$ P! fceph dashboard ac-user-create admin  administrator -i password.txt
1
5 V# T$ D6 o; U/ C: U7 P& w7 Z$ I; Q2
查看服务访问方式
& P: X, U1 a; lceph mgr services
6 S2 Q+ S/ i: @! ?) c1
通过web访问Ceph Dashboard，用户名密码为admin/123456
https://10.40.65.148:8443
- W/ E9 ~0 \3 f2 w6 b! ^; Y' J5 R% t8 H

admin

部署Ceph mon服务
5 x1 M1 F6 y( h( ~2 M2 }  d9 c安装Ceph-mon服务程序（所有设备执行）
' s4 E) a: e" @
( h1 {  t) S) H& G: E  p& ]4 vyum install -y ceph-mon
% d4 ^' c, j9 k/ l9 x0 W- F$ p1
( }; _& \+ N0 V$ o初始化Mon服务（Ceph 01执行）

* Y: V  S! |; p6 K2 \; d/ `4 j9 V4 F3 N生成uuid

* O4 f% u8 m4 B1 yuuidgen
. F# {* D( l% l& p# _& Z) `> 9bf24809-220b-4910-b384-c1f06ea80728
9 \7 \) C6 q" z+ ^7 u1
2
+ d6 b( S/ d$ H( l* ?& b, g创建Ceph配置文件
. f  F5 J5 i8 T; D+ Z; ^
# _5 ~& M( @% L4 t6 H" B- q& l3 Ccat >> /etc/ceph/ceph.conf <<EOF
[global]
3 _, A6 ~5 h( kfsid = 9bf24809-220b-4910-b384-c1f06ea80728
! r. i) b* k9 P6 T6 n& I- E2 y. Pmon_initial_members = ceph01,ceph02,ceph03
6 d, g  p6 A: H8 w- Qmon_host = 10.40.65.156,10.40.65.175,10.40.65.129
public_network = 10.40.65.0/24
1 A1 ?# h' N6 B7 M, Nauth_cluster_required = cephx
' I) [' V/ q5 f/ y0 nauth_service_required = cephx
+ k' K1 k: r  T- U$ dauth_client_required = cephx
osd_journal_size = 1024
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 64
* o5 C8 u- u. E0 E8 `1 p: qosd_pool_default_pgp_num = 64
1 M# L- c2 x  X5 G' s" Zosd_crush_chooseleaf_type = 1
, m* v7 d% F( U2 e' @. y4 W+ k. t; bEOF
1
2
3
( h5 J4 C- ]4 J; H4
5
; Q6 W. l( c( q7 x! z, |6
5 L1 `' R+ \+ q& h, ?% a7
8
) [/ R" u. X8 ?% }9 r$ R! I1 j9
10
11
$ m  P8 L/ u6 G: e7 g12
0 j0 Y) q# m& s# B- g: o# G! o, Y13
5 ?$ g/ y- M& Y% g4 {, K# R14
' B; f3 h: ]; `4 ?- j+ [15
16
! ^) b, U0 O, Q/ A! ]. z* D( j4 z创建集群Monitor密钥。

ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
1
8 N! ^: p# u. R3 D$ d/ ^创建client.admin用户、client.bootstrap-osd用户密钥，添加到集群密钥中。
* G& M  b  y+ U4 y# i
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
( Y# `! Y5 {. \7 Mceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
4 Y1 K: m0 F2 V1
% W$ M7 A! O  J" }# z; o2
" A2 j. J# W7 {ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
! M5 l9 M" J$ f6 l# l1 |. Hceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
: Z1 j: p& b# e2 }% x+ z9 ~1
3 B9 N" ?8 {5 x$ i2
3 a0 v$ [& \( i# z使用主机名、主机IP地址、FSID生成monitor map。

monmaptool --create --add ceph01 10.40.65.156 --add ceph02 10.40.65.175 --add ceph03 10.40.65.129 --fsid 9bf24809-220b-4910-b384-c1f06ea80728 /tmp/monmap
+ ~$ l+ Y' t: a  @1
初始化并启动monitor服务

sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph01
) m  ]9 M, \) }* Dchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i ceph01 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
, i* a6 r0 Q( {. ]8 Gls /var/lib/ceph/mon/ceph-ceph01/
4 H7 v0 _. D6 s, ]5 k4 C  Z1
2
$ @, B& {. D- S: L2 \* y3
4
0 ^& s: k8 O. U! g$ Esystemctl start ceph-mon@ceph01
" c" V0 a( H$ O1 X4 M% Ssystemctl enable ceph-mon@ceph01
( @8 G) O2 C; i7 D/ D* `systemctl status ceph-mon@ceph01
1
- m6 j, b  B, A- T  I+ F+ s2
3
" R4 D4 b5 q7 J/ e3 F" \. V同步配置文件、密钥、monmap到其他节点中（Ceph 01执行）

复制ceph.client.admin.keyring、client.bootstrap-osd key、ceph.mon.keyring、monitor map、ceph.conf到另外2个节点
* Y, `% F5 G0 h6 j" `
3 R, m- a4 ]- H! Qscp /etc/ceph/ceph.client.admin.keyring root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.client.admin.keyring root@ceph03:/etc/ceph/
4 G  n# k+ c1 v( z6 o# \1
2
" b# r: W3 ^+ w# @: {  X! _9 Sscp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph02:/var/lib/ceph/bootstrap-osd/
1 R* n- i& r+ k, [, H2 bscp /var/lib/ceph/bootstrap-osd/ceph.keyring root@ceph03:/var/lib/ceph/bootstrap-osd/
1
2
8 ]) s! K. L/ A' o. m( uscp /tmp/ceph.mon.keyring root@ceph02:/tmp/
$ Q+ K7 ~3 `4 x0 Iscp /tmp/ceph.mon.keyring root@ceph03:/tmp/
% x: z0 P% L8 z1
. X1 Q0 r3 B9 U/ S2
7 a/ v8 `, {7 b  l1 c  @- bscp /tmp/monmap root@ceph02:/tmp/
scp /tmp/monmap root@ceph03:/tmp/
8 m) U- ?* ^; b4 l1
, A% t6 K/ I7 o# y$ K+ Y( d: }2
scp /etc/ceph/ceph.conf root@ceph02:/etc/ceph/
scp /etc/ceph/ceph.conf root@ceph03:/etc/ceph/
6 g; P& R( w' T/ k) R( Q1
2
+ v. y% S  y7 t& t/ y) u3 ^8 D% v启动其他节点的monitor服务（Ceph 02执行）

+ ~: \9 |' U6 k4 o  O8 S. {/ Esudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph02
# ^6 X& @5 f+ Fchown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i ceph02 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-ceph02/
1
2
3
( E) ]* j  [1 E' b- j: p4
" T$ J8 c- X# w9 B% {0 Usystemctl start ceph-mon@ceph02
# _, Z* [, q' Gsystemctl enable ceph-mon@ceph02
systemctl status ceph-mon@ceph02
1
, n- B9 v$ I9 }5 L2 M  s# e4 q2 d2
' P" _. l" Q' g- n3
% u- h7 P0 s# w) k启动其他节点的monitor服务（Ceph 03执行）
" d) _9 R; X" g2 i' m! I  E
. ?  |3 Q" ^8 nsudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph03
chown ceph.ceph -R /var/lib/ceph /etc/ceph /tmp/ceph.mon.keyring /tmp/monmap
sudo -u ceph ceph-mon --mkfs -i ceph03 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
  h9 D5 T; a4 E4 P& C" `ls /var/lib/ceph/mon/ceph-ceph03/
* O2 `  K3 {) ^' h( p- ]  ^3 [1
% T; i" t' y6 `2 Q2
3
4
systemctl start ceph-mon@ceph03
systemctl enable ceph-mon@ceph03
+ `7 Q0 r5 V" G  R% O1 d+ v3 \systemctl status ceph-mon@ceph03
* y1 ^( T. B+ S  D0 C! M% h1
% }2 Q7 g; J* Z7 |6 o- ]) o2
  }: F5 Y7 i9 {( |' I3
7 Q. C' d! [8 _; \/ f; n查看当前集群状态（任意节点执行）
通过ceph -s命令查询集群状态，可见集群services中3个mon服务已启动。

ceph -s
> cluster:
5 U. U1 g9 Q" a; Z3 N>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
>                 health: HEALTH_OK
/ g- u6 \& n* k/ c2 y: q' I' t+ {>
" F# n# a; k1 x# f> services:        
8 X8 l' ]" Y- b2 \! g>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)        
>                 mgr:                               
/ e) z& A  ^$ v>                 osd:
>
7 b0 s+ F5 C$ ^0 ?> data:
1 `! c8 E( [. e/ a+ g>                  pools:   
>                 objects:
% z' |9 x1 t7 z; t6 r>                 usage:   
5 v9 i. W+ s3 h4 Q>                 pgs:
1
+ V9 j) }2 M! k+ x" f2
3
7 r/ j1 L( I7 s. b* r9 R4 R4
( ?& ~# W# ]# o& u" }5
6
+ S4 V( E- `, W5 G' j) C9 q+ r7
2 I5 H* o6 i6 r+ e6 P6 `! w+ L8
9
* K% q/ \+ u% `* I5 v6 j  m6 J10
) |7 P7 l& @$ ^0 L11
12
/ ~. c1 K  g, `% \8 @13
14
2 x& k! E0 [, |; v3 H6 m15
+ n: n6 T' ^1 P# u" N7 g% ^/ k3 A/ K部署Ceph mon服务（ceph-volume 自动化创建）
7 M: l' T6 q2 J8 }6 i( s安装Ceph-osd服务程序（所有设备执行）
' H$ p, |1 T. `6 h6 m' l  k! Zyum install -y ceph-osd
1
1 \. n+ R4 A; f/ r; r' I" S9 d0 n初始化osd服务（所有设备执行）
$ {1 @/ o  `- t- D通过fdisk等工具查看磁盘盘符，然后利用ceph-volume工具自动化创建osd服务。
% {* B0 l. E# M7 F, O6 nceph-volume lvm create --data /dev/sda
6 y* Z0 B3 E1 |% nceph-volume lvm create --data /dev/sdb
, y6 x3 ^& s3 {ceph-volume lvm create --data /dev/sdc
1
2
4 [& v5 Q' j0 g! O6 s' C, t; T9 v; `3
查看当前集群状态（任意节点执行）
9 O0 {4 W: }' f, Y3 Q* B通过ceph osd tree命令查询集群状态，可见集群services中所有osd服务已启动。
0 i7 |: [; i* J" E* q6 N0 {# U: a3 _* iceph osd tree
> ID CLASS WEIGHT   TYPE NAME       STATUS REWEIGHT PRI-AFF
> -1       16.36908 root default
> -3        5.45636     host ceph01
- \  T" \: H: D) s& j>  0   hdd  1.81879         osd.0       up  1.00000 1.00000
>  1   hdd  1.81879         osd.1       up  1.00000 1.00000
>  2   hdd  1.81879         osd.2       up  1.00000 1.00000
/ I7 j+ @# v! `7 g' S> -5        5.45636     host ceph02
>  3   hdd  1.81879         osd.3       up  1.00000 1.00000
>  4   hdd  1.81879         osd.4       up  1.00000 1.00000
>  5   hdd  1.81879         osd.5       up  1.00000 1.00000
; J6 t. r& y" x+ A! @> -7        5.45636     host ceph03
3 M/ w0 [" T5 U3 C% w5 t>  6   hdd  1.81879         osd.6       up  1.00000 1.00000
>  7   hdd  1.81879         osd.7       up  1.00000 1.00000
- ?5 o6 @9 {3 `8 k' c>  8   hdd  1.81879         osd.8       up  1.00000 1.00000
1
1 V; _- h9 f" S+ [& N2
  L8 i1 R3 W3 z0 H  W3
4
5 \. n/ P) s- J- X: i' M5
! ~1 d: ~- o$ b' N# l6
1 Z) I$ [% _" a; E7
7 S, q2 L' ~& z. }- m7 V1 P8
9
/ y5 y# n- z$ y5 O10
11
  ^# Z0 k1 ^( `6 k: u# N4 l% h# }12
13
14
- |( Q) v, i4 U" K7 c/ `2 `" q9 y15
部署Ceph mgr服务并开启Dashboard
安装Ceph-mgr服务程序（所有设备执行）
yum install -y ceph-mgr
1
初始化并启动主MGR服务（Ceph01执行）
9 v( H& p$ R7 X5 zmkdir -p /var/lib/ceph/mgr/ceph-ceph01
chown ceph.ceph -R /var/lib/ceph
( M$ O% g- p( Bceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph01.keyring --gen-key -n mgr.ceph01 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
ceph auth import -i /etc/ceph/ceph.mgr.ceph01.keyring
ceph auth get-or-create mgr.ceph01 -o /var/lib/ceph/mgr/ceph-ceph01/keyring
1
2
3
4
5
systemctl start ceph-mgr@ceph01
systemctl enable ceph-mgr@ceph01
; Z! B' x8 ~4 @) T0 q6 i, usystemctl status ceph-mgr@ceph01
% t6 [1 Z$ H! w0 w7 c1
2
3
初始化并启动从MGR服务（Ceph02执行）
mkdir -p /var/lib/ceph/mgr/ceph-ceph02
chown ceph.ceph -R /var/lib/ceph
ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph02.keyring --gen-key -n mgr.ceph02 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
* f, w9 e  _9 c7 }: h1 w1 ^+ Uceph auth import -i /etc/ceph/ceph.mgr.ceph02.keyring
: `0 Y! }; C* \3 Z4 r% O# Mceph auth get-or-create mgr.ceph02 -o /var/lib/ceph/mgr/ceph-ceph02/keyring
1
  E0 E( K! X* d2
3
4
5
systemctl start ceph-mgr@ceph02
  v& N: ~& A5 v- F" y% e$ Fsystemctl enable ceph-mgr@ceph02
systemctl status ceph-mgr@ceph02
# G& r' ^8 z+ @& Y% C$ Q2 x% z1
, j( ]% a# O) w( O2
* o* L3 h1 E  e1 X9 m3
初始化并启动从MGR服务（Ceph03执行）
1 i7 W2 L1 p1 ]) `! C: a+ umkdir -p /var/lib/ceph/mgr/ceph-ceph03
: K8 E4 N0 k/ T8 L6 Jchown ceph.ceph -R /var/lib/ceph
5 p) o/ }# n( N2 T" }ceph-authtool --create-keyring /etc/ceph/ceph.mgr.ceph03.keyring --gen-key -n mgr.ceph03 --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *'
: I" d2 s- h* ?$ `ceph auth import -i /etc/ceph/ceph.mgr.ceph03.keyring
ceph auth get-or-create mgr.ceph03 -o /var/lib/ceph/mgr/ceph-ceph03/keyring
- F- O) e2 h( {8 R' ]7 U' H! v1
$ @! c& ^- ^" J* F* P2
3
4
5
1 g9 ~8 E. L/ @0 M4 L9 A+ I8 X! Bsystemctl start ceph-mgr@ceph03
systemctl enable ceph-mgr@ceph03
systemctl status ceph-mgr@ceph03
$ u) ~0 `7 `& d, E& X9 W6 h" s1
2
0 ]3 Y8 U9 `/ c& ?3
查看当前集群状态（任意节点执行）
3 E. ?# k0 Y6 }/ m通过ceph -s命令查询集群状态，可见集群services中3个mgr服务已启动。
: S( w+ p$ I( x# Lceph -s
> cluster:
8 k% G. x' g+ b4 x& w5 z>                 id:     cf3862c5-f8f6-423e-a03e-beb40fecb74a        
% D" s+ x; t4 e7 S>                 health: HEALTH_OK
>
+ Z5 ^5 k/ {% u! b3 Z> services:        
>                 mon:         3 daemons, quorum ceph03,ceph02,ceph01 (age 12d)        
>                 mgr:        ceph01(active, since 3w), standbys: ceph01、ceph02                       
>                 osd:
>
" d, F2 O" j  c: D1 m) v> data:
>                  pools:   
>                 objects:
>                 usage:   
- l0 O+ X3 I$ n4 [1 C5 M2 H>                 pgs:
7 p' B( j' A9 }% w& k1
2
. l. W7 V  }$ p% K- u, p, A3
) W2 d1 J- |- g' p) k4
( M' q) V! p" [5
4 [2 L5 Z! \- B. E; E* }4 p6
7
2 R$ u+ P1 U6 M! d$ i) N8 w& G8
: K1 _# s  C$ ~1 N% y3 J# _% z9
10
" P" J& f6 n, a$ |0 a11
12
13
14
15
6 c- k; x4 L4 }! ?. X使能Dashboard访问功能（任意节点执行）
开启mgr dashboard功能
ceph mgr module enable dashboard
1
$ Y: b) J: p! w( D+ e* B  S生成并安装自签名的证书
ceph dashboard create-self-signed-cert
1
配置dashboard
ceph config set mgr mgr/dashboard/server_addr 10.40.65.148
* H5 D( e& [0 h1 _# Eceph config set mgr mgr/dashboard/server_port 8080
ceph config set mgr mgr/dashboard/ssl_server_port 8443
$ [* l" J0 ~' A1
4 y/ z4 J0 n/ Z4 P% W; [4 B; N2
& d8 h; ^* l! K7 p4 H3
) b8 G# {0 o' v6 c9 m创建一个dashboard登录用户名密码
1 B/ j- T" H. E" M5 Vecho '123456' > password.txt
0 @7 j; e! j2 O% Dceph dashboard ac-user-create admin  administrator -i password.txt
1
  T) x: e' A8 N) b) a6 s' {2
查看服务访问方式
' P6 s+ s/ w* R' E; a# {0 q. ^ceph mgr services
1
8 w( Y: {. X$ a4 W* }% S通过web访问Ceph Dashboard，用户名密码为admin/123456
https://10.40.65.148:8443