创建过程
如果需要部署osd的机器是一个没有任何ceph组件的机器，需要将
/etc/ceph/ceph.conf传输过去，不然无法找到集群位置
将
/var/lib/ceph/bootstrap-osd/ceph.keyring转移到需要创建osd的机器上shell[root@ceph-0 /var/lib/ceph/osd]# scp /var/lib/ceph/bootstrap-osd/ceph.keyring ceph-1:/var/lib/ceph/bootstrap-osd/ceph.keyring ceph.keyring 100% 129 11.2KB/s 00:00创建osd
也可以批量创建，下面的是创建一个的命令
shellceph-volume lvm batch --bluestore /dev/sdb /dev/sdc /dev/sdd /dev/sdeshell[root@ceph-1 /var/lib/ceph]# ceph-volume lvm create --bluestore --data /dev/sde Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c56cc8ed-8e08-4962-829b-840a188f1641 Running command: /sbin/vgcreate --force --yes ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7 /dev/sde stdout: Wiping xfs signature on /dev/sde. stdout: Physical volume "/dev/sde" successfully created. stdout: Volume group "ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7" successfully created Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-c56cc8ed-8e08-4962-829b-840a188f1641 ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7 stdout: Logical volume "osd-block-c56cc8ed-8e08-4962-829b-840a188f1641" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-7 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin Running command: /bin/chown -h ceph:ceph /dev/ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7/osd-block-c56cc8ed-8e08-4962-829b-840a188f1641 Running command: /bin/chown -R ceph:ceph /dev/dm-4 Running command: /bin/ln -s /dev/ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7/osd-block-c56cc8ed-8e08-4962-829b-840a188f1641 /var/lib/ceph/osd/ceph-7/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-7/activate.monmap stderr: 2024-04-12T17:28:36.397+0800 7fd6a7980700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2024-04-12T17:28:36.397+0800 7fd6a7980700 -1 AuthRegistry(0x7fd6a0058eb8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: got monmap epoch 4 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-7/keyring --create-keyring --name osd.7 --add-key AQDC/hhmEK8oFRAAmlCQruWzhZEb8ew0/rSboQ== stdout: creating /var/lib/ceph/osd/ceph-7/keyring stdout: added entity osd.7 auth(key=AQDC/hhmEK8oFRAAmlCQruWzhZEb8ew0/rSboQ==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 7 --monmap /var/lib/ceph/osd/ceph-7/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-7/ --osd-uuid c56cc8ed-8e08-4962-829b-840a188f1641 --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/sde Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7/osd-block-c56cc8ed-8e08-4962-829b-840a188f1641 --path /var/lib/ceph/osd/ceph-7 --no-mon-config Running command: /bin/ln -snf /dev/ceph-abfa51c2-fe89-4a33-9773-96b7e43f2bc7/osd-block-c56cc8ed-8e08-4962-829b-840a188f1641 /var/lib/ceph/osd/ceph-7/block Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-7/block Running command: /bin/chown -R ceph:ceph /dev/dm-4 Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 Running command: /bin/systemctl enable ceph-volume@lvm-7-c56cc8ed-8e08-4962-829b-840a188f1641 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-7-c56cc8ed-8e08-4962-829b-840a188f1641.service → /lib/systemd/system/ceph-volume@.service. Running command: /bin/systemctl enable --runtime ceph-osd@7 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd@7.service → /lib/systemd/system/ceph-osd@.service. Running command: /bin/systemctl start ceph-osd@7 --> ceph-volume lvm activate successful for osd ID: 7 --> ceph-volume lvm create successful for: /dev/sde
创建失败
在创建的过程中，有创建失败的可能，但是 ceph-volumn lvm create会生成lvm卷，需要删除这个卷才能继续创建，使用lvremove
shell
[root@ceph-0 /etc/ceph]# ceph-volume lvm create --bluestore --data /dev/sdb
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8cf266e8-65c7-430b-8cf0-19e8402cf95c
Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-8cf266e8-65c7-430b-8cf0-19e8402cf95c ceph-b1f19f56-5894-4554-a8d4-681e9c9ddb6f
stderr: Calculated size of logical volume is 0 extents. Needs to be larger.
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
--> RuntimeError: command returned non-zero exit status: 5
[root@ceph-0 /etc/ceph]# ceph-volume lvm list
====== osd.0 =======
[block] /dev/ceph-b1f19f56-5894-4554-a8d4-681e9c9ddb6f/osd-block-53d67815-aff7-4203-9133-3d1a2b16a9a5
block device /dev/ceph-b1f19f56-5894-4554-a8d4-681e9c9ddb6f/osd-block-53d67815-aff7-4203-9133-3d1a2b16a9a5
block uuid gDEUWH-iuLX-mrVv-YmaJ-mwWc-S0q1-fuFZ1n
cephx lockbox secret
cluster fsid 8a212a0e-0fb5-4f4a-a1ee-53d7b3783a64
cluster name ceph
crush device class None
encrypted 0
osd fsid 53d67815-aff7-4203-9133-3d1a2b16a9a5
osd id 0
type block
vdo 0
devices /dev/sdb
[root@ceph-0 /etc/ceph]# lvremove /dev/ceph-b1f19f56-5894-4554-a8d4-681e9c9ddb6f/osd-block-53d67815-aff7-4203-9133-3d1a2b16a9a5
Do you really want to remove and DISCARD active logical volume ceph-b1f19f56-5894-4554-a8d4-681e9c9ddb6f/osd-block-53d67815-aff7-4203-9133-3d1a2b16a9a5? [y/n]: Y
Logical volume "osd-block-53d67815-aff7-4203-9133-3d1a2b16a9a5" successfully removed还有一种可能，乜有创建lv，只是加入了vg，也需要删除，先删除vg，再删除pv
shell
[root@ceph-3 /etc/ceph]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 49G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 24.5G 0 lvm /
sdb 8:16 0 20G 0 disk
sdc 8:32 0 20G 0 disk
sdd 8:48 0 20G 0 disk
sde 8:64 0 20G 0 disk
sr0 11:0 1 1024M 0 rom
[root@ceph-3 /etc/ceph]# vgs
VG #PV #LV #SN Attr VSize VFree
ceph-1f180dd3-9d21-47d9-96b5-7c5d8d6650ab 1 0 0 wz--n- <20.00g <20.00g
ceph-70b2aa3a-d025-4497-9402-23f86bcd8458 1 0 0 wz--n- <20.00g <20.00g
ceph-906dfe7e-35cf-4325-8f7c-390feb582663 1 0 0 wz--n- <20.00g <20.00g
ceph-b03a2187-4d8e-4972-b5be-506a3b604bb1 1 0 0 wz--n- <20.00g <20.00g
ubuntu-vg 1 1 0 wz--n- <49.00g <24.50g
[root@ceph-3 /etc/ceph]# vgs |grep ceph |awk '{print $1}'
ceph-1f180dd3-9d21-47d9-96b5-7c5d8d6650ab
ceph-70b2aa3a-d025-4497-9402-23f86bcd8458
ceph-906dfe7e-35cf-4325-8f7c-390feb582663
ceph-b03a2187-4d8e-4972-b5be-506a3b604bb1
[root@ceph-3 /etc/ceph]# vgremove `vgs |grep ceph |awk '{print $1}'`
Volume group "ceph-1f180dd3-9d21-47d9-96b5-7c5d8d6650ab" successfully removed
Volume group "ceph-70b2aa3a-d025-4497-9402-23f86bcd8458" successfully removed
Volume group "ceph-906dfe7e-35cf-4325-8f7c-390feb582663" successfully removed
Volume group "ceph-b03a2187-4d8e-4972-b5be-506a3b604bb1" successfully removed
[root@ceph-3 /etc/ceph]# pvs --noheadings -o pv_name --separator=" " -S "vg_name=''"
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
[root@ceph-3 /etc/ceph]# pvremove `pvs --noheadings -o pv_name --separator=" " -S "vg_name=''"`
Labels on physical volume "/dev/sdb" successfully wiped.
Labels on physical volume "/dev/sdc" successfully wiped.
Labels on physical volume "/dev/sdd" successfully wiped.
Labels on physical volume "/dev/sde" successfully wiped.
[root@ceph-3 /etc/ceph]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 ubuntu-vg lvm2 a-- <49.00g <24.50g