Ceph Journal Device Ownership

Some devices struggle with persistent ownership due to the driver.   I have some OCZ ssd used for journal that are affected by this.

So I’ve created a udev rule to assign them to the proper user during boot.

Add this to /etc/udev/rules.d/89-ceph-journal.rules

KERNEL=="oczpcie*?" SUBSYSTEM=="block" OWNER="ceph" GROUP="disk" MODE="0660"

Then retrigger it to test

udevadm trigger --action=add
ls -lh /dev/oczpcie_3_0_ssd*
brw-rw---- 1 ceph disk 251, 0 Jul 14 13:01 /dev/oczpcie_3_0_ssd
brw-rw---- 1 ceph disk 251, 1 Jul 14 13:01 /dev/oczpcie_3_0_ssd1
brw-rw---- 1 ceph disk 251, 4 Jul 14 13:01 /dev/oczpcie_3_0_ssd4
brw-rw---- 1 ceph disk 251, 5 Jul 14 13:01 /dev/oczpcie_3_0_ssd5
brw-rw---- 1 ceph disk 251, 6 Jul 14 13:01 /dev/oczpcie_3_0_ssd6
brw-rw---- 1 ceph disk 251, 7 Jul 14 13:01 /dev/oczpcie_3_0_ssd7
brw-rw---- 1 ceph disk 251, 8 Jul 14 13:01 /dev/oczpcie_3_0_ssd8
brw-rw---- 1 ceph disk 251, 9 Jul 14 13:01 /dev/oczpcie_3_0_ssd9

Ceph udev rule retrigger for mounting OSD and Journals

Ya so if your filestore OSD’s wont mount the journal run this:

udevadm trigger --action=add

If you’re using bluestore and unmounted the tmpfs run this:

ceph-volume lvm activate --all

Have an issue with starting you bluestore osd after reboot?
http://tracker.ceph.com/issues/23067

lvs -o lv_tags | awk -vFS=, /ceph.osd_fsid/'{ OSD_ID=gensub(".*ceph.osd_id=([0-9]+),.*", "\\1", ""); OSD_FSID=gensub(".*ceph.osd_fsid=([a-z0-9-]+),.*", "\\1", ""); print OSD_ID,OSD_FSID }' | sort -n | uniq

That gives you the OSD ID + FSID to activate. You can loop over it like:

for i in `lvs -o lv_tags | awk -vFS=, /ceph.osd_fsid/'{ OSD_ID=gensub(".*ceph.osd_id=([0-9]+),.*", "\\1", ""); OSD_FSID=gensub(".*ceph.osd_fsid=([a-z0-9-]+),.*", "\\1", ""); print OSD_ID,OSD_FSID }' | sort -n | uniq`;
do
ceph-volume lvm activate $i
done