Some devices struggle with persistent ownership due to the driver. I have some OCZ ssd used for journal that are affected by this.
So I’ve created a udev rule to assign them to the proper user during boot.
Add this to /etc/udev/rules.d/89-ceph-journal.rules
KERNEL=="oczpcie*?" SUBSYSTEM=="block" OWNER="ceph" GROUP="disk" MODE="0660"
Then retrigger it to test
udevadm trigger --action=add
ls -lh /dev/oczpcie_3_0_ssd*
brw-rw---- 1 ceph disk 251, 0 Jul 14 13:01 /dev/oczpcie_3_0_ssd
brw-rw---- 1 ceph disk 251, 1 Jul 14 13:01 /dev/oczpcie_3_0_ssd1
brw-rw---- 1 ceph disk 251, 4 Jul 14 13:01 /dev/oczpcie_3_0_ssd4
brw-rw---- 1 ceph disk 251, 5 Jul 14 13:01 /dev/oczpcie_3_0_ssd5
brw-rw---- 1 ceph disk 251, 6 Jul 14 13:01 /dev/oczpcie_3_0_ssd6
brw-rw---- 1 ceph disk 251, 7 Jul 14 13:01 /dev/oczpcie_3_0_ssd7
brw-rw---- 1 ceph disk 251, 8 Jul 14 13:01 /dev/oczpcie_3_0_ssd8
brw-rw---- 1 ceph disk 251, 9 Jul 14 13:01 /dev/oczpcie_3_0_ssd9
If you’re having inventory or hardware issues during discovery in UCS, there is a few commands that can help clear it up and get the system to properly inventory.
The latest method I’ve learned about from TAC is to do the following.
decommission slot 6/5
reset slot 6/5
acknowledge slot 6/5
Grab storage software from: https://www.micron.com/products/solid-state-storage/storage-executive-software
Use CLI:
mkdir mnt
sudo mount -o loop /path/to/the/iso mnt
mkdir tmp
cd tmp
gunzip -c ../mnt/boot/core.gz | cpio -i
sudo /path/to/msecli -U -i opt/firmware -n /dev/sdX -r
hdparm --user-master u --security-set-pass Eins /dev/X
hdparm --user-master u --security-erase Eins /dev/X
2 Methods:
1. Mdadm
mdadm --zero-superblock /dev/sdX
2. DD end of disk (this is where raid info lives)
dd if=/dev/zero of=sdX bs=512 seek=$(( $(blockdev --getsz sdX) - 1024 )) count=1024
Your DHCPD slow af? Getting
"dhcpd: none: host unknown."
Just add this to /etc/hosts
255.255.255.255 none
Check this out: http://www.tldp.org/HOWTO/DHCP/x369.html
Ya so if your filestore OSD’s wont mount the journal run this:
udevadm trigger --action=add
If you’re using bluestore and unmounted the tmpfs run this:
ceph-volume lvm activate --all
Have an issue with starting you bluestore osd after reboot?
http://tracker.ceph.com/issues/23067
lvs -o lv_tags | awk -vFS=, /ceph.osd_fsid/'{ OSD_ID=gensub(".*ceph.osd_id=([0-9]+),.*", "\\1", ""); OSD_FSID=gensub(".*ceph.osd_fsid=([a-z0-9-]+),.*", "\\1", ""); print OSD_ID,OSD_FSID }' | sort -n | uniq
That gives you the OSD ID + FSID to activate. You can loop over it like:
for i in `lvs -o lv_tags | awk -vFS=, /ceph.osd_fsid/'{ OSD_ID=gensub(".*ceph.osd_id=([0-9]+),.*", "\\1", ""); OSD_FSID=gensub(".*ceph.osd_fsid=([a-z0-9-]+),.*", "\\1", ""); print OSD_ID,OSD_FSID }' | sort -n | uniq`;
do
ceph-volume lvm activate $i
done