Ceph — Basic Management of OSD location and weight in the crushmap

It’s amazing how crappy hard disk are!   No really!   We operate a 100 disk ceph pool for our object based backups and Its almost a weekly task to replace a failing drive.   I’ve only seen one go entirely unresponsive but normally we get read error and rear failures that stop the osd service and show up in dmesg as faults.


To change the weight of a drive:

ceph osd crush reweight osd.90 1.82

To replace a drive:

#Remove old disk
ceph osd out osd.31
ceph osd crush rm osd.31
ceph osd rm osd.31
ceph auth del osd.31
#Provision new disk
ceph-deploy osd prepare --overwrite-conf hostname01:/dev/diskname

Move a host into a different root bucket.

ceph osd crush move hostname01 root=BUCKETNAME

Juniper SRX Flow vs Packet Mode

The Juniper SRX as it comes forwards IP traffic based on flows between security zones.  It can be configured to forward traffic based on packets (no fancy security features).  In packet mode an SRX acts just like a router or layer 3 switch. This is useful for labs and learning.

Run the following command to get an idea of how your SRX is forwarding traffic.
> show security flow status

By default Inet (IPv4) traffic is the only traffic that is configured to forward traffic in flow mode.

To disable this simply delete all of the configuration under the security hierarchy.
# delete security
# commit
# run request system reboot

To enable other traffic types use the following commands

# set security fowarding-options family inet6 mode packet-based

# set security fowarding-options family mpls mode packet-based

# set security fowarding-options family iso mode packet-based

You must now commit the configuration and reboot the device.

There is another method to do this that allows you to use both flow and packet mode on the same family which requires firewall rule.  I will go over that in another post.

LLDP Cisco 3750 + Brocade VDX + Dell MXL + Juniper MX

Here is a brief overview of the  LLDP configuration needed for each device to give you similar information across all your devices.

Cisco 3750
lldp run
lldp tlv-select system-name
lldp tlv-select system-description
lldp tlv-select system-capabilities
lldp tlv-select port-description
lldp tlv-select management-address

Brocade VDX
protocol lldp
advertise optional-tlv management-address
advertise optional-tlv port-description
advertise optional-tlv system-capabilities
advertise optional-tlv system-description
advertise optional-tlv system-name

Dell MXL
protocol lldp
advertise management-tlv management-address system-capabilities system-description system-name
advertise interface-port-desc

Juniper MX
set protocols lldp port-id-subtype interface-name
set protocols lldp interface all