Quick and Dirty Ceph Deployment

Replace the disk names and ssd device name.   This will build a ceph cluster with 2 object redundancy in about 5 minutes.

ceph-deploy purge ceph0-mon0 ceph0-mon1 ceph0-mon2 ceph0-node0 ceph0-node1
ceph-deploy purgedata ceph0-mon0 ceph0-mon1 ceph0-mon2 ceph0-node0 ceph0-node1
ceph-deploy forgetkeys


ceph-deploy new ceph0-mon0 ceph0-mon1 ceph0-mon2

echo "osd pool default size = 2" >> ~/ceph.conf
echo "public network = 10.1.8.0/22" >> ~/ceph.conf
echo "cluster network = 10.1.12.0/22" >> ~/ceph.conf
echo "osd journal size = 12000" >> ~/ceph.conf

ceph-deploy install ceph0-mon0 ceph0-mon1 ceph0-mon2 ceph0-node0 ceph0-node1
ceph-deploy mon create-initial

ceph-deploy admin ceph0-mon0 ceph0-mon1 ceph0-mon2 ceph0-node0 ceph0-node1

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

ceph-deploy disk zap ceph0-node0:/dev/oczpcie_4_0_ssd
ceph-deploy disk zap ceph0-node0:/dev/sdb
ceph-deploy disk zap ceph0-node0:/dev/sdc
ceph-deploy disk zap ceph0-node0:/dev/sdd
ceph-deploy disk zap ceph0-node0:/dev/sde
ceph-deploy disk zap ceph0-node0:/dev/sdf
ceph-deploy disk zap ceph0-node0:/dev/sdg
ceph-deploy disk zap ceph0-node0:/dev/sdh
ceph-deploy disk zap ceph0-node0:/dev/sdi
ceph-deploy disk zap ceph0-node0:/dev/sdj
ceph-deploy disk zap ceph0-node0:/dev/sdk
ceph-deploy disk zap ceph0-node0:/dev/sdl
ceph-deploy disk zap ceph0-node0:/dev/sdm

ceph-deploy disk zap ceph0-node1:/dev/oczpcie_4_0_ssd
ceph-deploy disk zap ceph0-node1:/dev/sdb
ceph-deploy disk zap ceph0-node1:/dev/sdc
ceph-deploy disk zap ceph0-node1:/dev/sdd
ceph-deploy disk zap ceph0-node1:/dev/sde
ceph-deploy disk zap ceph0-node1:/dev/sdf
ceph-deploy disk zap ceph0-node1:/dev/sdg
ceph-deploy disk zap ceph0-node1:/dev/sdh
ceph-deploy disk zap ceph0-node1:/dev/sdi
ceph-deploy disk zap ceph0-node1:/dev/sdj
ceph-deploy disk zap ceph0-node1:/dev/sdk
ceph-deploy disk zap ceph0-node1:/dev/sdl
ceph-deploy disk zap ceph0-node1:/dev/sdm

ceph-deploy osd prepare ceph0-node0:/dev/sdb:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdb:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdc:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdc:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdd:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdd:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sde:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sde:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdf:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdf:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdg:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdg:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdh:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdh:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdi:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdi:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdj:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdj:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdk:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdk:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdl:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdl:/dev/oczpcie_4_0_ssd

ceph-deploy osd prepare ceph0-node0:/dev/sdm:/dev/oczpcie_4_0_ssd
ceph-deploy osd prepare ceph0-node1:/dev/sdm:/dev/oczpcie_4_0_ssd

Dell M1000e Manually Configure Set Minimal Fan Speed Control

You can manually configure the minimum fan speed of the m1000e so that the chassis maintains a lower operating temperature.

SSH The CMC with the cmc ip address and port 22.  User will be root and calvin unless changed.

Then run:

racadm config -g cfgThermal -o cfgThermalMFSPercent  75

This will set the minimum fan speed to 75%.  You can set it from 0-100%.  Obviously 0% is more like 35% but you won’t be able to tell.

You can view the requested fan speed by the servers in the chassis by running:

racadm getfanreqinfo

Example:

[Server Module Fan Request Table]

<Slot#>   <Server Name>   <Blade Type>       <Power State>  <Presence>   <Fan Request%>   

1         s2086.corp PowerEdgeM610      ON             Present      48               

2         s2087.corp PowerEdgeM610      ON             Present      48               

3         s2088.corp PowerEdgeM610      ON             Present      48               

[Switch Module Fan Request Table]

<IO>      <Name>                           <Type>             <Presence>   <Fan Request%>   

Switch-1  MXL 10/40GbE                     10 GbE KR          Present      30               

Switch-2  MXL 10/40GbE                     10 GbE KR          Present      30               

Switch-3  N/A                              None               Not Present  N/A              

Switch-4  N/A                              None               Not Present  N/A              

Switch-5  N/A                              None               Not Present  N/A              

Switch-6  N/A                              None               Not Present  N/A              

[Minimum Fan Speed %]

65

EMC XtremIO

6a00e552e53bd28833019aff2b85a5970bWell It’s been a while since I posted  but I got some new exciting stuff to talk about!

We’ve purchased our first EMC XtremIO array!   The system is 2x 20T Xbricks which gives us around 30TB raw capacity.  Compression and dedupe brings us to around 210TB usable storage, this is based on a  7:1  compress/dedupe factor which is proving to be the standard for the current software.

 

The system comes in up to an 8 brick configuration:

emc-XtremIO-f2

 

What each X-Brick looks like racked:

Each X-Brick contains:

  • Eaton high quality UPS unit.
  • 2 Storage Controllers.
  • DAE enclosure sas connected to the Storage Controllers.
  •  Once you go to a 2 or more brick system you will have 2 48G infiniban switches connecting the storage controllers together.

 

Update 12/1/14

 

So we’ve had our array in production for about 8 weeks now.  I have nothing but good things to say, performance has been absolutely incredible and stability / reliability has been everything promised.

The storage controller servers are Intel chassis with dual power supply, dual infiniban controllers and dual sas hba.  They appear to be running some sort of E5 cpu and boast 256GB of ram each.

Inside XtremIO Storage Controller

 

The system is pretty busy once cabled up, the architecture is very cluster oriented so there is a lot of redundancy in the cabling.

Xtremio Xbrick Cabling

Some pictures for the front of the array/Eaton UPS.

EMC EATON XtremIO UPS

 

20141202_180648

We are currently running around 550 production virtual machines that service 7,000 customer servers.   We are averaging 350-400MB/s read/writes at 20k io day in day out.   We’ve seen well over 20GB/s transfers and over 200k iop.

Storage vMotion and VAAI actions are extremely fast and completed almost instantly.  At the current time our data reduction/dedupe ratio is around 2.5:1 but I believe this numbers inaccurate as the total amount of data in our datastores is much larger than it shows stored.  🙂

Some UI Screenshots of our environment:

 

XtremIO UI Bandiwdth

XtremIO UI IOPS

xtremioui3

The lights on the disks in the UI blink based on activity, pretty cool eye candy.

Installing OpenVSwitch 2.3.1 LTS on CentOS 6

yum install kernel-headers kernel-devel gcc make python-devel openssl-devel kernel-devel, graphviz kernel-debug-devel automake rpm-build redhat-rpm-config libtool git

cd /root/

wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.64.tar.gz

tar xvf autoconf-2.64.tar.gz

cd autoconf-2.64/

./configure

make

make install

 

cd /root/

wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz -O /root/openvswitch-2.3.1.tar.gz

 

mkdir /root/rpmbuild/SOURCES

cp /root/openvswitch-2.3.1.tar.gz /root/rpmbuild/SOURCES/

rpmbuild -bb rhel/openvswitch.spec
rpmbuild -bb rhel/openvswitch-kmod-rhel6.spec

rpm -ivh /root/rpmbuild/RPMS/*.rpm

 

You can also use our public repo here for cloudstack.

http://mirror.beyondhosting.net/Cloudstack/

 

16 x 256GB Samsung 830 SSD Raid 6 with LSI 9266-8i Controller in a Dell R720 (16 Bay)

 

As a systems administrator it seems like I’m constantly battling IO contention and latency in our san and local storage environments. So As months roll by these new SSD drives keep getting cheaper and cheaper, offering better write wear and longer life spans for high write intensive environments, so finally I’m taking the plunge to begin converting our most intensive systems over to solid state.

In the process of exploring solid state disk the samsung 256GB 830 series really stuck out of the crowd. The 830 offers fantastic read and write latency and throughput as well as being one of the only SSD series on the market where both the flash and storage controller are by the same manufacture.

The main reason for chosing the samsung is this benchmark at extreme systems.

 

 

Update: 8/24/12

We ended up going back to the dell H710P after having a few issues with the uEFI bios not playing well with the controller at post.  Not to mention LSI webbios is a horrible pile of useless shit, this is 2012 why the hell do we have this prehistoric pile of crap UI on a raid controller.  Whoever at LSI approved that to be shipped on the cards should be forced to stand in a fire.

The H710P has dells lovely customized controller bios which is keyboard driven EASY to use and FAST to configure with.   Performance of the H710P is actually a little bit better than the 9266-8i while the hardware is identical.

Another major issue with the 9266 is when you would remove a drive *failure simulation* and replace it, the controller would mark the new drive as bad vs treating it as a fresh drive to rebuild on.  Without the CLI or MegaRaid Storage Manager this is a rather annoying problem to deal with as you would need to reboot the system to fix it in WEbiboss11!!111.. POS.

The H710P obviously works with dells unified system and can be accessed a number of ways without the operating system even knowing about it.

 The configuration:

  • 16x Samsung 830 256GB MLC SSD
  • Raid 6 with read and write caching (BBU backed).  64KB Block Size
  • Dell R720 16 Bay 8i SAS6 Expanded Backplane  2 Ports 16 devices.

The Benchmarks!

Here are some prelim benchmarks of the actual performance inside a VMware machine.

LSI 9266-8i

Children see throughput for 32 initial writers  =  214905.26 ops/sec
Parent sees throughput for 32 initial writers   =  198172.68 ops/sec
Min throughput per process                      =    6392.06 ops/sec
Max throughput per process                      =    7173.76 ops/sec
Avg throughput per process                      =    6715.79 ops/sec
Min xfer                                        =  925970.00 ops

Children see throughput for 32 readers          =  734057.97 ops/sec
Parent sees throughput for 32 readers           =  734011.56 ops/sec
Min throughput per process                      =   22833.85 ops/sec
Max throughput per process                      =   23062.16 ops/sec
Avg throughput per process                      =   22939.31 ops/sec
Min xfer                                        = 1038205.00 ops

Children see throughput for 32 random readers   =   55662.96 ops/sec
Parent sees throughput for 32 random readers    =   55662.71 ops/sec
Min throughput per process                      =    1730.88 ops/sec
Max throughput per process                      =    1751.76 ops/sec
Avg throughput per process                      =    1739.47 ops/sec
Min xfer                                        = 1036073.00 ops

Children see throughput for 32 random writers   =   19827.16 ops/sec
Parent sees throughput for 32 random writers    =   19090.45 ops/sec
Min throughput per process                      =     584.53 ops/sec
Max throughput per process                      =     663.61 ops/sec
Avg throughput per process                      =     619.60 ops/sec
Min xfer                                        =  967988.00 ops

Dell H710P

Children see throughput for 32 initial writers  =  489124.60 ops/sec
Parent sees throughput for 32 initial writers   =  435746.51 ops/sec
Min throughput per process                      =   14005.25 ops/sec
Max throughput per process                      =   17028.75 ops/sec
Avg throughput per process                      =   15285.14 ops/sec
Min xfer                                        =  860278.00 ops

Children see throughput for 32 readers          =  678563.56 ops/sec
Parent sees throughput for 32 readers           =  678524.72 ops/sec
Min throughput per process                      =   21111.18 ops/sec
Max throughput per process                      =   21253.53 ops/sec
Avg throughput per process                      =   21205.11 ops/sec
Min xfer                                        = 1041599.00 ops

Children see throughput for 32 random readers   =   59482.27 ops/sec
Parent sees throughput for 32 random readers    =   59482.00 ops/sec
Min throughput per process                      =    1851.91 ops/sec
Max throughput per process                      =    1869.25 ops/sec
Avg throughput per process                      =    1858.82 ops/sec
Min xfer                                        = 1038852.00 ops

Children see throughput for 32 random writers   =   20437.99 ops/sec
Parent sees throughput for 32 random writers    =   19228.06 ops/sec
Min throughput per process                      =     610.33 ops/sec
Max throughput per process                      =     695.63 ops/sec
Avg throughput per process                      =     638.69 ops/sec
Min xfer                                        =  945641.00 ops

 

 

Update 7/20/13!

So we’ve been running this configuration in production for almost a year now without fault.   Performance remains fantastic and we’ve had 0 disk failures or faults.

We’ve began testing on the 840 PRO series of disk and so far testing has not been as favorable, having some minor issues with 512gb drives being kicked from the array or faulting for no apparent reasons.

I can confirm that the 840 pro series are NOT compatible with the 24 bay chassis, the backplane power is designed for 12v utilization and the samsung drives are 5v.  You will have random system lockups with a message about not enough system power available.  If you need to populate a 24 bay chassis we recommend looking at the intel emlc drives which utilize 12v power optimization.

$10 512MB VPS are a joke.

We’ve been offering VPS for a few months now and have been paying close attention to our competitors. After doing the math several times I’m still baffled as to how people can offer 512mb ram on a VPS and sell it for $10 and make a profit.

Say the each host node has 24GB of ram, this allows for roughly 44 VPS giving the node some free ram for cache. Now 44*10 = 440, $440 is what it cost to maintain a server with 24gb ram + bandwidth, power and high performance disk. Where do you make profit at this price?

And it seems like the cheap host always say “we don’t oversell”, well how do you not oversell something like that, whats the point of running a business if you don’t make any money? A lot of the providers are offering 512MB under xen, As far as I know its a LOT harder to oversell a Xen node than it is OpenVZ.

Regardless, even if you do somehow run this server for less than 300/mo I wouldn’t consider the profit made effective once you factor in paying people to handle support tickets ect.

Keep that in mind when your shopping for a VPS, paying the least you can does not mean your getting a good deal.

http://BeyondVPS.net

Migration from Rackspace Cloud to OpenVZ

Xen Virtualization Technology

Today I had a customer that needed me to move over his Virtual Machine (Virtual Private Server) from the Rack Space Cloud to one of our OpenVZ servers on Beyond Hosting.  While in theory this seemed pretty easy, it became all to clear that It wouldn’t be.  First, rack spaces cloud uses a strange configuration of Xen Para Virtualization, and for whatever reason their configuration does not easily convert to OpenVZ using the standard methods.

Upon importing, the tty configurations are broken and you must repair this before you can serial console into the container or ssh it.  After your fix tty you will need to reconfigure the networking configuration for venet0:0 with the OpenVZ commands.   In my case the /dev devices were correctly configured and I did not need to do any further editing for the container to work correctly.

OpenVZ Virtual Containers

However if you are importing a physical system or Xen HVM guest you will need to follow this guide provided by OpenVZ.  OpenVZ Convert OS to Container If you only have a few applications installed and minimal configuration it is easier to SCP over the configs along with home directory’s.  Create the respective users and chown all the home dir’s back to those users.  Besides, the more you do it the better you will be at using your OS!

Earlier I mentioned that the customer was having me  migrate from a Rack Space Cloud VPS, I just want to note that Rack Space has always been a highly recommended solid and reputable company.  While they still are, the shear size they have grown to has slightly tarnished their products quality.    The reason for the migration was simply cost for performance, he did not feel that the performance was worth what he was paying.

SolusVM New Update

Hi everyone, a long anticipated update has finally been released for SolusVM today.

New features in build SolusVM Enterprise v1.2.04

Us at Beyond Hosting have been delaying the launch of Xen Containers simply because there was no way to rate limit them.    It’s also nice to see how they build their PHP modules, everything for WHMCS is ioncube encoded and hard to modify if needed.