As a systems administrator it seems like I’m constantly battling IO contention and latency in our san and local storage environments. So As months roll by these new SSD drives keep getting cheaper and cheaper, offering better write wear and longer life spans for high write intensive environments, so finally I’m taking the plunge to begin converting our most intensive systems over to solid state.

In the process of exploring solid state disk the samsung 256GB 830 series really stuck out of the crowd. The 830 offers fantastic read and write latency and throughput as well as being one of the only SSD series on the market where both the flash and storage controller are by the same manufacture.

The main reason for chosing the samsung is this benchmark at extreme systems.

Update: 8/24/12

We ended up going back to the dell H710P after having a few issues with the uEFI bios not playing well with the controller at post.  Not to mention LSI webbios is a horrible pile of useless shit, this is 2012 why the hell do we have this prehistoric pile of crap UI on a raid controller.  Whoever at LSI approved that to be shipped on the cards should be forced to stand in a fire.

The H710P has dells lovely customized controller bios which is keyboard driven EASY to use and FAST to configure with.   Performance of the H710P is actually a little bit better than the 9266-8i while the hardware is identical.

Another major issue with the 9266 is when you would remove a drive *failure simulation* and replace it, the controller would mark the new drive as bad vs treating it as a fresh drive to rebuild on.  Without the CLI or MegaRaid Storage Manager this is a rather annoying problem to deal with as you would need to reboot the system to fix it in WEbiboss11!!111.. POS.

The H710P obviously works with dells unified system and can be accessed a number of ways without the operating system even knowing about it.

 The configuration:

• 16x Samsung 830 256GB MLC SSD
• Raid 6 with read and write caching (BBU backed).  64KB Block Size
• Dell R720 16 Bay 8i SAS6 Expanded Backplane  2 Ports 16 devices.

The Benchmarks!

Here are some prelim benchmarks of the actual performance inside a VMware machine.

LSI 9266-8i

Children see throughput for 32 initial writers  =  214905.26 ops/sec
Parent sees throughput for 32 initial writers   =  198172.68 ops/sec
Min throughput per process                      =    6392.06 ops/sec
Max throughput per process                      =    7173.76 ops/sec
Avg throughput per process                      =    6715.79 ops/sec
Min xfer                                        =  925970.00 ops

Children see throughput for 32 readers          =  734057.97 ops/sec
Parent sees throughput for 32 readers           =  734011.56 ops/sec
Min throughput per process                      =   22833.85 ops/sec
Max throughput per process                      =   23062.16 ops/sec
Avg throughput per process                      =   22939.31 ops/sec
Min xfer                                        = 1038205.00 ops

Children see throughput for 32 random readers   =   55662.96 ops/sec
Parent sees throughput for 32 random readers    =   55662.71 ops/sec
Min throughput per process                      =    1730.88 ops/sec
Max throughput per process                      =    1751.76 ops/sec
Avg throughput per process                      =    1739.47 ops/sec
Min xfer                                        = 1036073.00 ops

Children see throughput for 32 random writers   =   19827.16 ops/sec
Parent sees throughput for 32 random writers    =   19090.45 ops/sec
Min throughput per process                      =     584.53 ops/sec
Max throughput per process                      =     663.61 ops/sec
Avg throughput per process                      =     619.60 ops/sec
Min xfer                                        =  967988.00 ops

Dell H710P

Children see throughput for 32 initial writers  =  489124.60 ops/sec
Parent sees throughput for 32 initial writers   =  435746.51 ops/sec
Min throughput per process                      =   14005.25 ops/sec
Max throughput per process                      =   17028.75 ops/sec
Avg throughput per process                      =   15285.14 ops/sec
Min xfer                                        =  860278.00 ops

Children see throughput for 32 readers          =  678563.56 ops/sec
Parent sees throughput for 32 readers           =  678524.72 ops/sec
Min throughput per process                      =   21111.18 ops/sec
Max throughput per process                      =   21253.53 ops/sec
Avg throughput per process                      =   21205.11 ops/sec
Min xfer                                        = 1041599.00 ops

Children see throughput for 32 random readers   =   59482.27 ops/sec
Parent sees throughput for 32 random readers    =   59482.00 ops/sec
Min throughput per process                      =    1851.91 ops/sec
Max throughput per process                      =    1869.25 ops/sec
Avg throughput per process                      =    1858.82 ops/sec
Min xfer                                        = 1038852.00 ops

Children see throughput for 32 random writers   =   20437.99 ops/sec
Parent sees throughput for 32 random writers    =   19228.06 ops/sec
Min throughput per process                      =     610.33 ops/sec
Max throughput per process                      =     695.63 ops/sec
Avg throughput per process                      =     638.69 ops/sec
Min xfer                                        =  945641.00 ops

grep -v “*: nobody” /etc/userdomains |cut -d: -f2 | sort | uniq | xargs /scripts/pkgacct {} \;

This will remove the generic (nobody) user from the list, separate the user from their respective domain, and execute the pkgacct command.

The terminal its self came with a copy of embedded Windows that was trying to connect to a Citrix Server. Considering that is far from what I wanted, I needed to figure out how to get Linux on it. There are a few things you need to know about these terminals. They have an AMD Geode processor, 128 Megabytes of RAM, but more importantly IMO they only have 32 Megabytes of flash. This ultimately is the biggest limitation.

This means you’re really having to chop things down to even get a base OS and some sort of SSH client on them. There are really 2 vectors for getting another OS on these clients, the first is USB. Considering this is a USB1 terminal, it is horridly slow to do this way. It does work, but a basic flash drive takes forever to boot. The other way, the way that I ultimately settled on is PXE booting. There are a few advantages of this IMO, among them being:

-Faster. 100 megs versus 12 megs. Even with a basic initrd it’s obvious to me which wins.

-Ease of migration. You can use this with pretty much any desktop, laptop or terminal that will PXE boot. That’s very convenient if you want to step up to something a bit hotter, or use a random laptop as a spare console.

-Remote connectivity. If I wanted to use this in my garage remotely, all I would have to do is run an Ethernet cable. Makes it really easy to get a VT.  No OS installations to worry about, it “just works.”

-Expandable. We can set this up to connect to a remote X11 server later if we want to.

To actually start the install, we need to change the BIOS. The password for these systems is “Fireport” which makes it easy for us to log in. There’s not a ton of options here, so we will change the boot order and exit.

Linux selection is a mixed bag on these, the fact the Geode is an I586 architecture limits the options pretty significantly. I decided on Slackware as I’ve always liked it, you can use the latest version with an I586 system and it’s fairly easy to “chop.”

On to actually getting this thing booted up in a (non-windows) environment. First things we need to do are set up services. On SuSE this can be a bit more trying than Red Hat due to having to open some ports, but the process is essentially the same. I have a separate NIC as well due to running DHCP on my main network and not wanting to cause conflicts with that. The idea is that the main server will be 192.168.1.1, the terminal will be set up as 192.168.1.2.

/etc/dhcpd.conf:

ddns-update-style none;
default-lease-time 14400;
filename “pxelinux.0″;

# IP address of the dhcp server nothing but this machine.
next-server 192.168.1.1;
subnet 192.168.1.0 netmask 255.255.255.0 {
# ip distribution range between 192.168.1.1 to 192.168.1.100
range 192.168.1.2 192.168.1.100;
default-lease-time 10;
max-lease-time 10;
}

I also edited /etc/sysconfig/dhcpd to set up dhcpd to listen on eth1:

DHCPD_INTERFACE=”eth1″

The next thing I did was install the TFTP server. There’s not much to that, it’s an Xinetd service. Be sure that ports are open, if at all possible I like to try and nmap the server to make sure everything is open and running. After this, we need to add a few things to the config. The first is the pxelinux.0 file which goes in /tftproot/. After this, a pxelinux.cfg directory needs to get created. Add a file to this called default. Since I started with the hugesmp kernel (I would just use the regular huge kernel since this is unicore and uniproc) I set it up the following way:

default hugesmp.s
prompt 1
timeout 1
display message.txt
F1 message.txt
F2 f2.tx
label hugesmp.s
kernel kernels/hugesmp.s/bzImage
append initrd=initrd.img load_ramdisk=1 prompt_ramdisk=0 rw SLACK_KERNEL=hugesmp.s

I copied the initrd.img off the DVD as well as the “kernels” directory into /tftpboot in their entirety. You should be able to actually boot the Slackware initrd at this point, and run any of the setup aps you want. We however, are going to do a lot more with it. This will come in part II where we will tweak it to our means. The cool thing about this initrd is that it has an SSH server as well as an SSH client built into it.

The first thing we do is use vgdisplay to show original info for the Volume Group. Notice how when you look at this, the Free PE Size is 0MB.

[root@nfsen01 ~]# vgdisplay
— Volume group —
VG Name               VolGroup00
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  3
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               2
Max PV                0
Cur PV                1
Act PV                1
VG Size               2.88 GB
PE Size               32.00 MB
Total PE              92
Alloc PE / Size       92 / 2.88 GB
Free  PE / Size       0 / 0
VG UUID              XXXXXXXXXXXXXXXXXXXXXXXXXX

Assuming you are using sdb1 as your drive, and that you have LVM set up as the partition type on it already extending the Volume Group is as simple as:

vgextend VolGroup00 /dev/sdb1

And this will extend the volume across the entire disk. You should be able to run vgdisplay again and see your free PE size went up.

What you have to do next is extend the Logical Volume for the disk. This is optional depending on your objectives, if you wanted a common VG and wanted to create new volumes you can do it at your convenience now.

lvextend -L +931.51G /dev/mapper/VolGroup00-LogVol00

Assuming you’re running EXT3 you would use this command. For other file systems on top of LVM your milage may vary; Consult your documentation.

resize2fs /dev/mapper/VolGroup00-LogVol00 -p

After this is done you should be able to use df -h on the drive, and see your partition has been enlarged. This can even be done while the system is active, there’s no need for any boot CDs or the likes.

mysql> select * from IPs;
+———————————-+———————————-+————————–+
| ip_address                       | netmask                          | computer_name            |
+———————————-+———————————-+————————–+
| 11000000101010000000001000000101 | 11111111111111111111111100000000 | control.frontandback.net |
+———————————-+———————————-+————————–+
1 row in set (0.00 sec)

#/usr/bin/perl

#IP2DB 0.1.0 (C) Febuary 2011 Howard A Underwood II
#Free for use and modification under the Creative Commons 1.0 License. If you want to give me a shout out try aunderwoodii#at#gmail.com
#The purpose of this code is to convert an IP address and netmask pair into Binary to make it easily stored in the database in a processable manner. This is only for IPV4 atm and is just a proof of concept, I’d love to see your adaptations to real world applications. Feel free to give me your feedback at the above address.

#This requires DBI and DBD::MySQL. Use CPAN or your package manager of choice to get them.
use DBI;
use DBD::mysql;

#info to connect to the DB server. This assumes that your table is pre-created. If you need to create a database do the following:
#create database ips;
#CREATE TABLE IPs (ip_address BINARY(32) PRIMARY KEY, netmask BINARY(32), computer_name char(200));

$hostname=localhost;
$db=”ips”;
$port=”3306″;
$user=”dbuser”;
$password=”wouldn’tyouliketoknow”;

#info to put into the DB. There’s the IP here, netmask and the computer name. These variables and the ones above are going to be what you need to use to adapt the script to your needs.
$ip=”192.168.2.5″;
$netmask=”255.255.255.0″;
$compname=”control.frontandback.net”;

#Getting down to business. This first line takes the netmask and breaks it into 4 ocets.
my @netmask = split (/\./, $netmask);
#Now that we have 4 ocets, we process each one into binary. Future modifications include cleaning this code up so that it’s a loop rather than 4 instances.
$ocetnm0= unpack(“B*”, pack(“C”, $netmask[0]));
$ocetnm1= unpack(“B*”, pack(“C”, $netmask[1]));
$ocetnm2= unpack(“B*”, pack(“C”, $netmask[2]));
$ocetnm3= unpack(“B*”, pack(“C”, $netmask[3]));
#We recombine everything into 1 Binary number after this.
$totalnm= $ocetnm0.$ocetnm1.$ocetnm2.$ocetnm3;
#Just printing the post process # on the TTY for human verification
print “$totalnm\n”;

#Now we repeat the process for the IP its self. This will probably get condensed into one instance along with the above code eventually. Once again, not the most efficient way to do it but rather straight forward.
my @ip = split (/\./, $ip);
$ocet0= unpack(“B*”, pack(“C”, $ip[0]));
$ocet1= unpack(“B*”, pack(“C”, $ip[1]));
$ocet2= unpack(“B*”, pack(“C”, $ip[2]));
$ocet3= unpack(“B*”, pack(“C”, $ip[3]));
$total= $ocet0.$ocet1.$ocet2.$ocet3;
print “$total\n”;

#Basic DBI connection code. We are using the DBI script to connect to the databse
$dsn = “DBI:mysql:database=$db;host=$hostname;port=$port”;
$DBIconnect = DBI->connect($dsn, $user, $password)
#If we don’t like what we see bail out because we can’t connect.
or die “Connection denied to database $db \n;”;
#Add the entry to the table. Please note that if you use the above table it will probably not let you run this more than once for any given IP.
eval { $DBIconnect->do(“INSERT INTO IPs (ip_address,netmask,computer_name) VALUES (‘$total’,'$totalnm’,'$compname’);”) };
print “Data not added to the database: $@\n” if $@;

tail -f /etc/httpd/domlogs/domain.log

When you do this, you will see what IPs are querying the page and the source they are being referred from. Look for any thing that doesn’t look like a search engine. To actually block them after they are identified what you do is you block the attack based on a referrer in the .htaccess. See the convenient rewrite code I jacked off another web site (about the same I did when I really saw the attack.)

RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} attacker\.com [NC]
RewriteRule .* – [F]

So, why does this work you may ask? In the case of the scenario I saw the person was attacking a “high value” target. This means a page that hits the database and has dynamically generated content with no caching. Server side configuration CAN make these sort of attacks a lot harder to perpetrate as well. Anything that you can do to increase the robustness of a server will help with a DoS. When you add a rule like this where it denies access to the referrer basically what happens is you pull up static content instead. Static content uses virtually no resources compared to something PHP based and backed by a databse. It’s a good idea to know about this sort of attack, as I could see it being bigger in the future. Black hat SEO is very common these days, and if you have the SEO part down the resources to do the rest of this attack are virtually nothing compared to what it does. It could also be plausible we will see this attack combined with “conventional, network level” type DoSing to increase its effectiveness.

A)run it as the same user

B)Put it in the user’s homedir

C)Make it so that it was password protected and executable via PHP script so the user wouldn’t require any bash experience at all but could upload a list via FTP and just run it.

#!/bin/bash

for video in `cat /root/list.txt` #We will run a loop where each line in list.txt is run as a variable $video.
do
mv /home/user/public_html/media/videos/flv/$video.flv /home/user/public_html/media/videos/flv/$video.flv.old #back up old files
ffmpeg -y -b 1500 -r 25 -i  /home/gogreenc/public_html/media/videos/vid/$video.* -f flv -s 640×480 -deinterlace -ac 1 -ar 41400 /home/user/public_html/media/videos/flv/$video.flv #encode new file, 640X480 out, FLV format deinterlaced.
chown user:user /home/user/public_html/media/videos/flv/$video.flv #chown to the right user. Not required if running as the right user.
done

-Mail on success or failure and on old file deletion

-Connect to a remote DB

-Monitor the overall size

Well enough with the abstract, on to the shell!

#!/bin/bash
date=`date +%Y%m%d`
mysqldump –all-databases > /mysqlbackups/mysql-$date.sql
find /mysqlbackups/ -atime +30 -delete

If you notice, this takes up all of 4 lines. The first one is the she-bang, the second is establishing the date time stamp, the third dumps the databases and the last one purges any old backups. The only real variable you have to change here is the “+30″ so that it is the number of days you want to retain the backups for minus one.

First we need to enable performance monitoring:

tw_cli /c0 set dpmstat=on

Now we will show the information its providing.

tw_cli /c0 show dpmstat  type=ra
Drive Performance Monitor Configuration for /c0 ...
Performance Monitor: ON
Version: 1
Max commands for averaging: 100
Max latency commands to save: 10
Requested data: Running Average Drive Statistics

 Queue           Xfer         Resp
Port   Status           Unit   Depth   IOPs    Rate(MB/s)   Time(ms)
------------------------------------------------------------------------
p0     OK               u0     22      23      0.479        11
p1     OK               u0     24      93      1.344        12
p2     OK               u0     25      82      0.720        14
p3     OK               u0     24      83      1.108        16

BE SURE TO TURN OFF PERFORMANCE MONITORING WHEN YOU ARE DONE!

tw_cli /c0 set dpmstat=off

Different performance results:

This command only applies to 9000 series SX/SE/SA controllers, except for
type=ext, which applies only to SE/SA models.

This command allows you to request drive statistics of the specified type for
the specified port. These statistics can be helpful when troubleshooting
performance problems.

type= specifies which statistics should be displayed. The options are: inst for
Instantaneous, ra for Running Average, lct for Long Command Times,
histdata for Histogram Data, and ext for Extended Drive Statistics.

inst (Instantaneous). This measurement provides a short duration average.
ra (Running Average). Running average is a measure of long-term averages
that smooth out the data, and results in older results fading from the average
over time.

ext (Extended Drive Statistics). The extended drive statistics refers to
statistics of a drive’s read commands, write commands, write commands with
FUA (Force Unit Access), flush commands, and a drive sectors’s read, write,
and write commands with FUA.

lct (Long Command Times). This a collection of the commands with the
longest read/write response time.

histdata (Histogram Data). The histogram categorizes the read/write
execution times and group them together based on time frames.

<iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
0<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
1<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
2<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
3<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>

…….

30<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>

This is one of the slicker DoSes I’ve seen in a while. Because of the way it was set up it would be very difficult if not impossible to block on a network level and not traceable back to any particular IP on a network level (read:iptables, RTG or hardware firewall.) Within a few assumptions here this is what I believe to happen:

-Person sets up a web site with just a park page etc. on it.
-Person directs traffic to this using SEO. (back links, etc) to gain it status on search engines
-Person puts up the attack page similar to the above
-Every time a person from a search engine clicks the link, they load a few dozen copies of the page
-The iframe points to a “high value” target that generates a lot of load on the server, such as a forum or other dynamic content.

I personally saw this attack decimate a late model server with 16GB of RAM with enough IP distribution that it was not plausible to block it. It is viciously effective when planned out and done properly. It can also be done with virtually NO resources using a free shared hosting account. The person who loads it probably never realized they just made an attack on a server either. The plus side is that if you track it you can limit the damage done very easily provided you know what you are looking for. That will be my next blog.