Absolutely required software for dDos attacks

Config Security & Firewall (CSF)

Absolutely the best software I’ve used to date, automatically detects a plethora of patterns and automatically adds the IP to the iptables block list.  Has lots of extra features for detecting malicious file activity and SU logins as well as some basic checks to make sure your configurations are secure.

Check it out at Config Server

Apache Mod_Evasive

Extremely effective and useful module for automatically blocking IP’s that request the same file very rapidly.

Mod_Evasive

SNORT Intrusion Detection

Very effective and useful tool to monitor everything thats going on in your system and track down potential attempted intrusions.

SNORT.org

A common pitfall of the OpenVZ installation

A very very common error I see people make when installing OpenVZ on a 64 bit system is that they will assume the kernel used is a 64 bit kernel. By default OpenVZ does NOT install a 64 bit kernel, but rather a 32 bit one. If the wrong kernel is installed this will cause the machine to drop a kernel panic. This is mentioned in the documentation, however this is an issue I have seen clients have repeatedly.

Guidelines for system file modification

Just figure I would throw a few basic guidelines out there I use whenever I can. Not saying I always follow them as exceptions do happen and I (like many other people out there) am not perfect, but the quickest way to resolve a problem is to be able to get around it. My rules are this:

When changing a file:Always make a copy of it. Try to include something that explains its significance, such as the date or your name. Having a bunch of httpd.conf.bakN where N is the number in which they were created doesn’t really do any good to anyone

When deleting a file:Dont unless necessecary or explicitly requested. Moving a file to a new directory is a far better option. This also lets you be able to examine the files at a later date, and is quicker to perform than a raw delete in most cases I have seen.

When you’re migrating files:Don’t delete the source ones until the new ones are working properly, everything is migrated over and preferably some period of time has passed. Trust me you don’t want to be that guy who didn’t have the DNS server changed over and deleted the active site or you needed some extra files that weren’t there.

The dot:When running commands, before hitting return be sure that your commands are dotted properly. Running chown username ./* and running chown username /* are two commands that will have very different results! This is very common, there are many admins who forgot the dot! This goes for any “heavyweight” commands such as rm, chmod, chown, etc.

Remember the server is meaningless without customers who have their data on it. As a sysadmin it is ultimately YOUR responsibility to take care of ensuring data is in tact. I will cover what I believe to be “suitable” measures to safeguard your data in another blog some time down the road. Until then stay tuned.

cPanel and basic iptables

Save this to a file and run it, This will empty your iptables and set a solid set of secure rules that are compatible with cPanel servers running DNS clustering. If you run DNS locally be sure to allow 53 on TCP AND UDP!

/sbin/iptables -F
/sbin/iptables -X
/sbin/iptables -Z
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p tcp ! --syn -j REJECT --reject-with tcp-reset
/sbin/iptables -A INPUT -m state --state INVALID -j DROP
/sbin/iptables -P OUTPUT DROP
/sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -p tcp ! --syn -j REJECT --reject-with tcp-reset
/sbin/iptables -A OUTPUT -m state --state INVALID -j DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A FORWARD -p tcp ! --syn -j REJECT --reject-with tcp-reset
/sbin/iptables -A FORWARD -m state --state INVALID -j DROP
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
/sbin/iptables -A FORWARD -i lo -o lo -j ACCEPT

##Acceptable IP
/sbin/iptables -A INPUT -s x.x.x.xx -j ACCEPT #YOUR TRUSTED IP's

##General Web/File Services
/sbin/iptables -A INPUT -p tcp --dport 80  -j ACCEPT #HTTP
/sbin/iptables -A INPUT -p tcp --dport 443 -j ACCEPT #HTTPS
/sbin/iptables -A INPUT -p tcp --dport 21 -j ACCEPT #FTP
/sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT #SSH
/sbin/iptables -A INPUT -p tcp --dport 5666 -j ACCEPT #NRPE

##Email Services
/sbin/iptables -A INPUT -p tcp --dport 25 -j ACCEPT #SMTP
/sbin/iptables -A INPUT -p tcp --dport 110 -j ACCEPT #POP3
/sbin/iptables -A INPUT -p tcp --dport 143 -j ACCEPT #IMAP
/sbin/iptables -A INPUT -p tcp --dport 465 -j ACCEPT #SMTPs
/sbin/iptables -A INPUT -p tcp --dport 993 -j ACCEPT #IMAPs
/sbin/iptables -A INPUT -p tcp --dport 995 -j ACCEPT #POP3s

##cPanel Services
/sbin/iptables -A INPUT -p tcp --dport 2083 -j ACCEPT #cPanel
/sbin/iptables -A INPUT -p tcp --dport 2087 -j ACCEPT #WHM
/sbin/iptables -A INPUT -p tcp --dport 2096 -j ACCEPT #Webmail

##Allow Ping
/sbin/iptables -A INPUT -p icmp --icmp-type 8/0 -j ACCEPT

##Final Blocks
/sbin/iptables -A INPUT -j DROP
/sbin/iptables -A OUTPUT -j ACCEPT
/sbin/iptables -A FORWARD -j DROP

Guide on removing iptable rules

If you loose access to your server while using this, well that sucks.   Setup a cron job to stop the iptables service every 5 minutes just in case.

How to submit a support ticket properly

We all submit support tickets in one form or another as part of our jobs in the IT world. There just comes a point where everyone needs some help. This being said, as a person working in the industry for the lowest response times and maximum efficiency there are a few things that you should do when submitting a support ticket:

  • Root password
  • Ports of any services that have been changed
  • Any form of authentication OTHER than passwords required (eg suing up, etc.)
  • Symptoms being reported
  • If you made changes before things started happening what were they
  • Your Ip address if you are experiencing difficulties accessing the site
  • Did you ping your server/VPS and did you get a response? Was it within the typical time?

Something to keep in mind is that by including these answers you will save the technician time in trying to bring your server up as well as reducing the time it takes you to get your server brought back up. The first two are critical, it’s really bad if you have a server running out of RAM but your tech can’t access it because you changed the SSH port to 2020 instead of leaving it on 22. We are here for you the client, however if your server can’t be brought up with the information on hand you get asked some questions and your ticket gets moved aside until a response is had.

Mitigate a small DOS attack

Tonight at 8:45 our main server load alarm tripped, on Nagios, at Beyond Hosting.  By the time I was able to ssh in, load had exceeded 220.64 and the box was headed straight for kernel panic, this is a  “dual core”. Quickly throwing the IPTables firewall onto drop all I was able to prevent the box from locking up, after a few minutes (takes a little to recover from the load) of checking netstat and Apache logs we were able to narrow the attack down to 2 separate ip.
Simply added blocks for the 2 addresses with:

iptables -I INPUT -s x.x.x.x -j DROP

I played with apache mod_evasive a little bit during the attack, pretty solid addon and I highly recommend it.  We tried out (D)Dos deflate but no matter what settings you put it on, it seems to think you have 2500+ connections from 1 IP and black list everyone.

You can view how many connections you have on your server by running:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

Cool One Liners #1

Welcome to the first edition of Cool One Liners. This will be a collection of one line commands you can use via BASH or another shell/scripting language to do something useful. Creativity will definitely be a big merit. Todays one liner is:

cat /var/log/secure | grep Failed | grep sshd | grep root | awk ‘{print $11}’ | sort | uniq -c | sort -n

What does it do? This takes the secure log, sorts out failed login attempts and then makes it so that the IPs are sorted based on the number attempts. Handy to try and track down brute force attempts on an box running SSH. As an example, I generated a few failed logins.

[root@DNS01 log]: cat /var/log/secure | grep Fail

May  9 03:31:58 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May  9 03:32:00 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May  9 03:32:04 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2

After this I ran the command given. Notice how the IPs have the number to the left of them. If this were a list the number with the most logins is going to be at the bottom.

[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n

3 127.0.0.1

This command also serves an additional interesting use. Lets say someone is probing your machine, and they happen to be attempting to brute force some nonstandard account names in the hope of coming up with something on the system that is there and has a weak password. This script will also list any invalid users that attempt to log in as well. An example would be if I attempted to log in with the user root1. The output would look like:

[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n

3 127.0.0.1
3 root1

In another blog we will likely take this command, convert it into a shell script, and make it so it will run as a cron job and email us periodic digests.

Migration from Rackspace Cloud to OpenVZ

Xen Virtualization Technology

Today I had a customer that needed me to move over his Virtual Machine (Virtual Private Server) from the Rack Space Cloud to one of our OpenVZ servers on Beyond Hosting.  While in theory this seemed pretty easy, it became all to clear that It wouldn’t be.  First, rack spaces cloud uses a strange configuration of Xen Para Virtualization, and for whatever reason their configuration does not easily convert to OpenVZ using the standard methods.

Upon importing, the tty configurations are broken and you must repair this before you can serial console into the container or ssh it.  After your fix tty you will need to reconfigure the networking configuration for venet0:0 with the OpenVZ commands.   In my case the /dev devices were correctly configured and I did not need to do any further editing for the container to work correctly.

OpenVZ Virtual Containers

However if you are importing a physical system or Xen HVM guest you will need to follow this guide provided by OpenVZ.  OpenVZ Convert OS to Container If you only have a few applications installed and minimal configuration it is easier to SCP over the configs along with home directory’s.  Create the respective users and chown all the home dir’s back to those users.  Besides, the more you do it the better you will be at using your OS!

Earlier I mentioned that the customer was having me  migrate from a Rack Space Cloud VPS, I just want to note that Rack Space has always been a highly recommended solid and reputable company.  While they still are, the shear size they have grown to has slightly tarnished their products quality.    The reason for the migration was simply cost for performance, he did not feel that the performance was worth what he was paying.

Tracking Network Floods

When it comes to working at a DC, attacks are a fact of life. Someone will ultimately get annoyed at a client and start throwing down with the botnet, their home system etc. These attacks can be hard to isolate at times because they can be on nearly any service, as well as using a wide variety of protocols. These attacks can do anything from making a server slow to respond to taking an entire data center out. When a system goes down due to an attack, there are a few things that must be known:

  • Who is DoSing or who is being DoSed (don’t assume your machine is the “victim” insidious PHP scripts are plentiful these days)
  • The magnitude of the attack
  • What protocol they are using

To find these out, you need proper network infrastructure set up. To this end I like Cacti on the switches for ease of usability in finding overall traffic, even though NFsen is nice for a quick check by IP however it can be flawed in picking up distributed or spoofed attacks in my experience. When attempting to isolate the problems one should have a “tiered” approach where they start checking at the most basic level which is a single server. If the attack is larger  one should try to see if it’s a single rack/switch being attacked and seeing if it can back track to a single server being the target. At this point, there are a few options. Null routes are an option if all fails, even null routing the server’s IP so that other traffic doesn’t get affected can be done as a last resort. The option that you probably have the most control over is server side mitigation. This will be the topic of my next blog. See you then!