Sudo Make Install

Well, it’s been a long time since I posted the first article on this. My time or lack thereof got the best of me. To counter this attack is actually very very easy. The first thing you do is you find out who is the referrer. This is simply done by tailing the logs. If you have a single domain, this can be fairly easy. Otherwise my preferred method involves using “watch ls -l” and seeing which log grows the fastest. This tends to be the one getting hit, or a likely suspect. I will probably write a perl script later to check this and tell me which log grows the most in say 10 seconds eventually. After this, you can use tail in the manner of:

tail -f /etc/httpd/domlogs/domain.log

When you do this, you will see what IPs are querying the page and the source they are being referred from. Look for any thing that doesn’t look like a search engine. To actually block them after they are identified what you do is you block the attack based on a referrer in the .htaccess. See the convenient rewrite code I jacked off another web site (about the same I did when I really saw the attack.)

RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} attacker\.com [NC]
RewriteRule .* – [F]

So, why does this work you may ask? In the case of the scenario I saw the person was attacking a “high value” target. This means a page that hits the database and has dynamically generated content with no caching. Server side configuration CAN make these sort of attacks a lot harder to perpetrate as well. Anything that you can do to increase the robustness of a server will help with a DoS. When you add a rule like this where it denies access to the referrer basically what happens is you pull up static content instead. Static content uses virtually no resources compared to something PHP based and backed by a databse. It’s a good idea to know about this sort of attack, as I could see it being bigger in the future. Black hat SEO is very common these days, and if you have the SEO part down the resources to do the rest of this attack are virtually nothing compared to what it does. It could also be plausible we will see this attack combined with “conventional, network level” type DoSing to increase its effectiveness.

I’ve seen my fair share of clients that need basic MySQL backups but have no control panel or don’t want to bother with Control panel based backups. This is a really simple setup that lets you do DB backups and put them in a local directory of the server. It would likely be easily modified to rsync to another server as well if you wanted to. There are a ton of options that could be added to this, your imagination (and shell scripting capacity) are the only limitations. Some suggestions I have would be

-Mail on success or failure and on old file deletion

-Connect to a remote DB

-Monitor the overall size

Well enough with the abstract, on to the shell!

#!/bin/bash
date=`date +%Y%m%d`
mysqldump –all-databases > /mysqlbackups/mysql-$date.sql
find /mysqlbackups/ -atime +30 -delete

If you notice, this takes up all of 4 lines. The first one is the she-bang, the second is establishing the date time stamp, the third dumps the databases and the last one purges any old backups. The only real variable you have to change here is the “+30″ so that it is the number of days you want to retain the backups for minus one.

I was on a client server getting attacked, the DoS was heavily distributed. Since he’d mentioned something about someone linking to his web site, I was poking through the Apache logs. I noticed that one site was generating a huge amount of referrals. Investigating deeper, Ifound this on the referral site:

<iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
0<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
1<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
2<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>
3<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>

…….

30<br><iframe src=”http://www.domain.com” width=”1″ height=”1″ ></iframe>

This is one of the slicker DoSes I’ve seen in a while. Because of the way it was set up it would be very difficult if not impossible to block on a network level and not traceable back to any particular IP on a network level (read:iptables, RTG or hardware firewall.) Within a few assumptions here this is what I believe to happen:

-Person sets up a web site with just a park page etc. on it.
-Person directs traffic to this using SEO. (back links, etc) to gain it status on search engines
-Person puts up the attack page similar to the above
-Every time a person from a search engine clicks the link, they load a few dozen copies of the page
-The iframe points to a “high value” target that generates a lot of load on the server, such as a forum or other dynamic content.

I personally saw this attack decimate a late model server with 16GB of RAM with enough IP distribution that it was not plausible to block it. It is viciously effective when planned out and done properly. It can also be done with virtually NO resources using a free shared hosting account. The person who loads it probably never realized they just made an attack on a server either. The plus side is that if you track it you can limit the damage done very easily provided you know what you are looking for. That will be my next blog.

At work I had a client with a Debian system that needed a bunch of IPs added to it. Since it doesn’t really support ranges (at least that I can find) I came up with the following script.

#/bin/bash
j=42
for i in  {186..190}
do
j=$(expr $j + 1)
echo auto eth0:$j >> interfaces; echo iface eth0:$j inet static >> interfaces; echo address 192.168.41.$i >> interfaces; echo netmask 255.255.255.248 >> interfaces;
done

How it works is that j is the last IP in the ranges currently set in the interfaces file. The address is defined in the script, and the range is defined in the i= section. Just change the numbers to match what you want, put this into /etc/networking, run it and restart networking. This is only for five IPs but you could do hundreds or thousands this way if it was the desired affect. Or you can use a distro that supports ranges :>

At Beyond Hosting we have a lot of customers who use CSF (Config Server & Firewall)  after about 30 installations of CSF the md5 checker can really cause problems for iowait.   So below is a script to check all the files for the setting of the md5 checker,  you can adapt this to check any file really.

for i in /vz/private/*
do grep "LF_INTEGRITY" $i/etc/csf/csf.conf
echo $i
done

Tonight presents a valuable lesson. I had a box running heavy MySQL duty that would crash at odd times. I could get MySQL to start, but the processes would die, it wouldn’t terminate cleanly, and even on a freshly started copy it was giving me “out of memory” errors. After fighting this for some time (say hours) and assuming that it was me the user, I checked the system in a bout of frustration.

Being a Xeon, my first look after rebooting it was in the error log of the BIOS. It had a lone ECC error in the log. Where I couldn’t even run show databases; before it will go through a check and stay up now. I bring this up as it presents two invaluable lessons:

A)It’s usually the software or the sysadmin that screws a server up. Not the hardware. That being said it is best to consider it. This is the second time I’ve seen a machine with ECC RAM screw up like this in two years and several hundred servers later. I have seen maybe 20 ECC equipped machines that actually had DIMMs that were bad. Probably half that. With that being said MySQL tends to show it first.

B)ECC RAM is worth the extra outlay in the datacenter. This could have easily not been detected for a long period of time, and cost a client and the next client that would have been put on the server.

The default Apache settings that cPanel sets upon install are definitely something that can be improved on. With a few small tweaks, the efficiency with which Apache runs with can be greatly improved.

To start with, lets go ahead and open the Apache configuration file:

vim /usr/local/apache/conf/httpd.conf

This list is a composite of the settings we will be reviewing from fresh install on a cPanel server:

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 15

MinSpareServers 5

MaxSpareServers 10

StartServers 5

MaxClients 150

MaxRequestsPerChild 0

Please note, the settings that we will review in this article are by no means a complete list

of tweakable options in the Apache configuration file. The settings we will be focusing on are

the ones that control how Apache handles webpage requests.

Timeout

Timeout 300

Usually this value doesn’t require editing and a default of 300 is sufficient. Lowering the ‘Timeout’ value will cause a long running script to terminate earlier than expected.

On virtualized servers like VPS servers, lowering this value to 100 can help improve performance.

KeepAlive

KeepAlive On

This setting should be “On” unless the server is getting requests from hundreds of IPs at once.

High volume and/or load balanced servers should have this setting disabled (Off) to increase connection throughput.

MaxKeepAliveRequests

MaxKeepAliveRequests 100

This setting limits the number of requests allowed per persistent connection when KeepAlive is on. If it is set to 0, unlimited requests will be allowed.

It is recommended to keep this value at 100 for virtualized accounts like VPS accounts. On dedicated servers it is recommended that this value be modified to 150.

KeepAliveTimeout

KeepAliveTimeout 15

The number of seconds Apache will wait for another request before closing the connection. Setting this to a high value may cause performance problems in heavily loaded servers. The higher the timeout, the more server processes will be kept occupied waiting on connections with idle clients.

It is recommended that this value be lowered to 5 on all servers.

MinSpareServers

MinSpareServers 5

This directive sets the desired minimum number of idle child server processes. An idle process is one which is not handling a request. If there are fewer spareservers idle then specified by this value, then the parent process creates new children at a maximum rate of 1 per second. Setting this parameter to a large number is almost always a bad idea.

Ajusting the value for this setting to the following:

Virtualized server, ie VPS 5

Dedicated server with 1-2GB RAM 10

Dedicated server with 2-4GB RAM 20

Dedicated server with 4+ GB RAM 25

MaxSpareServers

MaxSpareServers 10

The MaxSpareServers directive sets the desired maximum number of idle child server processes. An idle process is one which is not handling a request. If there are more than MaxSpareServers idle, then the parent process will kill off the excess processes.

The MaxSpareServers value should be set as double the value that is set in MinSpareServers.

StartServers

StartServers 5

This directivesets the number of child server processes created on startup. This value should mirror what is set in MinSpareServers.

MaxClients

MaxClients 150

This directive sets the limit on the number of simultaneous requests that will be served. Any connection attempts over the specified limit will be queued. Once a process is freed at the end of a different request, the queued connection will then be served.

For virtualized servers such as VPS accounts, it is recommended to keep this value at 150. For all dedicated servers the recommended value for this setting is 250.

MaxRequestsPerChild

MaxRequestsPerChild 0

This directive sets the limit on the number of requests that an individual child server process will handle. After the number of requests reaches the value specified, the child process will die. When this value is set at 0, then the process will never expire.

Some Recommended Values

Virtualized server, ie VPS 300

Dedicated server with 1-4GB RAM 500

Dedicated server with 4+GB RAM 1000

After beating my head against a wall for a significant period of time because of a software RAID issue, I figured out how to set it up. Because it was so difficult I figure I would pass the savings on to you. OK here’s what we need to do. First thing is we need to figure out what our arrays are going to be made from. This is simply done with fdisk. After that we would vi /etc/mdamd.conf and add a line like this one:

DEVICE /dev/sdaN /dev/sdbN /dev/sdcN

and so on where sda/b/cN are the partitions our RAIDs are made of. Don’t worry about which is which mdadm will take care of that with the uuids.

Run this next:

mdadm –examine –brief –scan –config=partitions >> /etc/mdadm.conf

and it will put the information for the array into the config file. After this all we need to do is run:

mdadm -A /dev/mdN

for each device we want to set up. Mount as normal. Have a lot of fun

:^>

This is just a quickie I hijacked off another web site but it came in really handy since a client couldn’t access their wordpress admin panel and we don’t really support it. Just log in to SSH or PHPMyAdmin and run the query that follows:

wp_options SET option_value = ” WHERE option_name = ‘active_plugins’;

*BOOM* no more WP plugins. Then if you’re troubleshooting it go through and reactivate till it breaks again. If not tell em they are on their own.

If you are doing this via SSH you will need to go into the MySQL shell. That can usually be done by typing “mysql” or “mysql -u username -p” and entering the password for the user. If you need this you can grab it from wp-config. After that you need to select the db to use. You do that by going:

mysql>use dbname_here;

This will put you into the appropriate db. Then you run the query above. This is of course assuming you’re using the default schema for naming the tables. If  you are not wp_options needs to be changed to just “options” or schema_options as necessecary. If you don’t know what your schema is you can do:

mysql>show tables;

This guide will walk you through installing memcache and memcached on a dedicated or vps server.

For the wiki version of this guide visit: http://wiki.beyondhosting.net/Memcached_and_PHP_with_cPanel

root# yum install libevent libevent-devel -y

Head over to http://memcached.org/ and grab the latest version.

root# yum install libevent-devel libevent gcc make -y
root# wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz
root# tar xvf memcached-1.4.5.tar.gz
root # cd memcached-1.4.5
root# ./configure && make && make install

Open /etc/memcached.conf with your favorite editor and paste in:

#Memory a usar
-m 16
# default port
-p 11211
# user to run daemon nobody/apache/www-data
-u nobody
# only listen locally
-l 127.0.0.1

touch /etc/init.d/memcached
chmod +x /etc/init.d/memcached

Open /etc/init.d/memcached with your favorite editor and paste in:

#!/bin/bash
#
# memcached    This shell script takes care of starting and stopping
#              standalone memcached.
#
# chkconfig: - 80 12
# description: memcached is a high-performance, distributed memory
#              object caching system, generic in nature, but
#              intended for use in speeding up dynamic web
#              applications by alleviating database load.
# processname: memcached
# config: /etc/memcached.conf
# Source function library.
. /etc/rc.d/init.d/functions
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/memcached
DAEMONBOOTSTRAP=/usr/local/bin/start-memcached
DAEMONCONF=/etc/memcached.conf
NAME=memcached
DESC=memcached
PIDFILE=/var/run/$NAME.pid
[ -x $DAEMON ] || exit 0
[ -x $DAEMONBOOTSTRAP ] || exit 0
RETVAL=0
start() {
 echo -n $"Starting $DESC: "
 daemon $DAEMONBOOTSTRAP $DAEMONCONF
 RETVAL=$?
 [ $RETVAL -eq 0 ] && touch $PIDFILE
 echo
 return $RETVAL
}
stop() {
 echo -n $"Shutting down $DESC: "
 killproc $NAME
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f $PIDFILE
 return $RETVAL
}
# See how we were called.
case "$1" in
 start)
  start
  ;;
 stop)
  stop
  ;;
 restart|reload)
  stop
  start
  RETVAL=$?
  ;;
 status)
  status $prog
  RETVAL=$?
  ;;
 *)
  echo $"Usage: $0 {start|stop|restart|status}"
  exit 1
esac
exit $RETVAL

touch /usr/local/bin/start-memcached
chmod +x  /usr/local/bin/start-memcached

Open /usr/local/bin/start-memcached with your favorite editor and paste in:

#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci <jaybonci@debian.org>
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if ($> != 0 and $< != 0) {
 print STDERR "Only root wants to run start-memcached.\n";
 exit;
}
my $etcfile = shift || "/etc/memcached.conf";
my $params = [];
my $etchandle;
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/memcached.pid
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/memcached.pid";
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile {
 my ($logfile) = @_;
 $fd_reopened = $logfile;
}
sub reopen_logfile {
 my ($logfile) = @_;
 open *STDERR, ">>$logfile";
 open *STDOUT, ">>$logfile";
 open *STDIN, ">>/dev/null";
 $fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
 "logfile" => \&handle_logfile
};
if (open $etchandle, $etcfile) {
 foreach my $line (<$etchandle>) {
  $line =~ s/\#.*//go;
  $line = join ' ', split ' ', $line;
  next unless $line;
  next if $line =~ /^\-[dh]/o;
  if ($line =~ /^[^\-]/o) {
   my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
   $conf_directives->{$directive}->($arg);
   next;
  }
  push @$params, $line;
 }
}
unshift @$params, "-u root" unless (grep $_ eq '-u', @$params);
$params = join " ", @$params;
if (-e $pidfile) {
 open PIDHANDLE, "$pidfile";
 my $localpid = <PIDHANDLE>;
 close PIDHANDLE;
 chomp $localpid;
 if (-d "/proc/$localpid") {
  print STDERR "memcached is already running.\n";
  exit;
 } else {
  `rm -f $localpid`;
 }
}
my $pid = fork();
if ($pid == 0) {
 reopen_logfile($fd_reopened);
 exec "$memcached $params";
 exit(0);
} elsif (open PIDHANDLE,">$pidfile") {
 print PIDHANDLE $pid;
 close PIDHANDLE;
} else {
 print STDERR "Can't write pidfile to $pidfile.\n";
}

Now we start the memcached daemon. (it is ok if “shutting down memcached” says fail.)

[root@ ~]# /etc/init.d/memcached restart
Shutting down memcached:                                   [  OK  ]
Starting memcached:                                        [  OK  ]

Make sure its running by:

[root@srv01 init.d]# ps aux  | grep memcached
nobody    5966  0.5  0.3 18248 16444 pts/0   S    13:55   0:00 /usr/local/bin/memcached -u root -m 16 -p 11211 -u nobody -l 127.0.0.1

Now we will set memcached to run at startup:

[root@ ~]# /sbin/chkconfig memcached on
[root@ ~]# /sbin/chkconfig --list | grep memcached
memcached       0:off   1:off   2:on    3:on    4:on    5:on    6:off

Now we will install the memcache plugin for PHP.

Download the latest stable version of memcache from http://pecl.php.net/package/memcache

root# wget http://pecl.php.net/get/memcache-2.2.5.tgz
root# tar xvf memcache-2.2.5.tgz
root# cd memcache-2.2.5
root# phpize
root# ./configure && make && make install

Now open /usr/local/lib/php.ini with your favorite text editor and find the dynamic extension section.

Add this

extension=memcache.so

Now restart apache with:

service httpd restart

We will now check to make sure memcached is running.

Create an empty file called test.php and place this in it:

<? phpinfo();  ?>

root# php -f test.php  | grep "memcache support"
memcache support => enabled

If this command does not return any thing the memcache plugin did not load correctly.

You can now delete test.php, your memcache installation should be functional.

References:

http://www.vbseo.com/blogs/danny-bembibre/daemon-scripts-memcached-44/

http://bloke.org/linux/installing-memcached-on-centos-cpanel/