Why you pay money for ECC RAM

Tonight presents a valuable lesson. I had a box running heavy MySQL duty that would crash at odd times. I could get MySQL to start, but the processes would die, it wouldn’t terminate cleanly, and even on a freshly started copy it was giving me “out of memory” errors. After fighting this for some time (say hours) and assuming that it was me the user, I checked the system in a bout of frustration.

Being a Xeon, my first look after rebooting it was in the error log of the BIOS. It had a lone ECC error in the log. Where I couldn’t even run show databases; before it will go through a check and stay up now. I bring this up as it presents two invaluable lessons:

A)It’s usually the software or the sysadmin that screws a server up. Not the hardware. That being said it is best to consider it. This is the second time I’ve seen a machine with ECC RAM screw up like this in two years and several hundred servers later. I have seen maybe 20 ECC equipped machines that actually had DIMMs that were bad. Probably half that. With that being said MySQL tends to show it first.

B)ECC RAM is worth the extra outlay in the datacenter. This could have easily not been detected for a long period of time, and cost a client and the next client that would have been put on the server.

Optimize Apache For Heavy Traffic

The default Apache settings that cPanel sets upon install are definitely something that can be improved on. With a few small tweaks, the efficiency with which Apache runs with can be greatly improved.

To start with, lets go ahead and open the Apache configuration file:

vim /usr/local/apache/conf/httpd.conf

This list is a composite of the settings we will be reviewing from fresh install on a cPanel server:

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 15

MinSpareServers 5

MaxSpareServers 10

StartServers 5

MaxClients 150

MaxRequestsPerChild 0

Please note, the settings that we will review in this article are by no means a complete list

of tweakable options in the Apache configuration file. The settings we will be focusing on are

the ones that control how Apache handles webpage requests.

Timeout

Timeout 300

Usually this value doesn’t require editing and a default of 300 is sufficient. Lowering the ‘Timeout’ value will cause a long running script to terminate earlier than expected.

On virtualized servers like VPS servers, lowering this value to 100 can help improve performance.

KeepAlive

KeepAlive On

This setting should be “On” unless the server is getting requests from hundreds of IPs at once.

High volume and/or load balanced servers should have this setting disabled (Off) to increase connection throughput.

MaxKeepAliveRequests

MaxKeepAliveRequests 100

This setting limits the number of requests allowed per persistent connection when KeepAlive is on. If it is set to 0, unlimited requests will be allowed.

It is recommended to keep this value at 100 for virtualized accounts like VPS accounts. On dedicated servers it is recommended that this value be modified to 150.

KeepAliveTimeout

KeepAliveTimeout 15

The number of seconds Apache will wait for another request before closing the connection. Setting this to a high value may cause performance problems in heavily loaded servers. The higher the timeout, the more server processes will be kept occupied waiting on connections with idle clients.

It is recommended that this value be lowered to 5 on all servers.

MinSpareServers

MinSpareServers 5

This directive sets the desired minimum number of idle child server processes. An idle process is one which is not handling a request. If there are fewer spareservers idle then specified by this value, then the parent process creates new children at a maximum rate of 1 per second. Setting this parameter to a large number is almost always a bad idea.

Ajusting the value for this setting to the following:

Virtualized server, ie VPS 5

Dedicated server with 1-2GB RAM 10

Dedicated server with 2-4GB RAM 20

Dedicated server with 4+ GB RAM 25

MaxSpareServers

MaxSpareServers 10

The MaxSpareServers directive sets the desired maximum number of idle child server processes. An idle process is one which is not handling a request. If there are more than MaxSpareServers idle, then the parent process will kill off the excess processes.

The MaxSpareServers value should be set as double the value that is set in MinSpareServers.

StartServers

StartServers 5

This directivesets the number of child server processes created on startup. This value should mirror what is set in MinSpareServers.

MaxClients

MaxClients 150

This directive sets the limit on the number of simultaneous requests that will be served. Any connection attempts over the specified limit will be queued. Once a process is freed at the end of a different request, the queued connection will then be served.

For virtualized servers such as VPS accounts, it is recommended to keep this value at 150. For all dedicated servers the recommended value for this setting is 250.

MaxRequestsPerChild

MaxRequestsPerChild 0

This directive sets the limit on the number of requests that an individual child server process will handle. After the number of requests reaches the value specified, the child process will die. When this value is set at 0, then the process will never expire.

Some Recommended Values

Virtualized server, ie VPS 300

Dedicated server with 1-4GB RAM 500

Dedicated server with 4+GB RAM 1000

How to set up DM-RAID drives in a rescue environment

After beating my head against a wall for a significant period of time because of a software RAID issue, I figured out how to set it up. Because it was so difficult I figure I would pass the savings on to you. OK here’s what we need to do. First thing is we need to figure out what our arrays are going to be made from. This is simply done with fdisk. After that we would vi /etc/mdamd.conf and add a line like this one:

DEVICE /dev/sdaN /dev/sdbN /dev/sdcN

and so on where sda/b/cN are the partitions our RAIDs are made of. Don’t worry about which is which mdadm will take care of that with the uuids.

Run this next:

mdadm –examine –brief –scan –config=partitions >> /etc/mdadm.conf

and it will put the information for the array into the config file. After this all we need to do is run:

mdadm -A /dev/mdN

for each device we want to set up. Mount as normal. Have a lot of fun

:^>

How to disable wordpress in a single query.

This is just a quickie I hijacked off another web site but it came in really handy since a client couldn’t access their wordpress admin panel and we don’t really support it. Just log in to SSH or PHPMyAdmin and run the query that follows:

wp_options SET option_value = ” WHERE option_name = ‘active_plugins’;

*BOOM* no more WP plugins. Then if you’re troubleshooting it go through and reactivate till it breaks again. If not tell em they are on their own.

If you are doing this via SSH you will need to go into the MySQL shell. That can usually be done by typing “mysql” or “mysql -u username -p” and entering the password for the user. If you need this you can grab it from wp-config. After that you need to select the db to use. You do that by going:

mysql>use dbname_here;

This will put you into the appropriate db. Then you run the query above. This is of course assuming you’re using the default schema for naming the tables. If  you are not wp_options needs to be changed to just “options” or schema_options as necessecary. If you don’t know what your schema is you can do:

mysql>show tables;

Install memcached with CentOS and WHM/cPanel

This guide will walk you through installing memcache and memcached on a dedicated or vps server.

For the wiki version of this guide visit: http://wiki.beyondhosting.net/Memcached_and_PHP_with_cPanel

root# yum install libevent libevent-devel -y

Head over to http://memcached.org/ and grab the latest version.

root# yum install libevent-devel libevent gcc make -y
root# wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz
root# tar xvf memcached-1.4.5.tar.gz
root # cd memcached-1.4.5
root# ./configure && make && make install

Open /etc/memcached.conf with your favorite editor and paste in:

#Memory a usar
-m 16
# default port
-p 11211
# user to run daemon nobody/apache/www-data
-u nobody
# only listen locally
-l 127.0.0.1
touch /etc/init.d/memcached
chmod +x /etc/init.d/memcached

Open /etc/init.d/memcached with your favorite editor and paste in:

#!/bin/bash
#
# memcached    This shell script takes care of starting and stopping
#              standalone memcached.
#
# chkconfig: - 80 12
# description: memcached is a high-performance, distributed memory
#              object caching system, generic in nature, but
#              intended for use in speeding up dynamic web
#              applications by alleviating database load.
# processname: memcached
# config: /etc/memcached.conf
# Source function library.
. /etc/rc.d/init.d/functions
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/memcached
DAEMONBOOTSTRAP=/usr/local/bin/start-memcached
DAEMONCONF=/etc/memcached.conf
NAME=memcached
DESC=memcached
PIDFILE=/var/run/$NAME.pid
[ -x $DAEMON ] || exit 0
[ -x $DAEMONBOOTSTRAP ] || exit 0
RETVAL=0
start() {
 echo -n $"Starting $DESC: "
 daemon $DAEMONBOOTSTRAP $DAEMONCONF
 RETVAL=$?
 [ $RETVAL -eq 0 ] && touch $PIDFILE
 echo
 return $RETVAL
}
stop() {
 echo -n $"Shutting down $DESC: "
 killproc $NAME
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f $PIDFILE
 return $RETVAL
}
# See how we were called.
case "$1" in
 start)
  start
  ;;
 stop)
  stop
  ;;
 restart|reload)
  stop
  start
  RETVAL=$?
  ;;
 status)
  status $prog
  RETVAL=$?
  ;;
 *)
  echo $"Usage: $0 {start|stop|restart|status}"
  exit 1
esac
exit $RETVAL
touch /usr/local/bin/start-memcached
chmod +x  /usr/local/bin/start-memcached

Open /usr/local/bin/start-memcached with your favorite editor and paste in:

#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci <jaybonci@debian.org>
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if ($> != 0 and $< != 0) {
 print STDERR "Only root wants to run start-memcached.\n";
 exit;
}
my $etcfile = shift || "/etc/memcached.conf";
my $params = [];
my $etchandle;
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/memcached.pid
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/memcached.pid";
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile {
 my ($logfile) = @_;
 $fd_reopened = $logfile;
}
sub reopen_logfile {
 my ($logfile) = @_;
 open *STDERR, ">>$logfile";
 open *STDOUT, ">>$logfile";
 open *STDIN, ">>/dev/null";
 $fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
 "logfile" => \&handle_logfile
};
if (open $etchandle, $etcfile) {
 foreach my $line (<$etchandle>) {
  $line =~ s/\#.*//go;
  $line = join ' ', split ' ', $line;
  next unless $line;
  next if $line =~ /^\-[dh]/o;
  if ($line =~ /^[^\-]/o) {
   my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
   $conf_directives->{$directive}->($arg);
   next;
  }
  push @$params, $line;
 }
}
unshift @$params, "-u root" unless (grep $_ eq '-u', @$params);
$params = join " ", @$params;
if (-e $pidfile) {
 open PIDHANDLE, "$pidfile";
 my $localpid = <PIDHANDLE>;
 close PIDHANDLE;
 chomp $localpid;
 if (-d "/proc/$localpid") {
  print STDERR "memcached is already running.\n";
  exit;
 } else {
  `rm -f $localpid`;
 }
}
my $pid = fork();
if ($pid == 0) {
 reopen_logfile($fd_reopened);
 exec "$memcached $params";
 exit(0);
} elsif (open PIDHANDLE,">$pidfile") {
 print PIDHANDLE $pid;
 close PIDHANDLE;
} else {
 print STDERR "Can't write pidfile to $pidfile.\n";
}

Now we start the memcached daemon. (it is ok if “shutting down memcached” says fail.)

[root@ ~]# /etc/init.d/memcached restart
Shutting down memcached:                                   [  OK  ]
Starting memcached:                                        [  OK  ]

Make sure its running by:

[root@srv01 init.d]# ps aux  | grep memcached
nobody    5966  0.5  0.3 18248 16444 pts/0   S    13:55   0:00 /usr/local/bin/memcached -u root -m 16 -p 11211 -u nobody -l 127.0.0.1

Now we will set memcached to run at startup:

[root@ ~]# /sbin/chkconfig memcached on
[root@ ~]# /sbin/chkconfig --list | grep memcached
memcached       0:off   1:off   2:on    3:on    4:on    5:on    6:off

Now we will install the memcache plugin for PHP.

Download the latest stable version of memcache from http://pecl.php.net/package/memcache

root# wget http://pecl.php.net/get/memcache-2.2.5.tgz
root# tar xvf memcache-2.2.5.tgz
root# cd memcache-2.2.5
root# phpize
root# ./configure && make && make install

Now open /usr/local/lib/php.ini with your favorite text editor and find the dynamic extension section.

Add this

extension=memcache.so

Now restart apache with:

service httpd restart

We will now check to make sure memcached is running.

Create an empty file called test.php and place this in it:

<? phpinfo();  ?>
root# php -f test.php  | grep "memcache support"
memcache support => enabled

If this command does not return any thing the memcache plugin did not load correctly.

You can now delete test.php, your memcache installation should be functional.

References:

http://www.vbseo.com/blogs/danny-bembibre/daemon-scripts-memcached-44/

http://bloke.org/linux/installing-memcached-on-centos-cpanel/

cPanel Setup Remote MySQL server – THE EASY WAY

cPanel typically has some decent documentation that helps you get something done, but with the remote mysql server they really screwed the pooch.   So heres a very rough guide and aggressive way of setting up a remote mysql server with all the data from the local one.

http://docs.cpanel.net/twiki/pub/AllDocumentation/TrainingResources/TrainingSlides07/MySQL_Remote_Server.pdf

Not very helpful is it?   Okay here's how we do this.
DB# yum install mysql server -y
DB#service mysqld stop
Web# scp -r /var/lib/mysql/ root@db:/var/lib/mysql

At this point you have NO idea what the root password on the db mysql server is.

Reset it with this guide > http://www.cyberciti.biz/tips/recover-mysql-root-password.html

Install phpMyAdmin to the DB server so we don’t have to work with 19th century cli tools.

On the Web box go to  SQL > “Setup Remote MySQL server”  then type in the ip and ssh port, select  login as: root and select authentication method as password.  Those are your SSH credentials, you can use ssh keys and all that crap but its only for running a script one time on the remote server.

Once you do that it will fail and be all nasty, don’t get upset its literally just to taunt you.

DB# cat /root/.my.cnf

This will contain a password that you need to setup in a moment. Memorize it or something useful…

Login to phpMyAdmin on the DB server, then click permissions.

Okay so here’s the part where you need to use your brain to make a decision.  I don’t know how your setup works so I can’t really tell you what to do, but what I can show you should put you on the right path.   Essentially the IP you use to SSH over is what cPanel will stick in the config as the DB server address or FQDN (Fully Qualified Domain Name)  So under privileges we need to create a user with the following:

User Name: root
Host: the ip or FQDN you plan on feeding cPanel.  Personally I think IP makes more sense here!
Password: remember what I had you waste your time memorizing earlier? Yeah type that here.
Global Privileges, you need to "check all" this.  This makes the magic happen!

Now go back to cPanel setup for the database and run it again, it should work…

Why not to use a downloaded my.cnf part II

In case you don’t recall or weren’t reading when it was put up, I did an article on why a downloaded my.cnf is a bad idea. It’s time for a bit of appending to that. I am cranking out my usual support tickets helping little old ladies with their sites about their pinochle groups (no not really) when I get a support ticket from a client who is having issues with their databases not working properly.

Naturally, being the curious sort I am and them being a paying client, I log into the server and start examining things. The first thing I do is try to do a database check. This doesn’t go over too well, I get a bunch of errors about “corrupt .frm files.” Naturally my curiosity is piqued at this point, so I go and start looking through /var/lib/mysql/hosts.err file. What is extremely odd here is that there it is super short, and there is nothing there really. At this point I decide to log into mysql and try rotating out the log file. It indicated that the operation had completed successfully, so I am a bit baffled at this point. Looking in the my.cnf, it appears that someone downloaded a web based my.cnf and stuck it on the server. It also appears that the logs are going somewhere else at this point. After examining the correct logs, I find a bunch of errors about the .frms being corrupted. I then start looking for the MYIs so that I can see if I can rebuild the .frms with them. To my dismay I find out there are no MYIs for the DBs here just the .frms.

After updating the client on what I’d found, I start looking over the net for an answer. It turns out that if InnoDB doesn’t have the engine started it will default to MyISAM databases, which is where the .frms came from. Definitely getting closer at this point, so I start looking at the engine statuses and of course InnoDB is disabled.

After this the picture is starting to come into focus a bit more, something in the my.cnf was changed and it broke everything. But what? I had noticed that the server was running MySQL as root when I had done “ps aux | grep mysql” earlier so I had a suspicion. Referencing another server I confirmed what I thought and found out that a line was missing.

user=mysql

The net downloaded configuration didn’t have a set MySQL user, root didn’t have permission to access the file and therefore it wouldn’t load the InnoDB files and in turn the InnoDB engine. A single line was missing and this production server was likely down for hours while they tried to figure out what happened. The internet is not a substitute for knowing what you are doing. Yes it can help you in this regard, but at the same time it is not a substitute for knowledge or competency. I once had an instructor tell me there is no crib sheet for life, it rings as true now as it ever did when he said it.

Dell really impressed me today

I gotta say that over the years I feel that Dell has burned a hole into my heart producing cheapo equipment just to make the sale.   Well in the past 2-3 years it seems that Dell has really kicked the quality of their product line up a few notches.   Today I was working with a client on our Managed Colocation service, we needed to order his servers so we went ahead assembled them on the website then called dell.

To my surprise the business sales rep was quick to load up the saved cart and begin to check it over with us.   After we made sure everything was listed correctly the rep instantly told us that he could save $600 on the R710 and $400 on the R610, We didn’t even begin to haggle him.   After we agreed on that pricing we then made the one and only request to get more for our money, we asked for idrac enterprise on both servers for free, our wish was granted instantly.

Typically we buy all of our servers through a reseller but for this particular client I felt ordering direct would be a much better experience for them and it paid off.  I have to give my congrats to Dell on earning back my trust in there sales team, I’m absolutely happy to work with them and to use there new servers.

On a side note for any one looking at buying a new OEM server the R610/R710 are absolutely phenomenal buys, and they come with the newest intel E5600 series CPUs.
Update: June 17th 2010.

Dell shipped the servers 2 days ago (Wednesday) they some how got the incorrect address and sent them to the entirely wrong state, thankfully we caught it and had them redirected to the correct destination.   I think this is terrible on dells part, not to mention they confirmed the address multiple times.

Update: June 21st 2010.

The servers in fact arrived at the WRONG address.  After several calls to dell with no resolution I contacted one of my Business reps at dell and within an hour received a call with an “upgraded” server being shipped overnight to the CORRECT address.  They added a 2nd CPU and 15k SAS disk vs 7200 SATA Disk.
We will see when it gets here if it was worth the wait, the other server is scheduled to arrive tomorrow without issue.

Basic MySQL backups

Backing data up in MySQL is super important. The site content is often replaceable, however your board posts, ecommerce orders, etc. are far far harder to replace, if at all possible. The other nice thing is that with a lot of database driven web sites you retain most content even if you only back up the databases. You can still lose things like images, but by and large your posts will still be there even in event of a crash. Since bad code, etc. happens there are a few reasons I would recommend doing backups of the databases.

  • Making server side changes, including configuration tweaks that can influence system stability.
  • Making software changes, even adding plugins etc. to your site. I have seen odd things in the past caused by plugins and they can interact together with not always predictable results.
  • Periodical backups as part of a data protection plan. I would do more than less, by and large most databases are not huge for the data they contain. I have seen forums with about 100,000 posts have about 2GB of database; not really that much if you think about it.

There are a few ways to do backups, but the most common for people who don’t use command line interface is probably going to be PHPMyAdmin. Quite frankly, I do not like using it for database backup and restorations. There are two reasons for this pertaining to reliability

  • If the session timeout of PHP or Apache is too low, you will get an incomplete dump. This means that backup you may have of your databases is all of a sudden found to be truncated at a few megabytes. This is not a big deal for a lot of people with small DBs however if you’re rocking out with 50+ megabyte DBs this can be a huge issue.
  • Restoration can be dependent on if the person has large uploads and large timeouts enabled. Since most people have slower upload than down, it can be a real nuisance to get the DB back up to the server and restored. Not only that but If you have a server with low upload size limits and timeouts, it may be impossible to get that DB back up and restored to the system before you hit one of these limits.

The way I honestly prefer to back my databases up is mysqldump; this allows one to have a lot of control over what gets dumped, do a full backup without worrying about doing it individually etc. It also allows us to make a nice easy cron job to do these backups too if we want.

The way we back a particular DB is up is:

mysqldump dbname > backup.sql

This will put the database named dbname into backup.sql. I believe you can separate multiple DBs with a space as well, however I don’t normally use that functionality. Another thing you can do is you can back up everything with:

mysqldump --all-databases > backup.sql

and this will do everything on the server. Handy for migrations, etc. Definitely way easier than trying to do this via a web interface and having 1,000 DBs.

How we restore these is fairly simple as well, we just run the command as follows:

mysql dbname < backup.sql

If you have a full DB dump you wouldn’t specify anything for the name when restoring. Notice how the carrat points in the opposing direction though. Don’t screw this up otherwise you will likely end up with a blank file and have to re-dump or re-move the file. I usually make 2 copies on the destination server especially when it’s not easy to get a second copy. There are other ways of moving the DBs over, such as copying the raw MYI files but that will be another blog.