Install memcached with CentOS and WHM/cPanel

This guide will walk you through installing memcache and memcached on a dedicated or vps server.

For the wiki version of this guide visit:

root# yum install libevent libevent-devel -y

Head over to and grab the latest version.

root# yum install libevent-devel libevent gcc make -y
root# wget
root# tar xvf memcached-1.4.5.tar.gz
root # cd memcached-1.4.5
root# ./configure && make && make install

Open /etc/memcached.conf with your favorite editor and paste in:

#Memory a usar
-m 16
# default port
-p 11211
# user to run daemon nobody/apache/www-data
-u nobody
# only listen locally
touch /etc/init.d/memcached
chmod +x /etc/init.d/memcached

Open /etc/init.d/memcached with your favorite editor and paste in:

# memcached    This shell script takes care of starting and stopping
#              standalone memcached.
# chkconfig: - 80 12
# description: memcached is a high-performance, distributed memory
#              object caching system, generic in nature, but
#              intended for use in speeding up dynamic web
#              applications by alleviating database load.
# processname: memcached
# config: /etc/memcached.conf
# Source function library.
. /etc/rc.d/init.d/functions
[ -x $DAEMON ] || exit 0
[ -x $DAEMONBOOTSTRAP ] || exit 0
start() {
 echo -n $"Starting $DESC: "
 [ $RETVAL -eq 0 ] && touch $PIDFILE
 return $RETVAL
stop() {
 echo -n $"Shutting down $DESC: "
 killproc $NAME
 [ $RETVAL -eq 0 ] && rm -f $PIDFILE
 return $RETVAL
# See how we were called.
case "$1" in
  status $prog
  echo $"Usage: $0 {start|stop|restart|status}"
  exit 1
exit $RETVAL
touch /usr/local/bin/start-memcached
chmod +x  /usr/local/bin/start-memcached

Open /usr/local/bin/start-memcached with your favorite editor and paste in:

#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci <[email protected]>
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if ($> != 0 and $< != 0) {
 print STDERR "Only root wants to run start-memcached.\n";
my $etcfile = shift || "/etc/memcached.conf";
my $params = [];
my $etchandle;
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/";
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile {
 my ($logfile) = @_;
 $fd_reopened = $logfile;
sub reopen_logfile {
 my ($logfile) = @_;
 open *STDERR, ">>$logfile";
 open *STDOUT, ">>$logfile";
 open *STDIN, ">>/dev/null";
 $fd_reopened = $logfile;
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
 "logfile" => \&handle_logfile
if (open $etchandle, $etcfile) {
 foreach my $line (<$etchandle>) {
  $line =~ s/\#.*//go;
  $line = join ' ', split ' ', $line;
  next unless $line;
  next if $line =~ /^\-[dh]/o;
  if ($line =~ /^[^\-]/o) {
   my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
  push @$params, $line;
unshift @$params, "-u root" unless (grep $_ eq '-u', @$params);
$params = join " ", @$params;
if (-e $pidfile) {
 open PIDHANDLE, "$pidfile";
 my $localpid = <PIDHANDLE>;
 chomp $localpid;
 if (-d "/proc/$localpid") {
  print STDERR "memcached is already running.\n";
 } else {
  `rm -f $localpid`;
my $pid = fork();
if ($pid == 0) {
 exec "$memcached $params";
} elsif (open PIDHANDLE,">$pidfile") {
 print PIDHANDLE $pid;
} else {
 print STDERR "Can't write pidfile to $pidfile.\n";

Now we start the memcached daemon. (it is ok if “shutting down memcached” says fail.)

[[email protected] ~]# /etc/init.d/memcached restart
Shutting down memcached:                                   [  OK  ]
Starting memcached:                                        [  OK  ]

Make sure its running by:

[[email protected] init.d]# ps aux  | grep memcached
nobody    5966  0.5  0.3 18248 16444 pts/0   S    13:55   0:00 /usr/local/bin/memcached -u root -m 16 -p 11211 -u nobody -l

Now we will set memcached to run at startup:

[[email protected] ~]# /sbin/chkconfig memcached on
[[email protected] ~]# /sbin/chkconfig --list | grep memcached
memcached       0:off   1:off   2:on    3:on    4:on    5:on    6:off

Now we will install the memcache plugin for PHP.

Download the latest stable version of memcache from

root# wget
root# tar xvf memcache-2.2.5.tgz
root# cd memcache-2.2.5
root# phpize
root# ./configure && make && make install

Now open /usr/local/lib/php.ini with your favorite text editor and find the dynamic extension section.

Add this

Now restart apache with:

service httpd restart

We will now check to make sure memcached is running.

Create an empty file called test.php and place this in it:

<? phpinfo();  ?>
root# php -f test.php  | grep "memcache support"
memcache support => enabled

If this command does not return any thing the memcache plugin did not load correctly.

You can now delete test.php, your memcache installation should be functional.


cPanel Setup Remote MySQL server – THE EASY WAY

cPanel typically has some decent documentation that helps you get something done, but with the remote mysql server they really screwed the pooch.   So heres a very rough guide and aggressive way of setting up a remote mysql server with all the data from the local one.

Not very helpful is it?   Okay here's how we do this.
DB# yum install mysql server -y
DB#service mysqld stop
Web# scp -r /var/lib/mysql/ [email protected]:/var/lib/mysql

At this point you have NO idea what the root password on the db mysql server is.

Reset it with this guide >

Install phpMyAdmin to the DB server so we don’t have to work with 19th century cli tools.

On the Web box go to  SQL > “Setup Remote MySQL server”  then type in the ip and ssh port, select  login as: root and select authentication method as password.  Those are your SSH credentials, you can use ssh keys and all that crap but its only for running a script one time on the remote server.

Once you do that it will fail and be all nasty, don’t get upset its literally just to taunt you.

DB# cat /root/.my.cnf

This will contain a password that you need to setup in a moment. Memorize it or something useful…

Login to phpMyAdmin on the DB server, then click permissions.

Okay so here’s the part where you need to use your brain to make a decision.  I don’t know how your setup works so I can’t really tell you what to do, but what I can show you should put you on the right path.   Essentially the IP you use to SSH over is what cPanel will stick in the config as the DB server address or FQDN (Fully Qualified Domain Name)  So under privileges we need to create a user with the following:

User Name: root
Host: the ip or FQDN you plan on feeding cPanel.  Personally I think IP makes more sense here!
Password: remember what I had you waste your time memorizing earlier? Yeah type that here.
Global Privileges, you need to "check all" this.  This makes the magic happen!

Now go back to cPanel setup for the database and run it again, it should work…

What Chattr can do for you

Chattr is one of those great utilities in a security tool box. Since log files are a prime target of removal when root kits and other scripts are used on a system after the compromise, it’s an ideal way to help prevent destruction of the logs. Besides this, if you don’t want certain files tampered with you can use chattr to make them non writeable by anyone including root. This is done at a file system (inode) level regardless of what is there. My favorite options are:

+a. This sets a file append only. This is great for logs, and will catch a lot of script kiddies off guard. If overwriting of data is attempted it will deny the operation. as an example:

[[email protected] ~]# chattr +a ./new.file
[[email protected] ~]# echo “aghaklsjdfhadlwadjhad” >> new.file
[[email protected] ~]# echo “aghaklsjdfhadlwadjhad” > new.file
-bash: new.file: Operation not permitted

Since the first echo adds it to the end it will work. The second echo attempts to overwrite the file contents so it gives a permission error. Please note that this is even though the user has full permissions to access the file. to reverse this we use chattr -a new.file

The +i function is a little different, it will make the file totally unchangeable. If we have binaries that we consider sensitive this may be an option for it. I have had systems I needed to get back up with rootkits that would change files on the boot, this would take care of bringing them back up without allowing whatever was on the system to modify it.

The +u option allows for undeleting of files, this is another thing that may be good for sensitive information such as logs or key backup archives. Would be tempted to use it with +a as well for logs.

The +c option gives the files on disk compression. Never really played with it much however I would think it could be detrimental with large archives on RAM and CPU bound systems.

Fix Your Slow Firefox DNS Resolver

Recently I started having a problem where my Firefox would take 10-20 seconds to resolve each domain, this was extremely frustrating! After digging through my linux install I was unable to recreate the issue in any other program and finally blamed it on firefox.

Well after lots of searching I found this.
Firefox Slow DNS issues

The Fix:

Type this in your firefox URL bar.


Type this in under the filter:


Then double click the option and it should turn to “true”.
Magically firefox is no longer slow!

How Fast Are Your Disk?

I came across this utility on Web Hosting Talk and felt that a bunch of good benchmarks would be cool. Almost all of these benchmarks are with near idle disk.

Fujitsu 160GB 2.5″ 7200RPM
Intel ICH9R No Raid

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [152627MB], wait 30 seconds.............................
Results: 58 seeks/second, 17.06 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  172 MB in  3.02 seconds =  57.01 MB/sec

4 x 320GB 7200RPM WD RE3
3ware 9750 512MB Raid 10

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [610330MB], wait 30 seconds..............................
Results: 80 seeks/second, 12.42 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  560 MB in  3.01 seconds = 186.29 MB/sec

4x 72GB 10K SCSI
IBM 6i + BBU Raid 5

seeker /dev/sdb
Seeker v2.0, 2007-01-15
Benchmarking /dev/sdb [210018MB], wait 30 seconds..............................
Results: 111 seeks/second, 8.96 ms random access time
hdparm -t /dev/sdb
 Timing buffered disk reads:  260 MB in  3.02 seconds =  86.19 MB/sec

4x 72GB 10K SCSI
HP 641 + BBU Raid 10

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [35328MB], wait 30 seconds..............................
Results: 117 seeks/second, 8.54 ms random access time
hdparm -t /dev/cciss/c0d0
 Timing buffered disk reads:  206 MB in  3.10 seconds =  66.42 MB/sec

8x 320GB 7200RPM SATA
Dell Perc 5i 512MB Raid 5

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [35328MB], wait 30 seconds..............................
Results: 117 seeks/second, 8.54 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  740 MB in  3.00 seconds = 246.48 MB/sec

2x 72GB 10k SCSI
IBM 6i + BBU Raid 1

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [70007MB], wait 30 seconds..............................
Results: 118 seeks/second, 8.42 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  150 MB in  3.02 seconds =  49.66 MB/sec

2x 15k Maxtor Maxline SCSI
Compaq Smart Array Raid 1

seeker  /dev/cciss/c0d0
Seeker v2.0, 2007-01-15
Benchmarking /dev/cciss/c0d0 [70089MB], wait 30 seconds..............................
Results: 178 seeks/second, 5.61 ms random access time
hdparm -t /dev/cciss/c0d0
 Timing buffered disk reads:  206 MB in  3.01 seconds =  68.48 MB/sec

4x 300GB Seagate Cheetah 15k
Adaptec 5405 Raid 10

seeker /dev/sda
Seeker v2.0, 2007-01-15
Benchmarking /dev/sda [571392MB], wait 30 seconds..............................
Results: 198 seeks/second, 5.03 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  1236 MB in  3.00 seconds = 411.95 MB/sec

4x 146GB Seagate Cheetah 15k
Dell Perc 6i Raid 10

Seeker v2.0, 2007-01-15,
Benchmarking /dev/sda [278784MB], wait 30 seconds..............................
Results: 208 seeks/second, 4.80 ms random access time
hdparm -t /dev/sda
 Timing buffered disk reads:  888 MB in  3.00 seconds = 295.57 MB/sec

Intel x25-m 80GB SSD
Intel ICH10R

seeker /dev/sdc
Seeker v2.0, 2007-01-15
Benchmarking /dev/sdc [76319MB], wait 30 seconds..............................
Results: 5074 seeks/second, 0.20 ms random access time
hdparm -t /dev/sdc
 Timing buffered disk reads:  668 MB in  3.00 seconds = 222.47 MB/sec

How To: Config Server & Firewall Installation Guide

I’ve written this guide to help you you through installing and configuring CSF – Config Server Firewall, a popular alternative to APF. CSF comes with LFD and works with or without cPanel.

CSF Install Guide and How To

CSF – Config Server Firewall is a stateful packet inspection firewall, login/intrusion detection and security application for Linux servers. What does that mean in English? Simple – it’s a program that can greatly improve your dedicated server or VPS’s security.

It’s a firewall – so it can block/restrict ports you don’t want open, and prevents someone from using any port they want if they did break in.
It has intrusion detection – so it will scan the log files and monitor failed login attempts, such as FTP password guessing and block the IP.
Those are the 2 big things I like about CSF – and it has a nice interface for the non techie person, on cPanel servers.

If you have another firewall installed, like APF, CSF can help you automatically remove the existing firewall and install theirs instead. Keep in mind it won’t migrate over your configuration.


Installation is quite straightforward:

Login as the root user to SSH and run the following commands.

rm -fv csf.tgz
tar  -xzf csf.tgz
cd csf

If you would like to disable APF+BFD (which you will need to do if you have
them installed otherwise they will conflict horribly):


That’s it. You can then configure csf and lfd in WHM, or edit the files
directly in /etc/csf/*

Installation Completed

Don’t forget to:

1. Configure the TCP_IN, TCP_OUT, UDP_IN and UDP_OUT options in the csf configuration to suite your server

2. Restart csf and lfd

3. Set TESTING to 0 once you’re happy with the firewall

csf is preconfigured to work on a cPanel server with all the standard cPanel
ports open. It also auto-configures your SSH port if it’s non-standard on

You should ensure that kernel logging daemon (klogd) is enabled. Typically, VPS
servers have this disabled and you should check /etc/init.d/syslog and make
sure that any klogd lines are not commented out. If you change the file,
remember to restart syslog.

Now – login to your cPanel server’s WHM as root and go to the bottom left menu. If already logged in then reload the page. In Plugins – you will see:  ConfigServer Security&Firewall

The firewall is STOPPED by default – it is not running. We need to configured it, and then take it out of Test Mode.

Click on Firewall Configuration

ETH_DEVICE =: Set this to eth+

TCP_IN/TCP_OUT/UDP_IN/UDP_OUT = : These are the ports you want to leave open for your server to operate. If you change the default SSH port make sure to add it here. Also add any other services you might have running such as Shoutcast or game servers. By default most of the ports used should already be configured.

MONOLITHIC_KERNEL = : 0 Only change this to 1 if your firewall will not start – otherwise leave it as it.

LF_DSHIELD = 0: Change this option to 86400. This is an automatic updated list of known attacking IPs. Enabling this will stop them from being able to connect to your server.

Spam Protection Alerts

If you want to add some spam protection, CSF can help. Look in the configuraiton for the following:

LF_SCRIPT_ALERT = 0 change this to 1. This will send an email alert to the system administrator when the limit configured below is reached within an hour.

LF_SCRIPT_LIMIT = 100 change this to 250. This will alert you when any scripts sends out 250 email messages in an hour.

Configuration Complete – Almost

Scroll down to the bottom and click on Change to save the settings. Then click Restart csf+lfd

You should see a big page of ACCEPT and near the bottom you should see:

csf: TESTING mode is enabled – don’t forget to disable it in the configuration

Starting lfd:[  OK  ]

Click on Return

Now TEST all your services to make sure everything is working – SSH, FTP, http. After you do a few quick tests go back into the Firewall Configuration page.

TESTING = 1 change this to 0 and click Change at the bottom. Then Restart csf+lfd

That’s it, the firewall is successfully installed and running!!
Firewall Status: Running – you should see this on the main CSF page in WHM.


Removing csf and lfd is even more simple:

cd /etc/csf

How To: VNCServer on Fedora

Virtual Network Computing (VNC) is a desktop protocol to remotely control another computer. It transmits the keyboard presses and mouse clicks from one computer to another relaying the screen updates back in the other direction, over a network.” -WikiPedia-

This article describes in brief how to configure VNC server instances for one or multiple users on a remote machine, how to use VNC to start graphical applications on boot and finally how to enhance security by connecting to the server through encrypted SSH tunnels.


A user account should exist on the remote machine.
The RPM packages xorg-x11-xauth, vnc-server and vnc should be installed on the remote machine and your workstation respectively.

yum install -y xorg-x11-xauth vnc-server vnc xsetroot xterm 

Setting up the server

I assume that we have setup a remote user account, named “Tyler” and we want to start an X session through VNC for this user.

In Red Hat based distros in general, all we have to do is define the VNC server instances in /etc/sysconfig/vncservers. These will be started by the vncserver initscript. This has to be done as root. Edit this file so that it contains the following:

VNCSERVERARGS[1]="-geometry 1024x768 -depth 16"

With these we define that a vnc server instance should be started as user Tyler on display 1 and we also set some options for this server such as resolution and color depth. Each VNC server instance listens on port 5900 plus the display number on which the server runs. In our case, Tylers’s vnc server would listen on port 5901.

For multiple vnc instances /etc/sysconfig/vncservers would look like this:

VNCSERVERS="1:tyler 2:albatros 3:leopard"
VNCSERVERARGS[1]="-geometry 1024x768 -depth 16"
VNCSERVERARGS[2]="-geometry 800x600 -depth 8"
VNCSERVERARGS[3]="-geometry 1024x768 -depth 16"

These would listen on ports 5901, 5902, 5903 respectively.

User Configuration

There is one more thing that needs to be done on the remote machine. User Tyler’s vnc password needs to be set. So, as user Tyler give the command:

# su -l Tyler
# vncpasswd
# exit

We are prompted for a password. This is the password that we will use when we connect to Tyler’s vnc server instance. This password is saved in /home/Tyler/.vnc/passwd.

Start the VNC server

After the initial configuration is done we restart the vnc service. As root:

# service vncserver restart

To make VNC server to start on boot:

# chkconfig vncserver on

More User Configuration

After the VNC service is started, some new files are created in /home/Tyler/.vnc/ directory. These include Tyler’s vnc server log file, pid file and an X startup script. As user Tyler we edit the script in order to customize some settings. The default /home/Tyler/.vnc/xstartup script contains some commands that are executed when the VNC server is started. These include:

xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
twm &

xsetroot in this case sets the background color.
vncconfig is a supplementary program that can be used to control the vnc server. Apart from this, when run without arguments it acts as a helper application and its main purpose is to provide support for clipboard transfers between the client (vncviewer) and the vnc server.
xterm starts an xterm terminal.
twm starts the X server’s default window manager. We probably want to change that to a more user friendly window manager, eg fluxbox.

The VNC server, apart from letting us control a remote machine using a graphical interface, it serves as a way to start graphical applications on boot. For example, I want my favourite p2p program, amule, to start on boot. So, I add this to the /home/Tyler/.vnc/xstartup script. This is how my xstartup file looks like:

xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" -e ./menu &
amule &
fluxbox &

menu is a script of mine that is executed when xterm is started.
Remember to put the “&” symbol after each command, so that it goes to the background and the xstartup script continues on.

Restart the VNC service for the changes to take effect. As root:

# service vncserver restart

Connect to the VNC server

In our example, Tyler’s vnc server listens for connections on port 5903. So, open this port in the remote machine’s firewall.

We connect to the remote machine using a vnc viewer. Having installed the vnc package, connect to to the server with the following command:

# vncviewer

The general usage is :

vncviewer [Server's IP]:[Port]:[Display]

We are prompted for the password and eventually connect to the server. Closing the vncviewer’s window, does not affect the server or the programs we run on it. If we reconnect everything will be there.

Special Note: There is no need, actually it’s pointless and could give you some trouble, to logoff from your remote X session. If this happens, generally you need to restart the VNC service on the remote machine to get your remote desktop back. If you want to stop working on your remote desktop, just close the vncviewer’s window and you are done.


The VNC protocol is not a secure communication protocol. The use of a vnc password provides security at the level of server access (it’s vulnerable to brute-force attacks though), but the whole VNC session is transmitted in the clear, without encryption. The easiest, but most effective, way to secure our connection to the VNC server is to connect through an encrypted SSH tunnel. This way the whole session will be encrypted.

The rest assume that you have the SSH server up and running on your remote machine ( and you know what SSH tunnels are.

So, what we are going to do is to create an encrypted tunnel, and connect to our VNC server through it. We also want this tunnel to be automatically closed as soon as we shut down vncviewer. All this is done with the following command:

# ssh -f -L 25903: [email protected] sleep 10; vncviewer

This is what it does:

  • -L 25903: forwards our local port 25903 to port 5903 on the remote machine. In other words, it creates the tunnel.
  • -f forks the SSH session to the background, while sleep is being executed on the remote machine. This ssh option is needed because we want to execute the following command (vncviewer) in the same local machine’s terminal.
  • vncviewer connects to the forwarded local port 25903 in order to connect to the VNC server through the encrypted tunnel.

The sleep command is of major importance in the above line as it keeps the encrypted tunnel open for 10 seconds. If no application uses it during this period of time, then it’s closed. Contrariwise, if an application uses it during the 10 sec period, then the tunnel remains open until this application is shut down. This way the tunnel is automatically closed at the time we close vncviewer’s window, without leaving any SSH processes running on our workstation. This is pure convenience! More information can be found at the Auto-closing SSH Tunnels article.

Using SSH tunnels to conect to your VNC server has two advantages:

  1. The whole session is encrypted.
  2. Keeping port 5903 open on your remote machine is no longer needed, since all take place through the SSH tunnel. So, noone will know that you run a VNC server on the remote machine.

Further Reading

I recommend that you read the man pages. Everything is in there:

# man vncserver
# man Xvnc
# man vncconfig
# man vncviewer
# man ssh

Optimizing a Server: prelink

When most people think about server optimization, it’s almost always a question of tuning MySQL, deciding how many HTTPD processes should be running, limiting the maximum connections to the server, and installing a caching proxy like Varnish or Squid.

While all of these optimizations should be performed, they’re not the only optimizations we can perform on a server. In fact, there’s something simple that can improve performance of the system globally, without any configuration!

When a program is run, there are a few things that have to happen before it can actually start doing work. First, the program is loaded into server memory. Next, processes the binary, loading all of the libraries (.so files, shared objects) that it needs–a step called dynamic linking. Finally, the program will start running from its main() function.

While there’s not much that can be done to speed up loading the binary into memory, other than getting faster disks, the dynamic linking step can be made faster. How? With prelink.

prelink is run on every binary and library on the system, pre-computing the locations of all of the libraries, so that they can be loaded into the same place in memory (barring no conflicts) every time a program starts. This significantly reduces the amount of time needed for the dynamic linking step of the program loading process, allowing programs to reach execution much faster.

To install prelink, run:

yum install prelink

Next, force prelink to pre-link all binaries on the system right now by running:

/usr/sbin/prelink --all

All programs on the system are now prelink'd. The prelink package adds a cron job that runs daily to check for any binaries that may have changed since the last prelink run, and updates them as needed.

Enjoy faster process start up times, thanks to prelink.

Why not to use a downloaded my.cnf part II

In case you don’t recall or weren’t reading when it was put up, I did an article on why a downloaded my.cnf is a bad idea. It’s time for a bit of appending to that. I am cranking out my usual support tickets helping little old ladies with their sites about their pinochle groups (no not really) when I get a support ticket from a client who is having issues with their databases not working properly.

Naturally, being the curious sort I am and them being a paying client, I log into the server and start examining things. The first thing I do is try to do a database check. This doesn’t go over too well, I get a bunch of errors about “corrupt .frm files.” Naturally my curiosity is piqued at this point, so I go and start looking through /var/lib/mysql/hosts.err file. What is extremely odd here is that there it is super short, and there is nothing there really. At this point I decide to log into mysql and try rotating out the log file. It indicated that the operation had completed successfully, so I am a bit baffled at this point. Looking in the my.cnf, it appears that someone downloaded a web based my.cnf and stuck it on the server. It also appears that the logs are going somewhere else at this point. After examining the correct logs, I find a bunch of errors about the .frms being corrupted. I then start looking for the MYIs so that I can see if I can rebuild the .frms with them. To my dismay I find out there are no MYIs for the DBs here just the .frms.

After updating the client on what I’d found, I start looking over the net for an answer. It turns out that if InnoDB doesn’t have the engine started it will default to MyISAM databases, which is where the .frms came from. Definitely getting closer at this point, so I start looking at the engine statuses and of course InnoDB is disabled.

After this the picture is starting to come into focus a bit more, something in the my.cnf was changed and it broke everything. But what? I had noticed that the server was running MySQL as root when I had done “ps aux | grep mysql” earlier so I had a suspicion. Referencing another server I confirmed what I thought and found out that a line was missing.


The net downloaded configuration didn’t have a set MySQL user, root didn’t have permission to access the file and therefore it wouldn’t load the InnoDB files and in turn the InnoDB engine. A single line was missing and this production server was likely down for hours while they tried to figure out what happened. The internet is not a substitute for knowing what you are doing. Yes it can help you in this regard, but at the same time it is not a substitute for knowledge or competency. I once had an instructor tell me there is no crib sheet for life, it rings as true now as it ever did when he said it.

IO scheduler tuning in Linux

This is my personal FS box. It consists of 4X200GB SATA drives on an Adaptec AAR-2810 controller in RAID10. I am using bonnie++ to benchmark. The reason I am throwing this up is that when I switched schedulers to deadline with 3 drives in a RAID5 on this system (I changed to a 10 later) it picked up about 5% by going to a deadline scheduler in sustained writes. If you notice in this case while I had a few small gains aggregately I lost a significant amount of performance at one point or another compared to the default scheduler. Then again, this array has write performance at a level typical of a single drive if not a little slower. It makes up for it in the fact that it has roughly double the read of a typical drive, which can be far more important depending on the application. Since it’s just a dump box anyways I’m not overly concerned about IO. Check the results out and post your opinions. I will probably add other tuning specifics at a later date. Please note this does not under any circumstances make CFQ my preferred scheduler; it just means it was better in this case.

cfq (default)

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 34694 88 39045 27 19030 8 36912 68 63545 12 496.2 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 19929 100 +++++ +++ 32373 99 +++++ +++ +++++ +++ +++++ +++,1000M,34694,88,39045,27,19030,8,36912,68,63545,12,496.2,1,16,19929,100,+++++,+++,32373,99,+++++,+++,+++++,+++,+++++,+++

echo deadline > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 30986 88 38772 31 16724 7 29545 56 62167 11 482.5 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20401 99 +++++ +++ 26963 99 29849 99 +++++ +++ +++++ +++,1000M,30986,88,38772,31,16724,7,29545,56,62167,11,482.5,1,16,20401,99,+++++,+++,26963,99,29849,99,+++++,+++,+++++,+++

echo anticipatory > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 33134 94 38153 28 17225 8 30326 58 65073 10 346.8 0
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 19071 94 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++,1000M,33134,94,38153,28,17225,8,30326,58,65073,10,346.8,0,16,19071,94,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

echo noop > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 29617 86 39779 27 16359 7 29197 57 62977 10 496.6 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20130 99 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++,1000M,29617,86,39779,27,16359,7,29197,57,62977,10,496.6,1,16,20130,99,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++