Optimizing a Server: prelink

When most people think about server optimization, it’s almost always a question of tuning MySQL, deciding how many HTTPD processes should be running, limiting the maximum connections to the server, and installing a caching proxy like Varnish or Squid.

While all of these optimizations should be performed, they’re not the only optimizations we can perform on a server. In fact, there’s something simple that can improve performance of the system globally, without any configuration!

When a program is run, there are a few things that have to happen before it can actually start doing work. First, the program is loaded into server memory. Next, ld.so processes the binary, loading all of the libraries (.so files, shared objects) that it needs–a step called dynamic linking. Finally, the program will start running from its main() function.

While there’s not much that can be done to speed up loading the binary into memory, other than getting faster disks, the dynamic linking step can be made faster. How? With prelink.

prelink is run on every binary and library on the system, pre-computing the locations of all of the libraries, so that they can be loaded into the same place in memory (barring no conflicts) every time a program starts. This significantly reduces the amount of time needed for the dynamic linking step of the program loading process, allowing programs to reach execution much faster.

To install prelink, run:

yum install prelink

Next, force prelink to pre-link all binaries on the system right now by running:

/usr/sbin/prelink --all

All programs on the system are now prelink'd. The prelink package adds a cron job that runs daily to check for any binaries that may have changed since the last prelink run, and updates them as needed.

Enjoy faster process start up times, thanks to prelink.

The sendmail that just wouldn’t

Today, I came across the strangest problem I’d ever seen in years of administering sendmail. Normally, sendmail is very much set-and-forget, taking care of emails without any problems. Obviously, today wasn’t normal.

No, not at all. The server had sendmail installed, and properly configured, but wouldn’t send any emails. It’s as if it wasn’t even running.

ps -fea | grep sendmail

Nope, no sendmail. So, let’s start sendmail and check again.

/sbin/service sendmail start

Uh oh. The usual start up messages didn’t appear. Checking if sendmail was running again showed 0 sendmail instances.

After looking through /var/log/message and finding nothing about sendmail anywhere, as well as a stubbornly empty /var/log/maillog, I decided the sendmail binary must be corrupt so issued

yum reinstall sendmail

After yum downloaded and installed the new RPM, I was sure that sendmail would start right up without any problems. I start the service and… nothing happens.

Out of frustration, I decided to run sendmail on the terminal and see where it was failing.

[root@server ~]$ /usr/sbin/sendmail
bash: /usr/sbin/sendmail: No such file or directory

Well, that couldn’t be right. bash had even auto-completed that for me. sendmail had to be installed, as all the other files where there. Wait a minute….

[root@server ~]$ ls -l /usr/sbin/sendmail
lrwxrwxrwx 1 root root 21 May 30 11:51 /usr/sbin/sendmail -> /etc/alternatives/mta

Most modern servers include alternatives to help manage different programs that provide the same feature. This is accomplished through the /etc/alternatives directory that has symlinks to the actual binaries, as well as /var/lib/alternatives to configure the entire system, which is managed by /usr/sbin/alternatives.

The only problem on the server turned out to be a broken link, /etc/alternatives/mta, which was pointing to a nonexistant qmail installation. The entire problem was fixed by issuing

/usr/sbin/alternatives --set mta /usr/sbin/sendmail.sendmail

I wasn’t happy with this, as it wasn’t “autoconfigured” by alternatives automatically, so I copied /var/lib/alternatives/mta from an identical server that only has sendmail running and ran

/usr/sbin/alternatives --auto mta

alternatives properly detected sendmail, fixed the symlinks in /etc/alternatives and sendmail successfully launched.