linux commands

The sendmail that just wouldn’t

CoreCount 0 Comments

Today, I came across the strangest problem I’d ever seen in years of administering sendmail. Normally, sendmail is very much set-and-forget, taking care of emails without any problems. Obviously, today wasn’t normal.

No, not at all. The server had sendmail installed, and properly configured, but wouldn’t send any emails. It’s as if it wasn’t even running. 

ps -fea | grep sendmail

Nope, no sendmail. So, let’s start sendmail and check again.

/sbin/service sendmail start

Uh oh. The usual start up messages didn’t appear. Checking if sendmail was running again showed 0 sendmail instances.

After looking through /var/log/message and finding nothing about sendmail anywhere, as well as a stubbornly empty /var/log/maillog, I decided the sendmail binary must be corrupt so issued

yum reinstall sendmail

After yum downloaded and installed the new RPM, I was sure that sendmail would start right up without any problems. I start the service and… nothing happens.

Out of frustration, I decided to run sendmail on the terminal and see where it was failing.

[root@server ~]$ /usr/sbin/sendmail
bash: /usr/sbin/sendmail: No such file or directory

Well, that couldn’t be right. bash had even auto-completed that for me. sendmail had to be installed, as all the other files where there. Wait a minute….

[root@server ~]$ ls -l /usr/sbin/sendmail
lrwxrwxrwx 1 root root 21 May 30 11:51 /usr/sbin/sendmail -> /etc/alternatives/mta

Most modern servers include alternatives to help manage different programs that provide the same feature. This is accomplished through the /etc/alternatives directory that has symlinks to the actual binaries, as well as /var/lib/alternatives to configure the entire system, which is managed by /usr/sbin/alternatives.

The only problem on the server turned out to be a broken link, /etc/alternatives/mta, which was pointing to a nonexistant qmail installation. The entire problem was fixed by issuing

/usr/sbin/alternatives --set mta /usr/sbin/sendmail.sendmail

I wasn’t happy with this, as it wasn’t “autoconfigured” by alternatives automatically, so I copied /var/lib/alternatives/mta from an identical server that only has sendmail running and ran

/usr/sbin/alternatives --auto mta

alternatives properly detected sendmail, fixed the symlinks in /etc/alternatives and sendmail successfully launched.

Cool One Liners #1

Alex Underwood 1 Comment

Welcome to the first edition of Cool One Liners. This will be a collection of one line commands you can use via BASH or another shell/scripting language to do something useful. Creativity will definitely be a big merit. Todays one liner is:

cat /var/log/secure | grep Failed | grep sshd | grep root | awk ‘{print $11}’ | sort | uniq -c | sort -n

What does it do? This takes the secure log, sorts out failed login attempts and then makes it so that the IPs are sorted based on the number attempts. Handy to try and track down brute force attempts on an box running SSH. As an example, I generated a few failed logins.

[root@DNS01 log]: cat /var/log/secure | grep Fail

May  9 03:31:58 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May  9 03:32:00 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May  9 03:32:04 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2

After this I ran the command given. Notice how the IPs have the number to the left of them. If this were a list the number with the most logins is going to be at the bottom.

[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n

3 127.0.0.1

This command also serves an additional interesting use. Lets say someone is probing your machine, and they happen to be attempting to brute force some nonstandard account names in the hope of coming up with something on the system that is there and has a weak password. This script will also list any invalid users that attempt to log in as well. An example would be if I attempted to log in with the user root1. The output would look like:

[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n

3 127.0.0.1
3 root1

In another blog we will likely take this command, convert it into a shell script, and make it so it will run as a cron job and email us periodic digests.