Paying attention to your Nagios alerts can pay dividends in regards to server uptime and security. Just as an example, If you have an SSH alert come up it’s likely a very good idea to check your logs for failed login attempts. See cool-one-liners-1/ for more info. Chances are someone is eating your connections up and the monitoring server can’t connect.
Author: Alex Underwood
Why you shouldn’t download a MySQL config from the net
Working on servers, people seem to think there is a cheat sheet to it at times. While there are many ways to simplify and automate a lot of configurations on a server, MySQL is not the time nor the place to cut corners. If you bog MySQL in turn the system will feel the wrath of your misconfigurations. Here are a few super common mistakes.
- max_connections set way high-Why people think this is even remotely near a good idea is beyond me. It is common to go out to a server, find out it has either flat out hung or is spitting out of memory errors on the screen and has gone brain dead. Just because you can set your connections to 1500 doesn’t mean it’s a good idea. In fact it’s probably better for MOST people to set their connections to 150. The reason is simple; each connection available uses RAM. If a query comes by and there are 150 connections available, you will probably get an out of connections error and can log into the system, figure out what broke and fix it; If you hang the system you get an out of memory error and no real data to work off of. CPanel sets this to 500 by default, if you are using 500 connections (and the system is still up) YOU PROBABLY HAVE A PROBLEM.
- huge buffer sizes-Guess what if you have 100 megabytes of InnoDB tables setting your innodb_buffer_size to 5GB isn’t going to help you out. Plain and simple. If you start adding more tables you could possibly even run out of RAM (see above.)
- Big RAM-MySQL does not like RAM use above 2GB on 32 bit systems. They do not recommend running it because it can cause stability problems. In turn this means that there is little reason not to run a 64 bit OS any more unless you’re running a low end server. You NT guys are pretty much stuck with 2GB no matter if you’re running X86 or X64. That being said for hard core DB applications you should probably be putting it on a unix based OS anyways.
Besides this there is the single biggest reason of all:You will probably have lousy performance compared to a custom my.cnf. When tuning databases, one needs to keep in mind that the my.cnf is tailored to a combination of the database content, how the database is used and the server its self. If there was a magical my.cnf that would make any server work great don’t you think that Oracle/Sun/MySQL would have included it with the server software? We will go into a few things in later blogs such as software to help tuning MySQL, what the parameters mean and how to actually tune a database properly.
Mod_limitipconn
I have had a few clients with chronic security issues on machines, DoSes are by far the most common attack. It’s easy enough for anyone of average (or below average, up for debate) intelligence to carry out. A few zombie machines and some basic software and all of a sudden you can take down a decently massive web site.
There are a few caveats to mod_limitipconn, the biggest being if you run a site with a very large amount of images. Images are loaded last on a page, and also are loaded at once. This can be a perilous position because of the fact that mod_limitipconn can cause issues with dead images if not done properly. This is why you have to tune it. In general there is no CPanel integration so you’re going to be doing this by hand. For you people out there with other panels, your milage may vary.
First thing is installing it. This is a relatively simple process, be sure you have APXS installed so you can do this. Download the software from http://dominia.org/djao/limitipconn2.html and install it using their commands. You can also probably get packages from a place like Dag if that’s your thing. Certain distros such as Debian also come with a ready to rock version. After we get it installed, we need to go into httpd.conf and make some alterations.
MaxConnPerIP 10 NoIPLimit images/*
There are a few things to note here. For people that have static content on their systems, you would set an instance of “images” for each static content directory that you were using. This includes things like Javascript, CSS, and conventional images. The reason for this is if using mod_limitipconn for system hardening an attacker will usually not target an image. This can happen, but it’s also not as “profitable” as a DoS using a page because you don’t pull the images and other content on the page (increasing bandwidth usage) and static content tends to be super efficient compared to PHP or other server side scripting so it tends to be far more robust.
You can also do other cool tricks such as format based setting of the connection limits.
# local per-directory settings here MaxConnPerIP 1 OnlyIPLimit audio/mpeg video
By putting this in with the main confiugration above, you can set up additional subdirectories and limit connections based on media type. This can slow a leecher down significantly because they can’t pull everything at once.
Since we have the basics covered, just a few words on tuning the configuration. The biggest thing you need to do is view the sites you use this on. Sites with mod_limitipconn can behave erratic behavior if an initial setup is done but left untuned. This can include random broken images and other content not showing up. There are two things we are concerned with to tune; the first is that the most static content possible is set to very loose or no limitations, the second being that you get the base connections as low as sanely possible. If you can get it around 5 that would likely be the most pages someone would “sanely” load of a typical site without having issues. The other thing is that if you have embedded pages in your site you will have to increase this number significantly. Just go slowly and when the page fully loads add say 10-20% to that amount. If you get reports of broken content it would likely be safe to up this number some at that point to “dial in” that perfect configuration.
The internet as of 2000
Back when I was younger and working on gaining knowledge of such things as Linux, I used to listen to this online radio show all the time. It really gives a feel for the industry as of about the years 2000-2001 and how it was evolving at the time. The show has long since gone off the air, but they do have archives up still. ftp://ftp.dhbit.ca/DHBiT/ I also used to listen to Off The Hook, Emmanuel Goldstien’s radio show. I think it’s still on, but it’s been forever for that as well.
Setting up DNS for a dedicated server or VPS with Godaddy
The series of blogs I write is actaully a lot of tech support questions that are common. This question gets bonus points because it is actually very difficult to explain over the phone due to the complexity in which DNS works. That being said pictures are worth their weight in gold. For this reason I have included the pictures below as well captions explaining what you need to do.
First off this is the domain menu. If you look here, the name servers are already defined. Chances are that unless you have a new domain, they will be pointed to the DNS of your host or your old server. What we are going to do is scroll all the way down the page, at the bottom left where it says “Host Summary” then click “Add.” This will open the next menu.
There are two fields we are going to edit here. The first one is for the Host record that the DNS server is going to be called. Normally I use “NS1” and “NS2.” don’t sweat capitals or not here, it’s not case sensitive. The other field we need to change is the one labeled “Host IP 1.” After this is done, we click “ok” and simply repeat the procedure for the secondary name server. When I set a new dedicated up I prefer to set it up with the first two IPs as NS1 and NS2 respectively.
Now that this has been done, we can finally add our name servers. Please note that if the other information hasn’t been updated in Godaddy’s information that it can take some time for it to allow you to do this. If you have NS1 and NS2 showing up in the Hosts Summary but it gives you an error about the name server not existing this is why. If you already have this set up, the only other change required is to change the Host Records so that the IPs are pointing to the new server and wait on propagation.
A common pitfall of the OpenVZ installation
A very very common error I see people make when installing OpenVZ on a 64 bit system is that they will assume the kernel used is a 64 bit kernel. By default OpenVZ does NOT install a 64 bit kernel, but rather a 32 bit one. If the wrong kernel is installed this will cause the machine to drop a kernel panic. This is mentioned in the documentation, however this is an issue I have seen clients have repeatedly.
Guidelines for system file modification
Just figure I would throw a few basic guidelines out there I use whenever I can. Not saying I always follow them as exceptions do happen and I (like many other people out there) am not perfect, but the quickest way to resolve a problem is to be able to get around it. My rules are this:
When changing a file:Always make a copy of it. Try to include something that explains its significance, such as the date or your name. Having a bunch of httpd.conf.bakN where N is the number in which they were created doesn’t really do any good to anyone
When deleting a file:Dont unless necessecary or explicitly requested. Moving a file to a new directory is a far better option. This also lets you be able to examine the files at a later date, and is quicker to perform than a raw delete in most cases I have seen.
When you’re migrating files:Don’t delete the source ones until the new ones are working properly, everything is migrated over and preferably some period of time has passed. Trust me you don’t want to be that guy who didn’t have the DNS server changed over and deleted the active site or you needed some extra files that weren’t there.
The dot:When running commands, before hitting return be sure that your commands are dotted properly. Running chown username ./* and running chown username /* are two commands that will have very different results! This is very common, there are many admins who forgot the dot! This goes for any “heavyweight” commands such as rm, chmod, chown, etc.
Remember the server is meaningless without customers who have their data on it. As a sysadmin it is ultimately YOUR responsibility to take care of ensuring data is in tact. I will cover what I believe to be “suitable” measures to safeguard your data in another blog some time down the road. Until then stay tuned.
How to submit a support ticket properly
We all submit support tickets in one form or another as part of our jobs in the IT world. There just comes a point where everyone needs some help. This being said, as a person working in the industry for the lowest response times and maximum efficiency there are a few things that you should do when submitting a support ticket:
- Root password
- Ports of any services that have been changed
- Any form of authentication OTHER than passwords required (eg suing up, etc.)
- Symptoms being reported
- If you made changes before things started happening what were they
- Your Ip address if you are experiencing difficulties accessing the site
- Did you ping your server/VPS and did you get a response? Was it within the typical time?
Something to keep in mind is that by including these answers you will save the technician time in trying to bring your server up as well as reducing the time it takes you to get your server brought back up. The first two are critical, it’s really bad if you have a server running out of RAM but your tech can’t access it because you changed the SSH port to 2020 instead of leaving it on 22. We are here for you the client, however if your server can’t be brought up with the information on hand you get asked some questions and your ticket gets moved aside until a response is had.
Cool One Liners #1
Welcome to the first edition of Cool One Liners. This will be a collection of one line commands you can use via BASH or another shell/scripting language to do something useful. Creativity will definitely be a big merit. Todays one liner is:
cat /var/log/secure | grep Failed | grep sshd | grep root | awk ‘{print $11}’ | sort | uniq -c | sort -n
What does it do? This takes the secure log, sorts out failed login attempts and then makes it so that the IPs are sorted based on the number attempts. Handy to try and track down brute force attempts on an box running SSH. As an example, I generated a few failed logins.
[root@DNS01 log]: cat /var/log/secure | grep Fail
May 9 03:31:58 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May 9 03:32:00 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
May 9 03:32:04 DNS01 sshd[10706]: Failed password for root from 127.0.0.1 port 34900 ssh2
After this I ran the command given. Notice how the IPs have the number to the left of them. If this were a list the number with the most logins is going to be at the bottom.
[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n
3 127.0.0.1
This command also serves an additional interesting use. Lets say someone is probing your machine, and they happen to be attempting to brute force some nonstandard account names in the hope of coming up with something on the system that is there and has a weak password. This script will also list any invalid users that attempt to log in as well. An example would be if I attempted to log in with the user root1. The output would look like:
[root@DNS01 log]: cat /var/log/secure | grep Failed | grep sshd | grep root | awk '{print $11}' | sort | uniq -c | sort -n
3 127.0.0.1
3 root1
In another blog we will likely take this command, convert it into a shell script, and make it so it will run as a cron job and email us periodic digests.
Tracking Network Floods
When it comes to working at a DC, attacks are a fact of life. Someone will ultimately get annoyed at a client and start throwing down with the botnet, their home system etc. These attacks can be hard to isolate at times because they can be on nearly any service, as well as using a wide variety of protocols. These attacks can do anything from making a server slow to respond to taking an entire data center out. When a system goes down due to an attack, there are a few things that must be known:
- Who is DoSing or who is being DoSed (don’t assume your machine is the “victim” insidious PHP scripts are plentiful these days)
- The magnitude of the attack
- What protocol they are using
To find these out, you need proper network infrastructure set up. To this end I like Cacti on the switches for ease of usability in finding overall traffic, even though NFsen is nice for a quick check by IP however it can be flawed in picking up distributed or spoofed attacks in my experience. When attempting to isolate the problems one should have a “tiered” approach where they start checking at the most basic level which is a single server. If the attack is larger one should try to see if it’s a single rack/switch being attacked and seeing if it can back track to a single server being the target. At this point, there are a few options. Null routes are an option if all fails, even null routing the server’s IP so that other traffic doesn’t get affected can be done as a last resort. The option that you probably have the most control over is server side mitigation. This will be the topic of my next blog. See you then!