Sudo Make Install

This is just a quickie I hijacked off another web site but it came in really handy since a client couldn’t access their wordpress admin panel and we don’t really support it. Just log in to SSH or PHPMyAdmin and run the query that follows:

wp_options SET option_value = ” WHERE option_name = ‘active_plugins’;

*BOOM* no more WP plugins. Then if you’re troubleshooting it go through and reactivate till it breaks again. If not tell em they are on their own.

If you are doing this via SSH you will need to go into the MySQL shell. That can usually be done by typing “mysql” or “mysql -u username -p” and entering the password for the user. If you need this you can grab it from wp-config. After that you need to select the db to use. You do that by going:

mysql>use dbname_here;

This will put you into the appropriate db. Then you run the query above. This is of course assuming you’re using the default schema for naming the tables. If  you are not wp_options needs to be changed to just “options” or schema_options as necessecary. If you don’t know what your schema is you can do:

mysql>show tables;

Chattr is one of those great utilities in a security tool box. Since log files are a prime target of removal when root kits and other scripts are used on a system after the compromise, it’s an ideal way to help prevent destruction of the logs. Besides this, if you don’t want certain files tampered with you can use chattr to make them non writeable by anyone including root. This is done at a file system (inode) level regardless of what is there. My favorite options are:

+a. This sets a file append only. This is great for logs, and will catch a lot of script kiddies off guard. If overwriting of data is attempted it will deny the operation. as an example:

[root@DNS02 ~]# chattr +a ./new.file
[root@DNS02 ~]# echo “aghaklsjdfhadlwadjhad” >> new.file
[root@DNS02 ~]# echo “aghaklsjdfhadlwadjhad” > new.file
-bash: new.file: Operation not permitted

Since the first echo adds it to the end it will work. The second echo attempts to overwrite the file contents so it gives a permission error. Please note that this is even though the user has full permissions to access the file. to reverse this we use chattr -a new.file

The +i function is a little different, it will make the file totally unchangeable. If we have binaries that we consider sensitive this may be an option for it. I have had systems I needed to get back up with rootkits that would change files on the boot, this would take care of bringing them back up without allowing whatever was on the system to modify it.

The +u option allows for undeleting of files, this is another thing that may be good for sensitive information such as logs or key backup archives. Would be tempted to use it with +a as well for logs.

The +c option gives the files on disk compression. Never really played with it much however I would think it could be detrimental with large archives on RAM and CPU bound systems.

In case you don’t recall or weren’t reading when it was put up, I did an article on why a downloaded my.cnf is a bad idea. It’s time for a bit of appending to that. I am cranking out my usual support tickets helping little old ladies with their sites about their pinochle groups (no not really) when I get a support ticket from a client who is having issues with their databases not working properly.

Naturally, being the curious sort I am and them being a paying client, I log into the server and start examining things. The first thing I do is try to do a database check. This doesn’t go over too well, I get a bunch of errors about “corrupt .frm files.” Naturally my curiosity is piqued at this point, so I go and start looking through /var/lib/mysql/hosts.err file. What is extremely odd here is that there it is super short, and there is nothing there really. At this point I decide to log into mysql and try rotating out the log file. It indicated that the operation had completed successfully, so I am a bit baffled at this point. Looking in the my.cnf, it appears that someone downloaded a web based my.cnf and stuck it on the server. It also appears that the logs are going somewhere else at this point. After examining the correct logs, I find a bunch of errors about the .frms being corrupted. I then start looking for the MYIs so that I can see if I can rebuild the .frms with them. To my dismay I find out there are no MYIs for the DBs here just the .frms.

After updating the client on what I’d found, I start looking over the net for an answer. It turns out that if InnoDB doesn’t have the engine started it will default to MyISAM databases, which is where the .frms came from. Definitely getting closer at this point, so I start looking at the engine statuses and of course InnoDB is disabled.

After this the picture is starting to come into focus a bit more, something in the my.cnf was changed and it broke everything. But what? I had noticed that the server was running MySQL as root when I had done “ps aux | grep mysql” earlier so I had a suspicion. Referencing another server I confirmed what I thought and found out that a line was missing.

user=mysql

The net downloaded configuration didn’t have a set MySQL user, root didn’t have permission to access the file and therefore it wouldn’t load the InnoDB files and in turn the InnoDB engine. A single line was missing and this production server was likely down for hours while they tried to figure out what happened. The internet is not a substitute for knowing what you are doing. Yes it can help you in this regard, but at the same time it is not a substitute for knowledge or competency. I once had an instructor tell me there is no crib sheet for life, it rings as true now as it ever did when he said it.

This is my personal FS box. It consists of 4X200GB SATA drives on an Adaptec AAR-2810 controller in RAID10. I am using bonnie++ to benchmark. The reason I am throwing this up is that when I switched schedulers to deadline with 3 drives in a RAID5 on this system (I changed to a 10 later) it picked up about 5% by going to a deadline scheduler in sustained writes. If you notice in this case while I had a few small gains aggregately I lost a significant amount of performance at one point or another compared to the default scheduler. Then again, this array has write performance at a level typical of a single drive if not a little slower. It makes up for it in the fact that it has roughly double the read of a typical drive, which can be far more important depending on the application. Since it’s just a dump box anyways I’m not overly concerned about IO. Check the results out and post your opinions. I will probably add other tuning specifics at a later date. Please note this does not under any circumstances make CFQ my preferred scheduler; it just means it was better in this case.

cfq (default)

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 34694 88 39045 27 19030 8 36912 68 63545 12 496.2 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 19929 100 +++++ +++ 32373 99 +++++ +++ +++++ +++ +++++ +++
FS01.robotonet.com,1000M,34694,88,39045,27,19030,8,36912,68,63545,12,496.2,1,16,19929,100,+++++,+++,32373,99,+++++,+++,+++++,+++,+++++,+++

echo deadline > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 30986 88 38772 31 16724 7 29545 56 62167 11 482.5 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20401 99 +++++ +++ 26963 99 29849 99 +++++ +++ +++++ +++
FS01.robotonet.com,1000M,30986,88,38772,31,16724,7,29545,56,62167,11,482.5,1,16,20401,99,+++++,+++,26963,99,29849,99,+++++,+++,+++++,+++

echo anticipatory > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 33134 94 38153 28 17225 8 30326 58 65073 10 346.8 0
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 19071 94 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
FS01.robotonet.com,1000M,33134,94,38153,28,17225,8,30326,58,65073,10,346.8,0,16,19071,94,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

echo noop > /sys/block/sda/queue/scheduler

Version 1.03e ——Sequential Output—— –Sequential Input- –Random-
-Per Chr- –Block– -Rewrite- -Per Chr- –Block– –Seeks–
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
FS01.robotone 1000M 29617 86 39779 27 16359 7 29197 57 62977 10 496.6 1
——Sequential Create—— ——–Random Create——–
-Create– –Read— -Delete– -Create– –Read— -Delete–
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20130 99 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
FS01.robotonet.com,1000M,29617,86,39779,27,16359,7,29197,57,62977,10,496.6,1,16,20130,99,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Backing data up in MySQL is super important. The site content is often replaceable, however your board posts, ecommerce orders, etc. are far far harder to replace, if at all possible. The other nice thing is that with a lot of database driven web sites you retain most content even if you only back up the databases. You can still lose things like images, but by and large your posts will still be there even in event of a crash. Since bad code, etc. happens there are a few reasons I would recommend doing backups of the databases.

• Making server side changes, including configuration tweaks that can influence system stability.
• Making software changes, even adding plugins etc. to your site. I have seen odd things in the past caused by plugins and they can interact together with not always predictable results.
• Periodical backups as part of a data protection plan. I would do more than less, by and large most databases are not huge for the data they contain. I have seen forums with about 100,000 posts have about 2GB of database; not really that much if you think about it.

There are a few ways to do backups, but the most common for people who don’t use command line interface is probably going to be PHPMyAdmin. Quite frankly, I do not like using it for database backup and restorations. There are two reasons for this pertaining to reliability

• If the session timeout of PHP or Apache is too low, you will get an incomplete dump. This means that backup you may have of your databases is all of a sudden found to be truncated at a few megabytes. This is not a big deal for a lot of people with small DBs however if you’re rocking out with 50+ megabyte DBs this can be a huge issue.
• Restoration can be dependent on if the person has large uploads and large timeouts enabled. Since most people have slower upload than down, it can be a real nuisance to get the DB back up to the server and restored. Not only that but If you have a server with low upload size limits and timeouts, it may be impossible to get that DB back up and restored to the system before you hit one of these limits.

The way I honestly prefer to back my databases up is mysqldump; this allows one to have a lot of control over what gets dumped, do a full backup without worrying about doing it individually etc. It also allows us to make a nice easy cron job to do these backups too if we want.

The way we back a particular DB is up is:

mysqldump dbname > backup.sql

This will put the database named dbname into backup.sql. I believe you can separate multiple DBs with a space as well, however I don’t normally use that functionality. Another thing you can do is you can back up everything with:

mysqldump --all-databases > backup.sql

and this will do everything on the server. Handy for migrations, etc. Definitely way easier than trying to do this via a web interface and having 1,000 DBs.

How we restore these is fairly simple as well, we just run the command as follows:

mysql dbname < backup.sql

If you have a full DB dump you wouldn’t specify anything for the name when restoring. Notice how the carrat points in the opposing direction though. Don’t screw this up otherwise you will likely end up with a blank file and have to re-dump or re-move the file. I usually make 2 copies on the destination server especially when it’s not easy to get a second copy. There are other ways of moving the DBs over, such as copying the raw MYI files but that will be another blog.

Paying attention to your Nagios alerts can pay dividends in regards to server uptime and security. Just as an example, If you have an SSH alert come up it’s likely a very good idea to check your logs for failed login attempts. See cool-one-liners-1/ for more info. Chances are someone is eating your connections up and the monitoring server can’t connect.

Working on servers, people seem to think there is a cheat sheet to it at times. While there are many ways to simplify and automate a lot of configurations on a server, MySQL is not the time nor the place to cut corners. If you bog MySQL in turn the system will feel the wrath of your misconfigurations. Here are a few super common mistakes.

• max_connections set way high-Why people think this is even remotely near a good idea is beyond me. It is common to go out to a server, find out it has either flat out hung or is spitting out of memory errors on the screen and has gone brain dead. Just because you can set your connections to 1500 doesn’t mean it’s a good idea. In fact it’s probably better for MOST people to set their connections to 150. The reason is simple; each connection available uses RAM. If a query comes by and there are 150 connections available, you will probably get an out of connections error and can log into the system, figure out what broke and fix it; If you hang the system you get an out of memory error and no real data to work off of. CPanel sets this to 500 by default, if you are using 500 connections (and the system is still up) YOU PROBABLY HAVE A PROBLEM.
• huge buffer sizes-Guess what if you have 100 megabytes of InnoDB tables setting your innodb_buffer_size to 5GB isn’t going to help you out. Plain and simple. If you start adding more tables you could possibly even run out of RAM (see above.)
• Big RAM-MySQL does not like RAM use above 2GB on 32 bit systems. They do not recommend running it because it can cause stability problems. In turn this means that there is little reason not to run a 64 bit OS any more unless you’re running a low end server. You NT guys are pretty much stuck with 2GB no matter if you’re running X86 or X64. That being said for hard core DB applications you should probably be putting it on a unix based OS anyways.

Besides this there is the single biggest reason of all:You will probably have lousy performance compared to a custom my.cnf. When tuning databases, one needs to keep in mind that the my.cnf  is tailored to a combination of the database content, how the database is used and the server its self. If there was a magical my.cnf that would make any server work great don’t you think that Oracle/Sun/MySQL would have included it with the server software? We will go into a few things in later blogs such as software to help tuning MySQL, what the parameters mean and how to actually tune a database properly.

I have had a few clients with chronic security issues on machines, DoSes are by far the most common attack. It’s easy enough for anyone of average (or below average, up for debate) intelligence to carry out. A few zombie machines and some basic software and all of a sudden you can take down a decently massive web site.

There are a few caveats to mod_limitipconn, the biggest being if you run a site with a very large amount of images. Images are loaded last on a page, and also are loaded at once. This can be a perilous position because of the fact that mod_limitipconn can cause issues with dead images if not done properly. This is why you have to tune it. In general there is no CPanel integration so you’re going to be doing this by hand. For you people out there with other panels, your milage may vary.

First thing is installing it. This is a relatively simple process, be sure you have APXS installed so you can do this. Download the software from http://dominia.org/djao/limitipconn2.html and install it using their commands. You can also probably get packages from a place like Dag if that’s your thing. Certain distros such as Debian also come with a ready to rock version. After we get it installed, we need to go into httpd.conf and make some alterations.

MaxConnPerIP 10
NoIPLimit images/*

There are a few things to note here. For people that have static content on their systems, you would set an instance of “images” for each static content directory that you were using. This includes things like Javascript, CSS, and conventional images. The reason for this is if using mod_limitipconn for system hardening an attacker will usually not target an image. This can happen, but it’s also not as “profitable” as a DoS using a page because you don’t pull the images and other content on the page (increasing bandwidth usage) and static content tends to be super efficient compared to PHP or other server side scripting so it tends to be far more robust.

You can also do other cool tricks such as format based setting of the connection limits.

# local per-directory settings here
MaxConnPerIP 1
OnlyIPLimit audio/mpeg video

By putting this in with the main confiugration above, you can set up additional subdirectories and limit connections based on media type. This can slow a leecher down significantly because they can’t pull everything at once.

Since we have the basics covered, just a few words on tuning the configuration. The biggest thing you need to do is view the sites you use this on. Sites with mod_limitipconn can behave erratic behavior if an initial setup is done but left untuned. This can include random broken images and other content not showing up. There are two things we are concerned with to tune; the first is that the most static content possible is set to very loose or no limitations, the second being that you get the base connections as low as sanely possible. If you can get it around 5 that would likely be the most pages someone would “sanely” load of a typical site without having issues. The other thing is that if you have embedded pages in your site you will have to increase this number significantly. Just go slowly and when the page fully loads add say 10-20% to that amount. If you get reports of broken content it would likely be safe to up this number some at that point to “dial in” that perfect configuration.

Back when I was younger and working on gaining knowledge of such things as Linux, I used to listen to this online radio show all the time. It really gives a feel for the industry as of about the years 2000-2001 and how it was evolving at the time. The show has long since gone off the air, but they do have archives up still. ftp://ftp.dhbit.ca/DHBiT/ I also used to listen to Off The Hook, Emmanuel Goldstien’s radio show. I think it’s still on, but it’s been forever for that as well.

The series of blogs I write is actaully a lot of tech support questions that are common. This question gets bonus points because it is actually very difficult to explain over the phone due to the complexity in which DNS works. That being said pictures are worth their weight in gold. For this reason I have included the pictures below as well captions explaining what you need to do.

First off this is the domain menu. If you look here, the name servers are already defined. Chances are that unless you have a new domain, they will be pointed to the DNS of your host or your old server. What we are going to do is scroll all the way down the page, at the bottom left where it says “Host Summary” then click “Add.” This will open the next menu.

There are two fields we are going to edit here. The first one is for the Host record that the DNS server is going to be called. Normally I use “NS1″ and “NS2.” don’t sweat capitals or not here, it’s not case sensitive. The other field we need to change is the one labeled “Host IP 1.” After this is done, we click “ok” and simply repeat the procedure for the secondary name server. When I set a new dedicated up I prefer to set it up with the first two IPs as NS1 and NS2 respectively.

Now that this has been done, we can finally add our name servers. Please note that if the other information hasn’t been updated in Godaddy’s information that it can take some time for it to allow you to do this. If you have NS1 and NS2 showing up in the Hosts Summary but it gives you an error about the name server not existing this is why. If you already have this set up, the only other change required is to change the Host Records so that the IPs are pointing to the new server and wait on propagation.