Sort Files with Bash

I’ve always been a fan of collecting fonts, as I go across the net I find random font files and save them for later use. Seems like I can never find the fonts I want because they are scattered all over and unsorted. Got tired of having unorganized font files laying around, so I wrote this to organize them.

#!/bin/bash
#Edit path to the location of where you want your fonts organized
#create a folder called "Unsorted" and place all the files into that.
path="Fonts/"
cd "${path}Unsorted/"
for mFile in *
do
  #Check rather its a file or folder, if its a folder skip it!
  if ! [[ -f $mFile ]]; then continue; fi

  #Grab the first letter of the filename and set it to $mFirstChar/
  mFirstChar="${mFile:0:1}"

  #Convert all lowercase fist letters to upper case.
  mFL=$(tr "[:lower:]" "[:upper:]"<<<"$mFirstChar")

  #If the filename contains any chars such as "!@#$%^&*()" at the beginning
  # set the $mFl var to "MSC".
  if [[ $mFL != [[:alpha:]] ]]; then mFL="MSC"; fi

  #Make directories, prompt for overwrite and create dir if it does not exists.
  mkdir -ip "../${mFL}"

  #Move files to there new home.
  mv -v "${mFile}" "../${mFL}/${mFile}"
done

Cleaning spam out

I’ve seen my fair share of spam in my day, however in a lot of cases a server has a sysadmin that doesn’t quite know what to look for in order to track it down. In fact a lot of these machines are caught because they crash due to the load put on them. While my methods are probably not the best way of doing it, they do work and you can clean a queue out without just deleting everything and starting over. You can also use an email verifier and cleaning tool to help you manage your emails.

You can follow these simple steps if you want.

First thing’s first, on a CPanel/Exim (the most common setup you’ll see if you do CPanel hosting) we simply go into /var/spool/exim/input.  At this point you will see a bunch of directories, there should be about 62 here. The letters A through Z in both upper and lower case as well as 0 through 9. These are subcategories of mail. if we do

[root@dns01 input]: ls ./* -lh

here you will see a bunch of files that have -D and -H after them if your queue is being stuffed. If this is empty, then you may be spamming however at a lower rate. Now then, lets say that we happen to have a spamming problem, there are a few things we can do here. The first thing we need to do is isolate the source of the problem, if at all possible. After that removal of everything (directories and all) is an option especially if you’re doing low end or free shared hosting. If you have to preserve your ham though, you will have to clean. The first thing that needs to be done is to figure out what content is there. I prefer to view a few of these emails and find a topic in them. So lets say your email is about some Nigrian King who is poisoned. all we would have to do is run:

[root@dns01 input]: grep -ilr 'Nigeria' ./* > check

This will populate the file check with the names of the files containing “Nigeria” These are our probable spam messages. You may lose a few legitimate emails this way but the vast majority will be safe. If you use a phrase you are virtually assured that no legit emails will be deleted.

so we would do something like this:

[root@dns01 input]: sed -e 's/^/rm -f /' check

[root@dns01 input]: sed -e 's/\-D/\-\*/' check

[root@dns01 input]: chmod 755 check

[root@dns01 input]: ./check

This will go through and delete the emails and their associated headers. While this isn’t too hard, spammers are like mice, or roaches. Once they find a way in they are sure to be back. To this end we need to at least try and find the hole before we delete the evidence. We can actually take the messages and move them instead which would allow us to view at our convenience, or we can evaluate the headers in the input directory. There are two things I tend to look for.

  • Rogue PHP scripts
  • Open Relays

PHP scripts can be a hotbed of insecurity on many servers. Personally I’m not a fan of how a lot of people use or misuse them. This being said it’s a necessary evil because of how much function PHP adds to a site. This being said, the quickest and dirtiest thing to add is mail() headers. This can be done via Easy Apache or you can download the latest and greatest from http://choon.net/php-mail-header.php. This gives you something convenient to track with. Now then all you have to do is run

[root@dns01 input]: grep -ir php ./*

before you delete the email you will get every mention of the PHP file name in question. All you have to do is find the one(s) that have a lot of entries and they are either going to be for message boards or other lists or they are going to be spam scripts. Pretty simple.

In regards to open relays, just find a checker via your search engine of choice and go to town. Telnet can also make a fun checking tool to see if you can relay or not, but that is likewise another episode.

Some SED basics

One of my favorite tools for Sysadmin work is the Stream EDitor utility or just SED. SED is useful for many things, and is a stepping stone along the way to making variable based shell scripts as well. Don’t want to have to edit the nameservers on a million zone files? SED it. Need to do certain things to a million files at once? SED it. In conjunction with cat, find, and grep SED is devastatingly effective in finding and eliminating administrator headaches. Lets start out with something extremely basic.

sed -i ‘s/ns1.domain1.com/ns1.domain.com/’ /var/named/*.db

What does this do? It goes through and changes the instances of ns1.domain1.com to ns1.domain.com in DNS zone files. Please note I advise grepping any thing out that you are changing because if there are multiple instances of this in the file it will only do the first instance.  If you have this issue, you can always repeat the command and check again until all instances are found. the s indicates a spelling correction, the -i puts it back into the original file. If we just wanted to print to the TTY we would use the -e augment.

Well, that’s pretty cool, but what about some other situations that come up? Lets say we’re migrating a cpanel box. There are a ton of scripts out there, but we have some special needs. Say we want to run it with –skip-homedir because this is going to be a pseudo-manual migration and we’ll sync the homedir over later. All we have to do is make a copy of /etc/users and then do the following:

sed -i 's/$/\/scripts\/pkgacct /' users

sed -i 's/^/  \-\-skip\-homedir\' users

chmod +x users

./users

Yes this is a few commands, but we want to progressively look over what’s happening here. The $ means that /scripts/pkgacct is prepended to the beginning of each line. Notice that there is a space at the end of the command so that the user name doesn’t become part of the command we are trying to run and error out. Also notice the \es. These are an escape character that is used in order to allow the use of a special character such as /  – . or other characters that may otherwise be taken as part of the command. The second line is similar to the first in the fact that it will add to every line in a file as well as the use of escape characters, however ^ will add to the end of every line. The last thing we are doing is making our script executeable with the +x command (you can chmod 755 if you want and get similar results) and then runs it. If we were smart we would probably put a she-bang at the top (#!/bin/bash) so that it is run with BASH.

Our input file would look like this:

user1

user2

user3

user4

and the output would look like

/scripts/pkgacct user1 --skip-homedir

/scripts/pkgacct user2 --skip-homedir

/scripts/pkgacct user3 --skip-homedir

/scripts/pkgacct user4 --skip-homedir

There are a ton of uses for this, I love cleaning spam in Exim’s mail queue with this if you’re not allowed to BOFH the system and delete the “clean” email with the spam as well. That will be a later episode however.

Linux Daily Tip – Concatenate

A extremely useful tool for outputting the contents of a file is “Cat” short for Concatenate.  Cat will print the standard output onto the screen.

A useful example:

cat -n  -s /proc/cpuinfo | more

Options: -n will number the lines outputted while -s will suppress excess empty lines.

Its also useful to pipe the output into more to make it easier to read.

Want to learn more about pipe? Checkout Alex’s article here.

Daily PHP Tip – Error Supression Operator

A common problem with PHP is that people write code that uses whats called an  “undefined index”,  basically this means they are calling for an unassigned variable.  In PHP, the error-suppression operator is an “@” sign.    It’s best to give you an example as where to use it.

Improper code:

$colors .= "purple,";
$colors .= "red,green,blue";
echo $colors;

This will return “Notice: Undefined variable: colors”

To suppress this error since you know about it:

@$colors .= "purple,";
$colors .= "red,green,blue";
echo $colors;

This will return “purple,red,green,blue” without a Notice warning.   Using @ is also useful for PHP-GD Image functions for capturing image sizes and resizing.

MySQL query hint of the day

Today I had a client who had a query something like this;

SELECT `field1`,`field2`,`field3`,`field4` FROM `table_name` WHERE approve='1' LIMIT 0,10 ORDER BY RAND();?

This is exactly how NOT to do a query. For starters it is not indexed which means that it will do a full table search. Besides this the RAND() function then proceeds to randomize EVERY LINE in the table. it only takes a few queries to make this fall flat. At the very least push this to code, generate some random #s and then search for those rows based on an Index.

Choosing a web hosting provider for features

What’s Right For Me?

There are 5 different types of hosting services: Personal Hosting, Business Hosting, Virtual Private Server (VPS) Hosting, Dedicated Server Hosting. Choosing between these hosting options is often difficult, with many different features to compare. No worries! Here’s a simple way to decide between the hosting choices.  For this article we are going to use Beyond Hosting as an example,  We host our blog with them.

Personal Hosting

Personal Hosting is a “shared hosting” environment. This means that your website will be on a server that will be hosting other Personal Hosting websites. The server resources will be split evenly between all Personal Hosting accounts. This means that attempting to use more than your “fair share” will result in degraded performance. This makes Personal Hosting suitable for low traffic websites, websites with little or no dynamic content (PHP scripts, Python scripts, etc.), and websites that are relatively small.

View Personal Hosting Packages

Business Hosting

Business Hosting is also a “shared hosting” environment, but with extra features that Personal Hosting can’t support. This is mainly the Secure Socket Layer (SSL) and Static IP. SSL allows you to communicate securely over the Internet, performing financial transactions safely. This makes Business Hosting great for small merchants and businesses. Business Hosting also supports a Static IP, meaning that your website will be accessible anywhere throughout the world through a dedicated, unique address that is only yours. This improves rankings on search engines like Google, Yahoo, and Bing because you don’t share the IP with your shared hosting “neighbors” (other accounts on the same server). This makes Business Hosting indistinguishable from our dedicated products, but you are still sharing server resources.

View Business Hosting Packages

Virtual Private Servers

Virtual Private Servers offer all of the features of Business Hosting, and guarantee access to more powerful server resources. Virtual Private Servers put you in control of your own website and server by providing a fully customizable Linux server platform. While Virtual Private Servers do share the same physical server, like shared hosting environments, the number of websites-per-server is significantly lower, typically 25 for Virtual Private Servers but up to 1000 for shared hosting. Virtual Private Servers are a necessity for larger websites, websites with lots of dynamic content, or websites with larger disk space demands.

View Virtual Private Servers

Dedicated Servers

Dedicated Servers are the ultimate in hosting. You are in full control of what the server is hosting, with direct access to all hardware. Dedicated Servers are typically configured to meet performance requirements, and can always be customized for specific applications. Unlike the shared hosting services Personal Hosting and Business Hosting, Dedicated Servers allow for customized services to run, such as a dedicated MySQL or PostgreSQL server to improve performance of heavyweight applications. Dedicated Servers provide the same interface as Virtual Private Servers, but with the additional advantages of more processor power, more RAM, more bandwidth, and more hard disk space.

View Dedicated Servers

Article Provided by: Beyond Hosting