Tech From the Trenches

It wasn’t that long ago that I decided to partition a couple of our new servers using the LVM file system.  I have read that using the LVM (logical volume manager) is much for flexible than using standard ext3 or ext4 partitions.  This is true when you are using a physical server with physical hard drives, because when you run out of disk space, your only option is to add additional hard drives.  Using logical volumes makes it very easy to add new disks to existing volume groups and expand the volumes.

But, when your servers are in a virtual environment, you can easily make the hard drive larger with a setting change.  Expanding volumes and expanding ext partition both require server downtime too, so there aren’t many advantages.  Let’s just compare the steps taken to expand both file systems on a virtual server.

For logical volumes:

1. Increase the size of the vdisk, and reboot.

2. On guest OS, create a new partition to use free space (‘fdisk -l’ and ‘fdisk /dev/sda’), and reboot.

note: the new lvm partition must have an ID type “8e” in fdisk

note: if you are using and extended partition, you will have to boot to a partition utility to resize it first (gparted works great)

3. Create a physical volume on the new partition with pvcreate.

4. Extend the existing volume group to the new partition with vgextend.

5. Expand the logical volume to fill the volume group (use ‘lvextend -l +100%FREE /dev/vg/lv’ to use the remaining space so you don’t have to calculate the exact size.  ‘lvresize’ can also be used.)

6. Expand the filesystem to fill the logical volume (‘resize2fs -p /dev/vg/lv’)

note: use ‘lvdisplay’ to find the correct path to your volume groups

Here is the same process for standard partitions:

1. Increase the size of the vdisk, and reboot.

2. Boot to a partition utility and resize your current partition to fill the new disk size (I use a gparted boot cd iso).

note: if you have multiple partitions, you will have to move the partition around on the disk to resize in certain situations.

I can conclude the standard partitioning is much easier to resize.

Here is a good reference blog for LVM resizing:

http://echenh.blogspot.com/2010/04/how-to-extend-lvm-on-vmware-guest-os.html

Drupal is the tool of choice for our web site team, but it using alot of dynamic page content.  Consequently, that increases the memory usage for the apache server processes.  Each process on our production server uses 30-50MB.  The server currently has 4GB total memory, so that means it should be able to handle 80-130 web connections without swapping to disk.  This simple calculation is something that should be done for every web server to set the MaxClients limit.  Here are some other settings that we are using for Apache2 on our Drupal servers:

Timeout 30

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 5

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          100
ServerLimit         150
MaxRequestsPerChild   2000
</IfModule>

Settings the MaxRequestsPerChild variable to 2000 means that each Apache process will be “recycled” 2000 times before it is forced to spawn a new Apache process.  The default is 0, or unlimited, which can lead to memory leaks.

Also set the KeepAliveTimeout low (2-5), to keep processes available for new connections instead of waiting on existing connections.

Here are some good resources that I used:

http://www.devside.net/articles/apache-performance-tuning

http://httpd.apache.org/docs/2.0/misc/perf-tuning.html

http://www.howtoforge.com/configuring_apache_for_maximum_performance

If you would like to query for a particular record type for a domain, then this tip is for you.  In my specific case, I needed to make sure that my mail server was reading the txt record correctly for our domain.  You can use this host command to do so:

#host -t txt <domainname>

You should be able to do the same for any domain if you want to see if they have an SPF record in place to cut down on spoofed mail.

Plesk doesn’t provide an easy way to make virtualhost configuration changes in Apache.  When Plesk is installed the default Apache settings are left untouched.  Plesk adds a .conf file to Apache’s conf.d directory that specifies “Include” parameters for the each domain’s configuration file.  The Plesk config files are over written whenever changes are made via Plesk.  If you want to manually make any changes to the Apache configuration for a website, you need to create a vhost.conf file for it in the following directory:

/var/www/vhosts/mydomain.com/conf/

This directory also contains the Apache config files for the domain (these files have names ending with httpd.include).  When the Plesk domain settings are re-saved, Plesk will include the vhost.conf file in the resulting domain httpd.include file.  The easiest way to force the addition of the vhost.conf parameters is to run the following command:

/usr/local/psa/admin/sbin/httpdmng --reconfigure-domain <domain>

I recently had to change the loglevel to debug for a particular website and I was not allowed to add the variable to the site’s .htaccess file.  Adding a vhost.conf file that included “loglevel debug” did the trick.

We have migrated and updated our Moodle databases several times over the years, which has resulted in inconsistent database collations.  I am by no means a database expert, but I know how to find the answers to the problems at hand.  Here are some of the steps that I used to upgrade our moodle site from 1.9 to 2.0.

Create a new empty database for the upgraded site:

mysql> CREATE DATABASE dbname CHARACTER SET utf8 COLLATE utf8_general_ci;

Use mysqldump to create a backup file and then import the backup into your empty database.

# mysqldump -u user -p  moodledb > data.sql

# mysql -u user -p -h localhost –default-character-set=utf8 moodledbnew < data.sql

Then to make sure you don’t get collation errors during the moodle upgrade process perform one of the following (the second method worked well for me):

1. Backup the database; Dump the database to a .sql or text file; Perform a global search and replace all instances of utf8_general with utf8_unicode; Run a query on the database replacing the existing code

2. In the following script, replace “database_name” with the name of your database and edit the statement template in the parameters of the CONCAT function to have the target character set (and/or collation) of your choice. After that you can just take the statements and batch execute them in the MySQL client of your choice (eg. phpmyadmin, Navicat, the official MySQL client, etc.). Of course, you’re strongly advised to make a backup of your database before you start messing around with character set (and/or collation) conversions.

SELECT  CONCAT('ALTER TABLE `', t.`TABLE_SCHEMA`, '`.`', t.`TABLE_NAME`, '`  CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;') as stmt
FROM `information_schema`.`TABLES` t
WHERE 1
AND t.`TABLE_SCHEMA` = 'database_name'
ORDER BY 1

Of course, you will need to download the new moodle version, copy over your themes, modules, and blocks, and adjust your web server settings appropriately.

I had a heck of a time finding out how to see what the parameters were currently loaded for MySQL on our servers.  By default, when you install MySQL server from the Ubuntu repositories, the my.cnf config file specifies only a few variables.  And the rest just use a default value (my assumption).  We recently had a problem with one of our websites that was being bombarded with registration requests; the database became unresponsive because of the number of sleeping processes that used up all of the system memory.  Since MySQL uses a default wait_timeout value of 28800 (8hrs), the sleeping processes were allowed to eat up system resources.  There were no timeout parameters listed in the my.cnf file at all.  So, if a setting is not specified in your my.cnf file, how do you see what is loaded by default?  Just run the following command:

# mysqld –help
or
# mysqld –verbose –help

You will have to specify a user when you run the command.

We are always looking for quick and easy solutions to problems. Here at Essdack we have an internally used system for our essk.me service built on Drupal. Which works great. In the last few weeks, I have been looking for a solution for a test project with Spockholm that is a little more robust with features like filtering, public or private options, Json returned in shortneing and lengthening, perfomance, and url preview to name a few. We have been experimenting with a free open source url shortening and lengthening service called yourls (Your Own Url Shortner). This service was pretty easy to set up and had many of the features of a tinyurl or bit.ly. We found with a high volume service the database needed to be modified and indexes used. Otherwise, it is a great solution. Kudos to the developers. Check it out at Yourls.org

Recently, our users have been receiving a lot of backscatter email, which are bounce messages from email sent to invalid addresses that the user did not actually send in the first place.  I hope that made sense…  The messages that were sent out used our employee’s email address as the sender.  There is really nothing we can do to stop these spammers from imitating our email addresses and sending these messages.  What we can do is setup spam rules on our email server to identify the messages as spam, and other email admins would have to do the same for their email systems.

The first step is to create an SPF record for your DNS zone, there are some great tools online that make this very easy.  I like http://www.openspf.org.  The SPF setup tool that they provide make the it a quick process.  Once you have the record contents, add it to you DNS zone file as a text record.  I would recommend that you use “-all” in you record, which states email from your domain should only come from the servers specified in the SPF record.  This addition will trigger a hard fail when the SPF record is checked and the message is delivered by a server that is not listed in the record.  If you want to be more lenient, you can use “~all”, which states that servers not specified in you SPF record are not preferred and it will trigger a soft fail.  The problem with using a soft fail is that many spam servers will not add spam points for soft fail unless a custom spam score is added.

We use Zimbra with Spamassassin, and installing SPF query tools was simple.  You will need to find instructions for your specific system.  With Spamassassin, just add your SPF spam scores and other custom scores to /opt/zimbra/conf/spamassassin/local.cf.

Here is a sample:

score SPF_SOFTFAIL 2.000
score SPF_FAIL 5.000
score SPF_HELO_FAIL 5.000

You will want to change the score based on your required spam score.

Here is a quick how to tip.  Scripting between servers is handy for routine file transfers or backups, but in order to make scripts unattended, you will have to add your server credentials to your script in plain text or setup authentication keys for you server.  The authentication keys are very easy to create, but you must always remember to create your key on the client.  Then, you copy the key to ~/.ssh/authorized_keys on the destination server for the user you will that you specify in your script.  I know that seems backwards, but since the server has the client’s key, it has permission to login without a password.  So, moving on, here is the command to create the key on you client; you will then distribute this key to your servers.

sudo ssh-keygen -t dsa -f /root/.ssh/id_dsa -C "your comment"

Note: your root user home directory might have a different path than I specified above.

Next, you need to add the key to the authorized_keys file on the server; here is an easy method:

sudo cat /root/.ssh/id_dsa.pub | ssh root@remote_address 'cat - >> ~/.ssh/authorized_keys'

You should be able to login to the server from the client without using a password now, just make sure you run your command using sudo.

I am not much of a writer and I never will be, but here is a solution that I searched for long and hard.  It fixes problems when using automysqlbackup.sh to backup all databases, including “information_schema” and “mysql”.

I had to change this line:

OPT=”–quote-names –opt”

to this:

OPT=”–quote-names”

There is only one other change needed; I had to add an -o (or) statement, just replace this section:

# Database dump function
dbdump () {
touch $2
chmod 600 $2
if [ $1 = "information_schema" -o $1 = "mysql" ] ; then
NEWOPT=”–skip-opt –skip-lock-tables ${OPT}”
else
NEWOPT=”–opt $OPT”
fi

if [ -z "${USERNAME}" -o -z "${PASSWORD}" ] ; then
mysqldump –defaults-file=/etc/mysql/debian.cnf $NEWOPT $1 > $2
else
mysqldump –user=$USERNAME –password=$PASSWORD –host=$DBHOST $NEWOPT $1 > $2
fi
return 0
}