• Make sure you run the latest Mediawiki version. Mediawiki 1.16 introduced a new localisation caching system which is supposed to improve performance, so you definitely want this to get the best performance.
• Create a directory where Mediawiki can store the localisation cache (make sure it is writable by your web server). By preference store it on a tmpfs (at least if you are sure it will be big enough to store the cache), and configure it in LocalSettings.php:
  $wgCacheDirectory = "/tmp/mediawiki";
  Iif /tmp is on a tmpfs, you might add creation of this directory with the right permissions to /etc/rc.local, so that it still exists after a reboot.
• Enable file caching in Mediawiki’s LocalSettings.php:
  $wgFileCacheDirectory = "{$wgCacheDirectory}/html";
$wgUseFileCache = true;
$wgShowIPinHeader = false;
$wgUseGzip = true;
• Make sure you have installed some PHP accelerator for caching. I have APC installed and configured it in Mediawiki’s LocalSettings.php:
  $wgMainCacheType = CACHE_ACCEL;

Here is a benchmark before implementing the above configuration (with CACHE_NONE, but APC still installed):

$ ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 255 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12750 bytes

Concurrency Level: 1
Time taken for tests: 30.084 seconds
Complete requests: 255
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 3344070 bytes
HTML transferred: 3251250 bytes
Requests per second: 8.48 [#/sec] (mean)
Time per request: 117.978 [ms] (mean)
Time per request: 117.978 [ms] (mean, across all concurrent requests)
Transfer rate: 108.55 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
50% 118
66% 122
75% 125
80% 127
90% 132
95% 138
98% 145
99% 156
100% 170 (longest request)

And here a benchmark after implementing the changes:

ab -kt 30 http://site/wiki/index.php/Page
This is ApacheBench, Version 2.3 < $Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking site (be patient)
Finished 649 requests

Server Software: Apache/2.2.16
Server Hostname: site
Server Port: 80

Document Path: /wiki/index.php/Page
Document Length: 12792 bytes

Concurrency Level: 1
Time taken for tests: 30.015 seconds
Complete requests: 649
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 8538244 bytes
HTML transferred: 8302008 bytes
Requests per second: 21.62 [#/sec] (mean)
Time per request: 46.248 [ms] (mean)
Time per request: 46.248 [ms] (mean, across all concurrent requests)
Transfer rate: 277.80 [Kbytes/sec] received

Percentage of the requests served within a certain time (ms)
50% 45
66% 47
75% 49
80% 50
90% 56
95% 62
98% 68
99% 73
100% 82 (longest request)

So Mediawiki can deal with more than 2,5 times as much requests now.

Some people use Apache’s mod_disk_cache to cache Mediawiki pages, but I prefer Mediawiki’s own caching system because it is more standard and does not require patching Mediawiki, even if it might not get as much benefit as a real proxy or mod_disk_cache.

With iotop you can check which processes are reading and writing from the disks. I always press the o, p and a keys in iotop so that it only shows me processes doing I/O and so that it will show accumulated I/O instead of the bandwidth. With the left and right arrows I select on which columns to sort the list.

Once you have identified the processes wich are doing much I/O, you can check what they are reading or writing with strace, for example
# strace  -f -p $PID  -e trace=open,read,write

(you can leave out read and/or write if this gives too much noise)

This way I identified some locations where processes do lots of read and write operations on temporary files.

For nagios I placed /var/lib/nagios3/spool and /var/cache/nagios3 on a tmpfs, for Amavis /var/lib/amavis/tmp and for PostgreSQL /var/lib/postgresql/8.4/main/pg_stat_tm.

Other candidates you might want to consider: /tmp, /var/tmp and /var/lib/php5. There are probably many others, depending on which services you use.

First of all, you need two machines with exactly the same version of ISC dhcpd. Failover is unsupported when the two systems are running a different dhcpd version, as the implementation may have been changed.

On the primary dhcpd, you put this in dhcpd.conf:

failover peer "dhcp" {
    primary;
    address 123.456.789.001; # address of this server
    port 519;
    peer address 123.456.789.002; # address of the secondary dhcpd
    peer port 519;
   max-response-delay 60;
   max-unacked-updates 10;
   mclt 600;
   split 128;
   load balance max seconds 3;
}

key primaryhost {
    algorithm hmac-md5;
    secret "secret";
};

omapi-key primaryhost;
omapi-port 7911;

include "/etc/dhcp/dhcpd.master";

You have to change the address of the primary and the secondary dhcpd in the configuration file. In this example, both the primary and the secondary dhcpd are listening on port 519, so you will have to open this port in the firewall on both systems so that the two dhcp daemons can contact each other. A split value of 128 means that both the primary and secondary server will manage half of the available IP addresses. By setting a higher number (max 255), the primary server will manage more addresses, by lowering it, the secondary will take care of a bigger part. With this option, you can distribute the load over the two servers. mclt is the duration of leases in seconds that either of the DHCP servers will offer to the peer’s DHCP clients when the peer is down. max-response-delay defines the number of seconds after which a dhcpd will consider its peer as dead if it does not get any reply; load balance max seconds is the time which a load balancing is disabled: if a dhcpd fails to respond within this time to a DHCPDISCOVER packet, the peer dhcpd will respond in its place.

Now you need to create a key for connecting to OMAPI, which is dhcpd’s interface which can be used to send it commands at run-time. You will need the dnssec-keygen command, part of the bind9utils package in Debian. To create the key, first cd to the /etc/dhcp/ directory, and run
# dnssec-keygen -a HMAC-MD5 -b 512 -n HOST primaryhost

“primaryhost” should be the name of the key you used in the dhcpd.conf file, you can use use the hostname if you want. This will create two files, one with the extension key and one with the extension private. Open the key file, and copy the last part of the line in that file (i.e. everything after the last space). This is the key’s secret: you have to to put it in your dhcpd.conf file after the secret keyword.

This is how dhcpd.conf looks like on the secondary dhcpd:

failover peer "dhcp" {
    secondary;
    address 123.456.789.002;
    port 519;
    peer address 123.456.789.001;
    peer port 519;
    max-response-delay 60;
    max-unacked-updates 10;
    load balance max seconds 3;
}

key secondaryhost {
    algorithm hmac-md5;
    secret "secret";
};

omapi-key secondaryhost;
omapi-port 7911;
include "/etc/dhcp/dhcpd.master";

Create an OMAPI key on the secondary dhcpd host, just like you did on the primary host, and copy the key’s secret in the dhcpd.conf.

Now we have to create the dhcpd.master file. This file should be at all time identical on both hosts! It can look something like this:

default-lease-time 7200;
max-lease-time 14400;
option domain-name-servers 123.456.789.001, 123.456.789.002;
option domain-name "my.domain";
subnet 123.456.789.0 netmask 255.255.255.0 {
    authoritative;
    option subnet-mask 255.255.255.0;
    option broadcast-address 123.456.789.255;
    option routers 123.456.789.5;
    pool {
        failover peer "dhcp";
        range 123.456.789.50 123.456.789.150;
    }
}

Basically the only difference with your non-failover dhcpd.conf file, is that you add a line

failover peer "dhcp"

in every pool section. If you used another name than dhcp in the failover peer section in dhcpd.conf, replace dhcp by the name you have chosen.

Now we have to find a way to make sure that dhcpd.master on the secondary dhcpd is always in sync with the one on the primary dhcpd. In the future, I might start using something like cfengine to manage configuration files on my servers, but for now I wanted something simple to set up. Incron was the perfect solution. On the primary server, I created a /etc/incron.d/dhcpd file with these contents:

/etc/dhcp IN_CLOSE_WRITE /usr/local/sbin/dhcpd.master.incron.sh $@/$#

This will execute the given shell script for every time a file is modified and closed in the directory /etc/dhcp. Note that you cannot just put an inotify watch on /etc/dhcp/dhcpd.master itself, as the watch will be lost whenever dhcpd.master is deleted, and this is exactly what happens when you use vim to edit the file: when saving an existing file, vim creates a new temporary file, then deletes the original old file, and renames the temporary file to the original name.

/usr/local/sbin/dhcpd.master.incron looks like this:

#!/bin/bash
if [ $1"x" == "/etc/dhcp/dhcpd.masterx" ]
then
    scp /etc/dhcp/dhcpd.master dhcp@secondarydhcp.my.domain:/var/lib/dhcpd/dhcpd.master/
fi

We check whether the saved file is dhcpd.master, and in that case, we scp it to the secondary server.

On the secondary server, I created a user dhcp and I created the directory /var/lib/dhcpd/dhcpd.master/, writable by the dhcp users. I added root’s ssh certificate of the primary server to ~dhcp/.ssh/authorized_keys on the secondary server.

The install incron on the secondary server and create this incron configuration file:

/var/lib/dhcpd/dhcpd.master IN_CLOSE_WRITE /usr/local/sbin/dhcpd.master.incron.sh $@/$#

Then /usr/local/sbin/dhcpd.master.incron.sh looks like this:

#!/bin/bash
if [ $1"x" == "/var/lib/dhcpd/dhcpd.master/dhcpd.masterx" ]
then
    cp $1 /etc/dhcp/
    /etc/init.d/dhcpd restart
fi

Now as soon as you have modified dhcpd.master on the primary dhcp server, the file will be copied to the secondary server and the secondary dhcp daemon will be restarted immediately. You still have to take care of restarting the primary dhcp daemon yourself. I prefer not to do this automatically at every dhcpd.master modification, because otherwise you will have no dhcpd running at all anymore if there is a syntax error in dhcpd.master.

If you plan to do maintenance on one of the dhcp servers, which will take longer than the MCLT time, it can be interesting to let the other dhcpd know that its partner is down, so that it will immediately will claim the complete DHCP pool. You can also use this when you start up one of the dhcp servers while the other one is down. If you do not do this, dhcpd will refuse to offer new IP addresses until MCLT has passed.

For example, if the secondary server is down, run this on the primary server:

omshell < < EOF
server localhost
port 7911
key primaryhost "secret"
connect
new failover-state
set name = "dhcp"
open
set local-state = 4
update
EOF

Keep in mind this note of Barry Pederson's blog however:

When the downed server comes back up, the two servers automatically start communicating and eventually get themselves back into a normal state. But only after the recovering server has spent mclt time in recover-wait state, where it renews existing leases but won't offer up new ones. So you probably wouldn't want to go into a partner-down state if the other server will be down for less than that amount of time.

During some spare time at work I decided taking a look at migrating the MS SQL Server databases to PostgreSQL. And all in all it was not very complicated. Fortunately, the databases itself are not very large and have a rather simple structure. The first thing I did, was recreating the structure of the database in PostgreSQL. With a tool like pgAdmin3 this is pretty straightforward. An alternative is to export the database structure in SQL format and adapt the resulting file so that it contains valid PostgreSQL table creation statements. The latter method is explained on the PostgreSQL wiki.

To migrate the data itself, I wrote a little Perl script. Actually I do not have much knowledge about Perl but it is a fairly straightforward language, so this did not prove to be too difficult. I connect to the MS SQL Server database using DBI, the Perl Database Interface, with the ODBC and FreeTDS drivers. The script gets all records from the tables you define, and inserts them in the PostgreSQL database using the PostgreSQL DBI driver.

These are the required packages on a Debian Squeeze system:

libdbd-odbc-perl
libdbd-pg-perl
tdsodbc
unixodbc
tdsodbc

Define the FreeTDS ODBC driver in /etc/odbcinst.ini:

[FreeTDS]
Description	= TDS driver (Sybase/MS SQL)
Driver		= /usr/lib/odbc/libtdsodbc.so
Setup		= /usr/lib/odbc/libtdsS.so
UsageCount	= 2

Then define the database in odbc.ini:

[database]
Driver = FreeTDS
Trace = No
Server = hostname
Port = 1433
Database = databasename

The script assumes that all field names in the PostgreSQL database are using lowercase names and that the MS SQL database is using ISO8859-15 encoding and the PostgreSQL database is using UTF-8 encoding. For every table, I make a migrateTable call with 3 arguments: the table name, an array containing all field names and the name of the primary key field for which you created a sequence (named fieldname_seq).

Once the database itself was migrated, the applications using the database had to be modified to use PostgreSQL. This was not too difficult, because the applications were PHP scripts using ODBC. These additional packages were needed:

odbc-postgresql
php5-odbc

To configure the PostgreSQL ODBC driver, add this in /etc/odbcinst.ini:

[PostgreSQL Unicode]
Description	= PostgreSQL ODBC driver (Unicode version)
Driver		= /usr/lib/odbc/psqlodbcw.so
Setup		= /usr/lib/odbc/libodbcpsqlS.so
Debug		= 0
CommLog		= 0
UsageCount	= 0

Then define the databse in odbc.ini:

[database]
Driver              = PostgreSQL Unicode
ServerName          = localhost
Database            = dbname
Username            = username
sslmode             = require

Then be sure not to call odbc_connect with the SQL_CUR_USE_ODBC option (which was actually needed with FreeTDS to fix some weird errors), because it causes segfaults.

The result? WISE’s publications pages is now using PostgreSQL as a back-end!

On these systems, the mails were saved in ~/Mail. This directory did not contain a standard maildir structure or mbox files, but some weird mix of those two. I do not know whether this is typical for KMail or whether this was a peculiarity on these systems because the e-mails have been migrated from even other e-mail clients and versions in the past already.

The first step was to make a standard maildir out of this structure. I created ~/Maildir and searched for all directories called “cur” in ~/Mail. All parents of these directories are standard maildir folders, so I copied these to ~/Maildir. One of them was called inbox. In maildir, the inbox is just the parent directory and not some folder called inbox. So I moved the cur new and tmp subdirectories of ~/Maildir/inbox to ~/Maildir itself. Then I renamed all other subfolders in ~/Maildir so that they contained a dot before the actual folder name, as this is how subfolders are named in maildir.

Now I was still left with a few mbox mail folders. These are single files which contain a whole mailbox. In order to convert them, I used the mb2md script. I ran mb2md -s ~/Mail -R ~/Maildir/ to convert all remaining mbox folders to the maildir structure.

Then I had a nice maildir structure, but Evolution cannot directly import Maildir folders. However it should be possible to create a local Maildir account in Evolution, pointing to the ~/Maildir directory and then copying over all folders. I did not try this. Instead, I installed the Dovecot IMAP server and edited /etc/dovecot.conf. I set these options:

protocols = imap
mail_location = maildir:~/Maildir

and restarted dovecot. Then I configured an IMAP+ account on localhost in Evolution, and dragged and dropped all IMAP folders to the local folders.

KMail’s address book is stored in ~/.kde/share/apps/kabc. Double clicking on the most recent vcf file there, should be enough to open and import it in Evolution.

The old servers were running Debian Lenny, while the new server runs Debian Squeeze, which is frozen since yesterday. That was a nice coincidence. The new server actually hosts three virtual machines using the Linux KVM virtualization system. One of them hosts the website http://wise.vub.ac.be, a second VM hosts a file server, and a third VM is the gateway for a small internal network.

The migration went pretty well, except for the smaller problems which you can always expect with such things, for example a Java web application which had hard-coded host names which had changed. The update to PHP 5.3 caused some compatibility problems which I already fixed before the migration itself: Dokuwiki had to be updated to a recent version in order to not show any warnings. There was also a website based on Joomla 1.0, which is actually not supported anymore. As updating it to a recent version was not really an option, I found some posts on the web on how to fix the errors I was seeing and cooked up a simple patch.

The new server is using 140 Watts most of the time. Keeping in mind that in the future it will also replace a third old server, that will probably be a nice reduction of power consumption compared to the old situation.

Now that I have a new battery, it is time to take a look again at trying to lower power consumption as much as possible. On my system I am using Mandriva 2010.1 Cooker with GNOME 2.30 and laptop-mode-tools 1.54, but this howto should apply (sometimes with slight modifications) to other distributions too.

Setting up ondemand cpufreq governor

The first thing to check is whether CPU frequency scaling is enabled. This should be supported on most up to date CPUs, including Intel Pentium M and Intel Core series and all AMD Athlon 64 and Turion CPUs.

Install the cpufrequtils package:

# urpmi cpufrequtils


Now run the cpufreq-info command to see the current cpufreq settings of your processors. It should show the different clock frequencies supported by your processors and which cpufreq governor and which frequency is currently active. Even when not running on battery power, the “ondemand” cpufreq governor should be used so that it will use the lowest frequency when your system is mostly idle and will switch to the fastest frequency when your system gets loaded. To configure the ondemand governor by default, you can install the cpufreq package with urpmi. It will load the governor set in /etc/sysconfig/cpufreq at boot up, which is ondemand by default.

If cpufreq-info complains that no cpufreq driver is loaded, you can try to run

# modprobe acpi-cpufreq

if you have an Intel CPU or

# modprobe powernow-k8

if you have an AMD CPU and retry. These are the most widely used cpufreq drivers. If one of these is successful and it was not loaded automatically, you can add the right driver name to the /etc/modprobe.preload file so that it gets loaded automatically at the next boot.

Setting up gnome-power-manager

In GNOME open the menu System and then choose Preferences – Power Management.

You can already set up some power savings measures in the AC Power tab. Personally I have it set to go to sleep after 1 hour, to blank the screen when the laptop lid is closed and to put the display to sleep after 5 minutes of inactivity. I have set the display brightness to 100% and disabled dimming on idle.

In the Battery Power tab I have configured to go to sleep when inactive for 10 minutes and to blank the screen when the lid is closed. When battery power is critically low I let it hibernate (so that no data is lost when the battery runs out completely) and I let the hard drives spin down. The display goes to sleep after 1 minute of activity and the display is dimmed when idle.

If you have root access on your laptop, I recommend clicking on the Make Default button, so that it will use these settings for all users and also when you are not logged in yet.

Now there are some more advanced settings which are not shown in gnome-power-manager’s preferences utility. Start up gconf-editor (System – Preferences – Configuration Editor) and navigate to /apps/gnome-power-manager in the tree of the left side. Under that folder you will find the subfolder backlight. Select it to see more detailed settings at the right side. There I have set the brightness_dim_battery key to 80 for a still lower brightness when running on battery power. You will see the effects of changing the value immediately when you are running on battery, so you can try out different values to find what suites you best. The idle_brightness key (the brightness to be used when idle) is set to 20 and idle_dim_time (amount of seconds after which the screen backlight will be dimmed) to 15.

laptop-mode-tools

Make sure you have the laptop-mode-tools package installed. This tool will automatically implement all kinds of power saving settings when you system switches to battery power.

Open the file /etc/laptop-mode/laptop-mode.conf in a text editor as root. Only a few settings need to be tweaked.

To enable most power saving modules, set

ENABLE_AUTO_MODULES=1


When running on battery power, it is recommended to make sure all file systems are mounted with the noatime option, in order not to cause any writes every time a file is read. Set

CONTROL_NOATIME=1
USE_RELATIME=0


Laptop-mode-tools also controls the hard disk idle timeout. I set it to the same values as gnome-power-manager so that they will not override each other:

CONTROL_HD_IDLE_TIMEOUT=1
LM_AC_HD_IDLE_TIMEOUT_SECONDS=60
LM_BATT_HD_IDLE_TIMEOUT_SECONDS=60
NOLM_HD_IDLE_TIMEOUT_SECONDS=7200


The various laptop-mode-tools are configured in the files in /etc/laptop-mode-tools/conf.d. Again the defaults are pretty nice. I made these changes:

• I do not use Bluetooth very often and when I use it, I am always on AC power. So to save battery time I ldisable Bluetooth when running on battery power. In /etc/laptop-mode-tools/conf.d/bluetooth.conf, set
  
CONTROL_BLUETOOTH=1
BATT_ENABLE_BLUETOOTH=0
AC_ENABLE_BLUETOOTH=1
BLUETOOTH_INTERFACES="hci0"

• Thanks to the cpufreq package, we already set up the ondemand cpufreq governor. However when running on battery it is even better to run the conservative governor: instead of going to full speed when the machine gets loaded, the conservative governor will more gradually switch to the frequencies between the minimum and maximum frequencies. In /etc/laptop-mode-tools/conf.d/cpufreq.conf set
  
BATT_CPU_GOVERNOR=conservative

• In the file /etc/laptop-mode-tools/hal-polling.conf, HAL polling is disabled by default when running on battery. If you do not want this (it might prevent automounting of CD and DVD media when running on battery power, set
  
CONTROL_HAL_POLLING=0

  However, if you want to enable this, check that your CD/DVD drive is mentioned the polling devices configured in that file. By default it will disable polling on all /dev/scd? devices but on my system the DVD drive is called /dev/sr0, so I modified it to /dev/sr?. Unfortunately, laptop-module-tools does not yet support udisks-daemon, a replacement for HAL which also polls the CD/DVD drive.

Laptop-mode-tools can also disable certain services when running on battery power. This is interesting for scheduling daemons like crond and atd.

cd /etc/laptop-mode/batt-stop
ln -s /etc/init.d/atd .
ln -s /etc/init.d/crond .


After changing the laptop-mode-tools configuration, restart the service:

# /etc/init.d/laptop-mode-tools restart


Enabling caching in rsyslog

In order to prevent too much disk wake-ups because of logging it is recommended to enable caching for all log files in rsyslog. Edit /etc/syslog.conf and make sure there is a – before every /var/log file name. Then restart rsyslog:

# /etc/init.d/rsyslog restart


Run powertop to identify applications causing too many wake-ups

Install the powertop package and run powertop as root. After it has collected enough data it will show the processes causing the most CPU wake-ups. Stop any unneeded applications causing lots of wake-ups. Powertop will also give you power saving recommendations but you can actually ignore them: if you have set up laptop-mode-tools as explained above, all of them are implemented already.

Disable WiFi if you do not use it

And last but not least: switch off your wireless network card with the wifi kill switch on your laptop if you do not use it. This simple trick can easily give you more than 10% more battery life time. You can also disable it when you temporarily do not need the network connection, for example while reading long web pages or while writing e-mails.

With all these measures in place and with a bit of conservative wifi usage, it seems my battery will last more than 7 hours. That is a nice improvement!

It was a very interesting experience: working out procedures to migrate the mailboxes (from Dovecot on the Linux server and Cyrus on the Mac server to Cyrus on the new server), finding out how to set up one NIC in two different subnets (especially the routing is a little bit tricky), getting all services hooked up to LDAP and managed by GOSA, getting dhcpd to do exactly what we want in a shared-network set up, and much more.

The new server is a HP DL185 G5 with an AMD Opteron quad core CPU and 8 GB of RAM and hosts two KVM virtual machines, one for public services and another one running internal services. You can visit the two websites, which are also hosted on this machine of course, of the concerned research labs:

• Artificial Intelligence Laboratory
• Computational Modeling Lab

Maybe in the not too far away future, I should try to move the services hosted on the underpowered desktop machine running this website, also to a virtual machine…

I noticed that also simple commands, such as ps and getent passwd were hanging too and that I could not log in via SSH anymore. Fortunately, the existing SSH connections continued to work, so I was not locked out yet.

I straced getent passwd and noticed that it hung while trying to connect to the remote LDAP server. The problem was apparently that Lenny’s libnns_ldap tried to connect via LDAPS to port 389, while LDAPS uses port 636 by default. It seems that you need to specify the port number 636 now to make LDAPS work right, so for example: ldaps://remote.host:636. I fixed this in libnss-ldap.conf and pam_ldap.conf, and then I could finish the upgrade without any problem. Apparently this is a known problem.

So definitely modify your configuration first if you are using LDAP authentication and want to upgrade to Lenny. I should probably also fix my nsswitch.conf so that applications don’t start to hang if the LDAP server is unreachable…

In spite of this problem, the whole upgrade was done in less than 1 hour. Without that problem, I guess it would have taken about 20 minutes less. Quite impressive!