System update

First of all, it is important that your system is up to date. Update it to Debian testing if you have not done that yet. It will give your, amongst others:

• Updated eglibc 2.13, which includes functions optimized for instruction sets SSE2, SSSE3  and SSE4.2 provided by recent processors
• Updated GCC 4.5 (with GCC 4.6 being on the way)
• Updated graphics stack with Xserver 1.10, Mesa 7.10.2 and new X drivers and up to date pixman and cairo libraries, all improving performance
• A recent kernel which brings improvements to process and disk schedulers, better hardware drivers, transparant hugepages (see further), scalability improvements to the EXT4 and XFS file systems and the Virtual File System layer, vhost-net for reduced network latency for KVM virtual machines and more. Debian testing has a 2.6.38 kernel, while 2.6.39 is available in unstable and will migrate to testing in the near future.
• Parts of GNOME 2.32, such as Evolution which has improved start-up performance and important bug fixes (for example support of mailboxes larger than 2GB)
• Iceweasel 4 is available in Debian Experimental and the upcoming 5 version, bringing even more performance improvements, is already available in an external repository.

Transparant hugepages

Transparant hugepages is a feature introduced in Linux 2.6.38 which can improve performance of applications. The processor has a translation lookaside buffer (TLB) which is a CPU cache used to speed up mapping of virtual memory addresses to physical memory addresses. This TLB has a limited size. By transparently combining several small 4 KB pages to larger “hugepages”,  more pages can fit into the TLB. Transparent hugepages can be enabled on the fly, however it will only have effect on applications started after you have enabled this feature. For this reason, it is best to activate it right from the start by using a kernel boot parameter. With transparent_hugepage=always, the kernel will use transparant hugepages for any process. If you want to use transparent hugepages only for applications which explicitly indicate that they prefer hugepages, you can use transparent_hugepage=madvise. You have to add one of these boot parameters to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub in Debian, and run the update-grub command. At the next boot, you can take a look at the contents of the /sys/kernel/mm/transparent_hugepage file to verify that it is really enabled.

More information:

• https://lwn.net/Articles/423584/
• https://lwn.net/Articles/423592/

Tuning ondemand cpufreq governor

The ondemand cpufreq governor (which should be used on most systems by default; make sure you have Debbian’s cpufrequtils package installed) tends to switch back to slower CPU frequency speeds a bit too early in some cases, hurting performance. By setting the sampling_down_factor to a value higher than 1, you can prevent it from reducing the clock speed too quickly.

I have added this to my /etc/rc.local script:

if test -f /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
then
    echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
fi

On server systems, I even use 100 instead of 10.

More information:

• https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3f78a9f7fcee0e9b44a15f39ac382664e301fad5
• https://forums.gentoo.org/viewtopic-t-877539.html?sid=bb3b956d1960647972fc73b3458ff6c8

VM dirty ratio

By default the VM dirty ratio is set to 20% and the background dirty ratio is set to 10%. This means that when 10% of memory is filled with dirty pages (cached data which has to be flushed to disk), the kernel will start writing out the data to disk into the background, without interrupting processes. If the amount of dirty pages raises up to 20%, processes will be forced to write out data to disk and cannot continue other work before they have done so.

By increasing these values, more data will be cached into memory, and data will be written out in larger I/O streams. This can be a good thing on servers with fast storage and lots of memory.

To increase these values, create a file /etc/sysctl.d/dirty_ratio.conf with these contents:

vm.dirty_ratio = 40
vm.dirty_background_ratio = 15

Then with the command <code>sysctl -p /etc/sysctl.d/dirty_ratio.conf</code> you make these settings become in effect immediately.

On desktop systems, the default dirty_ratio of 20 and dirty_background_ratio of 10 should be reasonable. You do not want a too high dirty_ratio on desktop systems, because applications will stall for too long if they have to write out all these dirty pages at once.

More information:

• http://www.kernel.org/doc/Documentation/sysctl/vm.txt
• http://www.scribd.com/doc/46600383/RHEL-Perf-Tuning

CFS scheduler tuning

CFS (Competely Fair Scheduler) is the name of the Linux process scheduler. By default it is tuned for desktop workloads. For server systems where throughput is more important than latency, Red Hat’s tuned package proposes these sysctl settings for CFS for servers:
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000

More information:

• http://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt

ulatencyd

ulatencyd is a daemon which uses cgroups to give hints to the kernel’s process scheduler CFS to improve desktop latency and make applications feel more responsive. It will prevent individual applications from hogging the system, slowing down other applications . This is somewhat simpler than the much hyped (but controversial) autogroup kernel patch but this solution is much more extensive in that ulatencyd knows different applications and desktops and knows how to configure the scheduler to improve responsiveness.

On Debian, you install the ulatencyd package and start the ulatencyd init script. The ulatency package contains the ulatency client too, which shows you the cgroups ulatencyd has set up.

In my opinion, ulatencyd is great for desktop systems but I do not recommend to install this on server systems.

More information:

• https://github.com/poelzi/ulatencyd/wiki

KVM performance improvements

When using Linux guests in kvm virtual machines, it is important to configure the network interface and hard drive as virtio devices in order to experience the best performance. KVM also benefits from transparant hugepages: be sure to enable them both in the host as in the guest machine.

More information:

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaav/LPC/LPCKVMSSPV2.1.pdf
• http://www.linux-kvm.org/page/Tuning_KVM

VHostNet

VHostNet improves network latency and throughput of KVM guests. To enable it, you need to load the vhost_net kernel module in your host system. In Debian you can add vhost_net to /etc/modules to load it automatically when booting the system. Then if you use libvirt to manage your virtual machines, VHostNet will be used automatically when starting virtual machines. If you start qemu-kvm by hand, you will need to add the vhost=on option to the netdev device.

More information:

• http://www.linux-kvm.org/page/VhostNet

Raw devices, disable cache and choose deadline I/O scheduler

For best performance, raw devices are recommended instead of qcow2 or other image files. In libvirt/virt-manager I have defined a storage pool on an LVM volume group, and let virt-install create logical volumes on it containing raw images.

It is recommended to disable I/O caching in KVM because it reduces data copies and bus traffic. In the libvirt XML file for your virtual machine, set the cache=’none’ attribute for the driver tag for the disk device. You can also use virt-manager to make this change: look for the cache mode under the advanced options for the disk.

Benchmarks seems to indicate that it is best to to use the deadline I/O scheduler instead of the default CFQ scheduler.  Using deadline in the guest seems also beneficial.  To make deadline the default scheduler, edit /etc/default/grub.conf and add elevator=deadline to the GRUB_CMDLINE_LINUX_DEFAULT variable.

More information:

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbpioschedulers.htm

Native AIO

Native AIO offers better performance than the thread based AIO in KVM. However, it should only be enabled on raw images, because it can lead to disk corruption in some cases otherwise. This problem is supposed to be fixed in recent kernels according to information I got on the #kvm IRC channel, but better be safe than sorry.

To enable this, add the parameter io=’native’ to the driver tag of the disk in the XML file for the virtual machine in libvirt.

• https://access.redhat.com/kb/docs/DOC-40644

CPU features

By default, KVM only provides a common limited set of CPU instructions implemented by different CPU’s from Intel and AMD. This is needed to permit live migration of a virtual machine to hardware with a CPU which does not implement all instructions available in the original system. If you do not plan on doing doing that, you can enable all instruction sets of your host CPU in the virtual machines, so that your virtual machine can make use of all advanced features of your CPU (for example SSE3 and others). The easiest way to do this, is by using virt-manager. Click on “Copy host configuration” in the Processor – Configuration settings of the virtual machine. The next time you start up the virtual machine, it will have access to all extended instruction sets of your CPU.

KSM

Kernel Samepage Merging is a kernel feature which merges identical pages in memory. If you are using different virtual machines, with the same operating system and applications running in it, lots of memory pages will actually be identical. KSM will save memory by merging the identical pages.

To enable this on Debian, I have put this in my /etc/rc.local script:

echo 1 > /sys/kernel/mm/ksm/run
echo 200 > /sys/kernel/mm/ksm/sleep_millisecs


The last line is optional. It raises the interval during two different memory scans, so that the CPU is not too busy scanning for duplicate memory pages all the time.

More information:

• http://www.linux-kvm.org/page/KSM

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaat/liaatbpioschedulers.htm

zram disk

If you do not have much RAM available in your system, it is useful to compress part of the data in memory. This can be done by using a zram disk, which is a ram disk on which all data is transparently compressed. On this zram disk you create a swap partition which you give a higher priority than the normal on disk swap space. Once the available RAM (total RAM – RAM reserved for zram disk) is used, data will be swapped out to the zram disk, which is much faster than swap space on a rotating hard disk. This way, more data fits into the RAM.

On my 1 GB netbook system which runs a full GNOME desktop, I have reserved 512 MB for the zram disk. To do so, I added the following in /etc/rc.local:

echo $((512*1024*1024)) > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon -p 60 /dev/zram0

Of course, a better solution is to add RAM to your system, especially on server systems.

The old servers were running Debian Lenny, while the new server runs Debian Squeeze, which is frozen since yesterday. That was a nice coincidence. The new server actually hosts three virtual machines using the Linux KVM virtualization system. One of them hosts the website http://wise.vub.ac.be, a second VM hosts a file server, and a third VM is the gateway for a small internal network.

The migration went pretty well, except for the smaller problems which you can always expect with such things, for example a Java web application which had hard-coded host names which had changed. The update to PHP 5.3 caused some compatibility problems which I already fixed before the migration itself: Dokuwiki had to be updated to a recent version in order to not show any warnings. There was also a website based on Joomla 1.0, which is actually not supported anymore. As updating it to a recent version was not really an option, I found some posts on the web on how to fix the errors I was seeing and cooked up a simple patch.

The new server is using 140 Watts most of the time. Keeping in mind that in the future it will also replace a third old server, that will probably be a nice reduction of power consumption compared to the old situation.

During the next days, there might be some short downtime now and then while I continue configuring things

It was a very interesting experience: working out procedures to migrate the mailboxes (from Dovecot on the Linux server and Cyrus on the Mac server to Cyrus on the new server), finding out how to set up one NIC in two different subnets (especially the routing is a little bit tricky), getting all services hooked up to LDAP and managed by GOSA, getting dhcpd to do exactly what we want in a shared-network set up, and much more.

The new server is a HP DL185 G5 with an AMD Opteron quad core CPU and 8 GB of RAM and hosts two KVM virtual machines, one for public services and another one running internal services. You can visit the two websites, which are also hosted on this machine of course, of the concerned research labs:

• Artificial Intelligence Laboratory
• Computational Modeling Lab

Maybe in the not too far away future, I should try to move the services hosted on the underpowered desktop machine running this website, also to a virtual machine…

My new Dell Latitude E6400 laptop, is my first system which support hardware visualization. It also has 4 GB RAM, making this an ideal machine to start experimenting with KVM, the kernel based virtual machine system which is now integrated in Linux itself.

In the past I had already used different virtualization systems. VMWare Server is annoying because you have to rebuild the kernel modules by hand each time you install a new kernel and it’s not Free (as in Free Speech). Virtualbox handles the modules rebuild nicely thanks to the use of DKMS in Mandriva, but always randomly crashes on an assert, making it totally unusable for serious work. Also it does not really include a server mode, where virtual machines just run in the background. Xen is especially nice if you have a system which does not support hardware virtualization and you just want to run a paravirtualized Linux in it. However, the host system (Dom0) needs to run a very specific kernel version (normally version 2.6.18, although Debian has a 2.6.26 based one now). OpenVZ is nice for lightweight Linux server containers (some kind of chroot on steroids), but is not a complete virtualization solution.

Because KVM is built into the standard Linux kernel, this means there won’t be any problems with kernel modules which have to be installed separately. It’s a complete virtualization system which means that you can run all kind of operating systems on it, like for example also Windows. KVM is evolving very quickly and its performance and features are constantly improving. Thanks to libvirt and virt-manager, it also has a nice GUI front-end which make configuring and managing virtual machines very easy (libvirt and virt-manager also include support for Xen and OpenVZ, but I don’t know how well that works).

Installing and using KVM on Mandriva 2009.0

Before installing KVM, it’s important to verify that the CPU on your system has hardware virtualization support. Start up a terminal, and run


$ cat /proc/cpuinfo


If you have an Intel CPU, you should have vmx in the flags. If you are using an AMD CPU, you should have svm.

It’s best to use KVM, libvirt and virt-manager from Mandriva’s backports repositories. The versions included there, are the most recent ones, and fix a few problems which made networking not work out of the box with the ones included in the standard repositories. So start up the Mandriva Control Center (Tools – System Tools – Configure Your Computer), and in the Software category, launch the media sources configuration tool and enable the Main Backports and Contrib Backports sources by checking the checkboxes.

Before we install the necessary packages, it’s necessary to make a change to udev’s configuration file to prevent the virbr* and vnet* virtual ethernet devices and bridges being configured automatically by Mandriva’s configuration tools. To do so, edit the file /etc/sysconfig/udev_net and add virbr* and vnet* to the BLACKLIST_ALL line if they are not there yet. The line should look like this:

BLACKLIST_ALL="ppp*|ippp*|isdn*|plip*|hso*|lo*|irda*|dummy*|ipsec*|vmnet*|wifi*|
wmaster*|br*|vbd*|vtpm*|vif*|ax*|nr*|rose*|bce*|scc*|virbr*|vnet*"
Then install the kvm and virt-manager packages with the graphical software installation utility or with urpmi. Some extra dependencies will be installed, such as dnsmasq-base, libvirt-utils and python-virtinst.

After the installation, make sure that the libvirtd service is running. You can use the graphical tool in the Mandriva Control Center in the category System – Manager system services, or you can run “service libvirtd status” to verify whether it’s running and “service libvirtd start” to start it.

Now start the Virtual Machine Manager from Tools – Emulators. Choose File – Add Connection and set up a Local QEMU connection. Double click on the localhost/qemu connection in the virtual machine manager, and enter your root password. You can now create new virtual machines by clicking on the New button. To use KVM, choose a fully virtualized system, and choose KVM as hypervisor. If you want to run Mandriva 2009.0 or later in a KVM based virtual machine, you can probably choose Linux as OS type and Ubuntu Hardy as OS variant, because that should give better performance than the generic options.

I currently have Windows Vista and Debian Lenny running in KVM. It works very fast and very reliably. The only problem I encounter, is that sometimes I cannot switch back from my Windows virtual machine to my host system: the mouse and keyboard cannot leave the VNC window anymore and also the Ctrl-Alt keystroke does not release the mouse pointer. The only solution is to shut down the Windows system.