System update

First of all, it is important that your system is up to date. Update it to Debian testing if you have not done that yet. It will give your, amongst others:

• Updated eglibc 2.13, which includes functions optimized for instruction sets SSE2, SSSE3  and SSE4.2 provided by recent processors
• Updated GCC 4.5 (with GCC 4.6 being on the way)
• Updated graphics stack with Xserver 1.10, Mesa 7.10.2 and new X drivers and up to date pixman and cairo libraries, all improving performance
• A recent kernel which brings improvements to process and disk schedulers, better hardware drivers, transparant hugepages (see further), scalability improvements to the EXT4 and XFS file systems and the Virtual File System layer, vhost-net for reduced network latency for KVM virtual machines and more. Debian testing has a 2.6.38 kernel, while 2.6.39 is available in unstable and will migrate to testing in the near future.
• Parts of GNOME 2.32, such as Evolution which has improved start-up performance and important bug fixes (for example support of mailboxes larger than 2GB)
• Iceweasel 4 is available in Debian Experimental and the upcoming 5 version, bringing even more performance improvements, is already available in an external repository.

Transparant hugepages

Transparant hugepages is a feature introduced in Linux 2.6.38 which can improve performance of applications. The processor has a translation lookaside buffer (TLB) which is a CPU cache used to speed up mapping of virtual memory addresses to physical memory addresses. This TLB has a limited size. By transparently combining several small 4 KB pages to larger “hugepages”,  more pages can fit into the TLB. Transparent hugepages can be enabled on the fly, however it will only have effect on applications started after you have enabled this feature. For this reason, it is best to activate it right from the start by using a kernel boot parameter. With transparent_hugepage=always, the kernel will use transparant hugepages for any process. If you want to use transparent hugepages only for applications which explicitly indicate that they prefer hugepages, you can use transparent_hugepage=madvise. You have to add one of these boot parameters to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub in Debian, and run the update-grub command. At the next boot, you can take a look at the contents of the /sys/kernel/mm/transparent_hugepage file to verify that it is really enabled.

More information:

• https://lwn.net/Articles/423584/
• https://lwn.net/Articles/423592/

Tuning ondemand cpufreq governor

The ondemand cpufreq governor (which should be used on most systems by default; make sure you have Debbian’s cpufrequtils package installed) tends to switch back to slower CPU frequency speeds a bit too early in some cases, hurting performance. By setting the sampling_down_factor to a value higher than 1, you can prevent it from reducing the clock speed too quickly.

I have added this to my /etc/rc.local script:

if test -f /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
then
    echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
fi

On server systems, I even use 100 instead of 10.

More information:

• https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=3f78a9f7fcee0e9b44a15f39ac382664e301fad5
• https://forums.gentoo.org/viewtopic-t-877539.html?sid=bb3b956d1960647972fc73b3458ff6c8

VM dirty ratio

By default the VM dirty ratio is set to 20% and the background dirty ratio is set to 10%. This means that when 10% of memory is filled with dirty pages (cached data which has to be flushed to disk), the kernel will start writing out the data to disk into the background, without interrupting processes. If the amount of dirty pages raises up to 20%, processes will be forced to write out data to disk and cannot continue other work before they have done so.

By increasing these values, more data will be cached into memory, and data will be written out in larger I/O streams. This can be a good thing on servers with fast storage and lots of memory.

To increase these values, create a file /etc/sysctl.d/dirty_ratio.conf with these contents:

vm.dirty_ratio = 40
vm.dirty_background_ratio = 15

Then with the command <code>sysctl -p /etc/sysctl.d/dirty_ratio.conf</code> you make these settings become in effect immediately.

On desktop systems, the default dirty_ratio of 20 and dirty_background_ratio of 10 should be reasonable. You do not want a too high dirty_ratio on desktop systems, because applications will stall for too long if they have to write out all these dirty pages at once.

More information:

• http://www.kernel.org/doc/Documentation/sysctl/vm.txt
• http://www.scribd.com/doc/46600383/RHEL-Perf-Tuning

CFS scheduler tuning

CFS (Competely Fair Scheduler) is the name of the Linux process scheduler. By default it is tuned for desktop workloads. For server systems where throughput is more important than latency, Red Hat’s tuned package proposes these sysctl settings for CFS for servers:
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000

More information:

• http://www.kernel.org/doc/Documentation/scheduler/sched-design-CFS.txt

ulatencyd

ulatencyd is a daemon which uses cgroups to give hints to the kernel’s process scheduler CFS to improve desktop latency and make applications feel more responsive. It will prevent individual applications from hogging the system, slowing down other applications . This is somewhat simpler than the much hyped (but controversial) autogroup kernel patch but this solution is much more extensive in that ulatencyd knows different applications and desktops and knows how to configure the scheduler to improve responsiveness.

On Debian, you install the ulatencyd package and start the ulatencyd init script. The ulatency package contains the ulatency client too, which shows you the cgroups ulatencyd has set up.

In my opinion, ulatencyd is great for desktop systems but I do not recommend to install this on server systems.

More information:

• https://github.com/poelzi/ulatencyd/wiki

KVM performance improvements

When using Linux guests in kvm virtual machines, it is important to configure the network interface and hard drive as virtio devices in order to experience the best performance. KVM also benefits from transparant hugepages: be sure to enable them both in the host as in the guest machine.

More information:

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaav/LPC/LPCKVMSSPV2.1.pdf
• http://www.linux-kvm.org/page/Tuning_KVM

VHostNet

VHostNet improves network latency and throughput of KVM guests. To enable it, you need to load the vhost_net kernel module in your host system. In Debian you can add vhost_net to /etc/modules to load it automatically when booting the system. Then if you use libvirt to manage your virtual machines, VHostNet will be used automatically when starting virtual machines. If you start qemu-kvm by hand, you will need to add the vhost=on option to the netdev device.

More information:

• http://www.linux-kvm.org/page/VhostNet

Raw devices, disable cache and choose deadline I/O scheduler

For best performance, raw devices are recommended instead of qcow2 or other image files. In libvirt/virt-manager I have defined a storage pool on an LVM volume group, and let virt-install create logical volumes on it containing raw images.

It is recommended to disable I/O caching in KVM because it reduces data copies and bus traffic. In the libvirt XML file for your virtual machine, set the cache=’none’ attribute for the driver tag for the disk device. You can also use virt-manager to make this change: look for the cache mode under the advanced options for the disk.

Benchmarks seems to indicate that it is best to to use the deadline I/O scheduler instead of the default CFQ scheduler.  Using deadline in the guest seems also beneficial.  To make deadline the default scheduler, edit /etc/default/grub.conf and add elevator=deadline to the GRUB_CMDLINE_LINUX_DEFAULT variable.

More information:

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbpioschedulers.htm

Native AIO

Native AIO offers better performance than the thread based AIO in KVM. However, it should only be enabled on raw images, because it can lead to disk corruption in some cases otherwise. This problem is supposed to be fixed in recent kernels according to information I got on the #kvm IRC channel, but better be safe than sorry.

To enable this, add the parameter io=’native’ to the driver tag of the disk in the XML file for the virtual machine in libvirt.

• https://access.redhat.com/kb/docs/DOC-40644

CPU features

By default, KVM only provides a common limited set of CPU instructions implemented by different CPU’s from Intel and AMD. This is needed to permit live migration of a virtual machine to hardware with a CPU which does not implement all instructions available in the original system. If you do not plan on doing doing that, you can enable all instruction sets of your host CPU in the virtual machines, so that your virtual machine can make use of all advanced features of your CPU (for example SSE3 and others). The easiest way to do this, is by using virt-manager. Click on “Copy host configuration” in the Processor – Configuration settings of the virtual machine. The next time you start up the virtual machine, it will have access to all extended instruction sets of your CPU.

KSM

Kernel Samepage Merging is a kernel feature which merges identical pages in memory. If you are using different virtual machines, with the same operating system and applications running in it, lots of memory pages will actually be identical. KSM will save memory by merging the identical pages.

To enable this on Debian, I have put this in my /etc/rc.local script:

echo 1 > /sys/kernel/mm/ksm/run
echo 200 > /sys/kernel/mm/ksm/sleep_millisecs


The last line is optional. It raises the interval during two different memory scans, so that the CPU is not too busy scanning for duplicate memory pages all the time.

More information:

• http://www.linux-kvm.org/page/KSM

• http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaat/liaatbpioschedulers.htm

zram disk

If you do not have much RAM available in your system, it is useful to compress part of the data in memory. This can be done by using a zram disk, which is a ram disk on which all data is transparently compressed. On this zram disk you create a swap partition which you give a higher priority than the normal on disk swap space. Once the available RAM (total RAM – RAM reserved for zram disk) is used, data will be swapped out to the zram disk, which is much faster than swap space on a rotating hard disk. This way, more data fits into the RAM.

On my 1 GB netbook system which runs a full GNOME desktop, I have reserved 512 MB for the zram disk. To do so, I added the following in /etc/rc.local:

echo $((512*1024*1024)) > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon -p 60 /dev/zram0

Of course, a better solution is to add RAM to your system, especially on server systems.

Con Kolivas has been maintaining an alternative scheduler for some time. His Staircase scheduler was designed with interactivity in mind, especially for desktop systems, where people want their system to be quickly responsive under all kinds of workloads. This scheduler has been optimized a lot through the years, and as such is very stable. Still there are some rare cases where “starvation” is possible.

At the start of March, Con Kolivas published a new scheduler, which was called RSDL (Rotating Staircase DeadLine scheduler) at first, and has been renamed to SD (Staircase Deadline) afterwards. Based on the experience Con Kolivas gathered with his Staircase scheduler, SD is a more general purpose scheduler, trying to give absolute fairness to the different running tasks, without favouring any process (for example lots of other schedulers favour X). This way, no starvation issues should be possible with this scheduler. A lot of discussion followed after his announcement, and it became quickly clear that a lot of people were not happy with the current scheduler in Linux. Important kernel developers like Mike Galbraith, Nick Piggin, Ingo Molnar, Willy Tarreau and Andrew Morton joined the discussion and also posted other scheduler patches, sometimes not without some trolling and flaming as sometimes happen on such mailing lists. Con Kolivas’ scheduler was added to Andrew Morton’s mm kernel tree to get some more testing. The development of RDSL/SD went up and down sometimes, because of Con Kolivas’ health problems.

Ingo Molnar, which was rather critical of some of the ideas in the new scheduler at first, also recently began the development of CFS (Completely Fair Scheduler, which actually is based on the same basic concept of fairness. Con Kolivas announced that he would stop development of the RD scheduler, because of his health problems, and because his ideas would now continue to be used in the CFS scheduler. But things came out differently, and Con Kolivas continued development in the end. The result is that the SD scheduler is now at its 46′th version (v. 0.46), and it seems most problems have been fixed. Based on all the testing done on the kernel mailing list, it seems SD 0.46 is more mature than CFS 6. Even Willy Tarreau, maintainer of the 2.4 Linux kernel tree, said that thanks to SD, he did make Linux 2.6 the default kernel on his laptop, as he found the scheduler in mainline 2.6 too bad compared to 2.4. It’s unclear however which of these schedulers will be integrated in linux finally, and when this will happen.

Personally I think SD 0.46 should be integrated now in Linux 2.6.22 pre-releases. There has been a lot of testing and bug fixing, and it seems there are no serious bugs open anymore now. I also hope that Mandriva 2008 will come out with one of these new schedulers. The tmb kernel in Mandriva Cooker, already uses the SD scheduler now. People interested in this discussion, can subscribe to the ck mailing list where a lot of the discussion is happening. Sites like LWN.net and Kerneltrap also often post about the progress of this subject.

Compiling kdepim on an AMD64 system, seems to require a huge amount of memory (much more than on x86 32 bit). 1 GB of RAM and about 250 MB of swap did not prevent g++ eating up all of my memory (even when no other services were active!). Unfortunately, this made Linux become completely unstable: the hard drive started thrashing the whole time, and the system was completely unresponsive. I could only stop it by doing a hard reset. I am clearly not the only one hating this stupid Linux behaviour.

In the Mandriva Cooker channel (irc.freenode.org, #mandriva-cooker), couriousous suggested to execute

# echo 2 > /proc/sys/vm/overcommit_memory

The default value is 0: when an application asks more memory than is available, Linux will still try to allocate it, even if chances exist that it won’t be available. This seems to be done because some applications ask more memory than what they will really use. If in the end, no more memory or swap is availed, Linux’ OOM killer will kill some (random) processes to free up memory. By setting overcommit_memory to 2, the allocation of too much memory will fail immediately. The application can then react itself to the fact that not enough memory is available. The result was that instead of bringing my system to death, the g++ compiler just exited with the message that I was out of memory. Much nicer! I found a complete technical explanation about memory allocation in Linux on the web. To make this setting default, I put vm.overcommit_memory = 2 in my /etc/sysctl.conf. I do not understand why it is not the default value, making a system unresponsive for several (tens of) minutes does not seem very friendly…

Update 16 april 2007: It seems like this setting has severe problems as well. On my system with 384 MB RAM, which I use as server and desktop, several applications randomly crashed because they did not get the memory they wanted (Evolution, Tilda,…). Changing the setting back to 0, made these applications work correctly again. I suppose playing with the overcommit_ratio value as explained in the article which I mentioned above, can improve this behaviour. But anyway, it sucks that such things are so difficult to get right. This should really be working nicely out of the box

Anyway, in the end I got kdepim compiled on AMD64 by adding some more swap. From now on, I’ll be creating bigger swap partitions is Linux, it can realy be useful, even when you think you have enough memory…

I also built a freetype 2.3.4 RPM for Mandriva. Font rendering on my flat panel is now much nicer comparing with freetype 2.3.1 which is included in Mandriva 2007.1! Not that it was ugly before, but it’s a nice surprise to still see such big improvements, especially from a minor update.