Frederik's Blog

Since two days, I have merged the main servers used by two research laboratories at work. One server was an old Linux server which really needed a hardware upgrade, and the other one was a Mac Pro machine running a flaky OS X Leopard. The new server is of course running Linux: Debian Lenny.

It was a very interesting experience: working out procedures to migrate the mailboxes (from Dovecot on the Linux server and Cyrus on the Mac server to Cyrus on the new server), finding out how to set up one NIC in two different subnets (especially the routing is a little bit tricky), getting all services hooked up to LDAP and managed by GOSA, getting dhcpd to do exactly what we want in a shared-network set up, and much more.

The new server is a HP DL185 G5 with an AMD Opteron quad core CPU and 8 GB of RAM and hosts two KVM virtual machines, one for public services and another one running internal services. You can visit the two websites, which are also hosted on this machine of course, of the concerned research labs:

• Artificial Intelligence Laboratory
• Computational Modeling Lab

Maybe in the not too far away future, I should try to move the services hosted on the underpowered desktop machine running this website, also to a virtual machine…

Last week-end, Debian Lenny 5.0 was finally released. I use Debian on most servers I manage at work. A few of them were already using Lenny when it was still the testing branch, but most are still on Debian Etch. So this morning I decided to test upgrade one of the less critical Etch systems to Lenny. That system is only used to store back-up files from other systems, so it would not be a problem if that machine was off line for a couple of hours.
According to the release notes, you should rather use aptitude instead of apt to upgrade, so that’s what I did. All went well, until suddenly the package upgrade hung while installing new udev configuration files. I could Ctrl-C the process to continue, but from that moment on, more and more post installation scripts started hanging and had to be interrupted.

I noticed that also simple commands, such as ps and getent passwd were hanging too and that I could not log in via SSH anymore. Fortunately, the existing SSH connections continued to work, so I was not locked out yet.

I straced getent passwd and noticed that it hung while trying to connect to the remote LDAP server. The problem was apparently that Lenny’s libnns_ldap tried to connect via LDAPS to port 389, while LDAPS uses port 636 by default. It seems that you need to specify the port number 636 now to make LDAPS work right, so for example: ldaps://remote.host:636. I fixed this in libnss-ldap.conf and pam_ldap.conf, and then I could finish the upgrade without any problem. Apparently this is a known problem.

So definitely modify your configuration first if you are using LDAP authentication and want to upgrade to Lenny. I should probably also fix my nsswitch.conf so that applications don’t start to hang if the LDAP server is unreachable…

In spite of this problem, the whole upgrade was done in less than 1 hour. Without that problem, I guess it would have taken about 20 minutes less. Quite impressive!