Stresstesting IMAP clients

Armijn Hemel, October 4, 2009, 16145 views.

In this blogpost we describe research we did into performance of several graphical mail programs on Linux when dealing with large amounts of email messages using IMAP.

Tags: , , , , , , ,

The background for this article is that at Loco we are email junkies. A very large part of communication with customers is done via email, so having a good email infrastructure and good clients is very important to us.

Since we have been online since the early/mid-1990s and have been active in several free software communities we have accumulated quite some mails: posts from mailing lists that still might need to be read, mails that have never been deleted because cleaning takes too much time, or work related mails that contain information that is always needed after you have deleted it (so we don't delete it). The amount of mails that I currently have in all IMAP folders combined is more than 46000 and takes about 950 megabytes of diskspace.

In the past I have had to move quite a bit of mail around between IMAP servers, which usually worked just fine, apart from taking some resources. Other people I talked to had different experiences, including crashes and suspicions of emails being lost. This triggered the idea of testing a few graphical email programs for IMAP performance: how would they behave if you had a huge amount of mails that you wanted to move to a different account?

We made a test with a few scenarios, with interesting results. Most of the test results are mainly observations we made from time to time on the machines, with some gut feeling mixed in. In a next test we will try to provide more detail, charts and more accurate measurements and also try to include a few non-Linux clients.

Preparing the test

Generating email

The first thing you need for the test is a huge set of mails. With Python it is very easy to generate a lot of basic test mails in maildir format:

#!/usr/bin/python
 
import sys, os, random
import tempfile
 
from email import Message
from email.MIMEText import MIMEText
from email.mime.application import MIMEApplication
from email.MIMEMultipart import MIMEMultipart
from mailbox import Maildir
 
dir = tempfile.mkdtemp()
print dir
 
testmaildir = Maildir(dir + '/foobar', create=True)
 
for i in range(0, 12000):
        msg = Message.Message()
        msg.add_header('Return-Path', '')
        msg.add_header('Subject', 'test')
        msg.add_header('From', 'example@example.org')
        if random.random() < 0.5:
                msg2 = MIMEMultipart()
                msg2.attach(MIMEText('foobar'))
                msg2.attach(MIMEApplication(os.urandom(random.randint(1,1000000))))
        else:
                msgtext = ""
                for i in range(1, random.randint(1,10000)):
                        msgtext += random.choice('abcdefghijklmnopqrstuvwxyz!@#$%^&*()_- +=')
                msg2 = MIMEText(msgtext)
        msg.set_payload(msg2.as_string())
        testmaildir.add(msg)

The script contains some randomisation so not all emails are exactly the same. Some mails contain a random sized attachment, other contain some random text, both of course with upper boundaries with respect to size. The amount of mails is hardcoded in the above script (12000 mails in this case), but it is trivial to adapt it to generate a different amount of mail.

Please note that the messages that are generated are fairly basic and do not have headers that you usually see in mails, like message IDs, dates and so on. This should not matter for moving them around by a mail client though.

IMAP server installation

Our IMAP server of choice is Dovecot 1.2.5, on Fedora Rawhide (updated up until October 1 2009). The server is configured to accept plain text passwords. The server is also configured to use IMAP only, not IMAPS.

The IMAP server is a 3 GHz Pentium 4, with 1.5 GB of RAM installed and connected with a 100 Mbit connection to the network (switched).

Client setup

The client machine is a Toshiba Satellite L100-129 laptop, with an Intel Core Solo at 1.66 GHz and 1 GB of memory. It runs Fedora 11, with all updates installed until up October 1 2009. The client machine is connected with a 100 Mbit connection to the same network as the server (switched).

The email clients are tested in their native environment: Evolution is tested in a GNOME environment, KMail is tested in a KDE environment. Thunderbird and Claws Mail are tested in a GNOME environment as well.

Test scenarios

In this test I tested two scenarios: downloading 12000 mails generated by the Python script above (about 3.9 GB of data) and moving these mails to a different account.

Downloading mail

The first test is simple: how well does the mail client cope with downloading all mail.

Moving mail to a different account

The second test is a bit more extensive: how well does the mail client perform when mail is moved from one account to another account on the same server.

Evolution test results

The first test was with Evolution 2.26.3.

Downloading mail

Downloading headers for 12000 mails is something which took some time, but which went without a hitch. The load on the client did not see a spectacular increase, but hovered around 1.

Moving mail to a different account

Moving mail with Evolution from one account to another was very uneventful. The load on the client machine went up to one, for about an hour, while messages were copied to the server. The load was quite stable between 1 and 1.5. The only times the load went up significantly was when the GNOME screensaver kicked in (mental note: disable the screensaver when doing long running desktop tests).

KMail IMAP

Next up was KMail 4.3.1. In KMail there are two different implementations of IMAP: regular IMAP and disconnected IMAP.

Downloading mail

Downloading mail headers with the regular IMAP implementation seemed to be a tiny bit faster than Evolution and nothing special happened.

Moving mail to a different account

After the uneventful test with Evolution I was expecting KMail to behalve similarly, but what I saw was quite a surprise. First of all KMail became completely unresponsive after selecting 12000 mails and dragging to them to another account (which makes me wonder: does KMail first read all mails into memory?), but after a few minutes the interface came back and showed it was moving mails around. After 45 minutes KMail had only transferred half of what Evolution did and after around 2500 mails KMail simply crashed. After restarting KMail and transferring another 2500 mails KMail crashed again. This happened a few times, after which I just gave up.

I had heard similar stories from someone who had tried to copy a lot of emails with KMail about 3 months ago.

KMail disconnected IMAP

Another IMAP implementation that is used in KMail is disconnected IMAP. The differences with the regular IMAP code is that it is mainly used for offline use.

Downloading mail

Like with the other mailers downloading mail is not spectacular at all and just worked as expected.

Moving mail to a different account

With the regular IMAP test results in mind I was not expecting much from disconnected IMAP. I was told by KDE developers that the disconnected IMAP code is actually much more used and tested than the regular IMAP. The disconnected IMAP code did indeed perform better than regular IMAP, but just marginally. It only crashed once, but it also forced me once to cut power to the machine once (using the good old 'press the power button for 6 seconds' rule) because the machine had become completely unresponsive.

Thunderbird test results

Mozilla's Thunderbird is a popular cross platform mail client. In this test Thunderbird 3.0 was used.

Downloading mail

As with Evolution and KMail there were no issues with downloading the mails from the server.

Moving mail to a different account

Thunderbird seemed to be really fast with moving mails, but as the amount of transferred mails increased, so did its memory usage. After a few thousand mails Thunderbird was grinding to a halt. Running strace on the binary just spewed out tons of lines like this:

futex(0x92e88400, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb61c3c08, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0xb614fc14, 3256820)

which could indicate there are some threading problems, or perhaps memory leaks. I could not see those lines after restarting Thunderbird. After a restart Thunderbird seemed to finish the task without a hitch, but it became unresponsive after that and the process needed to be killed.

Claws Mail test results

The last mail program I tested was Claws Mail 3.7.2.

Downloading mail

As with Evolution, KMail and Thunderbird there were no issues with downloading the mails from the server.

Moving mail to a different account

When I started Claws Mail for the first time I had high hopes for the performance. The user interface is not very polished (I would even go as far as calling it ugly), so most work has probably gone in the other parts. And Claws Mail delivered. In my test it was faster than Evolution, it did not crash like KMail, or grind away like Thunderbird. The overal load on the client system was also lower than with the other clients (often below 1.00).

Conclusion

The conclusion from this test is fairly obvious: Evolution and Claws Mail outperform both KMail and Thunderbird when faced with the task of moving large amounts of mail between IMAP accounts. KMail seems to have a severe performance problem and Thunderbird could probably use a performance review as well. Of course, this does not mean that Evolution and Claws Mail are overall better mail clients because there are a lot of different aspects to a mail client that did not matter for this test.

Development of the mail clients that were used in this test is ongoing. I'm not sure if there will be any big rewrites of the IMAP code in Evolution, Claws Mail and Thunderbird in the near future. I know for sure that the IMAP and disconnected IMAP code in KMail will be replaced by an Akonadi backend, probably in KDE 4.5 (summer 2010). As soon as there are new major versions of all the mail programs (probably around Fedora 13 or 14) I will repeat my tests.

Social networking: Tweet this article on Twitter Pass on this article on LinkedIn Bookmark this article on Google Bookmark this article on Yahoo! Bookmark this article on Technorati Bookmark this article on Delicious Share this article on Facebook Digg this article on Digg Submit this article to Reddit Thumb this article up at StumbleUpon Submit this article to Furl

Talkback

respond to this article

Re: Stresstesting IMAP clients (bobo, 2009-10-05 18:40 CEST)
e-mail is outdated technology ... :)
Re: Stresstesting IMAP clients (Jazz13, 2009-10-05 19:57 CEST)
> e-mail is outdated technology ... :)
lol
BTW, Claws Mail version is 3.7.2 maybe?
Re: Stresstesting IMAP clients (Armijn Hemel, 2009-10-05 21:42 CEST)
> > e-mail is outdated technology ... :)
> lol
> BTW, Claws Mail version is 3.7.2 maybe?

Yes, you're absolutely right. I've changed it in the article.
Re: Stresstesting IMAP clients (Anonymous, 2009-10-05 23:00 CEST)
Note that by using random text in mails, you make those mails more difficult to compress, and thus you change the benchmark workload somewhat from real mails. Mails normally compress very well. :)

Also, I'd love to see this same test run on mutt. I think you'll find it very competitive on speed. Personally, I use mutt, and connect to my IMAP server via SSH with compression.
Re: Stresstesting IMAP clients (Aaron Toponce, 2009-10-06 03:53 CEST)
> Note that by using random text in mails, you make those mails more
> difficult to compress, and thus you change the benchmark workload
> somewhat from real mails. Mails normally compress very well. :)
>
> Also, I'd love to see this same test run on mutt. I think you'll
> find it very competitive on speed. Personally, I use mutt, and
> connect to my IMAP server via SSH with compression.

I find mutt extremely slow, even with caching folders. I have about 30,000 emails covering about 1.4GB of space and about 45-50 IMAP folders. When opening a folder, even with caching enabled, mutt is 2-3X slower than Thunderbird in my personal experience.
Re: Stresstesting IMAP clients (anonymous, 2009-10-07 08:56 CEST)
> Note that by using random text in mails, you make those mails more
> difficult to compress, and thus you change the benchmark workload
> somewhat from real mails. Mails normally compress very well. :)
>
> Also, I'd love to see this same test run on mutt. I think you'll
> find it very competitive on speed. Personally, I use mutt, and
> connect to my IMAP server via SSH with compression.

I use mutt and have found it a bit painful when dealing with a large imap inbox (like 20 000) of mail. It also would segfault occasionally when quitting. Also it's caching is a bit slow.
Re: Stresstesting IMAP clients (iq-0, 2009-10-06 09:15 CEST)
The trick with thunderbird is that, when moving large amounts of mail, quickly click on a mailbox not involved in the mail move. It seems that much of it's sluggishness and crashing is due to the continues updates of the user interface, it can easily lockup during these activities.
But the memory consumption is completely outrageous, after doing my periodic mail archiving (mostly to local mail folder nowadays), I simply have to restart to reclaim the few gigabytes of virtual memory in use, often with a residency of a 75%. That is a realy problem...
Re: Stresstesting IMAP clients (Nick, 2009-10-06 14:16 CEST)
It might be worth checking out Mailody as that was designed for IMAP
Re: Stresstesting IMAP clients (Nelson, 2009-10-06 16:36 CEST)
I would love to see results for text-based clients, particularly (al)pine and mutt.
Re: Stresstesting IMAP clients (thestonewell, 2009-10-06 18:52 CEST)
Can you run again with Opera 10?
Re: Stresstesting IMAP clients (Samat Jain, 2009-10-09 07:35 CEST)
The IMAP stack in KDE (aka the IMAP kioslave) that KMail uses has not been getting much development or maintenance for the last couple of years.

Akonadi is the new framework for PIM data management and communication. While KMail does not yet use it, hopefully it will be KDE 4.4 or 4.5, and these performance problems with it will go away.
Re: Stresstesting IMAP clients (Mark Banner, 2009-10-21 16:16 CEST)
It is good to see this sort of testing going on, but I'd really like to point out that you did not use Thunderbird 3.0. I don't know which version you used, but Thunderbird 3.0 hasn't been released yet.

Please can you quote the exact versions you used of all your applications - it is quite possible that something is fixed between the version you use and the final release version of that application.

Now I'm off to find somewhere to save this for possible inclusion in our performance tests later on.
Re: Stresstesting IMAP clients (Armijn Hemel, 2009-10-21 18:54 CEST)
> It is good to see this sort of testing going on, but I'd really
> like to point out that you did not use Thunderbird 3.0. I don't
> know which version you used, but Thunderbird 3.0 hasn't been
> released yet.
>
> Please can you quote the exact versions you used of all your
> applications - it is quite possible that something is fixed
> between the version you use and the final release version of that
> application.
>
> Now I'm off to find somewhere to save this for possible inclusion
> in our performance tests later on.

I used the Thunderbird which is shipped in Fedora 11: 3.0-2.8.b4. This is apparently a beta version of Thunderbird.

The version of KMail that was used: 4.3.1, Evolution: 2.26.3, Claws: 3.7.2.