Robin on Tech

A geeky prospect it may be, but it’s often fun to imagine the product development meetings which occurred to produce some of the services and solutions we’re most reliant on. Not necessarily the most common solution. Rather, the products which, through sheer innovation, defined, or redefined a whole product area.

For many technology companies, ‘innovation’ is just a throwaway buzzword to be inserted with requisite regularity into puff marketing brochures and website flash animations. But occasionally, new products tip up, fully deserving of the ‘innovative’ moniker, and it is these products whose genesis is worth considering.

For example, consider the questions Google engineers might have asked themselves when they started specifying the requirements for a Google email product.

• What do we like about current email implementations?

• What do we dislike?

• What would do we want to do but are prevented from doing? How do current systems limit us?

To an extent, Gmail redefined email by adding pseudo-limitless storage, and doing away with folders. Most other email providers were forced to increase their storage quotas, and introduce email ‘tagging’. But the Google engineers omitted a key ingredient: email’s best kept secret.

Gmail’s web interface is one of the best around, but for speed, efficiency and flexibility it doesn’t come close to matching Outlook, Thunderbird or any number of other desktop email clients. It’s here that most webmail providers hit a problem. If you use a webmail account with an email client, or want to access your email from multiple locations or multiple devices, it becomes very difficult to keep your folders synchronised. The reason for this problem is the Post Office Protocol (POP or POP3) which underpins most consumer and many business email infrastructures.

If you’ve ever been approached by a friend or relative in the middle of an email ‘disaster’ – then POP was almost certainly the culprit. Getting a new laptop and can’t work out to how to ‘move’ all the old mail on to the new machine? Blame POP. Been accessing email from several different places, and now ‘sent’ mails are missing from the ‘Sent’ folder? Blame POP. Your laptop has been stolen – and now years of archived emails are gone forever? Blame POP (and the thieves…) Emails that are available on one PC can’t be seen on another PC? Blame POP.

It’s astonishing how many email problems are caused by POP. When POP was originally specified, users were expected to access their emails via dial-up, from one place and one device. The concept of always-on internet and multiple login locations was never envisioned. Neither was the idea that we might be strolling around accessing our email from mobile devices.

POP works by storing all incoming emails on a server until the user logs in to read them with an email client. At that point, the emails are moved from the server on to the user’s PC. Outgoing emails are stored in a ‘Sent’ folder in the client on the user’s PC.

The problem comes when the user wants to access emails from a second PC (or any other device). All the old emails have been stored locally on the first PC – but not kept on the server, which means that they’re unavailable on the second device. Worse still, email received or sent on the second PC will be stored locally i.e. on the second PC (so it won’t be available on the first PC). Over time, all the PCs and devices become out-of-sync. Even if the user stick to using only one PC, everything will still be lost if the PC breaks or is stolen.

To get around these problems, a number of updates and ‘fixes’ (which would be better described as kludges) have been applied to POP. A ‘Leave on server’ option copies incoming emails onto the local drive rather than moving them. However, for reasons too lengthy to explain here, the ‘Leave on server’ kludge is not totally reliable, and forces the server and client to do a lot of unnecessary work in the background.

POP suffers a myriad of other inefficiencies and problems. For example…

• The email client is forced to download whole emails (including attachments) before the user can read them. It is not possible for the user to simply view the header, and then decide whether to download the rest of the message.

• It is not possible to reliably log into a single account from multiple devices simultaneously (rendering it impossible, for example, to retrieve email on a mobile handset while still logged in on a PC)

• It is difficult to manage folders (e.g. renaming, creating and deleting) on the server by using the client.

Fortunately, a solution to all these problems has been around for years. It’s an alternative to POP, called IMAP, and avoids all the issues which plague POP.

• Emails are never moved or copied on to the local PC. Instead, they stay on the server at all times. When an email is sent, a copy is stored in the ‘Sent’ folder on the server. When an email is received, it is viewed on the local device, but remains on the server.

• When the user logs in, only the email headers are downloaded. The body of each email (along with any attachments) is only opened when the user opens the email. No time is wasted while waiting for unwanted emails or spam to download.

• A single email account can be accessed from many locations and devices – even simultaneously – without any folders becoming out-of-sync.

• Folders can be managed directly from the client

• IMAP supports ‘push’ functionality so users are immediately notified when a new email arrives. There is no need for the client to constantly ‘poll’ the server.

This might seem too-good-to-be-true. Indeed, the most significant downside to IMAP is that it requires a degree of server speed and reliability (on the part of the email service provider) that wouldn’t be necessary for a POP account. Consequently, most consumer IMAP email providers are not free. I’ve used Fastmail for several years and am happy to pay $40 a year for their premium service. For alternatives, there’s a hugely detailed list of IMAP providers here.

A little Googling will reveal that IMAP has many devotees. Those who have discovered it, like to shout about it. It’s not surprising that IMAP access is the single most requested GMail feature. And there are plenty of articles (e.g. here, here and here) extolling its virtues. Even Microsoft Exchange Server supports it. Inexplicably, IMAP is still email’s best kept secret. Hopefully it won’t be for much longer.