Robin on Tech

A geeky prospect it may be, but it’s often fun to imagine the product development meetings which occurred to produce some of the services and solutions we’re most reliant on. Not necessarily the most common solution. Rather, the products which, through sheer innovation, defined, or redefined a whole product area.

For many technology companies, ‘innovation’ is just a throwaway buzzword to be inserted with requisite regularity into puff marketing brochures and website flash animations. But occasionally, new products tip up, fully deserving of the ‘innovative’ moniker, and it is these products whose genesis is worth considering.

For example, consider the questions Google engineers might have asked themselves when they started specifying the requirements for a Google email product.

• What do we like about current email implementations?

• What do we dislike?

• What would do we want to do but are prevented from doing? How do current systems limit us?

To an extent, Gmail redefined email by adding pseudo-limitless storage, and doing away with folders. Most other email providers were forced to increase their storage quotas, and introduce email ‘tagging’. But the Google engineers omitted a key ingredient: email’s best kept secret.

Gmail’s web interface is one of the best around, but for speed, efficiency and flexibility it doesn’t come close to matching Outlook, Thunderbird or any number of other desktop email clients. It’s here that most webmail providers hit a problem. If you use a webmail account with an email client, or want to access your email from multiple locations or multiple devices, it becomes very difficult to keep your folders synchronised. The reason for this problem is the Post Office Protocol (POP or POP3) which underpins most consumer and many business email infrastructures.

If you’ve ever been approached by a friend or relative in the middle of an email ‘disaster’ – then POP was almost certainly the culprit. Getting a new laptop and can’t work out to how to ‘move’ all the old mail on to the new machine? Blame POP. Been accessing email from several different places, and now ‘sent’ mails are missing from the ‘Sent’ folder? Blame POP. Your laptop has been stolen – and now years of archived emails are gone forever? Blame POP (and the thieves…) Emails that are available on one PC can’t be seen on another PC? Blame POP.

It’s astonishing how many email problems are caused by POP. When POP was originally specified, users were expected to access their emails via dial-up, from one place and one device. The concept of always-on internet and multiple login locations was never envisioned. Neither was the idea that we might be strolling around accessing our email from mobile devices.

POP works by storing all incoming emails on a server until the user logs in to read them with an email client. At that point, the emails are moved from the server on to the user’s PC. Outgoing emails are stored in a ‘Sent’ folder in the client on the user’s PC.

The problem comes when the user wants to access emails from a second PC (or any other device). All the old emails have been stored locally on the first PC – but not kept on the server, which means that they’re unavailable on the second device. Worse still, email received or sent on the second PC will be stored locally i.e. on the second PC (so it won’t be available on the first PC). Over time, all the PCs and devices become out-of-sync. Even if the user stick to using only one PC, everything will still be lost if the PC breaks or is stolen.

To get around these problems, a number of updates and ‘fixes’ (which would be better described as kludges) have been applied to POP. A ‘Leave on server’ option copies incoming emails onto the local drive rather than moving them. However, for reasons too lengthy to explain here, the ‘Leave on server’ kludge is not totally reliable, and forces the server and client to do a lot of unnecessary work in the background.

POP suffers a myriad of other inefficiencies and problems. For example…

• The email client is forced to download whole emails (including attachments) before the user can read them. It is not possible for the user to simply view the header, and then decide whether to download the rest of the message.

• It is not possible to reliably log into a single account from multiple devices simultaneously (rendering it impossible, for example, to retrieve email on a mobile handset while still logged in on a PC)

• It is difficult to manage folders (e.g. renaming, creating and deleting) on the server by using the client.

Fortunately, a solution to all these problems has been around for years. It’s an alternative to POP, called IMAP, and avoids all the issues which plague POP.

• Emails are never moved or copied on to the local PC. Instead, they stay on the server at all times. When an email is sent, a copy is stored in the ‘Sent’ folder on the server. When an email is received, it is viewed on the local device, but remains on the server.

• When the user logs in, only the email headers are downloaded. The body of each email (along with any attachments) is only opened when the user opens the email. No time is wasted while waiting for unwanted emails or spam to download.

• A single email account can be accessed from many locations and devices – even simultaneously – without any folders becoming out-of-sync.

• Folders can be managed directly from the client

• IMAP supports ‘push’ functionality so users are immediately notified when a new email arrives. There is no need for the client to constantly ‘poll’ the server.

This might seem too-good-to-be-true. Indeed, the most significant downside to IMAP is that it requires a degree of server speed and reliability (on the part of the email service provider) that wouldn’t be necessary for a POP account. Consequently, most consumer IMAP email providers are not free. I’ve used Fastmail for several years and am happy to pay $40 a year for their premium service. For alternatives, there’s a hugely detailed list of IMAP providers here.

A little Googling will reveal that IMAP has many devotees. Those who have discovered it, like to shout about it. It’s not surprising that IMAP access is the single most requested GMail feature. And there are plenty of articles (e.g. here, here and here) extolling its virtues. Even Microsoft Exchange Server supports it. Inexplicably, IMAP is still email’s best kept secret. Hopefully it won’t be for much longer.

In a previous post, I wrote that Apple’s iPhone is significant because it is the first truly data-centric phone to be aimed at consumers rather than business. The crucial thing about Apple’s strategy is their perception of the network operator – Cingular – as nothing more than a ‘pipe’ for voice and data. An interesting article in the Wall Street Journal has thrown some more light on this idea…

“Service providers usually hold enormous sway over how phones are developed and marketed — controlling every detail from processing power to the various features that come with the phone. Not so with Apple and Cingular… Cingular agreed to leave its brand off the body of the phone. Upsetting some Cingular insiders, it also abandoned its usual insistence that phone makers carry its software for Web surfing, ringtones and other services…”

“Mr. Jobs once referred to telecom operators as “orifices” that other companies, including phone makers, must go through to reach consumers. While meeting with Cingular and other wireless operators he often reminded them of his view, dismissing them as commodities and telling them that they would never understand the Web and entertainment industry the way Apple did”

The crucial point is that Apple knows that the networks don’t ‘get’ consumer data. Intead of seeing data as a valuable commodity – to be sold in great volume to as many customers as possible – the operators have tried to ration it, charge ludicrous prices and endlessly attempted to bundle their own content. Many consumers have been put off from using significant quantities of mobile data because of the operators’ interminable bungling:

• It has been much too expensive. For example, Orange UK’s ‘cheapest’ mobile data bundle is still £4 per month for 4mb…

• The tariffs have been opaque. Consumers have been unsure what they’re going to be billed for, and how much it would cost them. And because they perceive mobile data as expensive to begin with, they’ve been deterred from using it at all.

• The operators have created ‘walled gardens‘ and/or attempted to funnel traffic through their own WAP sites (e.g. Orange World). Most consumers aren’t savvy enough (or just can’t be bothered) to modify their mobile browser homepage. So for many users, an unedifying mobile data experience has begun and ended in the walled gardens of their own networks.

• The operators have concentrated on trying to sell their own content (e.g. ringtones and wallpapers) instead of promoting wider access to the web and email.

• The operators have crippled or impaired the data-capabilities of many handsets. For example, Orange completely disabled the Nokia Standby Screen (which provides convenient access to email) on many of it’s phones, and replaced it with the Orange Homescreen. Although the Orange Homscreen can now be switched off on newer handsets, the setting is buried in the configuration menu, so many users never discover it.

Until recently, the effect of these policies has been to drive customers away from using significant quantities of mobile data. The result of this self-inflicted failure to capitalise on their expensive 3G spectrum is that many operators have been forced to write down the value of those licenses.

Fortunately, things are starting to change. T-Mobile have led the way with their ‘Web ‘n’ Walk’ data bundles. Three have introduced the ‘X-Series’ mobile data package. And even Orange provides an (unpromoted) 1gb data bundle for £8 per month, providing you spend enough on voicecalls.

Jobs’ reference to operators as “orifices” may have been gratuitous, but it was not inaccurate. iPhone owners are going to be using far greater quantities of data than most owners of other consumer handsets. This might be partially attributable to the fact that the type of people who will own iPhones are the type of people who would use more data anyway. But, as I explained in a previous post the real reason is that Apple have created a phone which really makes mobile data a compelling proposition. Coupling easy-to-access, useful data-services with a cheap data tariff will persuade customers to use – and spend more on – data. The lower margins on mobile data will be vastly outweighed by the huge jump in volume.

Recent leaks suggests that Apple may not be the only new entrant to the handset manufacturing business. Google is apparently working on an interesting ‘Google Phone’ – which will similarly use the network operators as nothing more than “dumb pipes“.

At last, mobile data is becoming the commodity it always should have been, with the mobile operators acting only as pipes. The deal with Apple indicates Cingular’s tacit acceptance of its role as an “orifice” (or at least a ‘dumb pipe’!), not as a content provider. Few customers view their broadband ISP as anything other than a ‘pipe’ with which to access the internet. It’s about time that more mobile network operators allowed their subscribers to take the same view.