Analysing the analysts: Should we trust them?

The market for mobile ring tones and related products will be worth £3.7m by 2005, says GartnerGroup. It sounds credible enough and it's an interesting statistic - one that has even appeared in a couple of magazines. Trouble is, it isn't true, at least not according to Gartner, which doesn't even study the mobile accessories market.

No-one is quite sure where this figure came from, but such is the faith analysts inspire, it was a while before anyone exposed it as a fake. Who knows, major business decisions may even have been based on the forecast.

As technology develops, we rely on industry analysts and their predictions more. Businessmen trying to sell technology and related services use them. Politicians and civil servants do too. Journalists are the worst culprits for mindlessly regurgitating statistics.

The above figure happens to be made up, but how accurate are the real ones? Often, they're not.

Take a classic analyst miscalculation - videoconferencing. It's pretty easy (and possibly slightly unfair) to find examples that just didn't happen. Here are a few:

- In February 1997, IDC published a report forecasting about 600,000 desktop videoconferencing units to ship to business customers, and some 5.4 million to sell into the home market during 2001. Most analysts joined in the frenzy. A month later, Frost and Sullivan predicted the market would be worth an anticipated $43bn by 2003.

- Will Strauss, president of US-based market researchers Forward Concepts, pushed the boat out furthest. He told Newsbytes in March 1997: "Within three years videoconferencing will be standardised on all PC operating systems including Windows and Mac."

We couldn't get up to date figures. IDC hasn't published anything on videoconferencing for a while. Let's just say nothing like 5.4 million domestic terminals have ever been sold.

Various other technologies, like ISDN and ATM, have also underperformed expectations. Every analyst has their favourite prediction that went wrong - usually something strange, like integrated voice-data terminals or mobile base stations for the home.

For various reasons, they tend to over-hype flops more often than they dismiss things that turn out to be huge. A rare exception is the exponential growth of SMS text messaging. It was largely unforeseen two or three years ago, but a staggering 15 billion messages were sent globally in December 2000, analysts say. (Actually it was the GSM Association.)

There are more examples, but they're slightly harder to track down. Analysts like to shift categories to make old (wrong) predictions seem closer to the mark.

How do they come up with their projections? It's a sensitive issue, and quite a lot of analyst houses are reluctant to talk about it on the record, either in detail or at all.

When researchers come out with predictions for most markets they rely on a variety of factors. They'll look at current market size (if there is one). They'll look at how fast a market is expected to grow. Take something like suntan lotion - they'll look at when people use suntan lotion, how fast the holiday market will grow, how many people are potential suntan oil users, how many people will have disposable income to spend on suntan lotion, whether there are competing products.

All this is fed into massive spreadsheets. Out the end spurts a figure.

Mary-Ann O'Loughlin is chief analyst at Ovum, one of the few analyst houses prepared to discuss the business with us and how accurate predictions are. She didn't want to put a figure on forecast accuracy. "It's our best estimate," she said. "No forecast is infallible."

And apparently it's not really a numbers game, anyway.

"Ninety per cent of the value is in the qualitative part of our reports," she added. "It's more in the scenario than the numbers that come out at the end."

IDC also fought shy of making specific claims for its data accuracy. "As a services company we are driven by the capabilities and characters of our analysts as much as a rigorous methodology when commenting on the market," said IDC's Martin Hingley, VP EMEA Systems Group, in a written statement.

All the analysts we spoke to emphasise they work closely with the industries they comment on and have a position of trust allowing them to get 'inside information' from the major companies in sectors they cover.

So why do they sometimes get it spectacularly wrong? Besides drawing on their own expertise, they are known to follow what other analysts' publish and say. They sometimes even credit each other as sources, typically for figures. This means mistakes tend to be across the board, even though some analysts do go out on a limb sometimes.

A possible hole in their information - which might explain occasional misjudgements - is a lack of accurate information about what customers will actually use. Most examples of great analytical pratfalls concern consumer products which consumers just didn't bother with.

In the B2B arena analysts tend to have a much more accurate grasp of what's going on because they spend so much time talking to businesses. Indeed, through their consulting work they help to guide - to an extent - what businesses actually buy. They're the ones, primarily, who dream up the three letter acronyms - ERP, CRM, even TLA - which keep the IT industry busy. Directly or indirectly, they help to tell the vendors what to produce and the buyers what to buy.

"When we get it wrong, it's because we don't understand what people really want," Ovum's O'Loughlin admitted. "That's the hardest thing to discover."

There are ways to avoid this - market research, focus groups and so on - but they can be blunt tools. At best, they give a rough indication, and can seem unscientific.

"There's a lot of common sense, and a little bit of intuition [in our predictions]," added O'Loughlin. "My teenage daughters are an invaluable resource."

It seems business intelligence comes from many sources, and like anything, we develop trusted suppliers.