Analysing the analysts: Making up the numbers

The market for internet-enabled toasters will be worth $400bn worldwide by 2005, according to analysts. Well, not really. But how do these industry sages come up with such numbers? In the third and final part of his investigation, Ben King sifts through the available data and makes some predictions...

In general, analysts work on two kinds of figures - market sizing and market forecasts. A market sizing number shows how big a market was in the past quarter, or year, and a market forecast gives a suggestion of how big it might be in, say, three or five years' time.

How are the figures produced? For a market sizing number, published sales data from every internet toaster vendor and reseller are collected and added up.

To get a market forecast, other figures are worked in to a spreadsheet after vendors are asked how much they think their sales will increase. Then analysts estimate how fast a economy is growing, look at how fast similar, previous technologies took off, and voila - a couple of clicks on a spreadsheet later, out comes a billion dollar market forecast number to delight customers and impress the press.

Most of the future estimates, in fact, come from marketing departments at various companies. Of course, marketing people tend to be an optimistic breed. No marketer is going to underestimate their figures. So most of the 'raw' data that goes into spreadsheets is going to be exaggerated.

Even that is putting it lightly. Some elements will actually be made up. Pete Stauvers, former marketing operations manager at Viatel, tells it like this: "Do you tell the truth [to analysts]? Just inflating certain figures and deflating others so they tally with published figures is no whopper. We'll round things up instead of rounding things down, that kind of thing. So that instead of 75 customers, we would have 100."

The thought that marketing people are lying has obviously crossed analysts' minds. Steve Cramoysan, senior analyst at Gartner, said: "Some will try and inflate the figures. But we build up relationships with them, so with a lot of them we can look them in the eye and say: 'Are you really telling the truth?'"

Among other areas, Cramoysan tracks the market for the PBX, the nerve centre of companies' internal phone systems. "The PBX has been around for 15 years," said Cramoysan. "It would be difficult for anyone to pull the wool over our eyes in that market."

In a less established market the situation is more difficult. "Some newer markets are harder to validate, but they are usually replacements for old things," said Cramoysan. "So we ask, 'How long are the write-off periods for old stuff?' There were some wild forecasts for [the newer] IP-based PBXes, but we could see that they were impossible because people just wouldn't upgrade until their existing equipment was obsolete."

In truth, analysts have a range of ways of verifying figures. Public companies are legally obliged to publish a range of data, from quarterly reports to filings with stock exchanges. The penalties for lying on these are severe, and they can be used to verify analyst figures.

Competitors are also a useful source for reality checks, and liars are often caught out later when they publish statements that conflict with earlier ones. It's much harder to lie consistently than to tell the truth.

Yet ultimately, reality checks can never be perfect. "If a vendor wants to pull the wool over our eyes, they will succeed - for a year," admitted Gartner's Cramoysan. "But after that we will catch them out."

Does this mean the numbers are useless? Of course not, although it pays to be wary of them. The newer the market, the more unreliable the figure. But the newer markets are also the ones where information is thinnest on the ground, and analysts' predictions - if accurate - are most valuable.

The danger with these numbers is that people tend to take them as gospel truth, quoting them in marketing pitches, magazine articles and business plans without considering how they are produced. They forget to remind themselves that these analyst numbers are often little more than aggregates of marketing managers' guesses.

One former analyst with over 10 years' experience at two different analyst houses confirmed this is often the case. He said: "The marketing department will double the sales figures, and send them to the analysts. The research department will then get those figures back in analyst reports. But do they then halve them? No. They take them as gospel truth."

The result is an industry that soon starts believing its own lies and absurd bubbles of hype grow around technologies - examples are videoconferencing, WAP, and perhaps soon 3G.

"It behooves the manufacturer to validate their research before they use it," said Simon Gwatkin, VP IP platforms at Mitel Networks. "You need to use both qualitative and quantitative research, and your own business instinct, your gut feeling. If you base your business plan solely on predictions from analysts, then you're completely East Ham - two stops down the line from Barking."

No-one would argue that analysts' predictions are useless, merely that they're often as flawed as, say, journalists' predictions, which are rarely used as the basis for billion dollar business plans. Until they're recognised as such, the technology industry will persist in deluding itself, and basing its decisions on unreliable evidence.

Finally, another insider - this time head of analyst relations at a UK-based PR firm - warned: "Many market projections are subsequently proven false, and some are suspect from the start. The more reputable analysts, however, are open about their research methodology and attach an appropriate health warning. Analysts should be more transparent about how they come up with their forecasts. It's in their interests as much as anyone else's."