Brampton Factor: Man and machine

Martin Brampton discusses the complex relationship between humans and computers - and how to create reliable systems when we can't put complete faith in either.

People love automating things and there are some obvious benefits to be gained. One example is Network Rail implementing remote monitoring of trackside equipment. But systems of this kind have to overcome deep-rooted problems. They relate not only to technology but to the complex behaviour of human beings.

At one time, I worked on a project that provided real time monitoring for an overland pipeline carrying natural gas liquids. It taught me the significance and complexity of the failure modes of systems involving computers and communications. Designing systems that run well when everything is working is one thing. In real life, skilled designers build systems that cater for things going wrong, as they inevitably will.

One of the problems with my project was that the outstations, based on simple microprocessors, would lose track of the SDLC protocol in use and just go to sleep. The monitoring and valve control were then defunct until someone made the awkward road journey to visit the site. Our solution at that time was to use a simpler protocol where the outstation continued communicating regardless of what it received from the central site. The result was more robust and also could be checked with the use of very simple test equipment.

Those are technical issues that make the problems more difficult than they appear at first sight. Other issues arise from human behaviour. One example of this was dramatically illustrated at the Buncefield oil depot recently. The enquiry found that automatic systems to control the filling of a tank were not working but with such systems in place, people assumed that an overflow could not happen. Complete reliance on technology can clearly lead to disaster.

Leakage of radioactive material at Sellafield tells another story. First, the pipe that sprang a leak was poorly designed, so we must conclude that even in a critical safety environment, design does not always live up to the highest ideals. Then the failure to notice the leak for anything up to nine months was blamed on staff 'complacency'. And 'indications' were ignored. Again, the existence of sophisticated monitoring systems may well have made the problem worse rather than better.

An interesting feature of the Sellafield incident was that once suspicions were aroused, the solution was to use a camera so that people could look at what was happening. Sometimes there is no substitute for having a person take a look. The trouble with taking that argument too far is that people are not altogether reliable when it comes to taking a look.

Driving past a red signal was a factor in one of the major train crashes of recent years. This highlights two widely held beliefs that fail to stand up to scrutiny. One is that people can easily observe what is happening around them. This belief is presumably held by those who continue to use mobile phones while driving, even after tests on simulators have shown that such drivers perform no better than drunks.

The other belief is the idea that people can simply be blamed for making mistakes in such situations. While there is no getting away from the responsibility carried by those who do jobs such as train driving, we know that it is an unavoidable human characteristic to make mistakes. Anyone who believes that they do not make them is deluding themselves. Making people perfect is not a realistic solution, so creating situations in which they can perform better is far more relevant.

The core conclusion from this is that automated systems can have great value but do not solve every problem. Human judgement and its frailties are still the critical factor for safety. This is something that does not sit well with contemporary obsessions. In the world of work, there is an ongoing tendency to dismiss skill and experience. Much of the talk about 'modernisation' is really about shifting control away from individuals who may rely on skill and experience. Instead managers and systems are supposed to take over.

Whether this is wise is certainly questionable. Network Rail found itself in a position where an expert commented that "the condition of railway infrastructure was not well known when Network Rail took over from Railtrack". This in turn stemmed from drastic reductions made by Railtrack in its complement of experienced engineering staff. The result was a short term boost to profits and a long term decline in the quality of the rail network.

Much the same principle is applied to banking with the creation of call centres, often in remote parts of the world. The assumption is that the whole process of interacting with customers can be handled by a standard set of mechanisms. Recent surveys showing widespread dissatisfaction with this kind of treatment indicates that the customers do not agree.

In fact, there are some signs that people intuitively understand the value of skill and experience. The recent phenomenon of public fascination with chefs is an example. It seems clear to everyone that being a good chef is not a matter of following some standard procedure that could, in principle, be automated. On the contrary, successful chefs rely on judgement and experience.

A more controversial example is education, where there are certainly doubts about the belief that teaching must be managed by crude measurement systems at every turn. The reality is surely that the quality of teachers and heads is the critical factor in the success or otherwise of schools. No amount of management grafted on to that can really make a fundamental difference.

So we find ourselves with an obvious but often ignored truth. Computers and other automated systems are quite different from people but people are not dispensable. The machines are only manifestations of human intentions and have no life of their own. We go wrong when we try to put our faith totally in machines or people behaving like machines. And we miss opportunities if we fail to take advantage of what machines can do. The best results come from a constructive and imaginative integration of human beings and automatic systems.