Getting better at predicting the future

Phillip Tetlock started to explore how to get better at predicting the future in 1987. It was then that he started to collect forecasts from about 300 experts – initially about preventing a nuclear war but then extending to encompass about 27,500 much wider political and geo-political events. The results were published in his book “Expert Political Judgement” in 2005.

Tetlock first discusses arguments about whether the world is too complex for people to find the tools to understand political phenomena, let alone predict the future. He evaluates predictions from experts in different fields, comparing them to predictions by well-informed laity or those based on simple extrapolation from current trends. He goes on to analyze which styles of thinking are more successful in forecasting. Classifying thinking styles using Isaiah Berlin’s prototypes of the fox and the hedgehog, Tetlock contends that the fox–the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events–is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems. A famous diagram from this is on the lines below:

Here, calibration is the number of right predictions, and discrimination is the range of the predictions. So it is possible to have stellar discrimination and terrible calibration scores if you make bold and wrong predictions. As well as the lack of success of all forecasters compared with models, he also noted a perversely inverse relationship between the best scientific indicators of good judgement and the qualities that the media most prizes in pundits–the single-minded determination required to prevail in ideological combat.

Rather than decide that forecasting was too difficult for mere mortals, Tetlock started the Good Judgement Project in 2011. He has 20,000 volunteers who participate in an annual tournament, giving judgement on geopolitical issues and updating as and when appropriate. The early years of the tournament are already yielding exciting results.

For instance, – even brief training works – a 20 minute course on how to put a probability on a forecast, correcting for well-known biases, provides lasting improvements to performance.

A second insight is that teamwork helps – teams of forecasters who discussed and argued produced better predictions.

He has produced advice for forecasters summarised as CHAMP

• Comparisons are important
• Historical trends can help
• Average opinions
• Mathematical models should be taken into account
• Predictable biases exist and should be allowed for.

Our Cobwebs events held through 2012 to 2014 were focused particularly on cognitive biases – spotting them and overcoming them.

Written by Gill Ringland.

If you enjoyed this blog, sign up for our monthly newsletter at eSAMIsignup@samiconsulting.co.uk and/or browse our web site www.samiconsulting.co.uk

Advertisement