Can crowds of people trading on what they believe will happen collectively predict the future? A new Tel Aviv University study of more than one million forecasts suggests that they often come remarkably close. But when it comes to certain events, from international conflicts to elections, the odds may tell a different story.
A research team from Tel Aviv University's School of Psychological Sciences, led by Prof. Michael Gilead and including Justin Dickerson, Hilla Shinitzky, Maayan Peer, and Mattan S. Ben-Shachar, analyzed 1,014,122 forecasts across four major prediction platforms: Polymarket, Kalshi, Manifold, and Metaculus. Covering 12 domains over a decade, the study, published in the scientific journal Proceedings of the National Academy of Sciences (PNAS), examines how well collective predictions perform across topics ranging from elections and geopolitics to technology and security.
- Crowds can estimate probabilities surprisingly well. Across the full dataset, predicted probabilities closely matched real-world outcomes. Events given a 70% chance of happening occurred approximately 70% of the time.
- Some events are consistently overestimated. Forecasts involving military conflict, international relations, and technology tended to assign higher probabilities to events than the outcomes justified.
- Election forecasts show a different pattern. Prediction markets tended to underestimate the likelihood of election-related events across certain probability ranges, particularly when predicted chances were low or moderate.
- Predictions become more accurate as events approach. However, this improvement mainly reflects better identification of which events will happen, rather than better calibration of probabilities. Forecasts were already well calibrated after just 25% of their allotted forecasting period.
Prediction markets are increasingly used to estimate the likelihood of political, economic, and technological events. Rather than examining a handful of high-profile predictions, the researchers looked at more than a million forecasts to understand how these systems perform on a much larger scale.
Their findings reveal an important distinction: while prediction markets are generally good at estimating probabilities, their reliability varies depending on the type of event being predicted.
The findings do not suggest that prediction markets can predict the future with certainty. Instead, they show that collective forecasts can provide valuable information about what might happen, provided we understand their limitations.
By identifying the areas in which prediction markets systematically overestimate or underestimate the likelihood of events, the study offers a clearer picture of how to interpret these forecasts, especially when the outcomes could have significant political, economic, or technological consequences.
