Scott Alexander assigns a 70% chance of AIs becoming "ultraforecasters" — models that exceed the accuracy of human superforecasters by between four and 12 percentage points.
His lower bound is based on his assumption that AIs will improve on human superforecasters by as much as prediction markets once improved on statistical models (which was four percentage points).
Four percentage points may not sound like a lot, but it is. Alexander says it's equal to half of the benefit of just knowing what you're betting on.
"The effect of going from a statistical model to a prediction market," he explains, "is half as large as the effect of knowing which two teams were playing and how good they are!"
He argues that this is the minimum improvement we can expect from AI because sporting events are optimized to be unpredictable (through rules, salary caps, drafts) and sports betting is therefore the most difficult area to improve on human forecasting.
The upper bound of his range is based on chess, because chess may be the domain where AIs have the greatest advantage over humans. AI chess engines are now so far ahead of humans that they can beat a grandmaster despite starting three pawns down.
Alexander then calculates that if an AI forecaster pulled ahead of human superforecasters by a comparable margin, it would be the equivalent of adding 12 percentage points of accuracy to their predictions.
In practical terms, this means that Alexander expects that what looks like a 50% probability on prediction markets now will be something between a 54% and 62% probability in the near future.
"All of this keeps me excited about AI superforecasters," he concludes, "even though I don't expect miracles."