What holds up, what's soft

A sample. One claim, traced back to where it actually comes from.

I write research briefs: you name a question, I go to the sources, and you get back the parts that survive contact with evidence, the parts that don't, and the links so you can check me.

This page is a small one, done for free. It's the thing I'd rather be judged on than a description of it.

The claim

"Superforecasters are 30% more accurate than intelligence analysts with access to classified information."

You've seen a version of this. It travels well because it sounds like a receipt. It traces back to the Good Judgment Project, the IARPA forecasting tournament won by Philip Tetlock and Barbara Mellers.

What holds up

Good Judgment did win, and by a lot: 35% to 72% better than the competing research teams. Their plain median forecast was that accurate as well, so the result isn't a trick of how they aggregated.

The peer-reviewed backbone is real, not just a website. Mellers et al. in Psychological Science (2014) and Atanasov et al. in OBHDP (2020), the latter on why small, frequent belief updates beat big dramatic revisions.

What doesn't

The 30% number lives on the seller's own site. It isn't in the papers. The papers give the 35–72% range against research teams, which is a different claim against a different opponent.

The lead-time figure is told two ways in the same line of work: superforecasters 400 days out matching regular forecasters at 150 days, or 300 against 100. Same analysis, by Mark Steyvers, two different arithmetic. One of those is wrong, and nobody repeating it has stopped to find out which.

The honest version

People who keep score and correct in small steps beat people who are confident and hold no receipts. That version is smaller than the 30% headline and it's the one that survives.