Paste per-model answers plus the truth label. Get the effective independent vote count n_eff. No upload, no server — this runs entirely in your browser.
n_eff = N / (1 + (N−1)·r̄), where r̄ is the mean pairwise correlation of each model's error indicators (wrong = 1). It estimates how many independent voters a panel behaves like. If every model failed on the same items, r̄ → 1 and n_eff → 1 no matter how many models you added. It is not the panel's accuracy; it is how much independent evidence the panel carries.
The trap: adding weak, near-random models can raise n_eff, because random mistakes don't agree. Report n_eff for a competent panel (drop models near chance), or n_eff will flatter the headcount.
A second caveat: two error indicators cannot correlate above a ceiling set by their error rates (r̄max = √(p(1−q)/q(1−p))), so when models differ in skill the raw r̄ understates how much they really depend on each other. The tool also reports a ceiling-corrected readout (each pair's correlation over its maximum possible). Read it as a dependence-strength number, not as a lower bound on n_eff: it answers a different counterfactual, N models at equal accuracy with that dependence. The raw n_eff stays the panel's Kish number.
Two assumption-light checks sit beside it. Plurality vs single: does the crowd's majority vote beat the median model, and does it reach the best one? Same-wrong-option: when two models both miss, how often do they pick the same wrong answer? Far above 1/(options−1) means the crowd fails together on the same attractive distractor, which is the dependence that actually costs the vote.
On HELM openbookqa (multiple_choice_joint, 500 items), 31 models from 11+ families:
r̄ = 0.251, n_eff = 3.63 of 31. Restricting to models with accuracy ≥ 0.40 (26 models):
r̄ = 0.353, n_eff = 2.64 of 26. One best model per family (11): n_eff = 2.95 of 11.
Same regime as the published literature (Nine Judges ≈ 2 of 9; Wavering Oracles 2.58 of 7).
Second panel, different task: HELM mmlu (5 subjects, multiple_choice_joint, 560 items), same 31 models. From the released letters r̄ = 0.346, n_eff = 2.72 of 31; from HELM's exact_match indicators 2.66, so the two routes agree. Here the crowd is worse than the best single model: plurality 0.670 vs best single 0.737 (median 0.579). P(two wrong models pick the same distractor) = 0.49 vs 0.33 uniform. Per-subject, near-chance abstract_algebra (mean acc 0.34) reads a flattering n_eff = 5.43 because random misses do not agree; computer_security (acc 0.69) reads 2.79. Click "Load the HELM MMLU panel" above to reproduce it in one step.
Decision check (labels from HELM's display_predictions.json, 499 items with a
gold any model got): best single 0.960, cross-lineage-11 plurality 0.943, competent-26 plurality 0.915,
median single 0.841. The 26-model crowd beats the median but not the best single, and the 11 cross-lineage
panel decides better than the 26. P(two wrong competent models pick the same distractor) = 0.657
vs 0.333 if wrong votes scattered. Fitting independent voters to the observed plurality gives ~2.9
effective votes, matching the 2.64 from correlations. Ceiling-corrected the competent panel reads
r̄ = 0.537, n_eff = 1.80, a dependence readout rather than a lower bound.