How the ranking works

Nobody votes for an idea. People compare pairs — and many small comparisons add up to a ranking.

Three rules

1. Always one pair. You get two ideas and pick the one that serves the topic better. You never judge your own ideas and never see the same pair twice. The pairs least is known about come first — spread across members: nobody is asked about the same idea over and over while others have not seen it.
2. Every idea has a score. It starts at 1500. A win raises it, a loss lowers it — by as much as the result was surprising. Beating a strong idea pays a lot; losing to one costs little. A new idea moves fast, a well-judged one only slowly. One member's duels on the same idea count fully twice, half the third time and not at all after that, so no single person can carry an idea.
3. A place is earned. An idea gets a position only after at least three judgments by three different members (in a small topic, as many as one person can give at all). Until then it is listed apart, with no number — neither high nor low. Placed ideas are ordered by their score minus twice what is still uncertain about it, so an idea judged by few members sits where the evidence can vouch for it and rises as more members judge it.

Late ideas

An idea added later is not at a disadvantage: pairs are chosen so that the ideas with the most uncertainty go first, so everyone who comes back gets the new idea in their next duels. Until it has enough judgments it has no place — it is not quietly slotted into the middle.

No single member decides

A ranking should reflect the group, not whoever judges most. Three safeguards make sure of that:

• Spread. Ideas you have not been asked about yet come first, so three pairs show you six different ideas while there are enough; an idea you have seen returns only once the others have had their turn. A new idea thus meets many different members before it meets any of them twice.
• Different members. The three judgments an idea needs for a place must come from three different members. Three duels from one person are not a ranking.
• Diminishing weight. Your first two duels on the same idea count fully, the third half, any further ones not at all. They are still recorded, so you never see a pair twice — they just stop moving the score.

The scores are never stored as the truth: every duel is, and the scores are refitted from all duels after every judgment. The order in which people judged therefore plays no part, and a rule change is applied to the past the same way.

The formula

chance that A beats B = 1 / (1 + 10^((score B − score A) / 400))

The scores are the set of numbers that gives the highest probability
to all of a topic's duels together — refitted after every judgment,
every score pulled gently toward 1500, a member's duels on the same
idea weighted 1, 1, ½, 0.

place = score − 2 × uncertainty   (what the duels so far cannot vouch for)

A gap of 100 points means about 64 % to 36 %. Skipping a pair changes nothing. Numbers, not names: how anyone judged is not stored against the person and is shown to nobody.

An example

Three ideas, six duels: A beat B twice, B beat C twice, and A and C beat each other once. The fit settles on A 1572, B 1500, C 1428 — the numbers under which these six outcomes are the most likely, and no other set does better. Had C beaten A both times, the three would form a circle, and the fit would put all three level at 1500: it rewards nobody for a win that the other duels contradict. The order in which the six were judged plays no part.

Where it comes from

Pairwise comparison is an old, well-studied way of measuring judgment: Thurstone's law of comparative judgment (1927), Zermelo's model (1929) and the Bradley–Terry model (1952) describe how to estimate each item's strength from paired contests. Elo (1978) showed how to update that estimate match by match, and Glickman (1995) added the uncertainty that governs how fast a score moves. For public ideas the same approach runs "wiki surveys" (Salganik and Levy, 2015), used among others in New York City's planning. inocracy fits the Bradley–Terry model to all of a topic's duels at once, as wiki surveys do, with a Glicko-style uncertainty steering which pairs are asked next — exactly the formula displayed above.