Aumann's agreement theorem
Aumann's agreement theorem states that Bayesian reasoners with common priors and common knowledge of each other's opinions cannot agree to disagree. Intuitively: if I'm an honest seeker of truth, and you're an honest seeker of truth, and we believe each other to be honest, then we can update on each other's opinions and quickly reach agreement. Unless you think I'm so irredeemably irrational that my opinions anticorrelate with truth, then the very fact that I believe something is Bayesian evidence that that something is true, and you should take that into account when forming your belief. Likewise, fellow rationalists should update their beliefs on your beliefs, not as a social custom or personal courtesy, but simply because your rational belief really is Bayesian evidence about the state of the world, in the same way that a photograph or a reference book is evidence about the state of the world. The fact that disagreements on questions of simple fact are so common amongst humans, and that people seem to think this is normal, is an observation that should [No safe defense|strike fear into the heart] of every aspiring rationalist.
- Agreeing to Agree by Hal Finney
- The Proper Use of Humility by Eliezer Yudkowsky
- The Coin Guessing Game by Hal Finney, a toy problem illustrating the mechanics of Aumann agreement