Difference between revisions of "Priors"

From Lesswrongwiki
Jump to: navigation, search
 
(20 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{wikilink|Prior probability}}
 
{{wikilink|Prior probability}}
A [[Bayesian]] uses [[Bayes's Theorem]] to [[belief update | update beliefs]] based on the [[evidence]].  This requires that, even in advance of seeing the evidence, you have beliefs about what the evidence means - how likely you are to see the evidence, ''if'' various hypotheses are true - and how likely those hypotheses were, in ''advance'' of seeing the evidence. To calculate a [[posterior probability]] using [[Bayes's Theorem]], you need a [[prior probability]] and [[likelihood distribution]].
+
In the context of [[Bayes's Theorem]], '''Priors''' refer generically to the beliefs an agent holds regarding a fact, hypothesis or consequence, before being presented with evidence. Upon being presented with new evidence, the agent can multiply their prior with a [[likelihood distribution]] to calculate a new (posterior) probability for their belief.
  
Suppose you had a barrel containing some number of red and white balls.  If you start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability between 0 and 1, and you start out ignorant of this fixed probability (the parameter could anywhere between 0 and 1), then each red ball you see makes it ''more'' likely that the next ball will be red. (By [[Laplace's Rule of Succession]].)
+
==Examples==
 +
Suppose you had a barrel containing some number of red and white balls.  You start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability. Furthermore, you start out ignorant of this fixed probability (the parameter could be anywhere between 0 and 1). Each red ball you see then makes it ''more'' likely that the next ball will be red, following a [http://en.wikipedia.org/wiki/Rule_of_succession Laplacian Rule of Succession]. For example, seeing 6 red balls out of 10 suggests that the initial probability used for assigning the balls a red color was .6, and that there's also a probability of .6 for the next ball being red.
  
 
On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it ''less'' likely that the next ball will be red (because there are fewer red balls remaining).
 
On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it ''less'' likely that the next ball will be red (because there are fewer red balls remaining).
  
Thus our prior can affect how we interpret the evidence.  The first prior is an [[inductive prior]]; things that happened before are predicted to happen again with greater probability.  The second prior is anti-inductive; the more red balls we see, the fewer we expect to see in the future.
+
Thus our prior affects how we interpret the evidence.  The first prior is an inductive prior - things that happened before are predicted to happen again with greater probability.  The second prior is anti-inductive - the more red balls we see, the fewer we expect to see in the future.
  
In both cases, you started out believing something about the barrel - presumably because someone else told you, or because you saw it with your own eyes. But then their words, or even your own eyesight, was evidence, and you must have had prior beliefs about probabilities and [[likelihoods]] in order to interpret the evidence. So it seems that an ideal Bayesian would need some sort of inductive prior at the very moment they were born; and where an ideal Bayesian would get this prior, has occasionally been a matter of considerable controversy in the philosophy of probability.
+
As a real life example, consider two leaders from different political parties. Each one has his own beliefs - priors - about social organization and the roles of people and government in society. These differences in priors can be attributed to a wide range of factors, ranging from their educational backgrounds to hereditary differences in personality. However, neither can show that his beliefs are better than those of the other, unless he can show that his priors were generated by sources which track reality better<ref>Robin Hanson (2006). "Uncommon Priors Require Origin Disputes". Theory and Decision 61 (4) 319–328. http://hanson.gmu.edu/prior.pdf</ref>.
  
==Prior probability==
+
Because carrying out any reasoning at all seems to require a prior of some kind, ideal Bayesians would need some sort of priors from the moment that they were born. The question of where an ideal Bayesian would get this prior from has occasionally been a matter of considerable controversy in the philosophy of probability.
This phrase usually refers to a point estimate already based on considerable evidence - for example when we estimate the number of women who start out with breast cancer at age 40, in advance of performing any mammographies.
 
  
The probability that you start with before seeing the evidence.  One of the inputs into [[Bayes's Theorem]].
+
==Updating prior probabilities==
 +
In informal discussion, people often talk about "updating" their priors. This is technically incorrect, as one does not change their prior probability, but rather uses it to calculate a posterior probability. However, as this posterior probability then becomes the prior probability for the next inference, talking about "updating one's priors" is often a convenient shorthand.
  
Suppose there are a hundred boxes, one of which contains a diamond; and this is ''all'' you know about the boxes.  Then your prior probability that a box contains a diamond is 1%, or prior odds of 1:99.
+
==References==
 
 
Later you may run a diamond-detector over a box, which is 88% likely to beep when a box contains a diamond, and 8% likely to beep (false positive) when a box doesn't contain a diamond.  If the detector beeps, then this is [[evidence]] with a [[likelihood ratio]] of 11:1 in favor of a diamond, which sends the prior odds of 1:99 to [[posterior odds]] of 11:99 = 1:9.  But if someone asks you "What was your prior probability?" you would still say "My prior probability was 1%, but I saw evidence which raised the posterior probability to 10%."
 
 
 
Your "prior probability" in this case was actually based on a certain amount of information - i.e., someone ''told'' you that one out of a hundred boxes contained a diamond.  Indeed, someone told you how the detector worked - what sort of evidence a beep represented.  For a more complicated notion of prior beliefs, including prior beliefs about the meaning of observations, see "[[priors]]".  ("Prior probability" is more likely to refer to a single summary judgment of some variable's prior probability, versus a Bayesian's general "[[priors]]".)
 
  
 +
<references />
  
 
==Blog posts==
 
==Blog posts==
Line 34: Line 32:
 
*[[Inductive bias]]
 
*[[Inductive bias]]
 
*[[Belief update]]
 
*[[Belief update]]
*[[Prior probability]]
 
  
 
[[Category:Jargon]]
 
[[Category:Jargon]]
 
[[Category:Concepts]]
 
[[Category:Concepts]]
 
[[Category:Bayesian]]
 
[[Category:Bayesian]]

Latest revision as of 16:54, 27 October 2012

Smallwikipedialogo.png
Wikipedia has an article about

In the context of Bayes's Theorem, Priors refer generically to the beliefs an agent holds regarding a fact, hypothesis or consequence, before being presented with evidence. Upon being presented with new evidence, the agent can multiply their prior with a likelihood distribution to calculate a new (posterior) probability for their belief.

Examples

Suppose you had a barrel containing some number of red and white balls. You start with the belief that each ball was independently assigned red color (vs. white color) at some fixed probability. Furthermore, you start out ignorant of this fixed probability (the parameter could be anywhere between 0 and 1). Each red ball you see then makes it more likely that the next ball will be red, following a Laplacian Rule of Succession. For example, seeing 6 red balls out of 10 suggests that the initial probability used for assigning the balls a red color was .6, and that there's also a probability of .6 for the next ball being red.

On the other hand, if you start out with the prior belief that the barrel contains exactly 10 red balls and 10 white balls, then each red ball you see makes it less likely that the next ball will be red (because there are fewer red balls remaining).

Thus our prior affects how we interpret the evidence. The first prior is an inductive prior - things that happened before are predicted to happen again with greater probability. The second prior is anti-inductive - the more red balls we see, the fewer we expect to see in the future.

As a real life example, consider two leaders from different political parties. Each one has his own beliefs - priors - about social organization and the roles of people and government in society. These differences in priors can be attributed to a wide range of factors, ranging from their educational backgrounds to hereditary differences in personality. However, neither can show that his beliefs are better than those of the other, unless he can show that his priors were generated by sources which track reality better[1].

Because carrying out any reasoning at all seems to require a prior of some kind, ideal Bayesians would need some sort of priors from the moment that they were born. The question of where an ideal Bayesian would get this prior from has occasionally been a matter of considerable controversy in the philosophy of probability.

Updating prior probabilities

In informal discussion, people often talk about "updating" their priors. This is technically incorrect, as one does not change their prior probability, but rather uses it to calculate a posterior probability. However, as this posterior probability then becomes the prior probability for the next inference, talking about "updating one's priors" is often a convenient shorthand.

References

  1. Robin Hanson (2006). "Uncommon Priors Require Origin Disputes". Theory and Decision 61 (4) 319–328. http://hanson.gmu.edu/prior.pdf

Blog posts

See also