Probability theory

From Lesswrongwiki
Revision as of 06:23, 13 September 2012 by Joaolkf (talk | contribs)
Jump to: navigation, search

Probability theory is a field of mathematics, it studies random variables and processes. The evolution of these processes and variables over time take many possible paths departing from the same initial condition and doesn't follow deterministic rules. Such evolution can be mathematically represented in the form of a probability distribution.

The basic axioms of probability theory can be intuitively stated as:

  • The sum of the probabilities of all elementary events is 1.
  • For any pair of arbitrary events, the joint probability of both is given by the sum of the probabilities of the elementary events which are subsets of both of them at the same time.
  • For any pair of arbitrary events, the disjoint probability of both is given by the sum of the probabilities of the elementary events which are subsets of at least one of them.

Although most of the basics and axioms of probability theory are uncontroversial, the interpretations, usages, and relative importance given to each result vary. There are two main interpretations of the concept of probability: the Bayesian (subjectivist, epistemic or evidential) and the frequentist (objectivist) [1]. The latter was the major and standard view from late 19th century until the late 20th century, when the Bayesian interpretation gained popularity in many fields of science and philosophy[2]. Bayesian networks show themselves to be extremely fruitful in the field of Artificial Intelligence.

The Bayesian interpretation of probability theory aims to model the correct way rational thinking ought to be when bounded by ignorance and uncertainty [3]. It's considered by many others fields as providing the golden reference for rationality – i.e.: in the study of cognitive biases. Its main theorem, Bayes' theorem, follows directly from the basic Probability Axioms, and describe the basics relations between dependent probabilities. Bayesians see this result as prescribing a method on how to update one’s probabilities given evidence, and this update as a defining aspect of the concept of probability. Whereas frequenstists tend to based their view of probability in the Law of Large Numbers, holding the expected probability of an individual event must be close to the average of the results obtained from a large number of trials.

In the Bayesian interpretation, probability is seem as a belief about the credence of an event[4] [5], whereas frequentist interpretations hold that probability is an objective property of a physical system, a propensity on some accounts[6]. An event with Bayesian probability of .6 (or 60%) should be interpreted as stating "With confidence 60%, this event contains the true outcome", whereas a frequentist interpretation would view it as stating "Over 100 trials, we should observe event X approximately 60 times." The Bayesian interpretation holds one should recursively applied the theorem in order to update his beliefs, when presented with new evidence the previous estimation becomes a prior to the next. It’s possible to assign probabilities to ideas or prepositions which aren’t empirical. Given some prior assumptions, in some cases the Bayesian probability can be made to be equal with the empirical frequencies[7] [8].

Many philosophers have argued for a complementary view of probability[9] [10] [8] [7], where both interpretations have their places and value. In "A Subjectivist's Guide to Objective Chance", David Lewis has constructed a view where one can incorporate the frequentist interpretation (objective, chance) inside the Bayesian (subjective, credence) as special case. The frequentist probability is a case of a Bayesian probability conditionalized on truth or empirical evidence: "Chance is objectified subjective probability(...). Objectified credence is credence conditional on the truth." [8]

Blog posts

Notes and References

  1. It’s worth mentioning that inside the Bayesian interpretation, there are also views called objectivists.
  2. WOLPERT, R.L. (2004) “A conversation with James O. Berger”, Statistical science, 9, 205–218.
  3. KORB, Kevin & NICHOLSON, Ann. (2010) "Bayesian Artificial Intelligence". CRC Press. p. 28
  4. BERNARDO, J. M. & SMITH, A. F. M. (1994) “Bayesian Theory”. Wiley.
  5. JAYNES, E. T. (1996) ”Probability theory: The logic of science. Available from
  6. POPPER, Karl.(1959) "The propensity interpretation of probability" The British Journal of the Philosophy of Science, Vol. 10, No. 37. (May, 1959), pp.25-42. Available at:
  7. 7.0 7.1 COX, R. T. (1946) “Probability, frequency and reasonable expectation, American Journal of Physics, vol. 14, no. 1, pp. 1-13
  8. 8.0 8.1 8.2 LEWIS, David. (1980). “A Subjectivist's Guide to Objective Chance”. In JEFFREY, Richard C. (ed.), Studies in Inductive Logic and Probability. University of California Press. Available at:
  9. CARNAP, Rudolf. (1945). "The Two Concepts of Probability", Philosophy and Phenomenological Research 5, 513-532.
  10. JEFFREY, Richard C. (1965). "The Logic of Decision." New York: McGraw-Hill.

See also