Difference between revisions of "Scoring rule"

From Lesswrongwiki
Jump to: navigation, search
Line 1: Line 1:
 
{{wikilink}}
 
{{wikilink}}
In [[decision theory]], a '''scoring rule''' is a measure of someone's performance at making predictions under uncertainty (which comprises two distinct aspects, calibration and discrimination). A '''proper scoring rule''' encourages the forecaster to be honest, as [[expected value|expected]] payoff is maximized by accurately reporting personal belief about the predicted event, rather than by gaming the system.
+
In [[decision theory]], a '''scoring rule''' is a measure of performance of probabilistic predictions - made under uncertainty.
  
==See also==
+
As an example of a probabilistic prediction, consider a sports magazine dealing with horse races that gives the winning chance of each horse for each race the day before. If we gather data regarding those predictions and compare it to the actual results, we have a measure – a scoring rule - of the magazine’s performance. This scoring is almost always not linear, however, and there are many different transformations which are widely used.
  
*[[Technical explanation]]
+
==Proper scoring rules==
 +
A '''proper scoring rule''' is one that encourages the forecaster to be honest – that is, the [[expected value|expected]] payoff is maximized by accurately reporting personal beliefs about the predicted event, rather than by gaming the system.
 +
These rules include the Logarithmic scoring rule, Spherical scoring rule and Brier/Quadratic scoring rule.
  
 
==References==
 
==References==
 
+
* Bickel, E.J. (2007). "Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules". Decision Analysis, 4, (2), 49–65.
 
*{{cite journal
 
*{{cite journal
 
| author      = Tilmann Gneiting; Adrian E Raftery
 
| author      = Tilmann Gneiting; Adrian E Raftery
Line 19: Line 21:
 
}} ([http://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf PDF])
 
}} ([http://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf PDF])
 
*[http://yudkowsky.net/rational/technical A Technical Explanation of Technical Explanation]
 
*[http://yudkowsky.net/rational/technical A Technical Explanation of Technical Explanation]
 +
 +
==See also==
 +
*[[Technical explanation]]
  
 
{{stub}}
 
{{stub}}
 
[[Category:Concepts]]
 
[[Category:Concepts]]

Revision as of 01:08, 5 October 2012

Smallwikipedialogo.png
Wikipedia has an article about

In decision theory, a scoring rule is a measure of performance of probabilistic predictions - made under uncertainty.

As an example of a probabilistic prediction, consider a sports magazine dealing with horse races that gives the winning chance of each horse for each race the day before. If we gather data regarding those predictions and compare it to the actual results, we have a measure – a scoring rule - of the magazine’s performance. This scoring is almost always not linear, however, and there are many different transformations which are widely used.

Proper scoring rules

A proper scoring rule is one that encourages the forecaster to be honest – that is, the expected payoff is maximized by accurately reporting personal beliefs about the predicted event, rather than by gaming the system. These rules include the Logarithmic scoring rule, Spherical scoring rule and Brier/Quadratic scoring rule.

References

  • Bickel, E.J. (2007). "Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules". Decision Analysis, 4, (2), 49–65.
  • Tilmann Gneiting; Adrian E Raftery (March 2007). "Strictly Proper Scoring Rules, Prediction, and Estimation". Journal of the American Statistical Association 102 (477): 359-378.  (PDF)
  • A Technical Explanation of Technical Explanation

See also