Difference between revisions of "5-and-10"

From Lesswrongwiki
Jump to: navigation, search
 
Line 16: Line 16:
  
 
That is the informal, natural language statement of the problem. Whether the algorithm is actually vulnerable to the 5-and-10 problem depends on the details of what the algorithm is allowed to deduce about itself.
 
That is the informal, natural language statement of the problem. Whether the algorithm is actually vulnerable to the 5-and-10 problem depends on the details of what the algorithm is allowed to deduce about itself.
 +
 +
==References==
 +
* [https://agentfoundations.org/item?id=1399 Two Major Obstacles for Logical Inductor Decision Theory] by Scott Garrabrant

Latest revision as of 21:25, 28 September 2017

The 5-and-10 problem addresses the question of how to construct a theory of logical counterfactuals.

Let there be a decision problem which involves the choice between $5 and $10, a utility function that values the $10 more than the $5, and algorithm A optimizing for this utility function.

One version of the 5-and-10 problem is "I have to decide between $5 and $10. Suppose I decide to choose $5. I know that I'm a money-optimizer, so if I do this, $5 must be more money than $10, so this alternative is better. Therefore, I should choose $5."

Another version, sometimes known as the heavy ghost problem, raises a difficulty with certain types of UDT-like decision theories, when the fact that a counterfactual is known to be false makes the algorithm implement it.

The algorithm A reasons something like:

"Look at all proposition of the type '(A decides to do X) implies (Utility=y)', and find the X that maximises y, then do X."

When faced with the above problem, certain types of algorithm can reason:

"The utility of $10 is greater than the utility of $5. Therefore I will never decide to choose $5. Therefore (A decides to do 'choose $5') is a false statement. Since a false statement implies anything, (A decides to do 'choose $5') implies (Utility=y) for any, arbitrarily high, value of y. Therefore this is the utility maximising decision, and I should choose $5."

That is the informal, natural language statement of the problem. Whether the algorithm is actually vulnerable to the 5-and-10 problem depends on the details of what the algorithm is allowed to deduce about itself.

References