Back to LessWrong

Difference between revisions of "5-and-10"

From Lesswrongwiki

Jump to: navigation, search
Line 1: Line 1:
 
The 5-and-10 problem addresses the question of how to construct a theory of logical counterfactuals.  
 
The 5-and-10 problem addresses the question of how to construct a theory of logical counterfactuals.  
 +
 +
Let there be a decision problem which involves the choice between $5 and $10, a utility function that values the $10 more than the $5, and algorithm A optimizing for this utility function.
  
 
[http://lesswrong.com/lw/91c/so_you_want_to_save_the_world/5jdp One version of the 5-and-10 problem] is "I have to decide between $5 and $10. Suppose I decide to choose $5. I know that I'm a money-optimizer, so if I do this, $5 must be more money than $10, so this alternative is better. Therefore, I should choose $5."
 
[http://lesswrong.com/lw/91c/so_you_want_to_save_the_world/5jdp One version of the 5-and-10 problem] is "I have to decide between $5 and $10. Suppose I decide to choose $5. I know that I'm a money-optimizer, so if I do this, $5 must be more money than $10, so this alternative is better. Therefore, I should choose $5."
  
Another [http://lesswrong.com/lw/8wc/a_model_of_udt_with_a_halting_oracle/Another version], sometimes known as the heavy ghost problem, is a problem in certain types of UDT-like decision theories, when the fact that a counterfactual is known to be false makes the algorithm implement it.
+
Another [http://lesswrong.com/lw/8wc/a_model_of_udt_with_a_halting_oracle/Another version], sometimes known as the heavy ghost problem, raises a difficulty with certain types of UDT-like decision theories, when the fact that a counterfactual is known to be false makes the algorithm implement it.
  
Specifically, let there be a decision problem which involves the choice between $5 and $10, a utility function that values the $10 more than the $5, and an algorithm A that reasons something like:
+
The algorithm A that reasons something like:
  
 
:"Look at all proposition of the type '(A decides to do X) implies (Utility=y)', and find the X that maximises y, then do X."
 
:"Look at all proposition of the type '(A decides to do X) implies (Utility=y)', and find the X that maximises y, then do X."

Revision as of 21:37, 8 March 2016

The 5-and-10 problem addresses the question of how to construct a theory of logical counterfactuals.

Let there be a decision problem which involves the choice between $5 and $10, a utility function that values the $10 more than the $5, and algorithm A optimizing for this utility function.

One version of the 5-and-10 problem is "I have to decide between $5 and $10. Suppose I decide to choose $5. I know that I'm a money-optimizer, so if I do this, $5 must be more money than $10, so this alternative is better. Therefore, I should choose $5."

Another version, sometimes known as the heavy ghost problem, raises a difficulty with certain types of UDT-like decision theories, when the fact that a counterfactual is known to be false makes the algorithm implement it.

The algorithm A that reasons something like:

"Look at all proposition of the type '(A decides to do X) implies (Utility=y)', and find the X that maximises y, then do X."

When faced with the above problem, certain types of algorithm can reason:

"The utility of $10 is greater than the utility of $5. Therefore I will never decide to choose $5. Therefore (A decides to do 'choose $5') is a false statement. Since a false statement implies anything, (A decides to do 'choose $5') implies (Utility=y) for any, arbitrarily high, value of y. Therefore this is the utility maximising decision, and I should choose $5."

That is the informal, natural language statement of the problem. Whether the algorithm is actually vulnerable to the 5-and-10 problem depends on the details of what the algorithm is allowed to deduce about itself.