Difference between revisions of "Simpleton Gambit"

From Lesswrongwiki
Jump to: navigation, search
(Created page with "The Simpleton Gambit is a thought experiment in which an agent has the opportunity to take an action that will give it an extremely high reward (a larger reward than it would ...")
 
m
 
Line 1: Line 1:
The Simpleton Gambit is a thought experiment in which an agent has the opportunity to take an action that will give it an extremely high reward (a larger reward than it would be able to obtain by doing anything else), with the catch that it requires the agent to modify itself to be unable to take any further actions (aside from some default action). A utility-maximizing agent would accept the gambit, as it results in the highest possible utility. However, if an agent mistakenly believes that it is facing the Simpleton Gambit, and it accepts, it would face a very low payoff as a result of its inability to take actions in the future. Ring and Orseau (2011)<ref>[http://www.idsia.ch/~ring/Orseau,Ring%3BSelf-modification%20and%20Mortality%20in%20Artificial%20Agents,%20AGI%202011.pdf Ring and Orseau (2011)]</ref> discuss how various self-modifying agents would react to the Simpleton Gambit.
+
The Simpleton Gambit is a thought experiment in which an agent has the opportunity to take an action that will give it an extremely high reward (a larger reward than it would be able to obtain by doing anything else), with the catch that it requires the agent to modify itself to be unable to take any further actions (aside from some default action). A utility-maximizing agent would accept the gambit, as it results in the highest possible utility. <br>
 +
However, if an agent mistakenly believes that it is facing the Simpleton Gambit, and it accepts, it would face a very low payoff as a result of its inability to take actions in the future. Ring and Orseau (2011)<ref>[http://www.idsia.ch/~ring/Orseau,Ring%3BSelf-modification%20and%20Mortality%20in%20Artificial%20Agents,%20AGI%202011.pdf Ring and Orseau (2011)]</ref> discuss how various self-modifying agents would react to the Simpleton Gambit.
  
 
==See also==
 
==See also==

Latest revision as of 14:52, 9 August 2013

The Simpleton Gambit is a thought experiment in which an agent has the opportunity to take an action that will give it an extremely high reward (a larger reward than it would be able to obtain by doing anything else), with the catch that it requires the agent to modify itself to be unable to take any further actions (aside from some default action). A utility-maximizing agent would accept the gambit, as it results in the highest possible utility.
However, if an agent mistakenly believes that it is facing the Simpleton Gambit, and it accepts, it would face a very low payoff as a result of its inability to take actions in the future. Ring and Orseau (2011)[1] discuss how various self-modifying agents would react to the Simpleton Gambit.

See also

Delusion box