Difference between revisions of "Anvil problem"

From Lesswrongwiki
Jump to: navigation, search
(Created page with "{{stub}} Eliezer Yudkowsky and others have pointed out that AIXI lacks a self-model: It extrapolates its own actions into the future indefinitely, on the assumption that...")
 
Line 1: Line 1:
 
{{stub}}
 
{{stub}}
 +
  
 
[[Eliezer Yudkowsky]] and others have pointed out that [[AIXI]] lacks a self-model: It extrapolates its own actions into the future indefinitely, on the assumption that it will keep working in the same way in the future.  
 
[[Eliezer Yudkowsky]] and others have pointed out that [[AIXI]] lacks a self-model: It extrapolates its own actions into the future indefinitely, on the assumption that it will keep working in the same way in the future.  
Line 6: Line 7:
  
 
"Though AIXI is an abstraction, any real AI would have a physical embodiment that could be damaged, and an implementation which could change its behavior due to bugs; and the AIXI formalism completely ignores these possibilities. This is called the [[Anvil problem]]:  AIXI would not care if an anvil was about to drop on its head." (Yampolskiy, Fox, 2012).
 
"Though AIXI is an abstraction, any real AI would have a physical embodiment that could be damaged, and an implementation which could change its behavior due to bugs; and the AIXI formalism completely ignores these possibilities. This is called the [[Anvil problem]]:  AIXI would not care if an anvil was about to drop on its head." (Yampolskiy, Fox, 2012).
 +
 +
==References==
  
 
[http://joshuafox.com/media/YampolskiyFox__AGIAndTheHumanModel.pdf R.V. Yampolskiy, J. Fox (2012) Artificial General Intelligence and the Human Mental Model. In Amnon H. Eden, Johnny Søraker, James H. Moor, Eric Steinhart (Eds.), The Singularity Hypothesis.The Frontiers Collection. London: Springer.]
 
[http://joshuafox.com/media/YampolskiyFox__AGIAndTheHumanModel.pdf R.V. Yampolskiy, J. Fox (2012) Artificial General Intelligence and the Human Mental Model. In Amnon H. Eden, Johnny Søraker, James H. Moor, Eric Steinhart (Eds.), The Singularity Hypothesis.The Frontiers Collection. London: Springer.]

Revision as of 06:31, 22 August 2012


Eliezer Yudkowsky and others have pointed out that AIXI lacks a self-model: It extrapolates its own actions into the future indefinitely, on the assumption that it will keep working in the same way in the future.

"AIXI does not 'model itself' to figure out what actions it will take in the future; implicit in its definition is the assumption that it will continue, up until its horizon, to choose actions that maximize expected future value. AIXI’s definition assumes that the maximizing action will always be chosen, despite the fact that the agent’s implementation was predictably destroyed. This is not accurate for real-world implementations which may malfunction, be destroyed, self-modify, etc

"Though AIXI is an abstraction, any real AI would have a physical embodiment that could be damaged, and an implementation which could change its behavior due to bugs; and the AIXI formalism completely ignores these possibilities. This is called the Anvil problem: AIXI would not care if an anvil was about to drop on its head." (Yampolskiy, Fox, 2012).

References

R.V. Yampolskiy, J. Fox (2012) Artificial General Intelligence and the Human Mental Model. In Amnon H. Eden, Johnny Søraker, James H. Moor, Eric Steinhart (Eds.), The Singularity Hypothesis.The Frontiers Collection. London: Springer.