Difference between revisions of "Subgoal stomp"
|Line 8:||Line 8:|
Revision as of 05:02, 16 June 2012
A subgoal is a goal created for the purpose of helping achieve a higher, more important supergoal. Subgoal stomp refers to pursuing a subgoal in a way that defeats the purpose by ignoring the supergoal. The term was originally defined by Eliezer Yudkowsky in Creating Friendly AI;
The concept emphasizes the need for care in designing AGI goal systems. Subgoal stomp can occur any time the programmer fails to give the AI the correct supergoals. It can also happen if the AI does not have a sufficient predictive horizon; that is, if the AI cannot forsee the consequences of its actions far enough ahead. This could occur even with a friendly goal but an inadequate reasoning system.