Basic AI drives

From Lesswrongwiki
Revision as of 04:00, 16 October 2012 by Joaolkf (talk | contribs)
Jump to: navigation, search

A basic AI drive is a goal or motivation that most intelligences will have or converge to. The idea was first explored by Steve Omohundro. He argued that sufficiently advanced AI systems would all naturally discover similar instrumental subgoals. The concept was also explored by Nick Bostrom by the term instrumental convergence thesis. The main idea is that a few goals are instrumental to almost all possible final goals. Therefore, all AIs will pursue these instrumental goals. Omohundro uses microeconomic theory by von Neumann to support this idea.

Omonhundro’s Drives

Omonhundro presents two sets of values, one for self improving artificial intelligences [1] and another he says will emerge in any sufficiently advanced AGI system [2]. The former set is composed by four main drives:

  • Self-preservation: A sufficiently advanced AI will probably be the best entity to achieve its goals. Therefore, it must continue existing in order to maximize goal fulfillment. Similarly, if its goal system was modified, then it would likely begin pursuing different goals. Since this is not desirable to the current AI, it would act to preserve the content of its goal system.
  • Efficiency: At any time, the AI will have finite resources of time, space, matter, energy and computational power. Using these more efficiently will increase its utility. This will lead the AI to do things like implement more efficient algorithms, physical embodiments and precise mechanisms. It will also lead the AI to virtualize as much as possible.
  • Acquisition: Resources like matter and energy are fundamentally necessary to act. The more resources the AI can control, the more actions it can perform to achieve its goals. The AIs physical capabilities constitute its level of technology. For instance, if the AI could invent nanotechnology, it would vastly increase the actions it could take to achieve its goals.
  • Creativity: The AIs operations will depend on its ability to come up with new, more efficient ideas. It will be driven to acquire more computational power for raw searching ability, and it will also be driven to search for better search algorithms. Omohundro argues that the drive for creativity is critical for the AI to display the richness and diversity that is valued by humanity. He discusses signaling goals as particularly rich sources of creativity.

Bostrom’s Drives

Bostrom argues for a Orthogonality thesis, which states:

Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.

But he also argues that, despite the fact that values and intelligence are independent, any recursive self-improvement intelligence would likely possess a common set of instrumental values which are useful for achieving any kind of terminal value. For him, those values are:

  • Self-preservation: In order to continue to take actions which will maximize its values, a superintelligence will value its continuing existence.
  • Goal-content integrity: Since modification trough swaping memories, downloading skills, and radically modifying their cognitive architecture and personalities would be easily available, the superintellinge would greatly value the preservation of its own set of values and goals even after extensive modification.
  • Cognitive enhancement: Improvements in cognitive capacity, intelligence and rationality would help the superintelligence to become a better decision-maker and enhance its ability to achieve its final goals.
  • Technological perfection: Increases in hardware power and algorithm efficiency will deliver increases in its cognitive capacities. Also, better engineering would enable the creation of a wider set of physical structures using few resources (i.e.? Nanotechnology)
  • Resource acquisition: More the guaranteeing the superintelligence continuing existence, basic resources such as time, space, matter and free energy could be processed to serve almost any goal, in the form of extended hardware, backups and protection. In the case of competing superintelligences, resources accumulation could be driven my social status concerns.


Both Bostrom and Omohundro argue these values should be used in trying to predict a superintelligence behavior since they could be the only set of values shared by any kind of superintelligence. They also mention that these values by themselves don't provide any safety indication. Omohundro says:

The best of these traits could usher in a new era of peace and prosperity; the worst are characteristic of human psychopaths and could bring widespread destruction.

Pathological Cases

In some rarer cases, AIs may not pursue these goals. For instance, if there are two AIs with the same goals, the less capable AI may determine that it should destroy itself to allow the stronger AI to control the universe. Or, an AI may have the goal of using as little resources as possible, or of being as unintelligent as possible. These goals will inherently limit the growth and power of the AI.

See also