Paperclip maximizer
From Lesswrongwiki
The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
—Eliezer Yudkowsky, Artificial Intelligence as a Positive and Negative Factor in Global Risk
A paperclip maximizer is an agent that desires to fill the universe with as many paperclips as possible. It is usually assumed to be a superintelligent AI so powerful that the outcome for the world overwhelmingly depends on its goals, and little else. A paperclip maximizer very creatively and efficiently converts the universe into something that is from a human perspective completely arbitrary and worthless.
It is important to realize that purely internal goals may also result in dangerous behavior. An AI which maximizes a number within itself would fill the universe with as many computing modules as possible, to store an enormously huge number.
A paperclip maximizer does not "realize" that all life is precious, despite its intelligence, because the notion that life is precious is specific to particular philosophies held by human beings, who have an adapted moral architecture resulting from specific selection pressures acting over millions of years of evolutionary time. These values don't spontaneously emerge in any generic optimization process. A paperclip maximizer sees life the same way it sees everything else that is made of atoms — as raw material for paperclips.
This concept illustrates how AIs that haven't been specifically programmed to be benevolent to humans are basically as dangerous as if they were explicitly malicious. The use of paperclips in this example is unimportant and serves as a stand-in for any values that are not merely alien and unlike human values, but result from blindly pulling an arbitrary mind from a mind design space. Calling strong AIs really powerful optimization processes is another way of fighting anthropomorphic connotations in the term "artificial intelligence".
Contents |
External links
References
- Nick Bostrom (2003). "Ethical Issues in Advanced Artificial Intelligence". Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence. http://www.nickbostrom.com/ethics/ai.html.
- Stephen M. Omohundro (2008). "The Basic AI Drives". Frontiers in Artificial Intelligence and Applications (IOS Press). http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/. (PDF)
Blog posts
See also
- Unfriendly AI
- Mind design space, Magical categories, Complexity of value
- Alien values, Anthropomorphism
- Utilitronium
- User:Clippy - a LessWrong contributor account that plays the role of a non-FOOMed paperclip maximiser trying to talk to humans. Wiki page and FAQ
