Optimization daemons

From Lesswrongwiki
Revision as of 06:24, 9 March 2018 by Riceissa (talk | contribs) (External links)
Jump to: navigation, search

Optimization daemons are optimizers that result from heavy optimization pressure on a different system. For example, natural selection is an optimization process (that optimizes for reproductive fitness) that produced humans (who are capable of pursuing goals that no longer correlate reliably with reproductive fitness). In this case, humans are optimization daemons of natural selection. In the context of AI alignment, the concern is that an artificial general intelligence exerting optimization pressure may produce daemons that break alignment.[1]

History

Wei Dai brings up a similar idea in an SL4 thread.[2]

The optimization daemons article on Arbital was published probably in 2016.[1]

Jessica Taylor wrote two posts about daemons while at MIRI:

See also

References

  1. 1.0 1.1 "Optimization daemons". Arbital.
  2. Wei Dai. '"friendly" humans?' December 31, 2003.

External links

Some posts that reference optimization daemons:

Related ideas: