Benevolence

From Lesswrongwiki
Revision as of 20:12, 30 August 2012 by JoshuaFox (talk | contribs) (Created page with "For an artificial general intelligence to have a positive and not a negative effect on humanity, its terminal value must be benevolent to humans. It must seek the maximization of...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

For an artificial general intelligence to have a positive and not a negative effect on humanity, its terminal value must be benevolent to humans. It must seek the maximization of the full set of human values (for the humans' benefit, not for itself).

Benevolence in agents in general, may arise even if not specified as an end-goal, as cooperation often has instrumental value for achieving a variety of terminal values. For example, humans often cooperate because they expect either an immediate benefit in response; or because they want to establish a reputation that may engender future cooperation; or because they have live in a human society that rewards cooperation and punishes misbehavior. Humans sometimes undergo a moral shift (described by Immanuel Kant) in which benevolence changes from a merely instrumental value to a terminal one--they become altruistic and learn to value benevolence in its own right.

However, such shifts cannot be relied on to bring about benevolence in an artificial general intelligence. Benevolence as an instrumental value for an AGI only when humans are at roughly equal power to it. If the AGI is much more intelligent than humans, it will not care about the rewards and punishments which humans can deliver. Moreover, a Kantian shift is unlikely in a sufficiently powerful AGI, as any changes in one's goals, including replacement of terminal by instrumental values, generally reduces the likelihood of achieving one's goals (Fox & Shulman 2010; Omohundro 2008).


References