Difference between revisions of "Fragility of value"

From Lesswrongwiki
Jump to: navigation, search
(Redirected page to Complexity of value)
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
Human values are fickle things. They are the product of our lengthy evolutionary history, and a brief but intense social evolution. It shouldn't be surprising then that our values are an inconsistent mess of contradictions that still haven't been sorted out, by philosophy or psychology.
#REDIRECT [[Complexity of value]]
We have not yet compared values with any other intelligent life, and most of us find those values in our history repulsive. With no inherent morality (see the [[metaethics sequence]], in particular [http://lesswrong.com/lw/sb/could_anything_be_right/ these] [http://lesswrong.com/lw/sm/the_meaning_of_right/ three] [http://lesswrong.com/lw/rr/the_moral_void/ posts]), a person's values define them. Whatever else an [[AGI]] does, it must not change our values - [[the utility function is not up for grabs]].
A poorly constrained [[artificial general intelligence]] seeking to maximize the [[utility]] experienced by humanity might turn us into blobs of perpetual orgasm. A world with immortality designed without [[fun theory]] might find its citizens modifying themselves as to find utterly engrossing a pursuit such as making table legs.
== See Also ==
* [[Metaethics sequence]]
*[[Fun theory]]
*[[The utility function is not up for grabs]]
== Blog Posts ==
*[http://lesswrong.com/lw/y3/value_is_fragile/ Value is Fragile]
*[http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ The Hidden Complexity of Wishes]

Latest revision as of 06:20, 3 October 2012