Difference between revisions of "Ontological crisis"

From Lesswrongwiki
Jump to: navigation, search
(Added examples; hope they're OK)
Line 1: Line 1:
An '''Ontological crisis''' is the term for the crisis experienced by any agent when its understanding of reality changes. This change can make some of the agent's preferences and goals into nonsense. In the context of an [[AGI]], an ontological crisis could in the worst case pose an [[existential risk]] when old preferences and goals continue to be used. Another possibility is that the AGI loses all ability to comprehend the world, and would pose no threat. If an AGI reevaluates its preferences after its ontological crisis, very [[Unfriendly Artificial Intelligence| unfriendly]] behaviours could arise. Depending on the extent of the reevaluations, the AGI's changes may be detected and safely fixed, or go undetected until they manifest horribly.   
+
'''Ontological crisis''' is a term coined to describe the crisis an agent, human or not, goes through when its model - its ontology - of reality changes. When considering artificial agents, this ontology can be seen as a [[utility function]], which needs to be adapted and re-defined according to the new knowledge of the world.  
 +
In the human context, a clear example of an ontological crisis is a believer’s loss of faith in God. Their motivations and goals, coming from a very specific view of life suddenly become obsolete and maybe even nonsense in the face of this new configuration. The person will then experience a deep crisis and go through the psychological task of reconstructing its set of preferences according the new world view.   
  
Eliezer Yudkowsky has pointed out that we ourselves could experience an ontological crisis if we were to know reality as it was known in the far future. For example, should it be learned that all humans will be copied at the instant of their death and recreated healthy during the 29th century to live as immortals, you might value you life a lot less than your fame. True and universal immortality would cause severe ontological crises as the urgency posed by a definite life span disappeared - and far stranger things. However, the development of indefinite medicinal life extension may make people value their lives far more, and any lifestyle that will let them obtain it. An example of an ontological crisis often experiences today is that following a person's loss of faith in God. Their preferences, often based in their theology and desire of acceptance to [[Wikipedia:Heaven| heaven]], are subject to change as they begin to consider their own values.  
+
When dealing with artificial agents, we, as their creators, are directly interested in their goals. That is, as Peter de Blanc puts it, when we create something we want it to be useful. As such we will have to define the artificial agent’s ontology – but since a fixed ontology severely limits its usefulness we have to think about adaptability. In his 2011 paper, the author then proposes a method to map old ontologies into new ones, thus adapting the agent’s utility functions and avoiding a crisis.
 +
 
 +
This crisis, in the context of an [[AGI]], could in the worst case pose an [[existential risk]] when old preferences and goals continue to be used. Another possibility is that the AGI loses all ability to comprehend the world, and would pose no threat at all. If an AGI reevaluates its preferences after its ontological crisis, for example in the way mentioned above, very [[Unfriendly Artificial Intelligence| unfriendly]] behaviours could arise. Depending on the extent of the reevaluations, the AGI's changes may be detected and safely fixed. On the other hand, It could go undetected until they go wrong - which shows how it is of our interest to deeply explore ontological adaptation methods when designing AI. 
 +
 
 +
==Further Reading & References==
 +
*[http://arxiv.org/abs/1105.3821 Ontological Crises in Artificial Agents' Value Systems] by Peter de Blanc
  
 
== Blog posts ==
 
== Blog posts ==
 
*[http://lesswrong.com/r/discussion/lw/827/ai_ontology_crises_an_informal_typology/ AI ontology crises: an informal typology] by Stuart Armstrong
 
*[http://lesswrong.com/r/discussion/lw/827/ai_ontology_crises_an_informal_typology/ AI ontology crises: an informal typology] by Stuart Armstrong
 
*[http://lesswrong.com/lw/xl/eutopia_is_scary/ Eutopia is Scary] by Eliezer Yudkowsky
 
*[http://lesswrong.com/lw/xl/eutopia_is_scary/ Eutopia is Scary] by Eliezer Yudkowsky
== External Links ==
+
 
*[http://arxiv.org/abs/1105.3821 Ontological Crises in Artificial Agents' Value Systems]
+
==See also==
 +
[[Evolution]]
 +
[[Adaptation executioners]]

Revision as of 09:53, 22 September 2012

Ontological crisis is a term coined to describe the crisis an agent, human or not, goes through when its model - its ontology - of reality changes. When considering artificial agents, this ontology can be seen as a utility function, which needs to be adapted and re-defined according to the new knowledge of the world. In the human context, a clear example of an ontological crisis is a believer’s loss of faith in God. Their motivations and goals, coming from a very specific view of life suddenly become obsolete and maybe even nonsense in the face of this new configuration. The person will then experience a deep crisis and go through the psychological task of reconstructing its set of preferences according the new world view.

When dealing with artificial agents, we, as their creators, are directly interested in their goals. That is, as Peter de Blanc puts it, when we create something we want it to be useful. As such we will have to define the artificial agent’s ontology – but since a fixed ontology severely limits its usefulness we have to think about adaptability. In his 2011 paper, the author then proposes a method to map old ontologies into new ones, thus adapting the agent’s utility functions and avoiding a crisis.

This crisis, in the context of an AGI, could in the worst case pose an existential risk when old preferences and goals continue to be used. Another possibility is that the AGI loses all ability to comprehend the world, and would pose no threat at all. If an AGI reevaluates its preferences after its ontological crisis, for example in the way mentioned above, very unfriendly behaviours could arise. Depending on the extent of the reevaluations, the AGI's changes may be detected and safely fixed. On the other hand, It could go undetected until they go wrong - which shows how it is of our interest to deeply explore ontological adaptation methods when designing AI.

Further Reading & References

Blog posts

See also

Evolution Adaptation executioners