Difference between revisions of "The Hanson-Yudkowsky AI-Foom Debate"

From Lesswrongwiki
Jump to: navigation, search
(Post-book material: * Eliezer's 2017 post, [https://intelligence.org/2017/10/20/alphago/ AlphaGo Zero and the Foom Debate])
 
(18 intermediate revisions by 6 users not shown)
Line 1: Line 1:
In late 2008, an extensive and long-awaited debate about the [[Technological Singularity]] occurred on [[Overcoming Bias]], mainly between [[Robin Hanson]] and [[Eliezer Yudkowsky]]. It focused on the likelihood of [[hard takeoff|hard AI takeoff]] ("[[FOOM]]"), the need for a theory of [[Friendly AI|Friendliness]], and the future of [[AI]], [[brain emulations]], and [[recursive improvement]] in general. This debate is often used to illustrate the difficulty of resolving [[disagreement|disagreements]], even among expert rationalists.
+
In late 2008, an extensive and long-awaited debate about the [[Technological Singularity]] occurred on [[Overcoming Bias]], mainly between [[Robin Hanson]] and [[Eliezer Yudkowsky]]. It focused on the likelihood of [[hard takeoff|hard AI takeoff]] ("[[FOOM]]"), the need for a theory of [[Friendly AI|Friendliness]], and the future of [[AGI]], [[whole brain emulation]], and [[recursive self-improvement]] in general. A stand-up debate on the same themes took place at ''Jane Street Capital'' in June 2011, and Hanson made a further post in 2014.
  
The posts constituting this debate have been collected here to make it easier to follow for future audiences.
+
The best resource for most readers new to the debate will be [[MIRI]]'s 2013 [http://intelligence.org/ai-foom-debate/ The Hanson-Yudkowsky AI-Foom Debate eBook], available in PDF, EPUB, and MOBI formats. This includes
 +
 
 +
* the original series of blog posts,
 +
* a transcript of the 2011 debate
 +
* Kaj Sotala's summary and analysis of the debate, and
 +
* a 2013 technical report on AI takeoff dynamics (“intelligence explosion microeconomics”) written by Yudkowsky.
  
 
==Prologue==
 
==Prologue==
Line 19: Line 24:
 
* [http://www.overcomingbias.com/2008/11/ai-go-foom.html AI Go Foom] by [[Robin Hanson]]
 
* [http://www.overcomingbias.com/2008/11/ai-go-foom.html AI Go Foom] by [[Robin Hanson]]
 
* [http://lesswrong.com/lw/rk/optimization_and_the_singularity/ Optimization and the Singularity] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/rk/optimization_and_the_singularity/ Optimization and the Singularity] by [[Eliezer Yudkowsky]]
* [http://www.overcomingbias.com/2008/06/eliezers-meta-l.html Eliezer's Meta-Level Determinismi] by [[Robin Hanson]]
+
* [http://www.overcomingbias.com/2008/06/eliezers-meta-l.html Eliezer's Meta-Level Determinism] by [[Robin Hanson]]
 
* [http://lesswrong.com/lw/w2/observing_optimization/ Observing Optimization] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/w2/observing_optimization/ Observing Optimization] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/w3/lifes_story_continues/ Life's Story Continues] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/w3/lifes_story_continues/ Life's Story Continues] by [[Eliezer Yudkowsky]]
Line 39: Line 44:
 
* [http://lesswrong.com/lw/wd/disappointment_in_the_future/ Disappointment in the Future] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/wd/disappointment_in_the_future/ Disappointment in the Future] by [[Eliezer Yudkowsky]]
 
* [http://www.overcomingbias.com/2008/12/i-heart-cyc.html I Heart CYC] by [[Robin Hanson]]
 
* [http://www.overcomingbias.com/2008/12/i-heart-cyc.html I Heart CYC] by [[Robin Hanson]]
 +
* [http://www.overcomingbias.com/2010/02/is-the-city-ularity-near.html Is the City-ularity Near?] by [[Robin Hanson]]
 
* [http://lesswrong.com/lw/we/recursive_selfimprovement/ Recursive Self-Improvement] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/we/recursive_selfimprovement/ Recursive Self-Improvement] by [[Eliezer Yudkowsky]]
 
* [http://www.overcomingbias.com/2008/12/whither-manufac.html Whither Manufacturing?] by [[Robin Hanson]]
 
* [http://www.overcomingbias.com/2008/12/whither-manufac.html Whither Manufacturing?] by [[Robin Hanson]]
Line 61: Line 67:
  
 
==Conclusion==
 
==Conclusion==
 
+
 
* [http://www.overcomingbias.com/2008/12/what-core-argument.html What Core Argument?] by [[Robin Hanson]]
 
* [http://www.overcomingbias.com/2008/12/what-core-argument.html What Core Argument?] by [[Robin Hanson]]
 
* [http://lesswrong.com/lw/wp/what_i_think_if_not_why/ What I Think, If Not Why] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/wp/what_i_think_if_not_why/ What I Think, If Not Why] by [[Eliezer Yudkowsky]]
Line 70: Line 76:
 
* [http://www.overcomingbias.com/2008/12/we-agree-get-froze.html We Agree: Get Froze] by [[Robin Hanson]]
 
* [http://www.overcomingbias.com/2008/12/we-agree-get-froze.html We Agree: Get Froze] by [[Robin Hanson]]
 
* [http://lesswrong.com/lw/wq/you_only_live_twice/ You Only Live Twice] by [[Eliezer Yudkowsky]]
 
* [http://lesswrong.com/lw/wq/you_only_live_twice/ You Only Live Twice] by [[Eliezer Yudkowsky]]
 +
* [http://www.overcomingbias.com/2011/07/debating-yudkowsky.html Debating Yudkowsky] by [[Robin Hanson]]
 +
* [http://www.overcomingbias.com/2013/02/foom-debate-again.html Foom Debate, Again] by [[Robin Hanson]]
 +
 +
==Jane Street debate==
 +
 +
A stand-up debate between Hanson and Yudkowsky on the same themes took place on Wednesday June 30 2011 at ''Jane Street Capital''.
 +
 +
* [http://hanson.gmu.edu/ppt/JaneStreetDebate2011vid.wmv Video with better quality audio] 1.57G
 +
* [http://hanson.gmu.edu/ppt/JaneStreetDebate2011.WMA Poor quality WMA audio] 97M
 +
* [http://hanson.gmu.edu/ppt/JaneStreetDebate2011.mp3 Poor quality MP3 audio] 97M
 +
* [https://docs.google.com/document/pub?id=17yLL7B7yRrhV3J9NuiVuac3hNmjeKTVHnqiEa6UQpJk Full transcript]
 +
* [http://lesswrong.com/lw/bug/partial_transcript_of_the_hansonyudkowsky_june/ Partial transcript] by Chris Hallquist
 +
 +
==Summary and analysis==
 +
 +
In Spring 2012 [[Luke Muehlhauser]] asked [[Kaj Sotala]] to write a summary and analysis of the debate; he made a [http://lesswrong.com/lw/fih/a_summary_of_the_hansonyudkowsky_foom_debate/ draft of that analysis] available in November 2012. The final version of the summary is included in the [https://intelligence.org/ai-foom-debate/ ebook].
 +
 +
==Post-book material==
 +
 +
Not included in MIRI's 2013 book on the debate:
 +
 +
* Robin Hanson's 2014 blog post on the subject in response to the publication of "Superintelligence", [http://www.overcomingbias.com/2014/07/30855.html I Still Don’t Get Foom]
 +
* [[Katja Grace]]'s 2014 hosted discussion on Less Wrong, [http://lesswrong.com/r/discussion/lw/l4e/superintelligence_6_intelligence_explosion/ Superintelligence 6: Intelligence explosion kinetics]
 +
* Eliezer's 2017 post, [https://intelligence.org/2017/10/20/alphago/ AlphaGo Zero and the Foom Debate] and Robin Hanson's reply, [https://www.lesserwrong.com/posts/D3NspiH2nhKA6B2PE/what-evidence-is-alphago-zero-re-agi-complexity What Evidence Is AlphaGo Zero Re AGI Complexity?]
  
 
==See also==
 
==See also==

Latest revision as of 13:33, 29 January 2018

In late 2008, an extensive and long-awaited debate about the Technological Singularity occurred on Overcoming Bias, mainly between Robin Hanson and Eliezer Yudkowsky. It focused on the likelihood of hard AI takeoff ("FOOM"), the need for a theory of Friendliness, and the future of AGI, whole brain emulation, and recursive self-improvement in general. A stand-up debate on the same themes took place at Jane Street Capital in June 2011, and Hanson made a further post in 2014.

The best resource for most readers new to the debate will be MIRI's 2013 The Hanson-Yudkowsky AI-Foom Debate eBook, available in PDF, EPUB, and MOBI formats. This includes

  • the original series of blog posts,
  • a transcript of the 2011 debate
  • Kaj Sotala's summary and analysis of the debate, and
  • a 2013 technical report on AI takeoff dynamics (“intelligence explosion microeconomics”) written by Yudkowsky.

Prologue

Main sequence

Conclusion

Postscript

Jane Street debate

A stand-up debate between Hanson and Yudkowsky on the same themes took place on Wednesday June 30 2011 at Jane Street Capital.

Summary and analysis

In Spring 2012 Luke Muehlhauser asked Kaj Sotala to write a summary and analysis of the debate; he made a draft of that analysis available in November 2012. The final version of the summary is included in the ebook.

Post-book material

Not included in MIRI's 2013 book on the debate:

See also