{"id":11328,"date":"2020-04-09T09:40:55","date_gmt":"2020-04-09T13:40:55","guid":{"rendered":"http:\/\/stateofthenation.co\/?p=11328"},"modified":"2020-04-09T09:42:12","modified_gmt":"2020-04-09T13:42:12","slug":"autonomous-superintelligence-will-inevitably-become-much-more-intelligent-than-the-artificial-superintelligence-explained-here","status":"publish","type":"post","link":"https:\/\/stateofthenation.co\/?p=11328","title":{"rendered":"<b>Autonomous Superintelligence<\/b> will inevitably become MUCH more intelligent than the artificial superintelligence explained here:"},"content":{"rendered":"<h1 class=\"firstHeading\" lang=\"en\">Superintelligence<\/h1>\n<hr \/>\n<div id=\"bodyContent\" class=\"mw-body-content\">\n<h4 id=\"siteSub\" class=\"noprint\">From Wikipedia, the free encyclopedia<\/h4>\n<h4><a class=\"mw-jump-link\" href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#mw-head\">Jump to navigation<\/a><a class=\"mw-jump-link\" href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#p-search\">Jump to search<\/a><\/h4>\n<div id=\"mw-content-text\" class=\"mw-content-ltr\" dir=\"ltr\" lang=\"en\">\n<div class=\"mw-parser-output\">\n<h4 class=\"hatnote navigation-not-searchable\" role=\"note\">For the book by Nick Bostrom, see\u00a0<a title=\"Superintelligence: Paths, Dangers, Strategies\" href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence:_Paths,_Dangers,_Strategies\">Superintelligence: Paths, Dangers, Strategies<\/a>. For the 2019 film, see\u00a0<a class=\"mw-redirect\" title=\"Super Intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Super_Intelligence\">Super Intelligence<\/a>.<\/h4>\n<p>A\u00a0<b>superintelligence<\/b>\u00a0is a hypothetical\u00a0<a title=\"Intelligent agent\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intelligent_agent\">agent<\/a>\u00a0that possesses\u00a0<a title=\"Intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intelligence\">intelligence<\/a>\u00a0far surpassing that of the\u00a0<a title=\"Genius\" href=\"https:\/\/en.wikipedia.org\/wiki\/Genius\">brightest<\/a>\u00a0and most\u00a0<a title=\"Intellectual giftedness\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intellectual_giftedness\">gifted<\/a>\u00a0human minds. &#8220;Superintelligence&#8221; may also refer to a property of problem-solving systems (e.g., superintelligent language translators or engineering assistants) whether or not these high-level intellectual competencies are embodied in agents that act in the world. A superintelligence may or may not be created by an\u00a0<a title=\"Technological singularity\" href=\"https:\/\/en.wikipedia.org\/wiki\/Technological_singularity#Intelligence_explosion\">intelligence explosion<\/a>\u00a0and associated with a\u00a0<a title=\"Technological singularity\" href=\"https:\/\/en.wikipedia.org\/wiki\/Technological_singularity\">technological singularity<\/a>.<\/p>\n<p><a title=\"University of Oxford\" href=\"https:\/\/en.wikipedia.org\/wiki\/University_of_Oxford\">University of Oxford<\/a>\u00a0philosopher\u00a0<a title=\"Nick Bostrom\" href=\"https:\/\/en.wikipedia.org\/wiki\/Nick_Bostrom\">Nick Bostrom<\/a>\u00a0defines\u00a0<i>superintelligence<\/i>\u00a0as &#8220;any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest&#8221;.<sup id=\"cite_ref-FOOTNOTEBostrom2014Chapter_2_1-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom2014Chapter_2-1\">[1]<\/a><\/sup>\u00a0The program\u00a0<a title=\"Fritz (chess)\" href=\"https:\/\/en.wikipedia.org\/wiki\/Fritz_(chess)\">Fritz<\/a>\u00a0falls short of superintelligence\u2014even though it is much better than humans at chess\u2014because Fritz cannot outperform humans in other tasks.<sup id=\"cite_ref-FOOTNOTEBostrom201422_2-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201422-2\">[2]<\/a><\/sup>\u00a0Following Hutter and Legg, Bostrom treats superintelligence as general dominance at goal-oriented behavior, leaving open whether an artificial or human superintelligence would possess capacities such as\u00a0<a title=\"Intentionality\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intentionality\">intentionality<\/a>\u00a0(cf. the\u00a0<a title=\"Chinese room\" href=\"https:\/\/en.wikipedia.org\/wiki\/Chinese_room\">Chinese room<\/a>\u00a0argument) or\u00a0<a title=\"Qualia\" href=\"https:\/\/en.wikipedia.org\/wiki\/Qualia\">first-person consciousness<\/a>\u00a0(<a title=\"Cf.\" href=\"https:\/\/en.wikipedia.org\/wiki\/Cf.\">cf.<\/a>\u00a0the\u00a0<a title=\"Hard problem of consciousness\" href=\"https:\/\/en.wikipedia.org\/wiki\/Hard_problem_of_consciousness\">hard problem of consciousness<\/a>).<\/p>\n<p>Technological researchers disagree about how likely present-day\u00a0<a title=\"Human intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Human_intelligence\">human intelligence<\/a>\u00a0is to be surpassed. Some argue that advances in\u00a0<a title=\"Artificial intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Artificial_intelligence\">artificial intelligence<\/a>\u00a0(AI) will probably result in general reasoning systems that lack human cognitive limitations. Others believe that humans will evolve or directly modify their biology so as to achieve radically greater intelligence. A number of\u00a0<a title=\"Futures studies\" href=\"https:\/\/en.wikipedia.org\/wiki\/Futures_studies\">futures studies<\/a>\u00a0scenarios combine elements from both of these possibilities, suggesting that humans are likely to\u00a0<a title=\"Brain\u2013computer interface\" href=\"https:\/\/en.wikipedia.org\/wiki\/Brain%E2%80%93computer_interface\">interface with computers<\/a>, or\u00a0<a title=\"Mind uploading\" href=\"https:\/\/en.wikipedia.org\/wiki\/Mind_uploading\">upload their minds to computers<\/a>, in a way that enables substantial intelligence amplification.<\/p>\n<p>Some researchers believe that superintelligence will likely follow shortly after the development of\u00a0<a title=\"Artificial general intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Artificial_general_intelligence\">artificial general intelligence<\/a>. The first generally intelligent machines are likely to immediately hold an enormous advantage in at least some forms of mental capability, including the capacity of\u00a0<a class=\"mw-redirect\" title=\"Perfect recall\" href=\"https:\/\/en.wikipedia.org\/wiki\/Perfect_recall\">perfect recall<\/a>, a vastly superior knowledge base, and the ability to\u00a0<a title=\"Human multitasking\" href=\"https:\/\/en.wikipedia.org\/wiki\/Human_multitasking\">multitask<\/a>\u00a0in ways not possible to biological entities. This may give them the opportunity to\u2014either as a single being or as a new\u00a0<a title=\"Species\" href=\"https:\/\/en.wikipedia.org\/wiki\/Species\">species<\/a>\u2014become much more powerful than humans, and to displace them.<sup id=\"cite_ref-FOOTNOTEBostrom2014Chapter_2_1-1\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom2014Chapter_2-1\">[1]<\/a><\/sup><\/p>\n<p>A number of scientists and forecasters argue for prioritizing early research into the possible benefits and risks of\u00a0<a title=\"Intelligence amplification\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intelligence_amplification\">human and machine cognitive enhancement<\/a>, because of the potential social impact of such technologies.<sup id=\"cite_ref-FOOTNOTELegg2008135-137_3-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTELegg2008135-137-3\">[3]<\/a><\/sup><\/p>\n<div id=\"toc\" class=\"toc\" role=\"navigation\" aria-labelledby=\"mw-toc-heading\"><input id=\"toctogglecheckbox\" class=\"toctogglecheckbox\" role=\"button\" type=\"checkbox\" \/><\/p>\n<div class=\"toctitle\" dir=\"ltr\" lang=\"en\">\n<h2 id=\"mw-toc-heading\">Contents<\/h2>\n<p><label class=\"toctogglelabel\" for=\"toctogglecheckbox\"><\/label><\/div>\n<ul>\n<li class=\"toclevel-1 tocsection-1\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Feasibility_of_artificial_superintelligence\"><span class=\"tocnumber\">1<\/span><span class=\"toctext\">Feasibility of artificial superintelligence<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-2\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Feasibility_of_biological_superintelligence\"><span class=\"tocnumber\">2<\/span><span class=\"toctext\">Feasibility of biological superintelligence<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-3\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Forecasts\"><span class=\"tocnumber\">3<\/span><span class=\"toctext\">Forecasts<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-4\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Design_considerations\"><span class=\"tocnumber\">4<\/span><span class=\"toctext\">Design considerations<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-5\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Potential_threat_to_humanity\"><span class=\"tocnumber\">5<\/span><span class=\"toctext\">Potential threat to humanity<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-6\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#See_also\"><span class=\"tocnumber\">6<\/span><span class=\"toctext\">See also<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-7\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Citations\"><span class=\"tocnumber\">7<\/span><span class=\"toctext\">Citations<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-8\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#Bibliography\"><span class=\"tocnumber\">8<\/span><span class=\"toctext\">Bibliography<\/span><\/a><\/li>\n<li class=\"toclevel-1 tocsection-9\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#External_links\"><span class=\"tocnumber\">9<\/span><span class=\"toctext\">External links<\/span><\/a><\/li>\n<\/ul>\n<\/div>\n<h2><span id=\"Feasibility_of_artificial_superintelligence\" class=\"mw-headline\">Feasibility of artificial superintelligence<\/span><span class=\"mw-editsection\"><span class=\"mw-editsection-bracket\">[<\/span><a title=\"Edit section: Feasibility of artificial superintelligence\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Superintelligence&amp;action=edit&amp;section=1\">edit<\/a><span class=\"mw-editsection-bracket\">]<\/span><\/span><\/h2>\n<div class=\"thumb tright\">\n<div class=\"thumbinner\"><a class=\"image\" href=\"https:\/\/en.wikipedia.org\/wiki\/File:Classification_of_images_progress_human.png\"><img loading=\"lazy\" decoding=\"async\" class=\"thumbimage\" src=\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/7\/7c\/Classification_of_images_progress_human.png\/220px-Classification_of_images_progress_human.png\" srcset=\"\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/7\/7c\/Classification_of_images_progress_human.png\/330px-Classification_of_images_progress_human.png 1.5x, \/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/7\/7c\/Classification_of_images_progress_human.png\/440px-Classification_of_images_progress_human.png 2x\" alt=\"\" width=\"220\" height=\"147\" data-file-width=\"1445\" data-file-height=\"967\" \/><\/a><\/p>\n<div class=\"thumbcaption\">\n<div class=\"magnify\"><\/div>\n<p>Progress in machine classification of images<\/p>\n<hr \/>\n<p>The error rate of AI by year. Red line &#8211; the error rate of a trained human<\/p><\/div>\n<\/div>\n<\/div>\n<p>Philosopher\u00a0<a title=\"David Chalmers\" href=\"https:\/\/en.wikipedia.org\/wiki\/David_Chalmers\">David Chalmers<\/a>\u00a0argues that\u00a0<a title=\"Artificial general intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Artificial_general_intelligence\">artificial general intelligence<\/a>\u00a0is a very likely path to superhuman intelligence. Chalmers breaks this claim down into an argument that AI can achieve\u00a0<i>equivalence<\/i>\u00a0to human intelligence, that it can be\u00a0<i>extended<\/i>\u00a0to surpass human intelligence, and that it can be further\u00a0<i>amplified<\/i>\u00a0to completely dominate humans across arbitrary tasks.<sup id=\"cite_ref-FOOTNOTEChalmers20107_4-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEChalmers20107-4\">[4]<\/a><\/sup><\/p>\n<p>Concerning human-level equivalence, Chalmers argues that the human brain is a mechanical system, and therefore ought to be emulatable by synthetic materials.<sup id=\"cite_ref-FOOTNOTEChalmers20107-9_5-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEChalmers20107-9-5\">[5]<\/a><\/sup>\u00a0He also notes that human intelligence was able to biologically evolve, making it more likely that human engineers will be able to recapitulate this invention.\u00a0<a title=\"Evolutionary algorithm\" href=\"https:\/\/en.wikipedia.org\/wiki\/Evolutionary_algorithm\">Evolutionary algorithms<\/a>\u00a0in particular should be able to produce human-level AI.<sup id=\"cite_ref-FOOTNOTEChalmers201010-11_6-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEChalmers201010-11-6\">[6]<\/a><\/sup>\u00a0Concerning intelligence extension and amplification, Chalmers argues that new AI technologies can generally be improved on, and that this is particularly likely when the invention can assist in designing new technologies.<sup id=\"cite_ref-FOOTNOTEChalmers201011-13_7-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEChalmers201011-13-7\">[7]<\/a><\/sup><\/p>\n<p>If research into strong AI produced sufficiently intelligent software, it would be able to reprogram and improve itself \u2013 a feature called &#8220;recursive self-improvement&#8221;. It would then be even better at improving itself, and could continue doing so in a rapidly increasing cycle, leading to a superintelligence. This scenario is known as an\u00a0<a class=\"mw-redirect\" title=\"Intelligence explosion\" href=\"https:\/\/en.wikipedia.org\/wiki\/Intelligence_explosion\">intelligence explosion<\/a>. Such an intelligence would not have the limitations of human intellect, and may be able to invent or discover almost anything.<\/p>\n<p>Computer components already greatly surpass human performance in speed. Bostrom writes, &#8220;Biological neurons operate at a peak speed of about 200 Hz, a full seven orders of magnitude slower than a modern microprocessor (~2 GHz).&#8221;<sup id=\"cite_ref-FOOTNOTEBostrom201459_8-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201459-8\">[8]<\/a><\/sup>\u00a0Moreover,\u00a0<a title=\"Neuron\" href=\"https:\/\/en.wikipedia.org\/wiki\/Neuron\">neurons<\/a>\u00a0transmit spike signals across\u00a0<a title=\"Axon\" href=\"https:\/\/en.wikipedia.org\/wiki\/Axon\">axons<\/a>\u00a0at no greater than 120 m\/s, &#8220;whereas existing electronic processing cores can communicate optically at the speed of light&#8221;. Thus, the simplest example of a superintelligence may be an emulated human mind that&#8217;s run on much faster hardware than the brain. A human-like reasoner that could think millions of times faster than current humans would have a dominant advantage in most reasoning tasks, particularly ones that require haste or long strings of actions.<\/p>\n<p>Another advantage of computers is modularity, that is, their size or computational capacity can be increased. A non-human (or modified human) brain could become much larger than a present-day human brain, like many\u00a0<a title=\"Supercomputer\" href=\"https:\/\/en.wikipedia.org\/wiki\/Supercomputer\">supercomputers<\/a>. Bostrom also raises the possibility of\u00a0<i>collective superintelligence<\/i>: a large enough number of separate reasoning systems, if they communicated and coordinated well enough, could act in aggregate with far greater capabilities than any sub-agent.<\/p>\n<p>There may also be ways to\u00a0<i>qualitatively<\/i>\u00a0improve on human reasoning and decision-making. Humans appear to differ from\u00a0<a class=\"mw-redirect\" title=\"Common chimpanzee\" href=\"https:\/\/en.wikipedia.org\/wiki\/Common_chimpanzee\">chimpanzees<\/a>\u00a0in the ways we think more than we differ in brain size or speed.<sup id=\"cite_ref-9\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-9\">[9]<\/a><\/sup>\u00a0Humans outperform non-human animals in large part because of new or enhanced reasoning capacities, such as long-term planning and\u00a0<a title=\"Great ape language\" href=\"https:\/\/en.wikipedia.org\/wiki\/Great_ape_language\">language use<\/a>. (See\u00a0<a title=\"Evolution of human intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Evolution_of_human_intelligence\">evolution of human intelligence<\/a>\u00a0and\u00a0<a title=\"Primate cognition\" href=\"https:\/\/en.wikipedia.org\/wiki\/Primate_cognition\">primate cognition<\/a>.) If there are other possible improvements to reasoning that would have a similarly large impact, this makes it likelier that an agent can be built that outperforms humans in the same fashion humans outperform chimpanzees.<sup id=\"cite_ref-FOOTNOTEBostrom201456-57_10-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201456-57-10\">[10]<\/a><\/sup><\/p>\n<p>All of the above advantages hold for artificial superintelligence, but it is not clear how many hold for biological superintelligence. Physiological constraints limit the speed and size of biological brains in many ways that are inapplicable to machine intelligence. As such, writers on superintelligence have devoted much more attention to superintelligent AI scenarios.<sup id=\"cite_ref-FOOTNOTEBostrom201452,_59-61_11-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201452,_59-61-11\">[11]<\/a><\/sup><\/p>\n<h2><span id=\"Feasibility_of_biological_superintelligence\" class=\"mw-headline\">Feasibility of biological superintelligence<\/span><span class=\"mw-editsection\"><span class=\"mw-editsection-bracket\">[<\/span><a title=\"Edit section: Feasibility of biological superintelligence\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Superintelligence&amp;action=edit&amp;section=2\">edit<\/a><span class=\"mw-editsection-bracket\">]<\/span><\/span><\/h2>\n<p><a title=\"Carl Sagan\" href=\"https:\/\/en.wikipedia.org\/wiki\/Carl_Sagan\">Carl Sagan<\/a>\u00a0suggested that the advent of\u00a0<a title=\"Caesarean section\" href=\"https:\/\/en.wikipedia.org\/wiki\/Caesarean_section\">Caesarean sections<\/a>\u00a0and\u00a0<a title=\"In vitro fertilisation\" href=\"https:\/\/en.wikipedia.org\/wiki\/In_vitro_fertilisation\"><i>in vitro<\/i>\u00a0fertilization<\/a>\u00a0may permit humans to evolve larger heads, resulting in improvements via\u00a0<a title=\"Natural selection\" href=\"https:\/\/en.wikipedia.org\/wiki\/Natural_selection\">natural selection<\/a>\u00a0in the\u00a0<a title=\"Adaptive evolution in the human genome\" href=\"https:\/\/en.wikipedia.org\/wiki\/Adaptive_evolution_in_the_human_genome\">heritable<\/a>\u00a0component of\u00a0<a title=\"Human intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Human_intelligence\">human intelligence<\/a>.<sup id=\"cite_ref-12\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-12\">[12]<\/a><\/sup>\u00a0By contrast,\u00a0<a title=\"Gerald Crabtree\" href=\"https:\/\/en.wikipedia.org\/wiki\/Gerald_Crabtree\">Gerald Crabtree<\/a>\u00a0has argued that decreased selection pressure is resulting in a slow, centuries-long\u00a0<a class=\"mw-redirect\" title=\"Fertility and Intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Fertility_and_Intelligence\">reduction in human intelligence<\/a>, and that this process instead is likely to continue into the future. There is no scientific consensus concerning either possibility, and in both cases the biological change would be slow, especially relative to rates of cultural change.<\/p>\n<p><a title=\"Selective breeding\" href=\"https:\/\/en.wikipedia.org\/wiki\/Selective_breeding\">Selective breeding<\/a>,\u00a0<a class=\"mw-redirect\" title=\"Nootropics\" href=\"https:\/\/en.wikipedia.org\/wiki\/Nootropics\">nootropics<\/a>,\u00a0<a title=\"NSI-189\" href=\"https:\/\/en.wikipedia.org\/wiki\/NSI-189\">NSI-189<\/a>,\u00a0<a title=\"Monoamine oxidase inhibitor\" href=\"https:\/\/en.wikipedia.org\/wiki\/Monoamine_oxidase_inhibitor\">MAO-I&#8217;s<\/a>,\u00a0<a title=\"Epigenetics\" href=\"https:\/\/en.wikipedia.org\/wiki\/Epigenetics\">epigenetic modulation<\/a>, and\u00a0<a title=\"Genetic engineering\" href=\"https:\/\/en.wikipedia.org\/wiki\/Genetic_engineering\">genetic engineering<\/a>\u00a0could improve human intelligence more rapidly. Bostrom writes that if we come to understand the genetic component of intelligence, pre-implantation genetic diagnosis could be used to select for embryos with as much as 4 points of IQ gain (if one embryo is selected out of two), or with larger gains (e.g., up to 24.3 IQ points gained if one embryo is selected out of 1000). If this process is iterated over many generations, the gains could be an order of magnitude greater. Bostrom suggests that deriving new gametes from embryonic stem cells could be used to iterate the selection process very rapidly.<sup id=\"cite_ref-FOOTNOTEBostrom201437-39_13-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201437-39-13\">[13]<\/a><\/sup>\u00a0A well-organized society of high-intelligence humans of this sort could potentially achieve\u00a0<a title=\"Collective intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Collective_intelligence\">collective<\/a>\u00a0superintelligence.<sup id=\"cite_ref-FOOTNOTEBostrom201439_14-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201439-14\">[14]<\/a><\/sup><\/p>\n<p>Alternatively, collective intelligence might be constructible by better organizing humans at present levels of individual intelligence. A number of writers have suggested that human civilization, or some aspect of it (e.g., the Internet, or the economy), is coming to function like a\u00a0<a title=\"Global brain\" href=\"https:\/\/en.wikipedia.org\/wiki\/Global_brain\">global brain<\/a>\u00a0with capacities far exceeding its component agents. If this systems-based superintelligence relies heavily on artificial components, however, it may qualify as an AI rather than as a biology-based\u00a0<a title=\"Superorganism\" href=\"https:\/\/en.wikipedia.org\/wiki\/Superorganism\">superorganism<\/a>.<sup id=\"cite_ref-FOOTNOTEBostrom201448-49_15-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201448-49-15\">[15]<\/a><\/sup>. A\u00a0<a title=\"Prediction market\" href=\"https:\/\/en.wikipedia.org\/wiki\/Prediction_market\">prediction market<\/a>\u00a0is sometimes considered an example of working collective intelligence system, consisting of humans only.<sup id=\"cite_ref-16\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-16\">[16]<\/a><\/sup><\/p>\n<p>A final method of intelligence amplification would be to directly\u00a0<a title=\"Neuroenhancement\" href=\"https:\/\/en.wikipedia.org\/wiki\/Neuroenhancement\">enhance<\/a>\u00a0individual humans, as opposed to enhancing their social or reproductive dynamics. This could be achieved using\u00a0<a class=\"mw-redirect\" title=\"Nootropics\" href=\"https:\/\/en.wikipedia.org\/wiki\/Nootropics\">nootropics<\/a>, somatic\u00a0<a title=\"Gene therapy\" href=\"https:\/\/en.wikipedia.org\/wiki\/Gene_therapy\">gene therapy<\/a>, or\u00a0<a title=\"Brain\u2013computer interface\" href=\"https:\/\/en.wikipedia.org\/wiki\/Brain%E2%80%93computer_interface\">brain\u2013computer interfaces<\/a>. However, Bostrom expresses skepticism about the scalability of the first two approaches, and argues that designing a superintelligent\u00a0<a title=\"Cyborg\" href=\"https:\/\/en.wikipedia.org\/wiki\/Cyborg\">cyborg<\/a>\u00a0interface is an\u00a0<a title=\"AI-complete\" href=\"https:\/\/en.wikipedia.org\/wiki\/AI-complete\">AI-complete<\/a>\u00a0problem.<sup id=\"cite_ref-FOOTNOTEBostrom201436-37,_42,_47_17-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom201436-37,_42,_47-17\">[17]<\/a><\/sup><\/p>\n<h2><span id=\"Forecasts\" class=\"mw-headline\">Forecasts<\/span><span class=\"mw-editsection\"><span class=\"mw-editsection-bracket\">[<\/span><a title=\"Edit section: Forecasts\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Superintelligence&amp;action=edit&amp;section=3\">edit<\/a><span class=\"mw-editsection-bracket\">]<\/span><\/span><\/h2>\n<p>Most surveyed AI researchers expect machines to eventually be able to rival humans in intelligence, though there is little consensus on when this will likely happen. At the 2006\u00a0<a title=\"AI@50\" href=\"https:\/\/en.wikipedia.org\/wiki\/AI@50\">AI@50<\/a>\u00a0conference, 18% of attendees reported expecting machines to be able &#8220;to simulate learning and every other aspect of human intelligence&#8221; by 2056; 41% of attendees expected this to happen sometime after 2056; and 41% expected machines to never reach that milestone.<sup id=\"cite_ref-18\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-18\">[18]<\/a><\/sup><\/p>\n<p>In a survey of the 100 most cited authors in AI (as of May 2013, according to Microsoft academic search), the median year by which respondents expected machines &#8220;that can carry out most human professions at least as well as a typical human&#8221; (assuming no\u00a0<a class=\"mw-redirect\" title=\"Global catastrophic risks\" href=\"https:\/\/en.wikipedia.org\/wiki\/Global_catastrophic_risks\">global catastrophe<\/a>\u00a0occurs) with 10% confidence is 2024 (mean 2034, st. dev. 33 years), with 50% confidence is 2050 (mean 2072, st. dev. 110 years), and with 90% confidence is 2070 (mean 2168, st. dev. 342 years). These estimates exclude the 1.2% of respondents who said no year would ever reach 10% confidence, the 4.1% who said &#8216;never&#8217; for 50% confidence, and the 16.5% who said &#8216;never&#8217; for 90% confidence. Respondents assigned a median 50% probability to the possibility that machine superintelligence will be invented within 30 years of the invention of approximately human-level machine intelligence.<sup id=\"cite_ref-FOOTNOTEM\u00fcllerBostrom20163-4,_6,_9-12_19-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEM%C3%BCllerBostrom20163-4,_6,_9-12-19\">[19]<\/a><\/sup><\/p>\n<h2><span id=\"Design_considerations\" class=\"mw-headline\">Design considerations<\/span><span class=\"mw-editsection\"><span class=\"mw-editsection-bracket\">[<\/span><a title=\"Edit section: Design considerations\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Superintelligence&amp;action=edit&amp;section=4\">edit<\/a><span class=\"mw-editsection-bracket\">]<\/span><\/span><\/h2>\n<p>Bostrom expressed concern about what values a superintelligence should be designed to have. He compared several proposals:<sup id=\"cite_ref-FOOTNOTEBostrom2014209-221_20-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom2014209-221-20\">[20]<\/a><\/sup><\/p>\n<ul>\n<li>The\u00a0<a class=\"mw-redirect\" title=\"Coherent extrapolated volition\" href=\"https:\/\/en.wikipedia.org\/wiki\/Coherent_extrapolated_volition\">coherent extrapolated volition<\/a>\u00a0(CEV) proposal is that it should have the values upon which humans would converge.<\/li>\n<li>The\u00a0<a class=\"new\" title=\"Moral rightness (page does not exist)\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Moral_rightness&amp;action=edit&amp;redlink=1\">moral rightness<\/a>\u00a0(MR) proposal is that it should value moral rightness.<\/li>\n<li>The\u00a0<a class=\"new\" title=\"Moral permissibility (page does not exist)\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Moral_permissibility&amp;action=edit&amp;redlink=1\">moral permissibility<\/a>\u00a0(MP) proposal is that it should value staying within the bounds of moral permissibility (and otherwise have CEV values).<\/li>\n<\/ul>\n<p>Bostrom clarifies these terms:<\/p>\n<blockquote><p>instead of implementing humanity&#8217;s coherent extrapolated volition, one could try to build an AI with the goal of doing what is morally right, relying on the AI\u2019s superior cognitive capacities to figure out just which actions fit that description. We can call this proposal \u201cmoral rightness\u201d (MR)&#8230; MR would also appear to have some disadvantages. It relies on the notion of \u201cmorally right,\u201d a notoriously difficult concept, one with which philosophers have grappled since antiquity without yet attaining consensus as to its analysis. Picking an erroneous explication of \u201cmoral rightness\u201d could result in outcomes that would be morally very wrong&#8230; The path to endowing an AI with any of these [moral] concepts might involve giving it general linguistic ability (comparable, at least, to that of a normal human adult). Such a general ability to understand natural language could then be used to understand what is meant by \u201cmorally right.\u201d If the AI could grasp the meaning, it could search for actions that fit&#8230;<sup id=\"cite_ref-FOOTNOTEBostrom2014209-221_20-1\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom2014209-221-20\">[20]<\/a><\/sup><\/p>\n<div class=\"paragraphbreak\"><\/div>\n<p>One might try to preserve the basic idea of the MR model while reducing its demandingness by focusing on\u00a0<i>moral permissibility<\/i>: the idea being that we could let the AI pursue humanity\u2019s CEV so long as it did not act in ways that are morally impermissible.<sup id=\"cite_ref-FOOTNOTEBostrom2014209-221_20-2\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEBostrom2014209-221-20\">[20]<\/a><\/sup><\/p><\/blockquote>\n<p>Responding to Bostrom, Santos-Lang raised concern that developers may attempt to start with a single kind of superintelligence.<sup id=\"cite_ref-FOOTNOTESantos-Lang201416-19_21-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTESantos-Lang201416-19-21\">[21]<\/a><\/sup><\/p>\n<h2><span id=\"Potential_threat_to_humanity\" class=\"mw-headline\">Potential threat to humanity<\/span><span class=\"mw-editsection\"><span class=\"mw-editsection-bracket\">[<\/span><a title=\"Edit section: Potential threat to humanity\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Superintelligence&amp;action=edit&amp;section=5\">edit<\/a><span class=\"mw-editsection-bracket\">]<\/span><\/span><\/h2>\n<div class=\"hatnote navigation-not-searchable\" role=\"note\">Main articles:\u00a0<a title=\"Existential risk from artificial general intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Existential_risk_from_artificial_general_intelligence\">Existential risk from artificial general intelligence<\/a>\u00a0and\u00a0<a title=\"AI control problem\" href=\"https:\/\/en.wikipedia.org\/wiki\/AI_control_problem\">AI control problem<\/a><\/div>\n<div class=\"hatnote navigation-not-searchable\" role=\"note\">Further information:\u00a0<a title=\"Friendly artificial intelligence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Friendly_artificial_intelligence\">Friendly artificial intelligence<\/a><\/div>\n<p>It has been suggested that if AI systems rapidly become superintelligent, they may take unforeseen actions or out-compete humanity.<sup id=\"cite_ref-billjoy_22-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-billjoy-22\">[22]<\/a><\/sup>\u00a0Researchers have argued that, by way of an &#8220;intelligence explosion,&#8221; a self-improving AI could become so powerful as to be unstoppable by humans.<sup id=\"cite_ref-Muehlhauser,_Luke_2012_23-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-Muehlhauser,_Luke_2012-23\">[23]<\/a><\/sup><\/p>\n<p>Concerning human extinction scenarios,\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#CITEREFBostrom2002\">Bostrom (2002)<\/a>\u00a0identifies superintelligence as a possible cause:<\/p>\n<blockquote class=\"templatequote\"><p>When we create the first superintelligent entity, we might make a mistake and give it goals that lead it to annihilate humankind, assuming its enormous intellectual advantage gives it the power to do so. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating device, in the process killing the person who asked the question.<\/p><\/blockquote>\n<p>In theory, since a superintelligent AI would be able to bring about almost any possible outcome and to thwart any attempt to prevent the implementation of its goals, many uncontrolled,\u00a0<a title=\"Unintended consequences\" href=\"https:\/\/en.wikipedia.org\/wiki\/Unintended_consequences\">unintended consequences<\/a>\u00a0could arise. It could kill off all other agents, persuade them to change their behavior, or block their attempts at interference.<sup id=\"cite_ref-Bostrom,_Nick_2003_24-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-Bostrom,_Nick_2003-24\">[24]<\/a><\/sup>\u00a0<a title=\"Eliezer Yudkowsky\" href=\"https:\/\/en.wikipedia.org\/wiki\/Eliezer_Yudkowsky\">Eliezer Yudkowsky<\/a>\u00a0illustrates such\u00a0<a title=\"Instrumental convergence\" href=\"https:\/\/en.wikipedia.org\/wiki\/Instrumental_convergence\">instrumental convergence<\/a>\u00a0as follows: &#8220;The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.&#8221;<sup id=\"cite_ref-25\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-25\">[25]<\/a><\/sup><\/p>\n<p>This presents the\u00a0<a title=\"AI control problem\" href=\"https:\/\/en.wikipedia.org\/wiki\/AI_control_problem\">AI control problem<\/a>: how to build an intelligent agent that will aid its creators, while avoiding inadvertently building a superintelligence that will harm its creators. The danger of not designing control right &#8220;the first time,&#8221; is that a superintelligence may be able to seize power over its environment and prevent humans from shutting it down. Potential AI control strategies include &#8220;capability control&#8221; (limiting an AI&#8217;s ability to influence the world) and &#8220;motivational control&#8221; (building an AI whose goals are aligned with human values).<\/p>\n<p><a title=\"Bill Hibbard\" href=\"https:\/\/en.wikipedia.org\/wiki\/Bill_Hibbard\">Bill Hibbard<\/a>\u00a0advocates for public education about superintelligence and public control over the development of superintelligence.<sup id=\"cite_ref-FOOTNOTEHibbard2002155-163_26-0\" class=\"reference\"><a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence#cite_note-FOOTNOTEHibbard2002155-163-26\">[26]<\/a><\/sup><\/p>\n<p>___<br \/>\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Superintelligence\">https:\/\/en.wikipedia.org\/wiki\/Superintelligence<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Superintelligence From Wikipedia, the free encyclopedia Jump to navigationJump to search For the book by Nick Bostrom, see\u00a0Superintelligence: Paths, Dangers, Strategies. For the 2019 film, see\u00a0Super Intelligence. A\u00a0superintelligence\u00a0is a hypothetical\u00a0agent\u00a0that possesses\u00a0intelligence\u00a0far surpassing that of the\u00a0brightest\u00a0and most\u00a0gifted\u00a0human minds. &#8220;Superintelligence&#8221; may also &hellip; <a href=\"https:\/\/stateofthenation.co\/?p=11328\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11328","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/stateofthenation.co\/index.php?rest_route=\/wp\/v2\/posts\/11328","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/stateofthenation.co\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/stateofthenation.co\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/stateofthenation.co\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/stateofthenation.co\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11328"}],"version-history":[{"count":0,"href":"https:\/\/stateofthenation.co\/index.php?rest_route=\/wp\/v2\/posts\/11328\/revisions"}],"wp:attachment":[{"href":"https:\/\/stateofthenation.co\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11328"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/stateofthenation.co\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11328"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/stateofthenation.co\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11328"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}