Singularity Countdown

Part of the rationale for being a “transhumanist”, or, more broadly, having grandiose dreams for humanity’s future, is the extremely simple and mundane observation that the available matter and free energy in our general vicinity is far larger than what we have utilized of it thus far. The incoming solar energy is about a million times greater than global energy consumption, and the available hydrothermal energy to be extracted from the energy gradient between the mantle and the upper crust is many times that. These energy sources far exceed that available from all fossil fuels, uranium, and thorium combined. In the long run (less than a century?), solar and hydrothermal will become our primary energy sources, simply because nothing else will be able to meet our exponentially growing demand.

The biosphere contains just two trillion tonnes of carbon, but the oceans contain about 36 trillion tonnes of carbon (mostly as bicarbonate ion), and several trillion tonnes of additional carbon exist as fossil matter, including the leftovers from the catastrophic Azolla event 49 million years ago. Retrieving oceanic carbon and reintroducing it to the organic biosphere could allow us to reestablish beautiful forests over much of the surface of the planet. Historically, tropical forests extended to within 40 degrees of the equator, subtropical forests to 60, and other forests to the poles. Palm trees and turtles thrived at the North Pole. Our current ice, grass, and desert-covered Earth is a geophysical abnormality caused by an Ice Age that began 23 million years ago when Antarctica split from South America, permitting the creation of the Antarctic Circumpolar Current and leading to an “Icebox Earth” with glaciated poles. We have had greener ages, and we can bring them back with technology, particularly organic and inorganic self-replicating agents.

Though most environmentalists center their efforts around preserving currently existing biodiversity, forward-looking environmentalists should look towards not just preserving the already existing biodiversity, by setting environmental conditions conducive to the development of millions of new species and a planet covered in luxuriant foliage. By using vertical farming, which will be demonstrated as proof-of-concept within years, and closed-cycle manufacturing, we can minimize our footprint and sustain upwards of 100 billion people with negligible environmental impact. The current impression that the planet is overpopulated is a selection effect resulting from people living in crowded cities, concentrated by technological and economic necessity. Decentralized manufacturing and high-resolution virtual communication will allow a more evenly distributed populace.

Some, like environmentalist Bill McKibben — have said “Enough”, enough technology, enough life, enough progress. Unsurprisingly, I disagree. Looking back from the perspective of a world more than 20 times lusher and Nature-filled than today, with more than 20 times more people distributed evenly across huge tracts of land now practically empty, it will be hard to say, “we should have stopped when we were just at 5% of this potential”. There have been other times in history with just 5% of the biomass and life of today — immediately after major mass extinctions. If today’s world is “enough”, then why stop there? Why not revert back to a world with even less biodiversity and biomass? It would be a surprising coincidence if the current biomass is just right, rather than too little or too much. Those arguing otherwise are just products of their environment — the glacier, desert, and steppe-covered poverty of the Late Cenozoic.

“The main thing that stands between the human species and the creation of a supervirus is a sense of responsibility among individual biologists.”
– Richard Preston, The Demon in the Freezer, page 227

From a 2002 article by Danny Penman in The Guardian:

“A few months’ work in a makeshift laboratory is all it would take to produce a biological weapon that could kill hundreds of millions of people.

The scientific information is freely available and the raw materials easily sourced. The only difficult part would be mastering the necessary scientific skills, and they are taught on most biology degree courses.

One of the simplest ways of constructing a biological weapon would be to engineer an existing human disease and to make it even more lethal. Something as simple as the flu virus, when engineered with the gene for botulinum toxin, could wipe out a significant part of the human race. A low dose of this toxin is the main ingredient in cosmetic botox injections.

The genetic sequence for the toxin is freely available. This sequence could then be uploaded to a commonly available gene synthesizer, which would churn out millions of copies of the gene in a few hours. The flu virus would then be grown in the presence of this newly synthesized gene. As the virus reproduced, a few of the virus particles would absorb the gene. With a bit of luck, the budding terrorist would have produced a new biological weapon.”

Continue.

Human beings are inherently vulnerable to a significant number of lethal compounds. Given a highly contagious biological vector to distribute these compounds, the potential outcome is grim. Given adequate tools, knowledge, and time, groups could manufacture several such viruses and release them simultaneously in different areas, thwarting quarantine and antidote efforts. Governments know this — hence the Biological Weapons Convention. Bio-weapons are off-limits for warfare at the international level, but such rules might break down in the case of a large enough war, and will not be respected by rogue parties.

DARPA gave it to them:

“RICHARDSON, Texas, Oct. 2 — Zyvex Labs today announced the award of a $9.7M program funded by DARPA (Defense Advanced Research Projects Agency) and Texas’ ETF (Emerging Technology Fund). The goal of this effort is to develop a new manufacturing technique that enables “Tip-Based Nanofabrication” to accelerate the transition of nanotechnology from the laboratory to commercial products. Starting with the construction of ‘one-at-a-time’ atomically precise, ‘quantum dot’ nanotech-based products in volume at practical production rates and costs. Harnessing this capability will position the United States and Texas with the fundamental technology to develop next-generation quantum dot applications for military and commercial applications such as advanced communications, metrology, and quantum computers. The spin-off nanomanufacturing capabilities from that early application will result in revolutionary nanotech products in follow-on development.”

Continue.

In my opinion, current advocates of molecular nanotechnology (MNT) aren’t doing enough to address the risks. Christine Peterson of the Foresight Institute advocates an open source physical security model, which is helpful, but should be accompanied by more specific recommendations to form a seed around which further ideas can accrete. The Center for Responsible Nanotechnology has laid out the technological specs of MNT and called for more discussion, but has provided little in the way of concrete recommendations. Ray Kurzweil seems to just think that everything will pretty much automatically turn out fine.

Nanofactories (manufacturing units based on MNT) will need to have extensive, unhackable built-in safeguards in order to be safe. If they can be hacked and these hacked nanofactories cannot be recovered, that could be very bad (significantly worse that terrorists getting weapons-grade uranium). That’s a phrase I’d like MNT advocates to repeat publicly: “terrorists or tyrants getting their hands on unlocked nanofactories would be far worse than weapons-grade uranium”. Unlocked (or poorly regulated) nanofactories would be able to build devices that enrich uranium many times more effectively than current centrifuge technology. That’s somewhat of a problem, unless we plan to gather up all the uranium on the planet and keep it locked up in vaults.

If unhackable nanofactories cannot be built, then to push ahead on the technology would be irresponsible. Mainstream “experts” will be saying this in 5-15 years, but I’m saying it now.

It’s important to realize the obvious: that every human problem, every malady, every concern, every evil, is at root simply a suboptimal arrangement of atoms and molecules. If this sounds quasi-spiritual, it’s because it is — for millennia, pre-scientific humans have attributed all ills to various agents — the gods, magicians, and other humans. This is because these ills demand an explanation, and we didn’t have a plausible one, so we made it up. Now, at least in the abstract, we have a concrete, very likely correct answer: suboptimal atomic arrangements.

This realization is neither trivial nor too broad to be useless. If your problems are caused by the gods (that some people sadly still believe in…), then to solve them, you either need to give up, on engage in rituals (prayer, sacrifice, etc.) that have an empirical impact of precisely zero. The ultimate promise is that the gods or God will come at the end of time to make everything better. Unfortunately (?) for us, that will never happen.

The alternative is to slash all spirits from your worldview and model the world as a game board where all the pieces are humans. This too isn’t quite correct, as many who avoid the error of deification of Nature fall right into the trap of the fundamental attribution error, where everything that goes right or wrong becomes some human’s fault or credit. The attribution error is absolutely omnipresent in politics, because invoking it also invokes human political emotions that a leader can easily use to manipulate everyone who has never heard of the error. Since this is practically everyone, it’s politically rational to exploit it to its fullest, and a self-reinforcing feedback loop of error is created. Excuse me, but there are a lot of relevant forces in this world besides deliberate human choice. The shared biases of all human beings come to mind, as do biological realities such as the existence of malaria, and economic realities such as centralized manufacturing.

One sidenote on the notion that “all ills are caused by suboptimal atomic arrangements”. People will have different definitions of what is suboptimal, that is patently obvious. That doesn’t change the fact the subjective personal ills are caused by suboptimal atomic rearrangements, or that there’s a huge space in the center of the Venn diagram of shared humans goals that is specified by certain specific atomic arrangements. Simply because we can’t specify all these arrangements doesn’t mean they aren’t there.

Despite my recognition of a physicalist basis to all problems, I do not advocate a universal convergence towards One True Atomic Pattern or other such absolutist nonsense. I simply wish us to recognize that all shared human problems can be ultimately diagnosed and remedied using the scientific method plus remedial effort: use tests to determine the suboptimal atomic arrangements, then devise engineering solutions to rearrange current arrangements into a more optimal state. This holds true for mental phenomena as well as phenomena in the external world — my brain is “the external world” for others and it is entirely physical. Those who advocate an aphysical basis for consciousness are making the same mystical mistakes that our ancestors have yawn-inducingly made for thousands of years. I am special even if my consciousness has a purely physical basis.

It really does exist!

In the secret, back-room Singularitarian mailing lists and discussion venues, we often ask: “More publicity good? Or more publicity bad? How much publicity is optimal?”

There’s no question that our cause (building safe seed AI) has more exposure now than ever. While it can be hard, if not impossible, to distinguish references to Singularity a la Kurzweil from Singularity a la I.J. Good, the two concepts are meshed together and people really do get exposure to both, even if they come away thinking that Singularity means “transhumanism” instead of “recursively self-improving superintelligence”. And the people who are really in the know can actually tell the difference. For instance, Kevin Kelly, founding editor of WIRED, recently wrote about our version of the Singularity at his blog, the Technium. When the Intel CTO mentioned the Singularity coming by 2060, he was talking about Kurzweil’s Singularity, so in my mind that doesn’t really count.

The goal is to get ourselves enough exposure to get the funding and talent we need to implement Friendly AI as quickly and safely as possible, and no more. Any additional exposure is a risk, because it increases the chance that someone with a ton of money says, “AGI, that sounds like a great idea! Good thing Isaac Asimov did all the groundwork on that friendliness issue for us, so we can just plow ahead on the intelligence part!” Then, after a successful brute force implementation, the AI develops self-replicating robotics, creates trillions of dummies that meet the definition of “human” based on its training set, and goes about spending the rest of eternity converting the universe into sock puppets and making certain to obey them. (Which is pretty easy, considering that the AI controls both the dummies and the system doing the obeying.)

The answer to the “more publicity?” question depends greatly on how hard one wagers AGI to be, or more appropriately, what your probability distribution over difficulty levels is. The people who wager that AGI is relatively “easy”, as in, requiring about a dozen brilliant programmer-theorists a la Fellowship of the Ring, along with a good ten or twenty million dollars, won’t want our cause to gain much more publicity or exposure. Those who wager AGI is extremely hard, as in requiring thousands of programmer-theorists and billions of dollars, would obviously want as much exposure as possible, as it would be necessary to reach the finish line. I fall somewhere in the middle.

On Overcoming Bias, Eliezer Yudkowsky recently observed how he thought many people in the field of AGI were simply ordinary. In my worldview, this is great. My personal experience with SIAI employees and interns indicates they are anything but ordinary. That means the “good guys” — those who make a huge deal about AI Friendliness and warn that we could all be exterminated if we mess up AGI programming — are doing better than the “bad guys” — those who just want to create AGI because it sounds like an interesting research project and are anticipating nothing more than obedient robots with IQs of 90.

But, in my view, the “good guys” still don’t have enough resources and talent, so we need more exposure. Not exposure to the general public, but targeted exposure to highly educated audiences. In a certain sense, the meme is self-filtering. Our version of the Singularity can’t be boiled down to soundbites easily. It helps to have detailed background knowledge about things like philosophy of mind, reductionism, rationality, the human tendency towards anthropocentrism, Homo economicus, evolutionary psychology, and more. Average members of the general public may stumble upon blogs like this and try to understand what I’m saying, but based on what I’ve seen, they’re likely to seize on some tiny incidental point I made and ignore the bigger picture, thereby stopping the spread of the meme in its tracks. Insofar as it makes reckless drives towards AGI less probable, that’s a good thing.

In the end, I don’t think that a million dollars a year and a dozen supergeniuses is enough. We need more resources, more talent, because the challenge of AGI is huge. It looks like the probability of success (by anyone) before 2015 is quite low, and the good guys have a significant theoretical head-start. I think we can afford (and in fact require) more exposure, until the necessary philanthropists and supergeniuses step forward. A major software project is not cheap, and taking the planning fallacy into account, things are going to take more work than we suspect. But once we reach that threshold — stop! Don’t keep plugging ahead for exposure like a mindless robot. That’s just what we’re trying to avoid, y’know?

And wait — you said there are smart bloggers out there that actually aren’t writing about this stuff?

The IEET sometimes replicates blog posts of mine in their articles section. They do this because I said they could. I like it when they do that, because I think what I say is important and should be heard by more people. Otherwise I wouldn’t say it.

You can help me and other IEET fellows by caring about the IEET and what it does. There’s cool stuff there. For instance, IEET executive director James Hughes, who many of you may have heard of, recently did an interview with the Boston Globe on extinction risks. It helps when he and IEET fellows do things like that, because it lowers the chance that we all die. And not dying is awesome.

Say that the world’s foremost microbiology expert announced that he or she was working on a synthetic virus with the ability to wipe out all human life. Or that a space-capable country, say, the United States, decided that its new mission is to locate the largest possible near-Earth asteroid and alter its trajectory so it impacts the Earth. Or that the world’s foremost expert on machine learning decided that he is tired of humanity and wants to create an AGI that shreds us all to pieces, to put us out of our own misery. (Yes, the technology to make these risks real may not exist now, but imagine 10 or more years down the line — the near future.)

I’d want to be able to say, “So what? Go ahead. You’ll never succeed, we have safeguards against that in place, and they’ve been extensively tested.”

But I can’t, and we’re always in danger, and will continue to be until safeguards are in place.

A key difference between me and many others who think about extinction risk is that I wouldn’t be unpleasantly surprised at a Doomsday Announcement by a leading scientist — surprised as in I’d feel sad about the announcement, but I would have accounted for the possibility in advance. I’d say, “yes, it’s sad you have to be this way, but I knew all along that we needed safeguards in place that would account for this eventuality”.

Of course, the available resources at my disposal are far less than is necessary to put such safeguards in place. However, it could potentially be done for lower cost than one might think. By fueling our watchguard systems with superintelligence rather than human intelligence, they might actually work, rather than failing when the guy watching the cameras goes to take a leak (”human error”). Recursively self-improving superintelligence might be accessible merely through a well-funded seed AI or human intelligence enhancement effort. A few million or even a few thousand dollars can go a long way here.

Say the world’s leading microbiologist had a team of 100 geniuses ready to join him/her in his mission to wipe out us pesky humans, and they already started on the project yesterday. What are you going to do about it? Say “we need to optimistic in life, we can’t always worry about risks, lol”? The problem with that outlook is that it causes you to die.

In inbox-land, the place where emails happen, I have received a piece of “electronic mail”. This e-mail comes from Peter Voss, Founder & CEO of A2I2, a company formed in December 2001 with the “express goal of developing and commercializing an effective general intelligence software engine”. I’ve been following the company since it came into existence, because hey, making claims about AGI is a big deal. Here’s the email:

“Dear friends of A2I2,

We are nearing the commercialization phase of our project.

In the past many of you have expressed a desire to be involved in some way, and a few of you have helped us in various ways (thank you).

At this stage we could use assistance in four areas:

1) Help us identify a business that could serve as a pilot site.
2) Help us find a high-powered CEO to help with our commercial division.
3) Help us brainstorm various business issues — i.e., provide seasoned business advice.
4) Help us test our technology — no technical skills required!

If you feel that you are willing and able to help us, then send a short email introducing yourself to mail at adaptiveai dot com.

Please note that we require you to sign an NDA (non-disclosure).

Peter Voss and Tas Dienes”

Well, here’s your chance to get involved in an AGI company. I have no idea what they’re cooking up, or what it will do, but you can find more information on the company at their site. Peter Voss has said previously that AGI might be possible within a very short time, just 5-10 years. I’m skeptical, but Voss is no kook, so it’s worth at least considering what he has to say.

Convergence 08, our new unconference, will be held at the Computer History Museum in Mountain View, CA on November 15-16. Early bird registration (available until Oct. 20) is cheap, just $70 for students and $145 for general. After Oct. 20, the price goes to $120 for students and $190 for non-students. Buy tickets now, get conversations with smart people later.

Convergence 08 is co-sponsored by humanity+ (formerly called the WTA), the Singularity Institute, Immortality Institute, Foresight Nanotech Institute, Methuselah Foundation, and the Long Now Foundation. Keynote will be given by futurist Paul Saffo. Because it’s an unconference, attendees will get to structure much of the conference program themselves, giving presentations on a variety of topics. People will be giving presentations simultaneously so there’ll be no boredom — you can wander from talk to talk and stick to your favorite. This conference is being marketed as the first and only forum dedicated to NBIC (nano-bio-info-cogno) technologies.

The day before the conference there is also a seminar on extinction risks, led by the Institute for Ethics and Emerging Technologies (IEET) and the Center for Responsible Nanotechnology (CRN). The text for this academic seminar is the recently-released Global Catastrophic Risks.

Help make this conference a success by showing up and impressing us with how awesome and knowledgable you are. Squeeze extra value out by chatting up and taking photos with heavyweights like Ben Goertzel, Aubrey de Grey, Barney Pell, Nick Bostrom, James Hughes, and let’s not forget, that other guy, me.

As I see it, there are three main categories of risk: bio, nano, and AI/robotics. These man-made risks make up the vast majority of the threat magnitude over the coming century and deserve most of the attention.

Threats of low probability include asteroid strikes, supervolcano eruptions, alien invasions, simulation getting shut down, and many others. Though there is disagreement on whether nuclear war, particle accelerator disasters, or runaway climate change deserve to be counted as substantial-probability extinction threats over the coming century, I would say they are not.

A word on focusing on low probability threats alongside higher probability threats. Mentioning low probability threats just for the sake of comprehensiveness is rhetorically damaging. It distracts from the central thrust by introducing superfluous information. Worse, it can damage credibility of the entire message. Whether fair or unfair, we have seen the doom-worriers of the Large Hadron Collider heavily maligned by both scientists and laypeople in print and online. Even if an x-risk mitigator thought there was some probability of planetary doom due to the LHC, say one in a hundred thousand, the credibility sacrifice of pushing the issue is bound to detract from one’s ability to advocate mitigation of other, much higher-probability threats. So it should be avoided. Of course, if the LHC occupies a dominant portion of the risk pie in one’s personal estimate, it would be rational to devote attention to that, despite the credibility penalty.

Regarding natural vs. artificial threats, there is a credible argument that all natural threats are of substantially low probability. We’re still here. Homonids have been around for at least two million years, despite radically inferior numbers and technology for 99.9% of that time. If our ancestors could survive natural disasters, then we’ll be able to also, with our far superior technology and numbers. Asteroids capable of causing major extinctions only strike the Earth about once every hundred million years or less. In the 600 million or so years that there has been complex multicellular life, there have only been six major extinctions, if you include the present one that humans are causing.

For the central risks that I mentioned, which can also be abbreviated GNR (genetics, nanotechnology, robotics) or GRAIN (genetics, robotics, AI, nanotechnology), I recommend the three S’s: science, standards, and security. Scientific investigation of the risks provides a sound basis for further policy. This takes actual money and work, and won’t occur automatically. Taxpayers should foot the bill. Free market incentives for self-regulation are not enough. Industries have an incentive to downplay the magnitude of risk for short-term gain. I say this as a capitalist and advocate of science and progress. (In our polarized political climate, such disclaimers are unfortunately mandatory.)

After science comes standards. All of an industry, say the nanotechnology industry, or the synthetic biology community, needs to come up with some basic set of safety rules, both for individual workers and for the effects of their industry on the planet and environment as a whole. Examples of industry standards are too numerous to list. How much government involvement should be included in the approach will vary depending on your political philosophy. Too much meddling will cripple an industry and encourage clandestine workarounds, and too little meddling may cause an industry to adopt a “no rules” policy that maximizes profits while ignoring risk. If your libertarian philosophy causes an industry to pursue dangerous practices that increase global risk, then your philosophy has failed to adapt to the dangers of the future. If your interventionist philosophy causes an industry to become frustrated and transfer their operations to another country with no rules, then you’ve failed again. Insofar as it’s possible, discard your context-insensitive political beliefs and adopt context-sensitive, non-partisan approaches to these new challenges. Only then will enough people actually agree with you that the approach is adopted and makes a difference.

After standards comes security. The standards have to actually be enforced, or they are useless. If dangerous genetically engineered microbes are not kept under lock and key, unsavory individuals may get ahold of them and use them to fulfill nefarious ends. Security measures will be bolstered by transparency and increasing surveillance and sousveillance, a natural consequence when you combine human curiosity and cheaper/smaller cameras. Local and global agencies have to cooperate as effectively as possible to ensure that standards are being enforced, both in private and public realms.

Those are my thoughts for today. In summary:

1) if you’re an academic, author, or journalist, write about extinction risks,
2) if you’re otherwise involved in science and technology, help create structures to manage global risk,
3) if you’re someone that makes a decent salary or otherwise has money or resources, consider contributing some of it to efforts to mitigate extinction risk,
4) if you’re anybody else, think carefully about the issue and get informed.

The goal is a world where the annual probability of risk is extremely low, approaching zero. Even if the annual probability is just one in a million, then after a million years we’re likely to destroy ourselves. It would be great if the human race and our descendants and variants lives for a long time, billions of years, colonizing the universe and living happy and fulfilling lives. Our thoughts and actions at this crucial juncture could make the difference.

“I must say that this is the greatest factor — the way in which the expedition is equipped — the way in which every difficulty is foreseen, and precautions taken for meeting or avoiding it. Victory awaits him who has everything in order — luck, people call it. Defeat is certain for him who has neglected to take the necessary precautions in time — this is called bad luck.”
– from The South Pole, by Roald Amundsen

Space stations or lunar settlements won’t help mankind avoid numerous types of extinction risks. This is because 1) any colony would remain near-completely dependent on Earth unless very large and in possession of advanced nanotechnology, and 2) the greatest danger, from superintelligence, could easily reach its long arm into space and crush any human colony if it wanted to.

This is not a challenge we can run away from. We have to stay here and fix it. Space will not swoop down and save the day.

Regarding self-replicating threats, it’s likely that a deep underground self-sufficient bunker would be nearly equivalent in its protective value to a space station, not to mention thousands of times cheaper. On Earth, there is air, organic and inorganic building materials, water, radiation shielding, proximity to other humans, and many other amenities. Even if you completely nuked the face of the planet, it would still remain the most habitable neighborhood in the solar system, hands down. This might have something to do with the fact that we descend from a lineage that has lived here and adapted to the environment for billions of years.

When dealing with extinction risks, we have to be practical, not fanciful, with visions of expensive space stations or lunar bases. That’s reality.

Continuing on with the practical viewpoint, we have to get off our high horse and realize that another species could come along that will easily kick our asses. This species will not come from the skies but from our labs. Ignoring this threat is nothing more than anthropocentric conceit. All the nukes and guns and electromagnetic pulses in the world won’t save us from something that’s fundamentally smarter than we are. The new species will merely think of everything we could come up with to fight against it and plan far in advance to counteract these threats. By the time we realize we’re under attack, it will be way too late. No non-brain-damaged human would lose a battle of wits with a Homo erectus, and no Neo sapiens or Colossus will lose a battle of wits with humans.

Accepting the threat of superintelligence involves 1) understanding that human intelligence is finite, understandable, and ultimately engineerable, just like the body (surprise!), and 2) humans are not local instantiations of some Turing-complete Godhead that intelligent species lapse into the second they’re smart enough to take over their own planet, but actually close to the dumbest that a species can be and establish a civilization. Incremental evolutionary processes don’t provide huge intelligence boosts, so Homo sapiens is just a minor tweak on what came before us, a minor tweak just good enough to launch us into the civilizational feedback loop of local dominance. A major tweak would put us into an entirely new realm, but most thinkers seem to assume that such a major tweak will just result in more entities essentially the same as us, but with bigger, bald heads, the propensity to speak in calm, authoritative language, and wear shiny silver/purple clothing. But this is just another monkey in a suit, not a new being.

Asserting with idle confidence that superintelligence won’t be here for centuries, or ever, is just another repeat of anthropocentric conceit. This is just over-worshipping intelligence like the phenomenon of heavier-than-air flight was once over-worshipped (”they’re trying to be like angels”), life was over-worshipped (”humans will never be able to create life in a lab”), the Sun was over-worshipped (”mankind will never be able to harness the power that illuminates the Sun”), the division between the heavens and Earth was over-worshipped (”we’ll never fly to the Moon”), and so on. We pretend that mysteriousness is a property of the territory rather than the map, in a (sometimes subconscious) effort to protect the last segments of the natural world from being understood scientifically. Why do you think Star Wars was so popular, even among scientists? The mysterious “Force” trumped the most advanced technology in the Galaxy. In real life, technology wins, not the make-believe psychic force. Luke gets hit by a heat-seeking missile before he’s even near the Death Star. He goes boom.

But yes, let’s keep developing cybernetics, synthetic life, space travel, biotechnologies, and advanced robotics. We humans will always be on top, and when we create superintelligence, the magic of market forces and man-machine interfacing will ensure that it embodies our values. No need to panic, be alarmist, apocalyptic, or deluded. Everything will be just fine.

What happened to “Singularitarian” being defined as someone who follows the Singularitarian Principles? Dr. Jones and others who contributed to the IEEE Special Issue on the Singularity mostly use the term to describe Ray Kurzweil and his fans, but for five years between 2000 and 2005 the word applied to a different, yet overlapping group with radically distinct beliefs:

- no fixed timeline
- no argument that all of history has been predeterministically building up to this point
- no argument that technological progress is slowing down, speeding up, moving sideways, or any other such specific claims
- no particular attention given to pre-transhuman intelligence technologies except insofar as they influence when and how superintelligence is created
- central focus on superintelligence as a distinct technological milestone
- acceptance of the point that deliberately designed AGI may exist before neuromorphic AGI
- acceptance of the fact that we might completely blow ourselves up before the Singularity hits
- acceptance of the fact that the first superintelligence might not give a damn about us, and just decide to rearrange our atoms into something more to its liking (like tiling the universe with instantiations of the deity Yog-Sothoth, or something equally ridiculous)
- no magical rosy scenario where human upgrades and AGI research coincidentally fuse seamlessly in a way that happens to completely benefit mankind
- acknowledgment of the Everest-sized challenge of creating AGI that doesn’t eliminate us outright, rather than hand-waving it over with “maintaining an open free-market system for incremental scientific and technological progress, in which each step is subject to market acceptance, will provide the most constructive environment for technology to embody widespread human values” (The Singularity is Near, pg 420). Yeah, right.
- etc…

“Singularitarian” used to mean making minimal assumptions: that superintelligence is possible, it could have a huge impact on the world, and our actions now may influence the final outcome. Now, it comes with a huge set of baggage that I wouldn’t wish on anybody.

Insufficient resources are currently devoted to the mitigation of extinction risks. This is the argument of organizations such as the Lifeboat Foundation, the Singularity Institute, and the Oxford Future of Humanity Institute. In the web edition of the Bulletin of the Atomic Scientists, Sandberg, Matheny, and Cirkovic ask, “How can we reduce the risk of human extinction?” They offer many valuable suggestions, including expansion into space, developing secure bunkers and interdisciplinary research in quantitative risk assessment, probability theory, and technological forecasting. I’m frequently engaged in the latter, most recently by participating in a summer project funded by a grant from SIAI, and in an ongoing basis by encouraging donations to the Lifeboat Foundation.

Fundraising progress has occurred but is largely disappointing. The Future of Humanity Institute does not publish its annual budget, but judging by the staff, I’d guess their budget is about $600,000 a year, funded by James Martin. According to guidestar.org, the Singularity Institute’s revenue in 2006 was about $460,000, and the Lifeboat Foundation’s was far less, about $10,000. All these organizations could use a lot more funding, in the tens of millions.

Why is the situation so poor? My guess is not lack of wealth or will to address the problem, but lack of ideas for low-hanging fruit to pick that helps the cause. If we had better actionable ideas, individuals and foundations would be more inclined to step forward to fund them. Of course, there is also the problem of the “silly factor” regarding extinction risks, and the vagueness of tangibility of such a broad venture.

More reasons: prevention of extinction risk is not a positive goal, it’s a negative goal. People prefer to fund positive goals. Another possible reason is that many philanthropists are on the older side, and inclined to worry more about their own demise than the hypothetical extinction of the human race.

Another challenge is that the most prominent scientists that fear human extinction, Martin Rees and Stephen Hawking, haven’t done enough to address the problem. Hawking encourages development of space, but what about other measures? Little activity there. Same with Sir Rees. After a blast of activity around his recent book, little else follows. An absence of specific initiatives and public proclamations of support. Little of the entrepreneurial spirit we see in business.

My preferred initiative to counteract extinction risk is Friendly AI research, but I know that this idea is not popular to everyone in the risk mitigation community. How about research into self-sufficient closed systems, human intelligence enhancement, building safeguards into gene synthesis equipment, or pursuing other avenues that Sandberg et al present in their recent article? Perhaps a prominent scientist needs to pick one of these ideas and ask their wealthy friends to fund them. After all, it’s only our future.

Matt Mahoney has written “A Model for Recursively Self-Improving Programs”.

One of the interesting challenges in self-improving AI is that current decision theory models lack features for helping decision makers change the way they make decisions, or incorporate models of themselves into decision-making.

The Singularity Summit 2008 will be at Montgomery Theater in San Jose on Saturday, October 25th. Registration is $350 before September 30th, $500 after. If you’re attending, I’ll see you there.

All living things on this planet come into existence through a self-copying process known as reproduction. In sexually reproducing organisms, the copy isn’t exactly the same as what created it, but it does bear some similarity. In asexually reproducing organisms, the progeny is more or less a clone of what made it.

Our planet is kept in relative balance by millions of years of the mutual push-and-pull of competing organisms, organisms which have adapted to cope with each other. Also helpful is that organic proteins used by every organism in the Kingdom of Life have similar material properties relative to inorganic materials, such as say, steel or fullerene. So the incremental evolution of an organism that can overrun the Kingdom of Life with superior properties is unlikely.

However, the phenomenon has occurred at least once, in a limited context. After the Permian-Triassic extinction, 251 million years ago, about 70% of terrestrial vertebrates, including most the giant amphibians, went extinct. Most vegetation was completely wiped out, with rivers changing from a meandering to braided structure. However, one species rose above the destruction and became dominant: Lystrosaurus, a pig-sized animal with a shovel-shaped skull and massive forelimbs for burrowing. In some fossil beds, the animals makes up 95% of all vertebrate fauna. It is the only time that a single animal has dominated the world to such an extent.

The reasons for Lystrosaurus‘ survival are uncertain: some point to its barrel-shaped chest, which probably contained massive lungs suitable for extracting oxygen from the poisoned atmosphere. Whatever the case, it was the complete winner of the time by evolutionary standards, and dominated terrestrial faunas for millions of years in the early Triassic, many hundreds of times longer than human history.

If a single organism could dominate the planet to such an extent then, due to mere evolutionary chance, then in the future could we engineer organisms that can overcome the biosphere during an episode of weakness, or even at its strongest? I do think it’s possible, but the organism would either need to be smarter-than-human or reproduce more quickly than we can manage. A microbe that exploits new non-biological materials for its defensive or offensive strategies might do the trick, or perhaps Artificial Intelligence with the ability to trick humans into doing its bidding.

A reproductive threat could even come from humans alone. If scientists could find a way to accelerate the process of human maturation, a single country with the technology could overwhelm others in a historically short timeframe. This possibility seems to be frequently overlooked in discussions regarding genetic modification, which focus on mental or physical enhancement rather than acceleration of pre-existing processes like reproduction.

The current tenuous balance, where everyone reproduces at pretty much the same speed, is likely to be toppled this century by new technologies that accelerate reproductive speeds for microbes, plants, and animals, including humans. Some people have worried about overpopulation due to life extension, but to me, modifying the exponent is a lot more significant than extending the lifespan of each individual member in the self-replicating set.

I present this idea as a counterpoint to a frequent transhumanist refrain about “progress! forward! ho!” that seems to wholeheartedly embrace every conceivable enhancement technology without considering the encyclopedia-sized list of potential downsides that we must anticipate and address in advance.

When selling a product, it’s useful to do two things: flatter the audience, and convince them that they need what you’re selling.

Transhumanism is a hard sell because it is neither flattering to humanity nor needed by everyone. Here I mean transhumanism in the sense of wanting to eventually modify oneself substantially rather than just considering the issues abstractly.

Transhumanism is not flattering to humanity because its whole point is that humanity isn’t the be-all, end-all of existence. Not the end of the road. This contradicts thousands of years of Homo sapiens-obsessed, Homo sapiens-centric theological and social teachings and habits. This obviously can marginalize transhumanism: how can you convince someone that they should want to change when they identify humanness with the very foundation of their being?

Two: it’s not imminently necessary to modify our brains and bodies. (Unless we want to live longer than a century or so, and many people don’t.) Human existence isn’t so terrible, at least not if you live in a developed country, where things are pretty damn good by ancestral standards. A friend of mine argues that daily life now is probably at least half as good as human life can possibly be, in terms of subjective happiness, even given superabundance. I’m not entirely sure about that, but the fact that it’s even plausible demonstrates that radical expansion of our technological powers isn’t the first thing that comes to mind as necessary to most folks. They identify it with only incremental improvements to human happiness.

The real benefits would derive, I think, from throwing out the whole Darwinian structure of pain and pleasure tied to ancestral correlates of fitness and replacing it with something more reasonable and customized to the needs and desires of the user. But communicating this radical possibility is substantially more difficult than focusing on the foothills of self-modification — prosthetics, psychopharmacology, etc. The result is some degree of fragmentation in the transhumanist community — some transhumanists that think transhumanism is about prosthetics, only moreso, and others that are envisioning a complete restructuring of the human organism based on new foundational principles.

There is also conflict with respect to interacting with interested dabblers on the fringes: the prosthetic crowd would prefer to offer their vision, because it’s more palatable and supported by near-term demonstrable advances, while the restructuring crowd offers their own which they consider more philosophically significant and deep. It can be hard to tell who is who, but a 30-minute conversation over a beer or two is generally enough to figure out where someone is coming from.

I am working on a project that requires references on the comparative difficulty of neuromorphic AI (human brain-simulating, a la Blue Brain) vs. non-neuromorphic AI (not a simulation of the brain). As the terms used to discuss these issues are not always standard, and are sometimes even made up on the spot, locating references via conventional search is not easy. Do any come to mind?

Papers or books preferred; blog or forum posts can’t really be referenced.

Transhumanism, as it stands today, is a philosophy that emphasizes the future of human enhancement and the risks and opportunities presented when one species (Homo sapiens sapiens) differentiates into millions or billions of new species through technological self-transformation. Transhumanists argue that unless we blow ourselves up, this differentiation is very likely to occur, whether we like it or not. I should also emphasize that transhumanists are not unequivocally in favor of every possible enhancement pathway, and in fact may be more concerned about the potential downsides of these technologies than the upsides. Like me, for instance.

What will transhumanism be in the future? Well, if humanity survives and develops technologies that people can use to profoundly modify their bodies, including their mental and physical capacities, then transhumanism will not be a “futurist philosophy”, but merely everyday life. It will be hard not to care about the risks and benefits of self-modification technologies at that point, because they’ll be directing everything around us and the course of world history. Everyone will become de facto transhumanists, so the term will lose much of its present meaning. Do people that use computers call themselves “computerists”? No, but that’s what many of us are, whether we explicitly recognize it or not.

Transhumanists argue that we need a head start — why start caring about these issues when they’re already coming full force, and everyone is more or less forced to care about them? We can start now, analyzing the possibilities and deploying strategies to increase the probability that things go well and our species doesn’t go extinct. Some transhumans may not be so nice, and there’ll need to be systems in place to ensure that these individuals are kept in check and prevented from acquiring too much power. Traditional political banter is only peripherally relevant to this task, because politics as we know it is predicated on an all-human society with human motivations, human intelligence, human habits, and human levels of capability. We have to do something far more difficult — build a model of a world unlike anything else that has existed before. All of history has only contained one truly intelligent species — us. When we’re confronted with billions of new intelligent species, how will we react? Will those that had the foresight to prepare in advance be able to soften the technological torrent, bend it in a way that is beneficial both to humans and transhumans? I do think so, and that’s why these investigations are extremely important now.