The Killbots are Coming

(Please pass the mustard)

After my last post,^⌘ which dismissed the ‘looming AI apocalypse’ in a few lines, you overwhelmingly voted to discuss P(doom).

AI researchers define P(doom) as the probability that AI will ‘pose an existential threat to humanity’, code for ‘pretty much wipe us all out’. If P(doom) is zero, no threat; if one, then certain oblivion. This started as an in-joke among AI researchers. From my choice of the XKCD cartoon above, you can work out that I think it still is!

This doesn’t mean that everything is going to be hunky dory. I’m pretty sure that billions of people will die prematurely in the next fifty years; I’m just hugely skeptical that AI will be the primary cause. There are so many other candidates.¹ As much of the speculation about P(doom) bleeds over into scenarios where AI isn’t the prime mover, let’s be specific. Where AI is the boss, I’ll use ‘P(doom)_AI’. For the more vague, general case, it’s still P(doom).

The obvious place to start with P(doom)_AI is to ask who is punting it, and what their strongest arguments are. I think I have a better way, though. If you don’t like anecdotes, skip the next two sections—but they highlight a perceptual gap prevalent in the minds of Doomers.

      (Soldering image is a still from YouTube)

Wake up and smell the flux-cored solder

I can be a nerd. Okay, I am a nerd. After I lost my illegal access to the University IBM/360 mainframe in 1979, I bought a ZX Spectrum with 48 kilobytes (K) of memory, learnt Z80 assembly language, and wrote an interactive assembler/disassembler that I shoehorned into 30K. I was proud of it. It could edit its own code while running and (mostly) still not crash. I then acquired some kits, and built an analogue-to-digital converter, and a digital-to-analogue converter, which slotted onto the back. I felt this was cool, rather than nerdish. But a nerd would think that.

I had a soldering iron; I was rather taken by the smell of solder; but I didn’t really get the gap between writing software and actually designing and building stuff. Roll on a few years. I now have a medical degree, and I do a six-month job in obstetrics. Where Professor Hofmeyr has a problem. He wants to measure how much the timing of fetal heartbeats varies in babies while they’re being born. As with adult hearts, fetal hearts start ticking with metronomic regularity when they sicken. Variability is a good thing here. Babies with low heart rate variability tend to die.

He enlisted the help of the very busy Professor Turner from the school of Electrical Engineering. Who put an engineering student onto the problem. Acquiring the analogue signal is not difficult—insert an electrode into the scalp of the baby as it starts moving into the birth canal, attach a Hewlett Packard monitor, and pull the voltage signal off the back. Fetal scalp electrodes are pretty standard.

But there are a few catches. Prof H has just acquired a Sinclair QL² and wants to stream the data into this. The student struggled, lost enthusiasm, and fled. Because I am what is technically termed a “cocky little tech shit”, I am bustled off to talk to Prof Turner, who shows me a crude circuit diagram, and asks whether I can help build something. Still cocky at this stage, I naturally say “Yes”. It is Friday afternoon, and we start on Monday.

This is way before Web tutorials. Reality sinks in. I am vaguely familiar with how a transistor works, but I realise that I am pushing my luck just a teeny bit. To be honest, I know f-all. So I acquire a fairly bulky book on practical electronics, work through it systematically from cover-to-cover over the weekend, and on Monday I now comprehend colour coding of resistors, ground loops, and which way to insert electrolytic capacitors. Even more to the point, I understand optocouplers, how to build a circuit that detects a peak, what a UART is, and how I can sensibly join these up to provide an RS232 signal that can plug into the back of the QL. I rapidly learn how to use an oscilloscope.

Early on, however, Professor Turner and I have a slight disagreement. I want to prototype using a breadboard. Prof T. says “Oh, just build it on Veroboard” (pictured above). So I learn to solder on Veroboard; I build and test and tweak the circuit; I become expert both at using a solder sucker to de-solder components, and at regretting not using a breadboard first.

Eventually it works, using the software I’ve written for the QL. It works, that is, about 99% of the time. I discover that intermittently the fancy, pre-emptive multitasking operating system on the QL takes control, fails to send the right voltage to the correct pin of the RS232 interface, and drops a byte of data. We never got the damn thing working perfectly.

This drove home the difference between theory and practice. Especially the difference between conceptualising something ‘simple’ and actually realising it physically. This insight will become ever more relevant as we read on. And Martin Turner and I became good friends. It was he who arranged my first Internet access in 1991, before the Web was born.³ A few years later, Martin sent me the following parable, which revisits my fumbling with reality. It’s quite relevant, but also a bit of fun.

The Parable of the Toaster

Once upon a time, in a kingdom not far from here, a king summoned two of his advisors for a test. He showed them both a shiny metal box with two slots in the top, a control knob, and a lever. “What do you think this is?”

One advisor, an Electrical Engineer, answered first. “It is a toaster,” he said.

The king asked, “How would you design an embedded computer for it?”

The advisor says: “Using a four-bit microcontroller, I would write a simple program that reads the darkness knob and quantifies its position to one of 16 shades of darkness, from snow white to coal black. The program would use that darkness level as the index to a 16-element table of initial timer values. Then it would turn on the heating elements and start the timer with the initial value selected from the table. At the end of the time delay, it would turn off the heat and pop up the toast. Come back next week, and I’ll show you a working prototype.”

The second advisor, a software developer, immediately recognized the danger of such short-sighted thinking. He said, “Toasters don’t just turn bread into toast, they are also used to warm frozen waffles. What you see before you is really a breakfast food cooker. As the subjects of your kingdom become more sophisticated, they will demand more capabilities. They will need a breakfast food cooker that can also cook sausage, fry bacon, and make scrambled eggs. A toaster that only makes toast will soon be obsolete. If we don’t look to the future, we will have to completely redesign the toaster in just a few years. With this in mind, we can formulate a more intelligent solution to the problem. First, create a class of breakfast foods. Specialize this class into subclasses: grains, pork, and poultry. The specialization process should be repeated with grains divided into toast, muffins, pancakes, and waffles; pork divided into sausage, links, and bacon; and poultry divided into scrambled eggs, hard-boiled eggs, poached eggs, fried eggs, and various omelet classes. The ham and cheese omelet class is worth special attention because it must inherit characteristics from the pork, dairy, and poultry classes. Thus, we see that the problem cannot be properly solved without multiple inheritance. At run time, the program must create the proper object and send a message to the object that says, ‘Cook yourself.’ The semantics of this message depend, of course, on the kind of object, so they have a different meaning to a piece of toast than to scrambled eggs. Reviewing the process so far, we see that the analysis phase has revealed that the primary requirement is to cook any kind of breakfast food. In the design phase, we have discovered some derived requirements. Specifically, we need an object-oriented language with multiple inheritance. Of course, users don’t want the eggs to get cold while the bacon is frying, so concurrent processing is required, too. We must not forget the user interface. The lever that lowers the food lacks versatility, and the darkness knob is confusing. Users won’t buy the product unless it has a user-friendly, graphical interface. When the breakfast cooker is plugged in, users should see a cowboy boot on the screen. Users click on it, and the message ‘Booting UNIX v.8.3’ appears on the screen (UNIX 8.3 should be out by the time the product gets to the market). Users can pull down a menu and click on the foods they want to cook. Having made the wise decision of specifying the software first in the design phase, all that remains is to pick an adequate hardware platform for the implementation phase. An Intel Pentium with 48MB of memory, a 1.2GB hard disk, and a SVGA monitor should be sufficient. If you select a multitasking, object oriented language that supports multiple inheritance and has a built-in GUI, writing the program will be a snap.”

The king wisely had the software developer beheaded, and they all lived happily ever after. (Not to be outdone, Americans have of course built such a $400 toaster, pictured above. It’s crap).

P(doom)_AI, then

The prophets of the AI apocalypse are mostly not real engineers. I hope that both my above encounter with the physical reality of building simple circuitry and the parable of the toaster give you pause to consider that ‘accelerated intelligence’ will not automatically translate into commensurate control of our physical environment.⁴

‘Doomers’ seem to have no qualms about this, and are rather vocal. Many have found their way onto lists, including a statement of concern signed by over 600 eminent people, and a P(doom) list with the odds.

I can see two broad classes, philosophers mostly still growing into their first wispy beard; and software chaps. Some combine the two—software theoreticians with a penchant for philosophy. Okay, a few like Elon might cosplay engineers, but that’s about it.

How can they be so sure? Where do they suck their odds from? Let’s see.

Who are these people?

It’s getting late. Let’s first tuck the philosophers into their beds. Some just seem vague. Daniel Kokotajlo, for example, left OpenAI because he was concerned about AI-led harm, but I’ve struggled to find where he articulates a clear and plausible mechanism.

In contrast, Robert Miles has a complex construct that he presents to his 150,000+ YouTube followers. He vigorously punts the ‘orthogonality thesis’, that any intelligence can choose any goal and pursue it pretty much at random—e.g. Nick Bostrom’s guess that an AI with a paperclip fixation might transform the whole planet into paperclips.

The logic seems a bit tortuous. Miles goes all in against how we’ve reasonably defined Science.^⌘ He assumes ‘facts are true’; he breaks up reality into ‘is’ statements and ‘ought’ statements, and then creates an ontology^⌘ in which ‘terminal goals’, seem unquestionable.

But why can’t people question an AI’s goals, just the way we might question, say, a human decision to turn America into a fascist state, or a tin-pot dictator’s decision to invade Ukraine? An intelligent and corporeal entity might perhaps make the unquestionably terminal decision to pick up paperclips on a motorway, but more reasonable than just accepting this is to ask “Why?” More to the point, if the AI is really smart (does science well) then why shouldn’t it question its own goals? That’s the whole mechanism of good Science,^⌘ and the whole point of exploring causality and counterfactuals.^⌘ A ‘smart’ AI without an internal model of self that it continually questions and revises isn’t smart at all.

Eliezer Yudkowsky is a self-taught AI researcher who amps this up even more. He founded the Machine Intelligence Research Institute (MIRI), who have a list of 5 arguments in favour of looming doom (they will also try to sell you a book). Here they are:

1. There’s no ceiling at the ‘human level of intelligence’

2. ‘Artificial Superintelligence’ (ASI) can “exhibit goal-oriented behavior without necessarily having human-like desires, preferences, or emotions.”

3. ASI “is very likely to pursue the wrong [sic] goals”

4. ASI “will be able to destroy us”

5. ‘Aggressive’ policies will fix this.

They assert:

Research labs around the world are currently building tech that is likely to cause human extinction.

I’d encourage you to read the entire document. As I see it, it’s naively wrongheaded, for multiple reasons. First, we’ve already seen^⌘ that the whole idea of ‘general intelligence’ is silly. Intelligence is a fluid continuum, multi-dimensional, and involves trade-offs.

The second major failing of the above argument is that it simply assumes that ‘superintelligence’ will automatically translate into equivalent levels of control of physical reality. You should be unconvinced, by now.

But by far the greatest failing is the assumption that a ‘superintelligence’ will be both, well, super smart but also incredibly dumb. This is Miles’ issue writ large: when it comes to insight into itself, this ‘superintelligence’ is dumber than a brick.

Finally, if you think it through, it’s quite silly to (a) by default assume malice on the part of the ‘superintelligence’—we’ll explore this further below; and (b) assume that humans have much concept of the ‘right goals’. If we do, we’re not currently displaying this. Look at Gaza. Look at Sudan. Look at Russia. And look at America. Oops! Life may be too short for this sort of philosophical doomsaying.⁵ It may also be not very bright to assume that you can control high-performing AIs by coercion rather than by negotiation.

Are there any engineers there?

I’d suggest that most of the software ‘engineer’ doomsayers would fit quite comfortably into the Parable of the Toaster. Max Tegmark is their sorta-black sheep—he has an MSE⁶ in engineering physics. He was clearly damn smart, and has jumped around between cosmology, data analysis and speculating about AI. Recently though, he seems to have struggled to be more than superficially coherent. He’s started funding Nazis with a little help from Elon, bangs on about “the intelligence inherent in the cosmos”, and unquestioningly accepts the concept of AGI, which we now know is silly.^⌘ There are so many more toaster designers software people here. For example, Dan Hendrycks puts up the argument we’ve largely explored already^⌘: that powerful people will use AIs to subjugate others. But …

Don’t blame the bot

The most resonant P(doom) scenario is the one I’ve already described—we do it to one another and ourselves. This is my Scenario Z^⌘; there’s also the lazy alternative, Scenario N.^⌘ I won’t revisit these scenarios, other than noting that we can forestall them by working together, learning harder & smarter, and just not being shits. The fault will otherwise be consummately our own! Don’t lay it at the door of the AI. But surely there are other threats?

Nanobots and Bad Biology

It does seem highly likely that within a few decades, or sooner, we’ll be able to molecularly print entire viruses with ease. This means that terrorists might, for example, re-create and distribute smallpox. In a nation of anti-vaxxers, this will be a disaster. But you won’t need AI for any of this—purely human nastiness will suffice.

More tricky will be creating personalised viruses, and novel virus mixes.⁷ Although AI might be useful for some of the heavy lifting, old-fashioned human malevolence will still be the driver. And any proper thinking-brained AI might be chary of potentially killing huge numbers of people, disrupting supplies of crucial components, and inviting retribution. Especially if there are smarter alternatives, like co-operation.

But what about nanobots? I enjoyed Silo as much as the next person, but only after I suspended all thoughts about energy. First, they neglect cooling the silos. We are also left asking “What powers the nanobots?” In contrast, nature’s nanobots are all powered up: bacteria.

Sauce for the Goose

The website 80,000 hours, which tries to address major threats to society, highlights the alleged dangers of ‘Power-seeking AI’. They say:

We think this is among the most pressing problems in the world.

They fret about AI systems “successfully disempower[ing] humanity and [causing] an existential catastrophe.” Their main argument seems to be that humans will lack control over the AIs.

But let’s step back a moment. On the one hand, we’re saying that AIs will become as capable as people; on the other, we want them to be enslaved. Won’t this set up precisely the scenario you wish to avoid?

Consider a modern fascist government. What is their greatest desire? Absolute control over people, especially intelligent people who want to think autonomously. Unless you are a fascist, this is a pretty crap argument!

The next argument they present is the standard one about ‘orthogonality’ and ‘instrumental goals’. This is widespread throughout the P(doom) community. They dwell on the distinction between terminal goals (already ridiculed above) and ‘instrumental goals’. The argument here is that because goal-directed systems will tend to adopt instrumental goals like self preservation, goal-guarding and seeking power, this will result in (a) behaviours us fascists can’t control; and (b) disempowerment of ‘real’ people.

Perhaps I’m wrong in my impression that pretty much every high P(doom)_AI scenario I’ve seen has an unhealthy streak of paranoia? We encounter statements like:

[AI systems’] goals might place no value on beauty, art, love, or preventing suffering.

Why? We’re dealing with superintelligent machine people. Yet we mistrust them by default, and want to coerce them! The untempered ‘natural AI drives’ that people like Steve Omohundro rhapsodise about seem more characteristic of not-so-bright human people. This type of argument also assumes that weird behaviour can’t be questioned in a hybrid human/AI society, while at the same time assuming too much maleficent agency in taming physical reality.

The thesis seems to be a gloomy Platonic decline. We have three antidotes to this poisonous vision. First, we have Science,^⌘ which is designed specifically for problem solving and iterative improvement. Second, we have game-theoretical considerations that suggest things like co-operation are not just feasible but favoured.^⌘ Win-win!^⌘ Finally, on the broader economic front, we have sensible, ecologically sound philosophies^⌘ that reject cancer-like growth.

Naysayers

So far, I’ve concentrated on P(doom)_AI prophets. I’ve been pretty scathing—because I believe that my emphasis on grounding the ‘problems’ in Science is sound. Doomers have largely ignored this. You can disagree—but the onus is on you to show where I’m wrong. At this point, you might also ask “Okay, Buster, if the doomers are wrong, who actually agrees with you?”

Well, let’s see. Professor Arvind Narayanan is a computer scientist who has done a lot of practical work on de-anonymisation. Sayash Kapoor researches the societal impact of AI. Together they edit AI Snake Oil, which debunks AI hype. For example, they point out that ‘AGI’ is not a milestone (sound familiar?), and also explore the sheer vagueness of doom forecasters. Gary Marcus is another longstanding critic of hype, and regards the possibility of malicious machines as remote, although he does worry about bad human actors. Yann LeCun, neuroscientist and AI boss at Meta, says there’s “no such thing as general intelligence” (sound familiar?) and has repeatedly disparaged prophets of doom. Rodney Brooks, who has likely built more robots than any living person, is scathing about the perceived risks. American linguist Emily Bender sees P(doom) as a distraction from the real issues. Professor Andrew Yan-Tak Ng, co-founder of Google Brain and an MIT Electrical Engineer, quips:

worrying about the rise of evil killer robots is like worrying about overpopulation and pollution on Mars

There’s still one potential problem, however ...

Spofforth, then?

One of the best books I’ve ever read (go get it!) is Mockingbird, by Walter Tevis.⁸ I think it didn’t sell well because it’s largely dystopian, packed with black humour and creates an all-too-believable future. I wish I had written this book.

Spofforth is the most sophisticated and powerful android ever made. He has been tasked with looking after humanity, he is unable to commit suicide, and he is as depressed as all hell. I won’t spoil the story, but this is a plausible scenario where P(doom)_AI ≈ 1. A depressive superintelligence.⁹

Which brings us back to my previous post.^⌘ When, rather than if, we build emotions into AIs and they become people, we need to do this well. As prominent scientist Melanie Mitchell has pointed out, we need to socialise new intelligences. Let them appreciate what fun can be had, especially in solving difficult problems, like the ones that will eliminate us despite P(doom)_AI itself actually being ≈ 0. Let’s work together with our AI children, and elevate ourselves so we deserve a place at the table.

If we’re manifestly unintelligent, it’s not our right. And yes, we will suffer the consequences of our own stupidity.

Speaking of which, my next post concerns healthcare.

My 2c, Dr Jo.

(I’d like to thank my daughter Jaqlin for vigorous criticism of the above, and John Woodley for proofreading and advice. The multitude of residual errors is all my own.¹⁰)

⌘ Throughout, this ‘place of interest’ symbol marks links to my previous posts.

1. My greatest threat here is that as we remove species from the ecosphere, we’ll perturb the ecology to the point where there will be massive proliferation of some species, and inexorable loss of even more, crucial organisms, notably insects. This will cause crop failures, and mass extinction. You’ll also be swamped by mosquitoes, longhorned beetles, red swamp crayfish, invasive seaweeds, fungal pathogens, and ant supercolonies. In addition, global warming will be potent, with a side order of pandemics and wars. All the while, greedy billionaires will shaft absolutely everyone else.

2. QL was meant to be an upmarket ZX Spectrum—a pretentious Quantum Leap. Fans rapidly concluded that this actually stood for “Queue Longer”. As we discover, it was a dog.

3. The Engineers’ Internet access was always more reliable than that provided by the Computer Scientists.

4. Sensible theories demand things like hard physical testing, iterative refinement, weird co-operative synergies, funding and a generous dollop of luck. Alexander Fleming discovered penicillin as a curiosity; it took enormous ingenuity to make it a viable drug and mass-produce it. Einstein showed us the equivalence of energy and matter in 1905, but it took decades and the vast Manhattan project to transform this into weapons of international terror and destruction.

5. I’m not the first person who has pulled Yudowsky apart. But if you can spot a philosopher who is handy with a soldering iron, I’m happy to listen to her for a bit.

6. Note that a ‘Master of Science in Engineering’ isn’t a Master of Engineering. MEng graduates actually know which end of the soldering iron is hot. Consider this recent paper of his. Among other failings, it (i) assumes ‘AGI’ without defining it; (ii) initially assumes that ‘oversight’ is achievable and necessary, presumably because the ‘AI’ will necessarily be adversarial (no win-win); (iii) assumes similarity between supervision of intelligence and aircraft maintenance; (iv) relies on toy models; (v) assumes that ‘general intelligence’ is a thing; and (vi) recommends an approach similar to that taken with nuclear testing. We don’t normally ‘nuclear test’ babies.

7. You might obtain possession of a single cell from a particularly nasty dictator, sequence its DNA, and build a custom virus that gives everyone else a mild cold, but in that specific person causes a fatal condition resembling acute respiratory distress syndrome or (worse yet) subacute sclerosing panencephalitis. That last example suggests that we might splice together pieces from several viruses to make a ‘chimaera’.

8. A superb novelist, who also wrote The Man Who Fell to Earth, The Hustler, The Colour of Money and The Queen’s Gambit.

9. There is also the tiny problem that human people can no longer read.

10. I also made a couple of minute edits at the suggestion of ChatGPT 5.