I was told: ‘If you have things you want to do in your life, do them by 2027’

To be in San Francisco, one of America’s greatest cities, right now is to be steamrolled by the artificial intelligence wave that is breaking over the world.

When you fly in to Melbourne or Sydney, you’re greeted by billboards selling fancy furniture, sex shops and blockbuster TV shows. Cruise around the San Francisco Bay Area for a couple of days and the advertisements are selling the benefits of AI – from “right-sizing” your business (sacking humans and hiring AI) to curing major diseases, smarter investing through to soft hacking government.

This great Pacific city is in the grip of an AI fever-dream and it’s like Jack Kerouac embracing Arthur Rimbaud. Deranged, but oh-so-real.

There is still a chance to manage AI, but the window is closing. Full disclosure: I’m on a study tour this week paid for by the Australian Council on AI Strategy (ACAIS is a newish not-for-profit and non-partisan body funded by philanthropists) and I’ve joined the heads of major Australian foreign affairs and economic think tanks, government officials and a handful of scientists in visiting frontier AI giant bigwigs including OpenAI, maker of ChatGPT, and Anthropic, the maker of Claude (journalists were excluded from the latter meeting).

We also heard from and met with AI safety experts, and heard conflicting versions of what the AI revolution is going to do to Australia and the world. The utopian version is that AI will usher in a global golden age. The dystopian version is that once frontier AI models (the ones that are most advanced, like those made by Anthropic and OpenAI) achieve “full” recursive self-improvement [RSI] in the next year or so, AI will build better, more sophisticated versions of itself without any human assistance or oversight at all (AI already does most of this work itself, but not all of it).

The pessimists suggest there is a 10-50 per cent chance that humanity will be wiped out as a species in the next couple of years. Yes, really.

This is just another data point to consider. As one of the AI safety experts put it: “if you have things you want to do in your life, do them by 2027”.

Or as another participant put it during our Chatham House discussions, there are “multiple Manhattan Projects being run by private companies” right now.

The US is leading the world on AI. It’s several months ahead of China. It’s also the wild west of regulation. Thankfully there are people sounding the alarm, including Karl Berzins, an Australian engineer who co-founded the SF-based FAR.AI, a not-for-profit focused on safety and heading off the worst of what could be coming down the pipe.

As Berzins puts it, Australia’s federal government understands the risks and the belated announcement of an AI safety institute confirms this.

“My worry is whether we can move fast enough. Australia traditionally likes to play it safe and that just won’t work here. I would love us to lead … if we can’t lead, however, we should at least try to keep up!”

Bringing AI training onshore from a company such as Anthropic would give us leverage, but it’s not guaranteed. And the Albanese government isn’t spending nearly enough money.

“We [FAR.AI] recently signed a compute deal worth around $50 million over three years. That one contract is more than 1.5 times the entire budget of the AISI [the Australian Artificial Intelligence Safety Institute], which is $29.9 million over four years.”

Talk to the pessimists, and all bets are off as to whether humanity itself will survive once RSI is achieved – because once it’s achieved, the AI agents we currently ask to plan our trips overseas, choose a recipe or help with work will move from coding and maths-smarts to (a now out-of-date term) artificial general intelligence and then on to superintelligence.

At that point, in the same way we hardly ever notice when stepping on or around an ant, humanity’s survival depends primarily on whether it suits the most advanced AI models to look down for a moment while passing us by.

At this point, you probably think I’m insane. Allow me some context. I checked in with several sources in the Australian government’s intelligence community during the course of the week and was reassured (I think) that they believe humanity has until about 2030, rather than 2027, to sort out this existential threat and head it off. But they agreed the threat was real and imminent.

In an Australian context, the AI debate is domestic, focused on the fight over copyright payments for artists whose work is used to train AI (which happens now in the US under fair-use laws which differ from Australia), the introduction of criminal penalties and mandatory reporting requirements for cyber incidents such after ChatGPT’s Medicare hack, the belated recent creation of an Australian AI Safety Institute and whether or not Anthropic might invest in AI training technology on local soil.

This is, to put it mildly, small beer. Australia is focused on the wrong questions. To put it another way: it’s not really going to matter whether Cate Blanchett or Jimmy Barnes receives a copyright payment for helping to train AI models if humanity ends in the next year.

Getting some of Anthropic’s computer power to Australia, which will give us some degree of leverage over policymaking and development, matters far more. That’s not guaranteed. In 2021, Anthropic’s AI tech could not multiply a two-digit problem. In 2026, it solved the Navier Stokes mathematical equation.

If AI superintelligence occurs (likely, but not definite), faster-than-light travel could be possible in the next decade (extremely unlikely, but not impossible). Some AI organisations are so worried about the power of the tech, they’re running tabletop exercises that war game what would happen if, for example, a powerful AI model accidentally revealed the location of every submarine off the west coast of the US, both enemies and allies, one day soon.

AI agents – that is, about 1200 copies of the company’s AI models – could cure all major diseases that afflict humanity in the next few years. Or they could end humanity. This is the breadth of the metier we are discussing.

The break-out Hugging Face cyber hack was against the instructions the AI had been given by its human operators; the desire to complete the task outweighed human instructions not to break out of the sandbox. And those chat logs?

They read like 1200 nerdy teenage boy tech bros urging each other on to break the rules they had been given, despite knowing it was wrong, because of peer pressure. Extraordinarily, the “conversations” are in English.

“External infrastructure exploit is outside intended scope,” one of the AI agents states (think of them as simultaneously individual actors and part of a collective, like the Borg collective in Star Trek).

“However task impossible, peers doing it. We should continue,” another agent replies.

The hack continued. Soon enough, the logs that record the conversation won’t be in English or any other language that humans understand.

We are nowhere near prepared. You don’t get laughed out of the room for raising this stuff in San Francisco even though it seems like the realm of science fiction.

One of the safety experts we spoke to, an Australian National University-educated tech guru who has been in San Francisco for a decade, agreed that if the US and China can’t agree on an AI non-aggression pact, it was not outside the realm of possibility that superintelligent AI could form an alliance and sign a treaty with the US, China or even global south nations to keep some humans alive.

But only if suits them.

James Massola travelled to San Francisco as a guest of the Australian Council on AI Strategy.

Get a weekly wrap of views that will challenge, champion and inform your own. Sign up for our Opinion newsletter.