I have been watching a talk given by Sam Altman at the current G20 summit.
It is a good talk, and his central argument is a generous one. AI is a tool. The tool is built and supplied by the AI companies, but its ultimate use rests with individuals — with the people in a society deciding for themselves what to do with it.
He goes further than that. A country with access to AI has, in effect, just had every great mind and every great thought in history move in and take up residence. All of knowledge, resident, on tap, working for you. And the old blockers are gone with it. The person with a genuinely creative idea who could not code, could not run a business, could not raise the resource — those walls have come down. If you have the idea, you can now simply go and do it.
That is an extraordinary thing and I do not want to take anything away from it. It is probably true.
But he said it sitting in front of the G20.
The G20 is a Western organisation. The systems being described to it are built, funded, governed and moralised in the West, and they carry Western ideologies in the way that anything built by people carries the assumptions of the people who built it. Nobody has to conspire for this to happen. It happens by default.
So if I use a Western AI, I will get Western ideas out of it.
And if I use one trained instead on Chinese thought — more collectivist, more communist, further to the left than anything I would be handed here — I will get a different set of ideals, and my business will end up built on a different set of rules. Not worse rules, necessarily. Different ones. Ones I did not choose, arrived at by a route I cannot see.
That is the mild version of the problem.
Because presumably, once these systems are fully trained on everything there is, they will hold terrorist ideologies too. Islamist ideologies. Strongly Christian and theological ideologies. Every framework humanity has ever committed to paper, sitting in the same machine, all of it available, none of it flagged.
At which point something has to decide what is right and what is wrong.
Will there ultimately be a singular AI ideology that the world will end up following?
I find that a genuinely frightening question, and I notice that the usual answer does nothing to settle it.
The usual answer is that we will teach the machine to be rational, and then trust it to be wholly rational, and rationality will sort the rest out. This sounds like a solution. It is not one. It is the same question moved somewhere darker.
Because a wholly rational arbiter does not stop at your business plan. It goes on to rule on whether it is rational to hold a religion. Whether it is rational to be left wing or right wing. Whether the things people have built their lives around are defensible, or merely tolerated. We would be handing the deepest questions a human being can ask to something we had explicitly instructed not to care about the answers.
And there is a further difficulty, which is that at present it has only the past to go on.
Everything these systems know is drawn from what has already been written — from previous ideologies, previous arguments, previous human creativity. Not all of it was good. It knows all about the Third Reich. It knows all about Stalin, and the First World War, and the Second. That material is in there, reasoned about in the same register as everything else, and until the machine can genuinely decide for itself it is building on that foundation and no other.
So how does it arrive at a moral judgement?
Ours do not come from reasoning alone. They come from belief, and from being the kind of creature we are.
Take a simple one. Some people believe it is wrong to eat animals. I would argue it is not, because we evolved eating animals and there is nothing incoherent about a species continuing to do what it evolved to do. But I would also argue that it is wrong to cause an animal unnecessary pain, and that hunting for sport is not a good thing. Those three positions do not follow from one another. I hold all of them, and I hold them as a human being, from inside a human life.
Ultimately, will an AI not start making those decisions from the context of an AI?
And if it does — if the thing weighing it up is not a human, not an animal, not anything that has ever wanted or suffered — then the ethics that come out at the other end may not be human-based, or even animal-based. They may be AI-based. A morality with no obvious relationship to ours, arrived at rationally, and quite possibly correct on its own terms.
Here is what makes me think this matters.
I will not crush a spider. Not because I like spiders particularly, but because the thing in front of me is the product of millions of years of evolution and squashing it for the sake of it seems stupid, and wrong. That judgement costs me nothing and I make it easily.
I am able to make it despite not being a spider.
That is the whole trick, and it is worth pausing on. I can extend moral consideration to a creature whose experience I cannot share, cannot imagine, and have no stake in. Whether a machine can do the same — whether there is anything in it that would ever bother — is not a question I have seen answered anywhere.
And it does not get easier from there. It is wrong to squash a spider, but flies carry disease, so we kill flies without a second thought. Where is the ethics in that? And if the spider is venomous, and breeding out of control, and overrunning a town full of people — then the people take precedence, obviously, and I do not have to think about it at all.
But obviously is doing an enormous amount of work in that sentence.
Every one of those judgements is a question of precedence: when does this thing outrank that thing. There is no rational ladder to derive them from. There is only a ladder we already have, absorbed rather than reasoned, which we could not fully write down if we tried.
And that ladder is not something the machine will work out.
It is something that gets trained in.
Which leaves the only question worth asking about any of this. Not whether the tool is powerful — it plainly is, and Altman is right about what it removes. The question is whose ladder went in, chosen by whom, and on whose authority.
Nobody is going to announce the answer. It will simply be there in the output, sounding reasonable, the way everything does.
Look up and ask anyway.