Should the AI-powered drones of the future have a licence to kill? The question is becoming ever more pressing as governments and the defence industry acknowledge that drone systems will play an increasingly crucial role in future warfare.

With drones being deployed in huge numbers in the Ukraine war and AI being used to assist bombing missions in the Iran conflict, there is an expectation among some observers that weapons will have to operate with increased operational autonomy, which means they will need something approximating a moral framework.

Last year Mustafa Suleyman, chief executive of Microsoft’s AI arm and a co-founder of the UK-based DeepMind, was unequivocal about the issue of machines making moral decisions. He said: “AIs cannot be people – or moral beings.”

David Omand, the former head of the UK spy agency, GCHQ, has told the Guardian he believes AI can create a “moral” configuration for unmanned weapons, while the UK armed forces minister, Al Carns, told the Financial Times recently there must be an option to “take the human out of the loop” in decision-making. This brings into focus the ethical and technological challenge of whether morality can be programmed into an autonomous weapons system.

Zee Talat, an academic specialising in machine learning at the University of Edinburgh’s school of informatics, argues that large language models – the technology that underpins modern generative AI systems such as chatbots – are fundamentally incapable of moral decision-making.

AI systems are trained on vast amounts of data that allow them to build a probabilistic model of what is the next most likely word or sentence in a sequence. This is not how humans make moral and ethical decisions, says Talat, even if the model has ingested all the philosophical tracts known to humanity.

“If you have a machine that’s probabilistic by nature it will veer towards the most likely answer in a situation. Do we think that morality follows probabilistic notions?” says Talat, who has recently co-authored a paper that argues ethical evaluation is an “open-ended, debate-based, sociopolitical process”, which AI cannot keep up with.

Andrew Rogoyski, of the Institute for People-Centred AI at the University of Surrey, says AI systems have become much more sophisticated since the arrival of ChatGPT in 2022 – as the emergence of so-called “reasoning” models shows. Nonetheless, can they replicate the nuance of moral decision-making?

“Morality is deeply complex, contested, culturally shaped, and something most humans never fully resolve, even for themselves,” he says. “Perhaps the real question is whether we understand morality well enough to codify it. Until we do, we cannot expect machines to embody something we ourselves cannot clearly articulate.”

This leads to the question of whether it is possible to have a universally recognised moral code for autonomous weapons. Programming morality into drones is problematic, according to Jessica Dorsey, an assistant professor of international law at Utrecht University in the Netherlands. Her concerns include determining whose morality the drone is following, a difficult process given the United Nations is still trying to achieve a global consensus on autonomous weapons governance.

Read More: Can autonomous AI-powered killer drones take morality onboard?