Moral Machines

Wendell Wallach and Colin Allen wrote Moral Machines at a moment when the phrase “machine ethics” still sounded, to most readers, like a category error. The book landed in 2008, published by Oxford University Press, and it argued — in patient, non-polemical prose — that the question of whether computers could act ethically had already been overtaken by events. Autonomous systems were being fielded in warfare, deployed on financial markets, embedded in medical decision support, and prototyped for the roads. The engineering was moving faster than the philosophy. Wallach, a bioethicist working out of the Yale Interdisciplinary Center for Bioethics, and Allen, a philosopher of cognitive science then at Indiana University, set out to survey what would be required to give machines something like moral competence, and to ask how much of that project could actually be built.

The book joined a very small conversation. Isaac Asimov’s Three Laws of Robotics had done the popular imagining decades earlier; philosophers such as James Moor had begun to sort out what could count as ethical behaviour in a machine; the Andersons, Michael and Susan Leigh, were assembling the Machine Ethics anthology that Cambridge University Press would publish three years later. But there was no widely-read, systematic treatment of what it would take to engineer moral judgement, and Wallach and Allen wrote to fill that gap. Their book became one of the standard references for the field it helped to define, and its vocabulary — operational versus functional morality, top-down versus bottom-up approaches, the artificial moral agent or AMA — went on to shape a decade of subsequent work.

The central argument is deceptively modest. Wallach and Allen do not claim that machines will soon become full moral agents in the sense that a human being is one. They claim, instead, that the space between a hammer and a saint is very wide, and that machines are already advancing into it. A landmine has no moral standing at all; its designers made every moral choice for it. A driverless car deciding how to distribute risk in a swerve is not a full moral agent either, but it is not a hammer. Somewhere along that continuum, the authors argue, machines acquire what they call functional morality: the capacity to assess morally salient features of a situation and respond to them, without a human in the loop. The book’s project is to describe that continuum honestly, to map the engineering strategies available for climbing it, and to argue that the work of climbing has to begin now, because deployment is already outpacing deliberation.

Around that thesis the book is organised in roughly a dozen chapters that move from framing to method to hard cases. The opening chapters ask why the question matters at all, and answer with a catalogue of near-term systems whose behaviour cannot plausibly be reduced to their operators’ intentions: military drones with target-recognition algorithms, high-frequency trading systems whose collective behaviour has already produced flash crashes and their equivalents, medical expert systems that grade the urgency of a symptom, service robots working in eldercare. The authors then draw the distinction that anchors the rest of the book. Operational morality is what a designer bakes in — safety interlocks, rule sets, prohibitions on certain outputs. Functional morality is what the machine does at runtime when the designer’s rules do not cover the case in front of it. Full moral agency, with consciousness, intention, and the capacity to be held responsible, remains a horizon rather than a target. Most of the book concerns the middle band.

Wallach and Allen then survey the philosophical raw material that engineers might draw on, and they are careful to treat the classical ethical traditions as engineering candidates rather than as truths to be adjudicated. Utilitarianism, in the Benthamite or Millian form, has an obvious appeal for a computational implementation: it treats moral decision as a species of maximisation, and machines are good at maximising. The authors walk through why the appeal is misleading — the intractable measurement problem, the difficulty of specifying the class of affected beings, the vulnerability of an aggregator to being gamed by a sufficiently clever counting scheme. Deontological approaches in the Kantian mould offer rule structures that map more naturally to code, and here the authors discuss Asimov’s Three Laws at length, less as a design proposal than as a case study in why elegant rule sets fail: Asimov’s own stories exist to dramatise the failures. Virtue ethics, which asks not what a rule requires but what a person of good character would do, is harder to reduce to an algorithm, but the authors take it seriously as a model for the developmental approach they describe later.

Having laid out the philosophical material, the book turns to the two families of engineering strategy that give the central chapters their spine. The top-down approach begins with a stated moral theory and tries to implement it. A system with a utilitarian core would compute expected utilities across possible actions and choose the maximum. A Kantian system would test candidate actions against a universalisability procedure. A rule-based system in the Asimov mould would encode a hierarchy of prohibitions and check each action against it. Wallach and Allen are respectful of the appeal but blunt about the record. Every top-down scheme runs into the frame problem in a moral register — the difficulty of enumerating in advance which features of a situation matter, and of specifying the rules with enough precision to survive their edge cases. A utilitarian who counts wrongly, a Kantian whose maxim is described at the wrong level of abstraction, or a rule-follower whose rules were written for a slightly different world all produce recognisable failure modes.

The bottom-up approach reverses the direction. Rather than encode a theory, the engineer builds a system that learns what to do from experience — through reinforcement learning, through evolutionary algorithms, through developmental scaffolding that mimics the way a child acquires moral competence. The authors are clearly interested in this family and give it a sympathetic hearing, but they are equally clear that bottom-up methods buy their flexibility with a loss of transparency. A learned policy cannot easily be audited against a principle; an evolved behaviour cannot easily be justified to someone it harms. Bottom-up systems are also creatures of their training environment in a way that engineers routinely underestimate. Wallach and Allen argue, in the end, for hybrid designs, in which a learned or developmental substrate is bounded and shaped by explicit constraints — the top-down layer functioning as a set of guardrails within which the bottom-up capacities operate. That hybrid recommendation has aged well; it prefigures a great deal of what the machine-learning safety community would go on to work on.

The middle-to-late chapters of the book turn to what the authors call the supra-rational faculties, and this is where Moral Machines sets itself apart from more narrowly computational treatments. Wallach and Allen take seriously the possibility that a machine capable of acting morally in the world would need capacities that a rule engine or a utility maximiser cannot straightforwardly supply. They discuss the role of emotion in moral cognition, drawing on Antonio Damasio’s work on how patients with damage to the ventromedial prefrontal cortex, deprived of the somatic signals that normally accompany decision, lose the capacity to make sensible practical choices even though their abstract reasoning remains intact. They discuss sociability, and the way moral judgement is shaped by the tracking of other agents’ beliefs and intentions — what philosophers call theory of mind. They discuss embodiment, and the possibility that the capacity to feel one’s way through the world, physically, contributes to moral perception in ways a disembodied inference engine cannot replicate. And they raise, without settling, the question of consciousness, and whether a moral agent needs the kind of inner life that permits genuine suffering, empathy, and remorse.

The examples the book uses are chosen to make the abstract concrete. The authors return repeatedly to military systems, which in the mid-2000s were the leading edge of autonomous decision under lethal stakes. The Predator and Reaper drones, the Phalanx close-in weapons system on US Navy ships, the SGR-A1 sentry gun stationed on the Korean demilitarised zone: each is a case in which some portion of the kill chain has already been delegated to a machine, and each is a case in which the delegation is presented as narrower than it is. The book discusses the Roomba and other early service robots as low-stakes analogues that nonetheless raise real questions about liability and consent. It discusses the 1987 Black Monday crash and other episodes of automated market failure as evidence that the effects of unsupervised algorithmic decision can be catastrophic without any single algorithm being at fault. Medical expert systems, from MYCIN onward, are treated as an early test bed for questions of responsibility that will eventually apply to much more capable systems. Autonomous vehicles get a chapter’s worth of attention that reads, with hindsight, as a preview of the debate the DARPA Urban Challenge and then Waymo and Tesla would push into the mainstream.

Wallach and Allen also raise, and take seriously, the question of the moral Turing test — whether one could, in principle, evaluate a machine’s moral competence by comparing its judgements to those of human judges without knowing which was which. They neither endorse nor dismiss the proposal; they use it to sharpen the question of what it is that we would be measuring, and they observe that human moral judgement is itself so variable and so context-dependent that a fair benchmark is not obviously available. Later chapters turn to the question of rights and responsibilities: if a machine acts morally, does it acquire any claim to moral consideration itself, and if it fails, who carries the liability? The authors decline to settle these questions but insist that the legal and ethical infrastructure needs to be built now rather than after a serious accident forces the issue.

The reception in the field was substantial and largely respectful. Peter Singer’s Wired for War, appearing the following year, made overlapping arguments for a general readership; Ronald Arkin’s Governing Lethal Behavior in Autonomous Robots, published in 2009, took the specifically military strand further and proposed an “ethical governor” architecture that Wallach and Allen’s framework helped make legible. The Andersons’ Machine Ethics anthology, published by Cambridge in 2011, cited Moral Machines throughout and cemented the vocabulary. In the decade after publication, as machine learning became the dominant AI paradigm and as autonomous vehicles moved from research demo to public road, the book’s hybrid recommendation and its emphasis on the limits of pure top-down rule sets came to look prescient. Nick Bostrom’s Superintelligence, in 2014, and Stuart Russell’s Human Compatible, in 2019, addressed a more speculative horizon but built on the same conviction that value alignment cannot be an afterthought. Where later authors have pushed back on Wallach and Allen, it has often been to argue that the book underestimates how quickly capability would outrun understanding, and how much of the alignment problem would fall to techniques — reinforcement learning from human feedback, constitutional methods, interpretability research — that were not on the horizon in 2008.

For a reader working through the literature on AI in war today, Moral Machines sits at the foundation. It is the book that established that the question was engineerable rather than merely philosophical, and it is the book whose distinctions — operational, functional, full; top-down, bottom-up, hybrid; the supra-rational faculties — remain in circulation. It pairs naturally with Arkin’s work on the military side, with the Andersons’ anthology for the philosophical texture, and with the later alignment literature for a sense of how far the field has moved. What it does not cover, because it could not, is the specific character of the deep-learning systems that would come to dominate the field: large models whose behaviour is emergent, whose training data is planetary in scale, and whose failure modes are statistical rather than logical. A reader looking for a treatment of prompt injection, reward hacking, or the alignment problem as understood in a frontier lab will need to look elsewhere.

What is likely to age well is the framing. The insistence that machines occupy a moral continuum rather than a binary, the argument that engineering choices now determine what kind of moral agents we will be surrounded by later, and the demand that ethical work proceed alongside technical work rather than trailing it are all more urgent in the era of autonomous drones over Ukraine and near-autonomous cars on public roads than they were when the book was written. What has dated is some of the specific technology — the expert-system examples, the pre-deep-learning assumptions about how machine cognition would develop — and some of the confidence with which the authors treated the harder problem of full moral agency as safely on the horizon. Moral Machines was written before its field had a name; that it can still be read as a working guide, nearly two decades on, is the measure of what Wallach and Allen got right.