Moral Machines

Wendell Wallach and Colin Allen wrote Moral Machines in the final years of the last decade, before the deep learning boom reshaped expectations of what software could do and well before autonomous weapons became a standing item on the agenda in Geneva. The book was published by Oxford University Press in 2008, with Wallach writing from Yale’s Interdisciplinary Center for Bioethics, where he had spent years running programmes on technology ethics, and Allen writing from Indiana University, where he worked as a philosopher of cognitive science and biology. The pairing matters, because the book belongs to two conversations that had, until then, mostly ignored each other. One was the applied ethics tradition that treated moral philosophy as a practical discipline — the kind of work that produced hospital ethics committees and research-oversight boards. The other was the technical community in artificial intelligence and robotics, which by the mid-2000s was shipping systems that behaved autonomously in open environments without having anything that could reasonably be called moral reasoning onboard. Wallach and Allen set out to persuade both audiences that the overlap between them was already a working problem, not a futurist exercise, and that putting it off would mean deploying ethically consequential machines with no principled design for how they handled consequences.

The central argument is more careful than the title suggests. The authors do not claim that robots will shortly become moral agents in any strong philosophical sense. They introduce a scaffolded idea instead: artificial moral agents, abbreviated AMAs, understood as systems that take moral considerations into account to a degree appropriate to their autonomy and their domain. The thesis is that engineers will have to build these long before any of the metaphysical questions about machine consciousness or genuine understanding are answered, because the operational pressure is already there — in medical devices that triage treatment, in trading systems that act in milliseconds, in battlefield platforms authorised to engage. The task they set themselves is to describe what designing morality into those systems would actually involve, which approaches from philosophy and cognitive science have anything to offer the engineer, and where the hard limits lie. They are explicit that this is a bridge-building book, not a manifesto. The reader is invited to see machine ethics as a tractable, if unfinished, engineering programme with a long philosophical history standing behind it.

The structure follows the shape of that programme. Early chapters work to establish that the question is not premature. The authors walk through examples where software and robotic systems had already produced morally weighty outcomes without any human in the decision loop: automated stock trading, autopilots that handled crisis moments, military systems like the Phalanx close-in weapon that had been authorising its own engagements since the 1980s. They then introduce a two-axis map that becomes the book’s organising diagram. One axis measures a system’s autonomy; the other measures its sensitivity to morally relevant features of its situation. The map yields three bands: operational morality, where the designers have anticipated the ethical dimensions and the machine simply follows the rules it was given; functional morality, where the system has some capacity to evaluate situations against moral criteria it was not explicitly instructed on; and full moral agency, the human case, which the authors treat as a horizon rather than a target. Most of the book sits in the middle band, which is where the engineering choices actually bite.

Within that middle band, Wallach and Allen set out three design strategies and spend chapters interrogating each. The top-down strategy begins with a moral theory and tries to encode it. Isaac Asimov’s three laws of robotics are the obvious reference, and the authors use them in the way philosophers of science use thought experiments — not as a proposal but as a tool for exposing problems. The laws fail not because Asimov was careless, which he wasn’t, but because any rule stated in natural language inherits the open texture of natural language, and because the laws must be ranked whenever they conflict, and the ranking cannot itself be a rule all the way down. The same critique is extended, more respectfully, to utilitarianism and deontology. A consequentialist machine needs a way to enumerate consequences, weight them across stakeholders, and discount uncertainty, all of which are computationally hard and philosophically contested. A Kantian machine needs to be able to recognise maxims and test them for universalisability, which assumes a capacity to represent its own reasoning in a form the test can be applied to. The authors do not reject top-down approaches; they describe the specific places each one bruises against reality.

The bottom-up strategy is the opposite move. Rather than install a theory, let the system develop moral competence through learning, imitation, or evolutionary processes. The authors draw here on virtue ethics in its Aristotelian form — the idea that moral character is a disposition built up by practice rather than a theorem derived from axioms — and on the connectionist and developmental traditions in cognitive science. They are candid that this family of methods has produced some suggestive demonstrations and nothing like a working moral agent. Learning systems trained on moral data inherit the biases of the data; evolutionary approaches produce competences their designers cannot inspect; developmental robotics is still mostly concerned with sensorimotor skills, not with ethical ones. But the authors argue that the bottom-up perspective buys something the top-down approach cannot: a story about how moral sensibility might be grounded in experience, embodiment, and social interaction, rather than imposed from outside. The practical conclusion is a hybrid. Any workable machine ethics will have to combine explicit rules, which give the system a stable spine and a form in which human regulators can inspect its behaviour, with learned sensitivity, which gives it a way to recognise the morally salient features of situations the rule-writers never anticipated.

The examples and case material are where the book earns its subtitle. Wallach and Allen devote substantial space to the medical setting, where ethics-aware software was further along than elsewhere. They examine MedEthEx, the system Michael Anderson, Susan Anderson, and Chris Armen were developing to advise clinicians on bioethical cases by reasoning from a corpus of prima facie duties in the Beauchamp and Childress tradition. They describe the project not as a solved problem but as an existence proof that moral reasoning can be implemented in a working piece of software when the domain is narrow enough and the duties are well-articulated enough. They set this alongside the longer tradition of expert systems in medicine, drawing a line from early rule-based diagnostic tools like MYCIN to the ethics advisors emerging in their own period. The treatment is careful. The authors note that these systems are decision support, not decision makers, and that this distinction matters legally and morally even if it blurs in practice when a busy clinician defers to the recommendation.

Military applications receive comparable attention and comparable restraint. The authors engage directly with Ronald Arkin’s then-emerging work at Georgia Tech on an “ethical governor” — a software architecture intended to sit between a battlefield robot’s target-recognition system and its weapon release, blocking engagements that would violate the laws of armed conflict or the rules of engagement for a given mission. They present Arkin’s position fairly: that human soldiers, under stress, in fear, in the fog of combat, often fail to apply the laws of war correctly, and that a machine that cannot panic and cannot hate might in principle apply them more consistently. They also present the objections, which include the brittleness of target recognition in cluttered environments, the near-impossibility of distinguishing combatants from civilians in irregular warfare, and the moral objection that delegation of killing to a machine corrodes the human responsibility that makes the laws of war meaningful in the first place. The book sits with this tension rather than resolving it, which is one of the reasons it has stayed useful as the policy conversation has moved on.

The book also widens its lens to the ways embodiment, affect, and social cognition contribute to human moral competence. Chapters on emotions, consciousness, and theory of mind step away from the rulebook and ask what else is in play when a human handles a moral situation well. The authors draw on work by Antonio Damasio on the role of somatic markers in judgement, on the social robotics work of Cynthia Breazeal, whose expressive robot Kismet was then the touchstone example of a machine that read and performed emotional cues, and on Alan Turing’s imitation game, which they reframe as a question about whether a system’s moral behaviour is convincing enough to count. These chapters are the book at its most speculative, and the authors say so. They are not predicting that affective computing will produce moral machines; they are arguing that any serious attempt to replicate the human case will have to engage with the fact that human moral competence is not purely cognitive, and that engineers who build only the cognitive layer will find that it does not sit stably on its own.

Reception since 2008 has placed the book in a specific role: the standard-reference starting point for machine ethics, cited by nearly everyone who entered the field afterwards, and in particular the sourcebook that organised the top-down, bottom-up, and hybrid vocabulary that the field now uses by default. Patrick Lin, Keith Abney, and George Bekey’s edited volume Robot Ethics drew heavily on the framework. Arkin’s 2009 Governing Lethal Behavior in Autonomous Robots engaged with Wallach and Allen directly, extending the military case into a full engineering proposal with the governor architecture at its heart. Nick Bostrom’s Superintelligence, which arrived six years later, pulled the conversation toward long-horizon existential risk and alignment theory, where Moral Machines had kept the focus on present and near-term systems; the two books are complementary rather than competing. Pushback has come from several directions. Some philosophers have argued that the AMA framework concedes too much by calling anything a moral agent that lacks consciousness and genuine moral understanding, and that the softer “ethical impact agent” vocabulary proposed by James Moor is more honest. Others, from the critical side of science and technology studies, have argued that the engineering framing obscures the political economy of autonomous systems — that the question is not whether a battlefield robot can be made to apply the laws of war, but who is served by deploying such robots at all. Wallach, in his later Dangerous Master, took the political dimension more directly in hand; Moral Machines itself largely leaves it to the reader.

For a reader coming to AI in warfare today, Moral Machines earns a particular place on the shelf. It is the book to read before the alignment literature and before the laws-of-armed-conflict analyses, because it supplies the conceptual vocabulary that makes those conversations coherent. It is less useful as a guide to current systems. Nothing in the 2008 text anticipates the specific capabilities and failure modes of large language models, and the authors’ treatment of learning systems is pitched at a period when neural networks were still a minor technique rather than the dominant paradigm. The book pairs well with Arkin on military applications, with Peter Asaro on legal and policy questions, with Shannon Vallor’s Technology and the Virtues for a sharper virtue-ethics treatment of the same ground, and with the more recent policy literature on meaningful human control, which can be read as the regulators’ answer to the engineering questions Wallach and Allen laid out.

What has aged well is the diagnostic frame. The two-axis map and the three-strategy taxonomy survive because they do not depend on which technologies happen to be in fashion; they describe where any ethically consequential autonomous system must sit, and how its designers can and cannot proceed. The hybrid conclusion has aged well too: no serious proposal today argues for pure rule-based or pure learning-based moral software, and the division of labour Wallach and Allen sketched is roughly the one working engineers assume. What has dated is the technological furniture. The specific systems discussed — Kismet, MedEthEx, early autopilots, the Phalanx — belong to a pre-smartphone, pre-deep-learning era, and the examples that would carry the same arguments now are different: large language models that produce fluent moral commentary without any clear reasoning underneath, loitering munitions that identify and strike targets at the edge of autonomy, recommendation systems whose moral consequences accumulate statistically rather than in single visible decisions. The scaffolding Wallach and Allen built is the right one to think about those systems with. The furniture needs replacing, and most of the field has been quietly doing so, which is itself a measure of how durable the frame has turned out to be.