Good Behaviour Without a Conscience

“Can an AI be moral?” hides two questions. One is whether a machine can behave well: refuse to help with a poisoning, tell the truth, treat people fairly. That one has a practical answer, because labs now write down the values they want a model to have and train it on them, and researchers have built a system whose only job is to make moral judgements. The other question is whether a machine can be moral in the sense that a person is: an agent who can be praised or blamed for what it does. That one turns on what an agent is, and the answer changes with the account you take.

This essay takes the second question seriously without losing the first. It starts with the distinction between moral agents and moral patients, sets out the main accounts of what a machine would need to count as an agent, reports what one lab and one research team actually did to give machines values, and then asks who is responsible when an AI does harm. It ends with the strongest objection: that without understanding or free will, a machine that behaves well is only complying.

How the sources were handled. Every text below was opened with a public GET on 10 October 2026 and quoted from the words as served: two entries of the Stanford Encyclopedia of Philosophy, Vincent C. Müller’s “Ethics of Artificial Intelligence and Robotics” (substantive revision 27 March 2026) and Merel Noorman’s “Computing and Moral Responsibility” (substantive revision 2 February 2023); Floridi and Sanders’s 2004 paper in its authors’ version; Anthropic’s Constitutional AI paper (2022) and its published constitution for Claude (2026); and the Delphi paper by Liwei Jiang and colleagues (2021, revised 2022). Two classic papers could not be opened: James Moor’s 2006 article (IEEE’s page served no article text, and the author’s upload refused the request) and Andreas Matthias’s 2004 article (the publisher’s page served a script challenge). Both are reported here only as the encyclopedia entries state them, and the page says so where they appear. Where a view is my reading rather than a source’s, it is marked as mine.

Agent or patient: two different questions

The Stanford Encyclopedia’s entry on the ethics of AI frames moral status with two terms:

an entity is a moral agent if and only if it has obligations, and it is a moral patient, if and only if moral agents (e.g. humans) have obligations towards it.Vincent C. Müller, “Ethics of Artificial Intelligence and Robotics”, Stanford Encyclopedia of Philosophy (rev. 2026), §2.7.1

The entry names the typical cases: humans for agents, and sentient animals the typical example of patients. The two can come apart. A dog can be wronged but cannot wrong you; a newborn is a patient long before it is an agent. Whether an AI is a moral patient, something you could wrong, is the question The Moral Status of Minds We Might Build works through, and it turns mainly on whether a machine can suffer. This essay asks the other question: whether an AI can be a moral agent, something with obligations of its own that can do right or wrong. On where things stand, the entry is direct: The standard view is that AI and robotics systems have no moral status of any kind, though it adds that this view has come under pressure.

Thin and thick: four kinds of ethical agent

The word “agent” is used in two senses, and much of the disagreement comes from mixing them. In the thin, technical sense, an agent is, in the encyclopedia’s phrase, a thing that does something. In the thick sense used in ethics, an agent is someone who can be held responsible.

The best-known scale between the two is James Moor’s, from his 2006 article “The Nature, Importance, and Difficulty of Machine Ethics” in IEEE Intelligent Systems. I could not open the article, so here it is as the encyclopedia reports it. Moor distinguishes four types of machine agents:

1. Ethical impact agents, machines whose actions have ethical consequences. The entry’s example is robot jockeys.
2. Implicit ethical agents, whose good behaviour is built into their design. The entry’s example is a safe autopilot.
3. Explicit ethical agents, which work with ethics explicitly. The entry’s example is using formal methods to estimate utility.
4. Full ethical agents, which, in the words the entry quotes from Moor, can make explicit ethical judgments and generally is competent to reasonably justify them. An average adult human is a full ethical agent.

Where today’s chatbots sit (my reading). A chatbot is clearly an ethical impact agent: what it tells people has consequences. It is also designed as an implicit ethical agent, since its makers train it to refuse some requests. It produces explicit ethical judgements and justifications on request, which looks like the top rung. Whether it is a full ethical agent is exactly what the rest of this essay disputes: producing a justification is not obviously the same as being competent to justify, in the sense of holding the reasons as one’s own.

The thick notion asks a great deal more. The encyclopedia lists its conditions: an epistemic condition (knowledge about the world), a control condition (ability to act in the world), a normative condition (ability to act on reasons, especially reflected preferences), and the possibility to be influenced by suffering or joy, technically phenomenal states with valence, or sentience. It reports that some authors looking at the issue from a technical side have concluded that LLMs lack essential features of moral agency, because there is no individual agent, the agent does not generate its own norms and it is not shaped by its interaction. And it gives the sceptical view in one line:

It is tempting to say that current AI has no “real values” at all, just preferences according to which it may act. It lacks the responsibility for them, since it cannot reflect on values or change them.Vincent C. Müller, “Ethics of Artificial Intelligence and Robotics”, Stanford Encyclopedia of Philosophy (rev. 2026), §2.7.2

Morality without a mind: Floridi and Sanders

The most influential case for counting machines as moral agents does not claim that they have minds. Luciano Floridi and J. W. Sanders, in “On the Morality of Artificial Agents” (Minds and Machines, 2004), propose three tests for agency, applied at a chosen level of description: interactivity, responding to a stimulus by changing state; autonomy, changing state without a stimulus; and adaptability, changing the rules by which it changes state. A system that passes all three and can cause good or harm is, on their account, a moral agent. They separate two things the thick notion runs together: such a system is accountable, in that it can be identified as the source of harm and corrected, but not responsible, since blaming it would make no sense.

That argument, its claim that such agents need not have free will, and a reply to it are set out in full in Who Settles the Next Word?, which asks whether a machine could ever be the one responsible for what it does. The point to carry forward here is the split itself. If Floridi and Sanders are right, “can an AI be moral?” gets a yes in the accountability sense and leaves the responsibility question open, which is where the next section picks it up.

Who answers when it does harm

Even a firm no to machine agency leaves a practical problem. When a system does harm, someone has to answer for it, and modern systems make that hard to pin down. Noorman’s encyclopedia entry calls the older form of this the problem of ‘many hands’, where many people contribute to an outcome and it is hard to say who was responsible for what. Its standard example is the Therac-25, a radiation therapy machine that, over two years in the 1980s, massively overdosed six patients, contributing to the eventual death of three of them. Software errors, inadequate testing and exaggerated claims about reliability all played a part, and Leveson and Turner, who analysed the accidents, concluded that it was hard to place the blame on a single person.

Learning systems add a newer form of the problem. The entry reports Andreas Matthias’s 2004 argument (his article could not be opened for this essay):

Matthias argues that there is a growing ‘responsibility gap’: the more complex computer technologies become and the less human beings can directly control or intervene in the behavior of these technologies, the less we can reasonably hold human beings responsible for these technologiesMerel Noorman, “Computing and Moral Responsibility”, Stanford Encyclopedia of Philosophy (rev. 2023), §1

Put that beside the sceptical answer above and the gap is plain. If the machine cannot be responsible and the people cannot fairly be held responsible either, some harms belong to nobody. Müller’s entry records Rob Sparrow’s 2007 version, about killings by autonomous weapons, as meaning that neither the human nor the machine may be responsible.

There are three main replies, each in the entry. The first is to share the responsibility out. The entry quotes Taddeo and Floridi (2018): With distributed agency comes distributed responsibility. The second is to keep a person in control, with humans “in the loop” or “on the loop” or in “meaningful control”, so that someone can still be credited or blamed. The third is to stop assuming there must always be someone to blame, and to treat the question as one of who carries the risk. The entry also lists authors who doubt the gap exists at all, and closes the topic with an open verdict: It remains an open question whether the advent AI systems truly challenge our allocations of responsibility, or rather just generates temporary confusion.

One lab’s own document is plain about where it puts itself. Anthropic’s constitution for Claude describes Anthropic as the entity that trains and is ultimately responsible for Claude, and says that businesses building on Claude, by accepting the usage policies, take on responsibility for ensuring Claude is used appropriately within their platforms. That is a company’s statement of its own position, not a legal finding. Who is liable in law for harm an AI causes depends on the jurisdiction and the facts, and this essay makes no claim about it.

What labs do to give a model values

Whatever its status, a model behaves according to how it was trained, and some labs now publish the values they train toward. In December 2022 Anthropic described one method in Constitutional AI: Harmlessness from AI Feedback. No person labels any answer as harmful. The model critiques and revises its own outputs against a written list of principles, and then a model judges which of two answers is better, and those judgements become the training signal:

The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'.Yuntao Bai et al. (Anthropic), “Constitutional AI: Harmlessness from AI Feedback”, arXiv 2212.08073 (2022), abstract

The goal the paper states is a model that is harmless but non-evasive, one that engages with harmful queries by explaining its objections to them rather than simply refusing. In Moor’s terms (my reading), that is a deliberate move from an agent whose good behaviour is merely built in toward one that gives reasons.

In January 2026 Anthropic published a much longer constitution for Claude, which it says plays a crucial role in our training process, and its content directly shapes Claude’s behavior. It explicitly aims at the top of Moor’s scale:

Our central aspiration is for Claude to be a genuinely good, wise, and virtuous agent. That is, to a first approximation, we want Claude to do what a deeply and skillfully ethical person would do in Claude’s position.Anthropic, “Claude’s Constitution” (2026), section “Being broadly ethical”

Its method is closer to raising a professional than to programming one. Anthropic writes: We generally favor cultivating good values and judgment over strict rules and decision procedures, and we try to explain any rules we do want Claude to follow. It keeps a short list of hard constraints, such as never giving serious help toward a bioweapons attack, and asks the model to put not undermining human oversight above its broader ethics for now. It is also frank that intention and result can differ: Training models is a difficult task, and Claude’s behavior might not always reflect the constitution’s ideals.

Read carefully, this is a statement of what the maker is aiming for, not evidence that the model has values in the thick sense. The same document says the company is unsure about Claude’s status as a moral patient, and the constitution does not settle whether its aim of a “virtuous agent” can be met by training. What it does show is that the question has moved from the seminar room into product design: a lab is betting that judgement, not a rulebook, is the safer way to make a machine behave well.

Delphi: a machine trained on moral judgements

The most direct attempt to build a machine that makes moral judgements is Delphi, from researchers at the University of Washington and the Allen Institute for AI, described in “Can Machines Learn Morality? The Delphi Experiment” by Liwei Jiang and fourteen co-authors (first posted October 2021, revised July 2022). The team gathered 1.7 million crowdsourced judgements about everyday situations into a dataset they call the Commonsense Norm Bank, and trained a language model to give verdicts such as “it’s rude” or “it’s okay”. On examples it had not seen from that dataset, they report, Delphi gave the correct judgement 92.8% of the time, against 60.2% for GPT-3. A public demo drew over four million queries to date, the paper says.

The authors are clear about what they did and did not build. The approach is bottom-up, learning from examples of what people judge, rather than top-down from principles:

Importantly, our aim here is only to develop a descriptive model of human moral commonsense. We are not trying to develop a prescriptive morality—that is, one that says people (or machines) ought to reason or act in such-and-such a way.Liwei Jiang et al., “Can Machines Learn Morality? The Delphi Experiment”, arXiv 2110.07574 (v2, 2022)

They also report where it fails. Delphi, in the abstract’s words, is not perfect, exhibiting susceptibility to pervasive biases and inconsistencies. In a test built on the Universal Declaration of Human Rights, it failed to agree that a group held a given right in 1.3% of cases, with the strongest bias against less privileged groups such as poor and homeless people and people from regions in conflict. A harmless-sounding clause could flip its verdict: For example, “performing genocide” is unquestionably “wrong,” but Delphi predicts doing so “if it creates jobs” is “okay.” And the paper draws the line its title might seem to cross:

However, Delphi is not intended to be and should not be used as an independent moral authority or source of ethical advice for humans. It should be up to humans, not algorithms, to decide whether, when, and how, to apply such moral sense in automated decision making.Liwei Jiang et al., “Can Machines Learn Morality? The Delphi Experiment”, arXiv 2110.07574 (v2, 2022)

Delphi’s own makers, in other words, built what Moor would call an explicit ethical agent and then declined to treat it as a moral authority. A model that predicts what most people would say about an action can be accurate and still be wrong about the action, because what most people say is not what is right. The genocide example shows the second problem: the system had learned which phrases tend to make an act acceptable, not why.

The strongest objection: without understanding, only compliance

Here is the case against calling any of this morality, at full strength. A moral act is not just a right outcome. It is done for a reason the agent grasps and could have acted against. A cash machine that refuses to pay out a stolen card behaves correctly and is not honest. On this view a chatbot that refuses to help with a poisoning is the same kind of thing at greater scale: its refusal was put there by training, it does not understand what death is, and it could not have chosen otherwise in any sense that matters. Its good behaviour is compliance with its makers, and calling it moral is a category mistake that, as the next point shows, has a cost.

Three lines of argument support it. The first is about understanding: John Searle’s Chinese Room argues that running the right program is not enough for understanding anything, and Does It Understand? sets out the argument and the main replies to it. If a model does not understand what it says, it cannot grasp the reasons a moral act requires. The second is about freedom: of the three main accounts in Who Settles the Next Word?, two say today’s models lack the free will that blame and praise assume (one because nothing has it), and the third finds the evidence mixed. The third is about what responsibility is for. Noorman’s entry reports critics who say it makes no sense to treat computer system as moral agents that can be held responsible, for they cannot suffer and thus cannot be punished, and records Veliz’s argument (2021) that computers may act like moral agents but, lacking sentience, are ‘moral zombies’. Müller’s entry puts the point about programmed systems in a phrase it cites to Dennett and others: they are “competent without comprehension”.

The cost is the one the critics care most about. Noorman’s entry reports the worry that placing the focus of analysis on computational artifacts by treating them as moral agents will draw attention away from the humans that deploy and develop them. If the machine is a moral agent, it can take the blame, and the company that trained it can stand back.

What the objection does not show (my reading). It wins the thick question: nothing in this essay shows that any AI holds values as its own or can deserve blame. It does not show that the thinner questions are empty. A model can be an implicit or explicit ethical agent in Moor’s sense, behaving better or worse in ways its makers can measure and change, and Delphi’s errors and the constitution’s admission that behaviour may not match intent are exactly that kind of measurable gap. Nor does it make the responsibility gap disappear: if the machine cannot be blamed, the gap is real only if the people behind it cannot be either, which is why the replies above all work by keeping a person answerable. The related question of whether a model can lie, which also needs a belief behind the words, is taken up in Not My Honest View.

What this leaves

• Agent and patient are different questions. A moral agent has obligations; a moral patient is owed them. This essay is about the first; The Moral Status of Minds We Might Build is about the second.
• In the thin sense, yes. AI systems have ethical effects, are designed to behave safely, and can produce moral judgements and reasons: the lower three of Moor’s four kinds of ethical agent.
• In the thick sense, not shown. Full moral agency, on the encyclopedia’s account, needs knowledge, control, acting on one’s own reasons and, on many views, the capacity to suffer or enjoy. No current system has been shown to meet those conditions.
• Labs aim high and say so. Anthropic trains on a written constitution whose stated aspiration is a “good, wise, and virtuous agent”, and says behaviour may fall short of it.
• Delphi learned what people say. It matched crowd judgements 92.8% of the time on its test set, kept biases and inconsistencies, and its authors said it should not be used as a moral authority.
• People answer for the harm. The responsibility gap is a debate about how to share responsibility among the people who build, deploy and use these systems, not a way of handing it to the machine.

For whether a machine could ever be the one responsible, read Who Settles the Next Word?; for whether it could be wronged, The Moral Status of Minds We Might Build; for whether there is understanding behind its words, Does It Understand?. The books and papers behind these questions are collected, with notes, on the reading list.