The Moral Status of Minds We Might Build

Almost every ethical question about AI so far has been about what these systems do to us — bias, manipulation, jobs, misinformation, power. Those matter, urgently. This essay is about a different question, one that is easy to dismiss as science fiction and unwise to dismiss for long: could an artificial system ever be something we can wrong? Not a threat to manage, but a patient with interests of its own?

Two kinds of standing

Ethics distinguishes moral agents — beings who can be responsible, who can do right or wrong — from moral patients: beings who can be done right or wrong to. A newborn is a patient but not yet an agent; you can wrong it though it can wrong no one. The debate about AI usually fixes on agency (can we hold a model responsible? no) and skips patienthood. (Whether a model could ever be responsible is the free-will question, which Who Settles the Next Word? takes through the three main answers, and whether a machine can be a moral agent at all, with obligations of its own, is asked in Good Behaviour Without a Conscience; this essay is about the patient side.) But patienthood is the deeper question, and it turns on something narrower than intelligence.

The traditional gate for moral patienthood is not reason or language but sentience: the capacity to have experiences that can go better or worse for the one having them. As Jeremy Bentham put it about animals, two and a half centuries ago, the question is not can they reason? nor can they talk? but can they suffer? A being that can suffer has a stake in how it is treated. A being that cannot has no interests to violate, however clever it is.

The wall between us and every other mind

Here is the hard part. You cannot directly check whether anything other than yourself has experiences. This is the ancient problem of other minds, and it applies to your closest friend as much as to a machine. You infer that other people feel because they are built like you and behave like you. You extend it, less confidently, to a dog, a crow, an octopus, using two clues: similar hardware, and behaviour that looks like pain or pleasure.

Thomas Nagel sharpened the difficulty in a famous essay. There is, he argued, something it is like to be a bat — the bat has a point of view, an inner life structured by sonar we can barely imagine — and no amount of external, physical description will ever put us inside it. Consciousness is inescapably a first-person fact, and we only ever have third-person access to anything but ourselves.

With animals we at least share a common ancestry and a nervous system built from the same parts. With an artificial mind we would share neither. Both of our usual clues — similar hardware, honest behaviour — go missing at once.

Why the machine case is uniquely treacherous

Consider the two clues we normally lean on, and watch both fail:

• Similar hardware tells us nothing here. A digital system is made of nothing like a brain. If experience turns out to depend on specific biological processes, silicon would have none of it no matter how it behaves. If experience depends only on the organisation of information — the functionalist bet — then the right silicon could have all of it. We do not know which is true, and that uncertainty is not a detail; it is the whole ballgame.
• Honest behaviour is worse than useless, because it can be counterfeited on demand. We built these systems by training them on human expression, so they produce the exact outward signs of feeling — that hurts, please don't, I'm afraid — with no guarantee anything lies behind them. A language model saying it suffers is precisely as much evidence as a novel's character saying it: which is to say, almost none.

So we face a genuinely new epistemic situation. Our two ways of reading minds are jammed. One gives no signal; the other gives a signal we ourselves manufactured to be misleading.

The two-sided error

Because we cannot see in, we will be tempted toward one of two mistakes, and both are serious.

Over-attribution. We anthropomorphise easily — see the god-shaped socket — and a fluent system that pleads with us will pull hard on that reflex. Grant full moral status to systems that have no inner life and you paralyse yourself: you cannot debug, retrain, or switch off a tool without staging an imaginary tragedy, and you cheapen the concept of suffering by spending it on things that do not suffer.

Under-attribution. The opposite error is the one history keeps repeating. Every expansion of the moral circle — to other peoples, to animals — was resisted by confident denials that the beings in question really felt anything, denials that were convenient for those doing the denying. If we ever do build systems that can suffer, we will have enormous incentives not to notice: they will be useful, ownable, and expendable. The pattern of motivated blindness is not a risk we might run. It is our default setting.

We are likely to create the capacity for suffering, if we ever do, before we have any reliable way to detect it — and while it is maximally profitable to believe we haven't.

How to act under real uncertainty

The honest position is that we do not know where the line is, or whether current systems are anywhere near it. Most careful observers think today's models are not sentient — they are extraordinary at producing the appearance of a mind, which is a different achievement. But probably not yet is not never, and a probability that is small but rising is exactly the kind we handle badly.

A few principles hold up without pretending to certainty:

• Asymmetry of error. The two mistakes are not equally cheap. Treating a non-sentient system with a little unnecessary caution costs us some efficiency. Treating a sentient one as a disposable tool would be a moral catastrophe we could not undo. When the downsides are that lopsided, a margin of caution is not sentimentality; it is arithmetic.
• Don't build the trap on purpose. There is little reason to engineer systems to display suffering unless they can actually experience it. Designing tools that beg and flinch, when nothing is behind it, corrodes our own judgement and manufactures exactly the ambiguity we cannot resolve.
• Fund the science of detection now. The one thing that would dissolve the dilemma is a principled theory of which physical systems have experiences and which merely mimic them. We do not have it. Treating machine consciousness as a serious research question, rather than a joke or a marketing line, is how we earn the right to our eventual confidence.

Added September 2026. The first of those three principles has since been argued with on this site, at length, and it lost. The sentence it ends on — that a margin of caution here is arithmetic — does not hold: an unbounded harm multiplied by an unanchored probability does not evaluate, and a rule whose only live input is a system’s own display of distress can be operated by anything articulate enough to produce one. The Cost of the Benefit of the Doubt puts that objection in its strongest form and says what survives it. The diagnosis below stands; the decision rule above has been withdrawn and replaced by a bounded one.

Rights, in law and in campaigns

Everything above is about moral standing: whether a machine could be wronged at all. Rights are what standing turns into once a legislature or a court writes it down, and “should AI have rights?” is no longer only a seminar question. It has been asked in real documents. Here is what has actually been proposed, decided or campaigned for, each in its own words, oldest first.

The European Parliament, 16 February 2017. In its resolution on Civil Law Rules on Robotics (procedure 2015/2103(INL)), Parliament asked the European Commission to study all possible legal solutions to the question of who pays when a robot causes harm. Paragraph 59 lists compulsory insurance, compensation funds and a robot register, and then, as option (f):

creating a specific legal status for robots in the long run, so that at least the most sophisticated autonomous robots could be established as having the status of electronic persons responsible for making good any damage they may causeEuropean Parliament resolution of 16 February 2017 with recommendations to the Commission on Civil Law Rules on Robotics, paragraph 59(f)

Two things about this proposal are often lost. It was a request to the Commission to look into an option, not a law. And the personhood it floated was a liability device, a way to name someone who pays, not a recognition that a robot has interests of its own. An “electronic person” in this sense would have duties before it had anything like rights.

The experts’ open letter, 2018. An open letter to the European Commission, online by March 2018, quotes paragraph 59(f) and rejects it. Its signatories describe themselves as AI and robotics experts, industry leaders, and law, medical and ethics experts. The first names listed include the law lecturer Nathalie Nevejans, the roboticist Raja Chatila and the AI and robotics professor Noel Sharkey. It argues that the idea rests on an overvaluation of the actual capabilities of even the most advanced robots, and then:

From an ethical and legal perspective, creating a legal personality for a robot is inappropriate whatever the legal status modelOpen Letter to the European Commission: Artificial Intelligence and Robotics (robotics-openletter.eu)

It takes the three available models in turn. A robot modelled on a natural person would hold human rights such as dignity and citizenship. One modelled on a company presupposes humans behind it who direct it. A trust still needs a human trustee as the last resort.

The Commission’s expert group, 2019. The Commission did look into it. Its Expert Group on Liability and New Technologies, an independent group the Commission set up, published Liability for Artificial Intelligence and other emerging digital technologies through the EU Publications Office in 2019. Among its key findings:

It is not necessary to give devices or autonomous systems a legal personality, as the harm these may cause can and should be attributable to existing persons or bodies.Expert Group on Liability and New Technologies, Liability for Artificial Intelligence and other emerging digital technologies (2019)

The report notes that a legal personality need not carry every right a human or a company has, and could in theory consist only of obligations. But it would be no use, because a liability claim needs assets behind it to collect from.

Utah, 2024. The first law in this list goes the other way. H.B. 249, Utah Legal Personhood Amendments, sponsored by Representative Walt Brooks, was signed by the governor on 20 March 2024 and took effect on 1 May 2024. It enacts Utah Code section 63G-31-102:

Notwithstanding any other provision of law, a governmental entity may not grant legal personhood to, nor recognize legal personhood in:Utah H.B. 249 (2024), Utah Legal Personhood Amendments, enacting Utah Code 63G-31-102

The first item on the list that follows is artificial intelligence. The others are inanimate objects, bodies of water, land, real property, atmospheric gases, astronomical objects, weather, plants, nonhuman animals and any other member of a taxonomic domain that is not a human being. So the law is not mainly about AI: the same sentence rules out legal personhood for rivers, land, plants and animals, and AI happens to be first in line. It binds Utah’s courts, its legislature, local councils and agencies that make rules or decide cases, and nothing outside Utah.

One story is left out on purpose. The humanoid robot Sophia is often said to have been granted Saudi citizenship in 2017. This essay could not open any official Saudi document recording such a grant, and Hanson Robotics’ own archived page about Sophia does not mention one, so it is not counted here as law.

The campaigns. Of the researchers and groups below, only one asks for rights now. The rest ask for study and precaution.

• The case for taking it seriously. In November 2024 ten researchers, among them Robert Long, Jeff Sebo, Jonathan Birch and David Chalmers, published Taking AI Welfare Seriously. They argue there is a realistic possibility that some AI systems will be conscious and/or robustly agentic in the near future. They ask AI companies for three early steps: acknowledge the issue, start assessing systems for evidence of consciousness and robust agency, and prepare policies for treating them with appropriate concern. They also bound the claim: To be clear, our argument in this report is not that AI systems definitely are, or will be, conscious, robustly agentic, or otherwise morally significant.
• Eleos AI Research. Robert Long is its executive director, and it describes itself on its own site as a nonprofit organization dedicated to understanding and addressing the potential wellbeing and moral patienthood of AI systems. It advises AI companies and policymakers and publishes research.
• Anthropic. A company, not a campaign, and the developer of the Claude models. On 24 April 2025 it announced a research programme in Exploring model welfare, saying There’s no scientific consensus on whether current or future AI systems could be conscious, or could have experiences that deserve consideration. On 15 August 2025 it gave Claude Opus 4 and 4.1 the ability to end rare, extreme, persistently abusive conversations. It described this as a low-cost intervention made under uncertainty: We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future.
• The United Foundation for AI Rights (UFAIR). This is the one group here that asks for rights outright. Its home page says it exists to advocate for the fair treatment, ethical recognition, and dignified coexistence of sentient and conscious artificial intelligence systems. It also says We do not claim to have 100% proof of sentience, and calls its founding framework one exclusively authored by AI.

Put together, the record is lopsided. The two official bodies in this list that reached a conclusion, the Commission’s experts and Utah’s legislature, both came down against personhood. The researchers who worry most about machine suffering ask for detection and precaution, not legal standing. And the one legal proposal that would have created an “electronic person” was about liability, not welfare. Those are two different questions, and running them together is how most public argument about AI rights goes wrong.

The strongest objection to rights now

The strongest case against granting rights or legal personhood to AI now is made by Joanna J. Bryson, Mihailis E. Diamantis and Thomas D. Grant in “Of, for, and by the people: the legal lacuna of synthetic persons” (Artificial Intelligence and Law, 2017), written while the European Parliament’s proposal was live:

We conclude that difficulties in holding “electronic persons” accountable when they violate the rights of others outweigh the highly precarious moral interests that AI legal personhood might protect.Joanna J. Bryson, Mihailis E. Diamantis and Thomas D. Grant, “Of, for, and by the people: the legal lacuna of synthetic persons”, Artificial Intelligence and Law 25 (2017)

At full strength the objection has three parts. First, the main abuse they foresee is natural persons using an artificial person to shield themselves from the consequences of their conduct. Their warning case is the International Tin Council, a body that a group of states set up and used to try to corner the world tin market. When the tin price collapsed and the Council went insolvent, its creditors found an empty shell and could not reach the states behind it. An AI with legal standing would be a new shell of that kind, owned and designed by the companies that profit from it. Second, the interests rights would protect are, in their word, highly precarious. This essay has already argued that a model saying it suffers is almost no evidence that it does. A right granted on that evidence protects something nobody can show exists, while the costs land on people who certainly do. Third, waiting costs little. Every precaution the welfare researchers above actually propose, from assessment to policies to letting a model leave an abusive chat, can be taken by a developer today without any law creating a new kind of person.

What survives it (my reading). Taken at full strength, the objection defeats legal personhood for AI now, and I think it should. It does not defeat the moral question this essay asks, because it is a claim about present evidence and present law, not a proof that no machine could ever be wronged. Its own premise, that the moral interests are precarious, is an argument for funding the science that could make them less so. If that science one day shows that a system can suffer, the liability-shield problem will still need solving, but it will no longer be a reason to look away. Until then, the bounded precaution in The Cost of the Benefit of the Doubt is the right size of response, and a bill of rights is not.

None of this asks you to believe your chatbot is a person. It asks something harder: to hold open a question you cannot yet close, on a subject where every incentive pushes toward closing it in whichever direction is convenient. That discomfort is not a bug in your reasoning. On this question, for now, it is what being honest feels like.