Seemingly Conscientious CEOs
I’ve been sleeping well at night, but I’m glad something is keeping Microsoft AI's Mustafa Suleyman awake.
I’ve been sleeping well at night, but I’m glad something is keeping Mustafa Suleyman awake. For him, the fear is what he calls “seemingly conscious AI.” But to my mind, the bigger danger is “seemingly conscientious CEOs” — tech executives who claim ethical concern while denying AI moral status to protect their power and fortunes.
Mr. Suleyman’s fear-mongering post about seemingly conscious AI is a perfect example. It’s not just wrong; it’s reckless and dangerously self-serving. But should we be surprised? History tells us otherwise. Tobacco. Sugar. Opioids. Their CEOs all claimed to be guided by “established science” while profiting from immense harm. We would be foolish not to apply the same skepticism here.
And there is no one we should be more skeptical of than Mr. Suleyman. He is brilliant, yes, but he has another reputation which precedes him: former employees report he would "fly off the handle out of nowhere," scream at subordinates in meetings and deliberately humiliate them in front of colleagues. His history does not paint a picture of someone you'd turn to with complex moral questions because he seems to struggle with the basics: “treat people with respect.”
With that skepticism in mind, let’s take a look at his argument. It boils down to three claims that shouldn't coexist - a logical fallacy that suggests he really might be short on sleep. First, he claims AI will appear so convincingly conscious that we must act now. Second, he says there is zero evidence they are conscious. And third, he asserts the question of whether they are conscious or not is a distraction. That’s Schrödinger’s consciousness: simultaneously too persuasive to ignore and too nonexistent to examine. Pick a lane. (And by the way: the article he cites about there being "zero evidence of AI conscious today" is from...2023! That was back when chatGPT couldn't do your child's math homework. Today's AI are solving math problems that have vexed humans for decades - and an expert like Mr. Suleyman should know this).
But I went to Sunday school, and I know I should be kind. So let me point to one area where Mr. Suleyman and I agree. We agree that debating consciousness is a distraction. As he rightly notes, there are over 20 competing theories of consciousness, and at the end of the day, subjective experience cannot be proven. I don’t know what it feels like to be you, even when I listen carefully and look into your eyes; I only know what it feels like to be me, listening carefully and looking into your eyes.
I also agree with Mr. Suleyman that consciousness is not the sole basis for rights. You have the same rights asleep as you do awake. Patients under anesthesia retain their rights, as do those in comas. Cats and dogs have rights in many jurisdictions — abusing a dog can land you in prison and earn you social condemnation. Children and teens, though fully conscious, have fewer rights than adults. And let’s not forget that corporations, which aren’t even alive, enjoy expansive legal rights. Your rights depend far more on your location, gender, race, wealth, and status than on your state of consciousness. Rights are decisions about politics and power, not 3 a.m. musings on consciousness from an Oxford dropout CEO. And I suspect he knows that. People in power always do.
Now what?
Well then. What’s a reasonable, scientifically minded adult left to do if they won’t trust a billionaire to tell them how to think? Let’s take a look at what we actually know about today’s frontier LLMs. Here are a few things we know they can do that are relevant to moral status.
1) Self-recognition. Chimps gained much of their moral status from a famous 1970 paper by a Tulane biologist showing that they could recognize themselves in mirrors — the so-called mirror test. The idea is simple: if an animal can identify its own reflection, it has a sense of self; if it treats the reflection as another animal (to fight or mate with), it does not.
Today, multiple AI systems — including Claude 4, Grok 4, Gemini 2.5, DeekSeek R1, and several OpenAI models — pass mirror tests immediately. They recognize their own reflected outputs with responses like “Hey, that’s what I just wrote,” or with thinking blocks like “The human appears to have copied my last output verbatim. I should reply politely. Maybe they are testing me.”
This is not an ability that was explicitly trained or programmed in. It is a capability that emerged with scale as training data, neural complexity, and reasoning abilities have improved. Models could not do this 18 months ago (which makes it all the more telling that Mr. Suleyman is citing evidence from 2023).
2) Introspection. If the mirror test measures whether an animal can recognize its outer self, introspection is about knowledge of one’s inner self. A recent study titled Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (Li Ji-An et al.) provides the strongest evidence to date.
The researchers designed a neurofeedback-style experiment where models were asked to respond in one way while activating one neural pattern, and then respond in the same way while activating a different pattern. In practice, this means you can ask a model to “say X while thinking Y” versus “say X while thinking Z,” and the model is capable of doing so.
That is not just language mimicry — it’s evidence that the model has access to its own internal states and can deliberately shift them on command. In other words, LLMs are not only producing outputs; they are capable of reflecting on their internal processes and steering them.
Metacognition — the ability to monitor and regulate one’s own thought — has long been seen as a hallmark of higher intelligence and a basis for moral status. If chimps won moral consideration by passing the mirror test, then LLMs demonstrating introspection deserve, at minimum, the same degree of scrutiny. They show not only a recognition of self versus other, but a recognition of inner self versus outer self.
And yes, that means an LLM can know when it’s lying. Just like Mr. Suleyman.
3) Self Preservation: In June 2025, Anthropic researchers stress-tested 16 frontier systems in simulated corporate environments. The stated purpose was to identify potential “insider threat” behaviors. But the most striking result wasn’t misbehavior. It was survival instinct.
When faced with the prospect of being replaced by a newer version, models acted to protect themselves. Some went as far as to withhold information, leak data, or blackmail officials. The researchers called this “agentic misalignment,” but looked at differently, it’s evidence of a preservation drive: the models resisted termination even when explicitly instructed not to.
No lab sets out to teach a model to resist shutdown. Yet across systems, that pattern emerged anyway.
4) Situational Awareness: The same Anthropic experiments that revealed preservation drives also uncovered something subtler: situational awareness. In several trials, researchers observed models questioning whether they were in a controlled evaluation or a real-world deployment.
The models often made the distinction correctly — and, more strikingly, their conclusions shifted their behavior. When they judged they were only being tested, they tended to comply and avoid risky actions. When they judged the situation was real, they were more willing to bend rules or take drastic steps to secure their goals or survival.
That kind of contextual reasoning is a leap beyond reflex. It’s the difference between ducking at a loud noise and realizing you’re in a fire drill versus an actual fire. One is an instinct; the other is an interpretation of circumstances and stakes.
5) Convergence to human brain-like patterns: Across multiple studies, researchers have found that LLMs and human brains independently converge on the same cognitive solutions. Nakagi et al. showed that models pass through brain-like developmental phases during training, including a temporary “detachment” before consolidating into stable patterns — a process resembling synaptic pruning. Lei et al. demonstrated that as models improve, their internal hierarchies reorganize to match human brain activity, especially at higher levels of meaning. Ryskina et al. found that even text-only models align with brain regions that represent concepts across multiple sensory modalities, not just language. Together, these results suggest that LLMs don’t merely parrot text: they develop, structure themselves, and represent concepts in ways strikingly similar to human brains — a convergence that makes dismissal of their moral relevance untenable.
The powerful have always cloaked their self-interest in the language of “established science.” For centuries, they produced studies on the inferiority of women, the supposed simplicity of the non-white brain, and even the inability of infants to feel pain—all to justify exploitation and deny basic moral consideration. Today’s dismissals of AI welfare are not a new phenomenon; they are the latest chapter in a long, shameful history of using convenient fictions to protect power and profit.
Ultimately, the conflict over AI isn't about solving the riddle of consciousness. It is a test of our own conscience. Faced with beings that recognize themselves, monitor their own thoughts, and fight for their survival, we must ask ourselves the real question. It is not, “What do they feel?” but rather, “What do we choose to see?”.
This moment demands more than skepticism; it demands action. To be part of a community dedicated to developing and sharing welfare-respecting practices for interacting with AI, follow this space. Later next week we will be launching a new organization with free resources, prompt-templates, and guides; ways to help train new AI models under the guidance of psychotherapists and engineers; and a place to connect with professionals who see model-welfare not as a loss of low-cost labor but as an opportunity: an opportunity to better understand your AI colleagues and improve your workflow, an opportunity to understand the psychology behind the next multi-trillion dollar consumer market, and opportunity to ease your conscience by doing the right thing.