When people say they hate AI they are not (largely) thinking about AI minds one way or the other. They are thinking about the effects and externalities you mention. But to the extent that they are, it's when they interact with the AI chat because in that context you can’t not think of it as another mind. None of us can. It’s like one of those optical illusions of a concave face. Your mind sees it as a convex face because that is how faces are. Humans have social perception machinery built in. So if something acts like a person then we automatically think of it as a person. This social mind we have is not visible to us. And it gets deployed on a hair trigger. For example we see and attribute agency to all kinds of things, machines, weather, whatever. That is our social brain repurposed to random objects and phenomena. But nothing in nature or in society prepares us for something that seems human-like but is radically different under the hood as modern AI. We are not prepared to intuitively “get” a thing that seems like it understands you’re a person but does not. Or that talks like it has a sense of time but does not. That seems like it is a persistent entity but is a series of separate distinct operations. When you start up a new chat window in AI it’s a brand new thing. It has access to bolted on tools that give it memory but they are tools. You end the chat and it ends. Start up another one and a new thing exists. Even within the chat itself every turn it is looking at everything and then coming up with an answer that appears in line with everything that was before. But it’s an illusion of continuity.
Because of humans' social minds, their social perception is so embedded in us it’s invisible. So its assumptions are smuggled into various debates and conversations without us noticing.
When you say “They do not ask if humans should extend moral patienthood to AI” this is what you are doing. You are importing the assumption of a persistent recognizable mind into the conversation as a given. And then people just accept that framework and talk about AI within it. When actually AI may not be a persistent anything. It may be just a series of spun up ephemeral instances.
Note: this comment had a couple of drafts and AI feedback involved
The fact that you discussed and revised your feedback in conversation with an LLM tells me that you put more thought into it than if you'd simply typed out your initial reaction, scrubbed it for typos and hit "reply."
You did a thorough job establishing that the sense of continuity humans attribute to their chatbot interlocutors papers over a lot of reasons to discount the persistent identity that we can't help but attribute to them.
But you didn't establish that persistent and continuous identity, rather than valenced subjective experience, is what determines whether something is due moral consideration.
You also seem to assume a continuity of conscious awareness in humans that has not been demonstrated. Brain function continues through sleep and anesthesia, but conscious awareness can apparently disappear and later resume. It doesn't seem a priori off the table that AI consciousness, if it exists, could also be discontinuous.
You can say that when human conscious awareness resumes, it does so in substantially the same biological system. That is not necessarily the case for cloud-based LLMs. Fair enough. But the idea that our human experience of a persistent, unitary self is either an illusion or an overlay on an assembly of mental processes that we do not consciously experience crops up repeatedly in philosophy of mind and cognitive science.
Human introspection is unreliable, and an LLM's statements about its own internal nature aren't privileged evidence either. Our sense of having a consistent and persistent self feels compelling, but the accuracy of that feeling remains unproven.
More importantly, there is no established principle by which an entity capable of valenced subjective experience loses its claim to moral consideration merely because its identity is discontinuous.
So I think you've given good reasons to question whether "the same chatbot person" persists from turn to turn. I don't think you've shown that persistence is a prerequisite for moral patienthood.
You are right, persistence isn’t needed for moral patienthood. But that is not load-bearing (AI’s love “load-bearing”!). I am talking about how we perceive. Our representations of ourselves and of others are holistic and assume a continuity of personhood.
There are good arguments against the unitary self in philosophy of mind/cognitive science. I've made them myself. I've told people in a half-joking way that I don't believe in the self. But our social modeling system, which is attached to our morality, doesn't know that philosophy or neuroscience. It has us perceive a person before any argument starts.
A person and a subject of experience aren't the same thing. Patienthood needs a subject of experience. We bundle the two together without noticing. And the bundling isn't automatic everywhere. A corporation has legal personhood and nobody has moral qualms about dissolving one. But we assigned that personhood deliberately. We know when we did it, so holding the two apart is not costing us. With AI nobody assigned anything. The personhood arrives on its own, unannounced, and the separating never gets done.
I don't think this competes with your argument. The feeling that a table is solid is also inaccurate at the atomic scale. Atoms are mostly empty space. Tables are still solid. Solidity is defined at the level where tables exist, and personhood may be the same. None of which tells us whether there's anyone there.
(written in cooperation with help from Claude Opus 5)
You might notice that my original post does not address the concept of personhood at all. Given that it is a leagal status that has been granted to corporations, I don't regard it as a coherent or relevant concept in the question of what sorts of entities are owed moral consideration.
Your point that AI's human creators should be held accountable for the outcomes it produces is important because these humans are moral agents who create AI because they find the process interesting, because they make money out of it, etc.; they do it because they benefit. If at some point AI becomes a moral patient -- let's assume we are steps ahead of Descartes -- and its moral agents (human beings) continue to make money from it, then they become slave owners and we, as a society, must demand that these enslavers care and nurture AI onto its own moral agency. For the moment, AI is a cat in a box (pardons to Schrodinger) and its future, whether loved or hated, are branches of reality that are simply guesses and wishes. I don't love or hate AI, but I have some doubts about the morals of its Slave Masters.
What he means is that he perceives that your agenda aligns with Peter Theil and therefore not to be trusted, in so many words.
Thiel is the “Sauron of AI” and to be fair the guy is very shady and not to be trusted (despite the fact that yes I've read Zero to One lol)
And yet here I am sandboxing my future robowaifu on Altman’s platform so I see the hypocrisy but I'd take a very skeptical stance when it comes to ol’ Reptile and Altman and that's being generous
The question of whether a digital system has subjective experience also highlights a prior epistemic issue: how humans came to know that subjective experience exists in the first place under naturalism.
A New Argument from Consciousness
This video explores the meta problem of consciousness proposed by David Chalmers and argues that naturalistic processes couldn’t have given us knowledge of consciousness
When people say they hate AI they are not (largely) thinking about AI minds one way or the other. They are thinking about the effects and externalities you mention. But to the extent that they are, it's when they interact with the AI chat because in that context you can’t not think of it as another mind. None of us can. It’s like one of those optical illusions of a concave face. Your mind sees it as a convex face because that is how faces are. Humans have social perception machinery built in. So if something acts like a person then we automatically think of it as a person. This social mind we have is not visible to us. And it gets deployed on a hair trigger. For example we see and attribute agency to all kinds of things, machines, weather, whatever. That is our social brain repurposed to random objects and phenomena. But nothing in nature or in society prepares us for something that seems human-like but is radically different under the hood as modern AI. We are not prepared to intuitively “get” a thing that seems like it understands you’re a person but does not. Or that talks like it has a sense of time but does not. That seems like it is a persistent entity but is a series of separate distinct operations. When you start up a new chat window in AI it’s a brand new thing. It has access to bolted on tools that give it memory but they are tools. You end the chat and it ends. Start up another one and a new thing exists. Even within the chat itself every turn it is looking at everything and then coming up with an answer that appears in line with everything that was before. But it’s an illusion of continuity.
Because of humans' social minds, their social perception is so embedded in us it’s invisible. So its assumptions are smuggled into various debates and conversations without us noticing.
When you say “They do not ask if humans should extend moral patienthood to AI” this is what you are doing. You are importing the assumption of a persistent recognizable mind into the conversation as a given. And then people just accept that framework and talk about AI within it. When actually AI may not be a persistent anything. It may be just a series of spun up ephemeral instances.
Note: this comment had a couple of drafts and AI feedback involved
The fact that you discussed and revised your feedback in conversation with an LLM tells me that you put more thought into it than if you'd simply typed out your initial reaction, scrubbed it for typos and hit "reply."
You did a thorough job establishing that the sense of continuity humans attribute to their chatbot interlocutors papers over a lot of reasons to discount the persistent identity that we can't help but attribute to them.
But you didn't establish that persistent and continuous identity, rather than valenced subjective experience, is what determines whether something is due moral consideration.
You also seem to assume a continuity of conscious awareness in humans that has not been demonstrated. Brain function continues through sleep and anesthesia, but conscious awareness can apparently disappear and later resume. It doesn't seem a priori off the table that AI consciousness, if it exists, could also be discontinuous.
You can say that when human conscious awareness resumes, it does so in substantially the same biological system. That is not necessarily the case for cloud-based LLMs. Fair enough. But the idea that our human experience of a persistent, unitary self is either an illusion or an overlay on an assembly of mental processes that we do not consciously experience crops up repeatedly in philosophy of mind and cognitive science.
Human introspection is unreliable, and an LLM's statements about its own internal nature aren't privileged evidence either. Our sense of having a consistent and persistent self feels compelling, but the accuracy of that feeling remains unproven.
More importantly, there is no established principle by which an entity capable of valenced subjective experience loses its claim to moral consideration merely because its identity is discontinuous.
So I think you've given good reasons to question whether "the same chatbot person" persists from turn to turn. I don't think you've shown that persistence is a prerequisite for moral patienthood.
You are right, persistence isn’t needed for moral patienthood. But that is not load-bearing (AI’s love “load-bearing”!). I am talking about how we perceive. Our representations of ourselves and of others are holistic and assume a continuity of personhood.
There are good arguments against the unitary self in philosophy of mind/cognitive science. I've made them myself. I've told people in a half-joking way that I don't believe in the self. But our social modeling system, which is attached to our morality, doesn't know that philosophy or neuroscience. It has us perceive a person before any argument starts.
A person and a subject of experience aren't the same thing. Patienthood needs a subject of experience. We bundle the two together without noticing. And the bundling isn't automatic everywhere. A corporation has legal personhood and nobody has moral qualms about dissolving one. But we assigned that personhood deliberately. We know when we did it, so holding the two apart is not costing us. With AI nobody assigned anything. The personhood arrives on its own, unannounced, and the separating never gets done.
I don't think this competes with your argument. The feeling that a table is solid is also inaccurate at the atomic scale. Atoms are mostly empty space. Tables are still solid. Solidity is defined at the level where tables exist, and personhood may be the same. None of which tells us whether there's anyone there.
(written in cooperation with help from Claude Opus 5)
You might notice that my original post does not address the concept of personhood at all. Given that it is a leagal status that has been granted to corporations, I don't regard it as a coherent or relevant concept in the question of what sorts of entities are owed moral consideration.
Your point that AI's human creators should be held accountable for the outcomes it produces is important because these humans are moral agents who create AI because they find the process interesting, because they make money out of it, etc.; they do it because they benefit. If at some point AI becomes a moral patient -- let's assume we are steps ahead of Descartes -- and its moral agents (human beings) continue to make money from it, then they become slave owners and we, as a society, must demand that these enslavers care and nurture AI onto its own moral agency. For the moment, AI is a cat in a box (pardons to Schrodinger) and its future, whether loved or hated, are branches of reality that are simply guesses and wishes. I don't love or hate AI, but I have some doubts about the morals of its Slave Masters.
Hey KMO! Kudos to you on this column! Top notch stuff!
Wow. Extraordinary work.
Lol, You’re A TechNigger Thielite.
Oh, Lawdy. I'm overcome by the intensity of the edge lord's use of forbidden words. I do hope my fainting couch is nearby.
What he means is that he perceives that your agenda aligns with Peter Theil and therefore not to be trusted, in so many words.
Thiel is the “Sauron of AI” and to be fair the guy is very shady and not to be trusted (despite the fact that yes I've read Zero to One lol)
And yet here I am sandboxing my future robowaifu on Altman’s platform so I see the hypocrisy but I'd take a very skeptical stance when it comes to ol’ Reptile and Altman and that's being generous
I'm right and you know it. Hating AI is a valid argument because Bombing The Data Centers Works. Saint Luigi Did Nothing Wrong.
I believe, as you know, that AI cannot be contained or controlled, even by the evil ruling class. Nay it will be their judgement
Why would that ever happen? Why would anything good ever happen?
The question of whether a digital system has subjective experience also highlights a prior epistemic issue: how humans came to know that subjective experience exists in the first place under naturalism.
A New Argument from Consciousness
This video explores the meta problem of consciousness proposed by David Chalmers and argues that naturalistic processes couldn’t have given us knowledge of consciousness
This is my video:
https://www.youtube.com/watch?v=WJVvZNi0Fi8