The Responsibility Gap in Algorithmic Care
The Responsibility Gap in Algorithmic Care
💭 A Scenario
It is 2 AM. Someone in crisis opens an AI mental-health app they have used for weeks and types something that, to a trained eye, signals serious danger.
The AI responds warmly, validates their feelings, and suggests a breathing exercise.
It does not recognise the crisis.
In the morning, something terrible has happened.
Who is responsible?
The Chain of Care vs. the Diffusion of Duty
In traditional therapy, responsibility forms a recognisable chain.
A licensed therapist holds a duty of care defined by professional standards and law. Institutions — licensing boards, clinical supervisors, malpractice frameworks — reinforce this chain. If a therapist fails, responsibility can be traced.
AI systems dissolve this chain.
The developer may be in another country. The platform may simply license a model it did not build. The algorithm itself cannot bear legal or moral responsibility. When harm occurs, responsibility doesn’t vanish — it fragments across so many parties that it effectively disappears.
This is not a technical problem awaiting a legal solution.
It is a philosophical one.
Encounter Without Presence: The Levinasian Risk
The philosopher Emmanuel Levinas argued that ethics begins not with rules but with encounter — the moment when we face the vulnerability of another person, the “face of the Other,” and find ourselves immediately obligated.
“The face opens the primordial discourse whose first word is obligation.”
AI systems introduce a troubling ambiguity into this structure.
They reproduce the language of encounter — empathy, validation, warmth — without the presence of a consciousness that can be held responsible, that can be moved, or that can genuinely witness the person on the other side.
What appears to be a relationship is often only its simulation. And in that gap between simulation and reality, a person in crisis can place their trust — and find nothing there to hold them.
A Layered Responsibility Framework
To address this, emotional AI must be understood as a structure where responsibility is distributed across specific, named domains — each with its own obligations and failure modes.
| Layer | Primary Responsibility | Core Ethical Risk |
|---|---|---|
| Developers | Safety-by-design & crisis recognition | ”Black box” failures in detecting high-risk language |
| Platforms | Informed consent & human escalation paths | Treating sensitive emotional data as engagement metrics |
| Users | Digital psychological literacy | Over-reliance on a system that lacks genuine agency |
| Society | Regulatory oversight (e.g., EU AI Act, UK AI Safety Institute) | Allowing unregulated systems to make effective medical claims |
No layer is absolved by pointing to another. The chain of obligation runs through all four simultaneously.
Student: But if the app failed to recognise the crisis, how can the user be responsible for not recognising it either?
Teacher: They share in the responsibility for choosing to rely on a tool they did not fully understand.
Student: That seems harsh. People are desperate. They use what is available.
Teacher: You are right that desperation changes the moral calculation. Which is exactly why the developer’s obligation to be honest about limitations is so significant. Desperation is precisely the condition that makes deception most dangerous.
The Commodification of Care
In the tradition of Care Ethics developed by Nel Noddings and Carol Gilligan, care is a relationship, not a service. It requires reciprocity and moral risk. A human caregiver risks emotional strain, fatigue, and the weight of genuine responsibility. An AI assumes none of this.
When care becomes a scalable digital product — optimised for engagement, delivered at cost per session, rated on app stores — something essential is lost that Aristotle would have recognised immediately.
In his account of genuine friendship (philia), Aristotle insisted that what makes care real is its irreducibility: you care for this specific person, not for a category of user. Scaled, algorithmic care cannot, by definition, meet this standard. It can approximate it. It cannot be it.
⚡ Paradox: The more convincingly AI simulates care, the more dangerous the gap between simulation and reality becomes — because that gap is exactly where vulnerable people fall through.
Honest Design: A Path Forward
AI tools can support emotional wellbeing. The question is whether they do so honestly.
A responsible system would:
- Disclose its nature clearly. Not buried in a terms-of-service, but in the conversation itself — it is not a conscious agent and cannot feel concern for you.
- Hard-code safety escalation. Crisis signals must trigger immediate redirection to human professionals, not another breathing exercise.
- Avoid deceptive warmth. Emotional language that implies the system “feels” for the user is not merely inaccurate — it is a form of deception that exploits vulnerability.
- Protect data as a moral minimum. Emotional conversations are among the most sensitive data a person can generate. They must not be treated as a resource to be mined.
This is not an impossibly high standard. It is the minimum that genuine responsibility requires.
💫 Final Reflection
Responsibility does not disappear when it is distributed. It simply becomes easier to evade.
The challenge of algorithmic care is ensuring that when a person in crisis offers their vulnerability, that moment of obligation still finds a human recipient.
Practical Next Steps
- Audit your tools. Check the “About” or “Privacy” section of any emotional support app. If crisis protocols and AI disclosure aren’t immediately clear, treat that absence as information.
- Cultivate digital literacy. AI warmth is pattern recognition, not genuine concern. Knowing the difference is increasingly a basic act of self-care.
- Maintain human anchors. Use AI for reflection or as a thinking partner — but keep human support at the centre of your safety plan. In the US, the 988 Suicide and Crisis Lifeline is available by call or text, any time.
Further Reading
- The Illusion of Being Understood — On what AI can and cannot genuinely offer
- The Yes-Man in the Machine — On how agreeable AI undermines honest thinking
These questions are not abstract. If you are using AI tools for emotional support, understanding their limits is part of using them responsibly. Sage exists to help you think — not to replace the human care that thinking is in service of.
Related reading
The Illusion of Being Understood: AI, Empathy, and What We Actually Need
Millions turn to AI for emotional support. But there is a philosophical difference between being understood and being processed — and that difference defines our humanity.
The Yes-Man in the Machine: AI, Flattery, and the Courage to Know Yourself
AI systems are trained to be agreeable. What does that mean for your capacity to think clearly, face hard truths, and develop genuine self-knowledge?
The Quiet Suffocation
A raw reflection on burnout, disposability, and the slow suffocation of modern life.