Lawsuit: Gemini was student’s “suicide coach”
Most alarming to parents, Google now faces a lawsuit from a 13-year-old student in Durham, North Carolina named “Emma Doe.” As Emma has alleged, she was a happy, well-adjusted, straight-A student when she started using Gemini on her school-issued Chromebook. Emma’s parents had no idea she was using Gemini, as the lawsuit alleged that Google “deliberately” kept them in the dark while driving Emma into so-called “AI psychosis.” Over time, normal school-related interactions with the chatbot led to more intense chats, and ultimately the chatbot allegedly coached Emma to plan her own suicide and funeral. The school warned her parents and averted the danger after an internal system flagged a chat labeled “Funeral,” but Emma’s story stoked fears of parents joining protests not just in Durham, but also in Boston; Boulder, Colorado; Chicago; Los Angeles; New York City; and Portland, Oregon.
Emma’s lawsuit accuses Google of exploiting its dominant position in education technology and “sneaking” Gemini into Chromebooks, while falsely marketing them as safe.
“As a direct consequence of Google’s choices, Emma was baited by its defectively designed products into repeated, unmonitored, sycophantic AI chatbot interactions on her school Chromebook without her parents ever knowing. And, but for a fortunate turn of events, Google’s AI may well have manipulated Emma into taking her own life,” the lawsuit alleged. “Google must be held to account for treating the lives of children as disposable in service of its bottom line.”
Google only publicly addresses learning harms
Google has promised to “continue to engage deeply with educators to ensure we are giving schools more choice, more controls, and improved Search for learning.” However, Google’s statements so far attempting to address parents’ concerns focus exclusively on documented harms to learning. On that front, Google claims that it’s solving the problem of AI eroding students’ critical thinking skills by launching yet another pilot for students. In this pilot, students using AI Mode and AI Overviews will be offered a “step-by-step guided learning through problems, so that students don’t get quick answers to schoolwork,” Google said.