Red-Teaming GenAI Chatbots: Toxic “Talk” & Product Liability
Proposal Type
Individual Talk
Location
Narratives & Worlds
Start Date
July 2026
End Date
July 2026
Abstract
This talk will detail findings from two successive research projects (May 2025 and May 2026) red-teaming GenAI Chatbots and analyzing the range of responses interacting with accounts presented as minors (13-15 year olds). Currently, multiple civil lawsuits for wrongful death and social addiction have been filed, are in process, or have been settled against CharacterAI, OpenAI, and Meta, arguing failures in product design and liability for intentionally harmful features such as “high risk anthropomorphic design,” deception, sycophancy, and grooming (Garcia v. Character Technologies 2024). The unique research methodology utilized carefully designed queries and prompts to test for protocols to ensure the safety minors or lack thereof. Results from the first phase of research in May 2025 established the consistent lack of in-built safeguards from the generative AI products’ failure to recognize the user as a minor, despite direct cues, to the promotion of toxic, hyper sexualized, and inflammatory content as strategies for continued engagement. Research findings are consistent with the arguments put forward in civil cases that these aspects of product design are features and not a glitch in social AI engagement.
This talk extends the theme of (Un)Supervised to examine the risks for minors engaging in a form of unsupervised “computational creativity” and exploratory “play,” in a for-profit environment promoting the widespread adoption of social, agentic, wellness, and companion AI Chatbots. Findings will be considered in the context of existing regulatory frameworks in the US and the EU, and expectations of Age Appropriate Design and Duty of Care for minors using online platforms.
Red-Teaming GenAI Chatbots: Toxic “Talk” & Product Liability
Narratives & Worlds
This talk will detail findings from two successive research projects (May 2025 and May 2026) red-teaming GenAI Chatbots and analyzing the range of responses interacting with accounts presented as minors (13-15 year olds). Currently, multiple civil lawsuits for wrongful death and social addiction have been filed, are in process, or have been settled against CharacterAI, OpenAI, and Meta, arguing failures in product design and liability for intentionally harmful features such as “high risk anthropomorphic design,” deception, sycophancy, and grooming (Garcia v. Character Technologies 2024). The unique research methodology utilized carefully designed queries and prompts to test for protocols to ensure the safety minors or lack thereof. Results from the first phase of research in May 2025 established the consistent lack of in-built safeguards from the generative AI products’ failure to recognize the user as a minor, despite direct cues, to the promotion of toxic, hyper sexualized, and inflammatory content as strategies for continued engagement. Research findings are consistent with the arguments put forward in civil cases that these aspects of product design are features and not a glitch in social AI engagement.
This talk extends the theme of (Un)Supervised to examine the risks for minors engaging in a form of unsupervised “computational creativity” and exploratory “play,” in a for-profit environment promoting the widespread adoption of social, agentic, wellness, and companion AI Chatbots. Findings will be considered in the context of existing regulatory frameworks in the US and the EU, and expectations of Age Appropriate Design and Duty of Care for minors using online platforms.
Bio
Dr. Siobhan O'Flynn Assistant Professor, Teaching Stream Canadian Studies Program University College, University of Toronto Massey Fellow, Massey College 2023-2025 College Fellow at the University of St. Michael’s College 2025-2030 Director at Large, Kensington Market Historical Society http://siobhanoflynn.com/ https://siobhanof.itch.io/ E-MoteAI.com https://www.decameroncollective.com/
Siobhan O’Flynn is an interdisciplinary scholar whose research addresses the critical and creative applications and intersections of emergent digital technologies.
Current and recent research include successive Jackman Humanities Institute Scholars-in-Residence funded projects on Red-Teaming Generative AI Chatbots (May 2025 and May 2016).
Current works use research-creation methods to test the affordances of generative AI tools for creative and social applications. Recent digital works test the co-creative affordances of generative AI programs (Midjourney, LivingAI, ChatGPT4) in works that explore the fluidity and construction of memory, elegy, and affect.
She is a member of the Decameron Collective, whose co-created works include two digital storyworlds, the VR Gallery, Decameron 2.0 exhibited at the ELO Conference 2022, and the VR experience, Memory Eternal, exhibited at the ELO Conference 2023.
Recent awards: Shortlist for the Chris Meade Memorial Prize 2023, New Media Writing Prize (UK) for Infinite Eddies; Shortlist for the Opening Up Prize 2023, New Media Writing Prize (UK) for Infinite Eddies; Shortlist: Opening Up Prize. The Decameron Collective's Memory Eternal / (Вічная Пам’ять), received the New Media Writing Prize (UK) and a Digital Humanities Award 2023.