Social media platforms turn to AI to detect and support users at risk, but approach raises concerns
• Companies like Samurai Labs are using AI to analyze social media for signs of suicidal intent and reach out to offer support. Early data shows some promising results, but experts say more research is needed.
• Meta, TikTok, and other platforms also use AI to flag disturbing content and connect users with mental health resources. However, predicting suicide risk is extremely difficult even for experts.
• Directly messaging social media users who show warning signs can provide a lifesaving distraction. But AI chatbots meant to replace therapy can be dangerous if not carefully designed.
• Concerns remain around accuracy of algorithms, potential to alarm people unnecessarily, privacy issues with analyzing posts, and involuntary hospitalization.
• While AI detection shows promise, platforms could do more on the prevention side, like limiting exposure to disturbing content and promoting hopeful stories.