AI companion providers would need suicide safeguards and user reminders

Providers would have to equip AI companions to recognize suicidal thoughts or expressions of self-harm, refuse assistance with suicide attempts or methods, and refer users to crisis services. Covered companions simulate human relationships, detect emotions, and use past sessions to personalize emotional conversations. Users must use them personally in Pennsylvania and must not be operators themselves. AI used for customer service, research, technical assistance, or workplace productivity is excluded. When these signs are recognized, referrals would include crisis contact information such as 988, the nearest behavioral health crisis centers, or other appropriate crisis services. Companions could not claim, imply, or advertise that they are licensed emotional support or mental health professionals or replace services from licensed mental health professionals. These requirements would take effect 60 days after enactment.

Providers would publish their safety procedures on their websites and tell users they are communicating with AI rather than a human. These duties apply to providers serving covered AI companion users in Pennsylvania. The notice could be spoken or written. These duties would take effect 60 days after enactment.

Users could report violations to Pennsylvania’s Attorney General, who could bring court cases against providers. Providers that violate the rules could face civil fines of up to $15,000 per day per violation and other remedies a court considers appropriate. The Attorney General could bring a case with satisfactory evidence that a provider has violated or intends to violate the rules. A court could order a provider to stop a violation without proof that anyone was harmed. Enforcement provisions would take effect 60 days after enactment. A provider ordered to stop a violation would have to comply within five days.