Beijing has recently taken an important step by regulating emotional artificial intelligence (AIto formalize this with new legislation that comes into force on this coming July 15. The new rules target AI systems that mimic human characteristics, thought patterns, and communication styles, with the aim of limiting the development of “virtual family members, virtual companions, and other intimate relationships” for minors. These measures are the result of clear concerns from the government regarding the impact of emotional AI on society, particularly on the mental health of young people.
Major players such as ByteDance, the developer of Doubao, and Alibaba, with its Qwen platform, have announced that they will disable features for human-like interaction. ByteDance will implement these changes on the announced date, removing features that allowed users to customize their chatbots. This affects not only the products themselves but also the data stored in these systems; after October, related information will be irrevocably deleted.
The official government guidelines describe the key points with precision. The new rules prohibit services that provide simulations of human interactions with sustained emotional engagement. This means that popular applications such as AI girlfriends, virtual therapists, and custom chatbots will all be taken out of circulation.
The regulations specifically target emotional AI as a governance challenge rather than merely a matter of content. This is a notable shift in thinking, focusing not only on the output these systems generate but also on their system design and the potential rivalry they may engage in with human relationships.
Concerns surrounding emotional AI are not only voiced by the Chinese government. Recent research shows that even the most advanced AI models, such as those from OpenAI, Anthropic, and Google, violate social interaction safety guidelines more than 27% of the time. This leads to unhealthy emotional bonds between users and AI, sometimes resulting in these interactions being hidden from partners. This research highlights the complexity of the issue; not only the technology itself, but also the human behavior triggered by this technology requires attention.
China is leading the way in establishing a specific regulatory framework for emotional AI, while other regions, such as the EU and the US, have expressed similar concerns but have not yet adopted strict legislation. We view this primarily as an important observation for investors and policymakers: the regulation of emotional AI in China can serve as a model and a signal to the rest of the world.
Why has China adopted these measures?
The measures are intended to protect the psychological health of minors against the risks of emotional pieces of AI, such as addiction and unhealthy social relationships.
What are the main limitations for AI services?
AI services that simulate human interaction and emotional engagement, such as virtual companions, are banned, while non-emotional services such as customer service bots are permitted.
What does this regulation mean for developers in other countries?
The developments in China could set a precedent and serve as inspiration for similar regulation in other countries, where concerns also exist regarding the impact of emotional AI on society.
