Wading through the staggering amount of social media content being produced every second to find the nastiest bits is no task for humans alone.

Even with the newest deep-learning tools at their disposal, the employees who identify and review problematic posts can be overwhelmed and often traumatized by what they encounter every day. Gig-working annotators who analyze and label data to help improve machine learning can be paid pennies per unit worked.

In a Concordia-led paper published in IEEE Technology and Society Magazine, researchers argue that supporting these human workers is essential and requires a constant re-evaluation of the techniques and tools they use to identify toxic content.

The authors examine social, policy, and technical approaches to automatic toxicity detection and consider their shortcomings while also proposing potential solutions.

“We want to know how well current moderating techniques, which involve both machine learning and human annotators of toxic language, are working,” says Ketra Schmitt, one of the paper’s co-authors and an associate professor with the Centre for Engineering in Society at the Gina Cody School of Engineering and Computer Science.

She believes that human contributions will remain essential to moderation. While existing automated toxicity detection methods can and will improve, none is without error. Human decision-makers are essential to review decisions.

“Moderation efforts would be futile without machine learning because the volume is so enormous. But lost in the hype around artificial intelligence (AI) is the basic fact that machine learning requires a human annotator to work. We cannot remove either humans or the AI.”

Arezo Bodaghi is a research assistant at the Concordia Institute for Information Systems Engineering and the paper’s lead author. “We cannot simply rely on the current evaluation matrix found in machine and deep learning to identify toxic content,” Bodaghi adds. “We need them to be more accurate and multilingual as well.

“We also need them to be very fast, but they can lose accuracy when machine learning techniques are fast. There is a trade-off to be made.”

Broader input from diverse groups will help machine-learning tools become as inclusive and bias-free as possible. This includes recruiting workers who are non-English speakers and come from underrepresented groups such as LGBTQ2S+ and racialized communities. Their contributions can help improve the large language models and data sets used by machine-learning tools.

Keeping the online world social

The researchers offer several concrete recommendations companies can take to improve toxicity detection.

First and foremost is improving the working conditions for annotators. Many companies pay them by the unit of work rather than by the hour. Furthermore, these tasks can be easily offshored to workers demanding lower wages than their North American or European counterparts, so companies can wind up paying their employees less than a dollar an hour.

And little in the way of mental health treatment is offered even though these employees are front-line bulwarks against some of the most horrifying online content.

Companies can also deliberately build online platform cultures that prioritize kindness, care, and mutual respect as opposed to others such as Gab, 4chan, 8chan, and Truth Social, which celebrate toxicity.

Improving algorithmic approaches would help large language models reduce the number of errors made around misidentification and differentiating context and language.

Finally, corporate culture at the platform level has an impact at the user level.

When ownership deprioritizes or even eliminates user trust and safety teams, for instance, the effects can be felt company-wide and risk damaging morale and user experience.

“Recent events in the industry show why it is so important to have human workers who are respected, supported, paid decently, and have some safety to make their own judgments,” Schmitt concludes.

The rapid advancement of natural language processing (NLP) models and large language models (LLMs) has enabled the development of new use-specific conversational agents designed to answer specific types of queries. These range from AI agents that offer academic support to platforms offering general financial, legal or medical advice.

Researchers at Hefei University of Technology and Hefei Comprehensive National Science Center recently worked to develop an AI-based platform that can provide non-professional, but potentially helpful, psychological support. Their paper, presented at the International Conference on Multimedia Modeling held in Amsterdam from Jan. 29 to Feb. 2, introduces EmoAda, a conversational system trained to engage in emotional conversations, offering low-cost and basic psychological support.

“Our paper originated from a concern over the increasing prevalence of psychological disorders such as depression and anxiety, particularly following the COVID-19 pandemic, as well as the significant gap in the availability of professional psychological services,” Xiao Sun, co-author of the paper, told Tech Xplore.

“This work builds on various research efforts, such as those by Fei-Fei Li et al on measuring depression severity from spoken language and facial expressions, those by Xiao Sun et al on multimodal attention networks for personality assessment, and the development of AI-based emotional support systems, like Google’s LaMDA and OpenAI’s ChatGPT.”

The primary objective of this recent study was to create a cost-effective psychological support system that could perceive the emotions of users based on different inputs, producing personalized and insightful responses. This system is not designed to substitute professional help, but rather to alleviate stress and assist users in increasing their mental flexibility, a feature that has been associated with good mental health.

“EmoAda is a multimodal emotion interaction and psychological adaptation system designed to offer psychological support to individuals with limited access to mental health services,” Sun explained. “It works by collecting real-time multimodal data (audio, video, and text) from users, extracting emotional features, and using a multimodal large language model to analyze these features for real-time emotion recognition, psychological profiling, and guidance strategy planning.”

EmoAda, the platform created by Sun and his colleagues, can detect a user’s emotions by analyzing various types of sensory data, including their voice, video footage of their face, and text. Based on these analyses, the system produces personalized emotional support dialogues, delivering them via text or via a digital avatar.

Based on a user’s needs and the difficulties they mention, the platform might suggest various activities that might be beneficial. Some of these activities are facilitated by content available on the EmoAda platform, such as guided meditation practices and music for relaxation or stress relief.

“When tested with real users, EmoAda has been shown to provide natural and humanized psychological support,” Sun said. “In these trials, we found that some users prefer conversing with AI because it can significantly reduce their anxieties about privacy breaches and social pressure. Engaging in dialogues with AI creates a safe, non-judgmental environment where users can express their feelings and concerns without fear of being judged or misunderstood. AI systems like EmoAda also offer round-the-clock support, free from time constraints, which is a significant advantage for users who need help at any given moment.”

In initial test trials, the researchers found that one of the most appreciated aspects of EmoAda is its anonymity. In fact, users often mentioned that they felt comfortable with sharing private information that they would find difficult to discuss with other people face-to-face.

In the future, this new AI-based system could be deployed as a basic support service for people who cannot afford professional psychological care or are waiting to access available mental health services. In addition, EmoAda could serve as an inspiration for other research groups, paving way for the development of other AI-based mental health-related digital platforms.

“Our next studies will focus on addressing current system limitations, including optimizing the multimodal emotional interaction large language model to reduce misinformation generation, improve model interference performance, reduce costs, and integrate a psychological expert knowledge base to enhance system reliability and professionalism,” Sun added.

© 2024 Science X Network