Jigsaw and Google using machine learning to identify toxic online comments

Formerly Google Ideas, Jigsaw is an Alphabet company aimed at using technology to “make people in the world safer.” Their latest project uses machine learning to identify toxic online comments and is intended to help foster harassment-free discussions.

In partnership with Google, Perspective is built with news organizations and readers in mind. One study notes that nearly half of Americans have experienced online harassment, with approximately 140 million people in the US being affected by it. For publications that still have comments, manual curation takes money, labor, and time.

Perspective reviews and scores comments based on how similar they are to ones previously marked toxic. It also takes into account messages that are “likely to make someone leave a conversation.” The model was trained by sifting through hundreds of thousands of comments previously labeled by human reviewers, with more data improving detection.

The technology was built using TensorFlow on Google’s Cloud Machine Learning Platform. Besides expanding to other languages, future improvements will allow models to identify unsubstantial and off-topic comments.

The API can be implemented in a multitude of ways. Straightforward solutions have Perspective flagging comments for human moderators or allowing readers to manually sort. A more novel idea has the machine learning tool provide immediate feedback to allow commenters to see the “potential toxicity of their comment as they write.”

Working with The New York Times, Perspective has allowed the paper’s moderation team to sort through comments more quickly. Labeled as an early-stage technology, publishers in Google’s Digital News Initiative have access to the API, with others being able to request access.