For years, hashing technology has allowed platforms to automatically detect child sexual abuse material (CSAM) to prevent children from being retraumatized online. However, quickly detecting new or unknown CSAM remained a greater challenge for platforms as more and more new victims fell victim. Now AI may be ready to change that.
Today, a leading child safety organization, Thorn, in partnership with a leading cloud-based AI solutions provider, Hive, announced the release of an AI model designed to flag unknown CSAM upon upload. It is the first use of AI technology that aims to expose unreported CSAM on a large scale.
An extension of Thorn's CSAM detection tool, Safer, the new “Predict” feature uses “advanced machine learning (ML) classification models” to detect “new or previously unreported CSAM and child sexual exploitation behavior (CSE), allowing A risk score is generated to make human decisions easier and faster.”
The model was trained in part using data from the National Center for Missing and Exploited Children's (NCMEC) CyberTipline, using real CSAM data to detect patterns in malicious images and videos. Once a suspicion of CSAM is spotted, a human reviewer stays on top to monitor. It could potentially be used to investigate suspected CSAM rings spreading online.
Of course, mistakes can be made, but Hive CEO Kevin Guo told Ars that extensive testing has been conducted to significantly reduce the number of false positives or negatives. While he wouldn't share statistics, he said platforms wouldn't be interested in a tool where “99 out of a hundred things the tool flags are not correct.”
Rebecca Portnoff, Thorn's vice president of data science, told Ars that it was a “no-brainer” to work with Hive on Safer. Hive offers content moderation models used by hundreds of popular online communities, and Guo told Ars that platforms have consistently requested tools to detect unknown CSAM, much of which currently lingers in blind spots online because the hash database will never reveal it.
