Getty Images | SOPA images
One of the creators of the Botometer — a web tool Elon Musk used to estimate Twitter’s spam rate for a lawsuit — has reportedly said Musk’s calculation “means nothing.” Kai-Cheng Yang, a Ph.D. candidate at Indiana University, “questioned the methodology used by Mr Musk’s team and told the BBC they had not approached him before using the tool,” according to a BBC article today.
A Musk lawsuit on Aug. 4 claimed that a Botometer analysis of Twitter fire hose data in the first week of July “shows that during that period, 33 percent of exposed accounts were responsible for fake or spam accounts.” But as Yang noted, the Botometer gives scores from 0 to 5 — with 5 being the most bone-like — and Musk’s court file didn’t mention where he drew the line between human and bone.
“To estimate the prevalence” [of bots] you have to pick a threshold to lower the score,” Yang told the BBC. “If you change the threshold from a three to a two, you get more bots and fewer people. Details clear,” Musk “has the freedom to to do what he wants. So the song means nothing to me,” Yang said.
“Technically, you can pick any threshold you want and get any result you want,” Yang said in an earlier interview with Yahoo. The Botometer is a project of the Observatory on Social Media and the Network Science Institute at Indiana University.
Botometer rated Musk as a likely bot
The Botometer itself once indicated “that Elon Musk’s own Twitter account was likely a bot and scored it 4/5,” as Twitter noted in a lawsuit. Musk’s Botometer score reportedly fluctuated between 0.5 and 4, showing that the tool rates Musk as humanoid on some days and more bony on others.
Twitter also pointed out that Musk and his team “did not state what score they use to conclude that an account is spam, so their claim is not verifiable.” Twitter further noted that an account can be a bot without being what the company considers a fake account or spam. Twitter gave examples such as bots “that report earthquakes as they happen or updates about the weather.”
Other types of legitimate accounts may be seen as likely bots by the Botometer. The Botometer gave my own verified Twitter account a bot score of 3 out of 5 today, and it rated the verified Ars Technica account 3.6 out of 5.
The FAQ of the Botometer website warns that any account above a certain number will not be called a bot. “It’s tempting to set an arbitrary threshold score and consider anything above that number a bot and anything below a human, but we don’t recommend this approach… We think it’s more informative to look at the distribution of scores across a sample of accounts,” says the FAQ.
Yang Surprised Musk Didn’t Make a Better Tool
Yang also spoke to CNN recently, expressing his surprise that Musk used the Botometer instead of creating something more accurate. “To be honest, you know, Elon Musk is really rich, right? I assumed he would spend money hiring people to build a cutting edge tool or methods on his own,” Yang told CNN.
The Botometer is best used “in addition to, not replacing, your own judgment,” says the tool’s FAQ, noting that “humans and machines have different strengths when it comes to pattern recognition. Some” clearly ” According to a human observer, bot/human accounts will fool a machine-learning algorithm. For example, Botometer sometimes categorizes ‘organizational accounts’ as bot accounts. Likewise, an algorithm can confidently classify some accounts that people have difficulty with.”
Twitter sued Musk in the Delaware Court of Chancery after trying to backfire on his commitment to buy the company for $44 billion. Musk has defended his bid to break the merger agreement by questioning Twitter’s public disclosure that less than 5 percent of its revenue-generating daily active users (mDAU) are spam or fake.
Twitter defends the accuracy of its estimates, saying they are based on “multiple human assessments (in replica) of thousands of randomly selected accounts each quarter using both public and private data.” Twitter also says Musk has no right to leave the merger agreement based on the number of spam accounts.
Musk has plans for a more thorough spam analysis, according to his court. Defendants’ experts are continuing their analysis even now and, pending production of additional data by Twitter (including ‘private’ data that Twitter makes available to its human reviewers and claims it is necessary to reduce the reported less than -5 percent spam and false user numbers), plan to conduct a more comprehensive analysis and expect to present updated estimates and findings in expert reports and at trial,” Musk’s attorneys wrote.
