Facebook Machine Learning Tool to Check Terrorists Posts

0

In this post, you will learn about details on Facebook machine learning tool to contain online terrorists propaganda. The following topics are discussed in this post:

  • High-level design of Facebook machine learning solution for blocking inappropriate posts
  • Threat model (attack vector) on Facebook ML-powered solution

ML Solution Design for Blocking Inappropriate Posts

The following is the workflow Facebook uses for handling inappropriate messages posted by terrorist organizations/users.

  • Train/Test a text classification ML/DL model to flag the posts as inappropriate if the posts is found to contain words representing terrorist propaganda.
  • In production, block the messages which the model could predict as inappropriate with very high confidence.
  • Flag the messages for data analysts processing if the confidence level is not very high. These messages are prioritized based on likelihood estimate of the messages being inappropriate.

Threat Model (Attack Vector) on Facebook ML-powered Solution

The following lists down the threat model using which terrorists could carry out attacks on the claimed ML model (by Facebook) thereby compromising both systems’ integrity and systems’ availability. Here the assumption is that ISIS representatives of Islamic background/religion are trying to hack into the ML-powered system.

  • The hackers (hypothetically speaking, the users categorized as terrorists) get control of users’ account belonging to any non-Islamic country. Alternatively, the hacker from any non-Islamic country opens up an account on the Facebook.
  • The hacker starts posting encoded messages with words of interests interleaved and hard enough for the data analyst to flag the post as inappropriate.
  • The ML model gets trained with the encoded messages.
  • In future, the hackers (terrorists) start exchanging messages thereby compromising ML systems integrity as the inappropriate messages pass through the system. This is a classic example of false negatives. The following diagram represents the same.

    Facebook ML System Integrity Compromised

  • Let’s say, Facebook in future finds out about the encoded messages.
  • It makes it difficult to block these encoded messages with the primary reason that valid message will also get blocked. This would compromise ML system availability. In simpler words, this would mean that Facebook ML system would make system unavailable for users posting valid messages. The following diagram represents the same.

    Facebook ML System Availability Compromised

Summary

In this post, you learned about how did Facebook go about designing a machine learning solution to contain online terrorist propaganda. The challenge is to make sure that the model is retrained appropriate appropriately and look for adversary data sets entered into the system by hackers (the terrorist organization) to train the model inappropriately.

Ajitesh Kumar

Ajitesh Kumar

Ajitesh has been recently working in the area of AI and machine learning. Currently, his research area includes Safe & Quality AI. In addition, he is also passionate about various different technologies including programming languages such as Java/JEE, Javascript and technologies such as Blockchain, mobile computing, cloud-native technologies, application security, cloud computing platforms, big data etc.

He has also authored the book, Building Web Apps with Spring 5 and Angular.
Ajitesh Kumar

Leave A Reply

Time limit is exhausted. Please reload the CAPTCHA.