Why Facebook Relies on A/B Testing?

What is A/B testing?

A/B testing is a strategy in marketing in which two versions, A and B, (the control and the treatment) are tested against each other. A/B testing, as the names implies, is a simple randomized experiment with two variants/versions, A and B, one of which version A might be currently used version (control) and, version B (treatment) is modified in some respect to study/test the users’ behavior. These tests are also called as split tests. These tests involve modification some of the following attributes of page to gaze/measure the users’ response:

• Page layouts
• Text colors
• Images
• Text fonts, placement
• Navigation

Following are some use-case scenarios for A/B testing:

• Advertisement effectiveness: A/B testing is effectively used for testing different ad versions to determine which ad has greater click-through-rate (CTR).
• Email campaign: A set of users could be sent email based on template A and another set of users could be sent email based on template B which is slight modification of template A from one or more of above attributes. One could then measure the users’ response to determine which email template is better and could use it for future purpose. Take a look at this page from mailchimp specifying the A/B testing offering to its customer.
• Purchase funnel of eCommerce website: On an ecommerce website , one could test different page attributes including those mentioned above, when user is trying to purchase one or more items using shopping feature.
• Compelling & engaging experience for online stores: Following could be some of the attributes that one could test using A/B testing:
  • Call-to-action (CTA) buttons
  • Page layout/navigation
  • Checkout process
  • Promotional offers
  • Catalogs (Product selection)
  • Product images

Why A/B testing is used by Facebook?

• Facebook Ads: Facebook supports A/B testing for user-created ads. This essentially means that users could create multiple version of ads and run them at the same time to measure the effectiveness of these ads in terms of number of page likes, CTR etc. Once users determine the most effective ads, he then runs those ads and cancel other remaining ads. Read more on A/B testing and Facebook ads on this page. One could do A/B testing with following data when creating the ads:
  • Ad body consisting of image, text etc. One could create different ad versions having different images and text.
  • Users’ Targeting : One could create different categories of users’ selection based on their age, location etc.
• Facebook mobile app testing & AirLock: Facebook releases new/enhanced features for its mobile app after every 4 weeks. With every release, it need to determine some of the following to figure out strategy for future releases:
  • How new features performed
  • Whether the fixes improved performance and reliability
  • Improvements to the user interface change how people use the app and where they spend their time

  All of the above objectives are achieved using the A/B testing infrastructure, namely AirLock, that Facebook engineering team created in-house. Airlock is a testing framework that lets Facebook Engg. team compare metric data from each version of the app and the various tests, and then decide which version to ship or how to iterate further. With AirLock, Facebook achieves some of the following goals of A/B testing:

  • Navigation model
  • 10-15 variations tested at the same time across millions of users

Who should adopt A/B testing?

Following are different class of companies who should consider A/B testing:

• eCommerce business
• Auction websites
• Businesses relying heavily on mobile apps. Example, Facebook, Gmail, Yahoo  etc.

Startups innovating with A/B testing

Following are different startups which are innovating with A/B testing and providing tools for websites to run such tests:

• Optimizely
• Appiterate