Lessons from Evernote Usage of Apache Thrift Framework

What is Evernote?

As per Wikipedia…Evernote is a suite of software and services designed for notetaking and archiving. A “note” can be a piece of formatted text, a full webpage or webpage excerpt, a photograph, a voice memo, or a handwritten “ink” note….

Evernote supports a number of operating system platforms (including OS X, iOS, Chrome OS, Android, Microsoft Windows, Windows Phone, BlackBerry, and webOS) and also offers online synchronisation and backup services. The diagram below illustrates Evernote support for multiple devices:

Look at this scenario. I, being an active user of Evernote, access and update same set of notes from desktop (client, Chrome browser), and mobile devices (my phone and iPad) as I access internet from one of these devices at different time of the day.  And, this is, primarily, why I love using Evernote is the flexibility that it provides to the users like me to work on my same set of notes using different devices anytime, anywhere.

The important point to note is the support for both “thin” clients (like web browsers) and “thick” synchronizing clients like the ones you need to install on your desktop. With such support, the end users are empowered to install & configure Evernote client applications on different devices and manage their notes efficiently. Following are some of the key characteristics of Evernote client application:

• Cross-platform: Service interfaces are defined using Thrift Interface Definition Language (IDL) and Apache Thrift is used to generate client and server stubs to work with different programming languages.
• Forward/backward compatibility: Thrift provides flexibility in modifying/adding structures, fields, service methods, and function parameters without breaking existing clients. This is one of the most powerful features of Thrift.
• Binary-efficient: Binary data representing many embedded images in the notes (resulting as an outcome of synchronizing between different clients) needed to be passed in equivalent size across the wire.
• Native bindings: With thrift, one is not required to take care of parsing/marshaling code on every different client.

The question, thus, arises is how does Evernote manage to support so many client applications on different OS platforms? What is the underlying integration technology? 

Apache Thrift Makes it All Possible!

As per Apache thrift website, The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages. Simply speaking, Thrift is an interface definition language, a code generation tool and a set of libraries that take care of serialization and implement the transport protocol. 

With Apache thrift, Evernote was able to fulfill following key application requirements:

1. Server-to-server communications in the backend
2. Client-server synchronization over the Internet
   • Handle heterogeneous data with big binary blobs in a smooth manner
   • Reliably support backward/forward compatibility
   • Support the application on multiple platforms

At a very high level, different Evernote client applications invokes Evernote Service API for managing notes on the server. To achieve something similar like Evernote, all that one needs is a set of API and client and server side components generated using Thrift in following manner:

• Define service definition file using Thrift IDL
• Use Thrift compiler to achieve following:
  • Generate service interface code (for the server) in appropriate programming language. In case of Evernote, it is Java that runs on server.
  • Generate stubs (for the client) in different programming languages for different OS platform.

Who all uses Apache Thrift?

Apart from Evernote, following are some of the companies/projects that uses Thrift framework for different purposes:

• Facebook
• Twitter
• Cassandra project
• Hadoop (Supports access to its HDFS API through thrift bindings)
• HBase (Leverages thrift for cross-language API)
• LastFM
• DoAT
• Scribe
• ThriftDB
• Junkdepot

Lessons Learnt…When to use Apache Thrift?

So, when would one want to use framework like Apache thrift?

• If the requirements look similar to Evernote where there are multiple different client applications to be written using different programming languages, Thrift may be one of the prominent options to evaluate.
• If there are complex data model with big binary attachments to be transported to the server, Thrift may prove to be a good choice.
• If you got server side applications written using different programming languages which need to talk internally with each other, Thrift may be a good choice for evaluation.
• If, however, the application that has few APIs and, need to exchange small/fewer messages may just live with REST based integration.

Great Thrift Links

• Thrift: The Missing Guide
• http://jnb.ociweb.com/jnb/jnbJun2009.html
• https://blog.evernote.com/tech/2011/05/26/evernote-and-thrift/