Top Twitter Open Source Projects You Must Know

Twitter engineers use, contribute to and release a lot of open source software and their GitHub repo is a proof of that. Twitter has 131 public repos, maintained by 116 active twitter members around the globe. Twitter as such started as a simple Ruby on Rails application but soon realized that – to meet the scale demand of twitter, they need to re-invent and revamp the entire platform. And while doing that, they had implemented and open sourced many great projects. Twitter also maintains an open source community Twitter handle @TwitterOSS.

Top Twitter open source projects are:

Scalding

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Finagle

Finagle is used in production at Twitter (and many other organizations), and is being actively developed and maintained. Finagle is an extensible RPC system for the JVM, used to construct high-concurrency servers. Finagle implements uniform client and server APIs for several protocols, and is designed for high performance and concurrency. Most of Finagle’s code is protocol agnostic, simplifying the implementation of new protocols.

Finatra

Finatra is a sinatra-inspired agile web framework for scala, it runs on top of Finagle.

Ambrose

Twitter Ambrose is a platform for visualization and real-time monitoring of MapReduce data workflows. It presents a global view of all the map-reduce jobs derived from your workflow after planning and optimization. As jobs are submitted for execution on your Hadoop cluster, Ambrose updates its visualization to reflect the latest job status, polled from your process.

Parquest

Parquet is a columnar storage format that supports nested data. Parquet metadata is encoded using Apache Thrift. Twitter created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem.

Summingbird

Summingbird is a library that lets you write MapReduce programs that look like native Scala or Java collection transformations and execute them on a number of well-known distributed MapReduce platforms, including Storm and Scalding.

Bootstrap

Bootstrap is a well-known web UI framework used by thousands of companies today. It is a sleek, intuitive and powerful mobile first front-end framework for faster and easier web development.

Bower

Bower is a package manager for the web originally created at Twitter and adopted by a large community. It offers a generic, un-opinionated solution to the problem of front-end package management. It exposes the package dependency model via an API, that can be consumed by a more opinionated build stack. Bower runs over Git and is package-agnostic. A packaged component can be made up of any type of asset and use any type of transport.

Flight

Flight is a lightweight, component-based, event-driven JavaScript framework that maps behavior to DOM nodes. It was created at Twitter, and is used by the Twitter.com and TweetDeck web applications.

Typeahead

Inspired by twitter’s autocomplete search functionality, typeahead.js is a flexible JavaScript library that provides a strong foundation for building robust typeaheads.

Twemcache

Twemcache  is the Twitter Memcached. Twemcache is based on a fork of Memcached v1.4.4 that has been heavily modified to make to suitable for the large scale production environment at Twitter.

FlockDB

FlockDB is a distributed graph database for storing adjacency lists. FlockDB is much simpler than other graph databases such as neo4j because it tries to solve fewer problems. It scales horizontally and is designed for on-line, low-latency, high throughput environments such as websites. Twitter uses FlockDB to store social graphs (who follows whom, who blocks whom) and secondary indices. As of April 2010, the Twitter FlockDB cluster stores 13+ billion edges and sustains peak traffic of 20k writes/second and 100k reads/second.

Check out our other articles on open source projects here.