Rainbird: How Twitter acquires, stores and uses the massive amount of data we produce

By MATT SCHULER, Digital Architect, Marx Layne

When you think about how many messages Twitter’s millions of users produce on a daily basis, it’s staggering.

Even at 140 characters, the massive amount of information has to be funneled through something, and now we know a bit more about what.

Yesterday at O’Reilly’s Strata 2011, Kevin Wiel gave a talk on “Realtime Analytics at Twitter.” Wiel is the Analytics Lead for Twitter.

From the Strata description, “Twitter is a realtime system, and it’s critical that our analytics be realtime as well. But with over 160 million users producing over 90 million tweets per day, we needed infrastructure that scaled horizontally in addition to being realtime. In this talk we’ll discuss the high-volume realtime analytics system we have built, as well as the benefits and challenges of this model over standard offline models. We’ll look at some products that we are building on top of this infrastructure, and discuss where we’re hoping to take this system in the future.”

His presentation shines a big light on how Twitter works with what it has. The slide show is embedded below, or you can download it here.

2017-09-19T12:19:21+00:00

Leave A Comment