« all talks

RAD – How We Replicate Terabytes of Data Around the World Every Day

This talk was held on Wednesday, March 30, 2016 at 7:00pm

IndeedEng is back! After a hiatus in 2015, we’re back in a brand new office ready to tell some new stories. Indeed is growing and solving compelling problems with interesting technology, and we want to engage with the Austin tech community to share what we’ve learned. 

At Indeed, we use massive amounts of data to build our products and services. At first, we relied on rsync to distribute these data to our servers. This rsync system lasted for ten years before we started to encounter scaling challenges. So we built a new system on top of BitTorrent to improve latency, reliability, and throughput. Today, terabytes of data flow around the world every day between our servers. In this talk, we described what we needed, what we created, and the lessons we learned building a system at this scale.