RAD – How We Replicate Terabytes of Data Around the World Every Day
This talk was held on Wednesday, March 30, 2016 at 7:00pm
IndeedEng is back! After a hiatus in 2015, we’re back in a brand new office ready to tell some new stories. Indeed is growing and solving compelling problems with interesting technology, and we want to engage with the Austin tech community to share what we’ve learned.
At Indeed, we use massive amounts of data to build our products and services. At first, we relied on rsync to distribute these data to our servers. This rsync system lasted for ten years before we started to encounter scaling challenges. So we built a new system on top of BitTorrent to improve latency, reliability, and throughput. Today, terabytes of data flow around the world every day between our servers. In this talk, we described what we needed, what we created, and the lessons we learned building a system at this scale.
Speakers
Jason Koppe is a Systems Administrator who has been with Indeed since late 2008. He's worked on infrastructure automation, monitoring, application resiliency, incident response and capacity planning.
Julie Scully started at Indeed in 2009 as a Software Engineer and currently is an Engineering Manager for teams in Austin and Tokyo. In Austin, she leads teams that build scalable, efficient, and resilient solutions for job data processing and data distribution.