Being Just Reliable Enough

One Saturday morning, as I settled in on the couch for a nice do-nothing day of watching college football, my wife reminded me that I had agreed to rake the leaves after putting it off for the last two weekends. Being a good neighbor and not wanting another homeowners’ association (HOA) violation (and it being a bye week for the Longhorns), I grabbed my rake and went outside to work.

fall leaves

There were a lot of leaves. I would say my yard was 100% covered in leaves. I began to rake the leaves and with a modest effort I was able to collect about 90% of the leaves into five piles, which I then transferred into those bags you buy at Home Depot or Costco.

The yard looked infinitely better, but there were still plenty of leaves in the yard. I had the rake, I had the bags, I was already outside, and I was already dirty, so I went to work raking the entire yard again to get the remaining 10% I had missed in the first pass. This took about the same amount of time, but wasn’t nearly as fulfilling. My piles weren’t as impressive, and I was only able to get 90% of the remaining leaves into piles and then into bags, but I had cleared 99% of the leaves.

Still having plenty of daylight and knowing I could do better, I went to work on that last 1%. Now, I don’t know if you know this about leaves, but single leaves can be slippery and evasive. When you don’t have a lot of leaves to clump together to get stuck in the rake it may take two, three, sometimes four passes over the same area to get any good leaf accumulation into your pile. This third pass over the yard was considerably more time consuming, but I was able to get 90% of that remaining 1%. I had now cleared 99.9% of the leaves in my yard.

As I sat back and admired my now mostly leaf-free yard, I could see some individual leaves that had escaped my rake and even some new leaves that had just fallen from the trees. There weren’t too many, but they were there. Wanting to do a good job, I started canvassing the yard on my hands and knees, picking up individual leaves one by one. As you can imagine, this was very tedious and it took much longer to do the whole yard, but I was able to pick up 90% of the remaining 0.1%. I had now cleared 99.99% of the leaves in my yard.

The sun was starting to set and all that was left were mostly little leaf fragments that could only really be picked up by tweezers.

I went inside and asked my wife, “Where are the tweezers?” “Why do you need tweezers to paint the fence?” she asked. “Paint the fence?” I thought. Oh, yeah. I had also agreed to paint the fence today. I told her I hadn’t started on the fence yet and wouldn’t be able to do that this weekend because it was getting late and the Cowboys were playing the next day. She was not happy.

Yes, this story is ridiculous and contrived, but it demonstrates some good points that we apply to how we manage system reliability and new feature velocity at Indeed.

Where did I go wrong? 

It was way before I thought about getting the tweezers. When I started raking, my definition of a successfully raked yard was too vague. I did not have a service level objective (SLO) specifying the percentage of my yard that could be covered in leaves and still be considered well-raked by my clients.

Should I have defined the SLO?

I could have defined the SLO, but I might have based it on what I was capable of achieving. I was capable of picking up bits and pieces of leaves with tweezers until I had a 99.999% leaf-free yard. I could have also gone in the other direction (if it wasn’t a bye week) and determined that raking 90% of the leaves would be sufficient. 

SLOs should be driven by the clients who care about them 

The clients in my story are my HOA and my wife. My HOA cites me when my yard is only 50% raked for an extended period of time. My wife says she is happy when I rake 99% of the leaves once a year. For the SLO, we would take the higher of the two. I could have quit raking leaves after the second pass when I reached 99% and had time to paint the fence (depending on the SLO for the number of coats of paint).

But, I still did a good job, right?

I did, but I far exceeded my undefined SLO of 99% by two 9s, and yet I was not rewarded. Sadly, I was punished, because my wife didn’t care about the work I did on that remaining 1% and was upset that I didn’t have the time to meet my other obligation of painting the fence.

This brings us to the moral of the story:

We need to have the right SLOs and work to exceed them, but not by much. 

At Indeed, when our SLOs describe what our users care about, we avoid the effort of adding unnecessary 9s. We then use that saved effort to deploy more features faster, achieving a balance between reliability and velocity.

About the author

Andrew Ford is a site reliability engineer (SRE) at Indeed, who enjoys solving database reliability and scalability problems. He can be found on the couch from the start of College Gameday to the end of the East Coast game most Saturdays from September to December.

Do you enjoy defining SLOs that your clients care about? Check out SRE openings at Indeed!

Being Just Reliable Enough—cross-posted on Medium.

Indeed Open Source: All Things Open 2019 Speakers

We’re excited to have three Indeed representatives presenting at All Things Open this year. Join us in Raleigh, NC October 13-15 for engaging discussions.

Flyer for All Things Open shows a cartoon astronaut hitchhiking to Raleigh, North Carolina to get to the conference

Sustaining FOSS Projects by Democratizing the Sponsorship Process

Tuesday, October 15 | 10:45am | Room 201
Speaker: Duane O’Brien, Indeed head of open source

Within a given company, there are typically only a few people involved in deciding which FOSS projects and initiatives to support financially. This year we decided to change all that and democratize the decision making process. We set up an internal FOSS Sustainability Fund and invited everyone to participate in the process.

Sustaining FOSS Projects by Democratizing the Sponsorship Process examines how we got executive buy-in for the fund, set it up, and encouraged participation. It also explores the fund’s impact on our engineering culture.

Using Open Source Tools for Machine Learning

Tuesday, October 15 | 10:45am | Room 301A
Speaker: Samuel Taylor, Indeed data scientist

Machine learning can feel like a magic black box, especially given the wealth of proprietary solutions and vendors. This beginner-friendly talk opens the box. It reveals the math that underlies these services and the open source tools you can use in your own work. It introduces machine learning through the lens of three use cases:

  • Teaching a computer sign language (supervised learning)
  • Predicting energy usage in Texas (time series data)
  • Using machine learning to find your next job (content-based filtering)

You’ll walk away prepared to practice machine learning in the real world.

Your Company Cares about Open Source Sustainability. But Are You Measuring and Encouraging Upstream Contributions?

Tuesday, October 15 | 2:15pm | Room 201
Speaker: Dani Gellis, Indeed software developer

You encourage the behavior that you measure. If you want your company to help sustain the open source projects you depend on, start by measuring how your employees participate in those projects.

How many of your engineers contribute to projects your company consumes? Do they only open issues, or do they contribute code? Are they part of the conversation? Are your non-engineers also involved in the open source community?

Your Company Cares about Open Source Sustainability demonstrates how we use open source tools to measure the velocity of our employees’ open source contributions, as well as how Indeed chose these tools. It covers the evolution of our tooling as our open source program has grown. And it reveals our exciting new initiatives to promote sustainable contributions.

You’ll leave with new ideas for measuring and improving your organization’s contributions to open source projects.

Indeed Open Source: All Things Open 2019—cross-posted on Medium.

IndeedEng: Proud Supporters of the Open Source Community

At Indeed, open source is at the core of everything we do. Our collaboration with the open source community allows us to develop solutions that help people get jobs.

As active participants in the community, we believe it is important to give back. This is why we are dedicated to making meaningful contributions to the open source ecosystem.

We’re proud to announce our continuing support by renewing our sponsorship for these foundations and organizations.


Apache Software Foundation logoThe ASF thanks Indeed for their continued generosity as an Apache Software Foundation Sponsor at the Gold level.

In addition, Indeed has expanded on their support by providing our awesome ASF Infrastructure team the opportunity to leverage job listing and advertising resources. This helped us bring on new hires to ensure Apache Infrastructure services continue to run 24x7x365 at near 100% uptime.

We are grateful for their involvement, which, in turn, benefits the greater Apache community.

— Daniel Ruggeri, VP Fundraising, Apache Software Foundation


Cloud Native Computing Foundation logo

CNCF is thrilled to have Indeed as a member of the Foundation. They have been a great addition to our growing end-user community. Indeed’s participation in this vibrant ecosystem helps in driving adoption of cloud native computing across industries. We’re looking forward to working with them to help continue to grow our community.

— Dan Kohn, Executive Director, Cloud Native Computing Foundation


Open Source Initiative logoIndeed’s active engagement with open source communities highlights that open source software is now fundamental, not only for businesses, but developers as well.

Like most companies today, Indeed is a user of and contributor to open source software, and interestingly, Indeed’s research of resumes shows developers are too—as job seekers highlight open source skills and experience to win today’s most sought after jobs across technology.

— Patrick Masson, General Manager at the OSI


Outreachy logo

We’re so happy that Indeed continues to join our sponsors—making it possible for us to provide critical opportunities to people who are impacted by systemic bias, underrepresentation and discrimination—and helping them get introduced to free and open source software.

— Karen Sandler, Executive Director, Software Freedom Conservancy


Python Software Foundation logo

Participation in the PSF Sponsorship Plan shows Indeed’s support of our mission to promote the development of the Python programming language and the growth of its international community.

Sponsorships, like Indeed’s, fund programs that help provide opportunities for underrepresented groups in technology and shows support for open source and the Python community.

— Betsy Waliszewski, Python Software Foundation


We’re committed

Our open source initiatives involve partnerships, sponsorships and memberships that support open source projects we rely on. We work to ensure that Indeed’s own open source projects thrive. And we involve all Indeedians. This year we began a FOSS Contributor Fund to support the open source community. Anyone in the company can nominate an open source project to receive funds that we award each month.

We’re committed to open source. Learn more about how we do it.

IndeedEng Supports the Open Source Community—cross-posted on Medium.