IndeedEng Culture: Make Better Mistakes

In the previous post, I described the importance of keeping teams independent. In this post, I talk about how Indeed encourages teams to turn mistakes into learning.

In most organizations, mistakes are never part of the plan. Plans are always designed to succeed. Plans never contain a chapter for “what if we are totally wrong about this or that.” Most of the time, by sheer force of habit; and sometimes, out of someone’s ego.

This idea is so brilliant that it cannot fail. Let’s rally everybody around it and celebrate how awesome it is. If it fails, let’s blame the project owner; it is certainly their fault. Let’s blame the engineers; they were not smart enough to understand the requirements or solve their technical problems. Let’s blame marketing or sales; these people had the product that the competition wanted and they just blew it.

Look back at how many mistakes NASA made before sending someone on the moon. Look at how many rockets SpaceX lost before its first successful return. These organizations succeed not because they don’t make mistakes, but because they have defined a process on how to make them and learn from them.

At Indeed, we turn making mistakes into a process, so that they become learnings.

  1. Identify the assumptions. Instead of residing in an echo chamber, discuss your idea with those who think it’s doomed to failure. Do not try to convince each other; it rarely happens. Instead, try to agree on the smallest experiment possible, which will scientifically demonstrate where the truth lies.
  2. Use a test framework. Comparing before and after is usually what people do, but it has its flaws: what about external factors? How can we run several experiments at the same time? We use A/B testing for everything we ship (except high priority bugs).
  3. Take single steps as much as possible. Change one thing at a time. Avoid having multiple assumptions or variables in the same experiment.
  4. Do not blame. The only true mistake is to not have learned anything. When something does not work, draw the conclusions and move on to the next experiment. If you fail fast enough, the mistakes are not personal setbacks for careers.

What’s next?

In the next post, I describe how to manage consolidation debt, a type of technical debt that expresses itself as redundancies and inconsistencies across your products.  

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

IndeedEng Culture: Keep Teams Independent

Editor’s note: Five years ago we launched this blog to talk about the technology we work on every day. Since then, we’ve grown more than ten times and continue to build the software that makes Indeed the #1 job site worldwide. To celebrate, we’re proud to host a week-long series of posts by Indeed Engineering Manager James Dingle. These posts highlight some core aspects of what makes Indeed Engineering a great place to work.

Word cloud based on all 5 years of IndeedEng blog content

During my year-and-a-half tenure at Indeed, I have discovered a new way to accomplish business goals with software. These principles and elements of Indeed’s culture are quite surprising; if you didn’t see them in action, you might believe any company trying them would be doomed to failure. Quite the opposite is true. While some of these principles are deliberate, some of them happened organically. This series describes my perspective on the principles of our engineering culture at Indeed.

Step One: Keep teams independent

Usually, in large organizations, the company defines a grand vision and then breaks it down into products, stories, features, and tasks. These tasks are transverse, and teams commit to other teams on deliverable dates.

When too many teams depend on each other, hitting dates becomes more important than the actual value delivered. As the deadline approaches, teams start to cut out pieces of their planned work, until the product attractiveness becomes lukewarm, its features unbalanced, and its user experience confusing.

The enterprise culture suffers, too. When teams slip on their due dates, the org can drift to finger-pointing and defensive management. In most software products, these dates are not bound to any external event, but some managers still behave as if the project is date-driven.

Engineering new products means a great deal of uncertainty about the difficulties ahead and how much time it will take to overcome them. Beyond a certain size, adding a buffer to estimates does not guarantee success. This is why smaller, independent teams are so important.

Independent teams (and uncertain dates) may frighten executives who like big milestones on a calendar. They might believe milestones are a source of clarity, motivation, and accountability, internally and externally. They might think that if the product turns into a flow of sand of features, they will not control it. They might think that if there are no dates, people will become lazy, wander eternally and never ship anything. But in reality, employees have multiple reasons to stay realistic.

Independent teams are beneficial for the team and for managers.

For the team, independence ensures:

  • less friction
  • more identification with the product by contributors, which in turn triggers more motivation and initiative
  • most decisions by the team, where context is better understood

For executive management, an independent team:

  • is accountable for the true business impact it delivers
  • diminishes the risk of a global impact on the company’s strategy if the team fails or is delayed
  • can take more risks

What you need to keep teams independent:

  1. An efficient deployment pipeline so teams can push or roll back their bits as they please.
  2. Goals and metrics only they can move.
  3. A test framework to evaluate the impact of new features and experiments.
  4. An efficient data collection and analysis pipeline.

Indeed has focused on developing the four abilities above while growing rapidly. Goals come top-down, and innovation rises bottom-up.

What’s next?

In the next post, I’ll explore the importance of making better mistakes.

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

Indeed Reads: October 2017

The Indeed Engineering blog lets us write about the technologies we get to work with every day. We talk about what works, what sometimes doesn’t work, and how we’re scaling our technologies as a fast growing company. We participate in industry-wide conversations through this blog. We also read a lot of other tech blogs.

Indeedians at the Austin Tech Campus

Here are a few interesting pieces recommended by Indeed engineers:

Engineering Personas

by Casey Rosenthal

We know that sales and marketing research often includes user personas for building software. This article looks at five engineering personas that also can help you with hiring decisions. These personas aren’t intended to define people, but they provide insight into the kinds of behaviors and characteristics that would most benefit your team.

Read post


Gray Failure

by Adrian Colyer

Working from the paper “Gray failure: the Achilles’ heel of cloud-scale systems” by Huang et al., Colyer explores the idea that breakdowns and performance anomalies in cloud-scale systems are often caused by gray failures, subtle failures perceived differently by various components of a system. Huang et al., describe these failures: “One entity is negatively affected by the failure and another entity does not perceive the failure.” Colyer takes this observation of gray failures and asserts that when we fail, we should fail properly.

Read post


How To: S3 Multi-Region Replication (And Why You Should Care)

by Jessica Lucci

Indeed’s Jessica Lucci provides step-by-step instructions to create a replica set by combining Amazon’s S3, SNS, SQS and Lambda technologies. As Amazon only provides two-region bucket replication, deploying your own replica set allows you to replicate data across as many regions as you need. A custom replica set also allows you to customize replication behavior via your Lambda functions.

Read post


The Pitfalls of A/B Testing in Social Networks

by Brenton McMenamin

A/B testing can be the perfect tool for most testing situations. This method helps determine whether a feature positively affects user behavior in a given test group. But if the purpose of your site is to get users to connect with as many other users as possible, like on the dating site OkCupid, keeping the feature away from your control group can lead to unreliable results. Tests for user-to-user features, such as chat or video, need interaction with as many people as possible to verify effectiveness. When A/B testing involves social interactions, it’s highly likely you’ll need to rethink and restructure your experiment methodology.

Read post


How to Do Code Reviews Like a Human (Part One)

by Michael Lynch

Code reviews are more than a technical process used to help identify bugs; they’re also social interactions that, when done mindfully, can improve the process itself. Delivering critical feedback is rarely easy, and when the feedback is focused on a technical subject, our instinct can often be to provide technical responses without considering the human interaction involved. But a human wrote the code that’s being reviewed, and the best way to iterate with that human is to follow guidelines that help you to be constructive and avoid miscommunications.

Read post


Floating Point Visually Explained

by Fabien Sanglard

Floating-point arithmetic is one of the trickier concepts in computer programming. This datatype has been called both esoteric and essential. To better understand floating points, Sanglard proposes visualizing the sections of the mathematical notation of a floating-point number (sign, exponent, and mantissa). Then, instead of exponent, Sanglard proposes thinking of a window between two consecutive power of two integers, and instead of a mantissa, thinking of an offset within that window. Redefining the components of a floating-point number in a more visual way provides a clearer path to explaining how this datatype works.

Read post

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone