This post was originally published on learningfromincidents.io.
Photo by Jared Erondu on Unsplash
As a Site Reliability Engineer (SRE) at Indeed, I often participate in the retrospective process that follows an incident. Retrospectives—in use at Indeed since late 2015—are a meaningful part of our engineering culture. I have never questioned their importance, but recently I was struck by shortcomings I saw in some retrospectives. For example:
- A retrospective meeting might use only ~30% of the allotted time.
- What is discussed might be gleaned from reading the incident ticket and retrospective document instead of attending the meeting.
- Too much focus is devoted to the conditions that “triggered” the incident.
- Signals used for deciding to hold a retrospective tend to direct focus toward incidents with high impact or high visibility.
- Were participants actually learning anything new? It became apparent to me that we were not using every incident to realize our full potential to learn.
I decided to explore why so that we could improve our process.
The typical retrospective
Retrospectives at Indeed are usually a one-hour discussion including up to several dozen participants. The meeting is open to anyone in the company, but usually participants have either been involved in the incident response or have a stake in the outcome.
Facilitators follow a prescribed process:
- Review the timeline.
- Review the remediation items in the template.
- Find owners for the remediation items.
- Open the room for questions.
Spotting opportunities for improvement
In Summer 2018 I visited one of our tech sites and was invited to several local retrospective meetings to discuss some recent incidents. As an SRE it wasn’t unusual for me (or members of my team) to be invited. I also had subject matter expertise in a technology related to the incidents.
The facilitators took about 5 minutes to review the timeline, spent 8-10 minutes reviewing the remediation items, and concluded with questions related to the specific technologies involved in the causal chain. I didn’t learn anything new. I could have gained the same information from reading the incident ticket and retrospective document. This was a rare opportunity when a unique and eager group of people gathered in a conference room ready to collaboratively investigate. Instead, we never achieved the full potential.
This result is not uniform across retrospectives. I’ve been present in retrospectives where the participants offered such rich detail that the conversation continued well beyond the one-hour time limit, culminating with a huddle outside of the conference room.
The facilitators for these particular retrospective meetings followed the process faithfully but had only utilized ~30% of the time. It was clear to me that the retrospective process itself needed improvement.
Nurturing a safety culture
To understand potential changes, I first solicited viewpoints on why we conduct retrospectives at Indeed. Reasons I heard are likely familiar to most software organizations:
- Find out what caused the outage
- Measure the impact
- Ensure that the outage never happens again
- Create remediation items and assign owners
These goals also reflect Indeed’s strong sense of ownership. When someone’s service is involved in an incident, there’s a concern that we were closer to the edge of failure than we thought we were. Priorities temporarily change and people are more willing to critically examine process and design choices.
It’s important to use these opportunities to direct efforts toward a deeper analysis into our systems (both people and technical) and the assumptions that we’ve made about them. These approaches to a different safety culture at Indeed are still relatively new and are evolving toward widespread adoption.
Recommendation: Decouple remediation from the retrospective process
One process change I recommend is around the creation of remediation items. The retrospective process is not necessary as a forcing function for driving accountability of finding and owning remediation items.
I consistently observe that the creation of remediation items occurs organically after Production is stabilized. Many fixes are obvious to teams in the hindsight following an incident.
I see value in decoupling these “after action” activities from the retrospective process for many reasons.
- The search for remediation items is often a tacit stopping point that halts further or deeper investigation.
- The accountability around owning remediation items should be tightly coupled to incident ownership.
- The retrospective process should be an optional activity. By making the retrospective process optional, teams that decide to engage in it are doing so because they see value in it rather than as an obligation or a checklist item.
- Participants are freed up to conduct a deeper investigation unencumbered by the search for remediation items and shallow explanations.
Recommendation: Lighten up the retrospective template
Another useful change is with the retrospective template itself.
Using retrospective templates can be a lot like filling out forms. The focus is directed toward completion of an activity rather than free exposition. A blank document encourages a different kind of sharing. I have witnessed incidents where responders were so motivated to share their thoughts and descriptions that they produced rich and detailed analysis simply by starting with a blank document.
If every incident is shaped like a snowflake, it’s impossible to develop a template that is equipped to capture its unique characteristics. A template constrains detail and triggers explanations through close-ended questions. A blank canvas is open-ended. A template is yet another tacit stopping point that hinders deeper analysis. I recommend that we apply templates to incident analysis, but that we use blank documents for the retrospective process.
Driving organizational change
I have learned a lot by working to drive change at Indeed as we’ve grown quickly. My efforts have benefitted from my tenure in the company, experience participating in hundreds of incidents, and connection to the literature. I have made headway but there is still a lot to do.
I attribute some of my progress so far to finding other advocates in the company and remembering to communicate.
Find advocates
Advocates are colleagues who align closely with my goals, acknowledge where we could be doing better, and share a vision of what could be. I had no trouble finding these advocates. They are colleagues who are willing to listen, have an open mind and have the patience to consider another perspective. I held numerous 1:1s with leaders and stakeholders across the organization. I found opportunities to bring these topics up during meetings. I gave tech talks and reached out to potential advocates whenever I visited one of our global Engineering offices.
Over-communicate
As much as I might think that I was communicating what I was working on, it was never enough. I found I had to constantly over-communicate. As I over-communicated and leveraged multiple media, I may have sounded repetitive to anyone in close proximity to my words. But this was the only way to reach the far edges of the organization who might not have otherwise heard me. Not everybody has time to read every email or internal blog post.
Looking ahead
Response to these changes has been largely positive. The focus during retrospectives is still anchored to the technological factors, when more attention could be paid to the human factors. I’m exploring different avenues for increasing the reach and effectiveness of these efforts. This includes working with our instructional design team to create a debrief facilitator program, communicating more often and more broadly, making more process changes, continuing to help teams produce and share high quality write-ups, and focusing on producing educational opportunities. At this point we’ve only scratched the surface and I’m looking forward to what we will accomplish.
About the author
Alex Elman is a founding member of the Site Reliability Engineering team at Indeed. He leads two teams: one that focuses on Resilience Engineering and one that supports the flagship Job Search product. For the past eight years Alex has been helping Indeed adopt reliability practices to cope with ever increasing complexity and scale. Follow Alex on Twitter @_pkill.