At Indeed, we tackle interesting and challenging problems, at scale. We move from idea to implementation as fast as possible. We ship incremental changes, pushing code to production frequently. Our small releases reduce risk and increase quality.
But before we work on any solution, we ask: how will we measure success? This question keeps our solutions focused on what matters — measurable results.
Our approach to software development might be called “measure-learn-evolve.” Our teams employ techniques from various software development methodologies, but no single published methodology rules. We collaborate, we iterate, and we measure. Sometimes we succeed, sometimes we fail, but we are always learning.
We don’t view process implementation and improvement as success. Process is a means to an end. Process doesn’t deliver a successful product. (People do.) Process doesn’t provide talent and passion. (People do.) But the right process and tools can help people do those things and provide predictable mechanisms for:
- planning what we need to do and setting relative priorities
- communicating what we are doing or might do
- remembering what we’ve done
- managing our risk
We use Atlassian’s Jira to implement these mechanisms. In Jira, we propose ideas, define requirements, and plan projects. We document dependencies, track work, and manage releases. We describe experiments and record results. Customizing Jira to our needs has helped us collaborate on success metrics and maintain our engineering velocity.
It wasn’t always this way. We started simple. We were a startup and we focused on getting stuff done, quickly.
As we grew, we didn’t want to lose this focus on getting things done quickly and with quality. But our ad hoc process was neither repeatable nor predictable. Inconsistencies abounded and we were not creating a memory for the future. So we began to model our development process in Jira.
Customizing Jira
We have our own Jira issue types, workflows, fields, and roles. These customizations allow us to plan, communicate, and deliver our software in the way we want.
Linking custom project types
We use two types of Jira projects for product development: a “planning project” that corresponds to the product, and an “engineering project” that corresponds to a deployable application or service.
Our planning projects contain Initiative and Experiment issues. We use the Initiative type to capture goals, plans, and success metrics for a product change. We plan product initiatives each quarter, and we iterate on them throughout the quarter. As part of that iteration, we use the Experiment type to describe specific ideas we want to test to optimize our products.
The engineering projects include issues that detail the implementation necessary for the initiatives and experiments. Each deployable application or service has a corresponding engineering project. Issue links connect related issues to one another. Jira provides multiple types of bi-directional links. The following table gives examples of how we use them.
| Issue link | Description |
|---|---|
| incorporates / incorporated by | Product initiatives incorporate engineering project issues. |
| depends upon / depended on by | Issues can depend upon on other issues. This can model feature development dependencies or deploy order dependencies, for example. |
| references / referenced by | An issue for a functional regression references the project issue that introduced the bug. |
Issue types and workflows
We use Jira’s standard issue types: Bug, Improvement, New Feature. The workflow for these standard issue types is a slight modification of a typical Jira workflow:
- We create an issue and assign it to a project lead. The issue transitions to a Pending Triage state.
- If we can target work to a near-term release, we triage the issue, setting its Fix Version and assigning it to a developer. The issue then moves to Pending Acceptance. We move other issues to On Backlog.
- The developer accepts the issue, moving it to Accepted when they make a plan to start work.
- When the code is complete, the developer resolves the issue, moving it to Pending Review.
- After code review, we transition the issue to Pending Merge.
- When we’re ready to create a release candidate, we merge changes into the release branch and deploy to the QA environment, transitioning the issue to Pending Verification.
- The QA analyst verifies the work and either reopens the issue or verifies it, transitioning it to Pending Closure.
- After we verify all issues in a targeted release, we can release the build to production and move all the issues to Closed.
We also use custom issue types to model our process. In a previous post, we described the ProTest issue type (short for Proctor Test). We use this custom issue type to request new Proctor A/B tests or to change test allocations.
We have another custom issue type and associated workflow for localization. As we continue to grow internationally, we need a localization process that doesn’t slow us down. Coordinating with many translators can be a challenge, so we model our translation process in Jira. Our Explosion issue type incorporates an issue for each target translation language. The workflow follows:
- We create an issue with English strings that require translation.
- We triage the issue and submit it for review.
- When the strings are ready to be translated, an automated step creates one Translation issue for each target language and links them all to the Explosion issue.
- Each “exploded” issue follows its own workflow: Accept, Resolve, Verify and Close.
- When all Translation issues are closed, we verify and close the Explosion issue.
The Explosion and Translation custom issue types and workflows help streamline a process with many participants. Because we triage by language and feature, translation issues do not block the release of an entire feature. Using Jira also allows us to integrate machine translation and outside translation services.
Team triage
Many of our development teams use dashboards and agile boards in Jira for easy access to issues associated with a product. During routine triage meetings, product development teams use these tools to prioritize and distribute development work.
Closing the memory loop
Each code commit in Git is traceable to a corresponding issue in Jira. Further, if the referenced Jira links to the initiative, the trail leads all the way to the initiative. This means that an engineer can review any code commit and follow the trail in Jira to understand all related implementation details, requirements, and business motivation.
Production deploys
Deploying code to production requires clear communication and coordination, and our Deploy issue type helps us track this process. Using Jira to track deploys results in smooth handoffs and transparency for all stakeholders.
A deploy ticket is associated with each Fix Version and has a unique workflow that facilitates communication for moving artifacts through the build and release process. We use issue links to document all sysadmin tasks necessary for a successful deployment. The deploy ticket has the same fix version as the other issues in the release.
Most teams plan their work weekly but deliver to production as they complete the work. On some regular cadence – semi-weekly, daily, or more often – the release manager creates a release candidate from all open merge requests. We developed an internal webapp that coordinates across Git (branch management), Jira (code changes and deploys), Crucible (code review), and Jenkins (build). Status changes to the deploy ticket trigger issue reassignments, promoting smooth handoffs.
This approach provides our teams with the information they need to assess and manage risk for their production releases. The QA analyst can better understand potential regressions that a change may cause. The release manager can have a holistic view of what’s changing and quickly react when issues arise. And small releases make bug investigation more straightforward.
Working in the open
Jira enables effective, efficient collaboration for our software development and deployment process. We use it to clarify requirements, discuss implementation choices, verify changes, and deploy to production.
Across teams and up and down the organization, our use of Jira provides transparency into the work that is getting done. By working in the open, we can achieve a shared understanding of plans, progress, and challenges for hundreds of active projects and initiatives.
Do what makes sense for you
Methodology and process only help when they provide repeatable and predictable mechanisms for planning, communication, and delivery. Jira has helped us establish these mechanisms.
Try to avoid taking a methodology “off the shelf” and implementing it. And don’t depend on tools to solve your problems. Instead, think about how your team needs to plan, communicate, and deliver. Then, define the best process and tools that serve your needs. Iterate on your process as needed. And stay focused on what really matters: success.
Adapted from Jack Humphrey’s presentation at Keep Austin Agile 2014.