Forget Methodology — Focus on What Matters

At Indeed, we tackle interesting and challenging problems, at scale. We move from idea to implementation as fast as possible. We ship incremental changes, pushing code to production frequently. Our small releases reduce risk and increase quality.

But before we work on any solution, we ask: how will we measure success? This question keeps our solutions focused on what matters — measurable results.

Our approach to software development might be called “measure-learn-evolve.” Our teams employ techniques from various software development methodologies, but no single published methodology rules. We collaborate, we iterate, and we measure. Sometimes we succeed, sometimes we fail, but we are always learning.

We don’t view process implementation and improvement as success. Process is a means to an end. Process doesn’t deliver a successful product. (People do.) Process doesn’t provide talent and passion. (People do.) But the right process and tools can help people do those things and provide predictable mechanisms for:

  • planning what we need to do and setting relative priorities
  • communicating what we are doing or might do
  • remembering what we’ve done
  • managing our risk

We use Atlassian’s Jira to implement these mechanisms. In Jira, we propose ideas, define requirements, and plan projects. We document dependencies, track work, and manage releases. We describe experiments and record results. Customizing Jira to our needs has helped us collaborate on success metrics and maintain our engineering velocity.

It wasn’t always this way. We started simple. We were a startup and we focused on getting stuff done, quickly.

As we grew, we didn’t want to lose this focus on getting things done quickly and with quality. But our ad hoc process was neither repeatable nor predictable. Inconsistencies abounded and we were not creating a memory for the future. So we began to model our development process in Jira.

Customizing Jira

We have our own Jira issue types, workflows, fields, and roles. These customizations allow us to plan, communicate, and deliver our software in the way we want.

Linking custom project types

We use two types of Jira projects for product development: a “planning project” that corresponds to the product, and an “engineering project” that corresponds to a deployable application or service.

Our planning projects contain Initiative and Experiment issues. We use the Initiative type to capture goals, plans, and success metrics for a product change. We plan product initiatives each quarter, and we iterate on them throughout the quarter. As part of that iteration, we use the Experiment type to describe specific ideas we want to test to optimize our products.

The engineering projects include issues that detail the implementation necessary for the initiatives and experiments. Each deployable application or service has a corresponding engineering project. Issue links connect related issues to one another. Jira provides multiple types of bi-directional links. The following table gives examples of how we use them.

Issue link Description
incorporates / incorporated by Product initiatives incorporate engineering project issues.
depends upon / depended on by Issues can depend upon on other issues. This can model feature development dependencies or deploy order dependencies, for example.
references / referenced by An issue for a functional regression references the project issue that introduced the bug.

Issue types and workflows

We use Jira’s standard issue types: Bug, Improvement, New Feature. The workflow for these standard issue types is a slight modification of a typical Jira workflow:

  1. We create an issue and assign it to a project lead. The issue transitions to a Pending Triage state.
  2. If we can target work to a near-term release, we triage the issue, setting its Fix Version and assigning it to a developer. The issue then moves to Pending Acceptance. We move other issues to On Backlog.
  3. The developer accepts the issue, moving it to Accepted when they make a plan to start work.
  4. When the code is complete, the developer resolves the issue, moving it to Pending Review.
  5. After code review, we transition the issue to Pending Merge.
  6. When we’re ready to create a release candidate, we merge changes into the release branch and deploy to the QA environment, transitioning the issue to Pending Verification.
  7. The QA analyst verifies the work and either reopens the issue or verifies it, transitioning it to Pending Closure.
  8. After we verify all issues in a targeted release, we can release the build to production and move all the issues to Closed.

We also use custom issue types to model our process. In a previous post, we described the ProTest issue type (short for Proctor Test). We use this custom issue type to request new Proctor A/B tests or to change test allocations.

We have another custom issue type and associated workflow for localization. As we continue to grow internationally, we need a localization process that doesn’t slow us down. Coordinating with many translators can be a challenge, so we model our translation process in Jira. Our Explosion issue type incorporates an issue for each target translation language. The workflow follows:

  1. We create an issue with English strings that require translation.
  2. We triage the issue and submit it for review.
  3. When the strings are ready to be translated, an automated step creates one Translation issue for each target language and links them all to the Explosion issue.
  4. Each “exploded” issue follows its own workflow: Accept, Resolve, Verify and Close.
  5. When all Translation issues are closed, we verify and close the Explosion issue.

The Explosion and Translation custom issue types and workflows help streamline a process with many participants. Because we triage by language and feature, translation issues do not block the release of an entire feature. Using Jira also allows us to integrate machine translation and outside translation services.

Team triage

Many of our development teams use dashboards and agile boards in Jira for easy access to issues associated with a product. During routine triage meetings, product development teams use these tools to prioritize and distribute development work.

Closing the memory loop

Each code commit in Git is traceable to a corresponding issue in Jira. Further, if the referenced Jira links to the initiative, the trail leads all the way to the initiative. This means that an engineer can review any code commit and follow the trail in Jira to understand all related implementation details, requirements, and business motivation.

Production deploys

Deploying code to production requires clear communication and coordination, and our Deploy issue type helps us track this process. Using Jira to track deploys results in smooth handoffs and transparency for all stakeholders.

A deploy ticket is associated with each Fix Version and has a unique workflow that facilitates communication for moving artifacts through the build and release process. We use issue links to document all sysadmin tasks necessary for a successful deployment. The deploy ticket has the same fix version as the other issues in the release.

Most teams plan their work weekly but deliver to production as they complete the work. On some regular cadence – semi-weekly, daily, or more often – the release manager creates a release candidate from all open merge requests. We developed an internal webapp that coordinates across Git (branch management), Jira (code changes and deploys), Crucible (code review), and Jenkins (build). Status changes to the deploy ticket trigger issue reassignments, promoting smooth handoffs.

This approach provides our teams with the information they need to assess and manage risk for their production releases. The QA analyst can better understand potential regressions that a change may cause. The release manager can have a holistic view of what’s changing and quickly react when issues arise. And small releases make bug investigation more straightforward.

Working in the open

Jira enables effective, efficient collaboration for our software development and deployment process. We use it to clarify requirements, discuss implementation choices, verify changes, and deploy to production.

Across teams and up and down the organization, our use of Jira provides transparency into the work that is getting done. By working in the open, we can achieve a shared understanding of plans, progress, and challenges for hundreds of active projects and initiatives.

Do what makes sense for you

Methodology and process only help when they provide repeatable and predictable mechanisms for planning, communication, and delivery. Jira has helped us establish these mechanisms.

Try to avoid taking a methodology “off the shelf” and implementing it. And don’t depend on tools to solve your problems. Instead, think about how your team needs to plan, communicate, and deliver. Then, define the best process and tools that serve your needs. Iterate on your process as needed. And stay focused on what really matters: success.


Adapted from Jack Humphrey’s presentation at Keep Austin Agile 2014.

Interns Help People Get Jobs

At Indeed, an internship isn’t just a summer job, it’s a full-time experience. We immerse interns into our world of software engineering. Interns experience an exciting company culture and explore all that Austin has to offer. More importantly, we provide challenging projects that allow interns to make their mark at the company (in fact, one of our first interns is now our CTO).

Group of smiling interns taking a selfie.

Indeed interns on a food truck tour at Micklethwait Craft Meats

Our goal is for interns to help people get jobs, have fun, and promote Indeed. To achieve these goals, we start with an engaging, challenging, and well-defined intern project.

Defining an intern project

The choice of project is crucial to a successful internship. We put a lot of effort into defining projects, because we want all our interns to have a successful summer. An ideal intern project is:

Self-contained. If the project depends too much on other projects, those dependencies could slow down the intern’s progress.

Well-defined. The intern doesn’t have to wait on requirements and can get to work right away.

Appropriately scoped. The intern should be able to complete the project in three months.

Fun and flexible. The project allows room for the intern’s creativity.

Before interns start on their projects, mentors introduce them to Indeed tools and processes. Each intern deploys a small, contained code change by the end of their first week. When the code goes live, per Indeed tradition, the intern rings a gong and everyone applauds.

After this training, interns dig into the technical challenges of their projects. Mentors provide continuous advice, introductions, and general support.

The art of mentorship

Our software engineers who volunteer to be mentors are passionate about helping others and growing their own skills. Mentoring develops skills that engineers can leverage in their everyday work. Mentors learn the value of investing time to support others. They learn when to help out and when to take a step back. And they learn how to inspire and motivate.

For example, a mentor customizes the phases of a project to fit an intern’s skills and interests. Mentors listen to interns, assess their work, and give them room to be creative. For some engineers, mentorship provides a leadership opportunity that informs their future career decisions.

Location, location, location

Sometimes, the most enlightening part of the internship isn’t the job or the technology. The Indeed internship program provides a realistic view into the work life of a software engineer. For Indeed interns, this life is set in Austin, a city renowned for live music and food trucks. In Austin, you can find roller derby, chamber music, and everything in between.

Indeed’s University Recruiting team organizes events so interns can get to know the city and each other. Past events included a party barge on Lake Travis, a Segway tour downtown, and go-kart racing. Interns work hard on their projects, and Indeed works hard to make their summer a blast.

Group on Segways approaching the Texas state capitol building.

Intern Segway tour of downtown Austin

Case study: One intern’s success story

Tom Werner (University of Iowa) interned at Indeed during the summer of 2015. His project involved building new features that employers could use during the hiring process — features that let employers view interviewer metrics and check interviewer availability.

This project was challenging in several ways. First, Tom was new to the world of front-end web development, but his project required creating an intuitive and simple webapp. Additionally, Tom collaborated with another team to gain access to interviewer data. Finally, he needed to ensure each employer could only access their own data. Implementing this access control required testing and fixing the existing data segmentation infrastructure.

Tom was quick to accept these challenges. As a self-directed and fast learner, he completed the core goals of this project in half of the scheduled time.

Now what?

Tom then had over a month to work on other useful features, based on his interests. He addressed security concerns, and he created admin features for the webapp. Specifically, Tom developed an access control list interface for user permissions. This interface allows designated employer admins to manage their own user permissions. Without this feature, employers would have to contact Indeed to make permission changes.

Tom also improved the interface employers use to manage information about their interview funnel. He added more filtering and configuration options, including creating, editing, and copying funnels. Tom’s work improved usability for employers as well as internal Indeed users.

Demo time

Because of Tom’s excellent work, his mentor asked him to demo his project to a large internal group that included the broader product team, recruiters, and executives. Everyone was impressed by Tom’s contributions to the product.

No limits on success

Tom’s talent and dedication made his internship a great success. Indeed gave him opportunities to showcase his abilities, and he went beyond our expectations. Tom delivered on business needs while bringing his own creativity to the project, a balance to which we always aspire.

Everybody wins

Tom was just one of 27 Austin interns in 2015 who helped Indeed achieve our goals. Another Indeed intern developed a webapp to display international marketing metrics. Xingtong Zhou (University of Michigan), who is now a full-time employee, created the webapp to help marketing analysts identify countries where we need to adjust our marketing investment. With better international data, we are able to make smarter investments in international growth.

Interns bring a fresh perspective that inspires innovation in our products and technologies. They contribute to the vibrancy and energy of our work environment. They return to school and share their experiences, helping us build a great campus reputation and attract the best full-time talent.

The ultimate goal of our internships, however, is to gain full-time Indeed employees. An internship is a lot like a three-month-long job interview, giving interns the unique opportunity to showcase all their skills. And we have the opportunity to give interns a glimpse into just how exciting it can be to work at Indeed. When interns return to Indeed as full-time employees, they have the added benefit of starting a job they already know is the right fit.

The bottom line: when interns have a great experience, Indeed thrives. If you’re interested in an internship at Indeed, please email university-tech-recruiting@indeed.com.

Indeed University: Building Data-Driven Products

July 2015 marked the kickoff of Indeed University (IU), our inaugural 12-week summer program to teach Indeed’s development culture to new Indeedians. Over 50 new Indeed software engineers from our Tokyo, San Francisco, Seattle, and Austin offices took part in the program, held at our Austin headquarters.

Indeed University onboarded new hires by acquainting them with Indeed’s culture, technology, and software development philosophy. Our three main goals were:

  • Accelerate the onboarding of new college grads
  • Provide leadership opportunities for current employees
  • Prototype new ideas for Indeed’s business

With these goals in mind, we set out to test a new means of onboarding Indeed’s software engineers.

Indeed U seal for 2015.

Onboarding

We like to pair new hires with mentors, who help with anything from dev environment setup to version control to A/B test definition. If a new hire has a question about something beyond their mentor’s expertise, the mentor helps them find the right expert. We encourage every new engineer to get a change into production in their first week, and the mentor supports them as they navigate that learning experience.

From Fall 2014 through Spring 2015, Indeed hired 51 new graduates from Computer Science and Engineering programs in the United States and Japan. With this growth, we could not sustain an onboarding process based on 1:1 mentorship.

But Indeed University did not lose the emphasis on taking ownership and getting code into production. Within the first two weeks of the program, every participant had completed their “gong project,” in which they defined and implemented an A/B test on Indeed. Then they dove into a hands-on curriculum, attended informative talks, and formed teams to build new products.

Group working on a project in front of a computer against a backdrop of greenery outside of the window.

IUers collaborating in their main work space

Gong project

We take pride in our ability to test everything from simple changes to new experiences. For IU, we embraced this philosophy and offered a blank canvas for our new hires to implement an A/B test in their first week.

The A/B tests aimed to help job seekers find jobs faster by emphasizing helpful filters, such as salary.

"You may filter your results by salary"

Screenshot of an Indeed University A/B test that encouraged job seekers to filter by salary

It’s an Indeed tradition for an engineer to ring the gong after their first line of code goes live on Indeed. At IU, everyone gathered around when a test was live. The IU participant (IUer) explained what A/B test they implemented and then rang the gong. Usually with great gusto.

Animated GIF of participant.

Ringing the gong

Designing and implementing these tests provided an opportunity for these new Indeedians to bring fresh perspectives to Indeed’s core products. What better way for a new hire to learn about data-driven development than to form a hypothesis, design a test, and analyze the results? IUers got a first-hand experience with our culture of data-driven development. It was exciting to see new ideas come out of these A/B tests.

Curriculum

After completing the gong project, IUers launched into the curriculum – a self-paced course designed by seasoned Indeed engineers. Exercises followed each section, making the curriculum more hands-on.

The curriculum began with engineering basics and covered the tools and technologies we use at Indeed to build webapps and services. Then it covered logging (using logrepo) and data analysis (using Imhotep).

Talks

IU talks introduced our product development philosophy and some interesting market opportunities. We wanted participants to think about new products Indeed could build.

Seated participants listening to presentation.

Paul D’Arcy presents on How People Look for Jobs (Photo: Hannah Spellman)

Tech talks and product deep dives provided more specific details about infrastructure and technology. Unlike typical startups, the IU teams had immediate access to Indeed data and users, as well as tools for deployment, testing, and authentication.

Social events

Social activities helped IUers get to know one another. They developed lasting connections before many of them headed off to Seattle, San Francisco, and Tokyo.

We planned Friday happy hours that were complete with freshly baked cookies from Tiff’s Treats, local brews, a slew of board games, and Wii U. We also took our new hires to our favorite local restaurants, including Franklin Barbecue, Torchy’s Tacos, and Michi Ramen. We went to a Round Rock Express baseball game. We also did standup paddleboarding, go-karting, and paintball. Last, but not least, we visited the Austin Panic Room.

Paddleboarding on the water at sunset.

Standup Paddle Board on Town Lake in Austin

Smiling group packed into outdoor seating.

Enjoying Franklin Barbecue

Developing new leaders

Twelve emerging leaders from the engineering, product, and online marketing organizations joined Indeed University to teach best practices and our style of iterative, data-driven development. These leads gained experience by directly managing 4-5 new hires. They also each advised up to 3 product teams.

The experience gave the leads their first taste of engineering management, including weekly 1-on-1s and quarterly evaluations. The 1:4 ratio allowed leads to develop relationships with the new hires. IUers talked with their leads about product and technology challenges as well as more mundane concerns like finding an apartment.

As product team advisors, leads challenged themselves to teach rather than tell. They encouraged teams to plan product iterations, prioritize issues, design tests, and analyze results.

In other words, leads taught participants how to think like Indeed engineers. They encouraged teams to be independent but unafraid to ask questions, to take risks and use data to measure outcomes, and to take ownership of their products.

New products

We encourage all engineers to imagine product changes that focus on our mission: helping people get jobs. IUers brainstormed new product ideas for Indeed. We held three brainstorming sessions. After each session, groups pitched solutions to problems they had identified.

Brainstorming wrapped up with a final round of pitches to Indeed’s senior leadership team, Indeed University leads, and other interested Indeedians.

Participants collaborating in an open setting.

Brainstorming in the IU lounge

Following brainstorming, IUers formed teams based on their interests and spent the remaining 9 weeks building working products. They served as their own product managers, team leads, designers, marketers, and testers.

At Indeed, we believe we must explore as many product ideas as we can, as quickly as we can. IU immersed participants in this culture of engineering velocity.

Engineering velocity: Deliver, Measure, Learn and then repeat

Before product development began, every team researched their market. Teams created Google Surveys, called Indeed employers, and ventured to local shopping malls to speak with retail managers and employees directly.

These conversations challenged some assumptions, helped drive initial product direction, and caused one early product pivot.

We challenged each team to build the minimal viable product (MVP) that would allow them to validate their idea. How would they demonstrate value? One word: data. Teams needed to collect data in order to confirm their product’s value.

Indeed University teams had a few “unfair advantages.” For one, they had Indeed’s data at their fingertips. They could use this data to do preliminary research or to tailor their design to a specific audience. Second, each team had a generous online marketing budget for traffic acquisition. Teams tested showing ads on Google, Facebook, and Indeed job search, to see where they could best connect with potential users.

Teams presented their work at weekly product reviews with Indeed executives and IU leads. In these weekly reviews, teams answered the following questions:

  • What did you do this week?
  • What data did you collect?
  • What did you learn from this data?
  • What are you doing based on the data?

In these meetings, we encouraged teams to start small, validate with data, and iterate. They learned that building successful software is about more than software architecture.

The product reviews encouraged discussion of A/B test results and opportunities for testing assumptions. Many teams experienced high bounce rates on their landing pages, and they ran A/B tests to test their hypotheses about these bouncing users. One team tested a new call to action on their landing page. Another tested delaying the sign-in requirement, allowing job seekers to experience the product before creating an account.

In the course of three months of Indeed University, the new hires built eleven new products:

  • A search engine for college students to find jobs
  • A data trends exploration tool for HR professionals
  • A site that helps high school students choose a college major based on jobs that interest them
  • A product that uses social sentiment analysis to give job seekers unfiltered comments about company reputation
  • A hiring tool that helps employers manage active candidate statuses
  • A product that asks job seekers a series of questions in order to match them to jobs based on their interests
  • An application that allows employees to track progress toward their goals
  • A mobile app for finding local retail and food service jobs
  • An automated phone screening solution to help employers efficiently evaluate candidates
  • A gig marketplace for job seekers to find small jobs in their area
  • A site that lets people visually explore career paths based on a current or desired position

Graduation

At the Indeed University Graduation Party, each team introduced the product they built and presented their learnings. Five products continued beyond IU for product validation through further development, testing, and iteration.

Group standing and presenting.

An IU team talks about what they learned at the Graduation Party

After IU, the participants joined their new teams in Tokyo, Seattle, San Francisco, and Austin. They take with them the connections, skills, and knowledge they gained during the program. They are on their way to having a real impact at Indeed.

Whether you are a student or an industry veteran, we are looking for talented engineers to help with our mission of helping people get jobs. To learn more, check out our open positions.