Finding Anomalies in User Behavior with Python


In the course of helping over 200 million unique visitors every month find jobs, we end up with a lot of data. The data we collect can tell us a lot about the behavior of our users, and for the most part we observe predictable patterns in that behavior. But unexpected changes could be evidence of failures in our system or an actual shift in user behavior. When the data shows something strange, we want to understand.

Identifying and acting on anomalies in user behavior is a complex problem. To help detect these anomalies, we take advantage of several open-source libraries, chief among them Twitter’s AnomalyDetection library.

Observing anomalies in user behavior

Detecting anomalies in a large set of data can be easy if that data follows a regular, predictable pattern over time. If we saw the same range of pageviews for our job listings every day, as simulated in Figure 1, it would be easy to identify outliers.


Figure 1. Single outlier

But most of the data we collect is determined by user behavior, and such data does not follow patterns that we can easily observe. A variety of factors influence user behavior. For example, each of the following factors might affect what we consider a “normal” range of pageviews, depending on the geographic location of the user:

  • What day of the week is it?
  • What time of day is it?
  • Is it a holiday?

We might understand some factors in advance. We might understand others after analyzing the data. We might never fully understand some.

Our anomaly detection should account for as many variations as possible, but still be precise enough to provide significant statistical outliers. Simply saying “traffic is normally higher on Monday morning” is too fuzzy: How much higher? For how long?

Figure 2 shows a variable range of expected data, while Figure 3 shows a range of actual data. Anomalies within the actual data are not immediately visible.


Figure 2. Expected data


Figure 3. Actual data

Figure 4 shows the actual and expected data overlaid. Figure 5 shows the difference between the two at any point in time. Viewing the data in this way highlights the significant anomaly in the final data point of the sequence.


Figure 4. Expected and actual overlaid


Figure 5. Difference between actual and expected data

We needed a sophisticated method to quickly identify and report on anomalies such as these. This method had to be able to analyze existing data to predict expected future data. And it had to be accurate enough so that we wouldn’t miss anomalies requiring action.

Step one: Solving the statistical problem

The hard mathematical part was actually easy, because Twitter already solved the problem and open sourced their AnomalyDetection library.

From the project description:

AnomalyDetection is an open-source R package to detect anomalies which is robust, from a statistical standpoint, in the presence of seasonality and an underlying trend. The AnomalyDetection package can be used in wide variety of contexts. For example, detecting anomalies in system metrics after a new software release, user engagement post an A/B test, or for problems in econometrics, financial engineering, political and social sciences.

To create this library, Twitter started with an extreme studentized deviate (ESD) test, also known as Grubb’s test, and improved it to handle user behavior data. Originally, the test used the mean value for a set of data to identify outliers. Twitter’s developers realized that using the median value was more precise for web use, as user behavior can be volatile over time.

The result is a resource that makes it easy to quickly identify anomalous results in time-based datasets with variable trends. Twitter’s data scientists use this data to perform their own internal analysis and reporting. For example, they report on tweets per second and the CPU utilization of their internal servers.

Twitter’s library allowed us to use historical data to estimate a highly complicated set of user behavior and quickly identify anomalous behavior. There was only one problem: Twitter wrote the library using R, while our internal alerting systems are implemented in Python.

We decided to port Twitter’s library to Python so that it would work directly with our code.

Step two: Porting the library

Of course, porting code from one language to another always involves some refactoring and problem solving. Most of our work in porting the AnomalyDetection library dealt with differences between how math is supported in R and in Python.

Twitter’s code relies on several math functions that are not natively supported in Python. The most important is a seasonal decomposition algorithm called seasonal and trend decomposition using loess (STL).

We were able to incorporate STL from the open-source pyloess library. We found many of the other math functions that Twitter used in the numpy and scipy libraries. This left us with only a few unsupported math functions, which we ported to our library directly by reading the R code and replicating the functionality in Python.

Taking advantage of the excellent work done by our neighbors in the open-source community allowed us to greatly reduce the effort required in porting the code. By using the pyloess, numpy, and scipy libraries to replicate the R math functions Twitter used, one developer completed most of the work in about a week.

Open-source Python AnomalyDetection

We participate in the open source community because, as engineers, we recognize the value in adapting and learning from the work of others. We are happy to make our Python port of AnomalyDetection available as open source. Download it, try it out, and reach out to us on GitHub or Twitter if you need any help.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

A Funny Thing Happened on the Way to Java 8

Indeed is migrating from Java 7 to Java 8. We’re migrating in phases, starting with running our Java 7 compiled binaries on JRE 8, the Java 8 runtime. Rather than switch everything at once, we started with a few canary apps to get a sense for the kinds of problems we might find.

We selected our job search web app as our first canary. Last May, we switched a single job search server in production to JRE 8. We’d been running with JRE 8 in a QA environment for several weeks without any trouble. But when we switched to JRE 8 in production, the system load rose from 5 (a normal level) to 20 (an abnormal level): high enough to require a fix before proceeding. Fixing the problem required unusual measures.

Production metrics in Datadog

I started by looking at Datadog, a third-party, cloud-based metric and event correlation service that we use for real-time monitoring. The following Datadog graph shows the system load by host for every job search server in one of our data centers. The outlier (blue line) is the server running JRE 8.


Figure 1. System load for job search servers in a data center

Here is the median response time by host. Again, the server running JRE 8 is the outlier.


Figure 2. Median response time by host

The job search servers that were still on JRE 7 had normal system loads and normal median response times, so the problem was specific to the JRE 8 server.

It’s not unusual for a Java process to slow down because of garbage collection (GC) activity. We publish our GC metrics to Datadog, and the GC graphs for the JRE 8 instances looked normal.

Production log files

Next, I checked the web application log files. I retrieved every log message from the four hours when we used JRE 8, plus the log messages for the previous 24 hours. There were too many messages to read by hand. Instead, I focused on changes in the distribution of errors. I extracted the name of the associated Java class for each error and then generated histograms for different time periods.

The histograms showed a slight increase in memcache client timeouts during the JRE 8 period. They also showed we used an open-source memcache client library that is no longer maintained. A Google search didn’t turn up any JRE 8 specific problems with the library. I glanced through its source code but didn’t see any problems. I decided that the increase in timeouts was most likely just a side effect of the higher system load.

Reproducing the problem in QA

Since the evidence from our production environment didn’t identify the problem, the next step was to try to reproduce it in our QA environment. As I mentioned earlier, the web app had been running normally on JRE 8 in QA. That wasn’t surprising, because performance problems might not manifest themselves until the system is under enough load, and our QA servers don’t get as much traffic as our production servers. I hoped that if I increased traffic beyond normal QA levels, we would see the same symptoms.

I queried our Imhotep system to find the relative frequencies of our most common incoming requests. Then, using JMeter, I simulated that mixture of traffic and ran it twice while monitoring system load, once each with instances on JRE 7 and JRE 8. In both cases, the web app processed about 50 requests per second. The system load was about the same in both cases, too.

Fifty requests per second is a higher traffic load than we normally simulate in QA, but it is still not at the level we receive in production. When we started this effort, our production job search instances each served approximately 350 requests per second. To reproduce the problem in QA, I would need to get closer to that traffic volume.

Ruling out JMeter as a bottleneck

Why was our application pegged at 50 requests/sec? Before blaming the QA environment, I needed to rule out JMeter as the bottleneck. I installed JMeter on a second server, configured it like the first JMeter instance, and then ran both at the same time. If the two instances combined had run at more than 50 requests/sec, I would know JMeter was the bottleneck. In that case, the next step would be to add more JMeter instances until we either saturated the web app or caused the system load to rise to 20.

It turned out JMeter wasn’t the bottleneck. When I ran both instances at the same time, the aggregate throughput was still 50 requests/sec.

Eliminating slow services

The job search web app depends on 22 services. If some of them are slow enough, the web app will slow down, too. A service in QA may be slower than its production counterpart. We sometimes replace slow services with fake but fast versions when isolating the dependent application for performance analysis. I decided to create fake versions of the five services that our application uses the most.

Using the new fake services, JMeter drove the webapp at 100 requests/sec. JRE 8 was about 15% slower than JRE 7, but the system load still wasn’t very high — and the difference between JRE 7 and JRE 8 varied a lot from one run to the next. Still, since the numbers were different, I profiled the performance of both configurations using YourKit.


YourKit is a Java profiler you attach to a Java process to collect and analyze runtime statistics. Once attached, you can create a performance snapshot: an aggregation of stack traces from every Java thread from multiple points in time. YourKit provides multiple ways to visualize a performance snapshot, including a hot spot view that shows which methods consumed the most time. Since our web app on JRE 8 consumed a lot more CPU than on JRE 7, I expected the respective hot spot views to differ.

The views were about the same, presumably because I still hadn’t put the systems under enough load in QA.

Taking performance profiles in production

At this point, I could have invested more time trying to reproduce the problem in QA, but instead I decided to use YourKit in production. We took performance snapshots in production with both JRE 7 and JRE 8. The only significant difference between the two was that the JRE 8 configuration seemed to use a lot more CPU. In fact, it used much more CPU than the performance snapshot could account for. It was as if the application was busy doing something in the background that YourKit couldn’t measure. We then decided to try profiling with the perf command.

Using the perf command

A Java process is more than a collection of Java code; it’s also a JVM composed of multi-threaded C++. A Java profiler can’t measure what that C++ code does; for that, you need a different kind of profiler, like the Linux perf command.

The perf command can collect call stacks from a Linux process. It uses the process’s debug symbol table to map a call stack’s addresses from virtual addresses to symbolic names of functions and methods. The JVM binary doesn’t contain a debug symbol table for your Java code, but there’s a way to query a Java process for its Java debug symbol table, too.

In production, we configured perf to capture the call stack for every web app thread every ~10ms for 30 seconds. That yielded about 11,000 call stacks. I aggregated the data into a flame graph (Figure 3).

In the flame graph, the y axis is stack depth. Any box at the top represents code that was on CPU when a sample was taken. The box below it is its caller, and so on. The flame graph in Figure 3 is really tall because our call stacks are deep. The x axis is just an alphabetical sort. The wider the box, the more call stacks it appears in. Green boxes are Java code, yellow is C++ code in the JVM, and red is C code in the JVM. Lightness and saturation are chosen for contrast and have no special meaning.


Figure 3. Flame graph showing job search call stacks

Figure 4 shows the bottom portion of the flame graph:


Figure 4. Flame graph zoomed view

The CompileBroker::compiler_thread_loop (left, in yellow area) appears in 39% of call stacks. That is the bytecode compiler, the part of the JVM that converts frequently used bytecode into machine instructions. The value 39% seemed like a lot, and it represented something in the background that we couldn’t measure with YourKit.

Why would it consume so much time? The bytecode compiler writes its machine code to a dedicated area of memory called the codecache, which has a configurable size. There are JVM settings to configure what happens when the codecache fills up, but by default, the JVM is supposed to make room by flushing older entries. YourKit showed that our codecache was close to full, so it seemed worthwhile to try increasing the maximum size.

Doubling the codecache

We doubled the codecache size in production from 64MB to 128MB, restarted the web app, and let it run over the weekend. The application behaved normally and the system load stayed at a normal level. We ran the perf command and flame-graphed the results (Figure 5).


Figure 5. Flame graph zoomed view after doubling the codecache

The width of the CompileBroker::compiler_thread_loop box (right) shows that the HotSpot compiler was less busy than before: about 20% of call stacks vs. about 40% before.

Codecache fluctuation

Even with a 128MB codecache, we eventually ran into trouble. Figure 6 shows what happened. The flat blue line is the maximum codecache size, and the purple curve below it is the amount of codecache used. At about 8am, the usage curve began to fluctuate and didn’t stabilize again until about 3am the next day.


Figure 6. Codecache graph from Datadog showing fluctuation

The codecache churn slowed response times. Figure 7 shows the median response times for every job search server in that data center. The outlier in orange is the JRE 8 server. It’s about 50% worse than the rest.


Figure 7. Job search web app median response times

The right size for our web app

Our new codecache size of 128MB was still only half of the Java 8 default. To try to address the performance problem, we decided to double the size again to 256MB. We hoped our codecache usage would level off below this size. If 256MB still wasn’t enough, we could try doubling once more. If that didn’t work, we would experiment with JVM optimizer settings to reduce the codecache usage.

One factor in our favor was our deployment schedule. We usually redeploy this web app at least once a week and often multiple times per week. Aside from the Christmas holidays, we never go two weeks without redeploying. We decided to configure a total of 8 instances with a 256MB codecache and let them run for at least a week. We didn’t detect any problems, so we declared it a success.

Why more codecache?

The default codecache size for JRE 8 is about 250MB, about five times larger than the 48MB default for JRE 7. Our experience is that JRE 8 needs that extra codecache. We have switched about ten services to JRE 8 so far, and all of them use about four times more codecache than before.

One big contributor to the 4x multiplier is tiered compilation. The JVM has two bytecode compilers: C1, optimized for fast startup, and C2, optimized for throughput of long-running processes. In earlier Java versions, C1 and C2 were mutually exclusive; you chose C1 with the ‑client switch or C2 with the ‑server switch. Java 7 introduced tiered compilation, which runs C1 at startup and then switches to C2 for the remainder of the process. Tiered compilation is intended to boost startup times for long-running server processes and, according to Oracle, can yield better long-term throughput than C2 alone. In Java 7 you have to explicitly enable it, but in Java 8 it is on by default. In QA, when I disabled tiered compilation on JRE 8, the codecache size dropped by 50%.

Java 8 also uses more codecache because it optimizes more aggressively than Java 7 does. For example, the InlineSmallCode option controls how short a previous compiled method needs to be in order to be inlined. Java 8 raised the default from 1000 to 2000, which means the Java 8 optimizer allows longer methods to be inlined than before. If you inline more methods, and the inlined methods are bigger, you need more codecache.

Do developers migrating to Java 8 need to worry about this?

If you don’t override the default codecache size and have 100MB per JVM to spare, you can probably ignore the codecache increase. Most Indeed services do not override the default codecache size. They will use more codecache on Java 8 than on Java 7, but the codecache is still a lot smaller than our average heap size.

If you do override the default codecache size but are not memory-constrained, it will probably suffice to remove the override. I recommend monitoring your codecache size just to be safe. This Github project includes an example that shows how to upload JMX codecache statistics to Datadog.

If you are already memory-constrained, you may want to measure how disabling tiered compilation affects your startup time, long-term performance, and memory consumption.


Connor Kelly and Mike Salsone in our Operations team supported this project, patiently putting up with lots of unusual requests.

Indeed software engineers David Wahler and Chen Yang helped me interpret the YourKit profiles. David also suggested using the perf command and helped me interpret the flame graphs.


Brendan Gregg is a Netflix engineer who specializes in performance measurement. His website is full of performance-related articles, and his talks on YouTube are worth watching. Brendan popularized using flame graphs for visualizing performance data. He also contributed JVM changes that made it possible to view Java symbols in perf data.

When I first learned about codecache, I checked the Oracle documentation for conceptual material and relevant JVM settings:

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

Building a Large-Scale Machine Learning Pipeline for Job Recommendations

(Editor’s note: This post was originally published on

With 200 million unique visitors every month, Indeed relies on a recommendation engine that processes billions of input signals every day — resulting in more external online hires than any other source.

To create this recommendation engine, we started with a minimum viable product (MVP) built with Apache Mahout and evolved to a hybrid, offline + online pipeline. Along the way, our changes affected product metrics, and we addressed challenges with incremental modifications to algorithms, system architecture, and model format. The lessons we learned in designing this system can apply to any high-traffic machine learning application.

From search engine to recommendation

Indeed’s production applications run in many data centers around the world. Clickstream data — and other application events from every data center — are replicated into a central HDFS repository, based in our Austin data center. We compute analytics and build our machine learning models from this repository.

Our job search engine is simple and intuitive, with two inputs: keywords and location. The search results page displays a list of matching jobs, ranked by relevance. The search engine is the primary way our users find jobs on Indeed.

Our decision to go beyond search and add job recommendations as a new mode of interaction was based on several reasons:

  • 25% of all searches on Indeed specify only a location and no search keywords. Many job seekers aren’t sure what keywords to use in their search.
  • When we deliver targeted recommendations, the job seeker’s experience is personalized.
  • Recommendations can help even the most sophisticated user discover additional jobs that their searches would not have uncovered.
  • With recommendations driving 35% of Amazon sales and 75% of Netflix content views, it’s clear they provide added value.

Recommending jobs is significantly different than recommending products or movies. Here are just a few of the things we took into careful consideration as we built our engine:

Rapid Inventory Growth. We aggregate millions of new jobs on Indeed every day. The set of recommendable jobs is changing constantly.

New Users. Millions of new job seekers visit Indeed every day and begin their job search. We want to be able to generate recommendations with very limited user data.

ChurnThe average lifespan of a job on Indeed is around 30 days. Content freshness matters a lot, because the older the job, the more likely it is to have been filled.

Limited Supply. One job posting is usually meant to hire one individual. This is different from books or movies, where as long as there is inventory it can be recommended to many users at the same time. If we over-recommend a job, we could bombard an employer with thousands of applications.

How to approach recommendation algorithms

Recommendations are a matching problem. Given a set of users and a set of items, we want to match users to their preferred items. There are two high-level approaches to this: content-based and behavior-based. They each have pros and cons, and there are also ways to combine these approaches to take advantage of both techniques.

Content-based approaches use data, such as user preferences and features of the items being recommended, to determine the best matches. For recommending jobs, using keywords of the job description to match keywords in a user’s resume is one content-based approach (note that users can upload their resume to the Indeed site). Using keywords in a job to find other similar jobs is another way to implement content-based recommendations.

Behavior-based approaches leverage user behavior to generate recommendations. These approaches are domain-agnostic, meaning the same algorithms that work on music or movies can be applied to the jobs domain. Behavior-based approaches do suffer from a cold start problem. If you have little user activity, it is much harder to generate good quality recommendations.

Mahout collaborative filtering

We started by building a behavior-based recommendation engine because we wanted to leverage our existing job seeker traffic and click activity. Collaborative filtering algorithms are well understood in this space.

Our first attempt at personalized recommendations was based on Apache Mahout’s user-to-user collaborative filtering implementation. We fed clickstream data into a Mahout builder that ran in our Hadoop cluster, and produced a map of users to recommended jobs. We built a new service to provide access to this model at runtime, and multiple client applications accessed this service to recommend jobs.

MVP results and roadblocks

As an MVP, the behavior-based recommendation engine showed us that it is important to start small and iterate. Building this system quickly and getting it in front of users demonstrated that these recommendations were useful to job seekers. However, we ran into several immediate problems using Mahout on our traffic:

  • The builder took around 18 hours on Indeed’s 2013 clickstream, which is about 3X smaller than present day.
  • We could only run the builder once a day, which meant that millions of new users joining Indeed every day wouldn’t see recommendations until the next day.
  • Millions of new jobs aggregated on Indeed were not visible as recommendations until the builder ran again.
  • The model we produced was a large map, which was around 50GB, and took several hours to copy over a WAN from the data center where it was built to our data centers around the globe.
  • Mahout’s implementation exposed a few tweakable parameters, like similarity thresholds. We could tweak the parameters of the algorithm, but we wanted the flexibility to test entirely different algorithms.

Implementing MinHash for recommendations

We addressed the most important problem first: the builder was too slow. We found that user-to-user similarity in Mahout is implemented by comparing every user to every other user in n2 time. For only U.S. traffic in Indeed (50 million UVs), this amounts to 15 * 1015 comparisons, which is intractable. This calculation is also batch in nature. Adding a new user or a new click event requires recalculating all similarities.

We realized that recommendations were an inexact problem. We were looking for ways to find the closest users to a given user, but we didn’t need 100% accuracy. We looked for ways to approximate similarity without having to calculate it exactly.

Principal contributor Dave Griffith came across MinHash from a Google News academic paper. MinHash, or minwise independent permutations, allows approximating Jaccard similarity. Applying this measure to jobs that two users clicked on at Indeed, we see that the more jobs these two users have in common, the higher their Jaccard similarity. Calculating Jaccard similarity for all pairs of users is O(n2), and with MinHash, we can reduce this down to O(n).

The MinHash of a set of items, given a hash function h1, is defined as taking the hash of all items in that set using that hash function, and then taking the minimum of those values. A single hash function is not sufficient to approximate the Jaccard similarity because the variance is too high. We have to use a family of hash functions to reasonably approximate Jaccard distance. With a family of hash functions, MinHash can be used to implement personalized recommendations with tweakable Jaccard similarity thresholds.

Mining Massive Datasets, a recent Coursera course from Stanford professors Leskovec, Rajaraman, and Ullman, explains MinHash in great detail. Chapter 3 (PDF) of their book, “Mining Massive Datasets,” explains the mathematical proof behind MinHash. 

Our implementation of MinHash for recommendations involved the following three phases:

Phase 1: Signature calculation/cluster assignment

Every job seeker is mapped to a set of h clusters, corresponding to a family of hash functions H. The following pseudocode shows this: 

H = {H1, H2, ..H20}
for user in Users
     for hash in H
          minhash[hash] = min{x∈Itemsi| hash(x)}

Where H is a family of h hash functions. At the end of this step, each user is represented by a signature set: a cluster consisting of h values.

Phase 2: Cluster expansion

Users that share the same signature are considered similar, and their jobs are cross recommended to each other. We expand each cluster with all the jobs from each user in that cluster.

Phase 3: Recommendation generation

To generate recommendations for a given user, we union all jobs from the h clusters that the user is in. We remove any jobs that this user has already visited to get the final set of recommended jobs.

Recommending jobs to new users

MinHash’s mathematical properties allow us to address recommending jobs to new users and recommending new jobs to all users. We update the MinHash signature for users incrementally as new clicks come in. We also maintain a map in memory of new jobs and their MinHash clusters. By keeping these two pieces of data in memory, we are able to recommend jobs to new users after they click on a few jobs. As soon as any new jobs posted throughout the day receive clicks, they are recommended to users.

After transitioning to MinHash, we had a hybrid recommendation model — consisting of an offline component that builds daily in Hadoop — and an online component implemented in memcache — consisting of the current day’s click activity. Both models are combined to compute the final set of recommendations per user. The recommendations for each user became more dynamic after we made this change, because they would update as users clicked on jobs that interested them.

With these changes, we learned that we could trade off a little bit of accuracy for a lot of performance. We also learned to complement a slower offline model with an online model for fresher results.

Engineering infrastructure improvements

The recommendation model that contained a map from each user to their recommendations was a large monolithic file. Because jobs are local to each country, we first attempted to shard our data into zones based on approximate geographic boundaries. Instead of running one builder for the entire world, we ran one builder per zone. Each zone consisted of multiple countries. As an example, the East Asian zone contained recommendations for China, Japan, Korea, Hong Kong, Taiwan, and India.

Even after sharding, some of our zones produced data files that were too big and took hours to copy from our Austin Hadoop cluster over a WAN to a remote data center in Europe. To address this, we decided to ship recommendation data incrementally rather than once per day. We reused sequential write ahead logs and log structured merge trees to implement this. This was already validated in other large production applications at Indeed, like our document service.

Instead of producing a large model file, we modified our builder to write small segments of recommendation data. Each segment file is written using sequential I/O and optimized for fast replication. These segments get reassembled into a log structured merge tree in recommendation services running in remote data centers.

This infrastructure change caused users to see their new recommendations hours faster in remote data centers. From our A/B testing of this change, we saw a 30% increase in clicks due to the fact that users received newer recommendations faster.

This improvement demonstrated that engineering infrastructure improvements can make as much of an impact on metrics as algorithm improvements.

A/B testing velocity

Building out the pipeline to compute and update recommendations was only the beginning. To improve the coverage and quality of recommendations, we needed to increase our A/B testing velocity.

We were making many decisions in the builder to tune the final set of recommendations. These decisions included similarity thresholds, the number of jobs to include in an individual’s recommendations, and different ways to filter out poor quality recommendations. We wanted to tweak and optimize every aspect of computing recommendations, but to do so would require building and shipping a new model per algorithm tweak. Testing multiple improvement ideas meant many times more disk and memory usage on the servers that handled requests from users.

We began to improve our A/B testing velocity by shipping the individual components of the recommendation calculation, rather than the final results. We changed the recommendation service to perform the final calculation by combining the pieces, instead of simply reading the model and returning results. Critical subcomponents of recommendations are cluster assignments per user, the mapping from each cluster to jobs in that cluster, and a blacklist for each user that contained jobs that should not be recommended for them. We modified our builder to produce these components and modified our service to put them together at request time to return the final list of recommendations.

By implementing this architectural change, we only shipped subcomponents that changed per A/B test. For example, if the test only tweaked what jobs got removed from a user’s recommendations, we would only ship the blacklist for the test group.

This change improved A/B testing velocity by orders of magnitude. We were able to test and validate several ideas that improved the quality and coverage of recommendations in a short period of time. Previously, we averaged testing one improvement in the model every quarter because of the overhead in setting up our experiments.

Our experience shows that A/B testing velocity should be considered when designing large machine learning systems. The faster you can get your ideas in front of users, the faster you can iterate on them.

This post summarizes a series of algorithmic and architectural changes we made as we built our recommendation engine. We build software iteratively at Indeed — we start with an MVP, learn from it, and improve it over time. As a result of these changes, job recommendations grew from a small MVP to contributing 14% of all clicks on Indeed, which is up from 3% in early 2014.

recommendations architecture

Architecture diagram for Indeed’s recommendation engine


Moving forward, we continue to refine our recommendation engine. We are prototyping a model using Apache Spark. We are building an ensemble of models, and we are refining our optimization criteria to combat popularity bias.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone