Imhotep: Scalable, Efficient, and Fast

This post is the first in a five-part series on improving the development process (and coaching developers) with metrics-driven insights.

Move fast and try things — that’s how we develop products at Indeed. We don’t believe in betting on a small number of great ideas. Instead, we bet on exploring lots of ideas as quickly as possible.

To be successful in this approach, we need innovative team members with diverse perspectives. We hire people who are excited to quickly explore ideas in service of our mission — to help people get jobs. Once they’re on board, we give them ownership and autonomy to do exactly that. And we give them the tools to track and analyze their experiments.

The right tools for the job

We’ve developed and open sourced some of these tools, including Imhotep, our data analytics platform. Imhotep enables rapid exploration and analysis of large time-series datasets. It includes a query language (IQL), a web-based UI, and a distributed backend. It is scalable, efficient, and fast.

Imhotep measure question learn improve Indeed Open Source

Imhotep is scalable

Imhotep scales horizontally by adding daemon instances that can run on commodity hardware or in the cloud. Indeed’s internal Imhotep cluster handles up to 5 million queries each week across thousands of datasets. Roughly 90% of those queries come from automated systems.

Our most popular dataset includes about 39 billion events just for the last year. That dataset alone receives around 25,000 distinct queries each month.

Imhotep is efficient

Because the data structure underlying Imhotep is an inverted index, the disk utilization is remarkably low for most time-series datasets. The dataset mentioned above, with 39 billion events and 384 possible fields per event, takes up 5.7 terabytes on disk. That works out to 146 bytes per event.

That kind of storage efficiency allows us to keep all the data for analysis and avoid sampling. Sampling is fine when you want to just look at aggregate trends. But if you want to actually dig down into your data and examine the outliers, you can’t reliably find them or see their effects if you sample.

Imhotep is fast

Imhotep’s speed lets us rapidly iterate and collaborate. Over a recent 90-day period at Indeed, our internal cluster saw around 2 million interactive Imhotep queries (queries done from the webapp). The median response time for those queries was 276 milliseconds.

A powerful cache implementation contributes to this blazing speed, with nearly 60% of interactive queries coming from the cache. But even uncached queries are quite fast, with a median response time of around 4 seconds. An uncached query over a long time span takes longer, but not that much longer. For uncached queries with a 365-day time span, the median response time is about 9 seconds.

How do we know all these stats about Imhotep performance? Because we have an Imhotep dataset for Imhotep usage. In just a few minutes, I was able to iteratively query that dataset to understand recent cluster performance.

Imhotep drives insight and improvement

Imhotep empowers us to experiment and quickly improve our products. We’ve also applied this data-driven approach to improving development processes. In the next post in this series, I explain more about how we use metrics to improve process.

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

Audit Your Web Applications with AVA

Hosting a web application is an excellent way to expose useful services to the public, but it comes at a cost: vulnerabilities in your web apps could allow attackers to access important systems, endangering your customers and your business.

AVA exposes vulnerabilities like a magnifying glass finding flaws

The Indeed Security team developed Another Vulnerability Auditor (AVA) to address this problem. By using AVA to automate application scans, we can continuously monitor production and QA systems for potential vulnerabilities. And, because we have released it as an open source tool, you can use it to monitor your applications as well.

How does it work?

AVA scans a set of application endpoints, defined in HTTP Archive (HAR) format. The HAR file catalogs the URL, headers, cookies, and POST data for HTTP requests. AVA uses this information to model endpoints and scan them using a combination of auditors and checks.


Auditors determine the HTTP elements that AVA audits. These include URLs, headers, cookies, and POST data.

Type What does it audit?
Cookie Individual cookies in the Cookie request header
Header Most request headers
JSON JSON data in the request body
Multipart Multipart form data in the request body
Parameter Parameters in the URL query string and request body
Response Aspects of a response (passive audit)
Text Plain text data in the request body
URL Request URL


Checks determine the types of security vulnerabilities AVA checks. These include cross-site scripting, open redirects, SQL injection, and shell injection.

Type What does it check for?
Code injection Code injection in Python’s pickle and eval statements
Header injection Header injection in responses
Open redirect Open redirects to arbitrary URLs
Path traversal Path traversal on the local filesystem
Shell injection Shell injection in Bash statements
SQL injection SQL injection in database queries
Cross-site scripting HTML and JavaScript injection in responses
XML external entity XML external entities in XML documents
Personally Identifiable Information (PII) Email addresses, credit cards, and Social Security numbers

How can I use it?

We designed AVA for use within automated systems. We automate AVA with Docker Swarm and Jenkins. However, you can use AVA anywhere Python can be installed.

Use in Docker Swarm

Indeed’s Security team uses Docker Swarm to automate AVA and scan public-facing applications daily. This allows us to identify vulnerabilities shortly after they are introduced. The pipeline has three components:

  • Enricher combines data from sources, such as WES, into endpoint definitions
  • Scheduler maintains a schedule and configuration
  • Vulnerability manager stores reports and displays vulnerability information

The process is as follows:

  1. The scheduler contacts the enricher and requests endpoint definitions for the current application.
  2. The enricher returns these definitions in HAR format.
  3. The scheduler pushes the HAR data and configuration settings to AVA.
  4. AVA runs the configured scan against the application and generates a report.
  5. AVA sends the report to the vulnerability manager for storage.

Diagram showing how AVA finds vulnerabilities with Docker Swarm

Use in Jenkins

We also use AVA in Jenkins to examine systems in our QA environment. This allows us to identify vulnerabilities before they reach production. The pipeline has two components: functional tests and AVA. The functional tests are a collection of Selenium-based test cases for verifying release candidates in QA.

The process is as follows:

  1. Functional tests run against the application.
  2. A proxy collects traffic from the tests and exports it as HAR files.
  3. AVA scans the application using the exported HAR files.
  4. AVA provides a report documenting the results of the scan.

Diagram showing how AVA finds vulnerabilities with Jenkins

How can I get AVA?

We’ve made AVA available and open source on Git. Download it, try it out, and reach out to us on GitHub or Twitter if you need any help. You can open an issue on the GitHub repository, or hit us up on Twitter.

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

@IndeedEng Presents: The Art and Science of Product Metrics in San Francisco and Seattle

Indeed Engineering is hosting talks on the art and science of product metrics in San Francisco and Seattle during the month of August.

Ketan Gangatirkar, VP of Engineering for Indeed’s Job Seeker products, will discuss choosing metrics that work, because every story of metrics gone wrong is really a story of badly chosen metrics. Attendees will leave this talk with a greater understanding of data science and concrete strategies for using data to improve their processes and products.

This talk aims to help product managers, tech leads, managers, and data scientists become better at their work. You’ll gain insights and knowledge if you’re in one of these roles or aspiring to be—or you’re just curious.

This is an extended edition of the talk presented at O’Reilly Strata, London.

San Francisco

This talk was held on Wednesday, August 1, 2018


This talk was held on Wednesday, August 29, 2018

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone