Improving Security with OAudit Toolbox

Trust is a big part of why Indeed continues to be the world’s number one job site. Users trust us to keep their information safe. We’ve always taken our responsibilities seriously, and as abuse of personal data continues to dominate the news, we cannot afford to lose diligence. That’s why we need to keep our own data safe — not just to protect Indeed employees and corporate data, but also to protect the data that our users entrust us with.

I work on Indeed’s Information Security team and in early 2017 started to address shortcomings in our Google GSuite implementation. Our solution — the OAudit Toolbox — is now available as an open source tool that you can use as well.

The problem: Third party apps can be risky

A major area of risk for Indeed’s GSuite implementation is integrating third party apps.

GSuite users can grant applications access to their account — anything from basic account info to full read/write access to Gmail. The OAuth Scopes presented in the authorization prompt control access to Google resources.

For businesses running GSuite, this presents a number of issues:

User education. Users may unknowingly grant access, or they might not understand the privacy or security implications of their choices.

Data sharing. Authorizing apps for certain scopes allows third parties access to sensitive data. Your business might not have the proper data sharing contracts in place for this kind of access.

Data exfiltration. Malicious applications can use this OAuth flow to effectively phish and exfiltrate data from Google accounts.

Limited tooling. At the time of my investigation, Google’s options for restricting apps were limited: the API only allowed for reactive blacklisting. The relatively new feature allowing for whitelisting of connected apps requires that you actually have a whitelist of apps.

Policy culture. Many companies allow overly permissive scope access. Implementing Google’s new whitelist functionality could require your security team to review, approve, and whitelist a giant backlog of apps — not to mention fielding requests from employees who feel their productivity depends on Sketchy Mail Tracker Pro™.

It wasn’t long before we had a real-world attack that highlighted these issues and helped us define our own solution.

The proof: A massive phishing attack

In May 2017, a massive phishing attack masquerading as a Google Docs invitation hit Google Apps users all over the world.


Users received an email, apparently sent from one of their contacts, requesting that they view a shared document. If they worked for a business using GSuite, this wasn’t an unusual request. What was unusual was this fake “Google Docs” requesting permission to access the recipient’s email and contacts. If users granted this access, the fake “Google Docs” app then sent the same phishing email onward to the victim’s contacts, using the victim’s account and masquerading as them.

As part of Indeed’s Security team, I was on the front lines of our company’s response to this attack. After several hours of panic and revocation of OAuth tokens, I felt oddly inspired. I wanted to fill in the gaps in GSuite’s available tooling, detect attacks like this one sooner, and find a way to educate users about the dangers of authorizing third party apps. Along with my coworker Dustin Decker, I started work on a set of tools that might get us closer to a perfect solution.

Our solution: OAudit Toolbox

OAudit Toolbox is a set of tools that detects third party app integrations and notifies users of their danger and scope.

How OAudit Toolbox works

OAudit Toolbox contains two major components:

  • Oaudit-collector indexes authorization events from the Google Admin API into Elasticsearch
  • Oaudit-notifier sends notifications with educational information about Oauth scopes and contains whitelisting/blacklisting logic

Blacklisting allows us to revoke access to a list of known bad apps in near-real time, including malicious apps and apps in violation of corporate policy. Access is only revoked to apps authorized after the app is defined in the blacklist.

Whitelisting allows us to stop notifications for trusted apps so that users don’t suffer alert fatigue.

  1. Oaudit-collector fetches authorization token event data using the Google Admin SDK API.
  2. Oaudit-collector indexes fetched data in Elasticsearch.
  3. Oaudit-notifier checks to see whether the authorized application is whitelisted, blacklisted, or unknown.
    • If the app is whitelisted, a notification is not sent. This is typically used for apps that have passed security review, and in the case of third party apps, have the appropriate data sharing agreements in place (if applicable).
    • If the app is blacklisted, a notification is sent to the user that the app is blacklisted and access has been revoked. This is used for malicious apps (such as the ones seen in the 2017 phishing attack) and apps that go against corporate policy.
    • Apps that are neither blacklisted nor whitelisted trigger a notification to the user informing them of the potential risk of authorizing untrusted applications and how to revoke unwanted access.

How OAudit Toolbox helps

OAuth Toolbox solves, or at least mitigates, each of the issues I identified with using GSuite.

Solution: User education

There are a million reasons why users might authorize third party apps:

  • A manager told them to
  • They don’t understand the authorization prompt
  • They haven’t heard of all the fun acronyms like DPA and MNDA that help us more securely share data with third parties
  • And many more!

With OAudit enabled, users receive an easy-to-understand visualization of the risks associated with each third party app, including user-friendly descriptions of the risks. Each scope is assigned a score based on the risk of sharing sensitive data, and highlighted with an associated color.

After enabling this feature, we saw a significant increase in questions from our users about whether apps are safe to use. We received more requests for our application security team to review third party apps. We were also contacted by teams outside of our engineering organization. Non technical users had previously felt uneasy about using some third party tools but lacked the technical or security context to explain why.

Solution: Data sharing

Users add tools to Google Apps to do things like improve spreadsheet visualizations, enable mail marketing campaigns, or send themselves Google Form results.

To the typical user, it’s not obvious that these integrations exist on a third party’s server rather than within GSuite itself. Some users assume that Google thoroughly vets each app. Other users don’t realize that once this data resides on third party equipment, they have no control over it beyond contractual agreements regarding further usage and sharing. For companies that must comply with GDPR, this issue goes beyond security and becomes regulatory.

Using the OAudit Toolbox has helped us socialize the concept of high-risk data sharing. At the same time, we have been able to work with our privacy and contracts teams to get the appropriate agreements in place where needed and revoke access to those apps that don’t pass our assessments. Retroactively revoking access to unapproved (but non-malicious) apps using the blacklist has been reasonably effective, as the expected functionality of these apps does not involve immediate data exfiltration.

Solution: Data exfiltration

While there are legitimately useful third party apps, there are also malicious apps masquerading as useful tools, such as the “Google Docs” app involved in the phishing attack of 2017.

You can always revoke a malicious app’s access, but doing so is minimally helpful because these apps are likely to exfiltrate data and/or abuse your account as soon as possible. The reliability of this method also depends on the timeliness of token logs — which, during the Google Docs phishing attack, were as far as 12 hours behind. More recently, token activity lag time is between 1-10 minutes, though Google claims this can be as much as a few hours.

Since the OAuth Toolbox also sends data to Elasticsearch, we recommend setting up ElastAlert or Watcher to detect never-before-seen apps being authorized or a spike in a single app being authorized in a short period of time.

By proactively warning users about dangers, OAudit Toolbox is more like an early alerting system for malicious apps, or IDS, and less like a proactive blocker, such as an IPS.

Solution: Available tooling

When development on OAuth Toolkit started, the ability to block app access was scattered across GSuite. A Google Apps administrator could block Marketplace Apps, Drive API, and Chrome extensions. That would have been too heavy-handed for us to successfully implement. Administratively, it was difficult to manage with minimal ROI since the solution was incomplete.

Google now allows you to block OAuth access to Google Apps such as Drive, GMail, and Contacts using the Google Admin Security panel. Optionally, you can block only “high risk” scopes, but there is no documentation as to which scopes are considered high risk. There is also a whitelist available to allow use of trusted apps.

This is a useful addition if and when your team is ready and able to:

  • Block all apps in use
  • Know what you need to whitelist
  • Have a workflow in place for approving new applications
  • Have application security resources for reviewing those apps

We were glad to see Google take these steps, but OAudit Toolkit provided us a solution that was easier to implement and less disruptive to workflow.

Solution: Policy culture

A major hurdle for implementing this tool and the subsequent review process wasn’t technical, but human. Going from an open, “bring your own app” culture to a more restrictive, seemingly bureaucratic process is a struggle — especially when transparency is a keystone of company culture.

We found a few practices to be helpful:

  • Use tools (such as OAudit Toolbox), training, tech talks, or internal blog posts to introduce users to the risk posed by third party apps.
  • Instead of adding a pile of legalese to your Acceptable Use policy, keep any actions users need to perform simple and available on an easily accessed page: the intranet home page, a wiki page, or an IT Support landing page.
  • Bucket frequently used, high-risk tools into categories such as “Mail Merge” or “Spreadsheet automation.” Doing so makes it easier to rip-and-replace them with a single, approved app rather than evaluate 50 unique apps.
  • Accept that you might not solve the problem overnight. Even if you can only mitigate the problem by allowing continued access to some apps and approving or restricting new apps, that still puts you in a better place than you were before.

Get started with OAudit Toolbox

We’ve made OAudit Toolbox open source and available so you can benefit from our experience. By integrating it with your company’s Google suite, you can take similar steps to improve organizational education and, hopefully, better resist the next phishing attack. Let’s stay ahead of these malicious actors and keep the web — or at least our little corners of it — safer.

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

Indeed at FOSSASIA 2018

Indeed is a proud sponsor of the FOSSASIA Open Tech Summit, from March 22-25 in Singapore. FOSSASIA is Asia’s premier developer event and covers tracks from AI, Cloud, Blockchain Science Tech, Web, and more.

On March 24, Indeed speakers will present the following topics. We encourage you to attend.

How to Overcome Obstacles and Take Control of Your Career in Tech

Speaker: Doug Gray

Doug Gray is Senior Vice President of Engineering at Indeed. As a member of Indeed’s senior leadership team, Doug defines the organizational and development processes for the engineering team. His areas of expertise include product revitalization, process change and team development. Previously, Doug was a senior leader at Versata Software. Prior to that, he held a number of senior positions at Trilogy, where he worked for over 12 years. He holds a Bachelor of Science degree in Computer Science from Stanford University.

Finding the perfect job can be difficult. Figuring out how to stand out from the competition and then navigating human biases, archaic hiring processes and other challenges can make landing that dream job seem like a long shot. ​ ​

Learn how you can take control of your tech career path. Gain key insights from new research on what drives top tech talent’s career decisions​ while also learning how technology can help you overcome obstacles that may occur during the hiring process so that you can be in control of finding your dream job.

Indeed MPH: Fast and Compact Immutable Key-Value Stores

Speaker: Alex Shinn

Alex Shinn is an engineer working on Indeed’s job recommendation system. A former Google search engineer, he is a generalist with experience in natural language processing, building scalable architectures, and various AI technologies from expert systems to modern machine learning. In his free time, he writes Sheme compilers and libraries and contributes to the Scheme standardization process.

When you need to scale an application with a lot of data, how do you decide on a storage solution? How can you both safely store and efficiently interact with large data sets? This usually boils down to a choice between SQL or NoSQL — but what if there was a third option?

In this session, developer Alex Shinn will discuss Indeed’s MPH-Table: an open source storage solution that uses minimal perfect hash functions to generate a fast and compact key/value store. Alex will discuss how we use MPH-Table at Indeed, what situations are ideal for it, additional optimizations it allows, and how it compares to alternate solutions. Attendees should enter with a fundamental understanding of existing scalable storage solutions, and will leave with a basic understanding of MPH-Table, when they might want to use it, and how other solutions compare.

Find the full schedule for FOSSASIA, and more information about Doug and Alex’s presentations here.

We’re hiring! We have openings in Singapore for Engineering Manager, Full-Stack Software Engineer, Quality Assurance Automation Engineer and Quality Assurance Engineer. Tokyo openings include Manager, Data Science and Manager, Site Reliability Engineering. Hyderabad openings include Product Manager and Senior User Experience Designer.

Cross-posted on Medium.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone

Indeed Expands its Commitment to Open Source

At Indeed, we’re committed to an active role in open source communities. We’re proud to announce that we’ve joined the Cloud Native Computing Foundation (CNCF), an open source software foundation dedicated to making cloud-native computing universal and sustainable.

Cloud Native Computing Foundation logo

The CNCF, part of The Linux Foundation, is a vendor-neutral home for fast-growing projects. It promotes collaboration among the industry’s top developers, vendors, and end users. CNCF members include many leading public cloud providers and private cloud companies.

Deciding to join the CNCF was easy for us. We rely on open source technologies like OpenTracing to build and deliver best-of-class products. We believe we can serve job seekers and employers best by ensuring that open source software projects continue to grow.

Open source is now the industry model for practical software development. No longer relegated to Linux companies like Red Hat, the commercial success of open source has accelerated thanks to adoption by large technology companies such as IBM and Oracle. More recently, companies like Walmart and Verizon rely on open source programs and host their own open source projects. Microsoft has also become a major open source contributor.

Our open source projects include Proctor, an A/B testing framework that enables data-driven product design; LSM Tree, a fast key-value store for high-volume random access reads and writes; and our newest release, MPH, an immutable key-value store with efficient space utilization and fast reads.

Cloud strategies that leverage open source projects play an increasingly important role in business success. As a result, the demand for employees with open source development skills is also growing. Indeed research indicates skills in the following areas are in high demand for open source development roles: Java, Python, Git, JavaScript, Node.js, Docker, AngularJS, Jenkins, Amazon Web Services, and Agile.

Our strategic open source initiative involves partnerships, sponsorships and memberships … like our new CNCF role. In the next few months, we plan to announce sponsorships that support critical open source projects and help us engage targeted audiences. We’ll also continue to expand our open source team and become more involved in open source communities.

For updates on Indeed’s open source projects, visit our open source site. If you’re interested in open source roles at Indeed, visit our hiring page.

Doug Gray is the Senior VP of Engineering at Indeed.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on Google+Share on RedditEmail this to someone