Trust is a big part of why Indeed continues to be the world’s number one job site. Users trust us to keep their information safe. We’ve always taken our responsibilities seriously, and as abuse of personal data continues to dominate the news, we cannot afford to lose diligence. That’s why we need to keep our own data safe — not just to protect Indeed employees and corporate data, but also to protect the data that our users entrust us with.
I work on Indeed’s Information Security team and in early 2017 started to address shortcomings in our Google GSuite implementation. Our solution — the OAudit Toolbox — is now available as an open source tool that you can use as well.
The problem: Third party apps can be risky
A major area of risk for Indeed’s GSuite implementation is integrating third party apps.
GSuite users can grant applications access to their account — anything from basic account info to full read/write access to Gmail. The OAuth Scopes presented in the authorization prompt control access to Google resources.
For businesses running GSuite, this presents a number of issues:
User education. Users may unknowingly grant access, or they might not understand the privacy or security implications of their choices.
Data sharing. Authorizing apps for certain scopes allows third parties access to sensitive data. Your business might not have the proper data sharing contracts in place for this kind of access.
Data exfiltration. Malicious applications can use this OAuth flow to effectively phish and exfiltrate data from Google accounts.
Limited tooling. At the time of my investigation, Google’s options for restricting apps were limited: the API only allowed for reactive blacklisting. The relatively new feature allowing for whitelisting of connected apps requires that you actually have a whitelist of apps.
Policy culture. Many companies allow overly permissive scope access. Implementing Google’s new whitelist functionality could require your security team to review, approve, and whitelist a giant backlog of apps — not to mention fielding requests from employees who feel their productivity depends on Sketchy Mail Tracker Pro™.
It wasn’t long before we had a real-world attack that highlighted these issues and helped us define our own solution.
The proof: A massive phishing attack
In May 2017, a massive phishing attack masquerading as a Google Docs invitation hit Google Apps users all over the world.
Users received an email, apparently sent from one of their contacts, requesting that they view a shared document. If they worked for a business using GSuite, this wasn’t an unusual request. What was unusual was this fake “Google Docs” requesting permission to access the recipient’s email and contacts. If users granted this access, the fake “Google Docs” app then sent the same phishing email onward to the victim’s contacts, using the victim’s account and masquerading as them.
As part of Indeed’s Security team, I was on the front lines of our company’s response to this attack. After several hours of panic and revocation of OAuth tokens, I felt oddly inspired. I wanted to fill in the gaps in GSuite’s available tooling, detect attacks like this one sooner, and find a way to educate users about the dangers of authorizing third party apps. Along with my coworker Dustin Decker, I started work on a set of tools that might get us closer to a perfect solution.
Our solution: OAudit Toolbox
OAudit Toolbox is a set of tools that detects third party app integrations and notifies users of their danger and scope.
How OAudit Toolbox works
OAudit Toolbox contains two major components:
- Oaudit-collector indexes authorization events from the Google Admin API into Elasticsearch
- Oaudit-notifier sends notifications with educational information about Oauth scopes and contains whitelisting/blacklisting logic
Blacklisting allows us to revoke access to a list of known bad apps in near-real time, including malicious apps and apps in violation of corporate policy. Access is only revoked to apps authorized after the app is defined in the blacklist.
Whitelisting allows us to stop notifications for trusted apps so that users don’t suffer alert fatigue.
- Oaudit-collector fetches authorization token event data using the Google Admin SDK API.
- Oaudit-collector indexes fetched data in Elasticsearch.
- Oaudit-notifier checks to see whether the authorized application is whitelisted, blacklisted, or unknown.
- If the app is whitelisted, a notification is not sent. This is typically used for apps that have passed security review, and in the case of third party apps, have the appropriate data sharing agreements in place (if applicable).
- If the app is blacklisted, a notification is sent to the user that the app is blacklisted and access has been revoked. This is used for malicious apps (such as the ones seen in the 2017 phishing attack) and apps that go against corporate policy.
- Apps that are neither blacklisted nor whitelisted trigger a notification to the user informing them of the potential risk of authorizing untrusted applications and how to revoke unwanted access.
How OAudit Toolbox helps
OAuth Toolbox solves, or at least mitigates, each of the issues I identified with using GSuite.
Solution: User education
There are a million reasons why users might authorize third party apps:
- A manager told them to
- They don’t understand the authorization prompt
- They haven’t heard of all the fun acronyms like DPA and MNDA that help us more securely share data with third parties
- And many more!
With OAudit enabled, users receive an easy-to-understand visualization of the risks associated with each third party app, including user-friendly descriptions of the risks. Each scope is assigned a score based on the risk of sharing sensitive data, and highlighted with an associated color.
After enabling this feature, we saw a significant increase in questions from our users about whether apps are safe to use. We received more requests for our application security team to review third party apps. We were also contacted by teams outside of our engineering organization. Non technical users had previously felt uneasy about using some third party tools but lacked the technical or security context to explain why.
Solution: Data sharing
Users add tools to Google Apps to do things like improve spreadsheet visualizations, enable mail marketing campaigns, or send themselves Google Form results.
To the typical user, it’s not obvious that these integrations exist on a third party’s server rather than within GSuite itself. Some users assume that Google thoroughly vets each app. Other users don’t realize that once this data resides on third party equipment, they have no control over it beyond contractual agreements regarding further usage and sharing. For companies that must comply with GDPR, this issue goes beyond security and becomes regulatory.
Using the OAudit Toolbox has helped us socialize the concept of high-risk data sharing. At the same time, we have been able to work with our privacy and contracts teams to get the appropriate agreements in place where needed and revoke access to those apps that don’t pass our assessments. Retroactively revoking access to unapproved (but non-malicious) apps using the blacklist has been reasonably effective, as the expected functionality of these apps does not involve immediate data exfiltration.
Solution: Data exfiltration
While there are legitimately useful third party apps, there are also malicious apps masquerading as useful tools, such as the “Google Docs” app involved in the phishing attack of 2017.
You can always revoke a malicious app’s access, but doing so is minimally helpful because these apps are likely to exfiltrate data and/or abuse your account as soon as possible. The reliability of this method also depends on the timeliness of token logs — which, during the Google Docs phishing attack, were as far as 12 hours behind. More recently, token activity lag time is between 1-10 minutes, though Google claims this can be as much as a few hours.
Since the OAuth Toolbox also sends data to Elasticsearch, we recommend setting up ElastAlert or Watcher to detect never-before-seen apps being authorized or a spike in a single app being authorized in a short period of time.
By proactively warning users about dangers, OAudit Toolbox is more like an early alerting system for malicious apps, or IDS, and less like a proactive blocker, such as an IPS.
Solution: Available tooling
When development on OAuth Toolkit started, the ability to block app access was scattered across GSuite. A Google Apps administrator could block Marketplace Apps, Drive API, and Chrome extensions. That would have been too heavy-handed for us to successfully implement. Administratively, it was difficult to manage with minimal ROI since the solution was incomplete.
Google now allows you to block OAuth access to Google Apps such as Drive, GMail, and Contacts using the Google Admin Security panel. Optionally, you can block only “high risk” scopes, but there is no documentation as to which scopes are considered high risk. There is also a whitelist available to allow use of trusted apps.
This is a useful addition if and when your team is ready and able to:
- Block all apps in use
- Know what you need to whitelist
- Have a workflow in place for approving new applications
- Have application security resources for reviewing those apps
We were glad to see Google take these steps, but OAudit Toolkit provided us a solution that was easier to implement and less disruptive to workflow.
Solution: Policy culture
A major hurdle for implementing this tool and the subsequent review process wasn’t technical, but human. Going from an open, “bring your own app” culture to a more restrictive, seemingly bureaucratic process is a struggle — especially when transparency is a keystone of company culture.
We found a few practices to be helpful:
- Use tools (such as OAudit Toolbox), training, tech talks, or internal blog posts to introduce users to the risk posed by third party apps.
- Instead of adding a pile of legalese to your Acceptable Use policy, keep any actions users need to perform simple and available on an easily accessed page: the intranet home page, a wiki page, or an IT Support landing page.
- Bucket frequently used, high-risk tools into categories such as “Mail Merge” or “Spreadsheet automation.” Doing so makes it easier to rip-and-replace them with a single, approved app rather than evaluate 50 unique apps.
- Accept that you might not solve the problem overnight. Even if you can only mitigate the problem by allowing continued access to some apps and approving or restricting new apps, that still puts you in a better place than you were before.
Get started with OAudit Toolbox
We’ve made OAudit Toolbox open source and available so you can benefit from our experience. By integrating it with your company’s Google suite, you can take similar steps to improve organizational education and, hopefully, better resist the next phishing attack. Let’s stay ahead of these malicious actors and keep the web — or at least our little corners of it — safer.
Improving Security with OAudit Toolbox cross-posted on Medium.