Dissecting Supply Chain Attacks: A Report on a Growing Sensitive Data Exposure Vector

Third-party risk has always been a concern for organizations, but since COVID and the rise of remote work, we’ve seen a dramatic acceleration in campaigns leveraging software supply chain attacks. Not just through open source vulnerabilities, but through closed source applications and services as well. To adapt to this new normal, it’s important to develop an understanding of supply chain attacks and protect yourself from them.

In this post, we’ll be going over recent GitHub campaigns that stemmed from two separate attacks. We’ll analyze these attacks in detail, cover their fallout for GitHub users, and provide advice for mitigating the impact of such consequences.

GitHub supply chain attack #1 - Heroku’s compromised OAuth tokens

What Happened?

Earlier this year Heroku, a cloud platform as a service (PaaS) owned by Salesforce, disclosed a cyberattack that took place on April 9th, 2022. The breach involved the exposure of OAuth user tokens for two GitHub integrations, Heroku and Travis CI.

Heroku has implied this breach provided the attacker access to data from their core database, which contained hashed and salted passwords belonging to customers. After some investigation, Heroku determined that the attack was more severe than initially thought, so they began forcing users to reset their passwords in an attempt to minimize the impact. However, the damage stemming from the OAuth tokens would present extended risk, as GitHub’s NPM organization was impacted by the theft of these tokens. Hackers stole an archive of NPM usernames, password hashes, and email addresses from NPM, as well as names and version numbers of published versions of all NPM private packages.

The attacker was likely able to gain access to Heroku’s system through a compromised token for a Heroku machine account. According to Travis CI, a Heroku service was also responsible for exposing a private application OAuth key responsible for integrating Heroku with Travis CI.

Story links: 

Security alert: Attack campaign involving stolen OAuth user tokens issued to two third-party integrators

npm security update: Attack campaign using stolen OAuth tokens

Why it matters

OAuth provides many of the biggest tech companies with ways to provide secure access to server resources. OAuth access tokens are leveraged by applications to make API requests on behalf of a user. Hackers with access to authentication tokens are likely granted access to critical systems at large. In the case of GitHub, it’s very likely for many private repositories to have secrets and credentials that grant access to other critical systems.

The repository listing activity was conducted by the attacker through abuse of third-party OAuth user tokens maintained by Heroku and Travis CI. Evidence suggests the attacker was able to use metadata to link customer repositories with OAuth tokens before the attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code.

The Attacker(s) approach according to GitHub

1. The attacker authenticated to the GitHub API using the stolen OAuth tokens issued to Heroku and Travis CI.
2. For orgs who had the affected Heroku or Travis CI OAuth apps authorized in their GitHub accounts, the attacker listed all the user’s organizations.
3. The attacker then selectively chose targets based on the listed organizations.
4. The attacker listed the private repositories for user accounts of interest.
5. The attacker then proceeded to clone some of those private repositories.

GitHub supply chain attack #1 - Travis CI’s credential leaks

What Happened?

In June 2022, security firm Aqua found that an unpatched issue in the Travis CI API left “tens of thousands” of developer’s user tokens exposed. This incident is completely distinct from the incident involving token exposure through Heroku back in April.

According to Aqua, more than 770 million logs of free tier users were publicly exposed, leaking tokens, secrets, and other credentials for popular cloud services.

Why it matters

Travis CI is one of the largest open source continuous integration services leveraged on GitHub and the service is leveraged by multitudes of people to connect to dozens of cloud services.

The type of data at risk

Examples of exposed data includes:

• GitHub access tokens and access keys to other third-party services (like AWS)
• Credentials to systems with critical databases SQL databases
• Docker Hub passwords 
• API keys to third party services like Twilio

Aqua provided a detailed analysis in their report, which contains the following graph: 

What is the risk from this type of attack?

Aqua, the security research team that discovered this issue, found that logs containing these secrets go back as far as January 2013. With a simple script, the security team at Aqua found that valid log requests could be made by unauthenticated users enumerating and appending any number to the URL of the API used to retrieve logs.

While not all logs contain secrets in clear text (some organizations use tools like Nightfall to redact text in logs), and while not all logs are for live services with sensitive data, the trivial ease by which an attacker could retrieve any number of logs makes this risk a serious concern. Aqua researchers estimated that at least 10% of the logs they discovered were for live services. This is apparently the only time Travis CI’s infrastructure has been exploited to reveal secrets, as documented by EdOverflow and a CVE from last year.

Aqua’s report illustrates an example scenario where a threat actor leverages Travis CI’s API for historic logs enumeration in order to access GitHub OAuth tokens, which then can be leveraged to escalate by providing the threat actor S3 access keys. While such a scenario is extremely plausible, since Travis CI integrates directly with a wide number of cloud applications, it’s entirely possible that with a simple script, any threat actor could have searched publicly available clear text logs for whatever types of credentials and access keys they wanted.

Story links:

https://blog.aquasec.com/travis-ci-security

Mitigating supply chain attacks in apps like GitHub

While supply chain attacks might catch organizations off guard, there are actions that can be taken to preemptively reduce data exposure risk. These include:

1. Examining applications that have access to your environments

SaaS platforms like GitHub make it easy to adopt a set-and-forget mentality when it comes to connecting third-party applications and services within your environments. It’s very easy to use an application for a one-time purpose and forget to revoke its access once you no longer need it. Depending on the scope of permissions provided by the application, this could pose a future security risk. That’s why it’s good to periodically review the applications that have access to your SaaS environments, and their scoped permissions, so you can have a decent idea of your security posture. Where appropriate, you should remove or limit access.

2. Periodically rotate secrets and OAuth tokens

Even assuming that an application has the appropriate permissions within your environment, you should rotate any secrets or tokens associated with it periodically. The good news is that the most common types of tokens and secrets often have a default expiration date. It might, however, sometimes be beneficial to manually manage how long your secrets remain valid.

3. Review audit logs and user activity for abnormal behavior

Within your SaaS environments, you should periodically review user activity and access to preemptively identify signs of suspicious behavior. For example, this could be the sudden appearance of new or unrecognized users, or existing users accessing content that they historically have never done.

Dividend Finance, a Nightfall customer, actually leveraged Nightfall for GitHub to identify indicators of compromise during the Heroku incident. Read more about GitHub security best practices, and how they responded to the incident.

4. Continuously scan your environments for credentials and secrets

Supply chain attacks like the ones highlighted in this post illustrate the need for continuous data security. That is to say, only by making sure that sensitive data is not easily accessible within your SaaS environments can you significantly reduce your risk of an escalated incident. This is something we talk about at length in a recent webinar on the subject of zero trust data security. You’ll need to invest in tools, like cloud-native data loss prevention, that can continuously monitor your environments for sensitive data like API keys in messages as well as in files like images and logs that might be embedded in your SaaS environments.