On Friday, November 3rd, Sumo Logic discovered that a compromised credential was used to access Sumo Logic’s AWS account. Since then, Sumo Logic rotated the exposed AWS credentials and locked down potentially affected infrastructure, and reported they didn’t detect access to customer’s data.
Nonetheless, Sumo Logic still suggested that customers rotate all Sumo Logic API access keys immediately.
For extra caution, since the investigation is still ongoing and until further details are given, Astrix also recommends rotating all types of non-human identities credentials saved on Sumo Logic’s systems, per Sumo Logic’s original guidelines:
- Installed collector credentials.
- Third-party credentials stored with Sumo for data collection by a hosted collector.
- Third-party credentials stored with Sumo for webhook configuration.
- Sumo Logic user passwords.
Next steps guide: How to successfully rotate each type of credentials
Sumo Logic integrates to and collects data from many different sources, both cloud or on-prem. Depending on the type of connection, a specific credential is necessary for the integration to work. This makes it especially hard for Sumo Logic administrators to handle the recommended approach to handle the recent breach.
As a first step, Astrix experts reviewed the necessary actions and dove deep into the different credentials that could have been affected by this breach. Below you will find a list explaining the different credentials required for rotation, including links to relevant documentation.
Sumologic API Access Keys
- What are these credentials?
These keys are simply used to access Sumo Logic’s API. They can be used as part of an automated service or via an installed collector (see below) - What’s the risk?
Threat actors controlling these keys could access data stored in your Sumo Logic account and view potentially sensitive data. - How do you rotate them?
As a Sumo Logic administrator, you can view all access keys created in your account. On that page, you can disable and delete all access keys to force all users and services who used them to create a new access key.
While on this management page, it is recommended to have your default deactivation period for access key to be below 30 days.
Installed collector credentials
- What are these credentials?
These collectors are usually installed directly on machines across your organization, digest local logs generated by the system and send them to Sumo Logic. To authenticate and register the collector, a Sumo Logic access key or installation token is required to serve as the credentials for this installed collector. - What’s the risk?
Threat actors that have access to Sumo Logic API access keys can call Sumo Logic APIs on behalf of the user that created the access key. These APIs can be used to read data off Sumo Logic, which may contain sensitive information about your critical systems. - How do you rotate them?
Sumo Logic API access keys can be deactivated through the UI. A new access key then needs to be generated and replaced in the user.properties file of the installed collector.
Third-party credentials stored on Hosted collectors
- What are these credentials?
These are credentials created as part of the process of setting up an integration between a SaaS or cloud platform and Sumo Logic, to pull logs from the platform to Sumo Logic for ingestion. They take many different forms depending on the platform – they can be API keys, temporary or permanent tokens, service accounts or users created specifically for the integration. - What’s the risk?
Depending on the specific platform, these credentials have access to your data across SaaS and cloud platforms. A threat actor can use these to read sensitive data from platforms such as Google Workspace, Azure, AWS, Github, Slack and others. In some cases, the threat actor can even use them to perform management actions, such as creating users, changing permissions, and editing configurations to weaken your organization’s security. - How do you rotate them?
This greatly varies depending on which platforms were collected by your Sumo Logic collectors. Astrix recommends filtering “Hosted collectors” on your Sumo Logic collectors management page. Then, for each hosted collector source displayed under them (which isn’t HTTP-based), view the credential used to configure the source. Deactivate or delete it from the platform, and review its documentation article under Hosted Collectors and Sources to find out how to create a new credential.
Note: Formerly active keys provided to Sumo Logic that are not used in existing connectors might still have access to your environment and should be removed. Astrix can easily provide visibility into these inactive keys.
Third-party credentials stored for webhook configuration
- What are these credentials?
Several Hosted collectors (i.e: Github, Gitlab, Hasura) utilize webhooks for log digestion, instead of creating a credential. - What’s the risk?
A webhook URL should be considered sensitive, since anyone knowing the URL and configuration of the webhook can send faulty messages to the digestion target – potentially affecting critical systems’ stability and security. - How do you rotate them?
Similar to a Hosted Collector’s credentials, look for “Hosted Collectors“ in the collectors management page, except now you need to review only “HTTP” sources. Click on “Regenerate URL” and update the new URL on the relevant SaaS platform.
Sumo Logic user passwords
- What are these credentials?
These are simply passwords your Sumo Logic users are using to log into the Sumo Logic console and UI. - What’s the risk?
Threat actors that have these passwords of users that do not have MFA enabled, could log in to your Sumo Logic dashboard and view potentially sensitive data. - How do I rotate them?
As a Sumo Logic admin, go to the Users and Roles page. Click the 3 dots next to each user and then choose “Reset Password”.This will send the user an email with their new password. While handling this case, it is recommended to force MFA on all your Sumo Logic users.
Key takeaways:
- Integrating SaaS platforms into your environment entails storing sensitive access credentials to your private data on third-party servers – a breach into their infrastructure also affects your data (aka supply chain attacks). Knowing which third-party platforms have access to your systems, and monitoring their access credentials (API keys, tokens, webhooks, service accounts, OAuth tokens etc) is crucial for your organization’s security.
- The way tokens and credentials are created and handled makes rotating credentials and resetting integrations a difficult task in most platforms. It involves many different platform admins in your organization and potential dependencies that could “break” stuff. This means you need to always be prepared for a situation where you need to rotate many credentials – fast.
- Exposed credentials have the potential to cause a major security breach. It’s important to have a system in place that continuously scans your environments for exposed secrets.
The Astrix Security platform is the first and only tool built to easily monitor and secure all non-human access to your core platforms and data. Astrix gives you visibility, security context and threat detection & remediation to all non-human access credentials, helping you prevent supply chain exploits just like this one. Book a live demo to learn how.
