The security of DigitalOcean customers and their data is a responsibility we approach with utmost dedication. When our customers’ security is threatened we respond swiftly, communicate with transparency, and take accountability, even if the incident root cause occurs beyond the boundaries of DigitalOcean systems.
Today, we wanted to share an update about how DigitalOcean and our customers were impacted by a recent security incident disclosed by Mailchimp. The key takeaways are below:
- On August 8th, DigitalOcean discovered that our Mailchimp account had been compromised as part of what we suspect to be a wider Mailchimp security incident that affected their customers, targeted at crypto and blockchain.
- From that Mailchimp incident, we suspect certain DigitalOcean customer email addresses may have been exposed. Out of an abundance of caution, we are currently sending email communications to those impacted.
- A very small number of DigitalOcean customers experienced attempted compromise of their accounts through password resets. These customers’ accounts have been secured, and have been contacted directly.
- As of August 9th, we have migrated email services away from Mailchimp.
- No customer information other than email address was compromised, however, we recommend increased vigilance against phishing attempts in the coming weeks, in addition to enabling two-factor authentication on your DigitalOcean account.
What happened?
At 3:30pm ET on August 8th, 2022 transactional emails from our platform, delivered through Mailchimp, stopped reaching our customers’ inboxes. This was discovered by an internal test run by engineering teams to monitor the health of our signup process. We quickly discovered our Mailchimp account had been suspended, with no access, and no other information being provided by Mailchimp. For DigitalOcean, and our customers, this meant email confirmations, password resets, email-based alerts for product health, and dozens of other transactional emails were not reaching their destination.
We received the following communication from Mailchimp:
During that same timeframe on August 8th, our Security Operations team was made aware of a customer who claimed their password had been reset, without their initiation. Recognizing a likely connection between our sudden loss of transactional email, and potentially malicious password resets, which are delivered via email, a security incident and investigation was launched in parallel with the teams addressing our email outage.
One of the first discoveries was a non-DigitalOcean email address that appeared on a regular email from Mailchimp on August 7th. The [@]arxxwalls.com email was not there on a similar Mailchimp email on August 6th. This led us to strongly believe our Mailchimp account was compromised.
Soon after we discovered an issue with our Mailchimp account on August 8th, we initiated contact with Mailchimp, both via traditional support channels and other escalation methods. On August 10th, we had our first actionable response, and conversation with the Mailchimp/Intuit Legal team to better understand the incident and impact on our account. We were formally notified on August 10th by Mailchimp of the unauthorized access to our and other accounts by what we understand to be an attacker who had compromised Mailchimp internal tooling.
The investigation into errant password resets on August 8th led us to a single IP address initiating a password reset against a limited set of DigitalOcean accounts. Our internal logging indicated the attacker IP address x.213.155.164 had successfully changed the password, but in the case below, failed to access the account due to the second-factor authentication on the account. The attacker did not attempt to complete the second factor.
Correlating password reset events from the attacker IP address via our API logging, we confirmed the small number of DigitalOcean accounts targeted by malicious password resets. Though not all resets were successful.
Our security incident response team took action to secure these accounts and have communicated separately with these customers from our broader notification about email address exposure. We can confirm the attacks against DigitalOcean customer account passwords stopped after August 7th.
While the security incident and investigation proceeded, the broader email outage incident management team decided to immediately migrate critical services away from Mailchimp to another email service provider. After working around the clock, our critical transactional emails were back online with another provider at 11pm ET August 9th.
What did we learn?
We have three key takeaways from this incident.
- The ecosystem is fragile, and chains of trust, when broken, can have significant downstream consequences. Our threat models and security visibility must improve in our third-party SaaS and PaaS environments.
- Our business continuity plans need to better account for downtime from third parties. As an infrastructure provider, we invest heavily in keeping our infrastructure available, but modeling loss of services where we rely on third parties can be improved.
- Two-factor authentication saved a handful of customers targeted by the attacker from complete account compromise. We will lean in with our customers to expand 2fa adoption.
Additionally, related but not as a direct result of this incident, we are evaluating two-factor authentication on-by-default for all DigitalOcean customer accounts. If you do not have 2fa enabled on your account, please enable it now.
Each day, our DigitalOcean team logs on with a commitment to acting as responsible data stewards. We understand that our customers’ trust and loyalty comes with transparency and reliability. If, after reading this communication, you have additional questions, please contact us via the information below – a member of our team will be happy to assist.
security-notifications@digitalocean.com
Thanks, stay safe, and happy coding.
DigitalOcean Security
