Late last month, a company using the Google Cloud Platform had an incident where a critical application was immediately suspended by Google without warning. This was a cause for alarm given how many apps today run in the cloud. Google today announced a series of improvements to prevent any more sudden suspensions.
The anonymous GCP customer detailed the experience on Medium, noting that the project taken offline was a dashboard used to “monitor hundreds of wind turbines and scores of solar plants scattered across 8 countries.” Used 24/7 by a number of teams at that company, they received an alert that the site was down and a “barrage of emails from Google saying there is some ‘potential suspicious activity’ and all my systems have been turned off.”
EVERYTHING IS OFF. THE MACHINE HAS PULLED THE PLUG WITH NO WARNING. The site is down, app engine, databases are unreachable, multiple Firebases say i’ve been downgraded and therefore exceeded limits.
In attempting to resolve, the remedy involved a strict series of steps with no margin for error or lenience. Namely, if that specific violation wasn’t resolved within three business days, the entire project would be deleted. As the customer notes, “You just can’t turn things off and then ask for an explanation.”
What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
Google Cloud is addressing this today by getting rid of “automatic fraud-based account suspensions for established customers with offline payment” and “delayed suspension for established online customers.” Other users will now have a second human review before suspension.
We’re also modifying who has authority to suspend an account, as well as refreshing our training for the teams that review flagged accounts and determine response actions; re-evaluating the signals, sources, and the tools we use to assess potential fraudulent activity; and increasing the number of options we can use to help new customers quickly and safely grow their usage while building an account history with Google.
Meanwhile, Google is fully rolling out 24×7 chat support for customers that receive account notices by September, as well as updating project suspension guidelines and improving ways for users to reach Google.