Orkestro - Notice history

On Tuesday 20 October, from 07:02 to 08:42 UK time, the Orkestro platform was unavailable. Our API and most of the dashboard returned errors, and order processing was disrupted.

As you may have seen in the news, this was caused by a “power event” at one of Google Cloud’s London data centres, which disrupted many services across the UK (BBC News). Our databases are set up to switch to a healthy data centre automatically if one fails. That switch is run by Google, and this time it didn’t complete. Google has since confirmed that its failover stalled for an hour because its own systems were overloaded by the outage, and that our databases then stayed offline for another half an hour before they came back at 08:42.

Google provides our infrastructure, but keeping Orkestro running for you is our job, and we’re sorry for the disruption this caused. Some status updates that delivery partners sent us during the outage were lost and couldn’t be recovered afterwards. We’ve since corrected the orders affected by those missing updates. Our team has carried out a thorough internal postmortem, and we’re taking action to make sure we’re better covered if a third-party outage like this happens again.

If you have questions about this incident or its impact, please contact us at support@orkestro.com.

This incident has been resolved by the cloud provider. Our system is now back and running. We're going to continue to investigate and conduct an internal post-mortem.

A high-priority ticket has been raised with the major cloud provider, while we continue to explore alternatives from our side. We'll continue to update you here.

There's strong evidence indicating that this is an update gone wrong by one of our major Cloud Providers. We'll continue to investigate and provide updates.