Issues with configuration updates propagating

2023-08-18T12:17:07Z

Aug 18, 12:17 UTC
Resolved - This issue has been resolved. We have temporarily switched all services back to our "old infrastructure" and are running additional tests as well as working with Fastly before we reopen the "new infrastructure".

We will also publish additional details once we conclude our internal post mortem process.

Aug 18, 10:33 UTC
Monitoring - Fastly has implemented a fix for the issue, all services are working as expected again.

We have temporarily disabled switching over to the new infrastructure and are working with Fastly to better understand what happened on their end, why it took so long to identify and rectify this and how we can better monitor and prevent this in the future. We well enable the new infrastructure again, once we are confident in any services we rely on.

Aug 18, 09:53 UTC
Update - We continue working with Fastly to resolve this issue. Please see https://www.fastlystatus.com/incident/376022 for updates from their team as well.

Aug 18, 07:29 UTC
Update - We are continuing to work on a fix for this issue.

Aug 18, 07:15 UTC
Update - The incident with KV stores, which are used for service configuration, is now spreading to additional edge locations and affecting overall service availability for services on the new infrastructure. We have disabled the new infrastructure to provide our partner more time to identify and resolve the issue on their end.

Aug 17, 16:03 UTC
Identified - Our infrastructure partner has identified the issue and is working on fixing it.

Aug 17, 13:56 UTC
Update - We are continuing to investigate this issue together with our infrastructure providers.

If you haven't made configuration changes to your service recently, you are not affected by this issue.

Aug 17, 11:55 UTC
Investigating - We are investigating an issue with configuration updates propagating to the respective services. If you didn't make configuration changes recently, your services are not impacted by this incident.