Google Workspace Status Dashboard

This page provides status information on the services that are part of Google Workspace. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://workspace.google.com/. For incidents related to Google Analytics, visit the Google Ads Status Dashboard.

Incident affecting Google Vault

Incident began at 2022-01-20 23:30 and ended at 2022-01-21 18:38 (times are in Coordinated Universal Time (UTC)).

Date Time Description
Jan 27, 2022 7:40 PM UTC

INCIDENT REPORT

SUMMARY

On Thursday, 20 January, Google Vault experienced elevated 403 errors, while attempting to download exports and saved cases, for a duration of 16 hours, 38 minutes. To our Google Vault customers who were impacted during this outage, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

ROOT CAUSE

Google Vault is a service that is used to allow companies to manage, retain, research, and export their company email, on-the-record-chats, and Google Drive File content. Google Vault is not a backup or disaster recovery system. The primary use is for eDiscovery and compliance investigations and requests. On Thursday, 20 January at 15:30 US/Pacific, Google Vault Engineering changed a Google Cloud Storage configuration as part of an ongoing outage investigation. It was unknown at that time that this change would result in a 403 response when export download is attempted.

REMEDIATION AND PREVENTION

The incident was first detected Thursday, 20 January at 19:00, when a customer reported that they could not download a Vault Export. Google engineers immediately started an investigation. The incident was escalated to the Google Vault team for further investigation on Friday, 21 January at 08:00. The root cause of the issue was determined at 10:35, as related to enabling a configuration. This configuration was disabled at 10:38, fully resolving the impact.

We would like to apologize for the length and severity of this incident. We are taking the following steps to prevent a recurrence and improve reliability in the future.

  • Improve the testing and configuration changes process including testing more frequently to better detect issues in staging environments before reaching production.
  • Extend monitoring to include these critical user journeys and alert on elevated errors.
Jan 21, 2022 9:15 PM UTC

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 20 January 2022 15:28 PT

Incident End: 21 January 2022 10:38 PT

Duration: 19 hours, 10 minutes

Affected Services and Features:

Google Vault - Vault Export

Regions/Zones: Global

Description:

Google Vault experienced elevated 403 errors. From preliminary analysis, the root cause of the issue is due to a configuration change in Google Cloud Storage.

Customer Impact:

Customers experienced ‘Forbidden Error 403’ when attempting to download the results of a vault export.

Additional details:

The configuration change was reverted ending all customer impact on Friday, 2022-01-21 10:38 US/Pacific.

Jan 21, 2022 6:49 PM UTC The problem with Google Vault has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Users were unable to download Vault Exports

Jan 21, 2022 5:58 PM UTC We're investigating reports of an issue with Google Vault. We will provide more information shortly. The affected users are able to access Google Vault, but are seeing error messages, high latency, and/or other unexpected behavior.

Users may be unable to download exports from Vault