Google Workspace Status Dashboard
Incident affecting Google Vault
Incident began at 2022-01-20 23:30 and ended at 2022-01-21 18:38 (times are in Coordinated Universal Time (UTC)).
|Jan 27, 2022||7:40 PM UTC|| |
On Thursday, 20 January, Google Vault experienced elevated 403 errors, while attempting to download exports and saved cases, for a duration of 16 hours, 38 minutes. To our Google Vault customers who were impacted during this outage, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.
Google Vault is a service that is used to allow companies to manage, retain, research, and export their company email, on-the-record-chats, and Google Drive File content. Google Vault is not a backup or disaster recovery system. The primary use is for eDiscovery and compliance investigations and requests. On Thursday, 20 January at 15:30 US/Pacific, Google Vault Engineering changed a Google Cloud Storage configuration as part of an ongoing outage investigation. It was unknown at that time that this change would result in a 403 response when export download is attempted.
REMEDIATION AND PREVENTION
The incident was first detected Thursday, 20 January at 19:00, when a customer reported that they could not download a Vault Export. Google engineers immediately started an investigation. The incident was escalated to the Google Vault team for further investigation on Friday, 21 January at 08:00. The root cause of the issue was determined at 10:35, as related to enabling a configuration. This configuration was disabled at 10:38, fully resolving the impact.
We would like to apologize for the length and severity of this incident. We are taking the following steps to prevent a recurrence and improve reliability in the future.
|Jan 21, 2022||9:15 PM UTC|| |
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.
(All Times US/Pacific)
Incident Start: 20 January 2022 15:28 PT
Incident End: 21 January 2022 10:38 PT
Duration: 19 hours, 10 minutes
Affected Services and Features:
Google Vault - Vault Export
Google Vault experienced elevated 403 errors. From preliminary analysis, the root cause of the issue is due to a configuration change in Google Cloud Storage.
Customers experienced ‘Forbidden Error 403’ when attempting to download the results of a vault export.
The configuration change was reverted ending all customer impact on Friday, 2022-01-21 10:38 US/Pacific.
|Jan 21, 2022||6:49 PM UTC|| The problem with Google Vault has been resolved. We apologize for the inconvenience and thank you for your patience and continued support. |
Users were unable to download Vault Exports
|Jan 21, 2022||5:58 PM UTC|| We're investigating reports of an issue with Google Vault. We will provide more information shortly. The affected users are able to access Google Vault, but are seeing error messages, high latency, and/or other unexpected behavior. |
Users may be unable to download exports from Vault
- Times are listed in Coordinated Universal Time (UTC)
- JSON History | Schema
- Send feedback
- Service information
- Service disruption
- Service outage