INCIDENT REPORT

SUMMARY

On Thursday, 20 January, Google Vault experienced elevated 403 errors, while attempting to download exports and saved cases, for a duration of 16 hours, 38 minutes. To our Google Vault customers who were impacted during this outage, we sincerely apologize – this is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

ROOT CAUSE

Google Vault is a service that is used to allow companies to manage, retain, research, and export their company email, on-the-record-chats, and Google Drive File content. Google Vault is not a backup or disaster recovery system. The primary use is for eDiscovery and compliance investigations and requests. On Thursday, 20 January at 15:30 US/Pacific, Google Vault Engineering changed a Google Cloud Storage configuration as part of an ongoing outage investigation. It was unknown at that time that this change would result in a 403 response when export download is attempted.

REMEDIATION AND PREVENTION

The incident was first detected Thursday, 20 January at 19:00, when a customer reported that they could not download a Vault Export. Google engineers immediately started an investigation. The incident was escalated to the Google Vault team for further investigation on Friday, 21 January at 08:00. The root cause of the issue was determined at 10:35, as related to enabling a configuration. This configuration was disabled at 10:38, fully resolving the impact.

We would like to apologize for the length and severity of this incident. We are taking the following steps to prevent a recurrence and improve reliability in the future.

• Improve the testing and configuration changes process including testing more frequently to better detect issues in staging environments before reaching production.
• Extend monitoring to include these critical user journeys and alert on elevated errors.

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 20 January 2022 15:28 PT

Incident End: 21 January 2022 10:38 PT

Duration: 19 hours, 10 minutes

Affected Services and Features:

Google Vault - Vault Export

Regions/Zones: Global

Description:

Google Vault experienced elevated 403 errors. From preliminary analysis, the root cause of the issue is due to a configuration change in Google Cloud Storage.

Customer Impact:

Customers experienced ‘Forbidden Error 403’ when attempting to download the results of a vault export.

Additional details:

The configuration change was reverted ending all customer impact on Friday, 2022-01-21 10:38 US/Pacific.