Incident Report

Summary

On 06 August at 08:00 US/Pacific, Chrome OS devices experienced issues with logins globally for a duration of 19 hours 29 minutes. During this time, users on ChromeOS devices were able to complete the full sign-in process, including any Multi-factor Authentication when required. However, some users were immediately redirected back to the login page after logging in.

Root Cause

Google’s internal authentication system uses an HTTP endpoint microservice that provides clients with a list of accounts currently in the authentication cookies. This endpoint is mainly used for various account switcher user interfaces, providing a list of accounts the current session has access to.

The root cause of the issue is an incorrect configuration made to the Google authentication service, which resulted in a malformed response to the queries from this microservice in a particular condition. As a result, Chrome OS was unable to get the account details from the internal Google authentication service, which caused the sign-in failures.

Remediation and Prevention

Google engineers were alerted to the outage via a support case on Tuesday, 06 August at 15:51 US/Pacific, and immediately started an investigation. Once the nature and scope of the issue became clear, the configuration change was reverted and Google engineers ensured that sign-in worked again.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Migrate the endpoints used in Google authentication systems to a different format so that the issue is not triggered
• Expand the integration testing environment to include more clients including Google Chrome so that all scenarios are identified
• Include additional control mechanisms to ensure that configuration changes are approved appropriately

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

• On devices which were already configured, users were unable to add new users from the login screen.
• Users were able to successfully authenticate however, and were sent back to the login screen before starting a session.
• Users were unable to setup the first user account on their device (the device owner), resulting in the devices being reset to factory defaults

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 2024-08-06 08:00

Incident End: 2024-08-07 03:29

Duration: 19 hours 29 minutes

Affected Services and Features:

Chrome OS

Regions/Zones: Global

Description:

Chrome OS devices experienced issues with login due to a recent change to a backend identity service, causing incorrect parsing of account details. Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

• During the full sign in process ChromeOS devices completed the sign in (even Multi Factor Authentication (MFA) if it was applicable) but were eventually being brought back to the login page

Summary:

ChromeOS sign in issues affecting multiple customers throughout the globe

Description:

We are experiencing an issue with ChromeOS.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-08-07 06:00 US/Pacific with current details.

Diagnosis:

During the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.

The issue is affecting ChromeOS versions 120 to 128 (potentially more versions).

Workaround:

None at this time.

Summary:

ChromeOS sign in issues affecting multiple customers throughout the globe

Description:

We are experiencing an issue with ChromeOS.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-08-07 01:30 US/Pacific with current details.

Diagnosis:

During the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.

The issue is affecting multiple ChromeOS versions

Workaround:

None at this time.

Summary:

ChromeOS sign in issues affecting multiple customers throughout the globe

Description:

We are experiencing an issue with ChromeOS.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-08-06 23:30 US/Pacific with current details.

Diagnosis:

During the full sign in process ChromeOS devices complete the sign in process (even Multi Factor Authentication (MFA) if applicable) but are eventually brought back to the login page.

The issue is affecting multiple ChromeOS versions

Workaround:

None at this time.