GitHub header
All Systems Operational
Git Operations ? Operational
API Requests ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Mar 26, 2024

No incidents reported today.

Mar 25, 2024

No incidents reported.

Mar 24, 2024

No incidents reported.

Mar 23, 2024

No incidents reported.

Mar 22, 2024

No incidents reported.

Mar 21, 2024

No incidents reported.

Mar 20, 2024

No incidents reported.

Mar 19, 2024

No incidents reported.

Mar 18, 2024

No incidents reported.

Mar 17, 2024

No incidents reported.

Mar 16, 2024

No incidents reported.

Mar 15, 2024
Incident with Actions and Pages
Resolved - This incident has the same root cause as this incident.. Please follow the link to view the incident summary.
Mar 15, 20:28 UTC
Update - Actions is operating normally.
Mar 15, 20:27 UTC
Update - Pages is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:09 UTC
Investigating - We are investigating reports of degraded performance for Actions
Mar 15, 20:07 UTC
Incident with Codespaces and API Requests
Resolved - On March 15, 2024, between 19:42 UTC and 20:24 UTC several services were degraded due to a regression in calling the permissions system.

New GitHub Codespaces could not be created, as were Codespaces sessions that required minting a new auth token.

Actions saw delays and infrastructure failures due to the upstream dependency on fetching tokens for the repository for runs to successfully execute.

GitHub Pages were affected due to the impact on Actions, resulting in 1266 page builds failing, which at the low point represented 33% of page builds failing. This resulted in page edits not being reflected on those impacted sites.

We deployed an application update that included a newer version of our database query builder. The new version uses a newer MySQL syntax for upsert queries that is not supported by the database proxy service we use for some of our production-environment database clusters. This incompatibility impacted the permissions cluster specifically, causing requests that attempted such queries to fail.

We responded by rolling back the deployment, restoring the previous query use, and thus mitigated the incident.

We have identified and corrected a misconfiguration of the permissions cluster in our development and CI environments that will ensure queries utilize the proxy service to prevent future syntax additions causing issues in production.
Mar 15, 20:24 UTC
Update - Codespaces is operating normally.
Mar 15, 20:21 UTC
Update - API Requests is operating normally.
Mar 15, 20:20 UTC
Update - We rolled back the most recent deployment and are seeing improvements across all services, and will continue to monitor for additional impact.
Mar 15, 20:17 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:11 UTC
Update - Codespaces is experiencing degraded availability. We are continuing to investigate.
Mar 15, 20:03 UTC
Update - API Requests is experiencing degraded availability. We are continuing to investigate.
Mar 15, 20:03 UTC
Update - API Requests is experiencing degraded performance. We are continuing to investigate.
Mar 15, 20:00 UTC
Investigating - We are investigating reports of degraded performance for Codespaces
Mar 15, 19:55 UTC
Mar 14, 2024

No incidents reported.

Mar 13, 2024
Incident with Pull Requests
Resolved - From March 12, 2024 23:39 UTC to March 13, 2024 1:58 UTC, some Pull Requests updates were delayed and did not reflect the latest code that had been pushed. On average, 20% of Pull Requests page loads were out of sync and up to 30% of Pull Requests were impacted at peak. An internal component of our job queueing system was incorrectly handling invalid messages, resulting in stalled processing.

We mitigated the incident by shipping a fix to handle the edge case gracefully and allow processing to continue.

Once the fix was deployed at 1:47 UTC, our systems fully caught up with pending background jobs at 1:58 UTC.

We’re working to improve resiliency to invalid messages in our system to prevent future delays for these pull request updates. We are also reviewing our monitoring and observability to identify and remediate these types of failure cases faster.
Mar 13, 01:58 UTC
Update - Pull Requests is operating normally.
Mar 13, 01:58 UTC
Update - We believe we've found a mitigation and are currently monitoring systems for recovery.
Mar 13, 01:53 UTC
Update - We're continuing to investigate delays in PR updates. Next update in 30 minutes.
Mar 13, 01:18 UTC
Update - We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Mar 13, 00:47 UTC
Update - We're continuing to investigate an elevated number of pull requests that are out of sync on page load.
Mar 13, 00:12 UTC
Update - We're seeing an elevated number of pull requests that are out of sync on page load.
Mar 12, 23:39 UTC
Investigating - We are investigating reports of degraded performance for Pull Requests
Mar 12, 23:39 UTC
Mar 12, 2024
Incident with API Requests, Git Operations, Webhooks and Copilot
Resolved - On March 11, 2024 starting at 22:45 UTC and ending on March 12, 2024 00:48 UTC various GitHub services were degraded and returned intermittent errors for users. During this incident, the following customer impacts occurred: API error rates as high as 1%, Copilot error rates as high as 17%, and Secret Scanning and 2FA using GitHub Mobile error rates as high as 100% followed by a drop in error rates to 30% starting at 22:55 UTC. This elevated error rate was due to a degradation of our centralized authentication service upon which many other services depend.

The issue was caused by a deployment of network related configuration that was inadvertently applied to the incorrect environment. This error was detected within 4 minutes and a rollback was initiated. While error rates began dropping quickly at 22:55 UTC, the rollback failed in one of our data centers, leading to a longer recovery time. At this point, many failed requests succeeded upon retrying. This failure was due to an unrelated issue that had occurred earlier in the day where the datastore for our configuration service was polluted in a way that required manual intervention. The bad data in the configuration service caused the rollback in this one datacenter to fail. A manual removal of the incorrect data allowed the full rollback to complete at 00:48 UTC thereby restoring full access to services. We understand how the corrupt data was deployed and continue to investigate why the specific data caused the subsequent deployments to fail.

We are working on various measures to ensure safety of this kind of configuration change, faster detection of the problem via better monitoring of the related subsystems, and improvements to the robustness of our underlying configuration system including prevention and automatic cleanup of polluted records such that we can automatically recover from this kind of data issue in the future.
Mar 12, 01:00 UTC
Update - We believe we've resolved the root cause and are waiting for services to recover
Mar 12, 01:00 UTC
Update - API Requests is operating normally.
Mar 12, 00:56 UTC
Update - Git Operations is operating normally.
Mar 12, 00:55 UTC
Update - Webhooks is operating normally.
Mar 12, 00:54 UTC
Update - Copilot is operating normally.
Mar 12, 00:54 UTC
Update - We're continuing to investigate issues with our authentication service, impacting multiple services
Mar 12, 00:14 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Mar 11, 23:55 UTC
Update - Webhooks is operating normally.
Mar 11, 23:31 UTC
Update - Copilot is experiencing degraded performance. We are continuing to investigate.
Mar 11, 23:21 UTC
Update - Git Operations is experiencing degraded performance. We are continuing to investigate.
Mar 11, 23:20 UTC
Update - Webhooks is experiencing degraded performance. We are continuing to investigate.
Mar 11, 23:09 UTC
Investigating - We are investigating reports of degraded availability for API Requests, Git Operations and Webhooks
Mar 11, 23:01 UTC
← Incident History Powered by Atlassian Statuspage