GitHub header
All Systems Operational
Git Operations ? Operational
Webhooks ? Operational
Visit www.githubstatus.com for more information Operational
API Requests ? Operational
Issues ? Operational
Pull Requests ? Operational
Actions ? Operational
Packages ? Operational
Pages ? Operational
Codespaces ? Operational
Copilot Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Aug 30, 2024

No incidents reported today.

Aug 29, 2024
Disruption with some GitHub services
Resolved - This incident has been resolved.
Aug 29, 21:54 UTC
Update - The connectivity issues have been resolved and we are back to normal.
Aug 29, 21:54 UTC
Update - We have implemented a potential fix and are continuing to monitor for success.
Aug 29, 21:45 UTC
Update - We have isolated the symptom of the connectivity issues and are working to trace down the cause.
Aug 29, 21:25 UTC
Update - While we have seen a reduction in reports of users having connectivity issues to GitHub.com, we are still investigating the issue.
Aug 29, 20:43 UTC
Update - We are continuing to investigate issues with customers reporting temporary issues accessing GitHub.com
Aug 29, 20:07 UTC
Update - We are getting reports of users who aren't able to access GitHub.com and are investigating.
Aug 29, 19:29 UTC
Investigating - We are currently investigating this issue.
Aug 29, 19:29 UTC
Aug 28, 2024
Disruption with some GitHub services
Resolved - On August 28, 2024, from 21:40 to 23:43 UTC, up to 25% of unauthenticated dotcom traffic in SE Asia (representing <1% of global traffic) encountered HTTP 500 errors. We observed elevated error rates at one of our global points of presence, where geo-DNS health checks were failing. We identified unhealthy cloud hardware in the region, indicated by abnormal CPU utilization patterns. As a result, we drained the site at 23:26 UTC, which promptly restored normal traffic operations.
Aug 28, 23:43 UTC
Update - Our mitigation has taken effect and impact is resolved.
Aug 28, 23:43 UTC
Update - Our mitigation is in place and we are seeing a reduction in overall impact. We are continuing to monitor until we are confident that the issue has been resolved.
Aug 28, 23:38 UTC
Update - We have identified a potential mitigation and are currently testing.
Aug 28, 23:33 UTC
Update - We are experiencing reduced overall impact, but continue to see a continued small impact to unauthenticated requests. We are continuing to investigate.
Aug 28, 23:14 UTC
Update - We are continuing to see customer impact and are still investigating.
Aug 28, 22:53 UTC
Update - We are continuing to investigate.
Aug 28, 22:37 UTC
Update - We are seeing cases of user impact in some locations are continuing to investigate.
Aug 28, 22:19 UTC
Update - We are seeing elevated traffic in some of our regions that we are in process of investigating.
Aug 28, 22:06 UTC
Investigating - We are currently investigating this issue.
Aug 28, 22:02 UTC
Aug 27, 2024
Disruption with some GitHub services
Resolved - This incident has been resolved.
Aug 27, 23:26 UTC
Update - We are no longer seeing any traffic going through the affected routes and this issue should be resolved.
Aug 27, 23:25 UTC
Update - All traffic should be rerouted by now; and we have seen a complete drain from the affected provider. We are performing final validations that networking traffic is back to normal.
Aug 27, 23:20 UTC
Update - We have drained connections that would be running through the affected routes and are waiting for caches to expire in order to validate that issues are resolved.
Aug 27, 23:03 UTC
Update - We have identified some potential connectivity issues among public Internet transit routes and are attempting to reroute.
Aug 27, 22:50 UTC
Update - Git Operations is experiencing degraded performance. We are continuing to investigate.
Aug 27, 22:46 UTC
Update - Issues is experiencing degraded performance. We are continuing to investigate.
Aug 27, 22:45 UTC
Update - We are experiencing some issues related to TLS/SSL encrypted connections and are currently investigating.
Aug 27, 22:38 UTC
Investigating - We are currently investigating this issue.
Aug 27, 22:37 UTC
Aug 26, 2024

No incidents reported.

Aug 25, 2024

No incidents reported.

Aug 24, 2024

No incidents reported.

Aug 23, 2024

No incidents reported.

Aug 22, 2024
Incident with Actions
Resolved - On August 22, 2024, between 16:10 UTC and 17:28 UTC, Actions experienced degraded performance leading to failed workflow runs. On average, 2.5% of workflow runs failed to start with the failure rate peaking at 6%. In addition we saw a 1% error rate for Actions API endpoints. This was due to an Actions service being deployed to faulty hardware that had an incorrect memory configuration, leading to significant performance degradation of those pods due to insufficient memory.

The impact was mitigated when the pods were evicted automatically and moved to healthy hosts. The faulty hardware was disabled to prevent a recurrence. We are improving our health checks to ensure that unhealthy hardware is consistently marked offline automatically. We are also improving our monitoring and deployment practices to reduce our time to detection and automated mitigation at the service layer for issues like this in the future.
Aug 22, 17:28 UTC
Update - We are investigating issues with failed workflow runs due to internal errors. We are seeing signs of recovery and continuing to monitor the situation.
Aug 22, 17:21 UTC
Investigating - We are investigating reports of degraded performance for Actions
Aug 22, 16:49 UTC
Aug 21, 2024
Incident with Actions
Resolved - On August 21, 2024, between 13:48 UTC and 15:00 UTC, Actions experienced degraded performance, leading to delays in workflow runs. On average, 25% of workflow runs were delayed by 8 minutes. Less than 1% of workflow runs exhausted retries and failed to start. The issue stemmed from a backlog of Pull Request events which caused delays in Actions processing the event queues that trigger workflow runs.

We mitigated the incident by disabling the process that led to the sudden spike in Pull Request events. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future. We are also identifying appropriate changes to rate limits and reserved capacity to reduce the breadth of impact.
Aug 21, 15:11 UTC
Update - We have seen recovery and Actions workflow runs are now running as expected.
Aug 21, 15:10 UTC
Update - We are seeing reduced delays for Actions workflow runs to get triggered. We are continuing to investigate how to further reduce impact on customers and recover more quickly.
Aug 21, 14:51 UTC
Update - We are investigating reports of users seeing delays for Actions workflow runs to get triggered.
Aug 21, 14:15 UTC
Investigating - We are investigating reports of degraded performance for Actions
Aug 21, 14:09 UTC
Aug 20, 2024

No incidents reported.

Aug 19, 2024

No incidents reported.

Aug 18, 2024

No incidents reported.

Aug 17, 2024

No incidents reported.

Aug 16, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage