Sep 10, 01:54 UTC Resolved - We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full pos…
Sep 4, 02:39 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 4, 02:23 UTC Monitoring - We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring. Sep 4, 01:42 UTC Investigating - We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as…
Sep 4, 01:40 UTC Investigating - We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Sep 3, 07:06 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 3, 06:44 UTC Monitoring - We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs. Sep 3, 06:09 UTC Update - Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating…
Sep 2, 15:51 UTC Resolved - Job dispatch for Hosted Agents is no longer experiencing delays. Sep 2, 15:43 UTC Monitoring - We are continuing to process through the backlog of delayed jobs and monitoring. Sep 2, 15:29 UTC Identified - We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs. Sep 2, 15:16 UTC Update - O…
Sep 1, 16:52 UTC Resolved - Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully. Sep 1, 16:38 UTC Monitoring - We are seeing successful email deliveries and are monitoring processing of our outbound email queue. Sep 1, 16:19 UTC Update - Our on-call engineers are working on failing over to an alternative upstream email…
Aug 27, 11:47 UTC Resolved - Processing of uploaded test execution data has caught up and is back to normal. Aug 27, 11:23 UTC Update - Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing. Aug 27, 10:56 UTC Monitoring - Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all d…
Aug 26, 03:13 UTC Resolved - Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now. Aug 26, 02:43 UTC Monitoring - Recovery is underway; we'll keep monitoring until recovery is complete. Aug 26, 02:36 UTC Update - We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery.We'll keep monitoring until all…
Aug 26, 00:18 UTC Resolved - We're considering this incident resolved. Aug 25, 23:38 UTC Monitoring - Systems have recovered and we are seeing new builds created. For a period of about 20min new builds could not be created due to a failure of our internal DNS services. No build requests were lost during this time and we are continuing to process the backlog Aug 25, 23:23 UTC Update - We have ident…
Aug 25, 08:16 UTC Resolved - This incident has been resolved. Aug 25, 07:42 UTC Update - Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects.We recommend restarting agent service to make agents reconnect. Aug 25, 06:41 UTC Monitoring - At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, whic…
Aug 17, 02:43 UTC Resolved - This incident has been resolved. Aug 17, 02:00 UTC Monitoring - We have confirmed that all background workers have caught up as of 1:13am UTC. Hosted Agents dispatch has returned to normal. We'll continue to to monitor the service and are checking remaining retries. Aug 17, 01:21 UTC Update - We’re investigating elevated latency in background worker processing between…
Aug 13, 16:40 UTC Resolved - We think the impact from the issue is over. Aug 13, 16:29 UTC Monitoring - We have identified and fixed the issue. We are monitoring and seeing signs of improvement. Aug 13, 16:14 UTC Identified - We've identified the cause of the issue and are actively mitigating. Aug 13, 15:41 UTC Investigating - We've spotted that something has gone wrong and there's elevated latenc…
Aug 12, 04:08 UTC Resolved - For the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully. At 23:01 11 August 2026 UTC we started routing
Aug 11, 02:04 UTC Resolved - This incident has been resolved. Aug 11, 01:04 UTC Update - The integration between Pipelines and Test Engine has been re-enabled, restoring the Tests tab on build pages. We’re monitoring the recovery to confirm the integration remains stable. Test results continue to be available directly through Test Engine. Aug 11, 00:50 UTC Update - The integration between Pipeline…
Aug 5, 22:00 UTC Resolved - Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.
Aug 3, 05:36 UTC Resolved - We've confirmed all impact from this incident has been resolved. Aug 3, 05:29 UTC Monitoring - The fix has been fully deployed and we are seeing signs of recovery. All events impacted by this issue will be automatically retried. We are continuing to monitor the fix for stability. Aug 3, 05:22 UTC Update - We've identified that the impact is more widespread that we initi…
Jul 28, 20:57 UTC Resolved - Notification latency for the affected shard is now under 5s. Jul 28, 20:38 UTC Identified - We are investigating delays to build and job notifications for a subset of customers.
Jun 18, 09:43 UTC Resolved - We have seen a full recovery of services. Jun 18, 09:17 UTC Monitoring - A fix has been deployed, services are recovering. Jun 18, 08:59 UTC Update - We've identified the issue, we're continuing working on rolling out a fix. Impact is restricted to the builds create API for a subset of tenants. Jun 18, 08:05 UTC Identified - We're seeing increased latency and error rat…
Jun 12, 00:16 UTC Resolved - The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency. Jun 11, 23:51 UTC Update - We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected. Jun 11, 23:49 UTC Monitoring - We've isolated the issue to elevated load on our REST API…
Jun 11, 00:51 UTC Resolved - Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s. Jun 11, 00:32 UTC Investigating - We're observing increased latency and error rates for a subset of our customers on the Agent API.…
May 30, 00:30 UTC Resolved - We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails.This issue has now been resolved.
May 28, 21:18 UTC Resolved - This incident has been resolved. May 28, 20:47 UTC Identified - We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon. May 28, 20:20 UTC Investigating - We are investigating delays to build and job notifications for a subset of customers.
May 26, 10:38 UTC Resolved - We think the impact from the issue is over. May 26, 10:37 UTC Update - We see processing time for all affected services has returned to normal as of 20 minutes ago. May 26, 10:08 UTC Monitoring - We've identified the problem and have completed the remediation steps, we are now monitoring as service resumes. May 26, 09:56 UTC Identified - We're observing increased laten…
May 20, 17:39 UTC Resolved - The incident is resolved May 20, 17:26 UTC Monitoring - We are seeing recovery across affected customers and continue to monitor May 20, 17:06 UTC Identified - We have identified the issue and applied mitigations and are monitoring recoveryWe have determined that only a subset of customers are affected by the notification latency. May 20, 16:40 UTC Investigating - We a…
May 15, 07:35 UTC Resolved - Processing of the backlog is complete. May 15, 06:51 UTC Monitoring - Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour.This has been a recurring issue; an architectural c…
May 13, 15:34 UTC Resolved - Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the situation since then and things have returned to baseline. May 13, 15:14 UTC Investigating -…
May 12, 16:09 UTC Resolved - The fix was successful and the backlog has now been cleared. May 12, 12:59 UTC Monitoring - We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog
May 9, 04:34 UTC Resolved - The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident. May 8, 08:07 UTC Monitoring - Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for…
May 8, 23:42 UTC Resolved - The delayed backlog is now cleared and Test Engine ingestion is operating normally. May 8, 21:10 UTC Investigating - We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience delays while we clear the ingestion backlog. At the current processing rate we expect the…
May 8, 13:06 UTC Resolved - The Test Engine backlog is now cleared and operating normally. May 8, 12:20 UTC Monitoring - We are currently experiencing delayed processing of Test Engine data. We have identified and applied a fix for the issue but are expecting to continue to experience continued processing delays while we clear the ingestion backlog. At the current processing rate we expect the bac…