Nov 4, 22:42 UTC Resolved - This incident is resolved. Nov 4, 20:58 UTC Update - The affected services have recovered, and we will continue to monitor the issue. Nov 4, 20:05 UTC Monitoring - We have seen recovery and are currently monitoring the impacted services. Nov 4, 19:13 UTC Investigating - We are investigating elevated error rates on the Agent API
Nov 4, 06:36 UTC Resolved - We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon. Nov 4, 06:17 UTC Investigating - We're seeing increased latency and error rates on our Agent API for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Nov 4, 00:07 UTC Resolved - We have seen recovery and latency has returned to normal. Nov 3, 22:56 UTC Monitoring - Delays in processing incoming webhooks and creating builds are improving for affected customers. We patched the webhook-processing deployment to increase capacity and processing lag has dropped. We are monitoring recovery as the remaining backlog clears. Nov 3, 22:38 UTC Investigatin…
Nov 3, 20:22 UTC Resolved - We have seen recovery and latency has returned to normal. Nov 3, 19:43 UTC Monitoring - We’ve scaled out the resources that process incoming webhooks and are seeing recovery. We’re now working through the backlog of queued jobs and continuing to monitor Nov 3, 18:54 UTC Update - We are continuing to investigate. At this time, the issue is limited to a small subset of cu…
Nov 3, 20:12 UTC Resolved - API errors have subsided and we're seeing recovery. Nov 3, 19:59 UTC Identified - We're seeing recovery with previously elevated API error rates. Nov 3, 19:42 UTC Update - We are continuing to investigate this issue. Nov 3, 19:40 UTC Investigating - We're investigating elevated error rates with our API.
Oct 29, 21:58 UTC Resolved - We are seeing the job dispatch times are back to normal. Oct 29, 21:41 UTC Update - We’ve applied mitigations and are seeing recovery. We’re continuing to monitor the issue Oct 29, 21:15 UTC Monitoring - We've spotted that something has gone wrong. We've applied the mitigation and currently monitoring the issue, and will provide an update soon. Oct 29, 20:33 UTC Identi…
Oct 28, 23:49 UTC Resolved - We identified that the majority of the impact was for a single customer and was caused by a database failover. We are investigating the cause and we will work with the impacted customer. Oct 28, 23:17 UTC Monitoring - We saw increased latency and error rates in REST and GraphQL for a subset of our customers starting around 22:30 UTC until 22:45. The platform has been s…
Oct 26, 17:07 UTC Resolved - Between 17:07-17:19 UTC, our API endpoints experienced elevated latencies and errors due to issues with a shared service. Our monitoring systems detected these issues, which triggered automated system remediations and resulted in the service self-healing. Although our APIs are once again stable, we are continuing to investigate the cause of this very brief service degr…
Oct 24, 23:40 UTC Resolved - Between 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database customers experienced error rates averages 2.5% during this period.
Oct 22, 17:37 UTC Resolved - Processing of Hosted Agents jobs are no longer delayed. Oct 22, 17:22 UTC Monitoring - We continue to see improvements and are monitoring processing of the Hosted Agents jobs backlog. Oct 22, 17:09 UTC Identified - We've identified the issue and have applied remediations. We are already beginning to see recovery in Hosted Agents job processing times for affected custom…
Oct 21, 22:38 UTC Resolved - All services that power Buildkite have been updated to distrust the affected certificates, and updated releases are available for the components we distribute that run in customer infrastructure:Buildkite agent: v4.7.0 and v3.138.2Buildkite CLI: v3.64.2bktec: v3.3.1elastic-ci-stack-for-aws: v7.8.0agent-stack-k8s: v0.52.0We recommend updating to these versions to preven…
Oct 21, 18:00 UTC Resolved - We received a report that API requests were taking abnormally long to process. Investigation revealed this to be isolated to a single customer shard, with impact between 2026-10-22 01:50 - 08:30 UTC.
Oct 20, 21:05 UTC Resolved - Adding additional capacity has resolved the notifications latency. Oct 20, 20:15 UTC Monitoring - We've identified the issue, and added some additional capacity. Notification latency has returned to normal levels. We are monitoring for stability. Oct 20, 19:57 UTC Investigating - We are investigating delays to build and job notifications for a subset of customers.
Oct 19, 15:45 UTC Resolved - We are getting reports from a few customers that Hosted Agent jobs are delayed in starting. We are investigating.
Oct 6, 23:59 UTC Resolved - The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents. Oct 6, 19:07 UTC Identified - We continue to observe degraded checkout performance. Our engineering teams are working on establishing an alternative network transit path to alleviate networking performance issues. Oct 6, 18:1…
Oct 1, 06:57 UTC Resolved - The error rate has returned to normal levels. Oct 1, 06:33 UTC Monitoring - The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update. Oct 1, 06:10 UTC Identified - Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related b…
Sep 30, 02:32 UTC Resolved - Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved. Sep 30, 01:43 UTC Monitoring - Outbound notification queues have now fully caught up for all impacted customers. Sep 30, 01:19 UTC Identified - We've resolved the underlying error, and outbound notifications queues…
Sep 28, 01:30 UTC Resolved - Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.
Sep 22, 20:50 UTC Resolved - Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC. Sep 22, 20:26 UTC Identified - Our engineers have addressed the issue and we're monitoring recovery. Sep 22, 20:07 UTC Investigating - We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.
Sep 22, 00:32 UTC Resolved - We did not experience any customer impact during this AWS us-east-1 outage. Sep 21, 23:47 UTC Monitoring - We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.
Sep 21, 23:46 UTC Resolved - This incident has been resolved. Sep 21, 23:12 UTC Monitoring - We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement. Sep 21, 22:35 UTC Investigating - We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide…
Sep 10, 01:54 UTC Resolved - We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full pos…
Sep 4, 02:39 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 4, 02:23 UTC Monitoring - We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring. Sep 4, 01:42 UTC Investigating - We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as…
Sep 4, 01:40 UTC Investigating - We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Sep 3, 07:06 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 3, 06:44 UTC Monitoring - We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs. Sep 3, 06:09 UTC Update - Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating…
Sep 2, 15:51 UTC Resolved - Job dispatch for Hosted Agents is no longer experiencing delays. Sep 2, 15:43 UTC Monitoring - We are continuing to process through the backlog of delayed jobs and monitoring. Sep 2, 15:29 UTC Identified - We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs. Sep 2, 15:16 UTC Update - O…
Sep 1, 16:52 UTC Resolved - Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully. Sep 1, 16:38 UTC Monitoring - We are seeing successful email deliveries and are monitoring processing of our outbound email queue. Sep 1, 16:19 UTC Update - Our on-call engineers are working on failing over to an alternative upstream email…
Aug 27, 11:47 UTC Resolved - Processing of uploaded test execution data has caught up and is back to normal. Aug 27, 11:23 UTC Update - Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing. Aug 27, 10:56 UTC Monitoring - Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all d…
Aug 26, 03:13 UTC Resolved - Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now. Aug 26, 02:43 UTC Monitoring - Recovery is underway; we'll keep monitoring until recovery is complete. Aug 26, 02:36 UTC Update - We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery.We'll keep monitoring until all…
Aug 26, 00:18 UTC Resolved - We're considering this incident resolved. Aug 25, 23:38 UTC Monitoring - Systems have recovered and we are seeing new builds created. For a period of about 20min new builds could not be created due to a failure of our internal DNS services. No build requests were lost during this time and we are continuing to process the backlog Aug 25, 23:23 UTC Update - We have ident…