Incident History for Buildkite

Job dispatch times have returned to normal and the incident is resolved.

Buildkite has applied mitigations for elevated job dispatch times affecting the Job Queue and is monitoring recovery.

Buildkite has applied mitigation for elevated job dispatch times affecting the Job Queue and is now monitoring the issue.

Buildkite has identified the cause of elevated job dispatch times affecting the Job Queue and is applying mitigations.

The incident is resolved; a database failover caused most impact for a single customer, and the cause is being investigated.

Increased latency and error rates affected REST and GraphQL for some customers; platform is stable and root cause is being investigated.

Elevated error rates from a scheduled database upgrade between 23:16 and 23:27 UTC are now resolved.

Hosted Agents job processing delays are resolved and jobs are no longer delayed.

Buildkite is monitoring improvements as the Hosted Agents jobs backlog continues to be processed.

Buildkite identified the issue causing delays with Hosted Agents jobs and applied remediations, with recovery beginning.

Incident resolved; all Buildkite services now distrust affected eMudhra certificates, and updated releases are available for customer infrastructure components.

Buildkite finished updating its hosted services to distrust affected eMudhra certificates; customer infrastructure components should still upgrade to the listed releases.

Buildkite is monitoring the eMudhra certificate compromise and has released updated agent, CLI, and bktec versions that distrust the affected certificates.

Adding capacity resolved the notifications latency; the incident is resolved.

Buildkite identified the notification delay, added capacity, and latency has returned to normal while monitoring for stability.

Buildkite is investigating delays to build and job notifications affecting a subset of customers.

The issue is resolved; provider network maintenance caused unexpected side effects impacting outgoing traffic from Hosted Agents.

Buildkite is investigating reports of degraded checkout performance with GitHub repositories when using Hosted Agents.

The GitHub webhook error rate has returned to normal levels and the incident is resolved.

Mitigation is rolling out for rejected GitHub webhook deliveries and recovery is being seen while monitoring continues.

A small number of incoming GitHub webhook deliveries are being rejected, preventing related builds; cause identified and mitigation being deployed.

Incident resolved; outbound notification latency is back to expected levels, though some customers may have received duplicate notifications.

Outbound notification queues have fully caught up for all impacted customers.

Buildkite identified the cause of delayed notifications and queues are catching up for impacted customers.

Buildkite is investigating delays to build and job notifications affecting a subset of customers.

Webhook ingestion and GraphQL query issues are resolved; services fully recovered after webhook ingestion was affected from 19:09 to 19:32 UTC.

Engineers have addressed the webhook ingestion and GraphQL query issue and are monitoring recovery.

Buildkite is investigating an issue affecting webhook ingestion and execution of GraphQL queries.

The AWS us-east-1 outage affecting Buildkite is resolved with no customer impact experienced.

Buildkite is monitoring a major AWS us-east-1 outage and taking action to prevent impact.

The incident affecting the Agent API has been resolved.

Latency for build creation has improved and Buildkite is monitoring the Agent API for continued recovery.

Buildkite is investigating increased latency and error rates affecting a subset of customers on the Agent API.

A postmortem has been published for the Buildkite service disruption affecting Web, Agent API, REST API, and Job Queue.

Buildkite has fully recovered from an autoscaling feedback loop that impacted Web, Agent API, REST API, and job queue.

Buildkite is monitoring improvements across affected services as they return to normal, while determining root cause.

Buildkite is continuing to investigate elevated error rates across multiple services, affecting Web, Agent API, REST API, and Job Queue.

Incident resolved. Jobs running on Hosted Agents are now being dispatched promptly.

Mitigations deployed and recovery observed across Hosted Agents; monitoring continues.

Buildkite is investigating increased latency and error rates for Hosted Agents.

Buildkite is investigating increased latency and error rates for a subset of customers.

Job dispatch for Hosted Agents is no longer experiencing delays.

We are continuing to process through the backlog of delayed jobs and monitoring.

We have identified the cause of delayed job dispatch for Hosted Agents and are working through the backlog.

Engineers are continuing to investigate delayed job dispatch for Hosted Agents; self-hosted agents are unaffected.