Incident History for Braintrust

A new public write-up was shared for the incident involving elevated errors and latency on the US data plane API.

The issue causing elevated errors and latency on the US data plane API has been resolved.

A fix has been applied and we are monitoring results after elevated errors and latency on the US data plane API.

We are investigating elevated error rates and increased latency on the US centrally-hosted data plane, causing some requests to fail or time out.

We are investigating elevated errors affecting the Braintrust control plane APIs, causing failures for some users accessing projects, prompts, settings, or creating resources.

The issue causing slow or failed data loading in the US-hosted service has been resolved.

We've applied a mitigation for the data-loading issue and are monitoring performance.

We are investigating slow or failed data loading affecting some customers on our US-hosted service, causing pages to load slowly, hang, or show errors.

The issue causing elevated errors in the Braintrust control plane APIs has been resolved.

We have mitigated the issue causing elevated errors in the Braintrust control plane APIs, and are continuing to monitor closely.

We are investigating elevated errors affecting the Braintrust control plane APIs, causing failures for some users.

The upstream APT package mirror has recovered and no longer impacts new Brainstore instances.

The upstream APT package mirror on AWS us-east-1 is degraded, but there is currently no downtime. We are monitoring and looking into alternatives.

The issue has been resolved and service has remained stable following the rollback.

The issue has been resolved and we are monitoring the service to confirm normal operation.

We identified an issue causing 5XXs from a recent Gateway deployment and are rolling it back.