Incident History for Axiom

Metrics Ingest issues on US-east-1 have been resolved and all ingest issues addressed.

Further performance issues have been identified affecting Metrics Ingest on US-east-1.

Metrics query performance since 3pm UTC is fully restored and the incident is resolved.

Fixes have addressed the degraded metrics ingest on US-east-1, and monitoring continues to confirm stability.

The elevated query errors incident is resolved; monitors now process in real time with no visible delays.

Queues have cleared and the system is being monitored for stability after elevated query errors.

Query errors are nominal, but second-order effects are causing delays with monitors and degraded US alerting.

The problem causing elevated errors from US queries has been identified and a fix is being deployed.

The incident affecting metrics dataset queries in us-east-1 is resolved and everything appears stable now.

A fix was deployed and error rates are dropping as monitoring continues for metrics dataset queries in us-east-1.

Queries on metrics datasets in us-east-1 are failing at an elevated rate; root cause identified and a fix is underway.

The metrics query service issue in the US region has been resolved.

Metrics query service in the US region is experiencing a partial outage; the issue has been identified and a fix is being deployed.

Metrics ingest and query issues are resolved; the query service failure to discover recent metric data is fixed.

Metrics ingest has been fixed and is back to normal.

Metrics ingest is experiencing a partial outage due to an infrastructure change; the problem has been identified and work is ongoing.

Query performance in the US region fully restored since 19:15 UTC; errors and missed monitor runs occurred between 18:05 and 19:15 UTC.

Rolled back the change behind US query errors; error rates normal since 19:15 UTC and we are monitoring.

Elevated error rates and slower responses for some queries in the US region are under investigation; ingest is unaffected and no data has been lost.

The EU monitor execution backlog has fully drained and processing latency is back within normal range.

A fix has been deployed for the EU monitor execution backlog, which is decreasing; monitoring continues until it clears.

Access to the Axiom Console via GitHub and other authentication methods is available, with a possible one-time email verification step.

Monitoring started; GitHub and other authentication methods are available, with a possible one-time email verification step.

Identified degraded authentication for Console UI; GitHub login unavailable, other methods work.

Identified degraded authentication for Console UI; GitHub login unavailable, other methods work.