Resolved

Last updated: August 10, 2026 at 09:45 PM UTC

Delayed scaling due to node failure

Affected Components

Inference and Training H100 HardwareInference and Training L40S Hardware

Updates

Resolved

August 10, 2026 at 09:45 PM UTC

All affected models have been scaling correctly for over 30 minutes with no residual prediction queues.

Monitoring

August 10, 2026 at 09:05 PM UTC

Scaling decisions were delayed nearly 1 hour after a controller failed to schedule on a soft-failed node; the node was cordoned and drained, and metrics are flowing again.

    Incident: Delayed scaling due to node failure