Resolved
Last updated: August 10, 2026 at 09:45 PM UTC
Delayed scaling due to node failure
Affected Components
Inference and Training H100 HardwareInference and Training L40S Hardware
Updates
Resolved
August 10, 2026 at 09:45 PM UTC
All affected models have been scaling correctly for over 30 minutes with no residual prediction queues.
Monitoring
August 10, 2026 at 09:05 PM UTC
Scaling decisions were delayed nearly 1 hour after a controller failed to schedule on a soft-failed node; the node was cordoned and drained, and metrics are flowing again.