Status: In progressMaintenance has startedAffected components H100 Hardware (Under maintenance) L40S Hardware (Under maintenance) A100 Hardware (Under maintenance) HTTP API (Under maintenance) CPU Hardware (Under maintenance) Official Models (Under maintenance) Billing (Under maintenance) Support Tickets (Under maintenance) Home Page (Under maintenance) Playground (Under maintenance) Streaming API…
Status: ResolvedWe've been below capacity for A100 hardware for the past 30 minutes and all queues have been drained. Thanks for your patience 💖Affected components A100 Hardware (Operational)
Status: ResolvedAll affected models have been scaling correctly for more than 30 minutes at this point, and we see no residual prediction queues. Thank you for your patience!Affected components H100 Hardware (Operational) L40S Hardware (Operational)
Status: ResolvedWe have been monitoring for a while and services seem to be back. Thank you for your patience
Status: ResolvedWe have restarted the affected component and service should be back to normal. Thank you for your patience
Status: ResolvedWe're looking stable now, and with caching back on. Thanks again for your patience!Affected components H100 Hardware (Operational) L40S Hardware (Operational) A100 Hardware (Operational)
Status: ResolvedThis issue is resolvedAffected components H100 Hardware (Operational) Official Models (Operational)
Status: ResolvedWe have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared
Status: ResolvedH100 capacity has returned to normal levelsAffected components H100 Hardware (Operational)
Status: ResolvedWe are back under max capacity for H100 hardware. Thank you for your patience!Affected components H100 Hardware (Operational)
Status: ResolvedThe H100 capacity is back in good health. Thanks for your patience!Affected components H100 Hardware (Operational)
Status: ResolvedSystem is back to operating normallyAffected components H100 Hardware (Operational) L40S Hardware (Operational)
Status: ResolvedThis issue has been resolved and queue times are back to normalAffected components Official Models (Operational)
Status: ResolvedMessage flows are healthy.Affected components Playground (Operational) H100 Hardware (Operational) Streaming API (Operational) HTTP API (Operational) L40S Hardware (Operational) A100 Hardware (Operational) T4 Hardware (Operational) CPU Hardware (Operational) Home Page (Operational)
Status: ResolvedH100 hardware contention has resolved. Thank you for your patience!Affected components H100 Hardware (Operational)
Status: ResolvedThere is no more contention for H100 hardware. Thank you for your patience!Affected components H100 Hardware (Operational)
Status: ResolvedAll A100 capacity is back. Thanks for your patience!Affected components A100 Hardware (Operational)
Status: ResolvedThe maintenance is complete and all systems are reporting healthy. Thank you for your patience!Affected components A100 Hardware (Operational)