Pods remaining in a Pending state in ACP-Not Prod Cluster

Incident Report for ACP

Resolved

This incident has been resolved.
Posted Apr 24, 2025 - 11:05 BST

Update

We are continuing to monitor for any further issues.
Posted Apr 22, 2025 - 11:46 BST

Monitoring

Following the roll out of the fix, we can see that the number of Pending Pods has reduced to 0, and those pods that were stuck in a ContainerCreating state has also significantly reduced.

We will continue to monitor the cluster, but we're happy to report that this issue now appears to be resolved.
Posted Apr 22, 2025 - 11:45 BST

Identified

We have identified the issue in the ACP Not-Prod cluster that has caused the issue relating to Pods being stuck in a Pending state this morning. We have also rolled out a fix, and we're currently monitoring the cluster.

The fix appears to have solved the source of the problem, and we can confirm that the number of pods stuck with a 'Container Creating' status is starting to reduce, but it may take some time for the cluster to work through the number of pods.
Posted Apr 22, 2025 - 11:00 BST

Investigating

We're currently aware of an issue relating to the ACP Not-Prod cluster where pods are stuck in a Pending state and are showing an error stating there are not enough nodes available to schedule the pods.

We're currently investigating and we will update you when we identify what the problem is.
Posted Apr 22, 2025 - 09:30 BST
This incident affected: Kubernetes (not-prod).