[SOLVED]: no space left on deviceMountVolume.SetUp failed for volume Kubernetes

no space left on device is a common error. But no space left on device MountVolume.SetUp failed is  not so common error faced on Kubernetes pods. And the logs/events given by the pods doesn't provide much info which will direct us to find the root cause of the issue.

Possible fixes for no space left on deviceMountVolume.SetUp failed

Cleanup volumes and Restart the kubelet

If the issue is due to lack of space on our nodes, we can clean up the node space and verify the space availablilty using

 df -h

Once the space is available, restart the kubelet . This need to be done on the node where the faulty pod is scheduled.

systemctl restart kubelet

Verify if the pod template is proper

In most of the cases, the issue will be with the manifest files. Resources in the manifest files namely CPU and memory have to be proper. As there is no separate validation and, CPU and memory keys support string data, value of the fields can be erroneous

    resources:
          memory: "512mi" # incorrect; should be "512Mi"
          cpu: "200m"

Reference

no space left on device, which is unexpected. MountVolume.SetUp failed for volume in kubernetes cluster

Search on LinuxDataHub

Leave a Comment