Failed to mount volumes in pod ( HTTP error 404 )

Portworx on Kubernetes
1

Hi everyone,
I have already deploy portworx essentials with 5 storage nodes and it’s running well in my k8s cluster.

The only problem I’m facing right now is I can only use (only mount) PVC if the pods I deployed to cluster are on the storage nodes.
My cluster have total 10 nodes of worker VM. If the pods that used PVC are deployed to other nodes except storage nodes, the following error occurs…

Warning  FailedMount  52m (x386 over 2d18h)     kubelet, k8s-nonprod-worker-9  Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-2v7dh test-volume]: timed out waiting for the condition
  Warning  FailedMount  18m (x1384 over 2d19h)    kubelet, k8s-nonprod-worker-9  Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume default-token-2v7dh]: timed out waiting for the condition
  Warning  FailedMount  2m34s (x1989 over 2d19h)  kubelet, k8s-nonprod-worker-9  MountVolume.SetUp failed for volume "pvc-c16568fd-d6f0-48a9-b257-e6ecd04eadba" : HTTP error 404

I can’t add all worker nodes as storage nodes because px essential only allow 5 storage nodes.
Hope you guys can help me to solve this issue.

Thanks you all.

2

Hello @Wai_Yan_Win_Htain

You need to have Portworx running on the node that will use a Portworx volume (PVC).
Stork should automatically schedule a pod that uses a Portworx volume to one of the 5 nodes with Portworx deployed.
Could you please let me know what version of stork you have deployed?

kubectl -n kube-system describe deployment stork | grep Image

You need to have stork version 2.4.0 or up that has the admission controller to schedule the pod on the Portworx nodes, if you have earlier versions you can upgrade to the 2.4.0 or 2.4.1 (latest at time of this post).

Regards,

Alex.

1 Like
3

My stork version is 2.4.0 and I’m clearly understand what you said about stork.
If we want this Hyper Converged mode, we need to add schedulerName: stork right?
My deployment also include stork and it didn’t schedule Pods that used storage automatically.
All are working very fine and this is the only problem I need to deal with my portworx and kubernetes cluster.
So, if portworx is working fine on your environment, Can I know deployment architecture of your cluster?

4

Does your pod have affinity or anti-affinity rules? I mean do you have anything in deployment that would schedule the pod on the node that PX is not installed?

Basically you need PX installed on a node if you need to run a pod on it that requires a Portworx volume.

Since you are using PX essentials and it only allows for 5 PX nodes, you can run pods that require PVC only on those nodes.

1 Like
5

As I asked question on one portworx webinar, they said PVCs work on all nodes across k8s cluster, not just the storage nodes.
As you said, we can only scheduled pods which used PVC only on storage nodes …
So I’m confused now .

Can someone please explain me ? Which is right or wrong ?

My deployment hasn’t affinity or anti-affinity rules.
As I mentioned above, I have 10 worker nodes and used 5 nodes as PX storage nodes.
If the pods is scheduled on storage nodes, it’s working fine.
But , if the pods are scheduled to non-storage nodes, it return the error as first question.

6

Portworx works on non-storage nodes, but you need to have Portworx deployed on those nodes as well - we call it a storageless node and it will be included in the Portworx cluster and remote mount volumes for pods scheduled on those nodes.
In you case since you are using Portworx Essentials you are limited to 5 nodes only.

But since you have Stork 2.4.0 with the AdmissionController pods that are using Portworx volumes should only be scheduled in the 5 storage nodes, so we need to investigate why pods are scheduled on the other nodes.

First, could you please send me the output of the following command?

kubectl -n kube-system describe svc stork-service