PVC stuck in pending because of i/o timeout during PVC provisioning

Portworx on Kubernetes
1

Hi,

we are currently trying to get Portworx Essentials running on a managed Kubernetes cluster from DataLine. The PX cluster status is operational, showing 4 nodes with storage. However, when we try to create a PVC, it stays in “Pending” state with the following error message:

Failed to provision volume with StorageClass “px-secure-sc”: Get http://10.96.43.173:9001/v1/osd-volumes/versions: dial tcp 10.96.43.173:9001: i/o timeout

If I understand the architecture correctly, this means that kube-controller-manager cannot reach the Portworx service, and therefore cannot trigger the volume creation.

Can you suggest any further steps do debug and hopefully resolve this? We do not have access to the controlplane as this is a managed Kubernetes setup.

Best
Dominik

2

Can you share the output of kubectl get pods -n kube-system and share the spec file which was used for installation.

3

I’m working together with Dominik on this.

Output of kubectl get pods -n kube-system:

portworx-7c8mv                                            1/1     Running   0          27h
portworx-7qwl5                                            1/1     Running   0          27h
portworx-api-65xs6                                        1/1     Running   0          27h
portworx-api-7mvw7                                        1/1     Running   0          27h
portworx-api-hjtpb                                        1/1     Running   0          27h
portworx-api-v4z5s                                        1/1     Running   0          27h
portworx-k6lx6                                            1/1     Running   0          27h
portworx-msgqn                                            1/1     Running   0          27h

spec:
https://install.portworx.com/2.5?mc=false&kbver=1.17.8&oem=esse&user=5dad7baa-935f-11ea-97e6-f6e09c7a4e5e&b=true&f=true&j=auto&c=px-cluster-a224c99b-86b5-4bc0-9d59-3624abe8a213&stork=false&st=k8s

4

Can you apply this yaml and check:

5

After applying this were you able to provision the PVC ?

6

Hi,

yes that helped bringing up the the pvc.

Thank you!

1 Like
7

Hi, we are having the same error

Failed to provision volume with StorageClass "austral-px": Get
  "http://10.0.27.21:9001/v1/osd-volumes/versions": dial tcp 10.0.27.21:9001:
  i/o timeout

We apply the yaml bu we get this error and still can no bound the PVC

serviceaccount/portworx-pvc-controller-account created
clusterrole.rbac.authorization.k8s.io/portworx-pvc-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/portworx-pvc-controller-role-binding created
Warning: resource deployments/portworx-pvc-controller is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
deployment.apps/portworx-pvc-controller configured