ReadWriteMany pvc creation failing with rpc error: code = Internal desc in GKE

Portworx Install
, ,
1

We are trying to check if the portworx storage class can provision PVC with ReadWriteMany access mode. We are using Google cloud with GKE for the kubernetes engine.

Currently I am able to create PVC with ReadWriteOnce, however when I tried ReadWriteMany the pvc goes into Pending state, and when do a describe on the pvc resource I see error.

failed to provision volume with StorageClass "px-csi-db": rpc error: code = Internal desc = Failed to create volume: exit status 1

Debugging information

PVC resource yaml (taken from Volume Lifecycle Basics - Portworx Documentation)

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: px-check-pvc
spec:
  storageClassName: px-csi-db
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

PVC goes to pending

➜ ✗ k get pvc
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
px-mysql-pvc   Pending                                      px-csi-db      4s

error checked from k describe pvc px-mysql-pvc

kubernetes events

Events:
  Type     Reason                Age               From                                                                               Message
  ----     ------                ----              ----                                                                               -------
  Warning  ProvisioningFailed    16s               persistentvolume-controller                                                        Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "px-mysql-pvc": the object has been modified; please apply your changes to the latest version and try again
  Warning  ProvisioningFailed    16s               persistentvolume-controller                                                        Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "px-mysql-pvc": the object has been modified; please apply your changes to the latest version and try again
  Normal   ExternalProvisioning  14s               persistentvolume-controller                                                        Waiting for a volume to be created either by the external provisioner 'pxd.portworx.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   ExternalProvisioning  10s               persistentvolume-controller                                                        Waiting for a volume to be created either by the external provisioner 'pxd.portworx.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   ExternalProvisioning  3s (x2 over 16s)  persistentvolume-controller                                                        Waiting for a volume to be created either by the external provisioner 'pxd.portworx.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          1s (x5 over 16s)  pxd.portworx.com_px-csi-ext-569d65f8cb-7dsm4_15400128-e1fe-4435-9db4-f1136383f9df  External provisioner is provisioning volume for claim "portworx/px-mysql-pvc"
  Warning  ProvisioningFailed    1s (x5 over 16s)  pxd.portworx.com_px-csi-ext-569d65f8cb-7dsm4_15400128-e1fe-4435-9db4-f1136383f9df  failed to provision volume with StorageClass "px-csi-db": rpc error: code = Internal desc = Failed to create volume: exit status 1

I am not sure if the above given link is having a bug or something as it is using the storageclass as px-csi-db. However I have also tried to create the shared4 storageclass and used that as well exact same error as above (Create sharedv4 PVCs - Portworx Documentation).

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: px-sharedv4-pvc
spec:
  storageClassName: portworx-rwx-rep2
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

Cluster and environment details

  • GKE kubernetes version: 1.28.14-gke.1376000 (this is compatible and supported version as per - Prerequisites for Google Kubernetes Engine - Portworx Documentation. I have ensured the node configurations, and port opened for both TCP and UDP from 100-13000 so there shouldn’t miss in the port provisioning.
  • All the pods for portworx are in Running nothing in error or anything like that. Also I can provision ReadWriteOnce volumes without issues.
  • Portworx version: 3.2

I am using portworx essential and generated the spec file and commands from Portworx central.

Following the documentation - Install Portworx on Google Kubernetes Engine - Portworx Documentation

2

This is now already resolved. Issue was with the rpcbind not properly running inside the GKE nodes. Need to ssh ingo the GKE nodes terminal and essentially kill it once and start again (along with that also start the mountd service too otherwise the pvc won’t mount to the pod even though it got provisioned).

ps -ef | grep rpcbind
sudo kill <pid-from-above>
sudo systemctl restart rpcbind
sudo systemctl restart nfs-mountd
rpcinfo -p # now lists a bunch of ports for mountd and nfs and some more services

Good luck :tada: