No volume plugin matched

I have set up a 3-node Portworx cluster on K3s v1.21.4 on RHEL 8.4 Nodes. The cluster is operational according to pxctl status and Lighthouse.

But I’m unable to attach volumes to Pods. pxctl volume create works and the volume is shown in Lighthouse. When I try to attach the volume created with pxctl following error occurs:

volumes:
  - name: pvc
    portworxVolume:
      volumeID: "1023447001357235295"
      fsType: "ext4"

Warning FailedMount 4m9s (x37 over 38m) kubelet Unable to attach or mount volumes: unmounted volumes=[pvc], unattached volumes=[kube-api-access-4hr9w pvc]: failed to get Plugin from volumeSpec for volume "pvc" err=no volume plugin matched

And when I try to dynamically provisions a PV, following error occurs:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: px-replicated

Warning ProvisioningFailed 2m43s (x182 over 47m) persistentvolume-controller no volume plugin matched name: kubernetes.io/portworx-volume

CSI is enabled in the storagecluster CRD:

spec:
  featureGates:
    CSI: "true"

Logs of csi-node-driver-registrar:

I0922 07:39:04.669562       1 main.go:113] Version: v2.1.0
I0922 07:39:04.670113       1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0922 07:39:04.670138       1 connection.go:153] Connecting to unix:///csi/csi.sock
W0922 07:39:14.670312       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:39:24.670257       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:39:34.670275       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:39:44.670334       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:39:54.670444       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:04.670337       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:14.670322       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:24.670299       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:34.670310       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:44.670303       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0922 07:40:54.670299       1 connection.go:172] Still connecting to unix:///csi/csi.sock
I0922 07:41:01.192140       1 main.go:144] Calling CSI driver to discover driver name
I0922 07:41:01.192190       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0922 07:41:01.192207       1 connection.go:183] GRPC request: {}
I0922 07:41:01.197095       1 connection.go:185] GRPC response: {"manifest":{"driver":"pxd"},"name":"pxd.portworx.com","vendor_version":"1.4.0"}
I0922 07:41:01.197185       1 connection.go:186] GRPC error: <nil>
I0922 07:41:01.197197       1 main.go:154] CSI driver name: "pxd.portworx.com"
I0922 07:41:01.197234       1 node_register.go:52] Starting Registration Server at: /registration/pxd.portworx.com-reg.sock
I0922 07:41:01.198441       1 node_register.go:61] Registration Server started at: /registration/pxd.portworx.com-reg.sock
I0922 07:41:01.198527       1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: ""
I0922 07:41:01.442746       1 main.go:80] Received GetInfo call: &InfoRequest{}
I0922 07:41:01.489249       1 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

+1 here. Recreated the cluster 3 times. Was trying out for a customer, not a very good first impression.

Exactly the same issue.

  • v1.19.15+k3s2 + v1.22.2+k3s2

Same issue here, no volume plugin matched name during pvc creation
k3s v1.21.2+k3s1