I’m running a 3-node cluster on EKS. Got no errors during installation, have my 3 px-cluster pods running and the 3 storagenodes are in Online status. When I run pxctl status on one of the nodes I get that all of them are ‘Online Up’ but one of them that is ‘Online No Storage’:
Status: PX is operational
Telemetry: Healthy
Metering: Disabled or Unhealthy
License: Trial (expires in 31 days)
Node ID: e73c6dd4-3dfa-4238-a95a-8dc8094ea41f
IP: 192.168.17.185
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 200 GiB 14 GiB Online us-east-1f us-east-1
Local Storage Devices: 4 devices
Device Path Media Type Size Last-Scan
0:1 /dev/nvme2n1 STORAGE_MEDIUM_NVME 50 GiB 11 Nov 24 12:00 UTC
0:2 /dev/nvme3n1 STORAGE_MEDIUM_NVME 50 GiB 11 Nov 24 12:00 UTC
0:3 /dev/nvme1n1 STORAGE_MEDIUM_NVME 50 GiB 11 Nov 24 12:00 UTC
0:4 /dev/nvme4n1 STORAGE_MEDIUM_NVME 50 GiB 11 Nov 24 12:00 UTC
total - 200 GiB
Cache Devices:
* No cache devices
Kvdb Device:
Device Path Size
/dev/nvme5n1 32 GiB
* Internal kvdb on this node is using this dedicated kvdb device to store its data.
Cluster Summary
Cluster ID: px-cluster-129c26c4-7e84-426d-8c64-d17e4bfed4f2
Cluster UUID: 60fe4c69-3d03-4526-be6f-648560e3832c
Scheduler: kubernetes
Total Nodes: 2 node(s) with storage (2 online), 1 node(s) without storage (1 online)
IP ID SchedulerNodeName Auth StorageNode Used Capacity Status StorageStatus Version Kernel OS
192.168.17.185 e73c6dd4-3dfa-4238-a95a-8dc8094ea41f ip-192-168-17-185.ec2.internal Disabled Yes 14 GiB 200 GiB Online Up (This node) 3.2.0.0-2ded0fe 6.1.112-122.189.amzn2023.x86_64 Amazon Linux 2023.6.20241010
192.168.33.105 8cbda9ab-d430-4333-bb29-97c134533325 ip-192-168-33-105.ec2.internal Disabled Yes 14 GiB 200 GiB Online Up 3.2.0.0-2ded0fe 6.1.112-122.189.amzn2023.x86_64 Amazon Linux 2023.6.20241010
192.168.62.23 44ac1b72-8bf1-4864-978c-afbea3c6f472 ip-192-168-62-23.ec2.internal Disabled No 0 B 0 B Online No Storage 3.2.0.0-2ded0fe 6.1.112-122.189.amzn2023.x86_64 Amazon Linux 2023.6.20241010
Global Storage Pool
Total Used : 28 GiB
Total Capacity : 400 GiB
Collected at: 2024-11-11 13:06:38 UTC
I tried to create a PVC following the docs but it stays at ‘Pending’ state and when I run `kubectl describe pvc’ I get a few concerning messages:
Warning ProvisioningFailed 14m persistentvolume-controller Error saving claim: Operation cannot be fulfilled on persistentvolumeclaims "px-check-pvc": the object has been modified; please apply your changes to the latest version and try again
Normal ExternalProvisioning 4m44s (x41 over 14m) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'pxd.portworx.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Normal ExternalProvisioning 4m43s (x43 over 14m) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'pxd.portworx.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Normal Provisioning 86s (x12 over 14m) pxd.portworx.com_px-csi-ext-69dcb7566d-7l6sq_a939ecaa-80dd-49df-a710-76f9d534673b External provisioner is provisioning volume for claim "default/px-check-pvc"
Warning ProvisioningFailed 86s (x12 over 14m) pxd.portworx.com_px-csi-ext-69dcb7566d-7l6sq_a939ecaa-80dd-49df-a710-76f9d534673b failed to provision volume with StorageClass "px-csi-db": rpc error: code = Internal desc = Failed to create volume: could not find enough nodes to provision volume
Any clues on how to move forward?