We are using 4 Portworx nodes in Kubernetes, some time ago one has changed status to Online (StorageDown). What does this mean and how to fix this issue? Kubernetes cluster is in IBM Cloud
Ivan_Puzyrny1@cloudshell:~$ kubectl exec portworx-cwttl -it -n kube-system -- /opt/pwx/bin/pxctl status
Status: PX is operational
License: PX-Enterprise IBM Cloud (expires in 796 days)
Node ID: ed15c634-997d-4beb-b914-b8934ff94825
IP: 10.187.152.137
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 21 TiB 13 TiB Online dal13 us-south
Local Storage Devices: 6 devices
Device Path Media Type Size Last-Scan
0:1 /dev/sdh1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
0:2 /dev/sdc1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
0:3 /dev/sdd1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
0:4 /dev/sde1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
0:5 /dev/sdf1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
0:6 /dev/sdg1 STORAGE_MEDIUM_SSD 3.5 TiB 28 Jan 22 15:40 UTC
* Internal kvdb on this node is sharing this storage device /dev/sdf1 to store its data.
total - 21 TiB
Cache Devices:
* No cache devices
Cluster Summary
Cluster ID: pw-elastic-data
Cluster UUID: 88883914-8aec-48f7-9aa2-206d52ad9800
Scheduler: kubernetes
Nodes: 4 node(s) with storage (3 online)
IP ID SchedulerNodeName Auth StorageNode Used Capacity Status StorageStatus Version Kernel OS
10.187.152.137 ed15c634-997d-4beb-b914-b8934ff94825 10.187.152.137 Disabled Yes 13 TiB 21 TiB Online Up (This node) 2.7.2.0-1408b62 4.15.0-154-generic Ubuntu 18.04.5 LTS
10.187.152.188 9b038895-1fad-4de1-bd46-42f73fa8c85f 10.187.152.188 Disabled Yes 12 TiB 21 TiB Online Up 2.7.2.0-1408b62 4.15.0-154-generic Ubuntu 18.04.5 LTS
10.187.152.236 60066ffa-6805-4ba5-8ba0-a571e0474c5a 10.187.152.236 Disabled Yes 0 B 0 B Online (StorageDown) Full or Offline 2.7.2.0-1408b62 4.15.0-156-generic Ubuntu 18.04.6 LTS
10.187.152.212 57b8ca05-8d39-4971-b06f-f95212b26880 10.187.152.212 Disabled Yes 17 TiB 21 TiB Online Up 2.7.2.0-1408b62 4.15.0-154-generic Ubuntu 18.04.5 LTS
Warnings:
WARNING: Internal Kvdb is not using dedicated drive on nodes [10.187.152.212 10.187.152.137 10.187.152.188]. This configuration is not recommended for production clusters.
Global Storage Pool
Total Used : 42 TiB
Total Capacity : 63 TiB