PX cluster unable to ready - Readiness probe failed: HTTP probe failed with statuscode: 503

Portworx on Kubernetes
1

Trying to Install Portworx and configure it for ASG and volumes on Amazon EKS worker nodes. Encountered issues and errors when getting PX cluster to be ready.

kubectl get pods -o wide -n kube-system -l name=portworx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
portworx-4bv2f 0/1 Running 0 7m48s 10.0.1.127 ip-10-0-1-127.us-west-2.compute.internal
portworx-5c6vp 0/1 Running 0 7m48s 10.0.2.98 ip-10-0-2-98.us-west-2.compute.internal
portworx-bflh8 0/1 Running 0 7m48s 10.0.3.143 ip-10-0-3-143.us-west-2.compute.internal

the PX cluster and API unable to get ready
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system autopilot-564ffc474d-7nzk4 1/1 Running 0 8m
kube-system aws-node-2lxdv 1/1 Running 0 9m53s
kube-system aws-node-ghbfc 1/1 Running 0 9m51s
kube-system aws-node-wjhph 1/1 Running 0 9m51s
kube-system coredns-5946c5d67c-2454q 1/1 Running 0 55m
kube-system coredns-5946c5d67c-dbsns 1/1 Running 0 55m
kube-system etcd-operator-55978c4587-cl6qh 1/1 Running 0 8m53s
kube-system kube-proxy-nsz7m 1/1 Running 0 9m53s
kube-system kube-proxy-wv2m7 1/1 Running 0 9m51s
kube-system kube-proxy-x78st 1/1 Running 0 9m51s
kube-system portworx-4bv2f 0/1 Running 0 8m
kube-system portworx-5c6vp 0/1 Running 0 8m
kube-system portworx-api-cnl28 0/1 Running 0 8m
kube-system portworx-api-j5tjn 0/1 Running 0 8m
kube-system portworx-api-v6h2m 0/1 Running 0 8m
kube-system portworx-bflh8 0/1 Running 0 8m
kube-system portworx-etcd-cluster-d8v4ljz9db 1/1 Running 0 8m32s
kube-system portworx-etcd-cluster-llzqd559v2 1/1 Running 0 8m
kube-system portworx-etcd-cluster-npv4kp9xxt 1/1 Running 0 8m16s
kube-system portworx-pvc-controller-57778475bf-fw4zv 1/1 Running 0 8m
kube-system portworx-pvc-controller-57778475bf-kslmk 1/1 Running 0 8m
kube-system portworx-pvc-controller-57778475bf-sl8gl 1/1 Running 0 8m
kube-system stork-c4c4d6d75-mccvp 1/1 Running 0 8m
kube-system stork-c4c4d6d75-qtgvn 1/1 Running 0 8m
kube-system stork-c4c4d6d75-zwtrw 1/1 Running 0 8m
kube-system stork-scheduler-6c6f566fb4-kz6z5 1/1 Running 0 8m
kube-system stork-scheduler-6c6f566fb4-ssvtv 1/1 Running 0 8m
kube-system stork-scheduler-6c6f566fb4-x8rjz 1/1 Running 0 8m

printscreen4
printscreen41366×768 57.8 KB

kubectl describe pods -l name=portworx -n kube-system
Warning Unhealthy 3m25s (x20 over 12m) kubelet, ip-10-0-3-143.us-west-2.compute.internal Readiness probe failed: HTTP probe failed with statuscode: 503
Warning Unhealthy 3m23s (x3 over 4m23s) kubelet, ip-10-0-3-143.us-west-2.compute.internal Liveness probe failed: Get http://127.0.0.1:9001/status: dial tcp 127.0.0.1:9001: connect: connection refused
Normal Killing 3m23s kubelet, ip-10-0-3-143.us-west-2.compute.internal Container portworx failed liveness probe, will be restarted

2

From the logs/snap attached looks it is not able to reach ETCD endpoint. Do you have an external ETCD server ? What is your current Cluster size ?

3

Hi Sanjay,

Etcd.database.coreos.com/v1beta2 cluster version. Cluster Size 3.
Here my etcd-cluster & etcd-operator .yaml configuration:-

etcd-cluster
image

4

etcd-operator

image
image864×540 24.5 KB

5

Update for others:

When your planning to use External ETCD then those ETCD servers needs to be outside of your K8s cluster, If you don’t have a very large K8s cluster size then you don’t have to use External ETCD with Portworx, you can just use the Portworx Internal KVDB and you should be good.

After using the Internal Portworx KVDB, Portworx Cluster is up and Running. (Regenerated the new spec file using build in KVDB option)