Readiness probe failed: HTTP probe failed with statuscode: 503 on GKE

Portworx on Kubernetes
1

Hi Team,

I’m installing portworx on GKE version: 1.12.10-gke.17. I was able to generate a spec file and applied that using kubectl after doing so i noticed some my portworx pods do not get into a ready state. Below is a view when i run kubectl get po.

NAME READY STATUS RESTARTS AGE
portworx-24w79 0/1 Running 3 51m
portworx-p7s6f 0/1 Running 3 51m

portworx-api-9dw68 0/1 Running 0 57m
portworx-api-78cbn 0/1 Running 3 57m

When i run kubectl describe po against the portworx pods i get the error below.

portworx pods -: Readiness probe failed: HTTP probe failed with statuscode: 503

portworx api pods -: Readiness probe failed: Get http://127.0.0.1:9001/status: dial tcp 127.0.0.1:9001: connect: connection refused

I already added a firewall rule to the GKE vpc to allow tcp port 9001-9022 on both servers. Would appreciate any assistance towards resolving this.

Thanks.

2

Hi @Lawal_Adekunle_Oluse - Can you share the logs from portworx pod ? Also share the spec file which was applied for the installation.

Regards
Sanjay

3

Hi Sanjay,

Thanks a bunch. Kindly see the link below.

https://drive.google.com/drive/folders/1d24pZohv3LF1HYKA4z5Hj07ndUem43zc?usp=sharing

4

Hi Lawal,
It seems, the external kvdb service is not accessible by Portworx service.

@gke-excellentbillionare-default-pool-0c583e99-vzdp portworx[15740]: time="2019-12-05T22:49:33Z" level=warning msg="Test connection from KVDB failed." err="client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint." fn=kv-utils.TestKvdbConnection id=
@gke-excellentbillionare-default-pool-0c583e99-vzdp portworx[15740]: time="2019-12-05T22:49:34Z" level=warning msg="Test connection from KVDB failed." err="client: response is invalid json. The endpoint is probably not valid etcd cluster endpoint." fn=kv-utils.TestKvdbConnection id=
@gke-excellentbillionare-default-pool-0c583e99-vzdp portworx[15740]: time="2019-12-05T22:49:35Z" level=error msg="Failed to test connection with KVDB, tried 2" func=InitAndBoot package=boot
@gke-excellentbillionare-default-pool-0c583e99-vzdp portworx[15740]: time="2019-12-05T22:49:35Z" level=error msg="Could not init boot manager" error="**Failed to test connection with KVDB, tried 2**"
@gke-excellentbillionare-default-pool-0c583e99-vzdp portworx[15740]: PXPROCS: px daemon exited with code: 1

We see you mentioned external KVDB as Key Value Store: etcd:http://10.3.243.150:19999
Is this a standalone ETCD server ? How have you implemented this Etcd server ?

Also, please note that we currently support only Ubuntu OS in the GKE service. Make sure that you customise the GKE nodes to have Ubuntu as OS.

Thanks & Regards
Varun

5

Hi Varunjain,

Thanks a bunch, i fixed the etcd now everything works as it should.

$kubectl get pods -o wide -n kube-system -l name=portworx

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
portworx-4kf9b 1/1 Running 0 45m 10.128.15.225 gke-excellentbillionare-default-pool-e83513b3-jjj7
portworx-mqn2j 1/1 Running 0 45m 10.128.15.224 gke-excellentbillionare-default-pool-a9e777bf-ndjn

Regards