Readiness probe error: Get "http://127.0.0.1:17001/status": dial tcp 127.0.0.1:17001: connect: connection refused

Portworx Install
,
1

Hi,

We are experiencing issues deploying Portworx Enterprise on both OpenShift Container Platform (OCP) version 4.14 and Kubernetes. Could someone please assist us with finding a solution?

Portworx-api pod is in CrashLoopBackOff state with readiness probe error (Get “http://127.0.0.1:17001/status”: dial tcp 127.0.0.1:17001: connect: connection refused).
The portworx-api and pxcsi-ext pods are in CrashLoopBackOff state.
The StorageCluster and StorageNode is stuck in Initializing phase.

Below are the pod logs:

**# oc get po -A | grep -i px**
openshift-operators                                px-csi-ext-97fff4cf8-9x4vg                                        1/4     CrashLoopBackOff   1365 (75s ago)     42h
openshift-operators                                px-csi-ext-97fff4cf8-v79mj                                        1/4     CrashLoopBackOff   1368 (2m28s ago)   42h
openshift-operators                                px-csi-ext-97fff4cf8-wh2vn                                        1/4     CrashLoopBackOff   1365 (2m5s ago)    42h
openshift-operators                                px-plugin-85d78c474b-gb757                                        1/1     Running            0                  42h
openshift-operators                                px-plugin-85d78c474b-tqfbv                                        1/1     Running            0                  42h
openshift-operators                                px-plugin-proxy-69987b8b6c-lm9x5                                  1/1     Running            0                  42h

**# oc get po -A | grep -i port**
kube-system                                        portworx-proxy-4245c                                              0/1     Running            0                 42h
kube-system                                        portworx-proxy-8jxps                                              0/1     Running            0                 42h
kube-system                                        portworx-proxy-c2t67                                              0/1     Running            0                 42h
kube-system                                        portworx-proxy-g2jqb                                              0/1     Running            0                 42h

openshift-operators                                portworx-6rtqk                                                    0/1     Running            164 (46s ago)     42h
openshift-operators                                portworx-7cjjd                                                    0/1     Running            163 (15m ago)     42h
openshift-operators                                portworx-api-7zkt4                                                0/2     **CrashLoopBackOff   456 (2m36s ago)   42h**
**openshift-operators                                portworx-api-mrkh2                                                0/2     CrashLoopBackOff   456 (3m3s ago)    42h**
**openshift-operators                                portworx-api-twhfr                                                1/2     Running            456 (5m18s ago)   42h**
**openshift-operators                                portworx-api-wcxmf                                                0/2     CrashLoopBackOff   456 (2m4s ago)    42h**
openshift-operators                                portworx-f4854                                                    0/1     Running            164 (16s ago)     42h
openshift-operators                                portworx-operator-864bb59b95-hkrn9                                1/1     Running            0                 42h
openshift-operators                                portworx-p4dwx                                                    0/1     Running            164 (16s ago)     42h

**#oc describe pod -n openshift-operators  portworx-api-twhfr** 

  Warning  BackOff     6m6s (x10594 over 42h)  kubelet  Back-off restarting failed container csi-node-driver-registrar in pod portworx-api-twhfr_openshift-operators(97f3b8eb-4e8a-4629-81a0-be6256144d1c)
  Warning  ProbeError  57s (x26768 over 42h)   kubelet  Readiness probe error: Get "http://127.0.0.1:17001/status": dial tcp 127.0.0.1:17001: connect: connection refused
body:

**# oc logs -n openshift-operators   px-csi-ext-97fff4cf8-9x4vg csi-external-provisioner**

W0724 06:46:21.621306       1 feature_gate.go:241] Setting GA feature gate Topology=true. It will be removed in a future release.
I0724 06:46:21.621419       1 feature_gate.go:249] feature gates: &{map[Topology:true]}
I0724 06:46:21.621445       1 csi-provisioner.go:154] Version: v3.6.1
I0724 06:46:21.621454       1 csi-provisioner.go:177] Building kube configs for running in cluster...
W0724 06:46:31.623192       1 connection.go:183] Still connecting to unix:///csi/csi.sock
W0724 06:46:41.623213       1 connection.go:183] Still connecting to unix:///csi/csi.sock
W0724 06:46:51.623295       1 connection.go:183] Still connecting to unix:///csi/csi.sock
E0724 06:46:51.623338       1 csi-provisioner.go:215] context deadline exceeded
2

@Shreyashirwadkar csi-ext and portworx-api pods will go to healthy state only once px cluster pods (oc get pods -l name=portworx) are moved to Ready state.

To see why px cluster pods are in unhealthy state. You can either check those pod logs or portworx journal logs using below command.

journalctl -lefu portworx*

Just to make sure that pre-requisites are met for your cluster

3

Thanks @ [tomjoseph]
We are not able bring the pods in ready state due to connection refused to port 17001:

#oc describe pod -n openshift-operators portworx-api-twhfr**

Warning BackOff 6m6s (x10594 over 42h) kubelet Back-off restarting failed container csi-node-driver-registrar in pod portworx-api-twhfr_openshift-operators(97f3b8eb-4e8a-4629-81a0-be6256144d1c)
Warning ProbeError 57s (x26768 over 42h) kubelet Readiness probe error: Get “http://127.0.0.1:17001/status”: dial tcp 127.0.0.1:17001: connect: connection refused

Are there any additional steps required to open these ports?

4

@Shreyashirwadkar Will you be able to share the portworx journal logs to check the exact error from any nodes where px-cluster is not getting ready?

jouranlctl -lu portworx*