PX services not running on Pods deployed from portworx operator on openshift 4.2

Portworx on Kubernetes
1

Hello,

I am following the documentation to install portworx on openshift 4.2

the operator got successfully installed
the storage cluster instance was also created & it seems it created all the required objects
[ocpadmin@bastion-vm ~] oc get pods -n openshift-operators | grep portworx portworx-6h7wn 0/1 Running 0 4m21s portworx-9jzxp 0/1 Running 0 4m21s portworx-api-9ltrj 0/1 Running 0 4m26s portworx-api-gsnrq 0/1 Running 0 4m26s portworx-api-sdzgr 0/1 Running 0 4m26s portworx-api-shqdv 0/1 Running 0 4m26s portworx-api-tsjmk 0/1 Running 0 4m26s portworx-api-vjm6s 0/1 Running 0 4m26s portworx-br5xh 0/1 Running 0 4m21s portworx-nwr67 0/1 Running 0 4m21s portworx-operator-b4f9565fc-9hn6z 1/1 Running 0 46h portworx-pvc-controller-76ff89b5b5-6w2ln 1/1 Running 0 4m20s portworx-pvc-controller-76ff89b5b5-g9d4n 1/1 Running 0 4m20s portworx-pvc-controller-76ff89b5b5-mv2c6 1/1 Running 0 4m20s portworx-wdfhk 0/1 Running 0 4m21s portworx-z2r2k 0/1 Running 0 4m21s [ocpadmin@bastion-vm ~]

image
image928×458 24.2 KB

but seems the portworx service are running inside pods
[ocpadmin@bastion-vm ~] oc logs portworx-9jzxp | tail -n 40 time="2020-02-12T08:33:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:34:03Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:34:13Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:34:22: PX-Watchdog: Waiting for px process to start @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:34:22: PX-Watchdog: (pid 315): Begin monitoring time="2020-02-12T08:34:23Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:34:33Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:34:43Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:34:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:35:03Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:35:13Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:35:23Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:35:25: PX-Watchdog: (pid 315): PX REST server died or did not started. return code 7. Timeout 480 time="2020-02-12T08:35:33Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:35:43Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:35:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:35:56: PX-Watchdog: Waiting for px process to start @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:35:56: PX-Watchdog: (pid 315): Begin monitoring time="2020-02-12T08:36:03Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:36:13Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:36:23Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:36:33Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:36:43Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:36:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:36:59: PX-Watchdog: (pid 315): PX REST server died or did not started. return code 7. Timeout 480 time="2020-02-12T08:37:03Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:37:13Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:37:23Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:37:30: PX-Watchdog: Waiting for px process to start @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:37:30: PX-Watchdog: (pid 315): Begin monitoring time="2020-02-12T08:37:33Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:37:43Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:37:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:38:03Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:38:13Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:38:23Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:38:33Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" @ocp01-7wglq-worker-eastus1-sls85 portworx[1705925]: 2020-02-12_ 8:38:33: PX-Watchdog: (pid 315): PX REST server died or did not started. return code 7. Timeout 480 time="2020-02-12T08:38:43Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" time="2020-02-12T08:38:53Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9011/v1/cluster/nodehealth: dial tcp 127.0.0.1:9011: connect: connection refused" [ocpadmin@bastion-vm ~]

even the pxctl status is showing PX services not running inside the container

[ocpadmin@bastion-vm ~]$ oc rsh portworx-9jzxp
sh-4.4# whereis pxctl
pxctl: /opt/pwx/bin/pxctl
sh-4.4# /opt/pwx/bin/pxctl status
PX is not running on this host
sh-4.4#

need help to make this up.

thanks,
Sumit