How to recover a portworx installation that has gone sideways?

Installed Portworx on an IBM IKS cluster that initially didn’t have any storage nodes. Added the storage nodes, but portworx is not settling to a working state:

Here is the current state:
λ kubectl get pods -n kube-system | grep portworx
portworx-22wfb 0/1 Running 962 10d
portworx-5dvc5 0/1 Running 962 10d
portworx-69f64 0/1 Running 66 16h
portworx-874ts 0/1 Running 75 19h
portworx-8d9n2 0/1 Running 962 10d
portworx-api-4mwn7 0/1 Running 933 10d
portworx-api-5xj47 0/1 Running 76 19h
portworx-api-7dfwz 0/1 Running 933 10d
portworx-api-7hd6j 0/1 Running 64 16h
portworx-api-924fb 0/1 Running 964 10d
portworx-api-9h5ds 0/1 Running 933 10d
portworx-api-bkhmj 0/1 Running 933 10d
portworx-api-ffsb6 0/1 Running 933 10d
portworx-api-jxnr9 0/1 Running 933 10d
portworx-api-ppzzr 0/1 Running 78 19h
portworx-api-rzqm4 0/1 Running 933 10d
portworx-api-tcvrl 0/1 Running 964 10d
portworx-c6hh7 0/1 Running 955 10d
portworx-km55z 0/1 Running 963 10d
portworx-lvpd9 0/1 Running 962 10d
portworx-q9h4v 0/1 Running 963 10d
portworx-qh6h2 0/1 Running 944 10d
portworx-sss2w 0/1 Running 76 19h
portworx-wqssp 0/1 Running 933 10d

λ kubectl exec portworx-22wfb -n kube-system – /opt/pwx/bin/pxctl status
PX is not running on host: Could not reach ‘HealthMonitor’
command terminated with exit code 1

Was looking for some documentation that states how to restart the portworx service, but searching on the docs site did not yield any guidance.

Was this Portworx installation ever working at any one point, or is this a fresh install? It sounds like the former but you said it’s gone sideways, so want to make sure there isn’t existing data that needs to be preserved… And if that’s the case, then the easiest thing would be to simply use the uninstallation procedure and re-install fresh, rather than trying to resuscitate the existing installation.

You can use the procedure found on this page for performing that action, and once raw (unformatted) storage is made available on the nodes, you can try re-installing.

BTW, as for the question about restarting portworx service, this would need to be done by logging in (via ssh) to each individual worker node, using systemctl restart, and may be more involved than the proposed approach above (depending on the number of nodes one needs to iterate through, by my count 12 in your example output).

Please also note however that Portworx Essentials (the section you are posting in) only supports up to 5 nodes (along with other limitations). You may want to consider portworx enterprise instead for this size cluster, which comes with a 30 day free trial.

sorry, I got sent this way by the portworx support portal for some reason, but this is Portworx Enterprise.

The portworx was a fresh install on an MZ cluster that started off with nodes without local storage. We have added storage nodes and now have a set of resources that checks all the prereqs, but I don’t have a single working pod to run pxctl on. This is a secure cluster so can’t ssh into nodes.