This node is already initialized but could not be found in the cluster map

Hi, I am trying to add a node into the existing PX Essential cluster. But the API pod always failed in initialization and the log of the cluster pod showed that

Mar 03 14:16:06 portworx[21315]: time=“2021-03-03T14:16:06Z” level=error msg=“This node is already initialized but could not be found in the cluster map.”

Mar 03 14:16:06 portworx[21315]: time=“2021-03-03T14:16:06Z” level=error msg=“Failed to initialize node in cluster. This node is already initialized but could not be found in the cluster map.”

Mar 03 14:16:06 portworx[21315]: time=“2021-03-03T14:16:06Z” level=error msg=“Init Failure: This node is already initialized but could not be found in the cluster map. Please remove /etc/pwx/.private.json to initialize a new node.”

I have tried to remove /etc/pwx/.private.json and reinitiate the node even the cluster, but the result was an error as same.

Then I try to wipe the node by

sudo /opt/pwx/bin/pxctl service node-wipe

And another error occurs.

Error: Node is not down and is expected to be down for this operation. Please stop Portworx and try again.

At last, the pod fell into an infinite error loop of

portworx[183199]: time=“2021-03-03T16:35:07Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”

systemd[1]: docker-29edcb4989f62d1b874e974036e81c7a08317c2aae74eca289274dd5cc64d10b.scope: Succeeded.

systemd[1]: docker-29edcb4989f62d1b874e974036e81c7a08317c2aae74eca289274dd5cc64d10b.scope: Consumed 4.024s CPU time.

systemd[1]: var-lib-docker-overlay2-1c5c5279dd438e69a3113e3057c109e52a550e22fc0ebff671094cbb79d66a4a-merged.mount: Succeeded.

systemd[1]: Started libcontainer container 37c3d297f31980a4ebe1c947b663b30e48d8e1bcebca3878698e666043cb9a5c.

time=“2021-03-03T16:35:17Z” level=warning msg=“Could not retrieve PX node status” error=“Node status not OK (STATUS_INIT)\n”

portworx[183199]: time=“2021-03-03T16:35:17Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”

Anyone can help to resolve this issue?

Thanks in advance.

Please stop portworx with systemctl stop portworx and then execute /opt/bin/pxctl sv nw --all
to wipe the node. The PX daemonset should automatically push Portworx installation to the node again.

Regards
Varun

It works, thanks @varunjain

The error messages suggest that the node is already initialized but not found in the cluster map, which prevents the addition of the node. Additionally, when attempting to wipe the node, you encounter another error stating that the node is not down as expected for the operation.

Perhaps these tips will help solve your API:

Remove the existing PX node configuration:
sudo rm /etc/pwx/.private.json

Stop the Portworx service on the node:
sudo systemctl stop portworx

Perform node wipe again:
sudo /opt/pwx/bin/pxctl service node-wipe

Note: Make sure that the Portworx service is stopped before running the node wipe operation.
After wiping the node, start the Portworx service:
sudo systemctl start portworx

Wait for the node to come up and check its status:
sudo /opt/pwx/bin/pxctl status

Once the node is up and running, retry adding the node to the cluster using the appropriate method for your PX Essential cluster setup.

If the issue persists, you may need to check the Portworx documentation or reach out to Portworx support for further assistance. The error loop you are experiencing could be due to a misconfiguration or an issue with the Portworx setup that requires deeper investigation.