Re-commissioned nodes fail to authenticate against the cluster

Portworx on Kubernetes
1

For some maintenance work we had to decommission and remove some nodes from our Kubernetes cluster. I went the route to remove them using Kubespray and then kubeadm reset, followed by

pxctl cluster delete [node-id]

Now, the nodes were re-added, and I re-installed the Storage Cluster manifest doing:

kubectl apply -f "https://install.portworx.com/?operator=true&mc=false&kbver=1.25.6&ns=portworx&oem=esse&user=REDACTED&b=true&s=%2Fdev%2Fmapper%2Fvgroot-lvpx0&kd=%2Fdev%2Fmapper%2Fvgroot-lvpx1&c=REDACTED&stork=true&csi=true&mon=true&tel=false&st=k8s&e=PURE_DEFAULT_ENABLE_FB_NFS_SNAPSHOT%3Dtrue%2CPURE_ISCSI_LOGIN_TIMEOUT%3D20&promop=true"

However, the “px-cluster” pods keep restarting, and when I do a “sudo pxctl status” on one of the re-added nodes, I get:

NODE    NodeStartFailure                                                        ALARM           1       Aug 30 14:18:36 UTC 2023        Aug 30 14:18:36 UTC 2023        Failed to start Portworx: Failed to initialize k8s bootstrap: Failed to create configmap px-bootstrap-pxclusterREDACTED: Unauthorized
NODE    InternalKvdbSetupFailed                                                 ALARM           1       Aug 30 14:18:36 UTC 2023        Aug 30 14:18:36 UTC 2023        Failed to bootstrap internal kvdb: Failed to initialize k8s bootstrap: Failed to create configmap px-bootstrap-pxclusterREDACTED: Unauthorized

What am I missing?

2

This post may help, it contains steps to clean a node’s configuration.