I am initializing kubernetes on the openstack cloud. However, it encountered the following error when setting up the kvdb.
The node specification is as below:
1 master node, 3 worker nodes labelled with px/metadata-node=true
The status log from one of the Portworx pod:
failed to setup internal kvdb: failed to provision internal kvdb: Init Failure: Please remove /etc/pwx/.private.json to initialize a new node. WARNING: Removing /etc/pwx/.private.json will remove metadata from this node.
Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: Init Failure: Please remove /etc/pwx/.private.json to initialize a new node. WARNING: Removing /etc/pwx/.private.json will remove metadata from this node.
Readiness probe failed: HTTP probe failed with statuscode: 503
I have tried to remove the portworx, generate the new specification and redeploy again but the error still exist.
The k8s worker node has three device attached: /dev/vda, /dev/vdb, /dev/vdc. the /dev/vdb is reserved for kvdb with 60GB storage.
Do anyone encounter the same issue before?
Many thanks in advance.
@lindaorny looks like you are re-trying installations multiple time on the same worker nodes causing about mentioned errors. and also you don’t have to label px/metadata-node=true , if you have only 3 worker nodes in the cluster, by default portworx will pick all the 3 nodes as metadata and storage nodes(which it require min 3 nodes).
all you need to wipe out and clean the old failed installations by following this docs page Uninstall on Kubernetes
or manually you need to delete the daemonset or operator(which ever way you installed). and then login to each worker node and make sure you wipe and clean it manually be below commands.
sudo systemctl stop portworx
sudo docker rm portworx.service -f
sudo rm /etc/systemd/system/portworx.service -f
sudo rm /etc/systemd/system/dcos.target.wants/portworx.service -f
sudo rm /etc/systemd/system/multiuser.target.wants/portworx.service
sudo systemctl daemon-reload
sudo /opt/pwx/bin/pxctl service node-wipe --all
sudo chattr -i /etc/pwx/.private.json
sudo rm -rf /etc/pwx
sudo umount /opt/pwx/oci
sudo rm -rf /opt/pwx
sudo rmmod px -f
then retry to install the portworx and then let me know how it goes.