Hi,
We just deployed portworx on our production k8s cluster which uses pure flash array as the cloud storage. We are using the operator to deploy portworx. We had initially labeled 3 nodes to be kvdb nodes (say node 116, 117, 118) with label “px/metadata-node=true”. However, for some reason portworx itself was not coming up on those nodes. So, we decided to disable portowx on those nodes, and chose 3 different nodes for kvdb (say node 101,102, 103). We labeled these nodes with “px/metadata-node=true” and removed the labels previously given to 116, 117, 118 and also disabled portworx on those nodes with “px/enabled=False”.
The problem is that node 101 started running a kvdb pod along with the portworx pod which was already running. However, nodes 102 and 103 are not running any kvdb pods, but portworx pod on 103 is running fine (102 is having some trouble). We looked at the operator logs, it looks like the operator hasnt recognized that node 102, 103 also are supposed to be kvdb nodes. We have tried restarting the operator pod too, but no luck. Does the operator support adding kvdb nodes to an existing portworx cluster? Whats the correct way of doing this? Is giving the label sufficient?
PX version: 2.9.1.1
px-operator version: 1.7.0
Thank you!