We are considering using internal KVDB for Portworx on openshift cluster (cloud deployment).
I went through the doc here which has pretty good info - Internal KVDB
Would like to discuss few things in detail
Storage for KVDB
We currently attach block vols to nodes for PX storage. Can KVDB use these attached block vols for persistent storage ? Doc says it is not recommended to have KVDb consume storage from PX pool. Does this mean we shouldn’t have worker node be both a metadata node and storage node ?
HA for KVDB
Thru labelling, if we manage to have workers that have KVDB disks to be spread across 3 different Availability zones (AZs), can we expect KVDB to be resilient for zonal worker failures (volumes have more than 1 replication factor) ?
- Worker upgrades/ unexpected failures
Docs indicate backups are stored on worker node. In our env, workers are replaced with new worker nodes during upgrades. Assuming the worst case scenario that all workers KVDB uses are down, is there a path to recovery ?
- Worst case scenario: Where we are unable to recover PX cluster/KVDB
In this case, if we manage to bring a new installation of PX, would there be data loss ? Assuming we have scheduled snapshots to cloud object storage, can we recover the volumes from cloudsnaps ?