Portworx kvdb Disaster Recovery

1

I have 4 worker nodes kubernetes cluster with Portworx storage. Every each of them contains three devices (/dev/vda, /dev/vdb and /dev/vdc). The /dev/vdb is 20GB which contains the kvdb data while the /dev/vdc is 120GB with the kubernetes pod volume.

The /dev/vdb on one of the node encounters bad superblock issue and thus needed to be replaced. The /dev/vdc looks fine. Could I have any clues to recover the kvdb volume to join back to the volume cluster? I saw there are backup done under /var/lib/osd/kvdb_backup but I am not so sure how to recover the /dev/vdb from the backup.

I tried to rename the dump to pwx_kvdb_disaster_recovery_golden.dump and restart Portworx service but failed to recover the cluster.

Thanks in advance.

2

Following up: I have tried using the command

pxctl service kvdb restore --kvdb_dump_file  <path of the kvdb backup>

with the -f flags still not working. The output is

Failed connect to kvdb instance ([]): implementation not supported

Furthermore,
The below message occur even if I created a totally new device. I fail to read the vkdb content.
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/vdb, missing codepage or helper program, or other error.