Quickly test PX-Enterprise 'cloudsnap' and 'objectstore' features to take volume backups

PX-Central
1

I recently encountered a situation where I accidentally upgraded one of my px-dev test clusters to the trial px-enterprise license. I wanted to preserve some test volumes, rather than re-create them on a fresh px-dev (re)installation. I used this opportunity to learn about 2 features of Portworx that I hadn’t previously looked into: Portworx native S3-API-compatible objectstore, and Portworx ‘cloudsnap’ feature for taking volume backups to any AWS S3 API compatible object store (or Azure blob storage, Google cloud storage, etc). Enterprise license holders for Portworx includes the capability to save/backup volumes using the point-in-time-consistent ‘cloudsnap’ feature, but even the px-dev version allows one to restore a cloudsnap to a cluster. The Portworx team really had some foresight here when including the cloudsnap restoration capability in the development license.

To use cloudsnap, you’ll need to first set up a secrets provider to store credentials that Portworx can use to access an objectstore. There are numerous secure key management providers, but in my quick test I just used the etcd backing my PX-Enterprise cluster to (insecurely) hold the credentials. In /etc/pwx/config.json on each node, you can add a secret section and Portworx will see your change without needing to restart Portworx:

"secret": {
    "cluster_secret_key": "spaceballs-12345",
    "secret_type": "kvdb"
},

Each cloudsnap run appears to originate from the node that has a volume presently mounted, but that might have just been my observation. On one of your Portworx nodes, run the commands:

pxctl secrets kvdb login
pxctl secrets set-cluster-key spaceballs-12345

And on each other node in the cluster, just ‘pxctl secrets kvdb login’. Not sure if this is required, I’ll update this post if I hear otherwise. It will warn you that this probably isn’t what you want, it’s not super secure, and won’t persist across terminal sessions/reboots.

Next, you’ll need to configure an objectstore. If you use AWS S3, this is pretty simple. If you want to use your Portworx (px-enterprise or px-dev) cluster’s native S3-compatible object store, or in my case a separate (px-dev) cluster to run the objectstore (so that I could later blow-away and re-install the current enterprise cluster back to px-dev and then restore the volumes to it) you can easily do that as I will now show:

On the cluster you want running the objectstore, first create a volume that will be used to hold the data:

pxctl volume create --size 100 objectstorevol

Next, create the objectstore, passing it the volume above:

pxctl objectstore create -v objectstorevol

You’ll see some output that will include the access key and secret key. You can ‘pxctl objectstore status’ to verify the objectstore service is running within Portworx.

Back on the Portworx cluster you want perform a cloudsnap from, do:

pxctl secrets kvdb login

/opt/pwx/bin/pxctl credentials create --provider=s3 --s3-disable-ssl --s3-region=us-east-1 --s3-access-key=7189PVKDYLYZFL3C2YI5 --s3-secret-key=HkyLd0OKqPKRAnsTasDzuGGXbKbz+K0zwNyk --s3-endpoint=1.2.3.4:9010 spaceballs

Using the information that was output when you configured the objectstore. You can verify that your ‘spaceballs’ objectstore credentials work with the objectstore by doing ‘pxctl cred v spaceballs’.

Finally, you are ready to take the cloudsnap of a volume. If you have more than one set of credentials stored using the ‘pxctl credentials create’ command above, you’ll need to specify WHICH credentials to use in the following command by appending ‘–cred-id spaceballs’ to the cloudsnap command, otherwise it will default to the only credential/objectstore that you’ve added:

pxctl cloudsnap backup somevolume

You can run ‘pxctl cloudsnap status’ and ‘pxctl cloudsnap history’ to see how it went.

When ready to restore your cloudsnap from an objectstore, it couldn’t be easier:

pxctl secrets kvdb login
pxctl cloudsnap restore -v somenewvolume -s 2cd58bc5-3e59-404c-8ae0-7895ece3d07a/989322175776036709-1009388294432296362

You’ll find the -s (snapshot ID) by performing a ‘pxctl cloudsnap list’ or catalog.

If I’ve missed anything, please let me know and I’ll update this article.

1 Like