How to reduce / move / increase portworx volume replicas from one node to another or specific node?

Portworx Volumes
1

How to reduce-remove / move / increase volume replicas from one node to another or specific node ?

You may want to move the replicas, if you are planning to decommission a node from the Portworx cluster, replacing the node, save some space on a specific node etc.

Some Facts:-
A volume's replica is the exact copy of the Portworx volume, its data, its type and even the volume-id is same across the nodes in the cluster.
A volume's snapshot also maintains the exact number of replicas on the same nodes where the volume's replicas are present.

Lets say for example you have REPL-2 volume, mysql_vol

Step:1 : Decide and identify the node from which volume replica is to be removed / reduced from.

Command: pxctl volume inspect mysql_vol

Volume  :  435917115280854955
        Name                     :  mysql_vol
        Size                     :  3.0 GiB
        Format                   :  ext4
        HA                       :  2
        IO Priority              :  HIGH
        Creation time            :  Apr 19 10:06:57 UTC 2020
        Shared                   :  no
        Status                   :  up
        State                    :  Attached: 18bb0ae4-ad09-4f5f-9b79-216f467312f8 (70.0.71.13)
        Device Path              :  /dev/pxd/pxd435917115280854955
        Reads                    :  0
        Reads MS                 :  0
        Bytes Read               :  0
        Writes                   :  0
        Writes MS                :  0
        Bytes Written            :  0
        IOs in progress          :  0
        Bytes used               :  860 KiB
        Replica sets on nodes:
                Set 0
                  ***Node           : 70.0.71.13 (Pool 4683722c-e0c3-4055-94ad-ede955efef2a )***
                  ***Node           : 70.0.70.243 (Pool 586eaad3-2425-4d07-9086-f43b3541ae8b )***
        Replication Status       :  Up

Lets say the replica from Node with IP-Address 70.0.70.243 is to be removed.

Command:

1. Obtain the Node-ID:
pxctl status | grep 70.0.70.243 | awk '{print $2}'

2. Remove / Reduce the replica from the node:

pxctl volume ha-update --repl=1 --node f9c0c8cc-0ca6-4f8d-8717-4414267a44c5 mysql_vol

Step: 2: Move / Increase the replica to specific node:

Now that you have removed the replica, you can increase the replica to a specific node.

Command:

1. Decide and obtain the Node-ID:
pxctl status | grep 70.0.70.241 | awk '{print $2}'

2. Increase the replica back to 2 on another specific node.
pxctl volume ha-update --repl=2 --node b87895de-fe88-4f1b-8e39-018720e018cf mysql_vol

Step:3: Verify the volume of node to see if the replicas have been moved.

Commands:

  • You should see the updated replica sets.
    pxctl volume inspect mysql_vol

  • This will list all the volumes present on the specific node
    pxctl volume list -v --node-id f9c0c8cc-0ca6-4f8d-8717-4414267a44c5

Updated Replica Sets:-

    Replica sets on nodes:
            Set 0
              Node           : 70.0.70.241 (Pool 99c62c24-7199-46d9-a158-0ab0ba02fd5c )
              Node           : 70.0.71.13 (Pool 4683722c-e0c3-4055-94ad-ede955efef2a )
    Replication Status       :  Up

The node 70.0.70.243 has been evacuated successfully as there are no volumes on this node any more.

pxctl v l -v --node-id f9c0c8cc-0ca6-4f8d-8717-4414267a44c5
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY STATUS SNAP-ENABLED
[root@worker-node-1 ~]#

2 Likes