Volume delete failed: mounted

#1

Hello,
I’m trying to delete previously created volumes, but without success. Please help!

Portworx dev version 2.3.2.0-c6ebdd7, 3 nodes, docker

I’ve tried to delete volume, and no places where it’s mounted. But I’m still getting error on volume delete, and the volume is indicated as attached on “pxctl volume list” and in “pxctl volume inspect”.

I run the following commands on the volume:

pxctl detach XXXXX
pxctl force-detach XXXXXX

and it doesn’t seem to do anything.

Also, I verified that there are no mount points mounted on the nodes and inside containers for this specific volume.

How can I remove the volume?

Thank you

#2

Hello Ygilbaum,

Could you share the output for pxctl volume inspect ?
You can find processes holding the volume on the node where it is attached find /proc -name mounts | xargs grep
Also is this an encrypted volume ?

Thanks and Regards
Varun

#3

Hi varunjain
Thank you for replying.
Here is the output of pxctl volume inspect clapsite_vl

pxctl volume inspect clapsite_vl
Volume  :  1033153168369041284
        Name                     :  clapsite_vl
        Size                     :  200 GiB
        Format                   :  ext4
        HA                       :  2
        IO Priority              :  HIGH
        Creation time            :  Dec 30 12:52:16 UTC 2019
        Shared                   :  yes
        Status                   :  up
        State                    :  Attached: 4f5b9272-7f4b-408e-94ad-1c2fd298de9e (10.3.14.183)
        Device Path              :  /dev/pxd/pxd1033153168369041284
        Labels                   :  name=clapsite_vl
        Reads                    :  18433
        Reads MS                 :  119100
        Bytes Read               :  75739136
        Writes                   :  581524
        Writes MS                :  30994728
        Bytes Written            :  216681918464
        IOs in progress          :  0
        Bytes used               :  130 GiB
        Replica sets on nodes:
                Set 0
                  Node           : 10.3.14.182 (Pool 109a6317-f19f-4b7c-9d64-5a86d8d75068 )
                  Node           : 10.3.14.183 (Pool 8da516cf-ec14-4d2a-81ef-0d89fd9ed649 )
        Replication Status       :  Up

And a partial output of find /proc -name mounts | xargs grep 1033153168369041284 from the node 10.3.14.183

...
/proc/8457/task/8457/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/8457/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/8818/task/8818/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/8818/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/8953/task/8953/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/8953/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11438/task/11438/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11438/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11468/task/11468/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11468/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11924/task/11924/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/11924/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
/proc/21854/task/21854/mounts:/dev/pxd/pxd1033153168369041284 /var/lib/osd/pxns/1033153168369041284 ext4 rw,relatime,discard,data=ordered 0 0
...
#4

Will you be able to join us on a zoom call ?

#5

Sure, thank you, Please schedule

#7

As discussed, we were able to see the volume mounted on pwx03 node. We restarted the px-dev container on the node and detach the volume from another. Please note, we do not recommend use of pxctl host attach and mount commands for production uses except local backup or import of data on the host. Please feel free to reach us out again if you have any queries. For more information, you can refer to docs.portworx.com

#8

Thank you very much for your help

#9

Actually, the original problem was created when I mounted a volume on host machine with pxctl host mount command, and unmount it with linux umount command. I propose to consider to make linux “umount” legitimate way to unmount volume on host, or to fail linux umount.

#10

You can use pxctl host unmount --path to unmount the host mounted volume. And then pxctl host detach command to detach it gracefully.