Symbolic link file ownership is root always in portworx storage

0

I have a linux machine where portworx volume is mounted with nfs4.
The exports file contents on the machine where portworx volume created is /var/lib/osd/mounts/engsharevol *(rw,no_root_squash,sync,no_subtree_check,fsid=1)
Mount is done using the command mount -t nfs4 :/var/lib/osd/mounts/engsharevol /opt/
In /opt, I created a file
touch /opt/test1.txt
ln -s -f /opt/test1.txt /opt/test2.txt
chown -h user1:group1 /opt/test2.txt
I see /opt/test1.txt is owned by user1 whereas test2.txt is owned by root user.
If I run the same commands outside portworx storage ie outside /opt directory, then both test1.txt and test2.txt is owned by user1.
Do I need to do any changes in the mount command or any setting should be change in the linux files to preserve the ownership as it does in the storage outside of portworx.
Many thanks

Hi @deepayr,
This is actually the behavior of NFS-Exports. I did an exact test to yours but using NFS-Export and you can see in the output I got the same behavior you did.

[root@node-1-2 /]# touch /mnt/nfs/var/nfsshare/test1.txt

[root@node-1-2 /]# ln -s -f /mnt/nfs/var/nfsshare/test1.txt /mnt/nfs/var/nfsshare/test2.txt

[root@node-1-2 /]# chown -h user1:group1 /mnt/nfs/var/nfsshare/test2.txt

[root@node-1-2 /]# ls -l /mnt/nfs/var/nfsshare
total 0
-rw-r--r-- 1 root  root    0 Nov 13 19:58 test1.txt
lrwxrwxrwx 1 user1 group1 31 Nov 13 20:14 test2.txt -> /mnt/nfs/var/nfsshare/test1.txt

Hi @deepayr,
One of my colleagues also wondering if test2.txt does not have updated ownership and test1.txt is changed? are you sure of that? I tried the same, and for me only test2.txt has updated ownership and not the target( test1.txt), which is correct as the command was to only change the ownership of symlink.

On a different note if you want to change the permission of both symlink and target, you’ll need to use command twice. once with -h and one without.
Here is the output of my NFS-Export test after using the command twice as mentioned:

[root@node-1-2 /]# chown user1:group1 /mnt/nfs/var/nfsshare/test2.txt

[root@node-1-2 /]# ls -l /mnt/nfs/var/nfsshare
total 0
-rw-r--r-- 1 user1 group1  0 Nov 13 19:58 test1.txt
lrwxrwxrwx 1 user1 group1 31 Nov 13 20:14 test2.txt -> /mnt/nfs/var/nfsshare/test1.txt

I tried chown command on both the files. It is changing only for test1.txt but not for test2.txt. I see opposite behaviour in your case. I tried both chown and chown -h to change the symbolic link file for ownership change, but its not working. Is it something related to the underlying driver used in portworx storage or is it specific to portworx version.
I tried mounting Filestorage with same attributes and I could change the ownership for both the files, but same not happening with portworx.

What is the portworx version your using ?

pxctl --version
pxctl version 2.1.5.0-3b73452 (OCI)

Hi @deepayr,
I just tried the same thing you had using Portworx volume and PX version you have. I was able to change the ownership by running the chown -h and chown in sequence. Are you sure you running the command twice with and without -h?

[root@node-1-2 ~]# mount -t nfs4 192.168.101.101:/var/lib/osd/mounts/px_vol1 /opt/

[root@node-1-2 ~]# touch /opt/test1.txt

root@node-1-2 ~]# ln -s -f /opt/test1.txt /opt/test2.txt

[root@node-1-2 ~]# chown -h user1:group1 /opt/test2.txt

[root@node-1-2 ~]# ls -al /opt/
total 4
drwxr-xr-x   2 root  root   4096 Nov 14 13:13 .
dr-xr-xr-x. 17 root  root    224 Jan 28  2019 ..
-rw-r--r--   1 root  root      0 Nov 14 13:13 test1.txt
lrwxrwxrwx   1 user1 group1   14 Nov 14 13:13 test2.txt -> /opt/test1.txt

[root@node-1-2 ~]# chown  user1:group1 /opt/test2.txt

[root@node-1-2 ~]# ls -al /opt/
total 4
drwxr-xr-x   2 root  root   4096 Nov 14 13:13 .
dr-xr-xr-x. 17 root  root    224 Jan 28  2019 ..
-rw-r--r--   1 user1 group1    0 Nov 14 13:13 test1.txt
lrwxrwxrwx   1 user1 group1   14 Nov 14 13:13 test2.txt -> /opt/test1.txt

I tried running twice. But still the same

[root@ieeug4-s-ises opt]# touch test.txt
[root@ieeug4-s-ises opt]# ln -s -f test.txt test2.txt
[root@ieeug4-s-ises opt]# chown -h dsadm:dstage test.txt
[root@ieeug4-s-ises opt]# ls -la
total 20
drwxr-xr-x. 4 root root 4096 Nov 14 04:02 .
dr-xr-xr-x. 20 root root 4096 Nov 13 08:34 …
drwxr-xr-x. 15 root root 4096 Nov 12 08:09 InformationServer
drwxr-xr-x. 3 root root 4096 Nov 12 09:38 myhome
-rw-r–r--. 1 root root 2474 Nov 12 09:38 myhome.tar
-rw-r–r--. 1 dsadm dstage 0 Nov 14 04:02 test.txt
-rw-r–r--. 1 1008 1007 0 Nov 13 23:07 test1.txt
lrwxrwxrwx. 1 root root 8 Nov 14 04:02 test2.txt -> test.txt
[root@ieeug4-s-ises opt]# chown -h dsadm:dstage test2.txt
[root@ieeug4-s-ises opt]# ls -la
total 20
drwxr-xr-x. 4 root root 4096 Nov 14 04:02 .
dr-xr-xr-x. 20 root root 4096 Nov 13 08:34 …
drwxr-xr-x. 15 root root 4096 Nov 12 08:09 InformationServer
drwxr-xr-x. 3 root root 4096 Nov 12 09:38 myhome
-rw-r–r--. 1 root root 2474 Nov 12 09:38 myhome.tar
-rw-r–r--. 1 dsadm dstage 0 Nov 14 04:02 test.txt
-rw-r–r--. 1 1008 1007 0 Nov 13 23:07 test1.txt
lrwxrwxrwx. 1 root root 8 Nov 14 04:02 test2.txt -> test.txt
[root@ieeug4-s-ises opt]# chown dsadm:dstage test2.txt
[root@ieeug4-s-ises opt]# ls -la
total 20
drwxr-xr-x. 4 root root 4096 Nov 14 04:02 .
dr-xr-xr-x. 20 root root 4096 Nov 13 08:34 …
drwxr-xr-x. 15 root root 4096 Nov 12 08:09 InformationServer
drwxr-xr-x. 3 root root 4096 Nov 12 09:38 myhome
-rw-r–r--. 1 root root 2474 Nov 12 09:38 myhome.tar
-rw-r–r--. 1 dsadm dstage 0 Nov 14 04:02 test.txt
-rw-r–r--. 1 1008 1007 0 Nov 13 23:07 test1.txt
lrwxrwxrwx. 1 root root 8 Nov 14 04:02 test2.txt -> test.txt
[root@ieeug4-s-ises opt]# chown root:root test.txt
[root@ieeug4-s-ises opt]# chown -h dsadm:dstage test2.txt
[root@ieeug4-s-ises opt]# ls -la
total 20
drwxr-xr-x. 4 root root 4096 Nov 14 04:02 .
dr-xr-xr-x. 20 root root 4096 Nov 13 08:34 …
drwxr-xr-x. 15 root root 4096 Nov 12 08:09 InformationServer
drwxr-xr-x. 3 root root 4096 Nov 12 09:38 myhome
-rw-r–r--. 1 root root 2474 Nov 12 09:38 myhome.tar
-rw-r–r--. 1 dsadm dstage 0 Nov 14 04:02 test.txt
-rw-r–r--. 1 1008 1007 0 Nov 13 23:07 test1.txt
lrwxrwxrwx. 1 root root 8 Nov 14 04:02 test2.txt -> test.txt

Is there anything I am missing while creating portworx volume or anything wrong I mentioned in /etc/exports file or something with kubernetes version or is it something related to system level settings ?
Is there a way I can show it to you by having a call or other means ?

Interesting let me reach out to you directly

I will join the slack now. I will be able to do screen share in 30 min.

As per our discussion, I upgraded the kernel version to 3.10.0-1062. I still see the same issue.

Hi,

Can you share additional below information:

OS - ??
Kubernetes/version - ?? How is it setup (Cloud/onpremises/what method used to deploy)
Portworx Version - You already mentioned this
Kernel version - ??
Selinux - ??
Share your /etc/exports content
Command use to create portworx volume (StorageClass/PVC creation used if any)
Mount command used to mount the volume on host

OS - RHEL V7.6
Kubernetes/version - Master HA - How is it setup (Cloud/onpremises/what method used to deploy)
Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T21:04:45Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T20:56:12Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Onpremises
Portworx Version - You already mentioned this
Kernel version - 3.10.0-1062.4.1.el7.x86_64
Selinux -
[root@hadr02-s-ises-worker1 ~]# sestatus
SELinux status: disabled
Share your /etc/exports content
/var/lib/osd/mounts/engsharevol *(rw,no_root_squash,sync,no_subtree_check,fsid=1)
Command use to create portworx volume (StorageClass/PVC creation used if any)
pxctl volume create finalvol --shared --size=100
pxctl host attach finalvol
mkdir -p /var/lib/osd/finaldir
pxctl host mount --path /var/lib/osd/finaldir finalvol
cd /var/lib/osd/finaldir
[root@ip-10-0-14-9 finaldir]# touch test1.txt
[root@ip-10-0-14-9 finaldir]# ln -s -f test1.txt test2.txt
[root@ip-10-0-14-9 finaldir]# chown -h dsadm:dstage test2.txt
[root@ip-10-0-14-9 finaldir]# ls -la
total 4
drwxr-xr-x. 2 root root 4096 Nov 15 06:58 .
drwxr-xr-x. 14 root root 200 Nov 15 06:56 …
-rw-r–r--. 1 dsadm dstage 0 Nov 15 06:58 test1.txt
lrwxrwxrwx. 1 root root 9 Nov 15 06:58 test2.txt -> test1.txt

Mount command used to mount the volume on host
,mount -t nfs4 :/var/lib/osd/mounts/finaldir /opt/

OS - RHEL V7.6
Kubernetes/version - Master HA - How is it setup (Cloud/onpremises/what method used to deploy)
Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T21:04:45Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T20:56:12Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Onpremises
Attached spec file
Portworx Version - You already mentioned this
Kernel version - 3.10.0-1062.4.1.el7.x86_64
Selinux -
[root@hadr02-s-ises-worker1 ~]# sestatus
SELinux status: disabled
Share your /etc/exports content
/var/lib/osd/mounts/engsharevol *(rw,no_root_squash,sync,no_subtree_check,fsid=1)
Command use to create portworx volume (StorageClass/PVC creation used if any)
pxctl volume create finalvol --shared --size=100
pxctl host attach finalvol
mkdir -p /var/lib/osd/finaldir
pxctl host mount --path /var/lib/osd/finaldir finalvol
cd /var/lib/osd/finaldir
[root@ip-10-0-14-9 finaldir]# touch test1.txt
[root@ip-10-0-14-9 finaldir]# ln -s -f test1.txt test2.txt
[root@ip-10-0-14-9 finaldir]# chown -h dsadm:dstage test2.txt
[root@ip-10-0-14-9 finaldir]# ls -la
total 4
drwxr-xr-x. 2 root root 4096 Nov 15 06:58 .
drwxr-xr-x. 14 root root 200 Nov 15 06:56 …
-rw-r–r--. 1 dsadm dstage 0 Nov 15 06:58 test1.txt
lrwxrwxrwx. 1 root root 9 Nov 15 06:58 test2.txt -> test1.txt

Mount command used to mount the volume on host
,mount -t nfs4 :/var/lib/osd/mounts/finaldir /opt/

OS - RHEL V7.6
Kubernetes/version - Master HA - How is it setup (Cloud/onpremises/what method used to deploy)
Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T21:04:45Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T20:56:12Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Onpremises
Attached spec file
Portworx Version - You already mentioned this
Kernel version - 3.10.0-1062.4.1.el7.x86_64
Selinux -
[root@hadr02-s-ises-worker1 ~]# sestatus
SELinux status: disabled
Share your /etc/exports content
/var/lib/osd/mounts/engsharevol *(rw,no_root_squash,sync,no_subtree_check,fsid=1)
Command use to create portworx volume (StorageClass/PVC creation used if any)
pxctl volume create finalvol --shared --size=100
pxctl host attach finalvol
mkdir -p /var/lib/osd/finaldir
pxctl host mount --path /var/lib/osd/finaldir finalvol
Mount command used to mount the volume on host
,mount -t nfs4 :/var/lib/osd/mounts/finaldir /opt/

I am not sure whether my reply is visible to you or not. I had replied to your queries

I don’t see any reply to my query.

OS - RHEL V7.6
Kubernetes/version - Master HA - How is it setup (Cloud/onpremises/what method used to deploy)
Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T21:04:45Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.0”, GitCommit:“ddf47ac13c1a9483ea035a79cd7c10005ff21a6d”, GitTreeState:“clean”, BuildDate:“2018-12-03T20:56:12Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}
Onpremises
Portworx Version - You already mentioned this
Kernel version - 3.10.0-1062.4.1.el7.x86_64
Selinux -
[root@hadr02-s-ises-worker1 ~]# sestatus
SELinux status: disabled
Share your /etc/exports content
/var/lib/osd/mounts/engsharevol *(rw,no_root_squash,sync,no_subtree_check,fsid=1)
Command use to create portworx volume (StorageClass/PVC creation used if any)
pxctl volume create finalvol --shared --size=100
pxctl host attach finalvol
mkdir -p /var/lib/osd/finaldir
pxctl host mount --path /var/lib/osd/finaldir finalvol
cd /var/lib/osd/finaldir
Mount command used to mount the volume on host
,mount -t nfs4 :/var/lib/osd/mounts/finaldir /opt/

Just an update for others, use the portworx version 2.2 and create a sharedv4 volume and not shared volume. Below are the steps you need to follow and make it work.

 root@sourcek8sw1:/# pxctl v c --sharedv4 sv4vol
 Volume (with Sharedv4 support) successfully created: 766007947747112982
 root@sourcek8sw1:/# pxctl host attach sv4vol
 Volume successfully attached at: /dev/pxd/pxd766007947747112982
root@sourcek8sw1:/# mkdir /var/lib/osd/mounts/sharedv4; pxctl host mount sv4vol --path /var/lib/osd/mounts/sharedv4
Volume sv4vol successfully mounted at /var/lib/osd/mounts/sharedv4
Automatically populates /etc/exports file with correct information:-                                                                                           root@sourcek8sw1:/# cat /etc/exports
/var/lib/osd/pxns/766007947747112982 0.0.0.0/0.0.0.0(fsid=123e4567-e89b-12d3-00a1-682b8497b816,rw,no_root_squash,no_subtree_check,crossmnt)
root@sourcek8sw1:/# 
root@sourcek8sw1:/# cd /var/lib/osd/mounts/sharedv4
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# ls -lrth
total 0
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# touch file1
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# ln -s -f file1 file2
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# ls -lrth
total 0
-rw-r--r-- 1 root root 0 Nov 15 10:06 file1
lrwxrwxrwx 1 root root 5 Nov 15 10:07 file2 -> file1
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# chown -h sharetest:sharetest file2
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# 
root@sourcek8sw1:/var/lib/osd/mounts/sharedv4# ls -lrth
total 0
-rw-r--r-- 1 root  root  0 Nov 15 10:06 file1
lrwxrwxrwx 1 sharetest sharetest 5 Nov 15 10:07 file2 -> file1

Now the above test was carried on HOST A, If you want to mount on HOST B, use the below commands

pxctl host attach sv4vol
mkdir /var/lib/osd/mounts/sharedv4; pxctl host mount sv4vol --path /var/lib/osd/mounts/sharedv4

If you want to use the same on portworx 2.1.x release then you need to follow the Pre-requisites section