FsGroup security context causing Portworx sharedv4 volume mount to timeout

Portworx Volumes
1

If a pod using a Portworx sharedv4 PVC has a securityContext set like this

  securityContext:
    fsGroup: 3000

you might get the following errors while mounting the sharedv4 PVC:

Events:
Type Reason Age From Message
---- ------ ---- ---- -------

Warning FailedMount 1m (x3 over 6m) kubelet, ip-172-22-53-20.ca-central-1.compute.internal Unable to mount volumes for pod "wordpress-pod": timeout expired waiting for volumes to attach or mount for pod "wordpress-pod". list of unmounted volumes=[sharedv-pvc]. list of unattached volumes=[config data default-token-296dz]

When the securityContext is set, kubernetes will walk every file in the PVC and change the uid/gid to the value specified as fsGroup prior to starting the Pod. The time taken for changing this ownership depends upon the number of files present in the PVC. With large number of files you might end up with the above error.

You can use the following method, to solve this issue:

From Portworx v2.2 , you can add a label on the sharedv4 volume as allow_others=true. This will make sure that a non root user, will be able to access the sharedv4 volume. To update a label on an existing volume you can run the following command

pxctl volume update --label allow_others=true <vol-id>

Once the volume is updated, stop the applications using the sharedv4 volumes and remove the securityContext field from the pod spec. You can then start the applications back and you should not see any timeouts while mounting the volume.

3 Likes