Encrypted Volume PVC pending

Portworx Volumes
1

Hi Portworx team,

I’m trying to use Encrypted Volume feature in Portworx enterprise with OpenShift Container Platform v4.6.16. Anyone can help to take a look at this problem? or any advice ?

Follow the steps in Encrypting PVCs using StorageClass with Kubernetes Secrets. After step 3, the PVC is staying in Pending status and there is warning message reported by oc describe pvc/secure-pvc command.

# oc describe pvc/secure-pvc
Name:          secure-pvc
Namespace:     portworx
StorageClass:  px-secure-sc
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/portworx-volume
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason              Age                From                         Message
  ----     ------              ----               ----                         -------
  Warning  ProvisioningFailed  30s (x6 over 96s)  persistentvolume-controller  Failed to provision volume with StorageClass "px-secure-sc": rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-ade4bd86-bdfe-4ef0-9c86-e61a47386890) create failed error: Unable to get secret [px-vol-encryption] due to: Failed to get secret [px-vol-encryption]. Err: secrets "px-vol-encryption" not found

but the secret px-vol-encryption can be found in portworx namespace.

# oc get secret/px-vol-encryption -n portworx -o yaml
apiVersion: v1
data:
  cluster-wide-secret-key: c1VwM3JjcjN0
kind: Secret
metadata:
  creationTimestamp: "2021-04-02T07:38:20Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:cluster-wide-secret-key: {}
      f:type: {}
    manager: oc
    operation: Update
    time: "2021-04-02T07:38:20Z"
  name: px-vol-encryption
  namespace: portworx
  resourceVersion: "40461"
  selfLink: /api/v1/namespaces/portworx/secrets/px-vol-encryption
  uid: 2fabf9b3-ebba-453d-8a15-62c1d1f02af9
type: Opaque

Here is StorageClass yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: px-secure-sc
provisioner: kubernetes.io/portworx-volume
parameters:
  secure: "true"
  shared: "true"
  repl: "3"

and PVC yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: secure-pvc
spec:
  storageClassName: px-secure-sc
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
2

Did you set up the cluster wide or per volume secrete ? as per docs page Kubernetes Secrets

can you also post your portworx daemonset install spec arguments?

3

I setup a cluster wide secret not per volume. the command is

kubectl -n portworx create secret generic px-vol-encryption --from-literal=cluster-wide-secret-key=sUp3r$3cr3t

and

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')  

kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl secrets set-cluster-key \
  --secret cluster-wide-secret-key

I use operator to install, this is arguments

  containers:
  - args:
    - -c
    - px-cluster-9cf1b028-291f-4168-a2c7-addbb17ac3b2
    - -x
    - kubernetes
    - -b
    - -s
    - /dev/sdc
    - -j
    - auto
    - -kvdb_dev
    - /dev/sdb
    - -secret_type
    - k8s
    - -r
    - "17001"
4

thanks . it think your OCP cluster missing this permissions as per our docs page. 1. Prepare your platform

Mainly it might be missing this portworx-pvc-controller-account permission as per above posted error message. apply below commands and recreate your pods again with encrypted pvc. and keep me post the out come.

Add the Portworx service accounts to the privileged security context
Portworx runs as a privileged container. Hence you need to add the Portworx service accounts to the privileged security context.

oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:px-account
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:portworx-pvc-controller-account
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:px-lh-account
oc adm policy add-scc-to-user anyuid system:serviceaccount:kube-system:px-lh-account
oc adm policy add-scc-to-user anyuid system:serviceaccount:default:default
oc adm policy add-scc-to-user privileged system:serviceaccount:kube-system:px-csi-account
5

Hi sensre,

I have ran those command in my environment, but I still run into same problem. Have you any other advice?

BTW: I don’t have seen those service account in my env.

# oc get sa px-account -n kube-system
Error from server (NotFound): serviceaccounts "px-account" not found
# oc get sa portworx-pvc-controller-account -n kube-system
Error from server (NotFound): serviceaccounts "portworx-pvc-controller-account" not found
# oc get sa px-lh-account -n kube-system
Error from server (NotFound): serviceaccounts "px-lh-account" not found
# oc get sa default -n default
NAME      SECRETS   AGE
default   2         19h
# oc get sa px-csi-account -n kube-system
Error from server (NotFound): serviceaccounts "px-csi-account" not found
# oc get sa -n kube-system | egrep -E 'px|portworx'
portworx                             2         3h9m
portworx-operator                    2         3h10m
6

Assuming the page in Encrypting PVCs using annotations with Kubernetes Secrets has misleading information in Step 1: Create cluster wide secret key in which the cluster wide secret key will be created in portworx namespace. It is key point that why I received the error message after deploy PVC.

I tried to create the cluster wide secret key in kube-system namespace in which StorageCluster object was also created. After deploy new PVC, the PVC can be created successfully.

@sensre would you help to confirm which one is correct step for create cluster wide secret key? If my assuming is correct, I think the topic can be closed.

7

By default, Portworx has permissions to read secrets under portworx namespace. If your secrets are stored in some other namespace, then you need to give Portworx permissions to read the secrets. To grant Portworx permission to read vol-secrets secret, under the example namespace, do the following:

cat <<EOF | kubectl apply -f -
# Role to access 'vol-secrets' secret under 'example' namespace
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: px-vol-enc-role
  namespace: example
rules:
- apiGroups: [""]
  verbs: ["get"]
  resources: ["secrets"]
  resourceNames: ["vol-secrets"]
---
# Allow portworx service account to access the secret
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: px-vol-enc-role-binding
  namespace: example
subjects:
- kind: ServiceAccount
  name: px-account
  namespace: kube-system
roleRef:
  kind: Role
  name: px-vol-enc-role
  apiGroup: rbac.authorization.k8s.io
EOF```
8

I tried Marc_Pango solution and it worked.
I also tried yours and it still not work until I did what Marc_Pango has suggested.
One thing not very clear to me is that who created that Portworx namespace. When I installed the operator, I don’t see that as prerequisite?

9

portworx operator creates the portworx namespace