Canont create encrypted volume: Key not found

Portworx on Kubernetes
1

Hello all,

I’m running in some troubles trying to set up encrypted volumes. I’m running Portworx essentials on AKS, installed with the portworx operator.

I followed the instructions on how to set up portworx encrypted volumes here

Though, when I create the PVC, it fails to provision with an error:

I0202 10:59:08.061612       1 portworx_util.go:58] Creating Portworx volume for PVC: secure-pvc
E0202 10:59:09.637656       1 portworx_util.go:108] Error creating Portworx Volume : rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-cd57a4e3-720c-49c9-9d1c-c08704e40144) create failed error: Key not found

So, I guessed that the secret is somehow not found. So running

/opt/pwx/bin/pxctl secrets kvdb list-secrets
Secret ID
px-cluster-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxX_px_secret
px-vol-encryption
/opt/pwx/bin/pxctl secrets kvdb get-secret
Enter Secret Id: px-cluster-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxX_px_secret
KvdbGetSecret: couldn't get: /getSecret with error: Invalid secretId

Not sure why this tells me “Invalid secretId”… Anyway

These are the logs from portworx-pvc-controller

I0202 10:59:08.061612       1 portworx_util.go:58] Creating Portworx volume for PVC: secure-pvc
E0202 10:59:09.637656       1 portworx_util.go:108] Error creating Portworx Volume : rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-cd57a4e3-720c-49c9-9d1c-c08704e40144) create failed error: Key not found

kubectl get secret -n portworx px-vol-encryption

apiVersion: v1
data:
  cluster-wide-secret-key: REDACTED
kind: Secret
metadata:
  name: px-vol-encryption
  namespace: portworx
type: Opaque

StorageCluster:

kind: StorageCluster
apiVersion: core.libopenstorage.org/v1
metadata:
  name: px-cluster-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxX
  namespace: kube-system
  annotations:
    portworx.io/install-source: portworx.io/install-source: "https://install.portworx.com/?operator=true&mc=false&kbver=1.17.13&oem=esse&user=zzzzzzzzzzzzzzzzzzzzzz&b=true&s=%22type%3DPremium_LRS%2Csize%3D150%22&kd=type%3DPremium_LRS%2Csize%3D150&c=px-cluster-zzzzzzzzzzzzzzzzzzzzzz&aks=true&stork=true&lh=true&mon=true&st=k8s&promop=true"
    portworx.io/is-aks: "true"
    portworx.io/misc-args: "--oem esse"
spec:
  image: portworx/oci-monitor:2.6.1.6
  imagePullPolicy: Always
  kvdb:
    internal: true
  cloudStorage:
    deviceSpecs:
      - type=Premium_LRS,size=150
    kvdbDeviceSpec: type=Premium_LRS,size=150
  secretsProvider: k8s
  stork:
    enabled: true
    args:
      webhook-controller: "false"
  userInterface:
    enabled: true
  autopilot:
    enabled: true
    providers:
      - name: default
        type: prometheus
        params:
          url: http://prometheus:9090
  monitoring:
    prometheus:
      enabled: true
      exportMetrics: true
  env:
    - name: AZURE_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: px-azure
          key: AZURE_CLIENT_SECRET
    - name: AZURE_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: px-azure
          key: AZURE_CLIENT_ID
    - name: AZURE_TENANT_ID
      valueFrom:
        secretKeyRef:
          name: px-azure
          key: AZURE_TENANT_ID
2

Hi,

Can you try the following command to verify if volume is created by command line or still getting any error?

/opt/pwx/bin/pxctl volume create --secure --size 1 encrypted_volume

Please post what is the result of above command.

3

Did you pass the "-secret_type", "k8s" arguments to the portworx container in the daemonset ? as per docs page https://docs.portworx.com/key-management/kubernetes-secrets