Failed to load PX filesystem dependencies

Portworx on Kubernetes
1

We are trying to install Portworx Enterprise on OpenShift 4.4 running on bare metal using the Operator. We installed the operator just fine and now are trying to deploy a StorageCluster. Below is the manifest:

# SOURCE: https://install.portworx.com/?mc=false&kbver=1.17.1%2B912792b&b=true&s=%2Fdev%2Fsdb&j=auto&kd=%2Fdev%2Fsdc&cache=undefined&c=px-cluster-8812c620-2f7c-41ac-92a7-3750d24f7a19&osft=true&operator=true&stork=true&csi=true&st=k8s
kind: StorageCluster
apiVersion: core.libopenstorage.org/v1alpha1
metadata:
  name: px-cluster-8812c620-2f7c-41ac-92a7-3750d24f7a19
  namespace: kube-system
  annotations:
    portworx.io/install-source: "https://install.portworx.com/?mc=false&kbver=1.17.1%2B912792b&b=true&s=%2Fdev%2Fsdb&j=auto&kd=%2Fdev%2Fsdc&cache=undefined&c=px-cluster-8812c620-2f7c-41ac-92a7-3750d24f7a19&osft=true&operator=true&stork=true&csi=true&st=k8s"
    portworx.io/is-openshift: "true"
spec:
  image: portworx/oci-monitor:2.5.1
  imagePullPolicy: Always
  kvdb:
    internal: true
  storage:
    devices:
    - /dev/sdb
    journalDevice: auto
    kvdbDevice: /dev/sdc
  secretsProvider: k8s
  stork:
    enabled: true
    image: openstorage/stork:2.4.2
  autopilot:
    enabled: true
    image: portworx/autopilot:1.2.1
    providers:
    - name: default
      type: prometheus
      params:
        url: http://prometheus:9090
  featureGates:
    CSI: "true"

The pods don’t come up and I ran pxctl it said it failed to load PX filesystem dependencies for kernel 4.18.0-147.20.1.el8_1.x86_64.

image
image1386Ă—191 16 KB

We are running on OpenShift 4.4.9

$ oc version
Client Version: 4.4.4
Server Version: 4.4.9
Kubernetes Version: v1.17.1+912792b

Any ideas where to look?

2

It looks like you may have a newer kernel than in our mirrors. While we check, can you see if you can install the headers and restart the portworx service on the nodes to see if it picks it up?

3

Absolutely, but I must admit, I’m not our resident Linux guru, just playing around with this trying to get it to work outside of hours for the team. What are the steps to install the headers?

4

Updated command: yum install -y kernel-devel-$(uname -r)

You can restart the portworx pods by issue systemctl restart portworx after you install the kernel headers.

We will update our mirrors if that version headers have been published.

5

I tried running the yum install command on both the worker node directly and in the portworx pod but it said yum command not found. The nodes are running Red Hat CoreOS

6

Pls try this: dnf -y install kernel-devel-(uname -r) or yum install -y kernel-devel-(uname -r)

7

or yum install -y kernel-devel-$(uname -r)

8

dnf: command not found, I tried that one before too. Should I be running these on the worker nodes? It seems that they aren’t installed.

image

I think I might reinstall OpenShift which will bring me back to 4.4.3. We were going to do that anyway

9

Can you try running: ```
/usr/bin/toolbox

The run the dnf command?
10

That works in that now I can run dnf, but it returned an error:

image
image1006Ă—213 10.3 KB

11

I am sorry but that kernel-devel package does not seem to be available. It appears RedHat has not uploaded the packages for that yet.

12

Thanks for helping me narrow it down. Tomorrow I’ll tear down this cluster and make a new one on a slightly older version and see what happens. I can update this with the results.

13

We have recently tested up to Openshift 4.4.6 and whatever kernel version that got bundled with that.

14

So we tore down our cluster, reconfigured an OpenShift 4.4.3 cluster, then we upgraded it to 4.4.6 and then tried to install Portworx. We got the same error as when we ran it on 4.4.9. The operator installed fine, but creating the StorageCluster failed.

Failed to load PX filesystem dependencies for kernel 4.18.0-147.8.1.e18_1.x86_64

I tried to install the kernel-devel headers. It failed this time too, but this time with a different kernel version.

MicrosoftTeams-image (1)
MicrosoftTeams-image (1)931Ă—165 50 KB

Here is our oc version just to confirm it is 4.4.6

# oc version
Client Version: 4.4.9
Server Version: 4.4.6
Kubernetes Version: v1.17.1+f63db30

Any thoughts? We may go back and retry again on 4.4.3

Is it worth noting that we are running this on VMs running on Hyper-V with NVMe backed virtual hard disks attached to the VMs?

15

Would be possible to see more of the logging from the failed node?

16

Here’s the logs from the portworx container in one of the px-cluster pods…

17

kmenzel , Thank you so much for the logs. The reason the FS dependency is failing is because “Secure Boot is enabled”. This prevents us from installing the dependency. This would have be disabled for us to run on this system.

@lab-worker-05 portworx[2916107]: Secure boot is enabled. PX filesystem dependency operations will fail when secure boot is enabled.
@lab-worker-05 portworx[2916107]: Failed to load PX filesystem dependencies for kernel 4.18.0-147.8.1.el8_1.x86_64

1 Like
18

That was it! I’m not sure how I didn’t see that in the logs, I read through them before and missed it. Thanks for your great support! We’re going to go through the docs and learn more about how to get everything set up and working in our environment.