Portworx does not initialise on OCP version 4.8.24

Portworx Install
1

Hello Forum
I am only seeing this issue on OCP4.8.24, tried on previous version (OCP4.8.13) and Portworx installs fine.

Portworx Enterprise
1.6.1 provided by Portworx

portworx-api(s) do not start reporting the following event

Readiness probe failed: Get “http://127.0.0.1:17001/status”: dial tcp 127.0.0.1:17001: connect: connection refused

px-cluster pods report connection issue also

@worker2.aivt2-321-120137.cp.com portworx[304327]: Downloading from: https://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.18.0-305.28.1.el8_4.x86_64/packages ...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Download kernel failed, trying again...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Download kernel failed...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Failed to install/locate kernel headers for 4.18.0-305.28.1.el8_4.x86_64
@worker2.aivt2-321-120137.cp.com portworx[304327]: Downloading from: https://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.18.0-305.28.1.el8_4.x86_64/version/10...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Trying from: https://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.18.0-305.el8_4.x86_64/version/10...
@worker2.aivt2-321-120137.cp.com systemd[1]: run-runc-661efa709cff2b28a1129ef8355c46ce20ba90920259980a4e27f038cb5c1ff4-runc.8POm8H.mount: Succeeded.
@worker2.aivt2-321-120137.cp.com portworx[304327]: Unavailable: https://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.18.0-305.el8_4.x86_64/version/10
@worker2.aivt2-321-120137.cp.com portworx[304327]: Downloading from: https://mirrors.portworx.com/build-results/pxfuse/for-installer/108f0df/x86_64/4.18.0-305.28.1.el8_4.x86_64/version/10...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Trying from: https://mirrors.portworx.com/build-results/pxfuse/for-installer/108f0df/x86_64/4.18.0-305.el8_4.x86_64/version/10...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Unavailable: https://mirrors.portworx.com/build-results/pxfuse/for-installer/108f0df/x86_64/4.18.0-305.el8_4.x86_64/version/10
@worker2.aivt2-321-120137.cp.com portworx[304327]: checking local archive, please wait...
@worker2.aivt2-321-120137.cp.com portworx[304327]: Failed to load PX filesystem dependencies for kernel 4.18.0-305.28.1.el8_4.x86_64
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx.service: Main process exited, code=exited, status=10/n/a
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx.service: Failed with result 'exit-code'.
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx.service: Consumed 8.380s CPU time
@worker2.aivt2-321-120137.cp.com systemd[1]: Stopping Portworx FIFO logging reader...
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx-output.service: Succeeded.
@worker2.aivt2-321-120137.cp.com systemd[1]: Stopped Portworx FIFO logging reader.
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx-output.service: Consumed 11ms CPU time
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx.socket: Succeeded.
@worker2.aivt2-321-120137.cp.com systemd[1]: Closed Portworx logging FIFO.
@worker2.aivt2-321-120137.cp.com systemd[1]: portworx.socket: Consumed 0 CPU time
@worker2.aivt2-321-120137.cp.com systemd[1]: Starting Network Manager Script Dispatcher Service...
@worker2.aivt2-321-120137.cp.com systemd[1]: Started Network Manager Script Dispatcher Service.
@worker2.aivt2-321-120137.cp.com systemd[1]: run-runc-661efa709cff2b28a1129ef8355c46ce20ba90920259980a4e27f038cb5c1ff4-runc.VAx5aD.mount: Succeeded.
@worker2.aivt2-321-120137.cp.com systemd[1]: NetworkManager-dispatcher.service: Succeeded.
@worker2.aivt2-321-120137.cp.com systemd[1]: NetworkManager-dispatcher.service: Consumed 49ms CPU time
time="2022-01-12T13:46:55Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:17001/v1/cluster/nodehealth: dial tcp 127.0.0.1:17001: connect: connection refused"
2

Seeing the same issue with OCP4.7.40

3

What is the portworx version with 1.6.1 you are using ? Can you share your StorageCluster spec ?

4

Yes 1.6.1

apiVersion: v1
items:
- apiVersion: core.libopenstorage.org/v1
  kind: StorageCluster
  metadata:
    annotations:
      portworx.io/install-source: https://install.portworx.com/?operator=true&mc=false&kbver=&b=true&f=true&j=auto&c=px-cluster-9ba97b1d-6b05-497b-990d-600e97e92b3e&osft=true&stork=true&st=k8s&
rsec=regcred
      portworx.io/is-openshift: "true"
    creationTimestamp: "2022-01-19T23:36:53Z"
    finalizers:
    - operator.libopenstorage.org/delete
    generation: 2
    name: px-cluster-9ba97b1d-6b05-497b-990d-600e97e92b3e
    namespace: kube-system
    resourceVersion: "32726"
    uid: 461f2c1d-4ad1-4c2c-8dc2-83d609ae7f1b
  spec:
    autopilot:
      enabled: true
      providers:
      - name: default
        params:
          url: http://prometheus:9090
        type: prometheus
    env:
    - name: SAAS_ACCOUNT_KEY_STRING
      valueFrom:
        secretKeyRef:
          key: account-key
          name: px-saas-key
    featureGates:
      CSI: "true"
    image: portworx/oci-monitor:2.6.3
    imagePullPolicy: Always
    imagePullSecret: regcred
    kvdb:
      internal: true
    placement:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: px/enabled
              operator: NotIn
              values:
              - "false"
            - key: node-role.kubernetes.io/infra
              operator: DoesNotExist
            - key: node-role.kubernetes.io/master
              operator: DoesNotExist
          - matchExpressions:
            - key: px/enabled
              operator: NotIn
              values:
              - "false"
            - key: node-role.kubernetes.io/infra
              operator: DoesNotExist
            - key: node-role.kubernetes.io/master
              operator: Exists
            - key: node-role.kubernetes.io/worker
              operator: Exists
    revisionHistoryLimit: 10
    secretsProvider: k8s
    startPort: 17001
    storage:
      forceUseDisks: false
      journalDevice: auto
      useAll: true
    stork:
      args:
        webhook-controller: "false"
      enabled: true
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    version: 2.6.3
  status:
    clusterName: px-cluster-9ba97b1d-6b05-497b-990d-600e97e92b3e
    clusterUid: fd38b4b3-3460-4d57-95ec-1acf0d6f1934
    desiredImages:
      autopilot: portworx/autopilot:1.3.0
      csiAttacher: quay.io/openstorage/csi-attacher:v1.2.1-1
      csiNodeDriverRegistrar: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
      csiProvisioner: quay.io/openstorage/csi-provisioner:v1.6.0-1
      csiResizer: quay.io/k8scsi/csi-resizer:v0.5.0
      csiSnapshotter: quay.io/k8scsi/csi-snapshotter:v2.1.0
      stork: openstorage/stork:2.6.2
    phase: Online
    storage: {}
    version: 2.6.3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
5

Looks like you are using 2.6.3 you need to use the latest version with OCP 4.8, please use the latest available version which is 2.9.

6

Is there a latest tag for this image ?

7

You can generate the new spec and compare all images and use it.
PX-Central.
There is no latest tag as far as I know.

8

Thank a million Sanjay :+1: :grinning:

1 Like