Portworx pod failed to startup

Portworx Essentials
1

install portworx essentail with kubectl apply -f portworkx-essentail.yaml

kubectl describe pod -n kube-system portworx-fd2t7 
Events:
  Type     Reason                             Age                  From                        Message
  ----     ------                             ----                 ----                        -------
  Normal   Pulling                            25m                  kubelet, ip-10-230-12-153   Pulling image "portworx/oci-monitor:2.5.7"
  Normal   Pulled                             25m                  kubelet, ip-10-230-12-153   Successfully pulled image "portworx/oci-monitor:2.5.7" in 1.463315221s
  Normal   Created                            25m                  kubelet, ip-10-230-12-153   Created container portworx
  Normal   Started                            25m                  kubelet, ip-10-230-12-153   Started container portworx
  Warning  Unhealthy                          25m                  kubelet, ip-10-230-12-153   Readiness probe failed: Get "http://127.0.0.1:9015/health": dial tcp 127.0.0.1:9015: connect: connection refused
  Normal   PortworxMonitorImagePullInPrgress  25m                  portworx, ip-10-230-12-153  Portworx image portworx/px-essentials:2.5.7 pull and extraction in progress
  Normal   Scheduled                          25m                  default-scheduler           Successfully assigned kube-system/portworx-fd2t7 to ip-10-230-12-153
  Warning  InternalKvdbSetupFailed            12m                  portworx, ip-10-230-12-153  Failed to setup internal kvdb: failed in initializing drives on this node: Drive /dev/sda2 size  mismatch: (10485760512 -7264534528) > 3221225472
  Warning  NodeStartFailure                   12m                  portworx, ip-10-230-12-153  Failed to start Portworx: failed in internal kvdb setup: failed in initializing drives on this node: Drive /dev/sda2 size  mismatch: (10485760512 -7264534528) > 3221225472
  Normal   PortworxMonitorImagePullInPrgress  10m                  portworx, ip-10-230-12-153  Portworx image portworx/px-essentials:2.5.7 pull and extraction in progress
  Warning  InternalKvdbSetupFailed            9m32s                portworx, ip-10-230-12-153  Failed to setup internal kvdb: failed in initializing drives on this node: Drive /dev/sda2 size  mismatch: (10485760512 -7264534528) > 3221225472
  Warning  NodeStartFailure                   9m32s                portworx, ip-10-230-12-153  Failed to start Portworx: failed in internal kvdb setup: failed in initializing drives on this node: Drive /dev/sda2 size  mismatch: (10485760512 -7264534528) > 3221225472
  Warning  Unhealthy                          24s (x149 over 25m)  kubelet, ip-10-230-12-153   Readiness probe failed: HTTP probe failed with statuscode: 503

I have created a 10 GB /dev/sda

On the worker node itself:

lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  9.8G  0 disk
├─sda1   8:1    0    3G  0 part
└─sda2   8:2    0  6.8G  0 part
sr0     11:0    1  364K  0 rom
vda    253:0    0   80G  0 disk
└─vda1 253:1    0   80G  0 part /
2

I have a clean and empty /dev/sda (wipe out everytgin with dd) and when installing Portworx, it tries to create /dev/sda2, and after after that, the size of the disk is incorrect. That is weird. Why is it incorrect?

I have unattach and re-attach the disk and it happens on all worker node with disk /dev/sda.

3

Hi Kaitaklam,
It looks like you chose auto create journal device when generating the spec. That results in creating a 3GB partition, the minimum disk size is 8G and after creating the partition your disk does not satisfy the minimum requirements. Your options are to either choose a larger disk - which is recommended - or uncheck the auto create journal device box when generating a spec.

4

Thanks, I will try. Is 20 GB sufficient?

Outlook for Android downloaden

5

With 40GB, I got the same issue. I will try with disabling the auto create journal device

@ip-10-230-13-151 portworx[15289]: time=“2020-09-24T06:50:24Z” level=error msg=“Failed to replace drive /dev/sda with /dev/sda2: Drive /dev/sda2 size mismatch: (41943040512 -38721814528) > 3221225472”

@ip-10-230-13-151 portworx[15289]: time=“2020-09-24T06:50:24Z” level=error msg=“Unable to start internal kvdb on this node” err=“failed in initializing drives on this node: Drive /dev/sda2 size mismatch: (41943040512 -38721814528) > 3221225472” fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=b5de556a-62ba-40f3-8c97-53954cbd7734

@ip-10-230-13-151 portworx[15289]: time=“2020-09-24T06:50:24Z” level=error msg=“failed in internal kvdb setup: failed in initializing drives on this node: Drive /dev/sda2 size mismatch: (41943040512 -38721814528) > 3221225472” func=InitAndBoot package=boot

@ip-10-230-13-151 portworx[15289]: time=“2020-09-24T06:50:24Z” level=error msg=“Could not init boot manager” error=“failed in internal kvdb setup: failed in initializing drives on this node: Drive /dev/sda2 size mismatch: (41943040512 -38721814528) > 3221225472”

@ip-10-230-13-151 portworx[15289]: PXPROCS: px daemon exited with code: 1

image002.jpg

6

Hi,

Make sure to cleanup before reapplying the new spec file.

curl -fsL https://install.portworx.com/px-wipe | bash

Use the above command from where you have kubectl.

7

I have unattach and re-attach the disk and it happens on all worker node with disk /dev/sda.

8

Can you share the logs from the worker node : journalctl -lu portworx* > /tmp/w1.log

9

I have disable autojournal in the requirement Portworx UI, generate px-essential, yaml and it works. SO this is no longer an issue for me. Thanks