Storage device failed size check

#1

I am attempting to run Portworx for the 1st time (this is on Openshift), but it is failing to start.
One of the error messages I found is here: /var/lib/osd/log/disk_processing.log

Warning: skipping device: /dev/mapper/px_vg-px_storage. Failed size check, greater than 8G needed.
Head node:
No disks specified, starting as a storage-less node.

The device is however about 50G in size.
How is that storage check implemented ?
Should I do any additional preparation of the device before Portworx can use it?

thanks
Andrew

#2

The device size check simply looks at the size exported through sysfs. There is no additional preparation required to use the device. Could you post output of the following command from the failing node:

  1. lsblk --bytes
  2. lvs --all
#3

here is the output:

# lsblk --bytes
NAME                MAJ:MIN RM         SIZE RO TYPE MOUNTPOINT
sda                   8:0    0  85899345920  0 disk
├─sda1                8:1    0   1073741824  0 part /boot
└─sda2                8:2    0  36506173440  0 part
  ├─rhel-root       253:0    0  10737418240  0 lvm  /
  ├─rhel-swap       253:1    0   2143289344  0 lvm
  ├─rhel-tmp        253:3    0   2147483648  0 lvm  /tmp
  ├─rhel-home       253:4    0   2147483648  0 lvm  /home
  ├─rhel-var        253:5    0  46166704128  0 lvm  /var
  └─rhel-var_log    253:6    0  16106127360  0 lvm  /var/log
sdb                   8:16   0  42949672960  0 disk
└─rhel-var          253:5    0  46166704128  0 lvm  /var
sdc                   8:32   0  53687091200  0 disk
└─px_vg-px_storage  253:2    0  53682896896  0 lvm
sdd                   8:48   0 107374182400  0 disk
└─pvdb_vg-pvdb_meta 253:7    0 107369988096  0 lvm
sr0                  11:0    1   1073741312  0 rom
# lvs --all
  LV         VG      Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  pvdb_meta  pvdb_vg -wi-a----- <100.00g
  px_storage px_vg   -wi-a-----  <50.00g
  home       rhel    -wi-ao----    2.00g
  root       rhel    -wi-ao----   10.00g
  swap       rhel    -wi-a-----   <2.00g
  tmp        rhel    -wi-ao----    2.00g
  var        rhel    -wi-ao----  <43.00g
  var_log    rhel    -wi-ao----   15.00g
#4

The output looks fine. For some reason we aren’t able to read the device size correctly. Will need some info to debug.

  1. Can you post the install arguments (specifically the storage device options).
  2. Check if there were any other device errors reported above the “Warning: skipping device” line. And post a few more lines from the log before the warn.
  3. Run the following cmd and post the output:
    • lsblk /dev/mapper/px_vg-px_storage
    • lsblk --nodeps --bytes --noheadings --output SIZE /dev/mapper/px_vg-px_storage
#5

The journal log appears to show portworx finding and using the disks, put the pod still does not come ready.

oc apply -f ‘https://install.portworx.com/2.3?mc=false&kbver=1.11.0%2Bd4cacc0&b=true&s=%2Fdev%2Fmapper%2Fpx_vg-px_storage&md=%2Fdev%2Fmapper%2Fpvdb_vg-pvdb_meta&m=ens192&d=ens192&c=px-cluster-720c8731-71f3-40bb-9ca5-32b929480cc4&osft=true&stork=true&lh=true&st=k8s’

Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Found: x86_64/3.10.0-862.el7.x86_64
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: px: loading out-of-tree module taints kernel.
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: px: module verification failed: signature and/or required key missing - tainting kernel
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: pxd: driver loaded version 50a0d25555242afedca14b772b6b96bd7e3bccb1
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Using cluster: px-cluster-720c8731-71f3-40bb-9ca5-32b929480cc4
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Data network interface: ens192
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Management network interface: ens192Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Using meta data device: /dev/mapper/pvdb_vg-pvdb_meta
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Using storage device: /dev/mapper/px_vg-px_storage
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: Using scheduler: kubernetes
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain portworx[962]: patch_fs_create not needed.
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: xor: automatically using best checksumming function:
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: avx : 15276.000 MB/sec
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: raid6: sse2x1 gen() 7285 MB/s
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: raid6: sse2x2 gen() 9062 MB/s
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: raid6: sse2x4 gen() 10492 MB/s
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: raid6: using algorithm sse2x4 gen() (10492 MB/s)
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: raid6: using ssse3x2 recovery algorithm
Jan 09 08:41:14 labAakllocpa01.lab.labA.domain kernel: Btrfs loaded, crc32c=crc32c-intel

[root@labAakllocpa01 cores]# lsblk /dev/mapper/px_vg-px_storage
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
px_vg-px_storage 253:2 0 50G 0 lvm
[root@labAakllocpa01 cores]# lsblk --nodeps --bytes --noheadings --output SIZE /dev/mapper/px_vg-px_storage
53682896896

[root@labAakllocpa01 cores]# ls -l /var/lib/osd/log/termination.log
-rw-r-----. 1 root root 225 Jan 9 11:22 /var/lib/osd/log/termination.log

[root@labAakllocpa01 cores]# cat /var/lib/osd/log/termination.log
Could not init boot manager (error=“Failed in internal kvdb setup: Storage less node detected. No
de cannot act as a kvdb node. Waiting for other storage nodes to start kvdb.: timeout waiting for
other kvdb nodes to start.”)

#6

Thanks Andrew. Let me set up some time to take a look at the node and get this resolved. I’ll reach out to you on slack.

#7

As this was an initial install with no data yet on the disks, we wiped the storage disks with ‘pxctl service nw’ and restarted the portworx service.
That seemed to clear up the issue.
pxctl status, now reports everything is OK.

thanks