Unhealthy portworx container

eeginux · April 15, 2021, 3:56pm

Hi,

I have one EKS cluster with three nodes, and i have attached one extra volume size 10G each. In the spec, tried two options to use any unused device and specify the new volume device. After installing the portworx essential, the portworx container is in unhealthy status with the folllowing error,

time=“2021-04-15T16:09:46Z” level=error msg=“cmd: /sbin/parted type=gp2,size=150 print output:nsenter: failed to execute parted: No such file or directory\n” Device=“type=gp2,size=150” Error=“exit status 1” Function=runParted

time=“2021-04-15T16:09:46Z” level=error msg=“Unable to start internal kvdb on this node” err=“failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1” fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=a554b56d-e4e5-4029-b786-7ef81dc623f6

time=“2021-04-15T16:09:46Z” level=error msg=“failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1” func=InitAndBoot package=boot

time=“2021-04-15T16:09:46Z” level=error msg=“Could not init boot manager” error=“failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1”

Any idea what might be the problem here?

sensre · April 15, 2021, 10:20pm

use any unused device means -A argument you need to pass it in the daemonset spec install . but in the below error shows cloud drive option type=gp2,size=150 . so there is mismatch in your install as you mentioned. can you please post your install spec and the complete PX logs here?

eeginux · April 16, 2021, 7:10am

This is the spec,

https://install.portworx.com/?mc=false&kbver=1.18.9-eks-d1db3c&oem=esse&user=db12953f-81e4-11eb-a2c5-c24e499c7467&b=true&kd=type%3Dgp2%2Csize%3D150&c=px-cluster-15291395-1740-4f4c-86f6-34f411aff807&eks=true&stork=true&csi=true&lh=true&st=k8s

Any place i can upload or attach the log file?

eeginux · April 19, 2021, 8:00pm

time="2021-04-16T09:11:41Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-15291395-1740-4f4c-86f6-34f411aff807 -a -secret_type k8s -kvdb_dev type=gp2,size=150 -b --oem esse -x kubernetes"
time="2021-04-16T09:11:41Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-15291395-1740-4f4c-86f6-34f411aff807 -a -secret_type k8s -kvdb_dev type=gp2,size=150 -b -x kubernetes"
time="2021-04-16T09:11:41Z" level=info msg="OCI-Monitor computed version v2.7.0-gd349b25f-dirty"
time="2021-04-16T09:11:41Z" level=info msg="REAPER: Starting ..."
time="2021-04-16T09:11:41Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc000429880}"
time="2021-04-16T09:11:41Z" level=info msg="Setting up container handler"
time="2021-04-16T09:11:41Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-04-16T09:11:41Z" level=info msg="Locating my container handler"
time="2021-04-16T09:11:41Z" level=info msg="Negotiated Docker API version: 1.32"
time="2021-04-16T09:11:41Z" level=info msg="Detected NetworkMode container:7034acb377d08ee08d76819a7615ce5349466a71d41d2ae6be53954575f54fe1 -> polling 7034acb377d0 for network settings"
time="2021-04-16T09:11:41Z" level=info msg="> Using Docker as container handler"
time="2021-04-16T09:11:41Z" level=info msg="Activating REST server"
time="2021-04-16T09:11:41Z" level=info msg="Detected NetworkMode container:7034acb377d08ee08d76819a7615ce5349466a71d41d2ae6be53954575f54fe1 -> polling 7034acb377d0 for network settings"
time="2021-04-16T09:11:41Z" level=info msg="Detected HostNetwork setting - will track portworx status via REST"
time="2021-04-16T09:11:41Z" level=info msg="Parsed Registry/Repo portworx from own image URN portworx/oci-monitor@sha256:2d8adb9064e2103b43203963af7abd403f4476a4e374a6f3c78e5417642fb134"
time="2021-04-16T09:11:41Z" level=info msg="Detected custom registry server/repo -- installing PX-OCI from portworx/px-essentials:2.7.0"
time="2021-04-16T09:11:41Z" level=info msg="Removed env variables: [PATH PORTWORX_API_PORT PORTWORX_API_PORT_9001_TCP PORTWORX_API_PORT_9001_TCP_ADDR PORTWORX_API_PORT_9001_TCP_PORT PORTWORX_API_PORT_9001_TCP_PROTO PORTWORX_API_PORT_9020_TCP PORTWORX_API_PORT_9020_TCP_ADDR PORTWORX_API_PORT_9020_TCP_PORT PORTWORX_API_PORT_9020_TCP_PROTO PORTWORX_API_PORT_9021_TCP PORTWORX_API_PORT_9021_TCP_ADDR PORTWORX_API_PORT_9021_TCP_PORT PORTWORX_API_PORT_9021_TCP_PROTO PORTWORX_API_SERVICE_HOST PORTWORX_API_SERVICE_PORT PORTWORX_API_SERVICE_PORT_PX_API PORTWORX_API_SERVICE_PORT_PX_REST_GATEWAY PORTWORX_API_SERVICE_PORT_PX_SDK PORTWORX_SERVICE_PORT PORTWORX_SERVICE_PORT_9001_TCP PORTWORX_SERVICE_PORT_9001_TCP_ADDR PORTWORX_SERVICE_PORT_9001_TCP_PORT PORTWORX_SERVICE_PORT_9001_TCP_PROTO PORTWORX_SERVICE_PORT_9019_TCP PORTWORX_SERVICE_PORT_9019_TCP_ADDR PORTWORX_SERVICE_PORT_9019_TCP_PORT PORTWORX_SERVICE_PORT_9019_TCP_PROTO PORTWORX_SERVICE_PORT_9020_TCP PORTWORX_SERVICE_PORT_9020_TCP_ADDR PORTWORX_SERVICE_PORT_9020_TCP_PORT PORTWORX_SERVICE_PORT_9020_TCP_PROTO PORTWORX_SERVICE_PORT_9021_TCP PORTWORX_SERVICE_PORT_9021_TCP_ADDR PORTWORX_SERVICE_PORT_9021_TCP_PORT PORTWORX_SERVICE_PORT_9021_TCP_PROTO PORTWORX_SERVICE_SERVICE_HOST PORTWORX_SERVICE_SERVICE_PORT PORTWORX_SERVICE_SERVICE_PORT_PX_API PORTWORX_SERVICE_SERVICE_PORT_PX_KVDB PORTWORX_SERVICE_SERVICE_PORT_PX_REST_GATEWAY PORTWORX_SERVICE_SERVICE_PORT_PX_SDK STORK_SERVICE_PORT STORK_SERVICE_PORT_443_TCP STORK_SERVICE_PORT_443_TCP_ADDR STORK_SERVICE_PORT_443_TCP_PORT STORK_SERVICE_PORT_443_TCP_PROTO STORK_SERVICE_PORT_8099_TCP STORK_SERVICE_PORT_8099_TCP_ADDR STORK_SERVICE_PORT_8099_TCP_PORT STORK_SERVICE_PORT_8099_TCP_PROTO STORK_SERVICE_SERVICE_HOST STORK_SERVICE_SERVICE_PORT STORK_SERVICE_SERVICE_PORT_EXTENDER STORK_SERVICE_SERVICE_PORT_WEBHOOK]"
time="2021-04-16T09:11:41Z" level=info msg="Preparing to download Portworx image..."
time="2021-04-16T09:11:41Z" level=info msg="REST: Changing install-state: ST_UNKNOWN -> ST_INSTALL"
time="2021-04-16T09:11:41Z" level=info msg="Detected imagePullPolicy Always"
time="2021-04-16T09:11:41Z" level=info msg="Attempting to retrieve latest portworx/px-essentials:2.7.0 image (pullPolicy Always)"
time="2021-04-16T09:11:41Z" level=info msg="Using anonymous Docker credentials"
time="2021-04-16T09:11:43Z" level=info msg="Remote Portworx image digest identical to current image's digest (pull skipped)"
time="2021-04-16T09:11:43Z" level=info msg="Got requested Portworx image portworx/px-essentials:2.7.0 with digest sha256:2700033ab42d0c6ccba5eb1a1990f4f56496d0697078b29957b914cbaf494091"
time="2021-04-16T09:11:43Z" level=info msg="Installed image digest sha256:2700033ab42d0c6ccba5eb1a1990f4f56496d0697078b29957b914cbaf494091 same as remote/pulled image's (no need to reinstall)"
time="2021-04-16T09:11:43Z" level=info msg="Installing Portworx OCI service to /opt/pwx..."
time="2021-04-16T09:11:43Z" level=info msg="Prepending computed kubelet directory mount /var/lib/kubelet:/var/lib/kubelet:shared"
time="2021-04-16T09:11:43Z" level=info msg="Adding computed k8s-secrets-vol as volatile mount regex"
time="2021-04-16T09:11:43Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/etc-hosts:/etc/hosts'"
time="2021-04-16T09:11:43Z" level=info msg="> run-local: /usr/bin/rsync -aL --inplace --delete /etc/hosts /opt/pwx/oci/mounts/etc/hosts"
time="2021-04-16T09:11:43Z" level=info msg="> Changed mount /var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/etc-hosts:/etc/hosts to /opt/pwx/oci/mounts/etc/hosts:/etc/hosts"
time="2021-04-16T09:11:43Z" level=info msg="RSYNC: Found volatile mount '/var/lib/docker/containers/7034acb377d08ee08d76819a7615ce5349466a71d41d2ae6be53954575f54fe1/resolv.conf:/etc/resolv.conf:ro'"
time="2021-04-16T09:11:43Z" level=info msg="> run-local: /usr/bin/rsync -aL --inplace --delete /etc/resolv.conf /opt/pwx/oci/mounts/etc/resolv.conf"
time="2021-04-16T09:11:43Z" level=info msg="> Changed mount /var/lib/docker/containers/7034acb377d08ee08d76819a7615ce5349466a71d41d2ae6be53954575f54fe1/resolv.conf:/etc/resolv.conf:ro to /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf"
time="2021-04-16T09:11:43Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/containers/portworx/06e061c3:/tmp/px-termination-log'"
time="2021-04-16T09:11:43Z" level=info msg="> run-local: /usr/bin/rsync -aL --inplace --delete /tmp/px-termination-log /opt/pwx/oci/mounts/tmp/px-termination-log"
time="2021-04-16T09:11:43Z" level=info msg="> Changed mount /var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/containers/portworx/06e061c3:/tmp/px-termination-log to /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log"
time="2021-04-16T09:11:43Z" level=info msg="RSYNC: Found volatile mount '/var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/volumes/kubernetes.io~secret/px-account-token-bh74c:/var/run/secrets/kubernetes.io/serviceaccount:ro'"
time="2021-04-16T09:11:43Z" level=info msg="> run-local: /usr/bin/rsync -aL --inplace --delete /var/run/secrets/kubernetes.io/serviceaccount/ /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount/"
time="2021-04-16T09:11:43Z" level=info msg="> Changed mount /var/lib/kubelet/pods/f3053361-10af-4442-a115-c6585a5c9b96/volumes/kubernetes.io~secret/px-account-token-bh74c:/var/run/secrets/kubernetes.io/serviceaccount:ro to /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount"
time="2021-04-16T09:11:43Z" level=warning msg="CPU shares limit NOT passed to Portworx service (use `--cpu-shares 2` param instead)"
time="2021-04-16T09:11:43Z" level=info msg="> run-host: /opt/pwx/bin/px-runc install -c px-cluster-15291395-1740-4f4c-86f6-34f411aff807 -a -secret_type k8s -kvdb_dev type=gp2,size=150 -b -x kubernetes -v /var/lib/kubelet:/var/lib/kubelet:shared -v /dev:/dev -v /opt/pwx/oci/mounts/etc/hosts:/etc/hosts -v /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf -v /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log -v /var/cores:/var/cores -v /var/run/dbus:/var/run/dbus -v /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount -e CSI_ENDPOINT=unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock -e KUBERNETES_PORT=tcp://10.100.0.1:443 -e KUBERNETES_PORT_443_TCP=tcp://10.100.0.1:443 -e KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1 -e KUBERNETES_PORT_443_TCP_PORT=443 -e KUBERNETES_PORT_443_TCP_PROTO=tcp -e KUBERNETES_SERVICE_HOST=10.100.0.1 -e KUBERNETES_SERVICE_PORT=443 -e KUBERNETES_SERVICE_PORT_HTTPS=443 -e KUBE_DNS_PORT=udp://10.100.0.10:53 -e KUBE_DNS_PORT_53_TCP=tcp://10.100.0.10:53 -e KUBE_DNS_PORT_53_TCP_ADDR=10.100.0.10 -e KUBE_DNS_PORT_53_TCP_PORT=53 -e KUBE_DNS_PORT_53_TCP_PROTO=tcp -e KUBE_DNS_PORT_53_UDP=udp://10.100.0.10:53 -e KUBE_DNS_PORT_53_UDP_ADDR=10.100.0.10 -e KUBE_DNS_PORT_53_UDP_PORT=53 -e KUBE_DNS_PORT_53_UDP_PROTO=udp -e KUBE_DNS_SERVICE_HOST=10.100.0.10 -e KUBE_DNS_SERVICE_PORT=53 -e KUBE_DNS_SERVICE_PORT_DNS=53 -e KUBE_DNS_SERVICE_PORT_DNS_TCP=53 -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -e PORTWORX_API_PORT=tcp://10.100.201.120:9001 -e PORTWORX_API_PORT_9001_TCP=tcp://10.100.201.120:9001 -e PORTWORX_API_PORT_9001_TCP_ADDR=10.100.201.120 -e PORTWORX_API_PORT_9001_TCP_PORT=9001 -e PORTWORX_API_PORT_9001_TCP_PROTO=tcp -e PORTWORX_API_PORT_9020_TCP=tcp://10.100.201.120:9020 -e PORTWORX_API_PORT_9020_TCP_ADDR=10.100.201.120 -e PORTWORX_API_PORT_9020_TCP_PORT=9020 -e PORTWORX_API_PORT_9020_TCP_PROTO=tcp -e PORTWORX_API_PORT_9021_TCP=tcp://10.100.201.120:9021 -e PORTWORX_API_PORT_9021_TCP_ADDR=10.100.201.120 -e PORTWORX_API_PORT_9021_TCP_PORT=9021 -e PORTWORX_API_PORT_9021_TCP_PROTO=tcp -e PORTWORX_API_SERVICE_HOST=10.100.201.120 -e PORTWORX_API_SERVICE_PORT=9001 -e PORTWORX_API_SERVICE_PORT_PX_API=9001 -e PORTWORX_API_SERVICE_PORT_PX_REST_GATEWAY=9021 -e PORTWORX_API_SERVICE_PORT_PX_SDK=9020 -e PORTWORX_SERVICE_PORT=tcp://10.100.168.174:9001 -e PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.100.168.174:9001 -e PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.100.168.174 -e PORTWORX_SERVICE_PORT_9001_TCP_PORT=9001 -e PORTWORX_SERVICE_PORT_9001_TCP_PROTO=tcp -e PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.100.168.174:9019 -e PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.100.168.174 -e PORTWORX_SERVICE_PORT_9019_TCP_PORT=9019 -e PORTWORX_SERVICE_PORT_9019_TCP_PROTO=tcp -e PORTWORX_SERVICE_PORT_9020_TCP=tcp://10.100.168.174:9020 -e PORTWORX_SERVICE_PORT_9020_TCP_ADDR=10.100.168.174 -e PORTWORX_SERVICE_PORT_9020_TCP_PORT=9020 -e PORTWORX_SERVICE_PORT_9020_TCP_PROTO=tcp -e PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.100.168.174:9021 -e PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.100.168.174 -e PORTWORX_SERVICE_PORT_9021_TCP_PORT=9021 -e PORTWORX_SERVICE_PORT_9021_TCP_PROTO=tcp -e PORTWORX_SERVICE_SERVICE_HOST=10.100.168.174 -e PORTWORX_SERVICE_SERVICE_PORT=9001 -e PORTWORX_SERVICE_SERVICE_PORT_PX_API=9001 -e PORTWORX_SERVICE_SERVICE_PORT_PX_KVDB=9019 -e PORTWORX_SERVICE_SERVICE_PORT_PX_REST_GATEWAY=9021 -e PORTWORX_SERVICE_SERVICE_PORT_PX_SDK=9020 -e PX_TEMPLATE_VERSION=v4 -e STORK_SERVICE_PORT=tcp://10.100.102.48:8099 -e STORK_SERVICE_PORT_443_TCP=tcp://10.100.102.48:443 -e STORK_SERVICE_PORT_443_TCP_ADDR=10.100.102.48 -e STORK_SERVICE_PORT_443_TCP_PORT=443 -e STORK_SERVICE_PORT_443_TCP_PROTO=tcp -e STORK_SERVICE_PORT_8099_TCP=tcp://10.100.102.48:8099 -e STORK_SERVICE_PORT_8099_TCP_ADDR=10.100.102.48 -e STORK_SERVICE_PORT_8099_TCP_PORT=8099 -e STORK_SERVICE_PORT_8099_TCP_PROTO=tcp -e STORK_SERVICE_SERVICE_HOST=10.100.102.48 -e STORK_SERVICE_SERVICE_PORT=8099 -e STORK_SERVICE_SERVICE_PORT_EXTENDER=8099 -e STORK_SERVICE_SERVICE_PORT_WEBHOOK=443 -e container=oci -e PX_IMAGE=portworx/px-essentials:2.7.0 -e CONTAINER_RUNTIME=docker -e PX_IMAGE_DIGEST=sha256:2700033ab42d0c6ccba5eb1a1990f4f56496d0697078b29957b914cbaf494091 -e KUBELET_DIR=/var/lib/kubelet"
time="2021-04-16T09:11:45Z" level=warning msg="LOW RESOURCES: Host's maximum CPU-cores (2) is below recommended minimum (4)"
time="2021-04-16T09:11:45Z" level=info msg="Rootfs found at /opt/pwx/oci/rootfs"
time="2021-04-16T09:11:45Z" level=info msg="PX binaries found at /opt/pwx/bin/px-runc"
time="2021-04-16T09:11:45Z" level=info msg="Initializing as version 2.7.0.0-ccee71c (OCI)"
time="2021-04-16T09:11:45Z" level=info msg="Enabling Sharedv4 NFS support ..."
time="2021-04-16T09:11:45Z" level=info msg="Setting up NFS service"
time="2021-04-16T09:11:45Z" level=info msg="> Initialized service controls via DBus{type:dbus,svc:nfs-server.service,id:0xc0004615c0}"
time="2021-04-16T09:11:45Z" level=info msg="Fixing docker.sock mount:"
time="2021-04-16T09:11:45Z" level=info msg="> Removing mount for /var/run/docker.sock:/var/run/docker.sock:[rbind rprivate]"
time="2021-04-16T09:11:45Z" level=info msg="> Adding mount for /run:/var/host_run:[bind rprivate]"
time="2021-04-16T09:11:45Z" level=info msg="> Soft-link /opt/pwx/oci/rootfs/run/docker.sock -> /var/host_run/docker.sock already exists"
time="2021-04-16T09:11:45Z" level=info msg="Checking mountpoints for following shared directories: [/var/lib/kubelet /var/lib/osd]"
time="2021-04-16T09:11:45Z" level=info msg="Found following mountpoints for shared dirs: map[/:{isMP=T,Opts=shared:1} /var/lib/kubelet:{isMP=f,Opts=shared:1,Parent=/} /var/lib/osd:{isMP=f,Opts=shared:1,Parent=/}]"
time="2021-04-16T09:11:45Z" level=info msg="SPEC UPDATED [088847186943fb8adb7be5cf70816341  /opt/pwx/oci/config.json]"
time="2021-04-16T09:11:45Z" level=info msg="> Updated env: add{PORTWORX_API_PORT=tcp://10.100.201.120:9001 PORTWORX_API_PORT_9001_TCP=tcp://10.100.201.120:9001 PORTWORX_API_PORT_9001_TCP_ADDR=10.100.201.120 PORTWORX_API_PORT_9020_TCP=tcp://10.100.201.120:9020 PORTWORX_API_PORT_9020_TCP_ADDR=10.100.201.120 PORTWORX_API_PORT_9021_TCP=tcp://10.100.201.120:9021 PORTWORX_API_PORT_9021_TCP_ADDR=10.100.201.120 PORTWORX_API_SERVICE_HOST=10.100.201.120 PORTWORX_SERVICE_PORT=tcp://10.100.168.174:9001 PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.100.168.174:9001 PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.100.168.174:9019 PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9020_TCP=tcp://10.100.168.174:9020 PORTWORX_SERVICE_PORT_9020_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.100.168.174:9021 PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_SERVICE_HOST=10.100.168.174 STORK_SERVICE_PORT=tcp://10.100.102.48:8099 STORK_SERVICE_PORT_443_TCP=tcp://10.100.102.48:443 STORK_SERVICE_PORT_443_TCP_ADDR=10.100.102.48 STORK_SERVICE_PORT_8099_TCP=tcp://10.100.102.48:8099 STORK_SERVICE_PORT_8099_TCP_ADDR=10.100.102.48 STORK_SERVICE_SERVICE_HOST=10.100.102.48} rm{AUTOPILOT_PORT=tcp://10.100.75.186:9628 AUTOPILOT_PORT_9628_TCP=tcp://10.100.75.186:9628 AUTOPILOT_PORT_9628_TCP_ADDR=10.100.75.186 AUTOPILOT_PORT_9628_TCP_PORT=9628 AUTOPILOT_PORT_9628_TCP_PROTO=tcp AUTOPILOT_SERVICE_HOST=10.100.75.186 AUTOPILOT_SERVICE_PORT=9628 AUTOPILOT_SERVICE_PORT_AUTOPILOT=9628 PORTWORX_API_PORT=tcp://10.100.100.242:9001 PORTWORX_API_PORT_9001_TCP=tcp://10.100.100.242:9001 PORTWORX_API_PORT_9001_TCP_ADDR=10.100.100.242 PORTWORX_API_PORT_9020_TCP=tcp://10.100.100.242:9020 PORTWORX_API_PORT_9020_TCP_ADDR=10.100.100.242 PORTWORX_API_PORT_9021_TCP=tcp://10.100.100.242:9021 PORTWORX_API_PORT_9021_TCP_ADDR=10.100.100.242 PORTWORX_API_SERVICE_HOST=10.100.100.242 PORTWORX_SERVICE_PORT=tcp://10.100.250.70:9001 PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.100.250.70:9001 PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.100.250.70 PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.100.250.70:9019 PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.100.250.70 PORTWORX_SERVICE_PORT_9020_TCP=tcp://10.100.250.70:9020 PORTWORX_SERVICE_PORT_9020_TCP_ADDR=10.100.250.70 PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.100.250.70:9021 PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.100.250.70 PORTWORX_SERVICE_SERVICE_HOST=10.100.250.70 PX_LIGHTHOUSE_PORT=tcp://10.100.64.235:80 PX_LIGHTHOUSE_PORT_443_TCP=tcp://10.100.64.235:443 PX_LIGHTHOUSE_PORT_443_TCP_ADDR=10.100.64.235 PX_LIGHTHOUSE_PORT_443_TCP_PORT=443 PX_LIGHTHOUSE_PORT_443_TCP_PROTO=tcp PX_LIGHTHOUSE_PORT_80_TCP=tcp://10.100.64.235:80 PX_LIGHTHOUSE_PORT_80_TCP_ADDR=10.100.64.235 PX_LIGHTHOUSE_PORT_80_TCP_PORT=80 PX_LIGHTHOUSE_PORT_80_TCP_PROTO=tcp PX_LIGHTHOUSE_SERVICE_HOST=10.100.64.235 PX_LIGHTHOUSE_SERVICE_PORT=80 PX_LIGHTHOUSE_SERVICE_PORT_HTTP=80 PX_LIGHTHOUSE_SERVICE_PORT_HTTPS=443 STORK_SERVICE_PORT=tcp://10.100.59.79:8099 STORK_SERVICE_PORT_443_TCP=tcp://10.100.59.79:443 STORK_SERVICE_PORT_443_TCP_ADDR=10.100.59.79 STORK_SERVICE_PORT_8099_TCP=tcp://10.100.59.79:8099 STORK_SERVICE_PORT_8099_TCP_ADDR=10.100.59.79 STORK_SERVICE_SERVICE_HOST=10.100.59.79}"
time="2021-04-16T09:11:45Z" level=info msg="PX-RunC arguments: -a -b -c px-cluster-15291395-1740-4f4c-86f6-34f411aff807 -kvdb_dev type=gp2,size=150 -secret_type k8s -x kubernetes"
time="2021-04-16T09:11:45Z" level=info msg="PX-RunC mounts: /dev:/dev /etc/exports:/etc/exports /opt/pwx/oci/mounts/etc/hosts:/etc/hosts /etc/iscsi:/etc/iscsi /etc/mdadm:/etc/mdadm /etc/nvme:/etc/nvme /etc/nvmet:/etc/nvmet /etc/pwx:/etc/pwx /opt/pwx/oci/mounts/etc/resolv.conf:/etc/resolv.conf /etc/target:/etc/target /opt/pwx/bin:/export_bin /proc:/hostproc /lib/modules:/lib/modules proc:/proc:nosuid,noexec,nodev /run/docker:/run/docker /run/lock/iscsi:/run/lock/iscsi /run/lvm:/run/lvm /run/mdadm:/run/mdadm /run/udev:/run/udev sysfs:/sys:nosuid,noexec,nodev cgroup:/sys/fs/cgroup:nosuid,noexec,nodev /opt/pwx/oci/mounts/tmp/px-termination-log:/tmp/px-termination-log /usr/src:/usr/src /var/cores:/var/cores /run:/var/host_run:bind /var/lib/iscsi:/var/lib/iscsi /var/lib/kubelet:/var/lib/kubelet:shared /var/lib/nfs:/var/lib/nfs /var/lib/osd:/var/lib/osd:shared /var/lock/iscsi:/var/lock/iscsi /var/log/journal:/var/log/journal:ro /var/run/dbus:/var/run/dbus /opt/pwx/oci/mounts/var/run/secrets/kubernetes.io/serviceaccount:/var/run/secrets/kubernetes.io/serviceaccount"
time="2021-04-16T09:11:45Z" level=info msg="PX-RunC env: CONTAINER_RUNTIME=docker CSI_ENDPOINT=unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock DBUS_SESSION_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket GOMAXPROCS=64 GOTRACEBACK=crash KUBELET_DIR=/var/lib/kubelet KUBERNETES_PORT=tcp://10.100.0.1:443 KUBERNETES_PORT_443_TCP=tcp://10.100.0.1:443 KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_SERVICE_HOST=10.100.0.1 KUBERNETES_SERVICE_PORT=443 KUBERNETES_SERVICE_PORT_HTTPS=443 KUBE_DNS_PORT=udp://10.100.0.10:53 KUBE_DNS_PORT_53_TCP=tcp://10.100.0.10:53 KUBE_DNS_PORT_53_TCP_ADDR=10.100.0.10 KUBE_DNS_PORT_53_TCP_PORT=53 KUBE_DNS_PORT_53_TCP_PROTO=tcp KUBE_DNS_PORT_53_UDP=udp://10.100.0.10:53 KUBE_DNS_PORT_53_UDP_ADDR=10.100.0.10 KUBE_DNS_PORT_53_UDP_PORT=53 KUBE_DNS_PORT_53_UDP_PROTO=udp KUBE_DNS_SERVICE_HOST=10.100.0.10 KUBE_DNS_SERVICE_PORT=53 KUBE_DNS_SERVICE_PORT_DNS=53 KUBE_DNS_SERVICE_PORT_DNS_TCP=53 LVM_USE_HOST=1 NFS_SERVICE=nfs-server.service PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PORTWORX_API_PORT=tcp://10.100.201.120:9001 PORTWORX_API_PORT_9001_TCP=tcp://10.100.201.120:9001 PORTWORX_API_PORT_9001_TCP_ADDR=10.100.201.120 PORTWORX_API_PORT_9001_TCP_PORT=9001 PORTWORX_API_PORT_9001_TCP_PROTO=tcp PORTWORX_API_PORT_9020_TCP=tcp://10.100.201.120:9020 PORTWORX_API_PORT_9020_TCP_ADDR=10.100.201.120 PORTWORX_API_PORT_9020_TCP_PORT=9020 PORTWORX_API_PORT_9020_TCP_PROTO=tcp PORTWORX_API_PORT_9021_TCP=tcp://10.100.201.120:9021 PORTWORX_API_PORT_9021_TCP_ADDR=10.100.201.120 PORTWORX_API_PORT_9021_TCP_PORT=9021 PORTWORX_API_PORT_9021_TCP_PROTO=tcp PORTWORX_API_SERVICE_HOST=10.100.201.120 PORTWORX_API_SERVICE_PORT=9001 PORTWORX_API_SERVICE_PORT_PX_API=9001 PORTWORX_API_SERVICE_PORT_PX_REST_GATEWAY=9021 PORTWORX_API_SERVICE_PORT_PX_SDK=9020 PORTWORX_SERVICE_PORT=tcp://10.100.168.174:9001 PORTWORX_SERVICE_PORT_9001_TCP=tcp://10.100.168.174:9001 PORTWORX_SERVICE_PORT_9001_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9001_TCP_PORT=9001 PORTWORX_SERVICE_PORT_9001_TCP_PROTO=tcp PORTWORX_SERVICE_PORT_9019_TCP=tcp://10.100.168.174:9019 PORTWORX_SERVICE_PORT_9019_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9019_TCP_PORT=9019 PORTWORX_SERVICE_PORT_9019_TCP_PROTO=tcp PORTWORX_SERVICE_PORT_9020_TCP=tcp://10.100.168.174:9020 PORTWORX_SERVICE_PORT_9020_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9020_TCP_PORT=9020 PORTWORX_SERVICE_PORT_9020_TCP_PROTO=tcp PORTWORX_SERVICE_PORT_9021_TCP=tcp://10.100.168.174:9021 PORTWORX_SERVICE_PORT_9021_TCP_ADDR=10.100.168.174 PORTWORX_SERVICE_PORT_9021_TCP_PORT=9021 PORTWORX_SERVICE_PORT_9021_TCP_PROTO=tcp PORTWORX_SERVICE_SERVICE_HOST=10.100.168.174 PORTWORX_SERVICE_SERVICE_PORT=9001 PORTWORX_SERVICE_SERVICE_PORT_PX_API=9001 PORTWORX_SERVICE_SERVICE_PORT_PX_KVDB=9019 PORTWORX_SERVICE_SERVICE_PORT_PX_REST_GATEWAY=9021 PORTWORX_SERVICE_SERVICE_PORT_PX_SDK=9020 PX_IMAGE=portworx/px-essentials:2.7.0 PX_IMAGE_DIGEST=sha256:2700033ab42d0c6ccba5eb1a1990f4f56496d0697078b29957b914cbaf494091 PX_LOGLEVEL=info PX_RUNC=true PX_SHARED=/var/lib/kubelet:shared:1;/var/lib/osd:shared:1 PX_TEMPLATE_VERSION=v4 PX_VERSION=2.7.0.0-ccee71c STORK_SERVICE_PORT=tcp://10.100.102.48:8099 STORK_SERVICE_PORT_443_TCP=tcp://10.100.102.48:443 STORK_SERVICE_PORT_443_TCP_ADDR=10.100.102.48 STORK_SERVICE_PORT_443_TCP_PORT=443 STORK_SERVICE_PORT_443_TCP_PROTO=tcp STORK_SERVICE_PORT_8099_TCP=tcp://10.100.102.48:8099 STORK_SERVICE_PORT_8099_TCP_ADDR=10.100.102.48 STORK_SERVICE_PORT_8099_TCP_PORT=8099 STORK_SERVICE_PORT_8099_TCP_PROTO=tcp STORK_SERVICE_SERVICE_HOST=10.100.102.48 STORK_SERVICE_SERVICE_PORT=8099 STORK_SERVICE_SERVICE_PORT_EXTENDER=8099 STORK_SERVICE_SERVICE_PORT_WEBHOOK=443 TERM=xterm container=oci"
time="2021-04-16T09:11:45Z" level=info msg="/etc/systemd/system/portworx.socket content unchanged [e80e04204b7a7d113db36c53f420d635 /etc/systemd/system/portworx.socket]"
time="2021-04-16T09:11:45Z" level=info msg="/etc/systemd/system/portworx-output.service content unchanged [7340d8be39a32f3b7d296ac8275bc2e1 /etc/systemd/system/portworx-output.service]"
time="2021-04-16T09:11:45Z" level=info msg="/etc/systemd/system/portworx-reboot.service content unchanged [1dc97b965f3c6ad99aa3a92a02b2e8b1 /etc/systemd/system/portworx-reboot.service]"
time="2021-04-16T09:11:45Z" level=info msg="/etc/systemd/system/portworx.service content unchanged [4c6515ccf9c8cb3f81745795a981723c /etc/systemd/system/portworx.service]"
time="2021-04-16T09:11:45Z" level=info msg="runC spec got updated - restart pending"
time="2021-04-16T09:11:45Z" level=info msg="Portworx service restart required due to configuration update."
time="2021-04-16T09:11:45Z" level=warning msg="Reloading + Restarting portworx service"
time="2021-04-16T09:11:45Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx-reboot.service,id:0xc0000334e0}"
time="2021-04-16T09:11:45Z" level=info msg="Previous portworx.service started 17h42m11s ago"
time="2021-04-16T09:11:58Z" level=info msg="Activating node-watcher"
time="2021-04-16T09:11:58Z" level=info msg="Portworx service is ACTIVE"
time="2021-04-16T09:11:58Z" level=info msg="REST: Changing install-state: ST_INSTALL -> ST_FINISH"
time="2021-04-16T09:11:58Z" level=info msg="Start tailing portworx.service logs"
time="2021-04-16T09:11:59Z" level=info msg="> Starting local log-tailer"
time="2021-04-16T09:11:59Z" level=info msg="> run-local: /px-log-tail --follow -P @ -tf  -u portworx.service -u portworx-output.service -u init.scope -n 20000 -p 14072"
time="2021-04-16T09:11:59Z" level=info msg="Install done - MAIN exiting"
time="2021-04-16T09:11:59Z" level=info msg="-- Flushing logs for PID 14072 [20000 lines] --"
time="2021-04-16T09:11:59Z" level=info msg="-- Start tailing the logs for portworx.service, portworx-output.service, init.scope --"

eeginux · April 19, 2021, 8:01pm

@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Skipping reassembly as no px array config found
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXD version:  c41976af9efc5ae25e332fe3d9bff2922779878bÍ ï¿½
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Checking fs version...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Module version check: Success
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Done checking fs version...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Using cluster: px-cluster-15291395-1740-4f4c-86f6-34f411aff807
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Using kvdb device: type=gp2,size=150
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Warning skipping device exists check for: type=gp2,size=150.
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Using scheduler: kubernetes
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Clearing lttng tmpfs location: /var/lib/osd/lttng...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: ******************************************************************
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: ****** Checking mdraid0 layout path for null **************
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: ******************************************************************
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: patch_fs already done
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Checking sysfs mount...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: sysfs on /sys/firmware type sysfs (ro,relatime)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: sysfs mounted read-only. remounting...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: mapping:
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Setting portmap: 9001
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: px starting in IO Flusher mode
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: "bootstrap": true,
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:00,749 CRIT Supervisor running as root (no user in config file)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:00,754 INFO supervisord started with pid 1
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,756 INFO spawned: 'reboot-diags' with pid 241
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,759 INFO spawned: 'relayd' with pid 242
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,761 INFO spawned: 'cron' with pid 243
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,774 INFO spawned: 'px-etcd' with pid 244
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,806 INFO spawned: 'lttng' with pid 245
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,841 INFO spawned: 'exec' with pid 246
@ip-192-168-188-33.eu-west-1.compute.internal cron[14735]: (CRON) INFO (pidfile fd = 3)
@ip-192-168-188-33.eu-west-1.compute.internal cron[14735]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,879 INFO spawned: 'cache_mon' with pid 247
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,902 INFO spawned: 'px-diag' with pid 257
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,923 INFO spawned: 'px-healthmon' with pid 259
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,982 INFO spawned: 'pxdaemon' with pid 263
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:01,985 INFO spawned: 'px-ns' with pid 264
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:02,048 INFO spawned: 'px_event_listener' with pid 266
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:02,061 INFO exited: reboot-diags (exit status 0; expected)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Tracefile cleanup: Tracing disabled, remove all previous traces...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Clean out lttng tmpfs location: ...
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PX_STORAGE_IO_FLUSHER=yes
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Starting as an IOFlusher process : /usr/local/bin/start_pxcontroller_pxstorage.py
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Process with PID 263, is a IO Flusher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,055 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,055 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,055 INFO success: px-etcd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,055 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: cache_mon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: px-diag entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: px-healthmon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: px-ns entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:03,056 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:03Z" level=info msg="px-ns Starting.."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:03Z" level=info msg="InitPxClient No authentication enabled"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Installed NS trace handler for SIGHUP
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Installed NS sig-handler for SIGUSR1
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Installed NS sig-handler for SIGUSR2
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Starting NS server
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:07,187 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Tracing is disabled, not starting trace processes.
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px-storage with pid 341
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: connect: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: /dev/tcp/localhost/9009: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: px-storage not started yet...sleeping
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px with pid 352
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started watchdog with pid 353
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:10: PX-Watchdog: Starting watcher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:10: PX-Watchdog: Waiting for px process to start
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:10: PX-Watchdog: (pid 352): Begin monitoring
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Registering [kernel] as a volume driver"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Registered the Usage based Metering Agent...."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Setting log level to info(4)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="read config from env var" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="read config from config.json" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Alerts initialized successfully for this cluster"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Node is not yet initialized" func=setNodeInfo package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Generated a new NodeID: c9c2d384-bdb0-4765-aac1-0518f37f0470"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Using GW interface device:[eth0]..."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Detected Machine Hardware Type as: kvm (Virtual Machine)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:11Z" level=info msg="Bootstraping internal kvdb service." fn=kv-store.New id=c9c2d384-bdb0-4765-aac1-0518f37f0470
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:22Z" level=info msg="Starting kvdb on this node..." fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=c9c2d384-bdb0-4765-aac1-0518f37f0470
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:22Z" level=info msg="could not inspect instance group for AWS due to: AccessDenied: User: arn:aws:sts::322922525048:assumed-role/eks-first-nodes-NodeInstanceRole-K5RI1SOASSCE/i-0d52f9a1e74a36966 is not authorized to perform: autoscaling:DescribeAutoScalingGroups\n\tstatus code: 403, request id: 0a066301-bb0c-45b7-997f-d6fd12150c47"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:22Z" level=info msg="Detected provider AWS, Geo {AWS eu-west-1 eu-west-1a default default default default default default <nil>}"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:22Z" level=info msg="Made 1 pools"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:22Z" level=info msg="Benchmarking drive  /dev/sdf"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:33Z" level=info msg="fio: test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128\nfio-2.2.10\nStarting 1 process\n\ntest: (groupid=0, jobs=1): err= 0: pid=402: Fri Apr 16 09:12:33 2021\n  read : io=132388KB, bw=13212KB/s, iops=3303, runt= 10020msec\n    slat (usec): min=1, max=14960, avg=300.33, stdev=813.68\n    clat (msec): min=5, max=52, avg=38.44, stdev=11.33\n     lat (msec): min=5, max=52, avg=38.74, stdev=11.41\n    clat percentiles (usec):\n     |  1.00th=[ 6112],  5.00th=[ 6624], 10.00th=[ 7776], 20.00th=[39168],\n     | 30.00th=[41728], 40.00th=[41728], 50.00th=[42240], 60.00th=[42240],\n     | 70.00th=[42240], 80.00th=[42752], 90.00th=[44288], 95.00th=[48384],\n     | 99.00th=[50944], 99.50th=[51456], 99.90th=[51968], 99.95th=[51968],\n     | 99.99th=[51968]\n    bw (KB  /s): min=11800, max=35224, per=99.72%, avg=13175.15, stdev=5190.01\n    lat (msec) : 10=10.51%, 20=0.20%, 50=86.66%, 100=2.62%\n  cpu          : usr=0.62%, sys=1.58%, ctx=9275, majf=0, minf=137\n  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%\n     issued    : total=r=33097/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0\n     latency   : target=0, window=0, percentile=100.00%, depth=128\n\nRun status group 0 (all jobs):\n   READ: io=132388KB, aggrb=13212KB/s, minb=13212KB/s, maxb=13212KB/s, mint=10020msec, maxt=10020msec\n"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:33Z" level=info msg="Storage pool WriteThroughput 12 MB/s"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:33Z" level=info msg="Mounting metadata pool: Cos:LOW Medium:STORAGE_MEDIUM_SSD RaidLevel:\"raid0\" uuid:\"23a8b354-a0c5-481e-979c-3ef3fb5e4705\" "
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:33Z" level=info msg="HAL:Created volume:/var/.px/0/.reserve"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:33Z" level=info msg="Initializing journal: /var/.px/0/log"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="Mounting metadata pool: Cos:LOW Medium:STORAGE_MEDIUM_SSD RaidLevel:\"raid0\" uuid:\"23a8b354-a0c5-481e-979c-3ef3fb5e4705\" "
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="HAL:Created volume:/var/.px/0/.metadata"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="Applying labels to Pool 0"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="Sync mdpoolid=0,initinprogress,pxpool=0,mdvol labels to Pool 0"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="Created Metadata Volume: /var/.px/0/.metadata"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=info msg="Node (192.168.188.33) joining kvdb cluster." fn=kv-store.Init id=c9c2d384-bdb0-4765-aac1-0518f37f0470
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=error msg="cmd: /sbin/parted type=gp2,size=150 print output:nsenter: failed to execute parted: No such file or directory\n" Device="type=gp2,size=150" Error="exit status 1" Function=runParted
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=error msg="\r                                                                          \r" Device="type=gp2,size=150" Error="exit status 1" Function=HasPartitions
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=error msg="Unable to start internal kvdb on this node" err="failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1" fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=c9c2d384-bdb0-4765-aac1-0518f37f0470
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=error msg="failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1" func=InitAndBoot package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:34Z" level=error msg="Could not init boot manager" error="failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: px daemon exited with code: 1
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:35,284 INFO exited: pxdaemon (exit status 1; not expected)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:35,285 INFO spawned: 'pxdaemon' with pid 458
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PX_STORAGE_IO_FLUSHER=yes
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Starting as an IOFlusher process : /usr/local/bin/start_pxcontroller_pxstorage.py
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Process with PID 458, is a IO Flusher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:35,389 INFO reaped unknown pid 341
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px-storage with pid 498
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: connect: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: /dev/tcp/localhost/9009: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: px-storage not started yet...sleeping
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:36,548 INFO exited: px-etcd (exit status 1; not expected)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:37,551 INFO spawned: 'px-etcd' with pid 529
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px with pid 579
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started watchdog with pid 580
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:38: PX-Watchdog: Starting watcher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:38: PX-Watchdog: Waiting for px process to start
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:12:38: PX-Watchdog: (pid 579): Begin monitoring
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Registering [kernel] as a volume driver"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:39,306 INFO success: px-etcd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Registered the Usage based Metering Agent...."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Setting log level to info(4)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="read config from env var" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="read config from config.json" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Alerts initialized successfully for this cluster"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Node is not yet initialized" func=setNodeInfo package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Generated a new NodeID: dc3cbf30-60db-4824-b9d9-cb71e754572f"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Using GW interface device:[eth0]..."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Detected Machine Hardware Type as: kvm (Virtual Machine)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:39Z" level=info msg="Bootstraping internal kvdb service." fn=kv-store.New id=dc3cbf30-60db-4824-b9d9-cb71e754572f
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:12:40,409 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:47Z" level=info msg="Starting kvdb on this node..." fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=dc3cbf30-60db-4824-b9d9-cb71e754572f
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:47Z" level=info msg="could not inspect instance group for AWS due to: AccessDenied: User: arn:aws:sts::322922525048:assumed-role/eks-first-nodes-NodeInstanceRole-K5RI1SOASSCE/i-0d52f9a1e74a36966 is not authorized to perform: autoscaling:DescribeAutoScalingGroups\n\tstatus code: 403, request id: 964f0793-e65e-4bab-857e-441dfb24dc3f"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:47Z" level=info msg="Detected provider AWS, Geo {AWS eu-west-1 eu-west-1a default default default default default default <nil>}"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:47Z" level=info msg="Made 1 pools"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:47Z" level=info msg="Benchmarking drive  /dev/sdf"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:58Z" level=info msg="fio: test: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128\nfio-2.2.10\nStarting 1 process\n\ntest: (groupid=0, jobs=1): err= 0: pid=685: Fri Apr 16 09:12:58 2021\n  read : io=132112KB, bw=13193KB/s, iops=3298, runt= 10014msec\n    slat (usec): min=1, max=22267, avg=300.88, stdev=933.37\n    clat (msec): min=4, max=64, avg=38.49, stdev=11.51\n     lat (msec): min=5, max=64, avg=38.80, stdev=11.60\n    clat percentiles (usec):\n     |  1.00th=[ 6240],  5.00th=[ 6496], 10.00th=[ 7392], 20.00th=[38144],\n     | 30.00th=[41728], 40.00th=[42240], 50.00th=[42240], 60.00th=[42240],\n     | 70.00th=[42752], 80.00th=[42752], 90.00th=[46336], 95.00th=[49408],\n     | 99.00th=[51456], 99.50th=[51968], 99.90th=[52480], 99.95th=[54016],\n     | 99.99th=[56064]\n    bw (KB  /s): min=12000, max=35224, per=100.00%, avg=13228.58, stdev=5326.44\n    lat (msec) : 10=10.56%, 20=0.08%, 50=85.95%, 100=3.41%\n  cpu          : usr=0.56%, sys=1.55%, ctx=8707, majf=0, minf=136\n  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%\n     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%\n     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%\n     issued    : total=r=33028/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0\n     latency   : target=0, window=0, percentile=100.00%, depth=128\n\nRun status group 0 (all jobs):\n   READ: io=132112KB, aggrb=13192KB/s, minb=13192KB/s, maxb=13192KB/s, mint=10014msec, maxt=10014msec\n"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:58Z" level=info msg="Storage pool WriteThroughput 12 MB/s"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:58Z" level=info msg="Mounting metadata pool: Cos:LOW Medium:STORAGE_MEDIUM_SSD RaidLevel:\"raid0\" uuid:\"e9776116-d288-493b-b092-7b03a962b69e\" "
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:58Z" level=info msg="HAL:Created volume:/var/.px/0/.reserve"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:58Z" level=info msg="Initializing journal: /var/.px/0/log"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:59Z" level=info msg="Mounting metadata pool: Cos:LOW Medium:STORAGE_MEDIUM_SSD RaidLevel:\"raid0\" uuid:\"e9776116-d288-493b-b092-7b03a962b69e\" "
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:59Z" level=info msg="HAL:Created volume:/var/.px/0/.metadata"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:59Z" level=info msg="Applying labels to Pool 0"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:59Z" level=info msg="Sync pxpool=0,mdpoolid=0,initinprogress,mdvol labels to Pool 0"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:12:59Z" level=info msg="Created Metadata Volume: /var/.px/0/.metadata"
time="2021-04-16T09:12:59Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9001/v1/cluster/nodehealth: dial tcp 127.0.0.1:9001: connect: connection refused"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=info msg="Node (192.168.188.33) joining kvdb cluster." fn=kv-store.Init id=dc3cbf30-60db-4824-b9d9-cb71e754572f
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=error msg="cmd: /sbin/parted type=gp2,size=150 print output:nsenter: failed to execute parted: No such file or directory\n" Device="type=gp2,size=150" Error="exit status 1" Function=runParted
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=error msg="\r                                                                          \r" Device="type=gp2,size=150" Error="exit status 1" Function=HasPartitions
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=error msg="Unable to start internal kvdb on this node" err="failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1" fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=dc3cbf30-60db-4824-b9d9-cb71e754572f
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=error msg="failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1" func=InitAndBoot package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:00Z" level=error msg="Could not init boot manager" error="failed to setup internal kvdb: failed to provision internal kvdb: failed to check if kvdb device type=gp2,size=150 has partitions: exit status 1"

eeginux · April 19, 2021, 8:01pm

@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: px daemon exited with code: 1
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:01,318 INFO exited: pxdaemon (exit status 1; not expected)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:01,319 INFO spawned: 'pxdaemon' with pid 743
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PX_STORAGE_IO_FLUSHER=yes
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Starting as an IOFlusher process : /usr/local/bin/start_pxcontroller_pxstorage.py
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: Process with PID 743, is a IO Flusher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:01,389 INFO reaped unknown pid 498
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px-storage with pid 749
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: connect: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: bash: /dev/tcp/localhost/9009: Connection refused
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: px-storage not started yet...sleeping
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:02,693 INFO exited: px-etcd (exit status 1; not expected)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:03,696 INFO spawned: 'px-etcd' with pid 758
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started px with pid 768
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: PXPROCS[INFO]: Started watchdog with pid 769
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:13:04: PX-Watchdog: Starting watcher
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:13:04: PX-Watchdog: Waiting for px process to start
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16_ 9:13:04: PX-Watchdog: (pid 768): Begin monitoring
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Registering [kernel] as a volume driver"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:05,050 INFO success: px-etcd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Registered the Usage based Metering Agent...."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Setting log level to info(4)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="read config from env var" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="read config from config.json" func=init package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Alerts initialized successfully for this cluster"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Node is not yet initialized" func=setNodeInfo package=boot
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Generated a new NodeID: 2afc8949-46c2-4f4e-880c-5925470a4ab2"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Using GW interface device:[eth0]..."
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Detected Machine Hardware Type as: kvm (Virtual Machine)"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:05Z" level=info msg="Bootstraping internal kvdb service." fn=kv-store.New id=2afc8949-46c2-4f4e-880c-5925470a4ab2
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:07,135 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
time="2021-04-16T09:13:09Z" level=warning msg="Could not retrieve PX node status" error="Get http://127.0.0.1:9001/v1/cluster/nodehealth: dial tcp 127.0.0.1:9001: connect: connection refused"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: 2021-04-16 09:13:13,431 INFO reaped unknown pid 363
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:14Z" level=info msg="Starting kvdb on this node..." fn=kvdb-provisioner.ProvisionKvdbWithoutLock id=2afc8949-46c2-4f4e-880c-5925470a4ab2
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:14Z" level=info msg="could not inspect instance group for AWS due to: AccessDenied: User: arn:aws:sts::322922525048:assumed-role/eks-first-nodes-NodeInstanceRole-K5RI1SOASSCE/i-0d52f9a1e74a36966 is not authorized to perform: autoscaling:DescribeAutoScalingGroups\n\tstatus code: 403, request id: 3b57102d-89ba-4ad4-9d4c-56b134f0ba8b"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:14Z" level=info msg="Detected provider AWS, Geo {AWS eu-west-1 eu-west-1a default default default default default default <nil>}"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:14Z" level=info msg="Made 1 pools"
@ip-192-168-188-33.eu-west-1.compute.internal portworx[13985]: time="2021-04-16T09:13:14Z" level=info msg="Benchmarking drive  /dev/sdf"

eeginux · April 19, 2021, 8:02pm

Attached the portworx container log here. split into three replies due to size limit

sensre · April 25, 2021, 12:35am

I see the problem in your spec yaml file, you are using both -a and asking to provision the drive-by portworx itself "-kvdb_dev", "type=gp2,size=150" . its kind of mixed-mode, for that you need to prepare the cluster as per this docs page and provide the IAM roles and policies or security credentials to work . Install Portworx on AWS ASG using the DaemonSet . you may want to clean everything and retry it.

eeginux · May 2, 2021, 3:19pm

When creating the spec with spec generator, there are three options,

create use a spec
consume unused
using existing disks

I understand ‘create use a spec’ need to follow the provided link. but i was using the other two options. Do i still need to follow the link? Could you please tell me which to choose in the following scenario,

eks nodes with default ebs volume
eks nodes with default ebs volume and one extra volume

sensre · May 26, 2021, 2:36am

Sorry for the delay. you have to use the same link. and if you provisioned or eks node comes with extra volume, you can use them. but those ebs volumes should not be mounted or formatted .

Topic		Replies	Views
Storage device failed size check Portworx on Kubernetes	6	1415	January 9, 2020
Reinstall of Portworx after removing KVDB disks Portworx Install	1	621	February 6, 2021
Failed to provision internal kvdb: Make sure the drives are cleaned up before reinstalling on this node." Portworx Essentials	1	1168	October 13, 2020
ProviderInternal Error: InvalidVolume.NotFound Portworx on Kubernetes	5	1051	February 15, 2021
Create PVC Fails with KVDB Connection Failed Message Portworx Essentials	0	510	November 9, 2021

Unhealthy portworx container

Related Topics