mm-sa
August 24, 2023, 6:59pm
1
kubectl describe pods -l name=portworx -n kube-system Output attached
Name: dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d-jjzln
Namespace: kube-system
Priority: 0
Node: dru-storage03aio3/10.200.99.64
Start Time: Thu, 24 Aug 2023 13:24:07 -0500
Labels: controller-revision-hash=db96bf466
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","cattle.io/creator":"norman","kubernetes.io/arch":"amd64","kubernetes.i...
Status: Running
IP: 10.200.99.64
IPs:
IP: 10.200.99.64
Controlled By: StorageCluster/dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Containers:
portworx:
Container ID: docker://0059f02214ceb9c81fbc44a1072685c6c2dd0f88233e6e7705792e96feaaa1d2
Image: docker.io/portworx/oci-monitor:3.0.0
Image ID: docker-pullable://portworx/oci-monitor@sha256:80a0586cf5b6b044120f673b7e6c224043ea7c22d7386a8742f6251c8330fe81
Port: <none>
Host Port: <none>
Args:
-c
dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
-x
kubernetes
-b
-d
ens192
-m
ens160
-s
size=150
-max_storage_nodes_per_zone
3
-secret_type
k8s
-rt_opts
default-io-profile=6
--oem
esse
State: Running
Started: Thu, 24 Aug 2023 13:39:38 -0500
Last State: Terminated
Reason: Error
Message: PX stopped working 8.8s ago. Last status: Could not init boot manager (error="error loading node identity: Cause: ProviderInternal Error: drive set '3fb57a37-f742-421a-a21c-879257de57fc' is locked by other node 'dru-storage03aio1' which is not us, not locking...")
Exit Code: 2
Started: Thu, 24 Aug 2023 13:24:08 -0500
Finished: Thu, 24 Aug 2023 13:39:37 -0500
Ready: False
Restart Count: 1
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
NODE_NAME: (v1:spec.nodeName)
PX_TEMPLATE_VERSION: v4
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PURE_FLASHARRAY_SAN_TYPE: ISCSI
Mounts:
/etc/ccm from ccm-phonehome-config (rw)
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerddir (rw)
/var/cores from diagsdump (rw)
/var/lib/containerd from containerdvardir (rw)
/var/lib/osd from varlibosd (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6wq5g (ro)
csi-node-driver-registrar:
Container ID: docker://cb4e90d96ae7d9fdc6a21d122334b30f52a273646c27a9bb5b1ce77eac742086
Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2
Image ID: docker-pullable://registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:a13bff2ed69af0cf4270f0cf47bdedf75a56c095cd95b91195ae6c713a9b1845
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Thu, 24 Aug 2023 13:24:08 -0500
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6wq5g (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerddir:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
containerdvardir:
Type: HostPath (bare host directory volume)
Path: /var/lib/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
varlibosd:
Type: HostPath (bare host directory volume)
Path: /var/lib/osd
HostPathType:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
varcache:
Type: HostPath (bare host directory volume)
Path: /var/cache
HostPathType:
timezone:
Type: HostPath (bare host directory volume)
Path: /etc/timezone
HostPathType:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
ccm-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-config
Optional: false
ccm-phonehome-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-phonehome
Optional: false
kube-api-access-6wq5g:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 27m kubelet Pulling image "docker.io/portworx/oci-monitor:3.0.0"
Normal Pulled 27m kubelet Successfully pulled image "docker.io/portworx/oci-monitor:3.0.0" in 283.050415ms (283.05476ms including waiting)
Normal Created 27m kubelet Created container portworx
Normal Started 27m kubelet Started container portworx
Normal Pulling 27m kubelet Pulling image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2"
Normal Pulled 27m kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2" in 303.055803ms (303.098964ms including waiting)
Normal Created 27m kubelet Created container csi-node-driver-registrar
Normal Started 27m kubelet Started container csi-node-driver-registrar
Normal PortworxMonitorImagePullInProgress 27m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Warning NodeStartFailure 25m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set '520679b4-ae54-47e4-b695-896394eceb88' is locked by other node 'dru-storage03aio1' which is not us, not locking...
Warning NodeStartFailure 24m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'efb51fc9-2825-4f3f-b7a4-e0d174728977' is locked by other node 'dru-storage03aio4' which is not us, not locking...
Warning NodeStartFailure 22m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume 5fd60442ef to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa6c on the host: <nil>
Warning NodeStartFailure 20m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume 6c59da7521 to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa6d on the host: <nil>
Warning NodeStartFailure 19m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume bc2250e3e0 to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa6f on the host: <nil>
Warning NodeStartFailure 17m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume 8cb8eb5576 to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa71 on the host: <nil>
Warning NodeStartFailure 15m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume 7f31856295 to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa73 on the host: <nil>
Warning NodeStartFailure 13m portworx Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume 93452bf05d to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa75 on the host: <nil>
Warning NodeStartFailure 12m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set '3fb57a37-f742-421a-a21c-879257de57fc' is locked by other node 'dru-storage03aio1' which is not us, not locking...
Normal PortworxMonitorImagePullInProgress 12m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Warning InternalKvdbSetupFailed 5m40s portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1
Warning Unhealthy 2m33s (x176 over 27m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning NodeStartFailure 63s (x7 over 11m) portworx (combined from similar events): Failed to start Portworx: failed to generate a new node identity: Cause: ProviderInternal Error: failed to convert path /dev/dm-1 for volume c86e689bcf to mapper path: failed to find path /dev/mapper/3624a93704d435e72b56640600007aa80 on the host: <nil>
mm-sa
August 24, 2023, 7:00pm
2
Additional output:
Name: dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d-rt4bb
Namespace: kube-system
Priority: 0
Node: dru-storage03aio1/10.200.99.65
Start Time: Thu, 24 Aug 2023 13:24:05 -0500
Labels: controller-revision-hash=db96bf466
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","cattle.io/creator":"norman","kubernetes.io/arch":"amd64","kubernetes.i...
Status: Running
IP: 10.200.99.65
IPs:
IP: 10.200.99.65
Controlled By: StorageCluster/dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Containers:
portworx:
Container ID: docker://662d3e88fba575d8d9ef9b1593a02ed6a20312a21a748eaab76928f94572085c
Image: docker.io/portworx/oci-monitor:3.0.0
Image ID: docker-pullable://portworx/oci-monitor@sha256:80a0586cf5b6b044120f673b7e6c224043ea7c22d7386a8742f6251c8330fe81
Port: <none>
Host Port: <none>
Args:
-c
dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
-x
kubernetes
-b
-d
ens192
-m
ens160
-s
size=150
-max_storage_nodes_per_zone
3
-secret_type
k8s
-rt_opts
default-io-profile=6
--oem
esse
State: Running
Started: Thu, 24 Aug 2023 13:39:37 -0500
Last State: Terminated
Reason: Error
Message: PX stopped working 9.3s ago. Last status: Could not init boot manager (error="failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1")
Exit Code: 2
Started: Thu, 24 Aug 2023 13:24:07 -0500
Finished: Thu, 24 Aug 2023 13:39:36 -0500
Ready: False
Restart Count: 1
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
NODE_NAME: (v1:spec.nodeName)
PX_TEMPLATE_VERSION: v4
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
PURE_FLASHARRAY_SAN_TYPE: ISCSI
Mounts:
/etc/ccm from ccm-phonehome-config (rw)
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerddir (rw)
/var/cores from diagsdump (rw)
/var/lib/containerd from containerdvardir (rw)
/var/lib/osd from varlibosd (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jtntj (ro)
csi-node-driver-registrar:
Container ID: docker://acce6da8ec5e5b16761695dffc081c02f0666d733f7f8dfb775bab0fa655bf19
Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2
Image ID: docker-pullable://registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:a13bff2ed69af0cf4270f0cf47bdedf75a56c095cd95b91195ae6c713a9b1845
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Thu, 24 Aug 2023 13:24:08 -0500
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jtntj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerddir:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
containerdvardir:
Type: HostPath (bare host directory volume)
Path: /var/lib/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
varlibosd:
Type: HostPath (bare host directory volume)
Path: /var/lib/osd
HostPathType:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
varcache:
Type: HostPath (bare host directory volume)
Path: /var/cache
HostPathType:
timezone:
Type: HostPath (bare host directory volume)
Path: /etc/timezone
HostPathType:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
ccm-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-config
Optional: false
ccm-phonehome-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-phonehome
Optional: false
kube-api-access-jtntj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 27m kubelet Pulling image "docker.io/portworx/oci-monitor:3.0.0"
Normal Pulled 27m kubelet Successfully pulled image "docker.io/portworx/oci-monitor:3.0.0" in 323.930779ms (344.39117ms including waiting)
Normal Created 27m kubelet Created container portworx
Normal Started 27m kubelet Started container portworx
Normal Pulling 27m kubelet Pulling image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2"
Normal Pulled 27m kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2" in 297.271986ms (297.276559ms including waiting)
Normal Created 27m kubelet Created container csi-node-driver-registrar
Normal PortworxMonitorImagePullInProgress 27m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Normal Started 27m kubelet Started container csi-node-driver-registrar
Warning InternalKvdbSetupFailed 22m portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 71259fa9-2cbe-402c-9699-6d61ab5a1c3e, dev path /dev/mapper/3624a93704d435e72b56640600007aa6a at /var/.px/0: error invalid argument
Warning NodeStartFailure 22m portworx Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 71259fa9-2cbe-402c-9699-6d61ab5a1c3e, dev path /dev/mapper/3624a93704d435e72b56640600007aa6a at /var/.px/0: error invalid argument
Warning NodeStartFailure 17m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'b1b50af4-61f1-4da8-a115-a8e02d731bbc' is locked by other node 'dru-storage03aio4' which is not us, not locking...
Normal PortworxMonitorImagePullInProgress 12m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Warning NodeStartFailure 7m12s portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set '57a7df79-382e-4d3e-9b01-92043da09b01' is locked by other node 'dru-storage03aio4' which is not us, not locking...
Warning InternalKvdbSetupFailed 2m41s portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 09f20316-37f9-4bbf-8444-43c147ff269a, dev path /dev/mapper/3624a93704d435e72b56640600007aa7e at /var/.px/0: error invalid argument
Warning NodeStartFailure 2m37s portworx Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 09f20316-37f9-4bbf-8444-43c147ff269a, dev path /dev/mapper/3624a93704d435e72b56640600007aa7e at /var/.px/0: error invalid argument
Warning Unhealthy 2m34s (x177 over 27m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning InternalKvdbSetupFailed 19s (x6 over 20m) portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1
Warning NodeStartFailure 15s (x6 over 20m) portworx Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1
Name: dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d-xbhj7
Namespace: kube-system
Priority: 0
Node: dru-storage03aio4/10.200.99.66
Start Time: Thu, 24 Aug 2023 13:24:07 -0500
Labels: controller-revision-hash=db96bf466
name=portworx
operator.libopenstorage.org/driver=portworx
operator.libopenstorage.org/name=dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
operator.libopenstorage.org/node-labels:
{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","cattle.io/creator":"norman","kubernetes.io/arch":"amd64","kubernetes.i...
Status: Running
IP: 10.200.99.66
IPs:
IP: 10.200.99.66
Controlled By: StorageCluster/dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
Containers:
portworx:
Container ID: docker://ef9775eee8d3faacc6d834deb984a4e51a493aa3cdebaeea2234072140eb41bc
Image: docker.io/portworx/oci-monitor:3.0.0
Image ID: docker-pullable://portworx/oci-monitor@sha256:80a0586cf5b6b044120f673b7e6c224043ea7c22d7386a8742f6251c8330fe81
Port: <none>
Host Port: <none>
Args:
-c
dr-storage03-1d06456b-62c5-41af-abdf-582b6f31de8d
-x
kubernetes
-b
-d
ens192
-m
ens160
-s
size=150
-max_storage_nodes_per_zone
3
-secret_type
k8s
-rt_opts
default-io-profile=6
--oem
esse
State: Running
Started: Thu, 24 Aug 2023 13:39:38 -0500
Last State: Terminated
Reason: Error
Message: PX stopped working 3.2s ago. Last status: Could not init boot manager (error="error loading node identity: Cause: ProviderInternal Error: drive set 'f2ccf06c-b33a-48eb-8cf1-44145674c237' is locked by other node 'dru-storage03aio1' which is not us, not locking...")
Exit Code: 2
Started: Thu, 24 Aug 2023 13:24:08 -0500
Finished: Thu, 24 Aug 2023 13:39:37 -0500
Ready: False
Restart Count: 1
Liveness: http-get http://127.0.0.1:9001/status delay=840s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://127.0.0.1:9015/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
PURE_FLASHARRAY_SAN_TYPE: ISCSI
PX_NAMESPACE: kube-system
PX_SECRETS_NAMESPACE: kube-system
NODE_NAME: (v1:spec.nodeName)
PX_TEMPLATE_VERSION: v4
CSI_ENDPOINT: unix:///var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
Mounts:
/etc/ccm from ccm-phonehome-config (rw)
/etc/crictl.yaml from crioconf (rw)
/etc/pwx from etcpwx (rw)
/etc/systemd/system from sysdmount (rw)
/host_proc from procmount (rw)
/opt/pwx from optpwx (rw)
/run/containerd from containerddir (rw)
/var/cores from diagsdump (rw)
/var/lib/containerd from containerdvardir (rw)
/var/lib/osd from varlibosd (rw)
/var/log from journalmount2 (ro)
/var/run/crio from criosock (rw)
/var/run/dbus from dbusmount (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/log from journalmount1 (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dnsg (ro)
csi-node-driver-registrar:
Container ID: docker://e288e7055a19c3f7bf755d2f78186b353dfa0d210dff05a024a27aad228844d9
Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2
Image ID: docker-pullable://registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:a13bff2ed69af0cf4270f0cf47bdedf75a56c095cd95b91195ae6c713a9b1845
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=/var/lib/kubelet/plugins/pxd.portworx.com/csi.sock
State: Running
Started: Thu, 24 Aug 2023 13:24:09 -0500
Ready: True
Restart Count: 0
Environment:
ADDRESS: /csi/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
Mounts:
/csi from csi-driver-path (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8dnsg (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
containerddir:
Type: HostPath (bare host directory volume)
Path: /run/containerd
HostPathType:
containerdvardir:
Type: HostPath (bare host directory volume)
Path: /var/lib/containerd
HostPathType:
criosock:
Type: HostPath (bare host directory volume)
Path: /var/run/crio
HostPathType:
crioconf:
Type: HostPath (bare host directory volume)
Path: /etc/crictl.yaml
HostPathType: FileOrCreate
optpwx:
Type: HostPath (bare host directory volume)
Path: /opt/pwx
HostPathType:
procmount:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType:
sysdmount:
Type: HostPath (bare host directory volume)
Path: /etc/systemd/system
HostPathType:
dbusmount:
Type: HostPath (bare host directory volume)
Path: /var/run/dbus
HostPathType:
varlibosd:
Type: HostPath (bare host directory volume)
Path: /var/lib/osd
HostPathType:
diagsdump:
Type: HostPath (bare host directory volume)
Path: /var/cores
HostPathType:
etcpwx:
Type: HostPath (bare host directory volume)
Path: /etc/pwx
HostPathType:
journalmount1:
Type: HostPath (bare host directory volume)
Path: /var/run/log
HostPathType:
journalmount2:
Type: HostPath (bare host directory volume)
Path: /var/log
HostPathType:
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: DirectoryOrCreate
csi-driver-path:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/pxd.portworx.com
HostPathType: DirectoryOrCreate
varcache:
Type: HostPath (bare host directory volume)
Path: /var/cache
HostPathType:
timezone:
Type: HostPath (bare host directory volume)
Path: /etc/timezone
HostPathType:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
ccm-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-config
Optional: false
ccm-phonehome-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: px-telemetry-phonehome
Optional: false
kube-api-access-8dnsg:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 27m kubelet Pulling image "docker.io/portworx/oci-monitor:3.0.0"
Normal Created 27m kubelet Created container csi-node-driver-registrar
Normal Pulled 27m kubelet Successfully pulled image "docker.io/portworx/oci-monitor:3.0.0" in 306.643819ms (457.442197ms including waiting)
Normal Created 27m kubelet Created container portworx
Normal Started 27m kubelet Started container portworx
Normal Pulled 27m kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2" in 324.262006ms (324.266014ms including waiting)
Normal PortworxMonitorImagePullInProgress 27m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Normal Pulling 27m kubelet Pulling image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.2"
Normal Started 27m kubelet Started container csi-node-driver-registrar
Warning NodeStartFailure 20m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'b5c5e34c-f967-4b85-83b3-de6b0e2ef490' is locked by other node 'dru-storage03aio1' which is not us, not locking...
Warning NodeStartFailure 18m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'f40f7081-865c-44e3-bbf0-ffd1ef5bd5d7' is locked by other node 'dru-storage03aio1' which is not us, not locking...
Warning NodeStartFailure 14m portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'f2ccf06c-b33a-48eb-8cf1-44145674c237' is locked by other node 'dru-storage03aio1' which is not us, not locking...
Normal PortworxMonitorImagePullInProgress 12m portworx Portworx image docker.io/portworx/px-essentials:3.0.0 pull and extraction in progress
Warning InternalKvdbSetupFailed 9m27s portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 7d446458-5222-4a09-89fe-dab09a7e902b, dev path /dev/mapper/3624a93704d435e72b56640600007aa77 at /var/.px/0: error invalid argument
Warning NodeStartFailure 9m23s portworx Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: failed to mount device UUID 7d446458-5222-4a09-89fe-dab09a7e902b, dev path /dev/mapper/3624a93704d435e72b56640600007aa77 at /var/.px/0: error invalid argument
Warning NodeStartFailure 5m41s portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'd18491b1-1e18-4070-9ebd-1a1cfa66b12e' is locked by other node 'dru-storage03aio3' which is not us, not locking...
Warning Unhealthy 2m33s (x176 over 27m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Warning InternalKvdbSetupFailed 2m (x4 over 21m) portworx failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1
Warning NodeStartFailure 118s (x4 over 21m) portworx Failed to start Portworx: failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: exit status 1
Warning NodeStartFailure 32s portworx Failed to start Portworx: error loading node identity: Cause: ProviderInternal Error: drive set 'c9efdd41-1ed1-4af0-a5f8-e97bff361e8f' is locked by other node 'dru-storage03aio1' which is not us, not locking...
mm-sa
August 24, 2023, 9:20pm
3
After attempting a new cluster w/ Ubuntu LTS 22.04 I am now receiving the following error when executing pxctl status
NODE InternalKvdbSetupFailed a3bef04a-0a24-4bb0-b57d-7a3369b877fc ALARM 1 Aug 24 21:14:45 UTC 2023 Aug 24 21:14:45 UTC 2023 failed to setup internal kvdb: failed to provision internal kvdb: failed in initializing drives on this node: Failed to format [-f --nodiscard /dev/mapper/3624a93704d435e72b56640600007aeaf]: No valid Btrfs found on /dev/mapper/3624a93704d435e72b56640600007aeaf