Portworx Could not talk to Docker/Containerd/CRI

mr.io · February 5, 2021, 9:06pm

I am trying to install Portworx operator on K3s, and I am getting this error:

portworx  Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted

K3s uses containerd, located at:

/run/k3s/containerd/containerd.sock

How do I resolve this, please? Is it possible to point portworx to /run/k3s/containerd/containerd.sock instead of /run/containerd/containerd.sock, if so, how?

 $ k get nodes -o wide
NAME              STATUS   ROLES                  AGE     VERSION            INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-10-97-0-212    Ready    control-plane,master   7h38m   v1.20.2-rc1+k3s1   10.97.0.212    <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-3-85     Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.3.85     <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-13-113   Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.13.113   <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-7-111    Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.7.111    <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1

sensre · February 6, 2021, 2:55am

By any chance, are you using rancher k3s clusters?

meanwhile, get your cluster details to debug further

List Portworx pods:

kubectl get pods -l name=portworx -n kube-system -o wide

• Describe Portworx pods:

kubectl describe pods -l name=portworx -n kube-system

• Get Portworx cluster status:

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status

• List Portworx volumes:

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list

• Portworx logs: Recent Portworx logs can be gathered by using this kubectl command:

kubectl logs -n kube-system -l name=portworx --tail=99999

mr.io · February 6, 2021, 11:33am

Yes I am using Rancher k3s, @sensre

kubectl get pods -l name=portworx -n kube-system -o wide

NAME                                                    READY   STATUS             RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx   1/2     CrashLoopBackOff   1          33s   10.97.13.113   ip-10-97-13-113   <none>           <none>
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5   1/2     CrashLoopBackOff   1          32s   10.97.7.111    ip-10-97-7-111    <none>           <none>
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6   1/2     CrashLoopBackOff   1          32s   10.97.3.85     ip-10-97-3-85     <none>           <none>

kubectl describe pods -l name=portworx -n kube-system
output: 
https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-describe_pod-out

I am unable to to get the Portworx cluster status because the pod is obviously not running… see the Gist above.

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status
~ $ kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6 -n kube-system' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("portworx")

List Portworx volumes:

$ PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
~ $ kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6 -n kube-system' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("portworx")

kubectl logs -n kube-system -l name=portworx --tail=99999

error: a container name must be specified for pod px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx, choose one of: [portworx csi-node-driver-registrar]

This command failed because there are two containers in the pod. So I specified the container as shown below:

kubectl logs -c portworx  -n kube-system -l name=portworx --tail=99999

output:

gist.github.com

https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-container_logs_portworx-out

container_logs_portworx.out

 kubectl logs -c portworx  -n kube-system -l name=portworx --tail=99999


time="2021-02-06T11:07:32Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:07:32Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:07:32Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:07:32Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:07:32Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0004e62e0}"
time="2021-02-06T11:07:32Z" level=info msg="Activating REST server"
time="2021-02-06T11:07:32Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"

This file has been truncated. show original

describe_pod.out

Name:         px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6
Namespace:    kube-system
Priority:     0
Node:         ip-10-97-3-85/10.97.3.85
Start Time:   Sat, 06 Feb 2021 10:50:48 +0000
Labels:       controller-revision-hash=7499b67c55
              name=portworx
              operator.libopenstorage.org/driver=portworx
              operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: true

This file has been truncated. show original

I added a description of the pods here

 k describe po px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5 -n kube-system
output:
https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-describe_pod-out

Lastly, the result of get events:

k get ev -n kube-system
output:

gist.github.com

https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-get-events-out

container_logs_portworx.out

 kubectl logs -c portworx  -n kube-system -l name=portworx --tail=99999


time="2021-02-06T11:07:32Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:07:32Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:07:32Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:07:32Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:07:32Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0004e62e0}"
time="2021-02-06T11:07:32Z" level=info msg="Activating REST server"
time="2021-02-06T11:07:32Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"

This file has been truncated. show original

describe-po.out

Name:         px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5
Namespace:    kube-system
Priority:     0
Node:         ip-10-97-7-111/10.97.7.111
Start Time:   Sat, 06 Feb 2021 10:50:48 +0000
Labels:       controller-revision-hash=7499b67c55
              name=portworx
              operator.libopenstorage.org/driver=portworx
              operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: true

This file has been truncated. show original

describe_pod.out

Name:         px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6
Namespace:    kube-system
Priority:     0
Node:         ip-10-97-3-85/10.97.3.85
Start Time:   Sat, 06 Feb 2021 10:50:48 +0000
Labels:       controller-revision-hash=7499b67c55
              name=portworx
              operator.libopenstorage.org/driver=portworx
              operator.libopenstorage.org/name=px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9
Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: true

This file has been truncated. show original

There are more than three files. show original

mr.io · February 6, 2021, 12:13pm

FWIW, I decided to create a symb link to the location that portworx expects the containerd.sock to be located on all my nodes (master and worker nodes inclusive) as shown below:

$ sudo ln -s /run/k3s/containerd/containerd.sock /run/containerd/containerd.sock

verified: 
$ sudo ls -ltr  /run/containerd/containerd.sock

lrwxrwxrwx 1 root root 35 Feb  6 11:49 /run/containerd/containerd.sock -> /run/k3s/containerd/containerd.sock

Then I deleted the cluster pods, with the hope that the new containers should mount the path…

kubectl delete  pods -l name=portworx -n kube-system  --force --grace-period 0

but it failed with the same error message …

~ $ k logs px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-cpr4k -n kube-system -c portworx
time="2021-02-06T11:51:59Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:51:59Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:51:59Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:51:59Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:51:59Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0004f1600}"
time="2021-02-06T11:51:59Z" level=info msg="Activating REST server"
time="2021-02-06T11:51:59Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-02-06T11:51:59Z" level=info msg="Locating my container handler"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use Docker as container handler failed" error="/var/run/docker.sock not a socket-file"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use ContainerD as container handler failed" error="stat /run/containerd/containerd.sock: no such file or directory"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use k8s-CRI as container handler failed" error="CRI-Socket path not specified"
time="2021-02-06T11:51:59Z" level=error msg="Could not instantiate container client" error="Could not initialize container handler"
time="2021-02-06T11:51:59Z" level=error msg="Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted."

```

sensre · February 6, 2021, 1:10pm

Got it, let me verify the k3s + containerd qualification and validate it internally.

mr.io · February 12, 2021, 9:56am

Hey @sensre, do you have an update on this?

sensre · February 15, 2021, 4:46pm

yes, the YAML should include -v /run/k3s/containerd/containerd.sock:/run/containerd/containerd.sock override together with the “standard” -v /run/containerd:/run/containerd – example:: https://install.portworx.com/2.6?mc=false&kbver=1.20.2%2Bk3s1&k=etcd%3Ahttp%3A%2F%2F70.[…]ter-e1454e17-6b01-40d2-a03a-7a3fe6786b9c&stork=true&st=k8s
Provide precise kubernetes version when generating YAML (version must be copied strictly “as is” - e.g.

kubectl version --short | awk -Fv ‘/Server Version: / {print $3}’
1.20.2+k3s1

… our YAML-generation will use the version’s extension ( +k3s1 ) to add the appropriate tweaks into the YAML. These K3S-specific tweaks are currently missing in your deployment, which is likely root-cause for PX deployment failure.