Portworx Could not talk to Docker/Containerd/CRI

I am trying to install Portworx operator on K3s, and I am getting this error:

portworx  Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted

K3s uses containerd, located at:

/run/k3s/containerd/containerd.sock

How do I resolve this, please? Is it possible to point portworx to /run/k3s/containerd/containerd.sock instead of /run/containerd/containerd.sock, if so, how?

 $ k get nodes -o wide
NAME              STATUS   ROLES                  AGE     VERSION            INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-10-97-0-212    Ready    control-plane,master   7h38m   v1.20.2-rc1+k3s1   10.97.0.212    <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-3-85     Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.3.85     <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-13-113   Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.13.113   <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1
ip-10-97-7-111    Ready    <none>                 7h33m   v1.20.2-rc1+k3s1   10.97.7.111    <none>        Ubuntu 18.04.3 LTS   4.15.0-1054-aws   containerd://1.4.3-k3s1

By any chance, are you using rancher k3s clusters?

meanwhile, get your cluster details to debug further

List Portworx pods:

kubectl get pods -l name=portworx -n kube-system -o wide
• Describe Portworx pods:

kubectl describe pods -l name=portworx -n kube-system

• Get Portworx cluster status:

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status

• List Portworx volumes:

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list

• Portworx logs: Recent Portworx logs can be gathered by using this kubectl command:

kubectl logs -n kube-system -l name=portworx --tail=99999

Yes I am using Rancher k3s, @sensre

kubectl get pods -l name=portworx -n kube-system -o wide

NAME                                                    READY   STATUS             RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx   1/2     CrashLoopBackOff   1          33s   10.97.13.113   ip-10-97-13-113   <none>           <none>
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5   1/2     CrashLoopBackOff   1          32s   10.97.7.111    ip-10-97-7-111    <none>           <none>
px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6   1/2     CrashLoopBackOff   1          32s   10.97.3.85     ip-10-97-3-85     <none>           <none>
kubectl describe pods -l name=portworx -n kube-system
output: 
https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-describe_pod-out

I am unable to to get the Portworx cluster status because the pod is obviously not running… see the Gist above.

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status
~ $ kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6 -n kube-system' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("portworx")

List Portworx volumes:

$ PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
~ $ kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-mfxs6 -n kube-system' to see all of the containers in this pod.
error: unable to upgrade connection: container not found ("portworx")
kubectl logs -n kube-system -l name=portworx --tail=99999

error: a container name must be specified for pod px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-q94sx, choose one of: [portworx csi-node-driver-registrar]

This command failed because there are two containers in the pod. So I specified the container as shown below:

kubectl logs -c portworx  -n kube-system -l name=portworx --tail=99999

output:

I added a description of the pods here

 k describe po px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-nrjq5 -n kube-system
output:
https://gist.github.com/iogbole/6857fbf846dad6823de2bdb586ed1aa6#file-describe_pod-out

Lastly, the result of get events:

k get ev -n kube-system
output:

FWIW, I decided to create a symb link to the location that portworx expects the containerd.sock to be located on all my nodes (master and worker nodes inclusive) as shown below:

$ sudo ln -s /run/k3s/containerd/containerd.sock /run/containerd/containerd.sock

verified: 
$ sudo ls -ltr  /run/containerd/containerd.sock

lrwxrwxrwx 1 root root 35 Feb  6 11:49 /run/containerd/containerd.sock -> /run/k3s/containerd/containerd.sock

Then I deleted the cluster pods, with the hope that the new containers should mount the path…

kubectl delete  pods -l name=portworx -n kube-system  --force --grace-period 0

but it failed with the same error message …

~ $ k logs px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9-cpr4k -n kube-system -c portworx
time="2021-02-06T11:51:59Z" level=info msg="Input arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s --oem esse"
time="2021-02-06T11:51:59Z" level=info msg="Updated arguments: /px-oci-mon -c px-cluster-03732d57-45f2-48a3-80a2-97a0e48e3ad9 -x kubernetes -b -A -f -j auto -secret_type k8s"
time="2021-02-06T11:51:59Z" level=info msg="OCI-Monitor computed version v2.6.3-ge63b7d82-dirty"
time="2021-02-06T11:51:59Z" level=info msg="REAPER: Starting ..."
time="2021-02-06T11:51:59Z" level=info msg="Service handler initialized via as DBus{type:dbus,svc:portworx.service,id:0xc0004f1600}"
time="2021-02-06T11:51:59Z" level=info msg="Activating REST server"
time="2021-02-06T11:51:59Z" level=info msg="> run-host: /bin/sh -c cat /etc/crictl.yaml || cat /var/vcap/store/crictl.yaml"
time="2021-02-06T11:51:59Z" level=info msg="Locating my container handler"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use Docker as container handler failed" error="/var/run/docker.sock not a socket-file"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use ContainerD as container handler failed" error="stat /run/containerd/containerd.sock: no such file or directory"
time="2021-02-06T11:51:59Z" level=info msg="> Attempt to use k8s-CRI as container handler failed" error="CRI-Socket path not specified"
time="2021-02-06T11:51:59Z" level=error msg="Could not instantiate container client" error="Could not initialize container handler"
time="2021-02-06T11:51:59Z" level=error msg="Could not talk to Docker/Containerd/CRI: Could not initialize container handler - please ensure '/var/run/docker.sock', '/run/containerd/containerd.sock' or '/var/run/crio/crio.sock' are mounted."

```

Got it, let me verify the k3s + containerd qualification and validate it internally.

Hey @sensre, do you have an update on this?

yes, the YAML should include -v /run/k3s/containerd/containerd.sock:/run/containerd/containerd.sock override together with the “standard” -v /run/containerd:/run/containerd – example:: https://install.portworx.com/2.6?mc=false&kbver=1.20.2%2Bk3s1&k=etcd%3Ahttp%3A%2F%2F70.[…]ter-e1454e17-6b01-40d2-a03a-7a3fe6786b9c&stork=true&st=k8s
Provide precise kubernetes version when generating YAML (version must be copied strictly “as is” - e.g.

kubectl version --short | awk -Fv ‘/Server Version: / {print $3}’
1.20.2+k3s1

… our YAML-generation will use the version’s extension ( +k3s1 ) to add the appropriate tweaks into the YAML. These K3S-specific tweaks are currently missing in your deployment, which is likely root-cause for PX deployment failure.