I’m running prometheus container as a single replica Docker swarm service that can migrate between portworx’s 3 nodes. Everything worked well, but after the last update of the container, I’m getting error in syslog of any node of three:
“dockerd[17597]: time=“2020-05-21T17:30:15.006531562Z” level=error msg=“fatal task error” error=“create prometheus_vl: VolumeDriver.Create: PX Service Unavailable” module=node/agent/taskmanager node.id=0vx87m0rrc8ath6dlk1ejb1qq service.id=n43sbobmk0yn1x5jmpx0oxi4l task.id=mdo58n32xeusv3kbnoiyst4kx”
The volume exists and visible on all three nodes.
Can you please advise me, how can I get my container working again without deleting data?
Some info:
px-dev v.2.3.6.0 running as docker containers on 3 nodes
Docker 19.03.6 on Ubuntu 18.04
pxd plugin is installed on the system
the mounting command from docker service create:
–mount type=volume,volume-driver=pxd,source=prometheus_vl,target=/prometheus
The output of pxctl volume list is:
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY STATUS SNAP-ENABLED
1068279827754234208 prometheus_vl 100 GiB 2 yes no HIGH up - detached no
Any ideas?