We are in process of moving our mysql database to docker swarm. We have performance issue:
We have 3 nodes of 1TB SSD drive each portworx cluster. They are connected over 10GBit network.
I created volume for mysql database, like this:
pxctl volume create cmtsdb_vl --sharedv4 -s 1024 -r 2 --io_priority high --io_profile=db_remote
The volume is mounted by docker swarm onto /var/lib/mysql
The volume is created on two machines of the 3 available.
The container runs on one of the two machines that host the volume.
Sustained writes work at acceptable 200GB speed:
root@db-cmts_nabp:/var/lib/mysql# time dd if=/dev/zero bs=1G count=10 of=delete_me oflag=sync
10+0 records in
10+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 51.4847 s, 209 MB/s
real 0m51.582s
user 0m0.005s
sys 0m27.818s
But mysql has very poor performance - miserable 2MBytes/s.
I suspect that’s because SYNC takes unusual long time to complete, when mysql is running, sync takes 600-1000 milliseconds:
root@db-cmts_nabp:/# time sync
real 0m0.660s
user 0m0.003s
sys 0m0.000s
I already tried to minimize number of SYNC issued by mysql:
innodb_flush_log_at_trx_commit=2
I also remounted the ext4 filesystem of the volume to have nobarrier option, but that did not help (I know it is unacceptable for production).
Do you have other suggestions for me to try?
Portworx Version: 2.6.x
Deployment Type: On Premise