K8s added a new node - IO,BTRFS error

Hi there,

We added a new node to the kubernetes cluster, we got a lot of errors during portworx operator, I saw the following errors while researching. We have another node using the same storage, there is no problem in the other node, has anyone encountered this error?

[Mon Nov 29 18:54:12 2021] blk_update_request: critical target error, dev sdc, sector 4038840
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504599, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504600, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504601, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504602, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504603, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504604, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504605, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504606, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504607, lost async page write
[Mon Nov 29 18:54:12 2021] Buffer I/O error on dev sdc1, logical block 504608, lost async page write
[Mon Nov 29 18:54:46 2021] sd 0:0:2:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=52s
[Mon Nov 29 18:54:46 2021] sd 0:0:2:0: [sdc] Sense Key : Hardware Error [current]
[Mon Nov 29 18:54:46 2021] sd 0:0:2:0: [sdc] Add. Sense: Internal target failure
[Mon Nov 29 18:54:46 2021] sd 0:0:2:0: [sdc] CDB: Write(10) 2a 00 00 3e 60 c0 00 04 00 00
[Mon Nov 29 18:54:46 2021] blk_update_request: critical target error, dev sdc, sector 4088000
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510744, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510745, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510746, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510747, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510748, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510749, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510750, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510751, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510752, lost async page write
[Mon Nov 29 18:54:46 2021] Buffer I/O error on dev sdc1, logical block 510753, lost async page write
[Mon Nov 29 19:07:33 2021] sdc: sdc1 sdc2
[Mon Nov 29 19:07:33 2021] sdc: sdc1 sdc2
[Mon Nov 29 19:07:33 2021] sdc:
[Mon Nov 29 19:07:38 2021] sdc: sdc1
[Mon Nov 29 19:07:51 2021] sdc: sdc1 sdc2
[Mon Nov 29 19:08:02 2021] BTRFS: device fsid ede6cabf-8963-4e14-b433-a869d713ab61 devid 1 transid 5 /dev/sdc2
[Mon Nov 29 19:08:02 2021] BTRFS info (device sdc2): not using ssd allocation scheme
[Mon Nov 29 19:08:02 2021] BTRFS info (device sdc2): disk space caching is enabled
[Mon Nov 29 19:08:02 2021] BTRFS info (device sdc2): has skinny extents
[Mon Nov 29 19:08:02 2021] BTRFS info (device sdc2): flagging fs with big metadata feature
[Mon Nov 29 19:08:02 2021] BTRFS info (device sdc2): creating UUID tree
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): not using ssd allocation scheme
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): disk space caching is enabled
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): has skinny extents
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): not using ssd allocation scheme
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): disk space caching is enabled
[Mon Nov 29 19:08:18 2021] BTRFS info (device sdc2): has skinny extents
[Mon Nov 29 19:08:35 2021] BTRFS info (device sdc2): not using ssd allocation scheme
[Mon Nov 29 19:08:35 2021] BTRFS info (device sdc2): disk space caching is enabled
[Mon Nov 29 19:08:35 2021] BTRFS info (device sdc2): has skinny extents
[Mon Nov 29 19:57:51 2021] sd 0:0:2:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Mon Nov 29 19:57:51 2021] sd 0:0:2:0: [sdc] Sense Key : Hardware Error [current]
[Mon Nov 29 19:57:51 2021] sd 0:0:2:0: [sdc] Add. Sense: Internal target failure
[Mon Nov 29 19:57:51 2021] sd 0:0:2:0: [sdc] CDB: Write(10) 2a 00 00 60 ae c0 00 00 80 00
[Mon Nov 29 19:57:51 2021] blk_update_request: critical target error, dev sdc, sector 6336192
[Mon Nov 29 19:57:51 2021] BTRFS error (device sdc2): bdev /dev/sdc2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[Mon Nov 29 19:57:51 2021] BTRFS: error (device sdc2) in btrfs_commit_transaction:2228: errno=-5 IO failure (Error while writing out transaction)
[Mon Nov 29 19:57:51 2021] BTRFS info (device sdc2): forced readonly
[Mon Nov 29 19:57:51 2021] BTRFS warning (device sdc2): Skipping commit of aborted transaction.
[Mon Nov 29 19:57:51 2021] BTRFS: error (device sdc2) in cleanup_transaction:1851: errno=-5 IO failure
[Mon Nov 29 19:57:51 2021] BTRFS info (device sdc2): delayed_refs has NO entry
[Tue Nov 30 21:36:46 2021] BTRFS error (device sdc2): cleaner transaction attach returned -30
[Tue Nov 30 21:37:41 2021] BTRFS info (device sdc2): not using ssd allocation scheme
[Tue Nov 30 21:37:41 2021] BTRFS info (device sdc2): disk space caching is enabled
[Tue Nov 30 21:37:41 2021] BTRFS info (device sdc2): has skinny extents
[Tue Nov 30 21:46:08 2021] sd 0:0:2:0: [sdc] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Tue Nov 30 21:46:08 2021] sd 0:0:2:0: [sdc] Sense Key : Hardware Error [current]
[Tue Nov 30 21:46:08 2021] sd 0:0:2:0: [sdc] Add. Sense: Internal target failure
[Tue Nov 30 21:46:08 2021] sd 0:0:2:0: [sdc] CDB: Write(10) 2a 00 00 60 71 40 00 01 60 00
[Tue Nov 30 21:46:08 2021] blk_update_request: critical target error, dev sdc, sector 6320448
[Tue Nov 30 21:46:08 2021] BTRFS error (device sdc2): bdev /dev/sdc2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0

It looks like Hard drive failure to my understanding.
Could you check and confirm on the console ?