kzzz
August 6, 2024, 6:56pm
1
I used Portworx as a distributed storage for container storage interface (CSI) drivers. When I upgraded my CAPI based cluster, it caused “repaving” — replacing old nodes in the cluster one by one with new nodes that have the new desired state in place. When worker machines were repaving, CAPI continuously kill px-<cluster-name> pods, which caused CSI plugin never getting re-registered and the node hanging forever. Therefore, cluster upgrades got stuck. Since CAPI drain nodes based on --ignore-daemonsets --delete-emptydir-data --force, is it possible for Portworx to make px-<cluster-name> pods be deployed via Daemonset?
opened 05:29PM - 06 Aug 24 UTC
kind/feature
needs-triage
needs-priority
### What would you like to be added (User Story)?
As a developer, I would like … to be able filter out certain pods (certain pods won't be deleted) during the node draining process.
### Detailed Description
I used Portworx as a distributed storage for container storage interface (CSI) drivers. When I upgraded my cluster, it caused “repaving” — replacing old nodes in the cluster one by one with new nodes that have the new desired state in place. When worker machines were repaving, CAPI continuously kill `px-<cluster-name>` pods, which caused CSI plugin never getting re-registered and the node hanging forever. Therefore, cluster upgrades got stuck.
In this case, if we can have a field in machine spec to filter out the pods that we don't want CAPI to delete during the node draining process, then pods like `px-<cluster-name>` can be re-registered and repaving will be done successfully.
We may add a field called `NodeDrainPodFilters` in `MachineSpec`. We can also add this field in `KubeadmControlPlaneTemplateMachineTemplate` struct
```go
type MachineSpec struct {
// NodeDrainPodFilters allows to specify filters for pods to be excluded during node drain
// +optional
NodeDrainPodFilters *metav1.LabelSelector `json:"nodeDrainPodFilters,omitempty"`
}
```
Then, we can pass it as an extra parameter in `kubedrain.Helper`
```go
drainer := []kubedrain.PodFilter{
SkipFuncGenerator(m.Spec.NodeDrainPodFilters),
},
```
```go
func SkipFuncGenerator(labelSelector *metav1.LabelSelector) func(pod corev1.Pod) kubedrain.PodDeleteStatus {
return func(pod corev1.Pod) kubedrain.PodDeleteStatus {
if pod.Labels == nil {
return kubedrain.MakePodDeleteStatusOkay()
}
if labels.Equals(labelSelector.MatchLabels, pod.ObjectMeta.Labels) {
return kubedrain.MakePodDeleteStatusSkip()
}
return kubedrain.MakePodDeleteStatusOkay()
}
}
```
### Anything else you would like to add?
_No response_
### Label(s) to be applied
/kind feature