Respond to nodeConditions changes

[nogazax]

Is your feature request related to a problem? Please describe.
When using nodeProblemDetector (NPD), there is a gap where NPD marks nodes using nodeCondition, and Descheduler (DSC) is watching taints.

Describe the solution you'd like
I want to fill this gap with a controller, which is alerted on nodeCondition changes.
Controller reads condition, if it matches a some criteria (implementation + restrictions TBD) it taints the node\deschedules the pods (customable taint\simply cordon TBD).

After tainting, DSC starts removing the pods from these nodes, and clusterAutoScaler will remove the node when underutilized.

Describe alternatives you've considered
non controller application - IMO, in this use case it's better subscribing to events rather than polling api server for them.
using forked version of Draino project - as an unmaintained project, with multiple security vulnerabilities, not the favorite option
add this functionality to NPD - stale PR for a long time

What version of descheduler are you using?

descheduler version:
V0.30.0

Additional context
I wrote a small POC.
NPD marks condition -> my controller recognized it and added taint to node -> DSC acts on pods without toleration

[a7i]

Would you be able to use the existing RemovePodsViolatingNodeTaints?

[nogazax]

I think so, but not sure if this is the right place to add it:

1. if yes, should it be renamed to RemovePodsViolatingNodeTaintsAndConditions?
2. another option is to create another plugin RemovePodsViolatingNodeCondition. reducing complexity, adds functionality, but kinda duplicates the business logic
3. last option is to add a small component which taints nodes based on conditions, and letting current controller do it's job.

WDYT?

[googs1025]

Maybe RemovePodsViolatingCustomNodeConditions is more appropriate. 🤔 The built-in NodeConditions will be caught and evicted by the nodeLifeCycleController.

[googs1025]

@a7i @ingvagabund any comment? 😄