[YUNIKORN-2879] [shim] yunikorn unschedulable pods pending forever

[zj619]

• task postfail or rejected reschedule

What is this PR for?

when task fail, yunikorn unschedulable pods pending forever,introduce retry

What type of PR is it?

• - Bug Fix
• [√ ] - Improvement
• - Feature
• - Documentation
• - Hot Fix
• - Refactoring

Todos

• - Task

What is the Jira issue?

Issue associated with this PR：https://issues.apache.org/jira/browse/YUNIKORN-2879

How should this be tested?

test cases

Screenshots (if appropriate)

task fail to retry

after retry, bind successful

Questions:

• - The licenses files need update.
• - There is breaking changes for older versions.
• - It needs documentation.

[craigcondit]

I'm -1 on this whole approach. It's far too complex and introduces unnecessary per-pod configuration. We're trying to simplify the shim these days and this drastically complicates it.

At the very least this needs a design doc first.

[pbacsko]

-1 also.

@zhuqi-lucas is already working on sth like this, see YUNIKORN-2804.

This change alone is not even enough, we need to notify the core to cancel the allocation first and then the whole allocation cycle can restart again. We don't want to see an un-bindable pod on the core side to occupy resources.

[pbacsko]

Also, next time:

1. Please add a short description to "What is this PR for?" and remove the template text.
2. Insert the link to the upstream JIRA ticket properly.

Thanks.