Clean up the residual annotations when resources are preempted by pp from cpp

[zhzhuang-zju]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
refer to #3845, pp will preempt cpp regardless of priority. And it will also clean up the labels used to mark cpp during the preemption process. However, currently only the labels are being cleaned up, and the annotations are being left behind. This results in the same resource having both the annotation marking pp and the annotation marking cpp, like

apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
  annotations:
    clusterpropagationpolicy.karmada.io/name: test-cpp
    policy.karmada.io/applied-placement: '{"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Aggregated"}}'
    propagationpolicy.karmada.io/name: test-pp
    propagationpolicy.karmada.io/namespace: default

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
From a mechanism standpoint, only pp can preempt cpp. Therefore, there is no need to consider cleaning up pp's labels and annotations when applying ClusterPolicy.

Does this PR introduce a user-facing change?:

[zhzhuang-zju]

test report:

1. create configmap test-cm

apiVersion: v1
data:
  a: b
kind: ConfigMap
metadata:
  name: test-cm
  namespace: default

2. create cpp test-cpp

apiVersion: policy.karmada.io/v1alpha1
kind: ClusterPropagationPolicy
metadata:
  name: test-cpp
spec:
  conflictResolution: Abort
  placement:
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaSchedulingType: Duplicated
  preemption: Never
  priority: 10
  resourceSelectors:
  - apiVersion: v1
    kind: ConfigMap
    name: test-cm
    namespace: default
  schedulerName: default-scheduler

3. wait util the configmap has bund to the cpp
4. create a pp test-pp

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: test-pp
  namespace: default
spec:
  conflictResolution: Abort
  placement:
    clusterTolerations:
    - effect: NoExecute
      key: cluster.karmada.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: cluster.karmada.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    replicaScheduling:
      replicaDivisionPreference: Aggregated
      replicaSchedulingType: Divided
  preemption: Always
  priority: 0
  resourceSelectors:
  - apiVersion: v1
    kind: ConfigMap
    name: test-cm
    namespace: default
  schedulerName: default-scheduler

and the marks on rb test-cm-configmap will be:

$ karmadactl get rb test-cm-configmap -oyaml  
apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
  annotations:
    policy.karmada.io/applied-placement: '{"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Aggregated"}}'
    propagationpolicy.karmada.io/name: test-pp
    propagationpolicy.karmada.io/namespace: default
  creationTimestamp: "2024-09-18T08:15:36Z"
  finalizers:
  - karmada.io/binding-controller
  generation: 4
  labels:
    propagationpolicy.karmada.io/permanent-id: ede05d0c-0d6c-4bb9-8001-2601a7cacfc5
    resourcebinding.karmada.io/permanent-id: 32fb4ef6-128c-4559-8172-a384cc51ceb4
  name: test-cm-configmap
  namespace: default

[zhzhuang-zju]

cc @whitewindmills

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 50.84746% with 29 lines in your changes missing coverage. Please review.

Project coverage is 35.43%. Comparing base (62ae95e) to head (69a07ed).
Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/detector/detector.go	0.00%	29 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5563      +/-   ##
==========================================
+ Coverage   35.21%   35.43%   +0.21%     
==========================================
  Files         645      646       +1     
  Lines       44885    44988     +103     
==========================================
+ Hits        15806    15940     +134     
+ Misses      27847    27798      -49     
- Partials     1232     1250      +18     

Flag	Coverage Δ
unittests	35.43% <50.84%> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zhzhuang-zju]

/retest

[zhzhuang-zju]

All review comments have been resolved. To facilitate the review process, I will rebase the commits before merging
cc @RainbowMango @whitewindmills

[RainbowMango]

/assign

[RainbowMango]

Generally looks good to me.

[RainbowMango]

/lgtm

[RainbowMango]

All review comments have been resolved. To facilitate the review process, I will rebase the commits before merging cc @RainbowMango @whitewindmills

Are you going to squash the commits?

[whitewindmills]

LGTM

[XiShanYongYe-Chang]

LGTM

[RainbowMango]

/lgtm
/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment