[scheduler, mtsource] Even Spread across Nodes in each zone for HA scheduler

[aavarghese]

Fixes #593
🎁 Implementing HA across nodes within a zone as a new scheduling strategy

        - name: SCHEDULER_POLICY_TYPE
          value: 'EVENSPREAD_BYNODE'

Continuation of #587

Proposed Changes

• Controller has another new env var to control "scheduler policy type" called 'EVENSPREAD_BYNODE',
• "scheduler policy type" is an enum with three values MaxFillup, EvenSpread and EvenSpreadByNode. For EvenSpreadByNode, scheduler places vreplicas for HA by spreading replicas equally across nodes within a zone. Within a node, vreplicas are filled up in each pod upto capacity.
• Placement type in KafkaSource status has a new type for NodeName when EvenSpreadByNode scheduling
• HA controls are also shared with autoscaler, and state accessor
• State accessor computes the number of nodes in cluster and stores a map of node name to zone info

Release Note

New SCHEDULER_POLICY_TYPE called 'EVENSPREAD_BYNODE' for a High Availability scheduler that uniformly distributes vreplicas across all nodes in each zone to reduce impact of failure when a node or zone fails.

[knative-prow-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Merging #707 (72dbf28) into main (699e408) will increase coverage by 0.10%.
The diff coverage is 93.75%.

❗ Current head 72dbf28 differs from pull request most recent head 89ffb36. Consider uploading reports for the commit 89ffb36 to get more accurate results

@@            Coverage Diff             @@
##             main     #707      +/-   ##
==========================================
+ Coverage   73.84%   73.94%   +0.10%     
==========================================
  Files         133      133              
  Lines        5880     5861      -19     
==========================================
- Hits         4342     4334       -8     
+ Misses       1315     1309       -6     
+ Partials      223      218       -5     

Impacted Files	Coverage Δ
pkg/common/scheduler/statefulset/state.go	84.90% <66.66%> (-0.81%)	⬇️
pkg/common/scheduler/statefulset/scheduler.go	86.10% <95.23%> (+3.19%)	⬆️
pkg/common/scheduler/statefulset/autoscaler.go	73.58% <100.00%> (-1.10%)	⬇️
pkg/source/adapter/adapter.go	58.42% <0.00%> (ø)
...onsolidated/dispatcher/consumer_message_handler.go	0.00% <0.00%> (ø)
pkg/channel/consolidated/utils/util.go	100.00% <0.00%> (+4.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d1daa0...89ffb36. Read the comment docs.

[aavarghese]

/cc @lionelvillard

[knative-metrics-robot]

The following is the coverage report on the affected files.
Say /test pull-knative-sandbox-eventing-kafka-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/common/scheduler/statefulset/scheduler.go	87.3%	89.9%	2.6
pkg/common/scheduler/statefulset/state.go	93.9%	92.3%	-1.6

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aavarghese
To complete the pull request process, please assign evankanderson after the PR has been reviewed.
You can assign the PR to them by writing /assign @evankanderson in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment