v1.6.1

To install or upgrade to the old version, see installation doc.

Change log since v1.6.0

Upgrade Notice

• FeatureGate PodWebhook=false will not disable ResourcesDeletionProtection. (#1526, @zmberg)
• Update go.mod require k8s version from 1.29 to 1.26, and remove go mod replace. (#1527, KaiShi)

Advanced Workload

• Fix when StatefulSet reserveOrdinals exist and whenScaled=Delete, scale down pvc failed. (#1531, @zmberg)

v1.5.3

To install or upgrade to the old version, see installation doc.

Change log since v1.5.2

Advanced Workload

• Fix when StatefulSet reserveOrdinals exist and whenScaled=Delete, scale down pvc failed. (#1531, @zmberg)

v1.6.0

To install or upgrade to the old version, see installation doc.

Change log since v1.5.2

Upgrade Notice

No, really, you must read this before you upgrade

• OpenKruise no longer supports Kubernetes versions 1.16, 1.17.
  However it's still possible to use OpenKruise with Kubernetes versions 1.16 and 1.17 as long as KruiseDaemon is not enabled(install/upgrade kruise charts with featureGates="KruiseDaemon=false")
• OpenKruise leader election default to use leases mode. (#1407, dsxing)
  For users with OpenKruise version 1.3.0 or lower, please first upgrade your OpenKruise to version 1.4 or 1.5 before upgrading to 1.6.0, so as to avoid unexpected multiple leader problem during the installation.
• Bump Kubernetes dependency to 1.26.10. (#1511, KaiShi)

Key Features

• Fix WorkloadSpread incorrect subset allocation after workload rolling updating. (#1197, veophi)
• ImagePullJob support force image pulling for images with the name as previous one. (#1384, ls-2018)
• Job Sidecar Terminator reports correct pod phase for sidecar containers with non-zero exit code. (#1303, @diannaowa)
• Support the deletion protection of service and ingress resources. (#1269, @kevin1689-cloud)

Performance Enhancement

• Optimize PodProbeMarker performance. (#1430, ls-2018)
• Optimize container launch priority performance. (#1490, FillZpp)

Other Changes

• Enhanced Operation

  • PodProbeMarker: Container probe support Tcp probing. (#1474, KaiShi)
  • PodProbeMarker: Sync podCondition when probe message of probeStates changed. (#1479, chrisliu1995)
  • PersistentPodState: Fix the problem that PersistentPodState can't get spec.replicas from unstructured object. (#1462, 0xgj)
  • Fix PodProbeMarker feature gate dependency . (#1429, ls-2018)

• Advanced Workload

  • Enforce Advanced DaemonSet spec.selector is immutable. (#1505, @hantmac)
  • Advanced StatefulSet maxUnavailable now counts unavailable pods with smaller ordinal in the update order during rolling upgrade. (#1480, @Yesphet)
  • Fix EphemeralJob event handler for deleting object. (#1401, FillZpp)

• Sidecar Container

  • Fix pod annotations injection abnormal for SidecarSet. (#1453, @a932846905)

• Application Protection

  • PodUnavailableBudget ignore deletion of not ready or inconsistent pods. (#1512, Spground)

• Others

  • Replace 'github.com/pkg/errors' with the standard Go library 'errors'. (#1518, dongjiang1989)
  • Upgrade minimum docker api version from 1.23 to 1.24. (#1510, hantmac)
  • Add UT in controller_revision_test file. (#1457, xiangpingjiang)
  • BroadcastJob controller define some parameters as Constant. (#1414, lilongfeng0902)
  • Kruise-daemon enable pprof. (#1416, dsxing)
  • Remove deprecated 'io/ioutil' pkg. (#1404, testwill)
  • Fix unnecessary use of fmt.Sprintf. (#1403, testwill)

v1.5.2

To install or upgrade to the old version, see installation doc.

Change log since v1.5.1

CVE FIX: Enhance kruise-daemon security (#1482, veophi)

Start kruise-manager as a non-root user

We start kruise-manger with a non-root user to further enhance the security of kruise-manager. (#1491, @zmberg)

v1.5.1

To install or upgrade to the old version, see installation doc.

Change log since v1.5.0

In version 1.5.1, the focus was on enhancing UnitedDeployment and addressing various bug fixes:

• Add the ability to plan the lower and upper bound of capacity to the subsets in UnitedDeployment (#1428, @veophi)

• Fix unexpected job recreation by adding controller-revision-hash label for ImageListPullJob. (#1441, @veophi)

• Add prometheus metrics for pub and deletion protection to enhance observability for pub & deletion protection (#1398, @zmberg)

• Add enable pprof flag for kruise daemon, now you can disable the pprof of kruise daemon (#1416, @chengjoey)

• Fix SidecarSet upgrade exception for UpdateExpectations to solve the problem of updating the image of the sidecar container (#1435, @zmberg])

• add audit log for pub and deletion protection to enhance observability for pub & deletion protection (#1438, @zmberg])

v1.5.0

To install or upgrade to the old version, see installation doc.

Change log since v1.4.0

Upgrade Notice

No, really, you must read this before you upgrade

• Disable following feature-gates by default: PreDownloadImageForInPlaceUpdate(#1244, @zmberg), ImagePullJobGate(#1357, @zmberg), DeletionProtectionForCRDCascadingGate(#1365, @zmberg), and ResourceDistributionGate(#1360, @zmberg)
• Bump Kubernetes dependency to 1.24.16, Golang version to 1.19(#1354, Kuromesi)

Key Features: Enhanced Multi-Domain Management

• WorkloadSpread:
  • Support any customized workloads that have scale sub-resource. (#1286, veophi)
  • Add validation for subset patch field. (#1237, chengleqi)
• UnitedDeployment:
  • Support scale sub-resource. (#1314), diannaowa)
  • Support patch field for each subset. (#1266, chengleqi)
  • Optimize UnitedDeployment replicas settings. (#1247, y-ykcir)

ImagePreDownload

• ImageListPullJob:
  • Many users have the need for batch pre-download images, and the current approach, i.e., ImagePullJob, has a relatively high threshold for use, We added a new CRD ImageListPullJob to batch pre-download images.
    You just write a range of images in one ImageListPullJob CR, its controller will generate corresponding ImagePullJob CR for each image automatically. (1222, @diannaowa)
• ImagePullJob:
  • Fix the matching logic for the imagePullSecret in ImagePullJob. (#1241, #1357)
  • Advanced Workload pre-download image support attach metadata in ImagePullJob. (#1246, YTGhost)

Advanced Workload

• SidecarSet:
  • Add condition and event for not upgradable pods when updating. (#1309, MarkLux)
  • Take effect of shareVolumePolicy on initContainers. (#1229, y-ykcir)
  • Allow sidecar containers to mount serviceAccountToken type volume. (#1238, y-ykcir)
  • SidecarSet updateStrategy support priorityStrategy. (#1325, y-ykcir)
• BroadcastJob:
  • Make OnFailure as default restartPolicy. (#1149, Shubhamurkade)
  • Fix BroadcastJob doesn't make pod on node that has erased taint. (#1204, weldonlwz)
• CloneSet & StatefulSet:
  • Regard the pod at preparing update state as update revision when scaling. (#1290, veophi)
  • Add updatedAvailableReplicas field in status. (#1317, nitishchauhan0022)

Kruise Daemon

• Connecting to Pouch runtime via CRI interface. (#1232, @zmberg)
• Compatible with v1 and v1alpha2 CRI API version. (#1354, veophi)

ResourceProtection

• Reject Namespace deletion when PVCs are included under NS. (#1228, kevin1689-cloud)

And some bugs were fixed by
(#1238, y-ykcir),
(#1335, ls-2018),
(#1301, wangwu50),
(#1395, ywdxz),
(#1304, kevin1689-cloud),
(#1348, #1343, Colvin-Y),
thanks!

v1.4.0

To install or upgrade to the old version, see installation doc.

Change log since v1.3.0

Upgrade Notice

No, really, you must read this before you upgrade

• Enable following feature-gates by default: ResourcesDeletionProtection, WorkloadSpread, PodUnavailableBudgetDeleteGate, InPlaceUpdateEnvFromMetadata,
  StatefulSetAutoDeletePVC, PodProbeMarkerGate. (#1214, @zmberg)
• Change Kruise leader election from configmap to configmapsleases, this is a smooth upgrade with no disruption to OpenKruise service. (#1184, @YTGhost)

New Feature: JobSidecarTerminator

In the Kubernetes world, it is challenging to use long-running sidecar containers for short-term job because there is no straightforward way to
terminate the sidecar containers when the main container exits. For instance, when the main container in a Pod finishes its task and exits, it is expected that accompanying sidecars,
such as a log collection sidecar, will also exit actively, so the Job Controller can accurately determine the completion status of the Pod.
However most sidecar containers lack the ability to discovery the exit of main container.

For this scenario OpenKruise provides the JobSidecarTerminator capability, which can terminate sidecar containers once the main containers exit.

For more detail, please refer to its proposal.

Advanced Workloads

• Optimized CloneSet Event handler to reduce unnecessary reconciliation. The feature is off by default and controlled by CloneSetEventHandlerOptimization feature-gate. (#1219, @veophi)
• Avoid pod hang in PreparingUpdate state when rollback before update hook. (#1157, @shiyan2016)
• Fix cloneSet update blocking when spec.scaleStrategy.maxUnavailable is not empty. (#1136, @ivanszl)
• Add 'disablePVCReuse' field to enable recreation of PVCs when rebuilding pods, which can avoid Pod creation failure due to Node exceptions. (#1113, @willise)
• CloneSet 'partition' field support float percent to improve precision. (#1124, @shiyan2016)
• Add PreNormal Lifecycle Hook for CloneSet. (#1071, @veophi)
• Allow to mutate PVCTemplate of Advanced StatefulSet & CloneSet. Note, Only works for new Pods, not for existing Pods. (#1118, @veophi)
• Make ephemeralJob compatible with k8s version 1.20 & 1.21. (#1127, @veophi)
• UnitedDeployment support advanced StatefulSet 'persistentVolumeClaimRetentionPolicy' field. (#1110, @yuexian1234)

ContainerRecreateRequest

• Add 'forceRecreate' field to ensure the immediate recreation of the container even if the container is starting at that point. (#1182, @BH4AWS)

ImagePullJob

• Support attach metadata in PullImage CRI interface during ImagePullJob. (#1190, @diannaowa)

SidecarSet

• SidecarSet support namespace selector. (#1178, @zmberg)

Others

• Simplify some code, mainly comparison and variable declaration. (#1209, @hezhizhen)
• Update k8s registry references from k8s.gcr.io to registry.k8s.io. (#1208, @asa3311)
• Fix config/samples/apps_v1alpha1_uniteddeployment.yaml invalid image. (#1198, @chengleqi)
• Change kruise base image to alpine. (#1166, @fengshunli)
• PersistentPodState support custom workload (like statefulSet). (#1063, @baxiaoshi)

v1.3.0

To install or upgrade to the old version, see installation doc.

Change log since v1.2.0

New CRD and Controller: PodProbeMarker

Kubernetes provides three Pod lifecycle management:

• Readiness Probe Used to determine whether the business container is ready to respond to user requests. If the probe fails, the Pod will be removed from Service Endpoints.
• Liveness Probe Used to determine the health status of the container. If the probe fails, the kubelet will restart the container.
• Startup Probe Used to know when a container application has started. If such a probe is configured, it disables liveness and readiness checks until it succeeds.

So the Probe capabilities provided in Kubernetes have defined specific semantics and related behaviors.
In addition, there is actually a need to customize Probe semantics and related behaviors, such as:

• GameServer defines Idle Probe to determine whether the Pod currently has a game match, if not, from the perspective of cost optimization, the Pod can be scaled down.
• K8S Operator defines the main-secondary probe to determine the role of the current Pod (main or secondary). When upgrading, the secondary can be upgraded first,
  so as to achieve the behavior of selecting the main only once during the upgrade process, reducing the service interruption time during the upgrade process.

So we provides the ability to customize the Probe and return the result to the Pod yaml.

For more detail, please refer to its documentation and proposal.

SidecarSet

• SidecarSet support to inject pods under kube-system,kube-public namespace. (#1084, @zmberg)
• SidecarSet support to inject specific history sidecar container to Pods. (#1021, @veophi)
• SidecarSet support to inject pod annotations.(#992, @zmberg)

AdvancedCronJob

• TimeZone support for AdvancedCronJob from upstream. (#1070, @FillZpp)

WorkloadSpread

• WorkloadSpread support Native StatefulSet and Kruise Advanced StatefulSet. (#1056, @veophi)

CloneSet

• CloneSet supports to calculate scale number excluding Pods in PreparingDelete. (#1024, @FillZpp)
• Optimize CloneSet queuing when cache has just synced. (#1026, @FillZpp)

PodUnavailableBudget

• Optimize event handler performance for PodUnavailableBudget. (#1027, @FillZpp)

Advanced DaemonSet

• Allow optional filed max unavilable in ads, and set default value 1. (#1007, @ABNER-1)
• Fix DaemonSet surging with minReadySeconds. (#1014, @FillZpp)
• Optimize Advanced DaemonSet internal new pod for imitating scheduling. (#1011, @FillZpp)
• Advanced DaemonSet support pre-download image. (#1057, @ABNER-1)

Advanced StatefulSet

• Fix panic cased by statefulset pvc auto deletion. (#999, @veophi)

Others

• Optimize performance of LabelSelector conversion. (#1068, @FillZpp)
• Reduce kruise-manager memory allocation. (#1015, @FillZpp)
• Pod state from updating to Normal should all hooked. (#1022, @shiyan2016)
• Fix go get in Makefile with go 1.18. (#1036, @astraw99)
• Fix EphemeralJob spec.replicas nil panic bug. (#1016, @hellolijj)
• Fix UnitedDeployment reconcile don't return err bug. (#991, @huiwq1990)

v1.2.0

To install or upgrade to the old version, see installation doc.

Changes since v1.1.0

New CRD and Controller: PersistentPodState

With the development of cloud native, more and more companies start to deploy stateful services (e.g., Etcd, MQ) using Kubernetes.
K8S StatefulSet is a workload for managing stateful services, and it considers the deployment characteristics of stateful services in many aspects.
However, StatefulSet persistent only limited pod state, such as Pod Name is ordered and unchanging, PVC persistence,
and can not cover other states, e.g. Pod IP retention, priority scheduling to previously deployed Nodes.

So we provide PersistentPodState CRD to persistent other states of the Pod, such as "IP Retention".

For more detail, please refer to its documentation and proposal.

CloneSet

• Ensure at least one pod is upgraded if CloneSet has partition < 100% (Behavior Change). (#954, @veophi)
• Add expectedUpdatedReplicas field into CloneSet status. (#954 & #963, @veophi)
• Add markPodNotReady field into lifecycle hook to support marking Pod as NotReady during preparingDelete or preparingUpdate. (#979, @veophi)

StatefulSet

• Add markPodNotReady field into lifecycle hook to support marking Pod as NotReady during preparingDelete or preparingUpdate. (#979, @veophi)

PodUnavailableBudget

• Support to protect any custom workloads with scale subresource. (#982, @zmberg)
• Optimize performance in large-scale clusters by avoiding DeepCopy list. (#955, @zmberg)

Others

• Remove some commented code and simplify some. (#983, @hezhizhen)
• Sidecarset forbid updating of sidecar container name. (#937, @adairxie)
• Optimize the logic of listNamespacesForDistributor func. (#952, @hantmac)

Thanks to all our contributors! 😊

v1.1.0

To install or upgrade to the old version, see installation doc.

Changes since v1.0.1

Project

• Bump Kubernetes dependencies to 1.22 and controller-runtime to v0.10.2. (#915, @FillZpp)
• Disable DeepCopy for some specific cache list. (#916, @FillZpp)

InPlace Update

• Support in-place update containers with launch priority, for workloads that supported in-place update, e.g., CloneSet, Advanced StatefulSet. (#909, @FillZpp)

CloneSet

• Add pod-template-hash label into Pods, which will always be the short hash. (#931, @FillZpp)
• Support pre-download image after a number of updated pods has been ready. (#904, @shiyan2016)
• Make maxUnavailable also limited to pods in new revision. (#899, @FillZpp)

SidecarSet

• Support shared volumes in init containers. (#929, @outgnaY)
• Support transferEnv in init containers. (#897, @pigletfly)
• Optimize the injection for pod webhook that checks container exists. (#927, @zmberg)
• Fix validateSidecarConflict to avoid a same sidecar container exists in multiple sidecarsets. (#884, @pigletfly)

Advanced DaemonSet

• Refactor daemonset controller and fetch upstream codebase. (#883, @FillZpp)
• Support preDelete lifecycle for both scale down and recreate update. (#923, @FillZpp)
• Fix node event handler that should compare update selector matching changed. (#920, @LastNight1997)
• Optimize dedupCurHistories func in ReconcileDaemonSet. (#912, @LastNight1997)

Advanced StatefulSet

• Support StatefulSetAutoDeletePVC feature. (#882, @veophi)

Kruise-daemon

• Support CRI-O and any other common CRI types. (#930, @diannaowa) & (#936, @FillZpp)

Other

• Add kruise_manager_is_leader metric. (#917, @hatowang)

Thanks to all our contributors! 😊