Skip to content

Releases: kmesh-net/kmesh

Release v0.5.0

27 Sep 09:59
@hzxuzhonghu hzxuzhonghu
v0.5.0
a49b6e8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Release v0.5.0 Latest
Latest

We are so excited to announce Kmesh v0.5.0. First thanks to our contributors for their hard work over the last two months. In release v0.5.0 we have made a lot of great enhancements, including command line tool kmeshctl, more complete E2E test coverage, better visualization of underlying eBPF information, observability enhancement, fully restart support, improve CNI installer, RBAC in XDP prog. In addition, in this release cycle, many critical bugs have been fixed, some key code has been refactored and more tests have been covered, making Kmesh more stable and robust. The highlights are as follows:

Zero-Down time during kmesh restart

It is amazing now Kmesh can gracefully reload eBPF map and prog after restart , and also no need to re-enroll namespaces or specific pods into kmesh after restart. As a result, the traffic flow is not interrupted during the restart, which is a big benefit to users. After kmesh-daemon restarted, the bpf map configurations will be automatically updated to date.

As early as release v0.4.0, after kmesh restarted it would require all the pods managed by kmesh to be restarted to be re-managed, because this kmesh manage is triggered by cni plug-in. Now it can be done in kmesh-daemon so that pods do not need to be restarted to be re-managed.

Observability enhancement

Now Kmesh supports L4 access log, allowing users to clearly visualize the traffic managed by Kmesh. Note that accesslog is not enabled by default. You can start the accesslog function by modifying the --enable-accesslog parameter of spec.containers.args in Kmesh. We will support using kmeshctl to enable accesslog dynamiclly . At the same time, Grafana addon adapted for Kmesh has been added to better visualize monitoring metrics in various dimensions. Also some key issues were fixed in observability, effectively improving its accuracy and stability.

Offload authorization execution into XDP prog:

As early as release v0.3.0, Kmesh already supported L4 RBAC, but the previous solution was to do rbac in the user space, which had some issues in terms of performance and functionality. Now we have offloaded it into xdp eBPF, and this feature will be truly available.
Currently, authorization rules are moved down to the eBPF map, which provides the capability of performing authorization completely in the eBPF program. When the authz result is to reject, the XDP prog directly drops the request packet so that the client can detect the connection failure.

Better debugability

Added a command line tool kmeshctl:

Kmesh has its own command line tool! Now you no longer need to exec into the corresponding Kmesh daemon pod to adjust the log level of the Kmesh daemon or dump the configuration. You can directly use kmeshctl:

# Adjust kmesh-daemon log level (e.g., debug | error | info)
kmeshctl log kmesh-6ct4h --set default:debug
# Dump config
kmeshctl dump kmesh-6ct4h workload

More features will be added to kmeshctl in the future, allowing users to better manage and debug Kmesh.

Better visualization of underlying bpf map:

Previously we have interfaces /debug/config_dump/ads and /debug/config_dump/workload to output the config content cached in Kmesh daemon. Due to various reasons, the config in Kmesh daemon cache and the actual eBPF may not be completely consistent. If we can get human-readable eBPF info, it will be more helpful for us to troubleshoot. Now we can get it through interfaces /debug/bpf/*.
It will also be integrated into kmeshctl later, making it easier to view. And can even be further expanded to determine whether the underlying eBPF is synchronized with the configuration in the Kmesh daemon.

Improve CNI installer:

As cni installer is Kmesh daemon, if the kmesh-daemon crash unexpectedly or the machine suddenly loses power, the CNI doesn’t have chance to uninstall the CNI config. If the kubeconfig’s token installed is expired, no pod can startup successfully after kmesh-daemon exit abnormally. So we make use of the following two methods to resolve:

  1. Do clean up the cni config at the end of start_kmesh.sh
  2. Add a separate go routine in cni installer, update the kubeconfig file once the token file is modified. This can make sure the kubeconfig file does not expire easily.

Support hostnetwork workloads

Now for Kmesh Dual-Engine mode, we support accessing a service with hostnetwork pods.

Performance improvement

In dual-engine mode, we largely optimized the bpf map update during Workload and Service response handling by using local cache instead of looping over the bpf map.

Critical Bug Fix

We have also made some big bug fixes:

  1. Prevent losing control of traffic during workload resource updates by not deleting the frontend map.

  2. Traffic from mamespaced waypoint will be redirected to waypoint again, it falls into a dead loop. Now we skipped managing traffic sent from waypoint.

  3. Fixed previously when waypoint processes non-HTTP tcp traffic, it would unexpectedly return HTTP/1.1 400 Bad Request. #681

What's Changed

Full Changelog * kmesh route samples by @lec-bit in https://github.com//pull/531 * Kmesh Observability by @LiZhenCheng9527 in https://github.com//pull/527 * fix unexpected log by @Okabe-Rintarou-0 in https://github.com//pull/535 * Fix TestPodSidecarLabelChangeTriggersAddIptablesAction flake by @hzxuzhonghu in https://github.com//pull/540 * Modifybpf map update to prevent potential bugs by @weli-l in https://github.com//pull/541 * add codecov config by @LiZhenCheng9527 in https://github.com//pull/537 * use latest waypoint image to run e2e by @YaoZengzeng in https://github.com//pull/554 * add document for deploying and developing in kind by @Okabe-Rintarou-0 in https://github.com//pull/559 * add Copyright check by @LiZhenCheng9527 in https://github.com//pull/561 * add security.md for kmesh by @LiZhenCheng9527 in https://github.com//pull/564 * Add bpf log level getter (#560) by @Okabe-Rintarou-0 in https://github.com//pull/562 * add document about using enhanced kernel by @Okabe-Rintarou-0 in https://github.com//pull/565 * update gitignore for enhanced kernel by @Okabe-Rintarou-0 in https://github.com//pull/572 * Add code spell check github workflow by @Okabe-Rintarou-0 in https://github.com//pull/573 * add badge in readme by @LiZhenCheng9527 in https://github.com//pull/576 * Provide a way to allow setting all logger level to debug by @hzxuzhonghu in https://github.com//pull/557 * Fix `make gen` problem by @Okabe-Rintarou-0 in https://github.com//pull/582 * fix make clean by @Okabe-Rintarou-0 in https://github.com//pull/587 * add some waypoint related E2E test cases by @YaoZengzeng in https://github.com//pull/580 * optimize workload update by @nlgwcy in https://github.com//pull/590 * remove arch info in build process by @Okabe-Rintarou-0 in https://github.com//pull/585 * kmesh security: pod manage by @lec-bit in https://github.com//pull/489 * fix bpf map look up failed by @LiZhenCheng9527 in https://github.com//pull/594 * add configuration to collect kmesh metrics using Prometheus by @LiZhenCheng9527 in https://github.com//pull/589 * Bump the k8s-io group with 3 updates by @dependabot in https://github.com//pull/609 * waypoint should not managed by Kmesh by @LiZhenCheng9527 in https://github.com//pull/611 * remove resync period by @hzxuzhonghu in https://github.com//pull/601 * Fix DNS cluster's endpoint ip addr check by @LiZhenCheng9527 in https://github.com//pull/604 * E2E test cases for service and pod ip access by @YaoZengzeng in https://github.com//pull/596 * remove build arch in documents by @Okabe-Rintarou-0 in https://github.com//pull/622 * Bypass only for sidecar by @hzxuzhonghu in https://github.com//pull/607 * Bump github.com/containernetworking/cni from 1.2.2 to 1.2.3 by @dependabot in https://github.com//pull/624 * update metric_key with direction & dst_port by @nlgwcy in https://github.com//pull/627 * E2E test cases for waypoint management by @YaoZengzeng in https://github.com//pull/625 * Support ipv6 in e2e test by @noobwei in https://github.com//pull/621 * Make kmesh cni and manage controller consitent during pod enrollment by @hzxuzhonghu in https://github.com//pull/623 * kmesh support restart by reload old bpf map and prog by @lec-bit in https://github.com//pull/475 * enable select some e2e cases to run or skip some cases by @YaoZengzeng in https://github.com//pull/638 * copy bytes optimize by @hzxuzhonghu in https://github.com//pull/633 * preclude pod with host network to be managed by kmesh by @hzxuzhonghu in https://github.com//pull/634 * remove bypass from bpf prog by @hzxuzhonghu in https://github.com//pull/635 * Enable cleanup in e2e by @noobwei in https://github.com//pull/649 * Fix kmesh daemon graceful exit by @hzxuzhonghu in https://github.com//pull/651 * Fix TestPodSidecarLabelChangeTriggersAddIptablesAction flake by @hzxuzhonghu in https://github.com//pull/636 * Fixed bug in bpf where IPv4 destination address was stored as IPv6 by @LiZhenCheng9527 in https://github.com//pull/648 * add some secure compilation options by @kwb0523 in h...
Read more

Contributors

zirain, YaoZengzeng, and 14 other contributors
Assets 5
Loading

Kmesh v0.5.0-rc.0

25 Sep 09:38
@hzxuzhonghu hzxuzhonghu
v0.5.0-rc.0
904cbff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Kmesh v0.5.0-rc.0 Pre-release
Pre-release

This is a pre-release for v0.5.0

Assets 2
Loading

Kmesh v0.4.1

30 Aug 08:22
@hzxuzhonghu hzxuzhonghu
v0.4.1
e57671f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
Kmesh v0.4.1

Bug Fixes:

  • Fixed waypoint failed to process ipv4-mapped ipv6 connections #785
  • Fixed service access when deploying namespace and service scoped waypoint in mixed manner #754
  • Fixed load balancing when svc endpoint count is zero #685
  • Fixed incorrectly handling ipv4 mapped ipv6 address #660
  • Fixed DNS cluster's endpoint ip addr check #616
  • Skip manage waypoint to prevent bpf prog fall into deadloop and interrupting traffic #611
  • Fix TestPodSidecarLabelChangeTriggersAddIptablesAction flake #543
  • Fixed workload update can cause traffic unmanaged by kmesh #592

Feature Enhancement:

  • Add service metric and use bpf ringbuf to report metrics #710
Assets 2
Loading

v0.4.0

10 Jul 10:15
@hzxuzhonghu hzxuzhonghu
v0.4.0
2a46e99
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.4.0

2024 Mid-Year Review: We are pleased to announce release v0.4.0 after more than two months's hard work from all our contributors. In release v0.4.0 we have done many enhancements such as refined pod management, management of DNS typed services, IPv6 protocol support, and data plane communication metrics monitoring. Additionally, improvements have been made in performance, maintainability, and testability, making Kmesh more robust. The key features included in this release are listed but not limited to the following:

Fine-Grained Namespace and Pod Manage:

In addition to namespace-level management, Kmesh now supports fine-grained pod-level management. You can manage pods using the following commands:

# Particular pod manage
kubectl label pod <podName> istio.io/dataplane-mode=kmesh -n {namespace}

# Namespace scoped pods manage
kubectl label ns <namespace> istio.io/dataplane-mode=kmesh

Supports IPv6 Communication in Workload Mode:

Kmesh now supports IPv6 communication, catering to both public and private cloud environments. Even for IPv4 services, Java-based services default to using the IPv6 protocol family. This enhancement ensures broader service management scenarios.

The ads mode will also support IPv6 in the near future, and also we will make Kmesh dual-stack compatible.

Performance Optimization for Rule Refresh in ads Mode:

Previously in the ads mode, Kmesh suffered from slow rule refresh because of map-in-map model used. This release, we significantly improves rule refresh performance by orders of magnitude, laying the groundwork for large-scale cluster management.

Fine-Grained Waypoint Traffic Capture:

To stay in sync with Istio 1.22, Kmesh now supports the latest workload API model. You can now use the “istio.io/use-waypoint” label to enable waypoint capture at the namespace, service, or pod level. This flexibility allows precise and on-demand use of waypoints.

DNS-Type Services Support:

In Kmesh’s ads mode, HTTP protocol is managed using eBPF and kernel modules, along with Listener, Route, and Cluster APIs. However, DNS-typed services posed a challenge because their clusters use domain names as endpoints, making DNS resolution during load balancing impossible within eBPF.

Kmesh introduces a DNS resolve module within the Kmesh daemon. It resolves domain names in user space and rewrites clusters's inlined endpoints to prevent resolving DNS early. As a result, Kmesh now supports Kubernetes ExternalName Services and Istio DNS Resolution ServiceEntry.

Observability Support:

Kmesh prioritizes observability as a critical feature in traffic management. It achieves low-cost connection monitoring using eBPF in the kernel and supports Prometheus-based metrics collection and aggregation. Future updates will enhance other observability capabilities, accesslog and traces.
To query monitoring information, use the following command:

kubectl exec -ti -n kmesh-system kmesh-6ct4h -- curl http://127.0.0.1:15020/status/metrics

Dynamic Log Level Adjustment:

Kmesh now allows dynamic adjustment of log levels for both the kmesh-daemon and eBPF prog. Now kmesh also redirect eBPF data plane logs to user space printing to avoid trace-pipe cost (requires kernel version 5.13 or higher).

# Adjust kmesh-daemon log level (e.g., debug | error | info)
kubectl exec -ti -n kmesh-system kmesh-6ct4h -- kmesh-daemon log --set default:debug
# Adjust kmesh eBPF data plane log level
kubectl exec -ti -n kmesh-system kmesh-6ct4h -- kmesh-daemon log --set bpf:debug

Additionally, there are many other improvements are there:

  • Kmesh has introduced an E2E testing framework, ensuring that each PR submission undergoes E2E tests to prevent regressions.
  • Reliability reconnections with Istiod have been addressed, and community documentation has been enriched.

What's Changed

Read more

Contributors

YaoZengzeng, kwb0523, and 13 other contributors
Assets 2
Loading

v0.3.1

11 May 07:43
@nlgwcy nlgwcy
v0.3.1
f0523f2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.3.1

Bug Fixes:

  1. Fix authz on server listening on both ipv4 and ipv6 port #279
  2. Fix incorrect domain matching in ads mode #315
  3. Repair kmesh manager not effect in sockops on oe 23.03 #303
  4. Fix DNS resolve failed from python #314
Assets 2
Loading

v0.3.0

19 Apr 16:26
@nlgwcy nlgwcy
v0.3.0
0caf0c9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.3.0

Enhancements

After a series of intense tests, we have released version 0.3.0 of Kmesh. This version introduces a four-seven layer separation governance architecture, supports distant L7 waypoint components, and includes optimizations in security, operations, and other areas. Additionally, it addresses issues related to kind deployments, making it easier for developers to use Kmesh more effectively. The key features included in this update are as follows:

  • Four-seven layer separation architecture

    Kmesh supports a workload model and integrates with distant L7 waypoints through the proxy protocol.

    • Lightweight interception forwarding: Kmesh performs L4 interception forwarding in the kernel using eBPF programs during the connection phase.

    • Simplified proxy protocol: Kmesh defines TLV-based meta-information (including communication source, destination addresses, etc.) to carry original link information required for waypoint operations. This meta-information is appended to the first data packet and sent to the waypoint.

    • L7 waypoint component

      In contrast to existing waypoint integrations based on the HBONE protocol, Kmesh introduces a waypoint component compatible with the proxy protocol. This component is developed with enhancements based on Envoy and provides services externally via port 15019.

  • Support for IP address-based traffic authorization

    • Support for access authorization based on source and destination address information.

  • Troubleshooting assistant: One-click bypass for mesh data plane

    • In service meshes, when there is an issue with data forwarding for a particular Pod, it is necessary to determine whether the problem is caused by the mesh data plane. Kmesh supports one-click bypass for the mesh data plane (Kmesh data plane/sidecar data plane), helping users troubleshoot more effectively.

      # kubectl label pod <pod_name> kmesh.net/bypass=enabled

Optimization Features:

  • Ease of Use Optimization for kind Deployments

    • Optimized the steps required to mount the BPF file system for deployments using kind, enabling one-click deployment of Kmesh.

      # helm install kmesh ./deploy/helm -n kmesh-system --create-namespace

  • Kmesh Governance Mechanism Update

    • Optimized the implementation mechanism of Kmesh governance, removing the restrictions of cgroupv1.

    • Added annotations on Pods to indicate whether they are governed by Kmesh.

      # kubectl get pods {podName} -o yaml
metadata:
  annotations:
    kmesh.net/redirection: enabled
    ...

Meanwhile, Kmesh has also optimized logging, added support for Ubuntu systems, and enhanced protection for UT test cases, making Kmesh more stable. Feel free to try out the new Kmesh image version!

# The Kmesh x86 image is used for openEuler 23.03 OS & other 5.10+ OS.
docker pull ghcr.io/kmesh-net/kmesh:v0.3.0

# Kmesh L7 waypoint image
docker pull ghcr.io/kmesh-net/waypoint-x86:v0.3.0

# Kmesh compilation environment image 
docker pull ghcr.io/kmesh-net/kmesh-build-x86:v0.3.0
Assets 2
Loading

v0.2.0

08 Feb 15:05
@nlgwcy nlgwcy
v0.2.0
6f90963
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0

On the eve of the Lunar New Year, we released version 0.2.0 of Kmesh, which has been optimized for installation, deployment, and security, and now supports basic workload functionality to better serve developers in using Kmesh. The main features included in this update are as follows:

  • More aligned with cloud-native usage patterns

    • Support for building Docker images

      Kmesh provides a compilation environment image, where all compilation processes take place within the compilation container and the output is directed to the out folder in the root directory, simultaneously generating a runnable Kmesh image.

      # make docker IMAGE={repo:tag}

    • One-click Helm installation

      # helm install kmesh ./deploy/helm -n kmesh-system --create-namespace

  • Enhanced xDS communication security

    Strengthened communication security with the service mesh control plane, establishing sessions with Istiod via secure channels.

  • Support for basic workload functionality

    In certain scenarios, Layer 7 governance capabilities are not mandatory. For such scenarios, the Istio community has introduced the workload model, which Kmesh now supports, specifically the L4 service forwarding capability within workloads. Specific usage instructions include:

    # Modify kmeshDaemonArgs value in value.yaml before Helm installation
containers:
       kmeshDaemonArgs: "-enable-kmesh-workload -enable-ads=false -enable-workload=true"

# Helm install
# helm install kmesh ./deploy/helm -n kmesh-system --create-namespace

  • Process optimization

    • Refactored the xDS incremental refresh process;
    • Removed dependency on .kubeconfig, automatically generating data plane identity information.

The image information released by Kmesh is also updated：

# The Kmesh x86 image is used for openEuler 23.03 OS.
docker pull ghcr.io/kmesh-net/kmesh:v0.2.0

# The x86 image for Kmesh online compilation and execution, supports OS kernel versions 5.10 and above.
docker pull ghcr.io/kmesh-net/kmesh-compatible:v0.2.0

# Kmesh compilation environment image 
docker pull ghcr.io/kmesh-net/kmesh-build-x86:v0.2.0
Assets 2
Loading

v0.1.0

26 Dec 14:18
@nlgwcy nlgwcy
v0.1.0
ff61192
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.0

Kmesh is a high-performance traffic management engine, based on eBPF+programmable kernel technology, sinking traffic management into the OS. The forwarding path within the service mesh changes from multi-hop to single-hop, greatly improving forwarding performance, suitable for delay-sensitive application scenarios.

Quick View of Kmesh:

Features:

  • One-click Deployment

    The community has released Kmesh deployment images, and supports one-click deployment of Kmesh through YAML here.

  • Namespace Enabling

    Supports enabling Kmesh's traffic takeover scope through namespaces.

  • Collaborative Work with Sidecar Data Plane

    Supports collaborative work with sidecar data plane. After enabling Kmesh, the traffic of newly created Pods within the namespace will be automatically taken over by Kmesh without passing through a sidecar proxy. If the sockmap feature is enabled during Kmesh deployment, it can accelerate the original sidecar traffic (reducing latency by about 15%).

  • Layer 4 Traffic Management

    Support for tcp_proxy management rules based on eBPF, including routing, grayscale, load balancing, etc.

  • Layer 7 Traffic Management

    Thanks to Kmesh's enhancement of the kernel, L7 traffic management can be fully completed within the eBPF prog without passing through any proxy software. Currently supports some models of http_connection_manager xds, including routing, grayscale, and load balancing under the http1.1 protocol.

  • Service Mesh Control Plane Connection

    Supports XDS protocol and can easily connect with service mesh control planes such as Istiod.

Docker Images:

Kmesh achieves the ability to completely sink traffic management below the OS through kernel enhancements. When releasing images, the range of OS for which the image is applicable must be considered. To this end, we consider releasing two types of images:

  • Supported OS versions with kernel enhancement modifications

    The current openEuler 23.03 OS natively supports the kernel enhancement features required by Kmesh. Kmesh release images can be directly installed and run on this OS. For a detailed list of supported OS versions with kernel enhancement modifications, please refer to this link.

  • Unsupported OS versions with kernel enhancement modifications

    To be compatible with different OS versions, Kmesh provides online compilation and running images. After Kmesh is deployed, it will automatically select Kmesh features supported by the host machine's kernel capabilities, to meet the demand for one image to run in different OS environments.

# The Kmesh x86 image is used for openEuler 23.03 OS.
docker pull ghcr.io/kmesh-net/kmesh:v0.1.0

# The x86 image for Kmesh online compilation and execution, supports OS kernel versions 5.10 and above.
docker pull ghcr.io/kmesh-net/kmesh-x86:v0.1.0

# The arm image for Kmesh online compilation and execution, supports OS kernel versions 5.10 and above.
docker pull ghcr.io/kmesh-net/kmesh-arm:v0.1.0
Assets 2
Loading