Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BPF docs - Add a note about IPVS and MKE docs to 3.28 #1634

Merged
merged 6 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions calico-enterprise/operations/ebpf/enabling-ebpf.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,11 @@ To re-enable it:
```bash
kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}'
```
:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

#### MKE

Expand Down
10 changes: 7 additions & 3 deletions calico-enterprise/operations/ebpf/install.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ and in particular, pushing the networking capabilities of the latest Linux kerne
**Unsupported platforms**

- GKE
- MKE
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're removing support for MKE, do we not need to remove from other areas? See Line 170 in enabling-ebpf.mdx. I can't make any sense of this. If there's a qualification, this must be made explicit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry. That was a mistake. we are not removing support for MKE. Reverted ^^

- TKG
- RKE

Expand All @@ -52,10 +53,7 @@ eBPF supports AKS with Calico CNI and {{prodname}} network policy. However, with
:::

**Unsupported features**
- Using NodeLocal DNSCache in your cluster
- Clusters with some eBPF nodes and some standard dataplane and/or Windows nodes
- IPv6
- Host endpoint `doNotTrack` policy (other policy types are supported)
- Floating IPs
- SCTP (either for policy or services)
- `Log` action in policy rules
Expand Down Expand Up @@ -417,6 +415,12 @@ Then, should you want to start `kube-proxy` again, you can simply remove the nod
</TabItem>
</Tabs>

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

## Next steps

**Recommended**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,11 @@ To re-enable it:
```bash
kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}'
```
:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

#### MKE

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,7 @@ eBPF supports AKS with Calico CNI and {{prodname}} network policy. However, with
:::

**Unsupported features**
- Using NodeLocal DNSCache in your cluster
- Clusters with some eBPF nodes and some standard dataplane and/or Windows nodes
- IPv6
- Host endpoint `doNotTrack` policy (other policy types are supported)
- Floating IPs
- SCTP (either for policy or services)
- `Log` action in policy rules
Expand Down Expand Up @@ -418,6 +415,12 @@ Then, should you want to start `kube-proxy` again, you can simply remove the nod
</TabItem>
</Tabs>

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

## Next steps

**Recommended**
Expand Down
6 changes: 6 additions & 0 deletions calico/operations/ebpf/enabling-ebpf.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,12 @@ If you are running MKE, you can disable `kube-proxy` as follows:
Follow the step procedure in [Modify an existing MKE configuration](https://docs.mirantis.com/mke/current/ops/administer-cluster/configure-an-mke-cluster/use-an-mke-configuration-file.html#modify-an-existing-mke-configuration) to download, edit, and upload your MKE configuration. During the editing step, add the following configuration:
`kube_proxy_mode=disabled` and `kube_default_drop_masq_bits=true`.

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

### Avoiding conflicts with kube-proxy

If you cannot disable `kube-proxy` (for example, because it is managed by your Kubernetes distribution), then you _must_ change Felix configuration parameter `BPFKubeProxyIptablesCleanupEnabled` to `false`. This can be done with `kubectl` as follows:
Expand Down
6 changes: 6 additions & 0 deletions calico/operations/ebpf/install.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -442,6 +442,12 @@ Then, should you want to start `kube-proxy` again, you can simply remove the nod
</TabItem>
</Tabs>

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

## Next steps

**Recommended**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ eBPF (or "extended Berkeley Packet Filter"), is a technology that allows safe mi
- [AKS with Azure CNI and Calico network policy](../../getting-started/kubernetes/managed-public-cloud/aks.mdx#install-aks-with-calico-for-network-policy) works, but it is not possible to disable kube-proxy resulting in wasted resources and suboptimal performance.
- [AKS with {{prodname}} networking](../../getting-started/kubernetes/managed-public-cloud/aks.mdx#install-aks-with-calico-networking) is in testing with the eBPF dataplane. This should be a better solution overall but, at time of writing, the testing was not complete.
- RKE (RKE2 recommended because it supports disabling `kube-proxy`)
- MKE

- Linux distribution/kernel:

Expand Down Expand Up @@ -149,6 +150,8 @@ The best way to do that varies by Kubernetes distribution:
`api-int.<cluster_name>.<base_domain>` should point to the API server or to the load balancer in front of the
API server. Use that (filling in the `<cluster_name>` and `<base_domain>` as appropriate for your cluster) for the
`KUBERNETES_SERVICE_HOST` below. Openshift uses 6443 for the `KUBERNETES_SERVICE_PORT`.
- MKE runs a reverse proxy in each node that can be used to reach the API server. You should use `proxy.local`
as the `KUBERNETES_SERVICE_HOST` and `6444` as the `KUBERNETES_SERVICE_PORT`.
- For AKS and EKS clusters you should use the FQDN of the API server's load balancer. This can be found with
```
kubectl cluster-info
Expand Down Expand Up @@ -289,6 +292,19 @@ To re-enable it:
kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}'
```

#### MKE

If you are running MKE, you can disable `kube-proxy` as follows:

Follow the step procedure in [Modify an existing MKE configuration](https://docs.mirantis.com/mke/current/ops/administer-cluster/configure-an-mke-cluster/use-an-mke-configuration-file.html#modify-an-existing-mke-configuration) to download, edit, and upload your MKE configuration. During the editing step, add the following configuration:
`kube_proxy_mode=disabled` and `kube_default_drop_masq_bits=true`.

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

### Avoiding conflicts with kube-proxy

If you cannot disable `kube-proxy` (for example, because it is managed by your Kubernetes distribution), then you _must_ change Felix configuration parameter `BPFKubeProxyIptablesCleanupEnabled` to `false`. This can be done with `kubectl` as follows:
Expand Down Expand Up @@ -381,5 +397,7 @@ To revert to standard Linux networking:
```
kubectl patch ds -n kube-system kube-proxy --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": null}}}}}'
```
2. If you are running MKE, follow the step procedure in [Modify an existing MKE configuration](https://docs.mirantis.com/mke/current/ops/administer-cluster/configure-an-mke-cluster/use-an-mke-configuration-file.html#modify-an-existing-mke-configuration) to download, edit, and upload your MKE configuration. During the editing step, add the following configuration:
`kube_proxy_mode` to `iptables`.

1. Since disabling eBPF mode is disruptive to existing connections, monitor existing workloads to make sure they re-establish any connections that were disrupted by the switch.
46 changes: 35 additions & 11 deletions calico_versioned_docs/version-3.28/operations/ebpf/install.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ and in particular, pushing the networking capabilities of the latest Linux kerne
- OpenShift
- EKS
- AKS
- MKE

- Linux distribution/kernel:

Expand Down Expand Up @@ -127,7 +128,7 @@ Select the appropriate tab below for distribution-specific instructions:
`kubeadm` supports a number of base OSes; as long as the base OS chosen (such as Ubuntu 20.04) meets the kernel
requirements, `kubeadm`-provisioned clusters are supported.

Since `kube-proxy` is not required in eBPF mode, you may wish to disable `kube-proxy` at install time. With `kubeadm`
Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `kubeadm`
you can do that by passing the ` --skip-phases=addon/kube-proxy` flag to `kubeadm init`:

```bash
Expand All @@ -140,7 +141,7 @@ kubeadm init --skip-phases=addon/kube-proxy
`kops` supports a number of base OSes; as long as the base OS chosen (such as Ubuntu 20.04 or RHEL 8.2) meets the kernel
requirements, `kops`-provisioned clusters are supported.

Since `kube-proxy` is not required in eBPF mode, you may wish to disable `kube-proxy` at install time. With `kops` you
Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `kops` you
can do that by setting the following in your `kops` configuration:

```yaml
Expand Down Expand Up @@ -172,6 +173,17 @@ default kernel used by Amazon Linux is recent enough to run eBPF mode, as is the
18.04 image did not have a recent-enough kernel (but that may have changed by the time you read this).

</TabItem>
<TabItem label="MKE" value="MKE-5">

The eBPF data plane is supported on MKE with any Linux operating system that meets the minimum kernel requirements.

Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `MKE` you
can do that by setting `--kube-proxy-mode=disabled` and `--kube-default-drop-masq-bits` when installing the cluster.

More details can be found in [the MKE documentation](https://docs.mirantis.com/mke/current/install/predeployment/configure-networking/cluster-service-networking-options.html)

</TabItem>

</Tabs>

### Create kubernetes-service-endpoint config map
Expand All @@ -188,7 +200,7 @@ scaled up/down. If you have multiple API servers, with DNS or other load balanci
the address of the load balancer. This prevents {{prodname}} from being disconnected if the API servers IP changes.

<Tabs>
<TabItem label="Generic or kubeadm" value="Generic or kubeadm-5">
<TabItem label="Generic or kubeadm" value="Generic or kubeadm-6">

If you created a cluster manually (for example by using `kubeadm`) then the right address to use depends on whether you
opted for a high-availability cluster with multiple API servers or a simple one-node API server.
Expand All @@ -202,21 +214,21 @@ opted for a high-availability cluster with multiple API servers or a simple one-
causing {{prodname}} to lose connectivity to the API server.

</TabItem>
<TabItem label="kOps" value="kOps-6">
<TabItem label="kOps" value="kOps-7">

When using `kops`, `kops` typically sets up a load balancer of some sort in front of the API server. You should use
the FQDN and port of the API load balancer: `api.internal.<clustername>`.

</TabItem>
<TabItem label="OpenShift" value="OpenShift-7">
<TabItem label="OpenShift" value="OpenShift-8">

OpenShift requires various DNS records to be created for the cluster; one of these is exactly what we need:
`api.<cluster_name>.<base_domain>` should point to the API server or to the load balancer in front of the
API server. Use that (filling in the `<cluster_name>` and `<base_domain>` as appropriate for your cluster) for the
`KUBERNETES_SERVICE_HOST` below. Openshift uses 6443 for the `KUBERNETES_SERVICE_PORT`.

</TabItem>
<TabItem label="AKS" value="AKS-8">
<TabItem label="AKS" value="AKS-9">

For AKS clusters, you should use the FQDN of your API server. This can be found by running the following command:

Expand All @@ -234,7 +246,7 @@ In this example, you would use `mycalicocl-calicodemorg-03a087-36558dbb.hcp.cana
`KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map.

</TabItem>
<TabItem label="EKS" value="EKS-9">
<TabItem label="EKS" value="EKS-10">

For an EKS cluster, it's important to use the domain name of the EKS-provided load balancer that is in front of the API
server. This can be found by running the following command:
Expand All @@ -253,6 +265,12 @@ Kubernetes master is running at https://60F939227672BC3D5A1B3EC9744B2B21.gr7.us-
In this example, you would use `60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com` for
`KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map.

</TabItem>
<TabItem label="MKE" value="MKE-11">

MKE runs a reverse proxy in each node which can be used to reach the api-server. `KUBERNETES_SERVICE_HOST` must be set to
`proxy.local` and `KUBERNETES_SERVICE_PORT` must be set to `6444`.

</TabItem>
</Tabs>

Expand Down Expand Up @@ -371,7 +389,7 @@ kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nod
Then, should you want to start `kube-proxy` again, you can simply remove the node selector.

</TabItem>
<TabItem label="kOps" value="kOps-11">
<TabItem label="kOps" value="kOps-12">

`kops` allows `kube-proxy` to be disabled by setting

Expand All @@ -383,7 +401,7 @@ kubeProxy:
in its configuration. You will need to do `kops update cluster` to roll out the change.

</TabItem>
<TabItem label="OpenShift" value="OpenShift-12">
<TabItem label="OpenShift" value="OpenShift-13">

In OpenShift, you can disable `kube-proxy` as follows:

Expand All @@ -398,7 +416,7 @@ kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"
```

</TabItem>
<TabItem label="AKS" value="AKS-13">
<TabItem label="AKS" value="AKS-14">

AKS with Azure CNI does not allow `kube-proxy` to be disabled, `kube-proxy` is deployed by the add-on manager, which will reconcile
away any manual changes made to its configuration. To ensure `kube-proxy` and {{prodname}} don't fight, set
Expand All @@ -410,7 +428,7 @@ kubectl patch felixconfiguration default --type merge --patch='{"spec": {"bpfKub
```

</TabItem>
<TabItem label="EKS" value="EKS-14">
<TabItem label="EKS" value="EKS-15">

In EKS, you can disable `kube-proxy`, reversibly, by adding a node selector that doesn't match and nodes to
`kube-proxy`'s `DaemonSet`, for example:
Expand All @@ -424,6 +442,12 @@ Then, should you want to start `kube-proxy` again, you can simply remove the nod
</TabItem>
</Tabs>

:::note

If you are running kube-proxy in IPVS mode, switch to iptables mode before disabling.

:::

## Next steps

**Recommended**
Expand Down