Clarify --node-ip vs --node-public-ip vs --flannel-iface

[intelfx]

Hi!

I'm (still) developing some automation around k3s management and it's not clear to me which network configurations a) are supported and b) translate to which combination of flags/configuration.

1. What exactly are --node-ip and --node-external-ip and when should I use either?
2. What is --flannel-iface and how does it relate to the two IP overrides above?

---

Or, more specifically: if I have two network interfaces on each node, where the first one (eth1) has a default route and the second one (eth2) is an isolated LAN connecting the nodes, is it possible to tell k3s to use the second interface for inter-node traffic and the first one only for cluster ingress/egress (because it might be less performant / more costly)?

Will the configuration differ depending on whether eth1 directly has a globally routable IPv4 vs. a private IPv4 behind a NAT with forwarded ports?

---

Or, by example:

Case 1

• node1: eth1 = 1.2.3.4, eth2 = 10.10.10.1/24
• node2: eth1 = 5.6.7.8, eth2 = 10.10.10.2/24

(Assume that node1 may reach node2 by either IP, but the second one is preferable.)

Case 2

• node1: eth1 = 10.1.2.3 (NATed from 1.2.3.4), eth2 = 10.10.10.1/24
• node2: eth1 = 10.5.6.7 (NATed from 5.6.7.8), eth2 = 10.10.10.2/24

(Assume that node1 may reach node2 either by 5.6.7.8 or 10.10.10.2 (second one being preferable), but not by 10.5.6.7.)

Apologies for a long-winded question.

[brandond]

--node-ip sets the node's private IP, overriding the default behavior of selecting the primary IP of the interface targeted by the default route.
--node-external-ip sets the node's external IP, for cases where the node is behind 1:1 NAT and has an external address. In cloud provider environments, this is usually discovered by the cloud controller via instance metadata, but this allows it to be set by hand in unmanaged environments. It is unset by default.
--flannel-iface sets the interface used by flannel for network overlay traffic.

There is also the --flannel-external-ip flag, which when set on the server, configures flannel's behavior on all nodes in the cluster. Flannel will target the node's external IPs for network overlay traffic, instead of the private IP. This is intended for use in environments where the nodes cannot reach each other on their private IPs.

[intelfx]

--node-ip sets the node's private IP, overriding the default behavior of selecting the primary IP of the interface targeted by the default route

In general, what should a node's private IP (and thus --node-ip) be set to (i.e. what's it used for)?

node-external-ip sets the node's external IP, for cases where the node is behind 1:1 NAT and has an external address

Is it required that --node-external-ip be set to the IP that is NATed to --node-ip (if any)? What will happen otherwise?

--flannel-iface sets the interface used by flannel for network overlay traffic.

And if it's not set, then flannel will use the interface of --node-ip? What happens if it is set — will flannel establish connections from --flannel-iface to other nodes' --node-ip, or to the respective IP discovered on --flannel-iface of other nodes?

---

Apologies once again, this all raises even more questions 😅

[brandond]

what should a node's private IP be set to

You don't have to set it at all, if you're happy with the default behavior of selecting the primary IP on the interface with the default route.

Is it required that --node-external-ip be set

You only need to set it if you want K3s to be aware of the external IP, and be able to use it for things like loadbalancer or flannel destination addresses.

And if it's not set, then flannel will use the interface of --node-ip

It will use the interface with that IP, correct.

will flannel establish connections from --flannel-iface to other nodes

Yes, it sets the interface that flannel both initiates traffic from, and expects traffic to. These are really more flannel questions though...

[intelfx]

You don't have to set it at all, if you're happy with the default behavior of selecting the primary IP on the interface with the default route.

I'm just trying to understand what it's intended for -- i.e. what are the consequences of choosing one or another.

if you're happy with the default behavior of selecting the primary IP on the interface with the default route.

My end goal is to configure k3s in such a way that all inter-node traffic goes via a specific isolated (i.e. non-default-route) interface, whereas the interface with the default route is only used for cluster ingress/egress.

I originally assumed that I could set --node-ip to the isolated LAN address and --external-node-ip to the public address, but if the latter is designed for 1:1 NAT cases, then I guess I need something else.

[appany-it]

@intelfx Did you find a solution to isolate overlay traffic?