Inclusion of kubectl as a non-upstream binary/wrapper

[djryanj]

I am trying to understand the project's decision to use a non-upstream kubectl wrapper/binary.

At a glance, it seems like it serves no obvious purpose, and leads to some unexpected behaviours on the systems depending on configuration; specifically messages like WARN[0000] open /etc/rancher/k3s/config.yaml: permission denied and WARN[0000] open /etc/rancher/k3s/k3s.yaml: permission denied if you attempt to run the included kubectl as non-root/non-sudo on a system with those files secured (as is also the recommendation).

There are a couple different issues and some other discussions around the web about this, but near as I can tell it's not documented well (or at all) and when attempting to understand the decision here, most of the linked discussions end with one of two comments/suggestions:

• It needs to do that because it needs to know if the actual server binaries are somewhere else (e.g., using --data-dir), and we're not changing it, or;
• make the files more open using filesystem security.

My Thoughts on the Matter

Expanding on the points above:

1. Including a k3s wrapper does make sense, and needing it to know where the system binaries are located so as to do stuff with them is logical. It also makes sense to require e.g., sudo or root to be able to use it, since one can probably safely assume that anyone needing to interact directly with the running server on a non-rootless system is a sysadmin and is doing stuff that requires elevated privileges, so the issues regarding access to k3s.yaml and config.yaml will be moot.
2. However, the inclusion of a non-standard kubectl is opaque to the user and does not seem to be necessary.
   • The primary problem here is that the team regularly asserts that their kubectl wrapper needs to know where the system binaries are, but I cannot understand why. kubectl does not (or maybe it would be better to say should not) interact with the system binaries at all; it talks to the API server at whatever endpoint is defined in KUBECONFIG for the cluster.
   • If I had to guess, the team was attempting to solve for users on a system and their KUBECONFIG env var not being populated, so the wrapper automatically checks at /etc/rancher/k3s/k3s.yaml for that file first. However, this feels like a bizarre solution to an issue that has already been solved; to wit:
     • The installer script (or binary) could do what e.g., kubeadm does and inform the user that they need to handle getting the config file somewhere and setting KUBECONFIG. It's pretty common and standard in the Kubernetes world to need to deal with KUBECONFIG anyway.
     • PATH is a thing, and the installer, upon seeing a --data-dir as part of the install, could easily check for and add that to PATH if it's not already there. In fairness, this may be less portable than it seems on the surface; just throwing it out as a possibility.
3. To include my thoughts on filesystem security, it's unfortunate that many users will say "just chmod 777 it and it works" without understanding the impacts of doing that. It is not possible to control that, but when that advice is regularly tossed about as a solution it points to a problem that can be solved by the team.

Ultimately this focuses on user experience, and while I applaud the team for trying to improve overall user experience by simplifying things for users, I'm not really sure the end result is actually a better one overall mostly due to the opaqueness of what's been done. Expected behaviours just aren't there, and it's unclear why not.

I look forward to understanding this better.

[brandond]

The primary problem here is that the team regularly asserts that their kubectl wrapper needs to know where the system binaries are, but I cannot understand why.

You should be able to figure out the "why" of all of this yourself fairly easily.

Remember that the k3s binary that you download from github is not actually K3s. Nor is it kubectl, or crictl, or any of the other commands. It is just a self-extracting wrapper that unpacks the actual binaries used by a running node.

root@systemd-node-1:/# curl -sL get.k3s.io | INSTALL_K3S_SKIP_START=true sh -s
[INFO]  Finding release for channel stable
[INFO]  Using v1.31.3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.31.3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.31.3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.

root@systemd-node-1:/# ls -lah /usr/local/bin/kubectl /usr/local/bin/crictl /usr/local/bin/ctr /usr/local/bin/k3s
lrwxrwxrwx 1 root root   3 Dec 13 17:23 /usr/local/bin/crictl -> k3s
lrwxrwxrwx 1 root root   3 Dec 13 17:23 /usr/local/bin/ctr -> k3s
-rwxr-xr-x 1 root root 67M Dec 13 17:23 /usr/local/bin/k3s
lrwxrwxrwx 1 root root   3 Dec 13 17:23 /usr/local/bin/kubectl -> k3s

root@systemd-node-1:/# export K3S_DEBUG=true

root@systemd-node-1:/# /usr/local/bin/kubectl version --client
INFO[0000] Acquiring lock file /var/lib/rancher/k3s/data/.lock
INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
...
DEBU[0002] Verified hash k3s is correct
DEBU[0002] Verified link kubectl is correct
...
DEBU[0002] Asset dir /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
DEBU[0002] Running /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl [kubectl version --client]
Client Version: v1.31.3+k3s1
Kustomize Version: v5.4.2

root@systemd-node-1:/# ls -lah /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/k3s
-rwxr-xr-x 1 root root 176M Dec  5 21:29 /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/k3s
lrwxrwxrwx 1 root root    3 Dec 13 17:26 /var/lib/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl -> k3s

But what if I want to use /opt/rancher/k3s instead of /var/lib/rancher/k3s?

root@systemd-node-1:/# echo data-dir: /opt/rancher/k3s > /etc/rancher/k3s/config.yaml

root@systemd-node-1:/# /usr/local/bin/kubectl version --client
INFO[0000] Acquiring lock file /opt/rancher/k3s/data/.lock
INFO[0000] Preparing data dir /opt/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
...
DEBU[0002] Verified hash k3s is correct
DEBU[0002] Verified link kubectl is correct
...
DEBU[0002] Asset dir /opt/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
DEBU[0002] Running /opt/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl [kubectl version --client]
Client Version: v1.31.3+k3s1
Kustomize Version: v5.4.2

What about as a nonroot user?

sles@systemd-node-1:~> export K3S_DEBUG=true

sles@systemd-node-1:~> ls -la /etc/rancher/k3s/config.yaml
-rw-r--r-- 1 root root 27 Dec 13 17:30 /etc/rancher/k3s/config.yaml

sles@systemd-node-1:~> /usr/local/bin/kubectl version --client
DEBU[0000] Asset dir /opt/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
DEBU[0000] Running /opt/rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl [kubectl version --client]
Client Version: v1.31.3+k3s1
Kustomize Version: v5.4.2

But what if I can't read config.yaml as my nonroot user? I don't have write access to /var/lib/rancher/k3s and I can't read the config file, how will I know where to find the system binaries in /opt/rancher/k3s?
That's the neat part, you don't:

sles@systemd-node-1:~> export K3S_DEBUG=true

sles@systemd-node-1:~> ls -la /etc/rancher/k3s/config.yaml
-rw------- 1 root root 27 Dec 13 17:30 /etc/rancher/k3s/config.yaml

sles@systemd-node-1:~> /usr/local/bin/kubectl version --client
WARN[0000] open /etc/rancher/k3s/config.yaml: permission denied
INFO[0000] Acquiring lock file /home/sles/.rancher/k3s/data/.lock
INFO[0000] Preparing data dir /home/sles/.rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
...
DEBU[0002] Verified hash k3s is correct
DEBU[0002] Verified link kubectl is correct
...
DEBU[0002] Asset dir /home/sles/.rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
WARN[0002] open /etc/rancher/k3s/config.yaml: permission denied
DEBU[0002] Running /home/sles/.rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl [kubectl version --client]
Client Version: v1.31.3+k3s1
Kustomize Version: v5.4.2

Oh but I don't like the warnings about not being able to read the system config file, they bother me.

sles@systemd-node-1:~> export K3S_CONFIG_FILE=$HOME/.rancher/k3s/config.yaml

sles@systemd-node-1:~> /usr/local/bin/kubectl version --client
DEBU[0000] Asset dir /home/sles/.rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17
DEBU[0000] Running /home/sles/.rancher/k3s/data/3b5b4bda92acada40dbf3251e7e040c82cd2a79760260b5cec3e42f7c1cd0a17/bin/kubectl [kubectl version --client]
Client Version: v1.31.3+k3s1
Kustomize Version: v5.4.2

[djryanj]

Thanks for taking the time to reply.

This may be semantics, but I didn't realize that in all the different discussions that kubectl as installed is actually a symlink to k3s which includes the kubectl functionality and that kubectl is actually "unpacked" each time it's run (or cached for the next time at that long location). I know that you and other members of the team have said parts of that in all the stuff I've read, but it didn't click.

Is that an accurate statement?

[brandond]

I didn't realize that in all the different discussions that kubectl as installed is actually a symlink to k3s

The install script itself tells you that it's creating a symlink, is that not a good clue? If you already have something at /usr/local/bin/kubectl it will leave it alone and skip creating the symlink (and also tell you that it is doing so).

kubectl is actually "unpacked" each time it's run (or cached for the next time at that long location).

Yes, that is what happens. Each release of K3s will unpack its bundled binaries to uniquely named directory. This includes both the k3s multicall binary (as the docs say: "It wraps Kubernetes and other components in a single, simple launcher") along with the userspace binaries(docs: "Host utilities such as iptables/nftables, ebtables, ethtool, & socat").

The k3s docker image skips the self-extracting step, and just has the multicall binary and userspace as the image content.

[djryanj]

🤦you know in all of the times I've installed k3s, I never noticed the [INFO] Creating /usr/local/bin/kubectl symlink to k3s line.

RTFM I guess, and now I can see the design decision more clearly.

I would still like to submit that slightly clearer documentation on it might be possible. Is the docs site PR-able? Would you accept one if I submitted one? Or is this enough and can be quashed since I am the only one with the issue (lol)?

[brandond]

Is the docs site PR-able?

Click the "Edit this page" button at the bottom of any page on the docs site.

[djryanj]

@brandond wanted to say thanks again for taking the time to answer this. I'll see about submitting some possible changes to the docs to help those like myself who are just smart enough to be dangerous but not smart enough to figure the rest out for themselves.