✨ Add bootCommands to cloud-init file generation #11271

davidumea · 2024-10-08T14:17:23Z

What this PR does / why we need it:
Adds the ability to provide bootcmd commands to cloud-init via the KubeadmConfig or KubeadmConfigTemplates custom resources. (Same for KubeadmControlPlane)

One thing that I know that does not work at the time of creating this PR is validating/mutating webhooks for KubeadmConfig and KubeadmConfigTemplates. Some pointers to where to change that would be appreciated.

Which issue(s) this PR fixes
Fixes #

k8s-ci-robot · 2024-10-08T14:17:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-10-08T14:17:32Z

Welcome @davidumea!

It looks like this is your first PR to kubernetes-sigs/cluster-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-10-08T14:17:33Z

Hi @davidumea. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chrischdi · 2024-10-08T14:21:48Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

davidumea · 2024-10-09T12:28:47Z

Are there version constraints around bootcmd? Does it already exist in cloud-init for a long time?

Edit: looks like it is already there for a long time (from the start?) at 0.7.7 https://cloudinit.readthedocs.io/en/0.7.7/topics/examples.html

Exactly I don't think that should be an issue as bootcmd has been part of cloud-init for a long time.

What happens if we render it but the cloud-init version in the machine does not support it? Does it make cloud-init fail as a whole or is it just ignored?

During my testing, if the bootcmd module fails e.g. due to invalid commands, cloud-init still continues with the following modules. I haven't been able to test running this with a cloud-init version that doesn't support bootcmd.

davidumea · 2024-10-17T12:30:16Z

@chrischdi Is there anything I can do to get this reviewed?

fabriziopandini · 2024-10-18T08:37:00Z

/ok-to-test

cc @enxebre @vincepri @randomvariable

randomvariable

Bootcmd module has been in cloud-init for as long as I can remember (which is back to 2010), so don't see an issue there.

I would like the documentation to be more explicit though, particularly about how this will have no effect on Ignition. Have suggested some changes.

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

neolit123 · 2024-10-18T15:05:09Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


shouldn't it be added only to v1beta1 which is the actively maintained API version?

If I remove this and run make verify, it removes bootCommands from the CRDs.

Might be an issue in the verify logic, what do you think?

diff --git a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml index 79c3b19e0..4e1852f77 100644 --- a/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml +++ b/bootstrap/kubeadm/config/crd/bases/bootstrap.cluster.x-k8s.io_kubeadmconfigs.yaml @@ -47,17 +47,6 @@ spec: KubeadmConfigSpec defines the desired state of KubeadmConfig. Either ClusterConfiguration and InitConfiguration should be defined or the JoinConfiguration should be defined. properties: - bootCommands: - description: |- - BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd - module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition. - items: - description: BootCommand defines input for each bootcmd command - in cloud-init. - items: - type: string - type: array - type: array clusterConfiguration: description: ClusterConfiguration along with InitConfiguration are the configurations necessary for the init command

etc

New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

I reverted my changes to the v1alpha3 and v1alpha4 versions and added conversions instead a9c426a

If I remove this and run make verify, it removes bootCommands from the CRDs.

Disregard this, I didn't realize that all versions of the crds were in the same manifest.

fabriziopandini · 2024-10-21T08:36:03Z

internal/apis/bootstrap/kubeadm/v1alpha3/kubeadmconfig_types.go

+	// BootCommands specifies extra commands to run very early in the boot process via the cloud-init bootcmd
+	// module. This is typically run in the cloud-init.service systemd unit. This has no effect in Ignition.
+	// +optional
+	BootCommands []BootCommand `json:"bootCommands,omitempty"`


New fields should not be added to older API versions.
Conversions should be implemented instead.
See eg

cluster-api/internal/apis/bootstrap/kubeadm/v1alpha3/conversion.go

Lines 47 to 56 in 3db18fb

dst.Files = restored.Files

dst.Users = restored.Users

if restored.Users != nil {

for i := range restored.Users {

if restored.Users[i].PasswdFrom != nil {

dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom

}

}

}

fabriziopandini · 2024-10-21T08:37:14Z

bootstrap/kubeadm/internal/cloudinit/cloudinit_test.go

@@ -34,6 +34,7 @@ func TestNewInitControlPlaneAdditionalFileEncodings(t *testing.T) {
 	cpinput := &ControlPlaneInput{
 		BaseUserData: BaseUserData{
 			Header:              "test",
+			BootCommands:        nil,


Can we have a test for BootCommands not nil for init, join CP, join workers?

I added non-nil values for BootCommands in TestNewInitControlPlaneCommands and TestNewJoinControlPlaneAdditionalFileEncodings, did you want me to add more tests or is this enough? I couldn't find one for "join workers" daf4f97

fabriziopandini · 2024-10-21T08:39:02Z

controlplane/kubeadm/internal/filters_test.go

@@ -548,7 +548,7 @@ func TestMatchInitOrJoinConfiguration(t *testing.T) {
    },
    JoinConfiguration: nil,
    Files:             nil,
-    ... // 10 identical fields
+    ... // 11 identical fields


I need to take some time to dig into impact of this change on existing clusters, because we need to make sure this change does not trigger rollouts
However, I'm not sure when I will get to it 😓

Do you think it's feasible to have this included in the next minor release?

… PostKubeadmCommands

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Oct 8, 2024

k8s-ci-robot requested a review from elmiko October 8, 2024 14:17

k8s-ci-robot requested a review from enxebre October 8, 2024 14:17

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 8, 2024

randomvariable suggested changes Oct 18, 2024

View reviewed changes

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go Outdated Show resolved Hide resolved

davidumea requested a review from randomvariable October 18, 2024 13:45

fabriziopandini added the area/provider/bootstrap-kubeadm Issues or PRs related to CAPBK label Oct 18, 2024

k8s-ci-robot removed the do-not-merge/needs-area PR is missing an area label label Oct 18, 2024

neolit123 reviewed Oct 18, 2024

View reviewed changes

fabriziopandini reviewed Oct 21, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 1, 2024

davidumea added 6 commits November 5, 2024 09:16

Add bootCommands cloud-init file generation

31406da

fixup: go lint

97311d6

fixup: lint: added missing newline

4367074

fixup: add changes from verify-gen

0c23af8

fixup: filter tests

504dfd5

fixup: clearer documentation for BootCommands, PreKubeadmCommands and…

19d52df

… PostKubeadmCommands

davidumea added 4 commits November 5, 2024 09:17

fixup: verify: reflect new documentation in the crds

51068a2

fixup: restore v1alpha3,v1alpha4 and add conversions

8ed0171

fixup: non-nil values for bootcommands in cloudinit tests

332e78d

fixup: lint

28edd6c

davidumea force-pushed the david/bootcommand-upstream branch from ebc85c2 to 28edd6c Compare November 5, 2024 08:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 5, 2024

fixup: verify crds

8918ff3

davidumea requested review from fabriziopandini and neolit123 November 18, 2024 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add bootCommands to cloud-init file generation #11271

✨ Add bootCommands to cloud-init file generation #11271

davidumea commented Oct 8, 2024 •

edited

Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

neolit123 Oct 18, 2024

davidumea Oct 21, 2024

fabriziopandini Oct 21, 2024

davidumea Oct 22, 2024

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Oct 21, 2024

fabriziopandini Oct 21, 2024

davidumea Oct 22, 2024 •

edited

Loading

fabriziopandini Oct 21, 2024

davidumea Nov 5, 2024 •

edited

Loading

	dst.Files = restored.Files

	dst.Users = restored.Users
	if restored.Users != nil {
	for i := range restored.Users {
	if restored.Users[i].PasswdFrom != nil {
	dst.Users[i].PasswdFrom = restored.Users[i].PasswdFrom
	}
	}
	}

✨ Add bootCommands to cloud-init file generation #11271

Are you sure you want to change the base?

✨ Add bootCommands to cloud-init file generation #11271

Conversation

davidumea commented Oct 8, 2024 • edited Loading

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

k8s-ci-robot commented Oct 8, 2024

chrischdi commented Oct 8, 2024 • edited Loading

davidumea commented Oct 9, 2024 • edited Loading

davidumea commented Oct 17, 2024

fabriziopandini commented Oct 18, 2024

randomvariable left a comment

Choose a reason for hiding this comment

neolit123 Oct 18, 2024

Choose a reason for hiding this comment

davidumea Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

davidumea Oct 22, 2024

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

davidumea Oct 22, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziopandini Oct 21, 2024

Choose a reason for hiding this comment

davidumea Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

davidumea commented Oct 8, 2024 •

edited

Loading

chrischdi commented Oct 8, 2024 •

edited

Loading

davidumea commented Oct 9, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

davidumea Oct 22, 2024 •

edited

Loading

davidumea Nov 5, 2024 •

edited

Loading