Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move and rename package repos #2121

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Move and rename package repos #2121

wants to merge 2 commits into from

Conversation

chewi
Copy link
Contributor

@chewi chewi commented Jul 15, 2024

Move and rename package repos to repos/{gentoo-subset,flatcar-overlay}

Catalyst 4 has totally changed the way repositories are handled. It only works when the name of the directory containing the repository matches the configured name of that repository. We already renamed coreos to coreos-overlay accordingly, but we actually want entirely different names and more convenient locations too.

The repositories are now known as gentoo-subset and flatcar-overlay, and they live under scripts/repos. Using the same name as upstream Gentoo would have been problematic, and just "flatcar" would have looked awkward in documentation.

How to use

Simply enter a recent SDK and run sudo ./bootstrap_sdk. Note that this will automatically update the SDK container's Portage configuration, so you will need to recreate the container to go back to an older branch.

Testing done

I have built an SDK manually. I also now trigger a Jenkins build.

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

@chewi
Copy link
Contributor Author

chewi commented Jul 16, 2024

The two-phase Jenkins job against 4012.0.0 passed. I know there are conflicts, but the nature of this change means they will frequently appear, so I'll deal with them once this is reviewed.

Copy link

github-actions bot commented Jul 16, 2024

Build action triggered: https://github.com/flatcar/scripts/actions/runs/10073858410

@chewi
Copy link
Contributor Author

chewi commented Jul 16, 2024

I just rebased this following the big package update. That really wasn't fun, so I would appreciate eyes on this soon.

@jepio
Copy link
Member

jepio commented Jul 16, 2024 via email

@ader1990
Copy link
Contributor

@chewi would it be possible to have two commits? one commit with the file renames and one commit with the actual logic needed to be reviewed? my browser bails on me after I try to scroll a little. And this way would be cleaner if we d need to point to the logic change.

@jepio
Copy link
Member

jepio commented Jul 17, 2024

It’s a good idea to get everyone to consider what the possible side effects or breakage due to this change could be before we merge it.

Thought of another one - the workflows that are being touched in this PR are used to file PRs against branches for all channels. The updated repo paths in the workflows will only work for main, unless this gets cherry-picked to all branches, which is risky.

@chewi
Copy link
Contributor Author

chewi commented Jul 17, 2024

Good idea with splitting the commit. That might make rebasing easier too. Pushed that now.

I was aware of emerge-gitclone, so I'll take a look at that. I was also looking at the flatcar-build-scripts. I hadn't realised the workflow scripts were shared though.

I had been trying to avoid creating symlinks, as I prefer to work through the breakage rather than stay in a semi-migrated state for ages. For these cases though, symlinks may be the simplest answer. We can remove them once we this has fully rotated through all the release channels. What about LTS releases though, are they also affected?

@chewi
Copy link
Contributor Author

chewi commented Jul 19, 2024

See flatcar/flatcar-build-scripts#160 for fixes to the build scripts. These should ideally be merged first. They are backwards-compatible, so they don't need to wait for this.

This commit just moves all the unmodified files into their new
locations. The interesting changes are in the next commit.

Signed-off-by: James Le Cuirot <[email protected]>
Copy link
Member

@jepio jepio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first pass

update_metadata Outdated Show resolved Hide resolved
update_distfiles Outdated Show resolved Hide resolved
update_chroot Outdated
@@ -103,17 +103,17 @@ EMERGE_DEFAULT_OPTS="--verbose --oneshot"
source "/etc/portage/make.conf.user"
EOF

sudo_clobber "/etc/portage/repos.conf/portage-stable.conf" <<EOF
sudo_clobber "/etc/portage/repos.conf/gentoo-subset.conf" <<EOF
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't have to be split into separate files per repo definition, a single file with a stable name (flatcar.conf) will do and it will be easier to handle changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Catalyst 4 sets them up this way, so it's easiest to follow suit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we're regenerating/clobbering them here, so whatever catalyst did gets overwritten. Catalysts output/logic is opaque from the pov of these scripts, so I don't think its easiest to rely on it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're currently not deleting all the existing *.conf files though. We could do, but I think that would be impolite, especially given that we've been talking about custom overlays recently.

Copy link
Member

@jepio jepio Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the churn happening in this area, maybe deleting these conf files is the sane approach. /etc is part of the container image (same with /board/xxx/etc) so user modifications get lost by design.

Custom overlays will require a proper design and integration with scripts - perhaps the user will add an overlay in scripts/repos/.... and then this will be taken into account when generating the files, but either way - the configs are managed by scripts.

setup_board Outdated Show resolved Hide resolved
sdk_lib/sdk_entry.sh Outdated Show resolved Hide resolved
build_library/build_image_util.sh Outdated Show resolved Hide resolved
build_library/catalyst.sh Outdated Show resolved Hide resolved
sdk_container/src/third_party/coreos-overlay Outdated Show resolved Hide resolved
sdk_lib/sdk_entry.sh Outdated Show resolved Hide resolved
@chewi
Copy link
Contributor Author

chewi commented Jul 22, 2024

flatcar/flatcar-dev-util#19 will need to be merged first so that I can get a new version of emerge-gitclone into this PR.

Copy link
Contributor

@ader1990 ader1990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see at least 12 references for /mnt/host/source/src/scripts/repos -> maybe this can be added as a configuration path once in one file in a future patch.

build_library/catalyst.sh Show resolved Hide resolved
@chewi
Copy link
Contributor Author

chewi commented Jul 22, 2024

Adding flatcar/flatcar-linux-infra#280 to the pile.

Catalyst 4 has totally changed the way repositories are handled. It only
works when the name of the directory containing the repository matches
the configured name of that repository. We already renamed coreos to
coreos-overlay accordingly, but we actually want entirely different
names and more convenient locations too.

The repositories are now known as gentoo-subset and flatcar-overlay, and
they live under scripts/repos. Using the same name as upstream Gentoo
would have been problematic, and just "flatcar" would have looked
awkward in documentation.

I have removed code referencing /mnt/host/source/config rather than fix
it up, as this is no location is no longer used anywhere.

Signed-off-by: James Le Cuirot <[email protected]>
@chewi
Copy link
Contributor Author

chewi commented Jul 24, 2024

I'm putting this (and the associated PRs) on the ice for the time being.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Testing / in Review
Development

Successfully merging this pull request may close these issues.

4 participants