Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Build own docker image and use ghcr #11768

Closed
wants to merge 63 commits into from
Closed

Conversation

faddat
Copy link
Contributor

@faddat faddat commented Apr 25, 2022

Description

Closes: #11657
Closes: #11688

This PR changes the Dockerfile in the repository root to a rolling-release Docker image that is built here and keeps itself up to date.

Simapp and other docker images that come from the SDK could start with this.

In CI, we could set it up to build this, and push it only when it is on main. The main branch's docker image becomes canonical for current and past versions of the cosmos-sdk, and can be used in tm-db and iavl testing, too.

Merging this

This PR comes in two stages:

  1. is merged to main and then we wait 60ish minutes to make sure cron is working
  2. when cron is verified working, we change all other images to consume ghcr.io/cosmos/cosmos-sdk, and this will make build times as least as fast as they were previously, but every image will be built and stored here, and updated every time that code is pushed to main.

Once this is merged we can immediately clear dependencies that aren't in this repository because we will get ghcr.io/cosmos/cosmos-sdk

images.

Matienance

These images do not have a tagged go version. Like developer laptops, they will always use the latest go, the latest rocksdb, and the latest leveldb. This way, our CI system will more closely reflect a developer laptop.


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title
  • added ! to the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • followed the guidelines for building modules
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • included comments for documenting Go code
  • updated the relevant documentation or specification
  • reviewed "Files changed" and left comments if necessary
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic
  • reviewed API design and naming
  • reviewed documentation is accurate
  • reviewed tests and test coverage
  • manually tested (if applicable)

@faddat faddat changed the title Update Dockerfile feat: Dockerfile Apr 25, 2022
@faddat faddat marked this pull request as draft April 25, 2022 22:30
@faddat
Copy link
Contributor Author

faddat commented Apr 25, 2022

Steps to putting this into action:

  • Decide if pinning versions is a hard requirement
  • push this to ghcr.io/cosmos/cosmos-sdk when it hits master
  • base other images on this one, throughout cosmos
    • simapp
    • simd
    • protobufs build image

Breaking is a feature, not a bug. When Go is out of date on my laptop, it yells at me, and I update it, and I imagine that is the same for >95% of developers using the cosmos-sdk

To enhance this property, we could use a cron job to build and push this image.

The possible downside to this is that we would need to keep our dependencies up to date.

to test this out:

docker pull faddat/archlinux
docker run faddat/archlinux /bin/bash
pacman -Syyu go rocksdb cleveldb git python base-devel
git clone https://github.com/cosmos/cosmos-sdk

build with rocksdb enabled, if you'd like to -- but use grocksdb branch

@github-actions github-actions bot added the T: CI label Apr 26, 2022
@faddat
Copy link
Contributor Author

faddat commented Apr 26, 2022

NB: in places where I'm using the "blank" image, we can change to the one in the repo root post merge

@faddat faddat changed the title feat: Dockerfile feat: Dockerfiles Apr 26, 2022
@faddat faddat changed the title feat: Dockerfiles feat: Build own docker image and use ghcr May 1, 2022
@faddat
Copy link
Contributor Author

faddat commented May 1, 2022

This PR eliminates the dependency on faddat/archlinux-docker

@@ -0,0 +1,22 @@
# Server list generated by rankmirrors on 2020-04-17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we create a new repo for this fork. IM weary of adding it directly in the sdk.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because of the mirrorlist?

I could update it. The reason that I'd like to put it here is that if it is here, we build a complete multiplatform development environmentn in the cosmos sdk each hour and in the future developers (like me) will not have to run around the internet trying to figure out where the db build system image is built (oop, it was tmdb's or wait was it iavl and who is tendermint-dev, anyway?)

@faddat faddat marked this pull request as draft May 10, 2022 13:30
@faddat
Copy link
Contributor Author

faddat commented May 10, 2022

There's still too much cruft here. I will clean it up and then bring it out of draft.

@tac0turtle
Copy link
Member

@faddat did you have a chance to talk with rick? His idea of layers would make this pr able to be pushed across the finish line

@@ -0,0 +1,35 @@
Server = https://mirror.arizona.edu/archlinux/$repo/os/$arch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, if we take the path of making our own image from scratch. They are the mirror lists for packages.


FROM golang:1.18-alpine
FROM faddat/archlinux
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this image needs changing? Seems fine here since this isn't used anywhere other than proto generation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is to move down to a single image that we use everywhere, so that there are fewer layers. We need to bootstrap, though. This is one of many urls that would change to something like ghcr.io/cosmos/cosmos-sdk or such.

@tac0turtle
Copy link
Member

@faddat any chance you want to complete this? otherwise lets close it

@faddat
Copy link
Contributor Author

faddat commented Jul 2, 2022

Yes, I do want to complete it.

@faddat faddat marked this pull request as ready for review July 2, 2022 19:36
@faddat
Copy link
Contributor Author

faddat commented Aug 8, 2022

This PR is kinda-sorta blocking #12842

Basically, this one will let us publish an up to date build environment from the main branch. In this case that'd be go 1.19, but I have seen others.

I am amenable to moving this to another repo, though I do basically maintain it at https://github.com/notional-labs/containers

Just want to again note that merging this means that we depend on an external repo-- but for about 1 hour. Once it is merged, we get our own images hosted on ghcr.

Those images stay up to date because we will be using a rolling-release linux, and they're made once an hour.

@tac0turtle
Copy link
Member

closing this for the time being. Doing a close pr round to clean up things

@tac0turtle tac0turtle closed this Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Docker in the Cosmos SDK
3 participants