Production file-based catalog specific for OCP versions #2926

mgencur · 2024-10-08T13:07:44Z

https://issues.redhat.com/browse/SRVCOM-3369

Proposed Changes

mgencur · 2024-10-08T13:12:41Z

The build error is cause by the fact that two versions replace the same previous version:

{
            "name": "serverless-operator.v1.33.2",
            "replaces": "serverless-operator.v1.33.0",
            "skipRange": ">=1.33.0 <1.33.2"
        },
        {
            "name": "serverless-operator.v1.35.0",
            "replaces": "serverless-operator.v1.34.0",
            "skipRange": ">=1.34.0 <1.35.0"
        },
        {
            "name": "serverless-operator.v1.34.0",
            "replaces": "serverless-operator.v1.33.0",
            "skipRange": ">=1.33.0 <1.34.0"
        }

mgencur · 2024-10-09T10:09:08Z

Getting this error in the github action:

401 Unauthorized" host=registry.redhat.io
time="2024-10-09T10:08:07Z" level=fatal msg="render catalog image: render reference

Probably we don't have the right credentials yet.

.github/workflows/validate.yaml

pierDipi · 2024-10-09T12:23:43Z

.github/workflows/validate.yaml

+        env:
+          REGISTRY_REDHAT_IO_USERNAME: ${{ secrets.REGISTRY_REDHAT_IO_USERNAME }}
+          REGISTRY_REDHAT_IO_PASSWORD: ${{ secrets.REGISTRY_REDHAT_IO_PASSWORD }}
+        run: make generated-files generate-catalog


I'd include generate-catalog in generated-files so that others don't need to learn about this new command to run ?

The command "generate-catalog" takes a looot of time. About 1/2 hour on my localhost. Maybe it will be faster in the github action but let's see.
The other problem is that it requires credentials to registry.redhat.io and those are not available in github actions that run "on-pull-request".
I'm going to push a little change to split this. We could run "generate-catalog" only periodically or on-push.

Pushed the split.

wow 1/2 hour

Alright. In github action it's much faster. About 5 minutes:

11:28:54.752 INFO: Generating catalog for OCP 4.13 time="2024-10-11T11:28:54Z" level=info msg="rendering index \"registry.redhat.io/redhat/redhat-operator-index:v4.13\" as file-based catalog" time="2024-10-11T11:29:49Z" level=info msg="wrote rendered file-based catalog to \"/tmp/knative.aIjkdvzz/tmp.IiMQGlnAOi\"\n" 11:30:27.698 INFO: Generating catalog for OCP 4.14 time="2024-10-11T11:30:27Z" level=info msg="rendering index \"registry.redhat.io/redhat/redhat-operator-index:v4.14\" as file-based catalog" time="2024-10-11T11:31:21Z" level=info msg="wrote rendered file-based catalog to \"/tmp/knative.aIjkdvzz/tmp.s4bv1h7HZi\"\n" 11:31:39.478 INFO: Generating catalog for OCP 4.15 time="2024-10-11T11:31:39Z" level=info msg="rendering index \"registry.redhat.io/redhat/redhat-operator-index:v4.15\" as file-based catalog" time="2024-10-11T11:32:31Z" level=info msg="wrote rendered file-based catalog to \"/tmp/knative.aIjkdvzz/tmp.vOKEBnWFRH\"\n" 11:32:48.077 INFO: Generating catalog for OCP 4.16 time="2024-10-11T11:32:48Z" level=info msg="rendering index \"registry.redhat.io/redhat/redhat-operator-index:v4.16\" as file-based catalog" time="2024-10-11T11:33:36Z" level=info msg="wrote rendered file-based catalog to \"/tmp/knative.aIjkdvzz/tmp.mKSK5ppXVI\"\n" 11:33:57.680 INFO: Generating catalog for OCP 4.17 time="2024-10-11T11:33:57Z" level=info msg="rendering index \"registry.redhat.io/redhat/redhat-operator-index:v4.17\" as file-based catalog" time="2024-10-11T11:34:12Z" level=info msg="wrote rendered file-based catalog to \"/tmp/knative.aIjkdvzz/tmp.TB6GDPKkiP\"\n"

pierDipi · 2024-10-09T12:27:15Z

olm-catalog/serverless-operator-index/v4.14/catalog/serverless-operator/catalog.json

This one doesn't seem to be a valid JSON, there are missing commas

and I guess the same for others, but if that's normal, nevermind my noob questions

In the same file I see old CI images

{ "name": "knative-openshift-ingress", "image": "registry.ci.openshift.org/knative/release-1.34.0:serverless-ingress" }, { "name": "knative-openshift", "image": "registry.ci.openshift.org/knative/release-1.34.0:serverless-knative-operator" }, { "name": "knative-operator", "image": "registry.ci.openshift.org/knative/release-1.34.0:serverless-openshift-knative-operator" },

shouldn't there be prod images only in this file as that's part of the final release?

I got the same, when I created them as described https://github.com/konflux-ci/olm-operator-konflux-sample/blob/main/docs/konflux-onboarding.md#create-the-fbc-in-the-git-repository

btw: those commands also offer a --output yaml parameter. So the catalog-templat might be easier to maintain later.

what about that comment ? #2926 (comment)

is it correct / what we expect ?

I guess it's here the reply

This will be automatically fixed later. When 1.34 is pushed to the production catalog it will be pulled when calling "generate-catalog" and the script will skip this version and won't add CI images there.

Yes. That's the answer.

Pushed the yaml variant ! I like it much more.

pierDipi · 2024-10-09T12:32:41Z

hack/lib/images.bash

+function latest_registry_ci_sha() {
+  input=${1:?"Provide image"}
+
+  image_without_tag=${input%:*} # Remove tag, if any
+
+  go_bin="$(go env GOPATH)/bin"
+  export GOPATH="$PATH:$go_bin"
+  digest=$(skopeo inspect --no-tags=true "docker://${input}" | jq -r '.Digest')
+  if [ "${digest}" = "" ]; then
+    exit 1
+  fi
+
+  echo "${image_without_tag}@${digest}"
+}


I think we could reuse latest_konflux_image_sha for this ?

I will check this. We don't want the "latest" tag though. We want a specific tag, e.g. release-1.34.0. The latest_konflux_image_sha automatically adds "latest"

creydr · 2024-10-09T13:16:17Z

hack/lib/images.bash

+  image_without_tag=${input%:*} # Remove tag, if any
+
+  go_bin="$(go env GOPATH)/bin"
+  export GOPATH="$PATH:$go_bin"


not sure, why this is needed?

I'm not sure. I copied it from the other functions in this file. Looks like it works without this as well. Maybe @pierDipi knows?

Removed the line in the new commit.

not everyone has the go/bin folder in their path and go install installs binaries in go/bin but I'm ok making it a prerequisite

.github/workflows/validate.yaml

mgencur · 2024-10-10T14:09:42Z

I built the catalog image locally and deployed to OpenShift, and it works!

After building the image I just ran these two commands that install the CatalogSource.

export INDEX_IMAGE=quay.io/mgencur/serverless-index:release-1.35.0-4.17
./hack/dev.sh

pierDipi · 2024-10-10T14:33:23Z

Awesome!

creydr · 2024-10-10T14:35:51Z

Awesome 🎉

pierDipi · 2024-10-10T14:51:49Z

/lgtm

openshift-ci-robot · 2024-10-10T18:31:01Z

/retest-required

Remaining retests: 0 against base HEAD dc130e1 and 2 for PR HEAD 4783016 in total

openshift-ci-robot · 2024-10-11T02:09:50Z

/retest-required

Remaining retests: 0 against base HEAD dc130e1 and 2 for PR HEAD 4783016 in total

mgencur · 2024-10-11T05:44:42Z

One thing that is missing from this PR is replacing konflux reference to the bundle image with registry.redhat.io (after generating the catalog). And also deploying ImageContentSourcePolicy for this. But I will do this in a follow-up PR.

I had to do a rebase because of some other changes so I added a couple of commits and fixed the above as well.

creydr

/lgtm

openshift-ci · 2024-10-11T06:43:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: creydr, mgencur, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [creydr,mgencur,pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

creydr · 2024-10-11T09:21:36Z

/test 416-test-upgrade-aws-416

openshift-ci bot added the do-not-merge/work-in-progress label Oct 8, 2024

openshift-merge-robot added the needs-rebase label Oct 8, 2024

openshift-ci bot requested review from matzew and pierDipi October 8, 2024 13:07

openshift-ci bot added the approved label Oct 8, 2024

mgencur force-pushed the production_catalog branch from 6a4c17d to 18ba81e Compare October 9, 2024 09:19

openshift-merge-robot removed the needs-rebase label Oct 9, 2024

creydr reviewed Oct 9, 2024

View reviewed changes

.github/workflows/validate.yaml Outdated Show resolved Hide resolved

pierDipi reviewed Oct 9, 2024

View reviewed changes

creydr mentioned this pull request Oct 9, 2024

Allow to pass opm image as a build arg #2911

Closed

creydr reviewed Oct 9, 2024

View reviewed changes

pierDipi reviewed Oct 10, 2024

View reviewed changes

.github/workflows/validate.yaml Show resolved Hide resolved

openshift-merge-robot added the needs-rebase label Oct 10, 2024

mgencur changed the title ~~[WIP] Production file-based catalog specific for OCP versions~~ Production file-based catalog specific for OCP versions Oct 10, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Oct 10, 2024

mgencur force-pushed the production_catalog branch from 59eda4d to 4783016 Compare October 10, 2024 13:49

openshift-merge-robot removed the needs-rebase label Oct 10, 2024

openshift-ci bot assigned pierDipi Oct 10, 2024

openshift-ci bot added the lgtm label Oct 10, 2024

mgencur added 2 commits October 11, 2024 06:20

Basic template

113f576

Generating for single OCP version

858a008

mgencur added 20 commits October 11, 2024 06:22

Clean up

696cfe2

Get to original state

d4f4564

Handle bundle with same replaces

f0ced10

Keep original index Dockerfile for CI

b1cfbb6

Re-generate 4.17 catalog

73f2577

Fix for comparing versions

f23b75e

Re-generate dockerfiles

60faed3

Regenerate all catalogs

37db37c

Bring back original index.yaml and Dockerfile for CI

256d9a3

Remove old directory structure

8254c84

Do not copy over Dockerfile for CI

1d22232

Download amd64 opm for github action

1849f46

Fix github action

7e0ea5a

Reeuse code for getting image sha

39167fb

Use yaml

3258e8b

Run generate-catalog in bump.yaml

0325bc2

Generate yaml catalogs

c5c6e87

Minor fix

c2bb044

Replace bundle image with registry.redhat.io variant

8871d2d

Regenerate catalog

c2e5e66

mgencur force-pushed the production_catalog branch from 4783016 to c2e5e66 Compare October 11, 2024 05:42

openshift-ci bot removed the lgtm label Oct 11, 2024

creydr reviewed Oct 11, 2024

View reviewed changes

openshift-ci bot assigned creydr Oct 11, 2024

openshift-ci bot added the lgtm label Oct 11, 2024

creydr mentioned this pull request Oct 11, 2024

Add FBC applications & components openshift-knative/hack#334

Merged

openshift-merge-bot bot merged commit 51a4642 into openshift-knative:main Oct 11, 2024
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Production file-based catalog specific for OCP versions #2926

Production file-based catalog specific for OCP versions #2926

mgencur commented Oct 8, 2024 •

edited

Loading

mgencur commented Oct 8, 2024

mgencur commented Oct 9, 2024

pierDipi Oct 9, 2024

mgencur Oct 9, 2024 •

edited

Loading

mgencur Oct 9, 2024

pierDipi Oct 10, 2024

mgencur Oct 11, 2024

pierDipi Oct 9, 2024

pierDipi Oct 9, 2024

pierDipi Oct 9, 2024

pierDipi Oct 9, 2024

creydr Oct 9, 2024

pierDipi Oct 10, 2024

pierDipi Oct 10, 2024

pierDipi Oct 10, 2024

mgencur Oct 10, 2024

mgencur Oct 10, 2024

pierDipi Oct 9, 2024

mgencur Oct 9, 2024

mgencur Oct 10, 2024

creydr Oct 9, 2024

mgencur Oct 10, 2024

mgencur Oct 10, 2024

pierDipi Oct 10, 2024

mgencur commented Oct 10, 2024

pierDipi commented Oct 10, 2024

creydr commented Oct 10, 2024

pierDipi commented Oct 10, 2024

openshift-ci-robot commented Oct 10, 2024

openshift-ci-robot commented Oct 11, 2024

mgencur commented Oct 11, 2024

creydr left a comment

openshift-ci bot commented Oct 11, 2024

creydr commented Oct 11, 2024

Production file-based catalog specific for OCP versions #2926

Production file-based catalog specific for OCP versions #2926

Conversation

mgencur commented Oct 8, 2024 • edited Loading

Proposed Changes

mgencur commented Oct 8, 2024

mgencur commented Oct 9, 2024

Choose a reason for hiding this comment

mgencur Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgencur commented Oct 10, 2024

pierDipi commented Oct 10, 2024

creydr commented Oct 10, 2024

pierDipi commented Oct 10, 2024

openshift-ci-robot commented Oct 10, 2024

openshift-ci-robot commented Oct 11, 2024

mgencur commented Oct 11, 2024

creydr left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Oct 11, 2024

creydr commented Oct 11, 2024

mgencur commented Oct 8, 2024 •

edited

Loading

mgencur Oct 9, 2024 •

edited

Loading