This repository contains resources to deploy and practice Openshift Pipelines.
- 1. Introduction
- 2. TL;DR Full Installation
- 3. Secrets for authentication
- 4. Monitoring
- 5. Useful Tasks and Pipelines
- 6. Running pipelines
- 7. Triggers
- Annex A: Installing Tekton CLI
- Annex B: Tutorial
- Annex C: Default ClusterTasks
- Annex D: Retrieve the etc-pki-entitlement secret
- Annex E: Run Tekton Pipelines on Quay Image Mirroring Notifications
Red Hat OpenShift Pipelines is a cloud-native, continuous integration and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details.
What do we understand as Cloud Native CI/CD?
-
Containers: Built for container apps and runs on Kubernetes.
-
Serverless: Runs serverless with no CI/CD engine to manage and maintain.
-
DevOps: Designed with microservices and distributed teams in mind.
To get more details, you can check the official documentation.
|
ℹ️
|
This repository has been fully migrated to the GitOps pattern. This means that it is strongly recommended to deploy ArgoCD in order to deploy these components in an standard way. You can install GitOps using this repository. |
As this is a public repo, it is not possible to upload all the credentials freely to the git repository. For that reason, the following script will install the Pipelines operator and configure it. After that, it will configure Grafana dashboards if Grafana is installed and create the appropriate secrets to access all the resources. Please, execute the following script:
./auto-install.sh|
❗
|
This task still lacks the configuration of the secrets. For that, you will need to create the
|
This is the folder structure you should have after configuring the secrets:
secrets/
├── entitlement-key.pem
├── entitlement.pem
├── git-secret-ssh.yaml
├── kustomization.yaml
├── quay-alopezme-pull-secret.yaml
└── rh-registry-sa.yamlThe content of the kustomization.yaml should be:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ocp-pipelines
resources:
# Red Hat Registry
- rh-registry-sa.yaml
# Quay Robot Account
- quay-alopezme-pull-secret.yaml
# Git repositories
- git-secret-ssh.yaml
secretGenerator:
- name: etc-pki-entitlement
files:
- entitlement-key.pem
- entitlement.pem
options:
disableNameSuffixHash: trueIn many practical use cases, you might need to pull from private Git repositories or might need to push to an external container registry such as Quay.io. In this section, we will summarize how to create the Secrets to configure all the credentials.
To use the registry.redhat.io registry, you have to have a Red Hat login. To consume container images from registry.redhat.io in shared environments such as OpenShift, it is recommended for an administrator to use a Registry Service Account, also referred to as authentication tokens, in place of an individual’s Customer Portal credentials.
The management of Service Accounts is available via the Registry Service Account management application.
apiVersion: v1
kind: Secret
metadata:
name: $SECRET_NAME
data:
.dockerconfigjson: $DOCKER_CONFIG_FILE_CONTENT
type: kubernetes.io/dockerconfigjsonOnce you create the file with its contents, you can apply it to the cluster like this:
oc apply -f secrets/rh-registry-sa.yaml -n ocp-pipelines
oc secrets link -n ocp-pipelines pipeline rh-registry-saFor more information, check the full KCS article.
To clone a private repository in the pipeline, the pipeline Service Account will need to be able to authenticate against the repository. There are two main options to get this authentication: Using a username+token (Or a PAT if using GitHub) or using an SSH private key.
oc create secret generic git-secret-ssh --type=kubernetes.io/ssh-auth --from-file=ssh-privatekey=$LOCATION_PRIVATE_KEY -n pipelines
oc annotate secret git-secret-ssh tekton.dev/git-0="$GIT_PRIVATE_URL"
oc secrets link -n ocp-pipeline pipeline git-secret-sshoc create secret generic gh-pat-secret -n ocp-pipelines \
--type=kubernetes.io/basic-auth \
--from-literal=username=$GITHUB_USERNAME \
--from-literal=password=$GITHUB_PAT
oc annotate secret gh-pat-secret tekton.dev/git-0="$GIT_PRIVATE_URL"
oc secrets link -n ocp-pipelines pipeline gh-pat-secretFor more information about the PAT creation and configuration, you can follow the instructions that we have in the following workshop guidelines.
Robot accounts are a way to access repositories without requiring a human user account. A robot account has its own credentials, generated by Quay and linked to an Organization. To create a Robot Account and get its credentials, you have to access the Quay web console. For this repository, we are going to use my personal Quay organization, which is located at: https://quay.io/user/alopezme.
Using an admin account, you can access the organization, go to the Robot Accounts section and click on Create Robot Account. After creating the Account, click on it to directly download the Kubernetes secret definition that you have to apply in your namespace.
Once you create the file with its contents, you can apply it to the cluster like this:
oc apply -f secrets/quay-alopezme-pull-secret.yaml -n ocp-pipelines
oc secrets link -n ocp-pipelines pipeline quay-alopezme-pull-secretFor more information, you can access the documentation of the on-premise installation of Quay.
If you have Grafana installed in the cluster, you can create the following Dashboards to monitor the status of the pipeline’s execution. There are two dashboards obtained from different sources:
-
tekton-overview.jsonfrom the Grafana dashboards marketplace. This chart tries to provide a full overview. -
tekton-dashboard.jsonfrom the jenkins-x-charts organization. This chart is focused on TaskRuns and PipelineRuns.
In general, Tekton has several relevant metrics. All of them are detailed in the upstream documentation.
This section contains several tasks that can be useful for the pipelines.
| Task | New | Description |
|---|---|---|
✔️ |
List files from a mounted workspace. |
|
✔️ |
Download the latest release of a GitHub release from a given organization/repository. |
|
✔️ |
Change permissions of a file |
|
❌ |
Patched version of the Skopeo copy task. The feature to bulk-copy several images in the same Pipeline execution is currently affected by this bug. |
|
❌ |
Patched version of the Buildah task where the Task also mounts the |
| Pipeline | Description |
|---|---|
Copy one container image from one container repository to another. |
|
Copy several images listed in a ConfigMap from one container repository to another. |
|
This pipeline builds an image by cloning a git repo and building the image using its Dockerfile and the default |
|
This pipeline modifies the default |
|
This pipeline also builds a container image and includes a binary downloaded from the GH releases section of any repository (By default, it installs |
|
Pipeline that copies the |
oc create -f pipelineruns/11-copy-image.yamloc create -f pipelineruns/12-copy-bulk-images.yamloc create -f pipelineruns/21-build-image.yamloc create -f pipelineruns/22-build-image-entitlements.yamloc create -f pipelineruns/31-copy-entitlements.yamloc create -f pipelineruns/23-build-image-download.yamlIf you don’t want to use the previous pipeline, you can do it manually with the following command:
oc get secret etc-pki-entitlement -n openshift-config-managed -o yaml | yq 'del(.metadata.creationTimestamp, .metadata.uid, .metadata.resourceVersion, .metadata.namespace, .metadata.managedFields)' | oc create -n pipelines -f -Triggers capture external events, such as a Git pull request, and process them to extract key pieces of information. Triggers consist of four different CRDs that work together:
-
The TriggerBinding resource extracts the fields from an event payload and stores them as parameters.
-
The TriggerTemplate resource acts as a standard for the way resources must be created.
-
The Trigger resource combines the TriggerBinding and TriggerTemplate resources, and optionally, the interceptors event processor.
-
The EventListener resource provides an endpoint or an event sink, that listens for incoming HTTP-based events with a JSON payload.
We have an example using a CronJob to automatically trigger the Pipeline every morning.
To get the most out of Openshift Pipelines, you will need to download and install the tkn command line tool. You can download it from the Tekton documentation or directly from your Openshift cluster:
If you want a tutorial to learn Openshift Pipelines, I recommend you this tutorial from Red Hat Scholars.
The Openshift Pipelines Operator configures several ClusterTasks by default. Here you can find a summary of them for documentation purposes:
$ tkn clustertasks list
NAME DESCRIPTION AGE
argocd-task-sync-and-wait This task syncs (de... 2 days ago
buildah Buildah task builds... 2 days ago
git-cli This task can be us... 2 days ago
git-clone These Tasks are Git... 2 days ago
helm-upgrade-from-repo These tasks will in... 2 days ago
helm-upgrade-from-source These tasks will in... 2 days ago
jib-maven This Task builds Ja... 2 days ago
kn This Task performs ... 2 days ago
kn-apply This task deploys a... 2 days ago
kubeconfig-creator This Task do a simi... 2 days ago
maven This Task can be us... 2 days ago
openshift-client This task runs comm... 2 days ago
pull-request This Task allows a ... 2 days ago
s2i-dotnet s2i-dotnet task fet... 2 days ago
s2i-go s2i-go task clones ... 2 days ago
s2i-java s2i-java task clone... 2 days ago
s2i-nodejs s2i-nodejs task clo... 2 days ago
s2i-perl s2i-perl task clone... 2 days ago
s2i-php s2i-php task clones... 2 days ago
s2i-python s2i-python task clo... 2 days ago
s2i-ruby s2i-ruby task clone... 2 days ago
skopeo-copy Skopeo is a command... 2 days ago
tkn This task performs ... 2 days ago
trigger-jenkins-job The following task ... 2 days agoIn some cases, the Insights Operator cannot retrieve the etc-pki-entitlement secret correctly. In such cases, it is possible to download the certificate from the Red Hat management console. If this is your case…
>> Click Here <<
In this section, we are going to explore the proposal by siamaksade in Red Hat’s Blog: Keep Your Applications Secure With Automatic Rebuilds and in his GitHub repository quay-mirror-pipeline. For more information:
>> Click Here <<