[Feature] Add e2e testing to Sage Monorepo

[tschaffter]

What product(s) is this feature for?

Sage Monorepo, Schematic

Description

The goal is to enable e2e (end-to-end) testing for the projects developed in Sage Monorepo. The first use cases is to enable e2e testing of the project schematic-api.

Specifications

• The solution should take into account that developers use Git Forking Workflow. This means that PRs opened from forks don't have access to secrets of the upstream repo.
• The secrets required for the e2e testing of a project should not be passed to projects (or products) that don't need them (least privilege)

Tentative workflow

Feature branches from forks are no longer merged into main. Instead, the would merge to an upstream branch named <product>/e2e-<feature>. Upon merging the feature branch, a second PR could be opened to merge <product>/e2e-<feature> into main.

Because the feature branch originates from the upstream repo, the GH workflow would have access to the secret. If the second feature branch is named <product>/e2e-*, then we could pass only the secrets relevant to the projects that belong to the product <product>. If needed, we could further reduce the access on a per-project basis.

If the e2e testing fails when opening the PR that would be merged to main, the developer could open a new PR with a patch that would merge into <product>/e2e-<feature>. Alternatively, selected developers could be allow to push directly to the branch <product>/e2e-<feature> without using PR, either to develop the feature or only to patch broken features.

Alternative designs

There is a pattern that consist in using GH labels to authorize a PR that originates from a fork to access the upstream secrets. The logic is that if the developer is allowed to change the labels of the PR, then they are allow to write to the upstream repo and thus should be able to access the upstream secrets. The issue with this pattern is that developers who should normally not have access to the secrets are "given" access to them on a per-PR basis. Once the PR is "authorized", the developer could modify the feature branch, e.g. to extract the secrets, which is a risk that we could only take with trusted developers.

There is a mechanism that allows feature branch from fork to access upstream secrets. The limitation is that the GH workflow should be designed in a way that there is no way the code tested may leak the secrets. This is the most promising pattern but I need to read more about it to understand its limitations.

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[tschaffter]

As per our discussion, @rrchai aims to add the e2e this week.

[tschaffter]

Added to Sprint 24.2

[tschaffter]

Update 2024-03-04

I increase the priority of this ticket for me this Sprint (24.03) as we need a reliable workflow to run e2e tests.

[tschaffter]

Solution

The scope of this task is now focused on streamlining the execution of Schematic e2e tests. This is now possible using the manual review process introduced in #2596 for running Sonar scans.