Skip to content

Visual regression testing

Jump to bottom
Sean Moore edited this page Apr 2, 2021 · 4 revisions

Overview

All components that are covered by regression testing have visual regression testing in place along with functional tests. Visual regression works by taking a screenshot of the desired element and comparing it to the baseline screenshot. This comparison automatically checks for a specified percent difference of any changes including:

  • positioning
  • colors theme
  • applied states (hover, focused or selected states)

note: there is no way to skip color or other style/theme checks because the visual regression compares the whole baseline image to the whole new image and returns a percentage value for the difference between the two.

Approved screenshots, which are used as expected component/element states, are called baseline screenshot.

Storing the baseline screenshots

All baseline screenshots are stored in the e2e/wdio/baselineScreenshots folder of the repository.

When the test executes for the first time, baseline screenshots are generated automatically and saved to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder.

For all following executions, new baseline screenshots will not be generated if the baseline screenshot with the same name already exists in this folder. Instead, a comparison of the new screenshot vs the baseline screenshot will be performed.

In case the tests fail, two screenshots will be generated, for each comparison action:

  • actual screenshot
  • difference screenshot
    These two types of screenshots will be saved in the .tmp/actual/<browser_name> and .tmp/diff/<browser_name> folders respectively.

The ‘actual screenshot’ will show you the current image of the element as it is now. The ‘difference screenshot’ will show the actual vs baseline image, highlighting the differences in pink.

All baseline screenshots that are generated for new components need to be committed to the repository along with the e2e test case code.

Cross Browser visual regression

Component/element visual regression baseline screenshots are created for each browser + OS combination. For example if one component is under visual regression and tests are executed for Chrome and FF browsers for Win and MacOs, four separate screenshots will be generated and need to be committed along with the test case code.

Visual regression test cases failure debug.

Visual regression test cases indicate the mismatch of the baseline screenshot and actual screenshot (in percentage). If the baseline screenshot and actual screenshot don’t match, the particular test case will fail with an error of mismatch percentage.

In case the visual regression test case is failing:

  • The test case needs to be re-executed from the local environment on the Saucelabs side on the env on which test case failed.
  • The difference screenshots need to be evaluated by the PR owner to determine if the difference is expected or not. In the .tmp/diff folder you will be able to find the screenshot with mismatch highlighted in purple color.

In case the PR introduces unexpected changes, it needs to be fixed without merging any unexpected change to the main branch.
In case the change to the component is expected, the baseline screenshot needs to be replaced with the new actual screenshot. (You can do this by moving the screenshot from the .tmp/actual folder to the e2e/wdio/baselineScreenshots/platform or core/<component_name>/<browser_name> folder). The newly added (newly approved baseline) screenshots need to be committed to this PR.

All baseline screenshots need to be generated on the Saucelabs side (the same environment on which actual testing form the CI will be performed). Images generated during the local tests execution can't be used as a baseline screenshots, unless all the environment parameters completely match (OS version, Browser version, screen resolution).

Step-by-Step debugging and updating visual regression

When a test fails due to visual regression:

Preconditions: setup Saucelabs locally as shown here.

  1. Go to the Travis CI log and read which components and test cases failed
  2. Go to the wdio.conf.js file in your IDE
  3. Change the config to meet your debugging needs (specify component, browser/OS)
    • i.e. specs: [ './e2e/wdio/<platform/core>/**/<failing-spec-name>.e2e-spec.ts', ]
  4. Run the test through Saucelabs
    • start app: ng serve --disable-host-check --ssl --no-watch
    • run tests: npm run e2e:wdio:saucelabs
  5. Check the log to see if there is additional info about failed tests
  6. After test completion, go to the .tmp/diff/<browser> folder (choose the folder of the browser you need)
  7. Look for the img with the name that matches the component under test
  8. Open the image You will see the baseline screenshot and actual screenshot together here with differences highlighted in pink color
  9. Check the difference, are the changes expected?
    • If the changes are not expected:
      • Make appropriate changes to the PR to fix the issue
      • Rerun test
    • If the changes are expected:
      • Continue to the next step
  10. Go to the .tmp/actual/<browser> folder
  11. Find the img with the same file name as the one you checked
  12. Copy the img file
  13. Go to the .e2e/wdio/baselineScreenshot/<browser> folder
  14. Paste the img file there
  15. Choose to overwrite the existing file Now the original baseline screenshot is replaced with the new, actual screenshot, which will be used as the baseline for all future runs.
Clone this wiki locally