feat: add workflow for updating automated test results in TestRail #11424
Conversation
|
CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
chrisleewilcox
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
chrisleewilcox
added
the
needs-dev-review
|
chrisleewilcox
added
the
No QA Needed
| @@ -46,6 +46,7 @@ pipelines: | |||
| stages: | |||
| - build_e2e_ios_android_stage: {} | |||
| - run_release_e2e_ios_android_stage: {} | |||
| - report_results_stage: {} | |||
There was a problem hiding this comment.
stage added for reporting results after build and tests
| report_results_stage: | ||
| workflows: | ||
| - run_testrail_update_automated_test_results: {} |
There was a problem hiding this comment.
Added workflow for posting automated test results.
| ### Report automated test results to TestRail | ||
| run_testrail_update_automated_test_results: | ||
| before_run: | ||
| - code_setup | ||
| steps: | ||
| - script@1: | ||
| title: 'Add Automated Test Results to TestRail' | ||
| inputs: | ||
| - content: |- | ||
| #!/usr/bin/env bash | ||
| echo 'REPORT AUTOMATED TEST RESULTS TO TESTRAIL' | ||
| node ./scripts/testrail/testrail.api.js |
There was a problem hiding this comment.
Workflow script.
There was a problem hiding this comment.
This script is chaining a few request in order to properly retrieve all automated test cases and record their test status.
| .get(automatedTestCasesEndpoint) | ||
| .then((response) => { | ||
| const automatedcaseids = response.data.cases.map( | ||
| (automatedcase) => automatedcase.id, | ||
| ); | ||
| console.log(`test case id count: ${automatedcaseids.length}`); | ||
| return axios.post(addTestRun, { | ||
| name: 'Automated Test Run on bitrise release_e2e_pipline', | ||
| description: 'Automated test run on release branch', | ||
| include_all: false, | ||
| case_ids: automatedcaseids, | ||
| }); |
There was a problem hiding this comment.
Get all the automated test cases in TestRail mobile project.
Then create a Test Run with all the automated test cases assigned to the test run.
| .then((response) => { | ||
| runID = response.data.id; | ||
| return axios.get(`${getAutomatedTestRun}${runID}`); | ||
| }) |
There was a problem hiding this comment.
Get the runID of the test run just created to be used for updating test results.
| .then((response) => { | ||
| const automatedResults = response.data.tests.map((test) => ({ | ||
| test_id: test.id, | ||
| status_id: 1, | ||
| comment: 'Passed on bitrise', | ||
| })); | ||
| return axios.post(`${addResults}${runID}`, { | ||
| results: automatedResults, | ||
| }); | ||
| }) |
There was a problem hiding this comment.
Update test results for the automated test cases using the runID.
Default the results to Pass since this script is only run when all test pass.
Description
Adding a workflow to
release_e2e_pipelineto post test results to TestRail. This initial implementation will focus on utilizing TestRail API to post test results to TestRail mobile project when all workflows in the bitriserelease_e2e_pipelineare successful. Test results will not be posted if any other workflow fails on therelease_e2e_pipelinewith this implementation.Automated Test Cases in TestRail will need to be reviewed and updated to ensure the most accurate reporting until we implement mapping of automated scripts to automated test cases in TestRail. Once a mapping mechanism has been implemented this will allow us to report failed test runs in a more effective manner.
Not able to run the pipeline from bitrise dashboard until this PR is merged. Test runs for this PR was done using curl command shared in the bitrise advanced modal for the pipeline.
Related issues
Fixes: https://github.com/MetaMask/mobile-planning/issues/1891
Manual testing steps
release_e2e_pipelineon release branchrelease_e2e_pipelineare successfulScreenshots/Recordings
Before
No workflow to save test results in TestRail
After
Failed pipeline doesn't post test results: https://app.bitrise.io/app/be69d4368ee7e86d/pipelines/4047bd66-20a1-405b-bd9d-c3e03942bc5a?tab=workflows
Successful pipeline post test results:
Pre-merge author checklist
Pre-merge reviewer checklist