Setup build automation script and CI workflow #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The script is setup to take in, as its first parameter, the name of the example to build. This is then checked against the valid examples and matched with the associated directory. The directory is then checked for in the current path - if not, it's an indication that the script was not sourced from the project root.
The script now sets up the virtual environment directory and installs the required python dependencies from requirements.txt. Note: This script may have compatibility issues with Windows that will be addressed in a separate issue.
This commit saw a few major changes, including a correction to the target board variable. The mount point is now passed in as an argument to the script as there's an issue with flashing with mbed-tools; please refer to issue #282 on the mbed-tools repository for more information. As a result, the binary is flashed manually.
The script has many caveats that are addressed as a comment on the associated issue (or pull request). These are mainly as a result of dependency conflicts between mbed-tools, mbed-os, and pyocd. The updated script follows the build instructions as listed in the documentation accompanying this example (with additional checks).
This commit is the first in a series of refactoring commits. It introduces a new tool, fota.sh. This commit sets up the script with an author's note and a function to display the tool's usage.
The options passed into the script are parsed and stored in the corresponding variables. Note that some of these variables have default values (as indicated by the usage message).
This commit also removes the print_options function, which was created to help debug the functionality of the parse_options function.
The file is setup with constants for text formatting.
The say function prints a message to the console with formatting based on the selected formatting mode. For now, the error message functionality of this function has been implemented. Refer to the comments in the script for more information.
The function prints an error message to stderr and exits or returns with a code of 1 based on the fail mode, which is passed as an input. If the file is being sourced, then "return" would be passed in; else "exit" would be passed in. This function invokes the say function created earlier with the error formatting mode.
Previously, the parse_options function required that the option be followed by an '=' symbol and the value (i.e -o=value or --option=value). The usage instructions separate the option and the value by a space and the function was modified to meet this requirement. The following stack overflow post proved quite useful in understanding how options are parsed - in fact, the "Bash Equals-Separated" part of the accepted answer was adapted for the intial implementation. https://stackoverflow.com/a/14203146
The parsing function would loop indefinintely if there was no value associated with an option. There is still a need to check if the default value of the options have been cleared, in which case the prior default must be restored or an error must be displayed indicating that the option was not followed by a value.
This commit rolls back the parsing function to it's state in commit a58f3b9. The equals-separated parsing seems a bit more robust than space-separated parsing. I did experiment with getopt, which didn't work as the version pre-installed on macOS doesn't support long option parsing. The 'enhanced' getopt would require a separate installation - at this phase, a python script seems easier.
The fota script now sources the utils script and uses the fail and setup_formatting functions to handle the error of unrecognised options in the parsing phase.
A note is displayed if no mount point is specfied indicating that the binaries will have to be flashed manually.
The message is now updated to use the note formatting and the second line was dropped as it was redundant.
The build instructions were adapted from those defined earlier in build.sh. This function has been verfied to work with the supported target board and supports building without a mount point.
Create mock_clean and mcuboot_clean; the latter is still a dummy function. The former has been implemented and verified to clean the target build directory and dependencies directories.
The build instructions for the requirements installation were adapted from those defined earlier in build.sh, albeit with minor changes.
These instructions go upto the point where the original virtual environment is restored. The last step would involve some input from the user to modify the mbed_app.json file in the application folder.
This is quite a significant commit - the script now prompts the user on whether it should auto-update the application/mbed_app.json file; this serves well for testing purposes where the "yes" would be piped into the script. Also, part of the script was adapted from the prior implementation in build.sh. Note that this introduces a new dependency "jq" that's not installed through pip - rather, it requires some manual intervention on the end-user's behalf.
These steps were adapted from the ones written earlier in build.sh.
The trap invokes clean upon unsuccessful termination.
If there are a number of build files to clean, there is quite a significant pause from when an error occurs to when control is handed over to the user. This message indicates to the end-user what's going on during that time.
Please refer to the comments in the .yml file for more information
Build process automation
…fota into ci-setup
Revert "CI Setup"
…fota into ci-setup
Fix dependency issue in workflow
The workflow is no longer triggered on a push to main. This was done to avoid duplicate runs when a pull request is merged.
|
|
||
| # This function would clean up the example builds and generated files | ||
| clean () { | ||
| say message "Cleaning builds and generated files..." |
There was a problem hiding this comment.
what is this? nevermind, figured it out
There was a problem hiding this comment.
The tool accepts a -c or --clean option to remove any generated build folders or files. It removes the venv folder and any example-specific generated folders and files by invoking the clean functions of each example - mock_clean and mcuboot_clean; the definitions of these functions are located in mock.sh and mcuboot.sh respectively. Also, it's worth mentioning that if the build were to fail at any point with an exit code of 1, it would trap into the cleanup routine, which would call this function.
|
This pull request will be closed in favour of a new one (#3), which involves changes to the commit history. Please refer to the remarks at the end of the new pull request's description for more information. Request closed 🛑 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Build Automation Script
Currently, to setup the two examples (MCUBoot and Mock) for demonstration, the build instructions in their associated documentation has to be followed step-by-step. This process can be quite cumbersome and isn't entirely ideal for testing.
Instead, this build process can be automated through a simple bash script that performs every step in the build instructions for the end-user - as a result, the
fota.shcli build tool was created. The tool takes in as arguments the example, target board, mount point of target board, and toolchain.There are slight kludges with how the MCUboot and Mock examples are built. The former is a direct result of dependency conflicts between mbed-tools, pyocd, and mbed-os. The latter is a result of issues with flashing the target binary to the connected board. The details of both are addressed below:
Dependency Conflicts: The difference in version requirements of the PyYAML dependency imposed by both mbed-os and pyocd led to the creation of a temporary virtual environment; this is used in the "creating and flashing the factory firmware" stage of the MCUboot example build process. This is a known issue and would require changes to the
requirements.txtfile in the mbed-os to resolve.Another minor conflict that doesn't hinder the build process is one between mbed-tools and mbed-os on the version requirement of the Click dependency; mbed-tools requires that the minimum version of Click to be greater than 7.1 while mbed-os requires it to be greater than 7.0. Now, the dependencies for mbed-os are installed after those of mbed-tools, which overwrites the newer version and generates a conflict. This would (again) require changes to the requirements file in the mbed-os repository to bump up the minimum version number of Click to 7.1.
Target Binary Flashing: For the Mock example, compiling with mbed-tools results in the following error:
The tool is looking for a hex file under the cmake build output whose name comes from project directory (in this case, "target"). However, the generated one is named
BLE_GattServer_FOTAService.hex. Again, this is a known issue and has been filed (282) in the mbed-tools repository by noonfom.GitHub CI Workflow
Currently, there is no CI workflow setup to make sure that both examples build successfully. The aim was to setup a GitHub actions workflow that uses the new CLI build tool
fota.sh(from branch build-process-automation) to build the examples and check for any errors; known issues (documented above) are ignored or have workarounds implemented to avoid them.A GitHub workflow "Build Examples" was created to build the two examples (as two separate jobs). Also, this workflow uses the mbed-os-env container as it bundles the GCC ARM toolchain, along with Python and CMake.
Remarks
The description of this pull request was adapted from the issues and pull requests made in my fork.
For the automation script, the aim was to write it in bash as it is widely available and easily maintainable. In the future, the script's functionality could be migrated into a CLI tool written in Python.
It was not a strict requirement to run the examples on the target board using the CI workflow by means of a Raspberry Pi rig connected to an NRF52840_DK target board. However, support for this can be added later considering that the new build tool supports building the examples without providing a mount point.