Add lib-builder docker image (#155)

* Test docker Image

* Fix Dockerfile

* Fix file permissions

* Fix bugs and add run script

* Update tools/docker/Dockerfile

* Add textual to docker image

* Reorder

* Fix

* Test

* Fix

* Add UI checks

* Fix docker image

* Fix exec in docker

* Move argument

* Add workflow

* Add checks to entrypoint

* Improvements to Dockerfile

* Test example run scripts

* Fix powershell script

* Change parameter name

* Add comment

* Add Readme

* Fix comparison

* Remove cache

* Fix comment

* Improve permission settings

* Add warning

* Update readme

* Fix repositories

---------

Co-authored-by: Me No Dev <[email protected]>

.github/workflows/docker.yml

@@ -0,0 +1,81 @@
+name: Build and push Docker image
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release/*'
+    tags:
+      - 'v*.*'
+
+env:
+  # Build the image for amd64 and arm64
+  BUILD_PLATFORMS: linux/amd64,linux/arm64
+  DOCKERHUB_REPO: ${{ github.repository_owner }}/esp32-arduino-lib-builder
+
+jobs:
+  docker:
+    # Disable the job in forks
+    if: ${{ github.repository_owner == 'espressif' }}
+
+    runs-on: ubuntu-latest
+    steps:
+      # Depending on the branch/tag, set CLONE_BRANCH_OR_TAG variable (used in the Dockerfile
+      # as a build arg) and TAG_NAME (used when tagging the image).
+      #
+      # The following 3 steps cover the alternatives (tag, release branch, master branch):
+      - name: Set variables (tags)
+        if: ${{ github.ref_type == 'tag' }}
+        run: |
+          echo "CLONE_BRANCH_OR_TAG=$GITHUB_REF_NAME" >> $GITHUB_ENV
+          echo "TAG_NAME=$GITHUB_REF_NAME" >> $GITHUB_ENV
+      - name: Set variables (release branches)
+        if: ${{ github.ref_type == 'branch' && startsWith(github.ref_name, 'release/') }}
+        run: |
+          echo "CLONE_BRANCH_OR_TAG=$GITHUB_REF_NAME" >> $GITHUB_ENV
+          echo "TAG_NAME=release-${GITHUB_REF_NAME##release/}" >> $GITHUB_ENV
+      - name: Set variables (main branch)
+        if: ${{ github.ref_type == 'branch' && github.ref_name == 'master' }}
+        run: |
+          echo "CLONE_BRANCH_OR_TAG=master" >> $GITHUB_ENV
+          echo "TAG_NAME=latest" >> $GITHUB_ENV
+
+      # Display the variables set above, just in case.
+      - name: Check variables
+        run: |
+          echo "CLONE_BRANCH_OR_TAG: $CLONE_BRANCH_OR_TAG"
+          echo "CHECKOUT_REF: $CHECKOUT_REF"
+          echo "TAG_NAME: $TAG_NAME"
+
+      # The following steps are the standard boilerplate from
+      # https://github.com/marketplace/actions/build-and-push-docker-images
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+      - name: Set up QEMU for multiarch builds
+        uses: docker/setup-qemu-action@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+      - name: Build and push
+        uses: docker/build-push-action@v5
+        with:
+          context: tools/docker
+          push: true
+          tags: ${{ env.DOCKERHUB_REPO }}:${{ env.TAG_NAME }}
+          platforms: ${{ env.BUILD_PLATFORMS }}
+          build-args: |
+            LIBBUILDER_CLONE_URL=${{ github.server_url }}/${{ github.repository }}.git
+            LIBBUILDER_CLONE_BRANCH_OR_TAG=${{ env.CLONE_BRANCH_OR_TAG }}
+
+      - name: Update Docker Hub repository description (master branch)
+        if: ${{ github.ref_type == 'branch' && github.ref_name == 'master' }}
+        uses: peter-evans/dockerhub-description@v4
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+          repository: ${{ env.DOCKERHUB_REPO }}
+          readme-filepath: ./tools/docker/README.md

tools/config_editor/app.py

@@ -86,7 +86,6 @@ def compose(self) -> ComposeResult:
 
     def on_mount(self) -> None:
         # Event handler called when the app is mounted for the first time
-        self.title = "Configurator"
         self.sub_title = "Main Menu"
         print("Main screen mounted.")
 
@@ -104,6 +103,7 @@ class ConfigEditorApp(App):
     # Options to be set by the command line arguments
     setting_target = ""
     setting_arduino_path = ""
+    setting_output_permissions = ""
     setting_arduino_branch = ""
     setting_idf_branch = ""
     setting_idf_commit = ""
@@ -131,6 +131,7 @@ def on_mount(self) -> None:
         print("IDF Branch: " + str(self.setting_idf_branch))
         print("IDF Commit: " + str(self.setting_idf_commit))
         print("IDF Debug Level: " + str(self.setting_debug_level))
+        self.title = "Configurator"
         self.push_screen("main")
 
 def arduino_default_path():
@@ -199,6 +200,13 @@ def main() -> None:
                         required=False,
                         help="Path to arduino-esp32 directory. Default: " + arduino_default_path())
 
+    parser.add_argument("--output-permissions",
+                        metavar="<uid:gid>",
+                        type=str,
+                        default="",
+                        required=False,
+                        help=argparse.SUPPRESS) # Hidden option. It is only supposed to be used by the docker container
+
     parser.add_argument("-A", "--arduino-branch",
                         metavar="<arduino branch>",
                         type=str,
@@ -256,14 +264,18 @@ def main() -> None:
         elif args.arduino_path == arduino_default_path():
             print("Warning: Default Arduino path not found. Disabling copy to Arduino.")
             app.setting_enable_copy = False
+        elif args.arduino_path == "/arduino-esp32": # Docker mount point
+            print("Warning: Docker mount point not found. Disabling copy to Arduino.")
+            app.setting_enable_copy = False
         else:
-            print("Invalid path to Arduino core: " + os.path.abspath(args.arduino_path))
+            print("Error: Invalid path to Arduino core: " + os.path.abspath(args.arduino_path))
             exit(1)
     else:
         app.setting_enable_copy = False
 
     # Set the other options
     app.setting_arduino_path = os.path.abspath(args.arduino_path)
+    app.setting_output_permissions = args.output_permissions
     app.setting_arduino_branch = args.arduino_branch
     app.setting_idf_branch = args.idf_branch
     app.setting_idf_commit = args.idf_commit

tools/config_editor/compile.py

@@ -1,6 +1,7 @@
 import sys
 import subprocess
 import os
+import re
 
 from rich.console import RenderableType
 
@@ -60,6 +61,11 @@ def print_success(self, message: str) -> None:
         self.button_widget.add_class("-success")
         #print("Success: " + message) # For debugging
 
+    def print_warning(self, message: str) -> None:
+        # Print warning message to the RichLog widget
+        self.log_widget.write("[b bright_yellow]" + message)
+        #print("Warning: " + message) # For debugging
+
     def print_info(self, message: str) -> None:
         # Print info message to the RichLog widget
         self.log_widget.write("[b bright_cyan]" + message)
@@ -72,7 +78,12 @@ def compile_libs(self) -> None:
 
         label = self.query_one("#compile-title", Static)
         self.child_process = None
-        if self.app.setting_target == ",".join(self.app.supported_targets):
+
+        if not self.app.setting_target:
+            self.print_error("No target selected")
+            label.update("No target selected")
+            return
+        elif self.app.setting_target == ",".join(self.app.supported_targets):
             target = "all targets"
         else:
             target = self.app.setting_target.replace(",", ", ").upper()
@@ -133,6 +144,30 @@ def compile_libs(self) -> None:
                 print("Error reading child process errors: " + str(e))
             label.update("Compilation failed for " + target)
         else:
+            if self.app.setting_output_permissions:
+                regex = r"^[1-9][0-9]*:[1-9][0-9]*$"  # Regex to match the uid:gid format. Note that 0:0 (root) is not allowed
+                if re.match(regex, self.app.setting_output_permissions):
+                    print_info("Setting permissions to: " + self.app.setting_output_permissions)
+                    chown_process = None
+                    try:
+                        chown_process = subprocess.run(["chown", "-R", self.app.setting_output_permissions, self.app.setting_arduino_path])
+                        chown_process.wait()
+                    except Exception as e:
+                        print("Error changing permissions: " + str(e))
+
+                    if chown_process and chown_process.returncode != 0:
+                        self.print_error("Error changing permissions")
+                        self.print_error("Please change the ownership of generated files manually")
+                    else:
+                        self.print_success("Permissions changed successfully")
+                elif self.app.setting_output_permissions == "0:0":
+                    self.print_warning("Permissions settings are set to root (0:0)")
+                    self.print_warning("Please change the ownership of generated files manually")
+                    self.print_warning("If you are compiling for Windows, you may ignore this warning")
+                else:
+                    self.print_error("Invalid permissions format: " + self.app.setting_output_permissions)
+                    self.print_error("Please change the ownership of generated files manually")
+
             self.print_success("Compilation successful for " + target)
             label.update("Compilation successful for " + target)
 
@@ -161,3 +196,8 @@ def compose(self) -> ComposeResult:
             self.button_widget = Button("Back", id="compile-back-button")
             yield self.button_widget
         yield Footer()
+
+    def on_mount(self) -> None:
+        # Event handler called when the screen is mounted
+        print("Compile screen mounted")
+        self.sub_title = "Compilation"

tools/docker/Dockerfile

@@ -0,0 +1,79 @@
+# To Do: Check if it is worth to use espressif/idf as base image (image size will be much bigger)
+FROM ubuntu:22.04
+
+# switch to root, let the entrypoint drop back to host user
+USER root
+SHELL ["/bin/bash", "-c"]
+
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN : \
+  && apt-get update \
+  && apt-get install -y --no-install-recommends \
+    bison \
+    ccache \
+    cmake \
+    curl \
+    flex \
+    git \
+    gperf \
+    jq \
+    libncurses-dev \
+    libssl-dev \
+    libusb-1.0 \
+    ninja-build \
+    patch \
+    python3 \
+    python3-click \
+    python3-cryptography \
+    python3-future \
+    python3-pip \
+    python3-pyelftools \
+    python3-pyparsing \
+    python3-serial \
+    python3-setuptools \
+    python3-venv \
+    wget \
+  && pip3 install --upgrade pip textual-dev \
+  && apt-get autoremove -y \
+  && rm -rf /var/lib/apt/lists/* \
+  && :
+
+# To build the image for a branch or a tag of the lib-builder, pass --build-arg LIBBUILDER_CLONE_BRANCH_OR_TAG=name.
+# To build the image with a specific commit ID of lib-builder, pass --build-arg LIBBUILDER_CHECKOUT_REF=commit-id.
+# It is possibe to combine both, e.g.:
+#   LIBBUILDER_CLONE_BRANCH_OR_TAG=release/vX.Y
+#   LIBBUILDER_CHECKOUT_REF=<some commit on release/vX.Y branch>.
+# Use LIBBUILDER_CLONE_SHALLOW=1 to peform shallow clone (i.e. --depth=1 --shallow-submodules)
+# Use LIBBUILDER_CLONE_SHALLOW_DEPTH=X to define the depth if LIBBUILDER_CLONE_SHALLOW is used (i.e. --depth=X)
+
+ARG LIBBUILDER_CLONE_URL=https://github.com/espressif/esp32-arduino-lib-builder
+ARG LIBBUILDER_CLONE_BRANCH_OR_TAG=master
+ARG LIBBUILDER_CHECKOUT_REF=
+ARG LIBBUILDER_CLONE_SHALLOW=
+ARG LIBBUILDER_CLONE_SHALLOW_DEPTH=1
+
+ENV LIBBUILDER_PATH=/opt/esp/lib-builder
+# Ccache is installed, enable it by default
+ENV IDF_CCACHE_ENABLE=1
+
+RUN echo LIBBUILDER_CHECKOUT_REF=$LIBBUILDER_CHECKOUT_REF LIBBUILDER_CLONE_BRANCH_OR_TAG=$LIBBUILDER_CLONE_BRANCH_OR_TAG && \
+    git clone --recursive \
+      ${LIBBUILDER_CLONE_SHALLOW:+--depth=${LIBBUILDER_CLONE_SHALLOW_DEPTH} --shallow-submodules} \
+      ${LIBBUILDER_CLONE_BRANCH_OR_TAG:+-b $LIBBUILDER_CLONE_BRANCH_OR_TAG} \
+      $LIBBUILDER_CLONE_URL $LIBBUILDER_PATH && \
+    git config --system --add safe.directory $LIBBUILDER_PATH && \
+    if [ -n "$LIBBUILDER_CHECKOUT_REF" ]; then \
+      cd $LIBBUILDER_PATH && \
+      if [ -n "$LIBBUILDER_CLONE_SHALLOW" ]; then \
+        git fetch origin --depth=${LIBBUILDER_CLONE_SHALLOW_DEPTH} --recurse-submodules ${LIBBUILDER_CHECKOUT_REF}; \
+      fi && \
+      git checkout $LIBBUILDER_CHECKOUT_REF && \
+      git submodule update --init --recursive; \
+    fi
+
+COPY entrypoint.sh $LIBBUILDER_PATH/entrypoint.sh
+
+WORKDIR /opt/esp/lib-builder
+ENTRYPOINT [ "/opt/esp/lib-builder/entrypoint.sh" ]
+CMD [ "python3", "tools/config_editor/app.py" ]

tools/docker/README.md

@@ -0,0 +1,52 @@
+<!-- This is a brief version of the Arduino Core for ESP32 documentation (add specific link)
+     intended to be displayed on the Docker Hub page: https://hub.docker.com/r/espressif/esp32-arduino-lib-builder.
+     When changing this page, please keep the documentation in sync.
+     (Keep the differences between Markdown and restructuredText in mind.)
+ -->
+
+# ESP-IDF Docker Image
+
+This is a Docker image for the [ESP32 Arduino Lib Builder](https://github.com/espressif/esp32-arduino-lib-builder). It is intended for building the static libraries of ESP-IDF components for use in Arduino projects.
+
+This image contains a copy of the esp32-arduino-lib-builder repository and already include or will obtain all the required tools and dependencies to build the Arduino static libraries.
+
+Currently supported architectures are:
+  - `amd64`
+  - `arm64`
+
+## Tags
+
+Multiple tags of this image are maintained:
+
+  - `latest`: tracks `master` branch of esp32-arduino-lib-builder
+  - `release-vX.Y`: tracks `release/vX.Y` branch of esp32-arduino-lib-builder
+
+## Basic Usage
+
+```bash
+docker run --rm -it -e "TERM=xterm-256color" -v <path to arduino-esp32>:/arduino-esp32 espressif/esp32-arduino-lib-builder:latest
+```
+
+The above command explained:
+
+  - `docker run`: Runs a command in a new container.
+  - `--rm`: Optional. Automatically removes the container when it exits. Remove this flag if you plan to use the container multiple times.
+  - `-i`: Runs the container in interactive mode.
+  - `-t`: Allocates a pseudo-TTY.
+  - `-e "TERM=xterm-256color"`: Optional. Sets the terminal type to `xterm-256color` to display colors correctly.
+  - `-v <path to arduino-esp32>:/arduino-esp32`: Optional. Mounts the Arduino Core for ESP32 repository at `/arduino-esp32` inside the container. Replace `<path to arduino-esp32>` with the path to the repository on the host machine. If not provided, the container will not copy the compiled libraries to the host machine.
+  - `espressif/esp32-arduino-lib-builder:latest`: The Docker image to use.
+
+After running the above command, you will be inside the container and can build the libraries using the user interface.
+
+By default the docker container will run the user interface script. If you want to run a specific command, you can pass it as an argument to the docker run command. For example, to run a terminal inside the container, you can run:
+
+```bash
+docker run -it espressif/esp32-arduino-lib-builder:latest /bin/bash
+```
+
+## Documentation
+
+<!-- Pending: Add link to the full documentation. -->
+
+For more information about this image and the detailed usage instructions, please refer to the Arduino Core for ESP32 documentation.

tools/docker/entrypoint.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+set -e
+
+# LIBBUILDER_GIT_SAFE_DIR has the same format as system PATH environment variable.
+# All path specified in LIBBUILDER_GIT_SAFE_DIR will be added to user's
+# global git config as safe.directory paths. For more information
+# see git-config manual page.
+if [ -n "${LIBBUILDER_GIT_SAFE_DIR+x}" ]
+then
+    echo "Adding following directories into git's safe.directory"
+    echo "$LIBBUILDER_GIT_SAFE_DIR" | tr ':' '\n' | while read -r dir
+    do
+        git config --global --add safe.directory "$dir"
+        echo "  $dir"
+    done
+fi
+
+# Check if the mount point /arduino-esp32 exists
+if [ -d "/arduino-esp32" ] && [[ "$@" == "python3 tools/config_editor/app.py"* ]]; then
+    # Running UI with mount point detected, adding -c and --output-permissions arguments
+    echo "Output folder permissions: `stat -c "%u:%g" /arduino-esp32`"
+    exec "$@" -c /arduino-esp32 --output-permissions `stat -c "%u:%g" /arduino-esp32`
+else
+    # Running UI without mount point detected or running another command
+    exec "$@"
+fi