Broken Variable Substitution in Subpipelines

[alyx]

When building a custom pipeline which uses a sub-pipeline (e.g., mycustompipeline -> fetch), having the same variable for inputs: may cause the second pipeline to fail. While I'm unsure of the exact cause, this seems like a race condition with setting the ${{inputs.}} variables when transitioning into a sub-pipeline, as the failure does not occur 100% of the time.

Renaming the input variables used by the parent pipeline to not collide with any sub-pipeline variables seems to bypass the issue.

Melange version output:

GitVersion:    v0.15.0
GitCommit:     unknown
GitTreeState:  unknown
BuildDate:     unknown
GoVersion:     go1.23.2
Compiler:      gc
Platform:      linux/amd64

This has also been tested on v0.14.7 / 71c88c32251b71894867172d355a13ee703f86ec in Docker.

Potential Build Failures Example

Note: As mentioned, this build does not always fail. It may succeed on occasion, with no changes to any involved files or commands.

mycustompipeline.yaml (example pipeline)

name: Example custom pipeline

needs:
  packages:
    - busybox

inputs:
  expected-sha256:
    description: |
      The expected SHA256 of the downloaded artifact.
  version:
    description: |
      The package version to download.
    default: ${{package.version}}

pipeline:
  - uses: fetch
    with:
      uri: https://downloads.apache.org/httpd/httpd-${{inputs.version}}.tar.gz
      expected-sha256: ${{inputs.expected-sha256}}

apache-httpd.yaml (example package)

package:
  name: apache-httpd
  version: 2.4.62
  epoch: 0
  description: "Apache HTTPd"

environment:
  contents:
    repositories:
      - https://packages.wolfi.dev/os
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

pipeline:
  - uses: mycustompipeline
    with:
      expected-sha256: 3e2404d762a2da03560d7ada379ba1599d32f04a0d70ad6ff86f44325f2f062d

Output of melange build --pipeline-dir=./pipelines --arch x86_64 ./apache-httpd.yaml:

2024/11/02 21:08:25 INFO git commit for build config not provided, attempting to detect automatically
2024/11/02 21:08:25 WARN git repository URL for build config not provided
2024/11/02 21:08:25 INFO melange is building:
2024/11/02 21:08:25 INFO   configuration file: ./apache-httpd.yaml
2024/11/02 21:08:25 INFO   workspace dir: /tmp/melange-workspace-2652354432
2024/11/02 21:08:25 INFO evaluating pipelines for package requirements
2024/11/02 21:08:25 INFO --cache-dir ./melange-cache/ not a dir; skipping
2024/11/02 21:08:25 INFO populating workspace /tmp/melange-workspace-2652354432 from .
2024/11/02 21:08:25 INFO building workspace in '/tmp/melange-guest-3609488816' with apko
2024/11/02 21:08:25 INFO setting apk repositories: [https://packages.wolfi.dev/os]
2024/11/02 21:08:25 INFO image configuration:
2024/11/02 21:08:25 INFO   contents:
2024/11/02 21:08:25 INFO     build repositories: []
2024/11/02 21:08:25 INFO     runtime repositories: [https://packages.wolfi.dev/os]
2024/11/02 21:08:25 INFO     keyring:      [https://packages.wolfi.dev/os/wolfi-signing.rsa.pub]
2024/11/02 21:08:25 INFO     packages:     [busybox wget]
2024/11/02 21:08:25 INFO   accounts:
2024/11/02 21:08:25 INFO     runas:  
2024/11/02 21:08:25 INFO     users:
2024/11/02 21:08:25 INFO       - uid=1000(build) gid=1000
2024/11/02 21:08:25 INFO     groups:
2024/11/02 21:08:25 INFO       - gid=1000(build) members=[build]
2024/11/02 21:08:25 INFO auth configured for: []
2024/11/02 21:08:26 INFO installing ca-certificates-bundle (20241010-r1)
2024/11/02 21:08:26 INFO installing wolfi-baselayout (20230201-r15)
2024/11/02 21:08:26 INFO installing glibc (2.40-r3)
2024/11/02 21:08:26 INFO installing ld-linux (2.40-r3)
2024/11/02 21:08:26 INFO installing libgcc (14.2.0-r4)
2024/11/02 21:08:26 INFO installing glibc-locale-posix (2.40-r3)
2024/11/02 21:08:26 INFO installing libxcrypt (4.4.36-r8)
2024/11/02 21:08:26 INFO installing libcrypt1 (2.40-r3)
2024/11/02 21:08:26 INFO installing busybox (1.37.0-r0)
2024/11/02 21:08:26 INFO installing libcrypto3 (3.4.0-r1)
2024/11/02 21:08:26 INFO installing libssl3 (3.4.0-r1)
2024/11/02 21:08:26 INFO installing wget (1.24.5-r5)
2024/11/02 21:08:26 INFO built image layer tarball as /tmp/apko-temp-1753404743/apko-x86_64.tar.gz
2024/11/02 21:08:26 INFO using /tmp/apko-temp-1753404743/apko-x86_64.tar.gz for image layer
2024/11/02 21:08:26 INFO ImgRef = /tmp/melange-guest-3358523063
2024/11/02 21:08:26 INFO running step "mycustompipeline"
2024/11/02 21:08:26 INFO running step "fetch" uses=mycustompipeline
2024/11/02 21:08:26 WARN /bin/sh: syntax error: bad substitution uses=mycustompipeline uses=fetch
2024/11/02 21:08:26 INFO deleting guest dir /tmp/melange-guest-3609488816
2024/11/02 21:08:26 INFO deleting workspace dir /tmp/melange-workspace-2652354432
2024/11/02 21:08:26 INFO removing image path /tmp/melange-guest-3358523063
2024/11/02 21:08:26 ERRO failed to build package: unable to run package apache-httpd pipeline: unable to run pipeline: unable to run pipeline: unable to run pipeline: exit status 2

No Build Failures Example

mycustompipeline.yaml

name: Example custom pipeline

needs:
  packages:
    - busybox

inputs:
  sha256:
    description: |
      The expected SHA256 of the downloaded artifact.
  version:
    description: |
      The package version to download.
    default: ${{package.version}}

pipeline:
  - uses: fetch
    with:
      uri: https://downloads.apache.org/httpd/httpd-${{inputs.version}}.tar.gz
      expected-sha256: ${{inputs.sha256}}