Python virtual environment isn't "activated" in kubernetes containers

[Cetlan]

Version report

Jenkins and plugins versions report:

Jenkins: 2.289.2
OS: Linux - 5.4.89+
---
promoted-builds:3.10
ssh-agent:1.23
code-coverage-api:1.4.0
ace-editor:1.1
authentication-tokens:1.4
pipeline-stage-step:2.5
jira:3.5
xunit:3.0.2
bouncycastle-api:2.20
build-timeout:1.20
javadoc:1.6
resource-disposer:0.16
mailer:1.34
kubernetes-client-api:5.4.1
view-job-filters:2.3
job-dsl:1.77
caffeine-api:2.9.1-23.v51c4e2c879c8
buildgraph-view:1.8
ghprb:1.42.2
pyenv-pipeline:2.1.2
shiningpanda:0.24
naginator:1.18.1
windows-slaves:1.0
run-condition:1.5
pam-auth:1.5.1
script-security:1.77
github:1.33.1
google-oauth-plugin:1.0.6
jdk-tool:1.0
google-compute-engine:4.3.8
docker-java-api:3.1.5.2
junit:1.51
workflow-durable-task-step:2.39
branch-api:2.6.5
workflow-basic-steps:2.23
dtkit-api:3.0.0
bootstrap5-api:5.0.2-1
git:4.7.2
matrix-project:1.19
structs:1.23
antisamy-markup-formatter:1.1
echarts-api:5.1.2-3
checks-api:1.7.0
command-launcher:1.2
docker-plugin:1.2.2
plain-credentials:1.7
pipeline-model-extensions:1.8.5
maven-plugin:3.12
embeddable-build-status:2.0.3
copyartifact:1.46.1
apache-httpcomponents-client-4-api:4.5.13-1.0
git-server:1.10
workflow-step-api:2.24
forensics-api:1.2.1
pipeline-build-step:2.13
throttle-concurrents:2.3
token-macro:2.15
docker-commons:1.17
extra-columns:1.23
pipeline-stage-tags-metadata:1.8.5
bootstrap4-api:4.6.0-3
pipeline-model-definition:1.8.5
data-tables-api:1.10.25-2
snakeyaml-api:1.29.1
pipeline-model-api:1.8.5
workflow-cps-global-lib:2.21
parameterized-trigger:2.41
okhttp-api:3.14.9
handlebars:3.0.8
ldap:2.7
plugin-util-api:2.4.0
conditional-buildstep:1.4.1
pipeline-input-step:2.12
oauth-credentials:0.4
workflow-job:2.41
cobertura:1.16
workflow-support:3.8
workflow-cps:2.93
pipeline-milestone-step:1.3.2
workflow-multibranch:2.26
font-awesome-api:5.15.3-4
ssh-credentials:1.19
jsch:0.1.55.2
ssh-slaves:1.32.0
greenballs:1.15.1
scm-api:2.6.4
github-api:1.123
lockable-resources:2.11
rebuild:1.32
popper2-api:2.5.4-3
external-monitor-job:1.4
credentials:2.5
jaxb:2.3.0
git-client:3.8.0
workflow-aggregator:2.6
jquery3-api:3.6.0-2
popper-api:1.16.1-2
jackson2-api:2.12.3
workflow-scm-step:2.13
kubernetes:1.30.1
matrix-auth:2.6.6
durable-task:1.38
trilead-api:1.0.13
workflow-api:2.46
pipeline-graph-analysis:1.11
kubernetes-credentials:0.9.0
github-branch-source:2.11.1
metrics:4.0.2.8
pipeline-rest-api:2.19
dashboard-view:2.17
credentials-binding:1.27
display-url-api:2.3.5
pipeline-stage-view:2.19
configuration-as-code:1.51
jjwt-api:0.11.2-9.c8b45b8bb173
variant:1.4
bulk-builder:1.5
ws-cleanup:0.39
sshd:3.0.4
cloudbees-folder:6.15
momentjs:1.1.1

• What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins controller LTS docker container and docker build agent

Reproduction steps

• Set up a kubernetes build cluster (unfortunately I don't know good reproduce steps here as I just used helm install on a test GKE account - you may not want to do the same)
• Create a new pipeline job using

pipeline {
    agent {
        kubernetes {
            yaml '''
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                  - name: python
                    image: python:2.7
                    args: ["sleep", "99d"]
'''
        }
    }
    
    stages {
        stage("Confirm virtualenv") {
            steps {
                container("python") {
                    sh "pip install virtualenv"
                    withPythonEnv("python") {
                        sh "python -c 'import sys; print(sys.executable)'"
                        sh "python-config --cflags"
                    }
                }
            }
        }
    }
}

• Run pipeline job

Results

Expected result:

Expected the commands inside the withPythonEnv block to print the executable and build flags for the virtual environment.

Actual result:

Executing sh script inside container python of pod python-info-29-r3hck-5kdvd-f2x82
Executing command: "python" "--version" 
exit
Python 2.7.18
Executing sh script inside container python of pod python-info-29-r3hck-5kdvd-f2x82
Executing command: "python" "-m" "virtualenv" "--python=python" "/home/jenkins/agent/workspace/python-info/.pyenv-python" 
exit
created virtual environment CPython2.7.18.final.0-64 in 254ms
  creator CPython2Posix(dest=/home/jenkins/agent/workspace/python-info/.pyenv-python, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
[Pipeline] {
[Pipeline] sh
+ python -c import sys; print(sys.executable)
/usr/local/bin/python
[Pipeline] sh
+ python-config --cflags
-I/usr/local/include/python2.7 -I/usr/local/include/python2.7 -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
[Pipeline] }
[Pipeline] // withPythonEnv

Given what I know about this plugin, I suppose this could be a kubernetes pipeline problem instead, with environment variables not getting passed through properly. And since this plugin hasn't been updated in a couple years, maybe this is no longer maintained? If that's the case, there is an obvious workaround of just not using this plugin, and explicitly handling the virtual environment myself in the sh commands. But thought I'd check if maybe this is a fixable bug, or if there's a known/official workaround.

(and yes, I'm aware that in theory there's no need for the virtualenv inside a container, because the container is isolated, but due to certain combinations of dependencies, scripts, etc, it's not feasible right now to try to set up a build without the virtualenv)

[danfoster]

I also see this problem. It appears the PYTHON_PATH envvar gets set, but the correct paths do not get prepended to PATH

[JCHacking]

Any update on this topic?
Is there a workaround to make it work?

[Nekto89]

I'm having issues with Docker. I assume that the reason might be the same. I'm using

docker.image('imagename:latest').inside('''-m=30G ...'') {
  sh '''python3 -m venv conan_venv'''
  //withPythonEnv('conan_venv/') {
  withPythonEnv(WORKSPACE+'/conan_venv/') {
    ...
  }
}

Do I understand correctly that Pyenv Pipeline Plugin currently doesn't support Docker Pipeline plugin? Or am I doing something wrong?

[Artalus]

Indeed it seems that the plugin does not work with Docker :(

    node('linux') {
        docker.image('python:3.8-slim-bullseye').inside {
            sh "echo python"
            withPythonEnv('python') {
                sh("""
                    which python
                    which python3
                """)
            }
        }
    }

16:26:22  [Pipeline] withDockerContainer
16:26:22  ubuntu-agent-18-1 does not seem to be running inside a container
16:26:22  $ docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/sandbox/tmp-python -v /home/jenkins/workspace/sandbox/tmp-python:/home/jenkins/workspace/sandbox/tmp-python:rw,z -v /home/jenkins/workspace/sandbox/tmp-python@tmp:/home/jenkins/workspace/sandbox/tmp-python@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** python:3.8-slim-bullseye cat
16:26:30  $ docker top e03c995ba0389789a334303d8f5d0b86213d3f938edcdbc916482b8ba1c13649 -eo pid,comm
16:26:30  [Pipeline] {
16:26:30  [Pipeline] sh
16:26:30  + echo python
16:26:30  python
16:26:30  [Pipeline] withPythonEnv
16:26:30  $ docker exec e03c995ba0389789a334303d8f5d0b86213d3f938edcdbc916482b8ba1c13649 python --version
16:26:30  $ docker exec e03c995ba0389789a334303d8f5d0b86213d3f938edcdbc916482b8ba1c13649 python -m venv /home/jenkins/workspace/sandbox/tmp-python/.pyenv-python
16:26:36  [Pipeline] {
16:26:36  [Pipeline] sh
16:26:37  + which python
16:26:37  /usr/local/bin/python
16:26:37  + which python3
16:26:37  /usr/local/bin/python3
16:26:37  [Pipeline] }
16:26:37  [Pipeline] // withPythonEnv
16:26:37  [Pipeline] }
16:26:37  $ docker stop --time=1 e03c995ba0389789a334303d8f5d0b86213d3f938edcdbc916482b8ba1c13649
16:26:40  $ docker rm -f e03c995ba0389789a334303d8f5d0b86213d3f938edcdbc916482b8ba1c13649
16:26:40  [Pipeline] // withDockerContainer

It successfully creates the venv directory, but fails to use it in sh steps.

[petacz]

The workaround that works for us:
environment { PATH = "$WORKSPACE/.pyenv-python3/bin:$PATH" }

and in the stage:
python3.11 -m venv $WORKSPACE/.pyenv-python3