pip3 install in the recipe fails and image is not created

[statsconchris]

Version of Singularity
singularity-ce version 3.9.5-bionic

Describe the bug
Since saturday my singularity recipe fails to create the image. The problem starts just at the pip installation line. For example, I may have the line pip3 install mysqlclient in the recipe and it stops building the image there
The error shows that I require an environment or use instead apt install python3-xyz. (the full error is copied at the end of this post as it is very long)
Is it possible to create an environment inside the container? I tried, added the following in the recipe and got a fatal error:

• python3 -m venv .venv
• source .venv/bin/activate
  /.post.script: 46: source: not found
  FATAL: While performing build: while running engine: exit status 127

I did not try apt install python3-xyz because I have several pip packages to be installed and some of them are not listed with apt install python3-xyz

How to solve this problem?

To Reproduce
add a pip3 install

Expected behavior
the image should be created

OS / Linux Distribution
Which Linux distribution are you using?
"Ubuntu 18.04.6 LTS"

Supplemental material
error while using pip3 install

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Pyt
hon installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
FATAL: While performing build: while running engine: exit status 1

[dtrudg]

Hello, as this is a usage question, and is not a bug in Singularity I'm going to transfer it to a discussion thread.

[dtrudg]

@statsconchris - the best thing here would be to share your definition file, so that people can see what you are trying to accomplish.

None of the issues suggests a problem with Singularity itself, but instead some limitations with the containers you are using to build on.

That error message you get with pip3 install is due to a change in python on Debian/Ubuntu distributions, where they are trying to discourage people from using the system pip to install packages into the system python locations (as this can cause conflicts).

Usually the source: not found error would be because you have a minimal container where /bin/sh is being used intead of /bin/bash as the shell. You can try using . instead of source. I.E. . .venv/bin/activate.

[statsconchris]

Indeed your solution solves the problem. How can I use /bin/bash in my container instead of a minimal setup?

If anybody wants to recreate the error, here I include a simple recipe file... the error will appear as soon as pip3 install is taken.

BootStrap: debootstrap                                         
OSVersion: stable                                              
MirrorURL: http://ftp.de.debian.org/debian/                    
                                                               
                                                               
%setup                                                        
#                   
                                                               
%files                                                        
#    
                                                               
%post                                                          
    # update repos ------------------------------------------- 
    apt update                                                 
    apt upgrade -y                                             
                                                               
    # install basic python libs -------------------------------
    apt install -y python3 python3-pip python3-venv 
                                                               
    # install python dependencies -----------------------------
    pip3 install numpy                                         
    pip3 install rpyc                                          
                                                               
    # clean-up ------------------------------------------------
    apt autoremove -y                                          
    apt clean                                                  
    #----------------------------------------------------------
                                                               
%environment                                                   
    #                             
                                                               
%runscript                                                     
    # empty                                                    

[dtrudg]

Indeed your solution solves the problem. How can I use /bin/bash in my container instead of a minimal setup?

My apologies... it's not because of a minimal container. It's because %post is run with /bin/sh and that's a symlink to dash on Debian, rather than bash in other distributions.

You can add -c /bin/bash to your %post line to make the script run under bash

%post -c /bin/bash
    # This will now run through /bin/bash

With regard to the def file, and the error with pip3.... there's nothing Singularity can do about that. It's a limitation that has been added in Debian's python3-pip. If this definition file previously worked, it means that Debian have changed their python3-pip since then.

[statsconchris]

Hi @dtrudg, while the image is created, there is an extra problem related to pip modules not found....

This extra issue is solved by adding in %post after the pip3 install:
echo '. /opt/.venv/bin/activate' >> $SINGULARITY_ENVIRONMENT

Note that I'm using as path /opt/ so the env creation should read:

python3 -m venv /opt/.venv
source /opt/.venv/bin/activate