Feature proposal: Shim launcher for Python

[J-M0]

• I was not able to find an open or closed issue matching what I'm seeing

I recently started working with Windows more and interacting with Git for Windows more and got to thinking about how to make it easier for Windows users to use git-filter-repo and get the full experience since installing it has proved problematic for Windows users. I would like to propose a small shim installed at /usr/bin/python and /usr/bin/python3 that would make using git-filter-repo work like it does on a UNIX install of git.

I have a rough draft here:

#!/bin/bash

# Look for all `python` commands in ${PATH}
found_pythons=($(type -ap python 2>/dev/null))

# Look for a valid python interpreter
for python in "${found_pythons[@]}"
do
    #  Make sure we don't call ourselves             Check that we aren't running the Microsoft Store shortcut
    if [[ ! ${python} =~ (\/usr)?\/bin\/python ]] && "${python}" --version &>/dev/null
    then
        exec "${python}" $@
    fi
done

# If we didn't find a `python`, try to use the py launcher
if command -v py &>/dev/null
then
    exec py $@
fi

echo "No python command found. Please install Python"
exit 1

This shell script will, in order

1. Find all python commands on the ${PATH}
2. Test each one to see if it is a valid python interpreter
3. If it finds a valid python, pass all arguments to the interpreter, else
4. Try to find the py launcher and pass all arguments to it, else
5. Exit with an error

I've tested it on my machine and it seems to work pretty well with a python.org Python and a Microsoft Store Python. I'm hoping with this shim, Git for Windows could start to include git-filter-repo in the same way it includes other extensions like git-flow and git-lfs, and that git-p4 could become a first-class citizen again.

[rimrul]

That's an interesting idea, but I have a few small remarks.

• The shim should probably check the ouput of python --version when running as python3 to ensure we don't run files with a python3 shebang through some old python2 we found.

• We might need to run non-MSYS2 python through winpty depending on wether were running in a tty.

[J-M0]

• The shim should probably check the ouput of python --version when running as python3 to ensure we don't run files with a python3 shebang through some old python2 we found.

See my reply here: #3661 (reply in thread)

• We might need to run non-MSYS2 python through winpty depending on wether were running in a tty.

Mm, yeah I keep forgetting about that one. I have yet to encounter that on Windows 10 after being away for so long but I remember hitting that on Windows XP.

[dscho]

Thank you for contributing the idea!

Check that we aren't running the Microsoft Store shortcut

Are you sure that this is still a valid concern, after msys2/msys2-runtime#27 was merged?

    exec py $@

Where does that py launcher come from?

[rimrul]

Thank you for contributing the idea!

Check that we aren't running the Microsoft Store shortcut

Are you sure that this is still a valid concern, after msys2/msys2-runtime#27 was merged?

Yes, because while we can now handle those execution aliases, the default version of that alias on a fresh windows doesn't link to a pre-installed python, but the windows store page to download python. The windows store can't execute the payload we want python to execute, so we check if this python knows --version. If it doesn't it's probably not a legit python.

    exec py $@

Where does that py launcher come from?

Python.org installers

https://docs.python.org/3/using/windows.html#from-the-command-line

[dscho]

I also wonder whether we really need to go through all the motions every single time python or python3 are called. Maybe we can do it only once, and then persist the path to the actual Python executable to use somewhere (whether in ~/bin/python or ~/bin/python3 as a simple exec /path/to/python "$@", or rather in a ~/.gitconfig type of file)?

[J-M0]

I'm unsure of how robust that would be. For official installs of Python at least, there several caveats I can think of with the official downloads from python.org:

1. Like Git for Windows, Python can be installed system-wide or just for a single user
2. Each version of Python installs into its own Python${MAJOR}${MINOR} folder completely isolated from any other versions
3. There are multiple ways to configure using python from the cmdline which the Python installers support:
   a. Putting the py launcher on ${PATH}
   b. Adding C:\path\to\Python${MAJOR}${MINOR} to the path
   c. Both of the above options at the same time
4. Since Python does not upgrade in place, the user may remove old python versions at any time which would make the stored /path/to/python go stale.

This reminded me though that Python records its installations in the registry for others to find

https://www.python.org/dev/peps/pep-0514/

which first got me thinking about how to parse that information. Then I thought maybe we could just borrow the py launcher from Python and tweak it if need be.

https://github.com/python/cpython/blob/main/PC/launcher.c

This would detect both python.org installed pythons and Microsoft Store pythons and we wouldn't have to worry about duplicating effort or possibly missing edge cases. It also handles parsing shebang lines on Windows so we get that for free too.

https://www.python.org/dev/peps/pep-0397/#shebang-line-parsing

I think I'm liking this idea the more I write about it.

[dscho]

Like Git for Windows, Python can be installed system-wide or just for a single user

Git for Windows' installer actually does not really support single-user installations. It is possible, but you really have to work hard to force the installer to do that.

I thought maybe we could just borrow the py launcher from Python and tweak it if need be.

https://github.com/python/cpython/blob/main/PC/launcher.c

This would detect both python.org installed pythons and Microsoft Store pythons and we wouldn't have to worry about duplicating effort or possibly missing edge cases. It also handles parsing shebang lines on Windows so we get that for free too.

https://www.python.org/dev/peps/pep-0397/#shebang-line-parsing

I think I'm liking this idea the more I write about it.

Sure, that sounds fine, as long as we can include it in a minimal fashion (from the link you gave it looks as if the launcher is part of the full mingw-w64-python3 package, but we obviously do not want to include all of its files).

[J-M0]

Git for Windows' installer actually does not really support single-user installations. It is possible, but you really have to work hard to force the installer to do that.

Huh, ok. I could have sworn there was an option for that, but I must've misremembered.

Sure, that sounds fine, as long as we can include it in a minimal fashion (from the link you gave it looks as if the launcher is part of the full mingw-w64-python3 package, but we obviously do not want to include all of its files).

As far as I can tell, the launcher is not part of any of the python packages since it wasn't designed for POSIX-style environments. That means it probably needs to be packaged on its own, which shouldn't be too difficult since it's only one file.

From some quick experimenting, it will probably have to be patched a little to run on the mingw64 toolchain since the launcher normally expects to be built with MSVC. I think I'll also work on making it pick a python depending on what name the launcher is invoked with, i.e. calling it as python2 or python3 does the right thing. I should be able to whip something up like that. I'll post a link when I have it

[dscho]

echo "No python command found. Please install Python"

It might be a really good idea to be more helpful here, by at least suggesting a link to the user, or even going so far as to offer taking the user there directly. Something like this:

git askyesno --title "Python required" \
    "No Python was found. Download and install from the Microsoft Store?" || exit 1
start https://www.microsoft.com/en-us/p/python-39/9p7qfqmjrfp7
# either wait in a busy loop, or ask the user to hit Enter when Python was installed, then re-exec

[rimrul]

git askyesno --title "Python required" \
    "No Python was found. Download and install from the Microsoft Store?" || exit 1
start https://www.microsoft.com/en-us/p/python-39/9p7qfqmjrfp7

That would require a check of the windows version at hand though. Older Windows versions don't work with that store. We probably also want some kind of link for the latest stable python instead of having to update that link all the time.

# either wait in a busy loop, or ask the user to hit Enter when Python was installed, then re-exec

Not sure if that works on the store version on Windows 8. It might have no app execution alias and no py launcher in that case and wont search the updated path.

[dscho]

Good points.

[dscho]

Final thought (for now): since the main aim for this thing is to support git filter-repo, shouldn't we make that command work? I.e. provide a git-filter-repo shim that will install Python and git-filter-repo if needed, then exec that?

[J-M0]

While my primary motivator was git-filter-repo, I wanted to make a general solution that would work for almost any git command written in Python. Like I said in my closing, I also wanted to be able to bring git-p4 back since it's a part of the core git suite.

[dscho]

Sure, but even for git-p4 we're not talking about any random Python script from the internet.

Most importantly, neither git filter-repo nor git p4 require interactive input, therefore we need not worry about using winpty.

And I really would love to have a way in Git for Windows where git filter-repo [...] would be simple and easy to use, by relieving users from the nitty-gritty details of installing and hooking up Python and the git_filter_repo package.

That would definitely improve the user experience, which I care about a lot.

[J-M0]

OK, I have the py launcher building under mingw64 with a PKGBUILD here: https://github.com/J-M0/git-for-windows-python-launcher

I'm going to see if I can make it automatically launch python 2 or 3 depending on if it's called as python2 or python3.

[dscho]

https://github.com/J-M0/git-for-windows-python-launcher

😢

[J-M0]

Oops, fixed

[dscho]

Hmm. I thought we'd make this a MINGW package rather than an MSYS one. More like https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-python-installer/PKGBUILD (and without requiring quilt).

Having had a look around, it seems to me as if there are better ways to get this, there seems to be a pyscript2exe.py script that is used elsewhere in the MINGW packages. Most likely there is a pip way to get that .exe, seeing as easy_install is deprecated (which otherwise could be used like the pyscript2exe.py script does it).

[J-M0]

Hmm. I thought we'd make this a MINGW package rather than an MSYS one. More like https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-python-installer/PKGBUILD

That was my intention too and it does build with MINGW's gcc. I've written PKGBUILDs before but I'm still learning how MSYS2 packaging works with all the other toolchains. It's also a little weird because I imagined the launcher living in /usr/bin even though it would be built with the MINGW toolchain which I realize is not how other packages are set up.

(and without requiring quilt).

What is wrong with quilt? I'm just using it to manage my patches so I don't have to write my own bash function to apply them manually. Also once I clean up my work, there probably will be fewer patches to apply as some were there only to help my testing.

Having had a look around, it seems to me as if there are better ways to get this, there seems to be a pyscript2exe.py script that is used elsewhere in the MINGW packages. Most likely there is a pip way to get that .exe, seeing as easy_install is deprecated (which otherwise could be used like the pyscript2exe.py script does it).

Using pip would generate .exe stubs on Windows based on the entry_points specification. They are essentially a very small Windows executable with a small python script glued on the end (literally, you can see the script with strings). git-filter-repo does declare an entry point, but using it also means installing git-filter-repo as a python package which the author doesn't really like supporting. Additionally, the stubs generated by the entry_points and pip install hard code a particular Python installation path, which means that git-filter-repo would need to be reinstalled whenever a Python version is removed and replaced on the system.

I'm trying to set this up so that git-filter-repo installs via make install as the author intended. To that end, I also started writing a PKGBUILD for git-filter-repo that I would hope would be included upstream in MSYS2 and by extension make it easy to include in the Git for Windows distribution.

I'm nearly finished with the changes I think need to be made to the py launcher. I just need to remove a testing patch or two and make sure the PKGBUILD is using the right toolchain. Once that's done I was going to submit it for feedback.

[J-M0]

Ok, sorry this took so long. I have the launcher in a state where I think we can start testing it. Right now my PKGBUILD will build the py launcher from the Windows python distribution, patching it so that it builds with MinGW and so that it doesn't call itself and forkbomb the machine. The launcher is installed in three places /usr/bin/python[23]? to catch most shebangs.

How can I include this in a build of the Git for Windows installer?