Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Use Python's limited C API #171

Open
drmfinlay opened this issue Oct 21, 2023 · 18 comments
Open

Proposal: Use Python's limited C API #171

drmfinlay opened this issue Oct 21, 2023 · 18 comments
Assignees
Labels
enhancement New feature or request

Comments

@drmfinlay
Copy link
Member

At the moment, Natlink must be compiled for each minor 32-bit version of Python. It has been a pain-point getting the right PYD file loaded. I wonder if it might be worthwhile to use Python's limited API, documented here and available since Python 3.2. If Natlink could be compiled this way, the same PYD file would work on Python 3.8, 3.9, 3.10 and so on.

It may not be possible to do this with Natlink, I'll need to look into the code first. If it can be done, I would be willing to work on this. I think it would simplify the project and make it easier to maintain going forward.

@drmfinlay
Copy link
Member Author

I've looked into this now. There are a few issues, but I think it's possible.

@LexiconCode LexiconCode added the enhancement New feature or request label Oct 21, 2023
@drmfinlay
Copy link
Member Author

I've started working on this here: https://github.com/drmfinlay/natlink2/tree/use_py3_stable_abi

My repository (natlink2) doesn't have many of the more recent changes. I used an older revision (without the PyConfig stuff) to reduce the amount of initial work, and to make building a PYD file easier.

Seems to be progressing nicely. The project cannot yet be built without a few prior changes to CMakeLists.txt.

The natConnect() function's optional parameter is a little tricky to deal with using only the limited API. @quintijn, do you know if there is any benefit to passing False (default) to natConnect()? Most users seem to pass True or 1.

@drmfinlay
Copy link
Member Author

drmfinlay commented Oct 25, 2023

I think I am now able to answer my own question. It looks like the only upside to not enabling thread safety is that the Natlink window's "Reload Everything" menu button works.

I have been able to successfully use a Natlink PYD file built for Python 3.4 (stable ABI) with Python 3.8, 3.9 and 3.11. Dragonfly's test suite passes normally. So that's nice. No thread safety yet, however. But then, Natlink doesn't have it on by default anyway.

@LexiconCode
Copy link
Member

Presumably, this might make porting to python 11/12 easier.

@drmfinlay
Copy link
Member Author

It should do, yes. Natlink would only need to have one installer for Python 3.

@dougransom
Copy link
Member

@quintijn Does unimacro spawn any threads? Or any of the timers?

@quintijn
Copy link
Contributor

quintijn commented Oct 26, 2023 via email

@drmfinlay
Copy link
Member Author

Sorry, yes, it is a difficult question. I've solved the problem I was having with thread safety. Threads, timers and the natConnect() function are working normally with my changes.

@drmfinlay
Copy link
Member Author

I've got it mostly working now in my natlink2 repo.
https://github.com/drmfinlay/natlink2/tree/use_py3_stable_abi

A PYD can be built and will work out of process on Python 3.3+. As for normal use as a DNS compatibility module, there is a dependency issue with python3.dll. I think this was probably solved a while ago in this repo.

@drmfinlay
Copy link
Member Author

I had to stop working on this in November. The main roadblock here is my limited understanding of how to change the installer logic. Perhaps someone might be able to help me with that at some point?

I wanted to mention, for consideration, an idea I had. I was thinking that, since the installer would work with essentially any 32-bit Python version 3.x, there should be an option in the installer for which version to install/use. Only one version can be used at a time, for complex reasons. Such functionality would also need to exist in the NatLink configurations utilities -- something similar to the "register" button in the old 4.x series GUI.

Let me know if you guys think this is still worth pursuing. I still think the changes would be useful, especially from a maintenance perspective.

@quintijn
Copy link
Contributor

Thanks for all these things, Dane. It is hard for me to judge the things. Also because I am still (and for a long time by now) busy getting a python3 version of Natlink working with all the things around it (like other packager, just now busy getting Unimacro things checked)

Do I understand well, that you make a special python install, just for Natlink to run on? I have other programs having such a dedicated python install. Then the Natlink python install does not have any connection with possible other python installs?

@quintijn
Copy link
Contributor

Dane, do you work with Dragon 16? Can you test if playEvents (and playString) work with you C Limited API??

@drmfinlay
Copy link
Member Author

Sorry for the late reply.

Yes, I normally use Natlink with a system Python installation that isn't used for much else. But that is mainly because Natlink requires 32-bit Python.

I don't work with Dragon 16. However, I can tell you that the issue with playEvents and playString would remain if Python's C limited API were to be used.

Using that API shouldn't do anything other than change the Python DLL requirement of the _natlink_core.pyd file from python310.dll to python3.dll, assuming Python 3.10 is used. That will allow using the same _natlink_core.pyd file with Python 3.8, Python 3.9, Python 3.10, and so on without recompiling.

@drmfinlay
Copy link
Member Author

Thinking about my proposal here, I think it is something that should have been done years ago when the C++ code was made compatible with Python 3. I don't think there's much point now.

@quintijn
Copy link
Contributor

quintijn commented Jun 6, 2024

Dane, I am rereading issues of natlink and related. Pffff, what a lot of work you did here, with no concrete result. Sorry I cannot judge all implications of this, but my great appreciations for this work, which will probably not be used then.

In general, we have two types of users,

  1. no programmers, they don't care which python is installed, even don't know what python is.
  2. programmers, who will use a 64bit version of python. Installing some 32bit version for natlink will not hurt, but more important, "we" can choose which one is suitable, as it will not be used for other applications.

I don't know what I want to say with this, only that some dedicated install will be sufficient.

Greetings, Quintijn

@drmfinlay
Copy link
Member Author

drmfinlay commented Jun 7, 2024 via email

@LexiconCode
Copy link
Member

LexiconCode commented Sep 27, 2024

Will this change require Python to be on path?

Anyone with Python developments going to have a 64-bit install on path.

@LexiconCode LexiconCode reopened this Sep 27, 2024
@drmfinlay
Copy link
Member Author

drmfinlay commented Sep 28, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants