Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use custom extension types in C extensions for free-threading python #126267

Closed
shinyano opened this issue Nov 1, 2024 · 12 comments
Closed
Labels
topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@shinyano
Copy link

shinyano commented Nov 1, 2024

Bug report

Bug description:

Python3.13t seems to be unable to use C extensions which use new extension types?

According to chapter 2.1. The Basics in official guide, I wrote 3 files: custom.c, pyproject.toml and setup.py which were identical to the code provided in this tutorial.

In the same location I executed D:\python3.13\python3.13t.exe -m pip install . The extension got installed successfully but when I executed import custom in python3.13t.exe, Python just exited without any output or errors:

C:\Users\zhang>python3.13t
Python 3.13.0 experimental free-threading build (tags/v3.13.0:60403a5, Oct  7 2024, 09:53:29) [MSC v.1941 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom

C:\Users\zhang>

Then I tried pip install with stable version of 3.13, and successfully imported custom module:

C:\Users\zhang>D:\python3.13\python.exe
Python 3.13.0 (tags/v3.13.0:60403a5, Oct  7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom
>>> ^Z

C:\Users\zhang>

I'm using Windows11 and Python3.13, setuptools==75.2.0. The complete code is as follows:

custom.c:

// custom.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    PyObject_HEAD
    /* Type-specific fields go here. */
} CustomObject;

static PyTypeObject CustomType = {
    .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = PyDoc_STR("Custom objects"),
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
};

static PyModuleDef custommodule = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "custom",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
};

PyMODINIT_FUNC
PyInit_custom(void)
{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0) 
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    if (PyModule_AddObjectRef(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

pyproject.toml:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "custom"
version = "1"

setup.py:

from setuptools import Extension, setup
setup(ext_modules=[Extension("custom", ["custom.c"])])

CPython versions tested on:

3.13

Operating systems tested on:

Windows

@shinyano shinyano added the type-bug An unexpected behavior, bug, or error label Nov 1, 2024
@Zheaoli
Copy link
Contributor

Zheaoli commented Nov 1, 2024

Would you mind to build the CPython with --pydebug flag and run the code and post the result here?

I can test it later.

@rruuaanng

This comment has been minimized.

@shinyano
Copy link
Author

shinyano commented Nov 1, 2024

Would you mind to build the CPython with --pydebug flag and run the code and post the result here?

I can test it later.

Sure, I'll try this out. Should I use 3.13 source code or newest?

@Zheaoli
Copy link
Contributor

Zheaoli commented Nov 1, 2024

Sure, I'll try this out. Should I use 3.13 source code or newest?

Both would be great, But you can try 3.13 branch first!

@colesbury
Copy link
Contributor

Hi @shinyano, thanks for the bug report. The Windows installer has an issue where not all the required macros are defined for the free threaded build. You can work around it by adjusting your setup.py to define the Py_GIL_DISABLED macro:

from setuptools import Extension, setup
import sysconfig

macros = []
if sysconfig.get_config_var("Py_GIL_DISABLED"):
    macros.append(("Py_GIL_DISABLED", 1))

setup(ext_modules=[Extension("custom", ["custom.c"], define_macros=macros)])

@Zheaoli
Copy link
Contributor

Zheaoli commented Nov 1, 2024

thanks for the bug report. The Windows installer has an issue where not all the required macros are defined for the free threaded build. You can work around it by adjusting your setup.py to define the Py_GIL_DISABLED macro:

Should we update the doc to remind this different or this would be fixed in 3.14?

@colesbury
Copy link
Contributor

colesbury commented Nov 1, 2024

I don't think the issue will be fixed with the Windows installer any time soon (if ever).

We should:

@colesbury
Copy link
Contributor

Here is the corresponding issue in setuptools: pypa/setuptools#4662

@Zheaoli
Copy link
Contributor

Zheaoli commented Nov 1, 2024

I don't think the issue will be fixed with the Windows installer any time soon (if ever).

We should:

SGTM

I have noticed that you have already commit to the setuptools. So I will update documentation.

@colesbury
Copy link
Contributor

The PR to distutils turned out to be easy. Let's not bother with changing the newtypes tutorial -- it's already fairly long and I think the distutils/setuptools change will probably happen relatively quickly.

@Zheaoli
Copy link
Contributor

Zheaoli commented Nov 1, 2024

The PR to distutils turned out to be easy. Let's not bother with changing the newtypes tutorial -- it's already fairly long and I think the distutils/setuptools change will probably happen relatively quickly.

SGTM, BTW I think distutils should release a new version ASAP. I will discuss this in distutils repo

@shinyano
Copy link
Author

shinyano commented Nov 4, 2024

After some test I noticed that I'm doing set Py_GIL_DISABLED=1 wrong. I mistakenly thought that "set Py_GIL_DISABLED=1" meant setting the environment variable Py_GIL_DISABLED to 1, but it's actually a macro definition.

from setuptools import Extension, setup
import sysconfig

macros = []
if sysconfig.get_config_var("Py_GIL_DISABLED"):
    macros.append(("Py_GIL_DISABLED", 1))

setup(ext_modules=[Extension("custom", ["custom.c"], define_macros=macros)])

This is exacly what I needed to do, thank you all for the discussion. And I'm sorry for taking up your time with such a basic error.

@shinyano shinyano closed this as completed Nov 4, 2024
@erlend-aasland erlend-aasland closed this as not planned Won't fix, can't repro, duplicate, stale Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

6 participants