libpython error when using "Python in q" functionality

[erichards97]

Describe the bug
When importing PyKx into a q session (i.e. \l pykx.q), a 'libpython error is returned. This can be fixed by symlinking libpython*.so (see below), but this shouldn't be necessary.

To Reproduce
Dockerfile

FROM ubuntu:22.04

WORKDIR /work

COPY prep/l64.zip /work
COPY prep/kc.lic /work

ENV QHOME=/work/kdb

RUN apt-get update && \
 apt-get install -y python3.10 python3.10-venv libpython3.10 unzip curl
 
RUN unzip l64.zip -d kdb && \
 python3.10 -m venv .venv && \
 . .venv/bin/activate && \
 pip install pykx find-libpython && \
 python -c "import pykx;pykx.install_into_QHOME(to_local_folder=True)" 

Then

sudo docker build . --tag pykx
sudo docker run --rm -it pykx

Then

. .venv/bin/activate
kdb/l64/q
\l pykx.q

--> Doesn't work with libpython error

To Fix

ln -s /usr/lib/x86_64-linux-gnu/libpython3.10.so.1 /usr/lib/x86_64-linux-gnu/libpython3.10.so
kdb/l64/q
\l pykx.q

--> Does work

Expected behavior
PyKx should work without having to symlink libpython. The find-libpython Python package successfully returns the correct path, and is mentioned in your docs as a dependency. Perhaps this should be used?

>>> from find_libpython import find_libpython
>>> find_libpython()
'/usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0'

If L5-12 from embedPy lives on in PyKx, that could explain the issue, as the "L" variable returns the incorrect path to libpython.

[cmccarthy1]

Hey @erichards97

Thanks for raising this! We can look into what we can do to improve the experience of this.

We have relied directly on find-libpython in the past (prior to 1.6.1) but we had seen issues with this not finding the correct shared object for the Python version that's being used in complex user environments and on Windows.

There is an environment variable PYKX_PYTHON_LIB_PATH which you can set to the path of the shared object that you wish to use should you not want to create the symlink. In your case this would be something akin to

export PYKX_PYTHON_LIB_PATH= '/usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0'

I'll try to provide an update in the week or so once we have had a chance to look into options

[cmccarthy1]

Hey @erichards97

Fundamentally I think we're a little stuck in the requirement for the split between how the search works but the following is a workaround which while manual for the moment I can look to include as part of a patch in the next couple of weeks

The following mimics what I'm considering adding

FROM ubuntu:22.04

WORKDIR /work

COPY prep/l64.zip /work
COPY prep/kc.lic /work

ENV QHOME=/work/kdb
ENV PYKX_USE_FIND_LIBPYTHON=True

RUN apt-get update && \
 apt-get install -y python3.10 python3.10-venv libpython3.10 unzip curl
 
RUN unzip l64.zip -d kdb && \
 python3.10 -m venv .venv && \
 . .venv/bin/activate && \
 pip install pykx find-libpython && \
 python -c "import pykx;pykx.install_into_QHOME(to_local_folder=True)" && \
 sed -i '71i if["true"~lower getenv `PYKX_USE_FIND_LIBPYTHON;setenv[`PYKX_PYTHON_LIB_PATH;raze system util.whichPython," -c\\"from find_libpython import find_libpython;print(find_libpython())\\""]];' /work/pykx.q

Essentially adding an env-var to choose the use of find-libpython and then using this to set the PYKX_PYTHON_LIB_PATH for a user rather than requiring you to dynamically set it.

[erichards97]

Thanks for digging into this @cmccarthy1 .

We've verified that find-libpython returns the correct path in all of our setups, so the proposed env var sounds good to us. Please let us know if this makes its way into a future PyKx release.

[erichards97]

Hi @cmccarthy1 - do you know if this functionality will be added in a future release? Many thanks

[cmccarthy1]

Hey @erichards97,

Yes, the intention is that we'll be adding something for this either in our 3.0 or 3.1 release so will have a fix for this before the end of the year.