Hatch installed project without error but dependencies can't be imported

[manuel-koch]

I'm trying to use hatch to publish multiple packages from a single monorepo, see https://github.com/manuel-koch/hatch-monorepo-sandbox.

This works somewhat but I have problems for a particular aspect of my use-case.
I'm able to publish two libs from my monorepo using hatch, using these two project toml:

• my-sandbox-lib
• my-sandbox-lib-addon

But when I try to use those libs in a new environment, installation using hatch from another project my-sandbox-tool works, but those libs still can't be imported afterwards.

Any hint regarding possible errors in my project.toml or my usage of hatch is appreciated !

For example running the following commands in the mentioned repo from above, results in a somewhat broken python environment where the libs are installed successfully, but still can't be imported:

#
# Setup environment
#
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ make reset_env
hatch env prune
hatch --verbose env create \
    && hatch run -- python -c 'import sys; import platform; print(f"Setup python {{platform.python_version()}} environment done: {{sys.executable}}")'
Finished creating environment: default
Looking in indexes: https://user:****@private-pypi/pypi/ds-pypi/simple
Obtaining file:///Users/manuelkoch/workspace/item/hatch-monorepo-sandbox/my-sandbox-tool
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
Collecting my-sandbox-lib (from my-sandbox-tool==1.0.0)
  Using cached https://private-pypi/api/pypi/ds-pypi/my-sandbox-lib/0.1.dev1%2Bg64f599b.d20240227/my_sandbox_lib-0.1.dev1%2Bg64f599b.d20240227-py3-none-any.whl (1.3 kB)
Collecting my-sandbox-lib-addon (from my-sandbox-tool==1.0.0)
  Using cached https://private-pypi/api/pypi/ds-pypi/my-sandbox-lib-addon/0.1.dev1%2Bg64f599b.d20240227/my_sandbox_lib_addon-0.1.dev1%2Bg64f599b.d20240227-py3-none-any.whl (1.4 kB)
Collecting numpy~=1.22 (from my-sandbox-lib->my-sandbox-tool==1.0.0)
  Using cached https://private-pypi/api/pypi/ds-pypi/packages/packages/75/5b/ca6c8bd14007e5ca171c7c03102d17b4f4e0ceb53957e8c44343a9546dcc/numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
Building wheels for collected packages: my-sandbox-tool
  Building editable for my-sandbox-tool (pyproject.toml): started
  Building editable for my-sandbox-tool (pyproject.toml): finished with status 'done'
  Created wheel for my-sandbox-tool: filename=my_sandbox_tool-1.0.0-py3-none-any.whl size=1512 sha256=276e6add5171b120be37a1e2d7f9f9a51ff9e44b96012e9fcf7a0278a59680dd
  Stored in directory: /private/var/folders/4l/pq2frs651d1ccv5rtxxmwjww0000gn/T/pip-ephem-wheel-cache-9w8momnn/wheels/3c/46/07/01144bb91424fd73f3a5df46ef9cdc8fdbe5875346f19b4b09
Successfully built my-sandbox-tool
Installing collected packages: numpy, my-sandbox-lib, my-sandbox-lib-addon, my-sandbox-tool
Successfully installed my-sandbox-lib-0.1.dev1+g64f599b.d20240227 my-sandbox-lib-addon-0.1.dev1+g64f599b.d20240227 my-sandbox-tool-1.0.0 numpy-1.26.4
Finished installing project in development mode
Finished checking dependencies
Setup python 3.12.2 environment done: /Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/bin/python

#
# Start shell in environemt
#
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ hatch shell
source "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/bin/activate"

manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ source "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/bin/activate"

#
# Run the script from the project that makes use of the libs
#
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ my-sandbox-tool
Traceback (most recent call last):
  File "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/bin/my-sandbox-tool", line 7, in <module>
    from my_sandbox_tool.cli import main
  File "/Users/manuelkoch/workspace/item/hatch-monorepo-sandbox/my-sandbox-tool/my_sandbox_tool/cli.py", line 2, in <module>
    from my_sandbox_lib.func import library_function
ModuleNotFoundError: No module named 'my_sandbox_lib'

#
# List files in current python environment
#
(my-sandbox-tool) 
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ ls -la "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/bin"
total 96
drwxr-xr-x  17 manuelkoch  staff   544 Feb 27 18:33 .
drwxr-xr-x   6 manuelkoch  staff   192 Feb 27 18:33 ..
-rw-r--r--   1 manuelkoch  staff  2308 Feb 27 18:33 activate
-rw-r--r--   1 manuelkoch  staff  1575 Feb 27 18:33 activate.csh
-rw-r--r--   1 manuelkoch  staff  3138 Feb 27 18:33 activate.fish
-rw-r--r--   1 manuelkoch  staff  2823 Feb 27 18:33 activate.nu
-rw-r--r--   1 manuelkoch  staff  1650 Feb 27 18:33 activate.ps1
-rw-r--r--   1 manuelkoch  staff  1337 Feb 27 18:33 activate_this.py
-rwxr-xr-x   1 manuelkoch  staff   350 Feb 27 18:33 f2py
-rwxr-xr-x   1 manuelkoch  staff   352 Feb 27 18:33 my-sandbox-tool
-rwxr-xr-x   1 manuelkoch  staff   355 Feb 27 18:33 pip
-rwxr-xr-x   1 manuelkoch  staff   355 Feb 27 18:33 pip-3.12
-rwxr-xr-x   1 manuelkoch  staff   355 Feb 27 18:33 pip3
-rwxr-xr-x   1 manuelkoch  staff   355 Feb 27 18:33 pip3.12
lrwxr-xr-x   1 manuelkoch  staff    44 Feb 27 18:33 python -> /opt/homebrew/opt/[email protected]/bin/python3.12
lrwxr-xr-x   1 manuelkoch  staff     6 Feb 27 18:33 python3 -> python
lrwxr-xr-x   1 manuelkoch  staff     6 Feb 27 18:33 python3.12 -> python

(my-sandbox-tool) 
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ ls -la "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/lib/python3.12/site-packages"
total 32
drwxr-xr-x  14 manuelkoch  staff   448 Feb 27 18:33 .
drwxr-xr-x   3 manuelkoch  staff    96 Feb 27 18:33 ..
drwxr-xr-x   3 manuelkoch  staff    96 Feb 27 18:33 __pycache__
-rw-r--r--   1 manuelkoch  staff    71 Feb 27 18:33 _my_sandbox_tool.pth
-rw-r--r--   1 manuelkoch  staff    18 Feb 27 18:33 _virtualenv.pth
-rw-r--r--   1 manuelkoch  staff  4443 Feb 27 18:33 _virtualenv.py
drwxr-xr-x   6 manuelkoch  staff   192 Feb 27 18:33 my_sandbox_lib-0.1.dev1+g64f599b.d20240227.dist-info
drwxr-xr-x   6 manuelkoch  staff   192 Feb 27 18:33 my_sandbox_lib_addon-0.1.dev1+g64f599b.d20240227.dist-info
drwxr-xr-x   9 manuelkoch  staff   288 Feb 27 18:33 my_sandbox_tool-1.0.0.dist-info
drwxr-xr-x  42 manuelkoch  staff  1344 Feb 27 18:33 numpy
drwxr-xr-x   8 manuelkoch  staff   256 Feb 27 18:33 numpy-1.26.4.dist-info
drwxr-xr-x   9 manuelkoch  staff   288 Feb 27 18:33 pip
drwxr-xr-x  10 manuelkoch  staff   320 Feb 27 18:33 pip-24.0.dist-info
-rw-r--r--   1 manuelkoch  staff     0 Feb 27 18:33 pip-24.0.virtualenv

(my-sandbox-tool) 
manuelkoch in /workspace/hatch-monorepo-sandbox/my-sandbox-tool (main●)
$ ls -Rla "/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/lib/python3.12/site-packages/my_sandbox_"*
/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/lib/python3.12/site-packages/my_sandbox_lib-0.1.dev1+g64f599b.d20240227.dist-info:
total 32
drwxr-xr-x   6 manuelkoch  staff  192 Feb 27 18:33 .
drwxr-xr-x  14 manuelkoch  staff  448 Feb 27 18:33 ..
-rw-r--r--   1 manuelkoch  staff    4 Feb 27 18:33 INSTALLER
-rw-r--r--   1 manuelkoch  staff  758 Feb 27 18:33 METADATA
-rw-r--r--   1 manuelkoch  staff  412 Feb 27 18:33 RECORD
-rw-r--r--   1 manuelkoch  staff   87 Feb 27 18:33 WHEEL

/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/lib/python3.12/site-packages/my_sandbox_lib_addon-0.1.dev1+g64f599b.d20240227.dist-info:
total 32
drwxr-xr-x   6 manuelkoch  staff  192 Feb 27 18:33 .
drwxr-xr-x  14 manuelkoch  staff  448 Feb 27 18:33 ..
-rw-r--r--   1 manuelkoch  staff    4 Feb 27 18:33 INSTALLER
-rw-r--r--   1 manuelkoch  staff  846 Feb 27 18:33 METADATA
-rw-r--r--   1 manuelkoch  staff  436 Feb 27 18:33 RECORD
-rw-r--r--   1 manuelkoch  staff   87 Feb 27 18:33 WHEEL

/Users/manuelkoch/Library/Application Support/hatch/env/virtual/my-sandbox-tool/oKYsAZ4X/my-sandbox-tool/lib/python3.12/site-packages/my_sandbox_tool-1.0.0.dist-info:
total 48
drwxr-xr-x   9 manuelkoch  staff  288 Feb 27 18:33 .
drwxr-xr-x  14 manuelkoch  staff  448 Feb 27 18:33 ..
-rw-r--r--   1 manuelkoch  staff    4 Feb 27 18:33 INSTALLER
-rw-r--r--   1 manuelkoch  staff  576 Feb 27 18:33 METADATA
-rw-r--r--   1 manuelkoch  staff  793 Feb 27 18:33 RECORD
-rw-r--r--   1 manuelkoch  staff    0 Feb 27 18:33 REQUESTED
-rw-r--r--   1 manuelkoch  staff   87 Feb 27 18:33 WHEEL
-rw-r--r--   1 manuelkoch  staff  121 Feb 27 18:33 direct_url.json
-rw-r--r--   1 manuelkoch  staff   61 Feb 27 18:33 entry_points.txt

I zipped the artifacts of the libs ( those are uploaded to my private PyPi repo ) and attached them here:
tar_and_wheel_artifacts_of_libs.zip.txt

[manuel-koch]

When I peek into the content of the lib wheels, I can't find any lib implementation - likely that is the cause for the import error.
But why is the wheel "empty" ?

[manuel-koch]

Looks like the following does the trick, editing the project.toml of a lib

from

[tool.hatch.build.targets.sdist.force-include]
"../src/my_sandbox_lib" = "src/my_sandbox_lib"

[tool.hatch.build.targets.wheel]
packages = ["src/my_sandbox_lib"]

to

[tool.hatch.build.targets.sdist.force-include]
"../src/my_sandbox_lib" = "my_sandbox_lib"

[tool.hatch.build.targets.wheel.force-include]
"../src/my_sandbox_lib" = "my_sandbox_lib"

Afterwards the import error from above is gone and the cli tool can happily import my_sandbox_lib and my_sandbox_lib_addon.
I've updated my example repo to reflect the patch.