You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I keep getting the error when trying to install from source starting from when pytorch was bumped to 2.4.0. It works fine for commits before that. Doing pip install numpy also didn't help.
Also tried running these which usually helps fix any compile errors previously.
/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/_subclasses/functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
Building wheels for collected packages: aphrodite-engine
Building editable for aphrodite-engine (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building editable for aphrodite-engine (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [197 lines of output]
/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/_subclasses/functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
cpu = _conversion_method_template(device=torch.device("cpu"))
running editable_wheel
creating /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info
writing /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/dependency_links.txt
writing entry points to /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/entry_points.txt
writing requirements to /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/requires.txt
writing top-level names to /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/top_level.txt
writing manifest file '/tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine.egg-info/SOURCES.txt'
creating '/tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine-0.6.1.post1+cu121.dist-info'
creating /tmp/pip-wheel-mhtbla3c/.tmp-945vfgf_/aphrodite_engine-0.6.1.post1+cu121.dist-info/WHEEL
running build_py
running build_ext
Using MAX_JOBS=20 as the number of jobs.
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build type: RelWithDebInfo
-- Target device: cuda
-- Found Python: /home/owen/miniconda3/envs/aphrolatest/bin/python (found version "3.11.9") found components: Interpreter Development.Module Development.SABIModule
-- Found python matching: /home/owen/miniconda3/envs/aphrolatest/bin/python.
-- Found CUDA: /home/owen/miniconda3/envs/aphrolatest (found version "12.1")
-- The CUDA compiler identification is NVIDIA 12.1.105
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /home/owen/miniconda3/envs/aphrolatest/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found CUDAToolkit: /home/owen/miniconda3/envs/aphrolatest/include (found version "12.1.105")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Caffe2: CUDA detected: 12.1
-- Caffe2: CUDA nvcc is: /home/owen/miniconda3/envs/aphrolatest/bin/nvcc
-- Caffe2: CUDA toolkit directory: /home/owen/miniconda3/envs/aphrolatest
-- Caffe2: Header version is: 12.1
-- /home/owen/miniconda3/envs/aphrolatest/lib/libnvrtc.so shorthash is 8144a3bc
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- Autodetected CUDA architecture(s): 8.6 8.6
-- Added CUDA NVCC flags for: -gencode;arch=compute_86,code=sm_86
CMake Warning at /tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:120 (append_torchlib_if_found)
CMakeLists.txt:67 (find_package)
-- Found Torch: /tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/lib/libtorch.so
-- Enabling core extension.
-- CUDA supported arches: 6.0;6.1;7.0;7.5;8.0;8.6;8.9;9.0
-- CUDA target arches: 86-real
-- CMake Version: 3.30.3
-- CUTLASS 3.5.1
-- CUDART: /home/owen/miniconda3/envs/aphrolatest/lib/libcudart.so
-- CUDA Driver: /home/owen/miniconda3/envs/aphrolatest/lib/stubs/libcuda.so
-- NVRTC: /home/owen/miniconda3/envs/aphrolatest/lib/libnvrtc.so
-- Default Install Location: install
-- Found Python3: /home/owen/miniconda3/envs/aphrolatest/bin/python3.11 (found suitable version "3.11.9", minimum required is "3.5") found components: Interpreter
-- Make cute::tuple be the new standard-layout tuple type
-- CUDA Compilation Architectures: 70;72;75;80;86;87;89;90;90a
-- Enable caching of reference results in conv unit tests
-- Enable rigorous conv problem sizes in conv unit tests
-- Using NVCC flags: --expt-relaxed-constexpr;-DCUTE_USE_PACKED_TUPLE=1;-DCUTLASS_TEST_LEVEL=0;-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1;-DCUTLASS_CONV_UNIT_TEST_RIGOROUS_SIZE_ENABLED=1;-DCUTLASS_DEBUG_TRACE_LEVEL=0;-Xcompiler=-Wconversion;-Xcompiler=-fno-strict-aliasing;-lineinfo
fatal: not a git repository (or any of the parent directories): .git
-- CUTLASS Revision: Unable to detect, Git returned code 128.
-- Configuring cublas ...
-- cuBLAS Disabled.
-- Configuring cuBLAS ... done.
-- Completed generation of library instances. See /tmp/tmpdryt1j0w.build-temp/_deps/cutlass-build/tools/library/library_instance_generation.log for more information.
-- Enabling C extension.
-- Enabling moe extension.
-- Configuring done (17.7s)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LIBNVTOOLSEXT
linked by target "_core_C"in directory /home/owen/aphro-latest/aphrodite-engine
-- Generating done (2.2s)
CMake Generate step failed. Build files cannot be regenerated correctly.
Traceback (most recent call last):
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 138, in run
self._create_wheel_file(bdist_wheel)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 341, in _create_wheel_file
files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 264, in _run_build_commands
self._run_build_subcommands()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 291, in _run_build_subcommands
self.run_command(name)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 98, in run
_build_ext.run(self)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "<string>", line 210, in build_extensions
File "<string>", line 193, in configure
File "/home/owen/miniconda3/envs/aphrolatest/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/owen/aphro-latest/aphrodite-engine', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpfcoaxk7c.build-lib/aphrodite', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmpdryt1j0w.build-temp', '-DAPHRODITE_TARGET_DEVICE=cuda', '-DAPHRODITE_PYTHON_EXECUTABLE=/home/owen/miniconda3/envs/aphrolatest/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=20']' returned non-zero exit status 1.
/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:973: _DebuggingTips: Problem in editable installation.
!!********************************************************************************
An error happened while installing `aphrodite-engine`in editable mode.
The following steps are recommended to help debug this problem:
- Try to install the project normally, without using the editable mode.
Does the error still persist?
(If it does, try fixing the problem before attempting the editable mode).
- If you are using binary extensions, make sure you have all OS-level
dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
- Try the latest version of setuptools (maybe the error was already fixed).
- If you (or your project dependencies) are using any setuptools extension
or customization, make sure they support the editable mode.
After following the steps above, if the problem still persists and
you think this is related to how setuptools handles editable installations,
please submit a reproducible example
(see https://stackoverflow.com/help/minimal-reproducible-example) to:
https://github.com/pypa/setuptools/issues
See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
********************************************************************************!!cmd_obj.run()
Traceback (most recent call last):
File "/home/owen/miniconda3/envs/aphrolatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in<module>main()
File "/home/owen/miniconda3/envs/aphrolatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/owen/miniconda3/envs/aphrolatest/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
return hook(wheel_directory, config_settings, metadata_directory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 459, in build_editable
return self._build_with_temp_dir(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 318, in run_setup
exec(code, locals())
File "<string>", line 460, in<module>
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 138, in run
self._create_wheel_file(bdist_wheel)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 341, in _create_wheel_file
files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 264, in _run_build_commands
self._run_build_subcommands()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 291, in _run_build_subcommands
self.run_command(name)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 98, in run
_build_ext.run(self)
File "/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "<string>", line 210, in build_extensions
File "<string>", line 193, in configure
File "/home/owen/miniconda3/envs/aphrolatest/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/home/owen/aphro-latest/aphrodite-engine', '-G', 'Ninja', '-DCMAKE_BUILD_TYPE=RelWithDebInfo', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmpfcoaxk7c.build-lib/aphrodite', '-DCMAKE_ARCHIVE_OUTPUT_DIRECTORY=/tmp/tmpdryt1j0w.build-temp', '-DAPHRODITE_TARGET_DEVICE=cuda', '-DAPHRODITE_PYTHON_EXECUTABLE=/home/owen/miniconda3/envs/aphrolatest/bin/python', '-DNVCC_THREADS=1', '-DCMAKE_JOB_POOL_COMPILE:STRING=compile', '-DCMAKE_JOB_POOLS:STRING=compile=20']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building editable for aphrodite-engine
Failed to build aphrodite-engine
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (aphrodite-engine)
The text was updated successfully, but these errors were encountered:
Numpy isn't an issue, that's mostly a benign error. The real issue is here:
-- Configuring done (17.7s) CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: LIBNVTOOLSEXT
It seems your setup is missing libnvtoolsext. Probably an incomplete CUDA installation. Would you mind trying ./runtime.sh aphrodite run EleutherAI/pythia-70m (or any other model)? That should create a conda env with a full cuda installation, then build aphrodite there.
Numpy isn't an issue, that's mostly a benign error. The real issue is here:
-- Configuring done (17.7s)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
LIBNVTOOLSEXT
It seems your setup is missing libnvtoolsext. Probably an incomplete CUDA installation. Would you mind trying ./runtime.sh aphrodite run EleutherAI/pythia-70m (or any other model)? That should create a conda env with a full cuda installation, then build aphrodite there.
Huh ok. So it's an issue with cuda toolkit installation? Somehow aprhodite versions with pytorch <2.4.0 works fine though. Which makes this confusing. I will try that and report back.
Your current environment
How did you install Aphrodite?
I keep getting the error when trying to install from source starting from when pytorch was bumped to 2.4.0. It works fine for commits before that. Doing pip install numpy also didn't help.
Also tried running these which usually helps fix any compile errors previously.
conda install -y -c "nvidia/label/cuda-12.1.1" cuda conda install nccl -y conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia -y
/tmp/pip-build-env-84gx5762/overlay/lib/python3.11/site-packages/torch/_subclasses/functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
The text was updated successfully, but these errors were encountered: