run inference on GPU or MPS #1155

mbui0327 · 2022-11-17T04:57:56Z

mbui0327
Nov 17, 2022

I could not find any document on running the diarization on GPU or MPS. The earlier version (v1) can be loaded into a torch pipeline and can be assigned device to it. Is there equivalent thing in v2.
Thanks

hbredin · 2022-11-17T08:02:25Z

hbredin
Nov 17, 2022
Maintainer

Setting the device used in pipelines manually is not currently supported.
pyannote.audio will automatically use GPU (not MPS) when available.
That being said, I'd be happy to look at a pull request adding this feature, if someone (you?) comes up with one.

2 replies

mbui0327 Nov 17, 2022
Author

Can you tell me where I should look into to do this? Took a quick look at audio/core/pipeline.py but could not find out how it is connected to torch pipeline. Thanks.

hbredin Nov 17, 2022
Maintainer

pyannote-audio/pyannote/audio/pipelines/speaker_diarization.py

Line 141 in d966b51

seg_device, emb_device = get_devices(needs=2)

fablau · 2022-11-27T00:57:26Z

fablau
Nov 27, 2022

If we can define the device, we could use the new pytorch optimized for M1 on Macs as explained here:

https://towardsdatascience.com/installing-pytorch-on-apple-m1-chip-with-gpu-acceleration-3351dc44d67c

0 replies

mbui0327 · 2022-11-28T17:01:16Z

mbui0327
Nov 28, 2022
Author

Yes, it is pretty trivial. I have it working. Will make a PR later this week. ~Bach

…

On Sat, Nov 26, 2022 at 4:57 PM Fabrizio Ferrari ***@***.***> wrote: If we can define the device, we could use the new pytorch optimized for M1 on Macs as explained here: https://towardsdatascience.com/installing-pytorch-on-apple-m1-chip-with-gpu-acceleration-3351dc44d67c — Reply to this email directly, view it on GitHub <#1155 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIF5GAIQT56X7YN4R5V6NDDWKKWYDANCNFSM6AAAAAASC6UFH4> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

0 replies

fablau · 2022-11-28T22:57:08Z

fablau
Nov 28, 2022

Awesome! Thank you!

…

On Nov 28, 2022, at 9:01 AM, Bach Bui ***@***.***> wrote: Yes, it is pretty trivial. I have it working. Will make a PR later this week. ~Bach On Sat, Nov 26, 2022 at 4:57 PM Fabrizio Ferrari ***@***.***> wrote: > If we can define the device, we could use the new pytorch optimized for M1 > on Macs as explained here: > > > https://towardsdatascience.com/installing-pytorch-on-apple-m1-chip-with-gpu-acceleration-3351dc44d67c > > — > Reply to this email directly, view it on GitHub > <#1155 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AIF5GAIQT56X7YN4R5V6NDDWKKWYDANCNFSM6AAAAAASC6UFH4> > . > You are receiving this because you authored the thread.Message ID: > ***@***.*** > com> > — Reply to this email directly, view it on GitHub <#1155 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMKWRG5F3DIKQWXS54T3K3WKTQORANCNFSM6AAAAAASC6UFH4>. You are receiving this because you commented.

0 replies

fablau · 2022-12-15T15:55:44Z

fablau
Dec 15, 2022

@mbui0327 Any news about all this?

0 replies

hbredin · 2022-12-21T17:33:21Z

hbredin
Dec 21, 2022
Maintainer

This PR adds a new pipeline.to(device) API. Would you mind testing it?

0 replies

fablau · 2022-12-22T01:41:21Z

fablau
Dec 22, 2022

Thank you! It works great! It is much, much faster! 20 seconds vs 3 minutes for a 40 seconds dialog. What an improvement! I just need to analyze the results because for some reason they seem to be different... I'll be back with more info.

0 replies

fablau · 2022-12-22T06:13:57Z

fablau
Dec 22, 2022

Yeah, for some reason, I get screwed results using the mps device. Is the code below the correct way to use it?

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization", use_auth_token="[my_token]")

pipeline.to("mps")

diarization = pipeline("audio_dialog.wav")

for turn, _, speaker in diarization.itertracks(yield_label=True):
	print(f"start={turn.start:.1f}s stop={turn.end:.1f}s speaker_{speaker}")

With the CPU, which takes over 3 minutes to process, I get the following correct results:

start=1.4s stop=13.3s speaker_SPEAKER_00
start=13.3s stop=15.3s speaker_SPEAKER_01
start=15.8s stop=32.0s speaker_SPEAKER_00
start=32.4s stop=37.5s speaker_SPEAKER_01

With the GPU (using pipeline.to("mps")) which takes just 20 seconds, I get the following results:

start=36.9s stop=37.5s speaker_SPEAKER_00

Any ideas?

0 replies

fablau · 2022-12-22T06:58:14Z

fablau
Dec 22, 2022

Well, after some research, I think the problem was that I didn't have PyTorch Nightly installed on my Mac. Fact is, after having created a new conda environment from scratch with the Nightly Pytorch on it, by performing:

pip3 install pyannote.audio

It seems to downgrade pytorch audio and other stuff so that once I execute the python script above, I get a bunch of errors:

/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/_internal/module_utils.py:99: UserWarning: Failed to import soundfile. 'soundfile' backend is not available.
  warnings.warn("Failed to import soundfile. 'soundfile' backend is not available.")
Traceback (most recent call last):
  File "diary.py", line 8, in <module>
    from pyannote.audio import Pipeline
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/pyannote/audio/__init__.py", line 29, in <module>
    from .core.inference import Inference
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/pyannote/audio/core/inference.py", line 34, in <module>
    from pyannote.audio.core.io import AudioFile
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/pyannote/audio/core/io.py", line 38, in <module>
    import torchaudio
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/__init__.py", line 1, in <module>
    from torchaudio import (  # noqa: F401
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/_extension.py", line 135, in <module>
    _init_extension()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/_extension.py", line 105, in _init_extension
    _load_lib("libtorchaudio")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/_extension.py", line 52, in _load_lib
    torch.ops.load_library(path)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torch/_ops.py", line 640, in load_library
    ctypes.CDLL(path)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/lib/libtorchaudio.so, 0x0006): Symbol not found: (__ZN2at4_ops9_pad_enum4callERKNS_6TensorEN3c108ArrayRefIxEExNS5_8optionalIdEE)
  Referenced from: '/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torchaudio/lib/libtorchaudio.so'
  Expected in: '/opt/homebrew/Caskroom/miniconda/base/envs/pyannote6/lib/python3.8/site-packages/torch/lib/libtorch_cpu.dylib'

Is pyannote compatible with PyTorch Nightly?

3 replies

adeviney Jan 13, 2023

@fablau I am having the same issue, and I am wondering -- why do you think the problem is (the lack of) PyTorch Nightly? 🤔

On my M1 mac, I am getting the same results you are after installing pyannote.audio as the current head of the develop branch and using pipeline.to("mps") (the pipeline being fit much faster, but the entire audio file being attributed to speaker 0).
However, the same thing also happened with Google Colab and their CUDA GPU.

So I am wondering if there is another issue?

fablau Jan 13, 2023

This is interesting @adeviney ... are you using the pipeline.to("mps") there too? If so, that's probably why... if you use a GPU, don't set the "mps" setting. And the fact you get the same kind of problem makes me think that maybe the problem is not an incompatibility with PyTorch Nightly but with the mps pipeline?

Could @hbredin chime in and comment on this?

adeviney Jan 13, 2023

Good point, although I'm pretty sure I used pipeline.to("cuda") for the Google colab test (or commented out that line altogether, assuming it still defaults to gpu if it's available). But I will confirm and see if I can figure it out later. :-)

petros94 · 2023-07-17T15:04:17Z

petros94
Jul 17, 2023

Has anyone managed to run pyannote on mps? 🤔

1 reply

fablau Jul 25, 2023

I haven't tried lately...

IKabitz · 2023-07-25T15:15:47Z

NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

Looks like an implementation needs to be made?

nahuel89p Sep 10, 2023

I think it's this? pytorch/pytorch#78044

nahuel89p · 2023-09-10T11:42:44Z

nahuel89p
Sep 10, 2023

Isn't it this issue?
pytorch/pytorch#78044

0 replies

jbeltran73-2 · 2024-11-01T18:00:33Z

jbeltran73-2
Nov 1, 2024

I have found this post that maybe would help

https://medium.com/buzzwordplus/speech-to-text-unlocking-speech-recognition-with-apples-machine-learning-framework-mlx-08416acbeb77

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run inference on GPU or MPS #1155

{{title}}

Replies: 13 comments 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

run inference on GPU or MPS #1155

Replies: 13 comments · 9 replies

hbredin Nov 17, 2022 Maintainer

mbui0327 Nov 17, 2022 Author

hbredin Nov 17, 2022 Maintainer

mbui0327 Nov 28, 2022 Author

hbredin Dec 21, 2022 Maintainer

Replies: 13 comments 9 replies

hbredin
Nov 17, 2022
Maintainer

mbui0327 Nov 17, 2022
Author

hbredin Nov 17, 2022
Maintainer

mbui0327
Nov 28, 2022
Author

hbredin
Dec 21, 2022
Maintainer