[Failing Test]: Unpin transformers in pydocs precommit

[tvalentyn]

What happened?

A recent transformers release https://pypi.org/project/transformers/ is causing an error:

WARNING: autodoc: failed to import module 'huggingface_inference' from module 'apache_beam.ml.inference'; the following exception was raised:
Traceback (most recent call last):
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1130, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/home/valentyn/.pyenv/versions/3.8.3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/pipelines/__init__.py", line 73, in <module>
    from .text_to_audio import TextToAudioPipeline
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/pipelines/text_to_audio.py", line 22, in <module>
    from ..models.speecht5.modeling_speecht5 import SpeechT5HifiGan
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/models/speecht5/modeling_speecht5.py", line 27, in <module>
    from ...activations import ACT2FN
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/activations.py", line 250, in <module>
    mish = get_activation("mish")
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/activations.py", line 238, in get_activation
    return ACT2FN[activation_string]
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/activations.py", line 210, in __getitem__
    return cls(**kwargs)
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/activations.py", line 161, in __init__
    if version.parse(torch.__version__) < version.parse("1.9.0"):
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/packaging/version.py", line 52, in parse
    return Version(version)
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/packaging/version.py", line 196, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/apache_beam/ml/inference/huggingface_inference.py", line 39, in <module>
    from transformers import Pipeline
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1120, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/target/.tox-py38-docs/py38-docs/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1132, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):
expected string or bytes-like object

Command exited with non-zero status 1
60.34user 2.97system 1:03.21elapsed 100%CPU (0avgtext+0avgdata 585712maxresident)k
10344inputs+150304outputs (17major+160626minor)pagefaults 0swaps
py38-docs: exit 1 (63.22 seconds) /home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python> time /home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/scripts/generate_pydoc.sh pid=223800
py38-docs: commands_post[0]> bash /home/valentyn/projects/beam/beam/beam/sdks/python/test-suites/tox/pycommon/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
.pkg: _exit> python /home/valentyn/projects/beam/beam/beam/build/gradleenv/2019945900/lib/python3.8/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  py38-docs: FAIL code 1 (205.81=setup[139.99]+cmd[0.00,0.32,2.23,0.02,63.22,0.02] seconds)
  evaluation failed :( (206.03 seconds)

Issue Failure

Failure: Test is continually failing

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[riteshghorse]

Looks like new pipeline 'text_to_audio' is added in 4.32 which broke our PyDocs. Although I would note that the error is originating from the code they have added for this. We should not upgrade to 4.32 until that is resolved.

[riteshghorse]

Found the root cause in the transformers library and notified them here huggingface/transformers#25669. Also sent a PR at huggingface/transformers#25670

[damccorm]

Repurposing this issue to track follow up; once huggingface/transformers#25669 or another fix is in, we should unpin transformers