Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mieb] google/siglip-large-patch16-384 fails on ImageNet10Clustering #1417

Open
Muennighoff opened this issue Nov 9, 2024 · 1 comment
Open
Labels
mieb The image extension of MTEB

Comments

@Muennighoff
Copy link
Contributor

ERROR:mteb.evaluation.MTEB:Error while evaluating ImageNet10Clustering: mean must have 1 elements if it is an iterable, got 3
Traceback (most recent call last):
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 91, in <module>
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch_size=32)
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run
    raise e
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 425, in run
    results, tick, tock = self._run_eval(
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 300, in _run_eval
    results = task.evaluate(
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate
    scores[hf_subset] = self._evaluate_subset(
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskImageClustering.py", line 54, in _evaluate_subset
    metrics = evaluator(model, encode_kwargs=encode_kwargs)
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/ClusteringEvaluator.py", line 41, in __call__
    image_embeddings = model.get_image_embeddings(
  File "/data/niklas/mieb/mteb/mteb/models/siglip_models.py", line 74, in get_image_embeddings
    inputs = self.processor(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/processing_siglip.py", line 113, in __call__
    image_features = self.image_processor(images, return_tensors=return_tensors)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 41, in __call__
    return self.preprocess(images, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/utils/generic.py", line 852, in wrapper
    return func(*args, **valid_kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/image_processing_siglip.py", line 231, in preprocess
    images = [
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/siglip/image_processing_siglip.py", line 232, in <listcomp>
    self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 111, in normalize
    return normalize(
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_transforms.py", line 398, in normalize
    raise ValueError(f"mean must have {num_channels} elements if it is an iterable, got {len(mean)}")
ValueError: mean must have 1 elements if it is an iterable, got 3

@isaac-chung isaac-chung added the mieb The image extension of MTEB label Nov 9, 2024
@gowitheflow-1998
Copy link
Contributor

fixed in #1424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mieb The image extension of MTEB
Projects
None yet
Development

No branches or pull requests

3 participants