[BUG] 501 (Not Implemented) Response when trying to load HF dataset

[jphme]

Distilabel Version 1.1.1.

Trying to load a private HF Dataset with LoadHubDataset .

I get

│ ╭─────────────────────────────────── locals ───────────────────────────────────╮                 │
│ │   config = None                                                              │                 │
│ │  headers = {'Authorization': 'Bearer hf_correct_token'} │                 │
│ │   params = {'dataset': 'xyz/private_dataset'}                             │                 │
│ │  repo_id = 'xyz/private_dataset'                                         │                 │
│ │ response = <Response [501]>                                                  │                 │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AssertionError: Failed to get 'xyz/private_dataset' dataset info. Make sure you have set the HF_TOKEN environment variable if it is a private dataset.

I have HF_Token in my enf, the Bearer token shown is correct and I can load the dataset just fine with load_dataset.

I REALLY think the hacky custom loading implementation should be disabled by default, this caused us so much headaches already... why don't add some opt-in streaming HF Dataset loading and just use load_dataset for everything else?

[alvarobartt]

Hi here @jphme sorry for the inconvenience! I believe that may be due to the fact that your dataset is private and Hugging Face now set the Datasets Server on private repositories for Pro users only I'm afraid :/

[gabrielmbmb]

Hi @jphme, we will work on something to avoid using the API.

[jphme]

Hi here @jphme sorry for the inconvenience! I believe that may be due to the fact that your dataset is private and Hugging Face now set the Datasets Server on private repositories for Pro users only I'm afraid :/

Ah, that might be it, thanks 👍 .

We are even Corporate Pro users. But don't you think the current implementation is flawed when
a) the library can't be used properly by the 90%+ of non-pro users and
b) the error message is completely non-descript

and all this for a nice-to-have-feature (don't having to load the dataset in full) that doesn't affect ~95+% (for us so far 100%) of usecases? ;-)

Sorry don't want to sound too negative, I just spent the better part of an hour trying to figure out why this isn't working.

[alvarobartt]

We are even Corporate Pro users. But don't you think the current implementation is flawed when
a) the library can't be used properly by the 90%+ of non-pro users and
b) the error message is completely non-descript

Fair, indeed the message has already been updated as we noticed about this a couple days ago and should roll out in the next distilabel release, but yes, we'll try to do better and offer a better (less hacky and, so on, most likely less efficient) solution.

[rasdani]

One solution could be to load the dataset in streaming mode and fetch a single row.
This way you get the features and don't have to load the whole dataset in advance.

Maybe you can get the features even without fetching a row.

[jphme]

We are even Corporate Pro users. But don't you think the current implementation is flawed when
a) the library can't be used properly by the 90%+ of non-pro users and
b) the error message is completely non-descript

Fair, indeed the message has already been updated as we noticed about this a couple days ago and should roll out in the next distilabel release, but yes, we'll try to do better and offer a better (less hacky and, so on, most likely less efficient) solution.

Sounds great, many thanks!. I think sometimes the tradeoff between optimizing performance + optimizing usability should be more in the direct of usability; but this is often very hard to know in advance...

[plaguss]

Closing this with #691, needs some more examples in the docs still.
The idea now is trying to download the data using get_dataset_info and if that fails, call load_dataset.