You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I created a custom step to read data from a jsonlines file in the local fileystem, but noted that load_dataset already has the functionality, we only need to expose it.
Describe the solution you'd like
Allow reading local files using LoadHubDataset, something similar to the following
I also wanted to bring up the fact that we now have LoadDataFromDicts and LoadHubDataset; most of the times, myself, as an user, end up writing LoadDataFromHub assuming the standard is LoadDataFrom ...; maybe this is a nice time to unify everything under the same naming pattern? I believe this will be easier and avoid conflicts in the future (we can set a deprecation warning on LoadHubDataset for the next couple of releases to avoid breaking any existing pipeline).
Closing this with #691. @rasdani will let you know once we add the docs, but the you could use the new LoadDataFromDisk to work with the s3 datasets saved to disk.
Is your feature request related to a problem? Please describe.
I created a custom step to read data from a jsonlines file in the local fileystem, but noted that
load_dataset
already has the functionality, we only need to expose it.Describe the solution you'd like
Allow reading local files using
LoadHubDataset
, something similar to the followingDescribe alternatives you've considered
Creating a custom step when needed.
Additional context
Ref: https://huggingface.co/docs/datasets/loading#local-and-remote-files
The text was updated successfully, but these errors were encountered: