How does AbstractFileSystem.cat_file work for http filesystems?

[lobis]

I would like to use fsspec (not exclusively) to fetch bytes ranges from files over http. It appears that the best suited api to use would be the cat_file which supports an start and end parameters.

I could not find any documentation on how this works in detail. I would love to know if fetching a given range of a file does A or B:

• A. Requests (downloads) the whole file and then returns a slice of this file.
• B. Requests only a given range of this file and returns it (if the server supports it).

In particular I am interested in performing requests for different ranges of a single file in parallel, currently using a single cat_file for each range. If I am downloading the whole file in each call then I need to rethink my strategy (previous implementation uses multi-part http requests).

I would love any comment in this regard.

Thanks,
Luis

[martindurant]

It does B, by including a Range header in the request (see here). This will be a problem when the server doesn't respect the request - sometimes you get an error back (which is fine), sometimes the wrong range (perhaps depending on encoding) and sometimes the whole file for every request - you should experiment first.

If you want to fetch many ranges from one or more files, you might well want to use cat_ranges, designed for this job. See also fsspec.utils.merge_offset_ranges, for ways to combine a set of ranges (adjacent requests should be combined and nearby ones maybe, since each request takes some overhead even when async).

[martindurant]

I see that cat_ranges seems to be missing a doc-string...

[lobis]

Thank you!