Welcome to light-speed-io Discussions!

[JackKelly]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

[maximedion2]

Hey there!

I know the project is currently on hold, but I was wondering if the code is currently in a working state? Not as something that can be used in any official capacity, but I'm trying to implement something much simpler (with io_uring) in a Zarr based query engine based on DataFusion, and I was looking at the code here for a few examples, and I wanted to make sure it's at least working before going any further.

Assuming it is, can I ask where file descriptors are handled in the workflow? For example, looking here, https://github.com/JackKelly/light-speed-io/blob/main/crates/lsio_uring/src/sqe.rs#L99, it's not clear to me where this file descriptor was ever set, starting from GetRanges, then going into GetRange, etc... The code seems to assume it's set, but I don't see it being set anywhere.

[JackKelly]

Yup, the code does work sufficiently to run some basic benchmarks!

The file descriptor is returned by the OpenAt operation, which is handled here:

https://github.com/JackKelly/light-speed-io/blob/main/crates%2Flsio_uring%2Fsrc%2Fget_ranges.rs#L82

[maximedion2]

Oh, yeah I see. I put together a very simple example, just to test out the how things work, based on a basic io-uring example from the docs, and I was using File::open() to open a file and then get the file descriptor, so that I can submit read operations to io uring. For the files descriptors to not go invalid in a loop, I either have to store them in a vec, or use mem::forget or something similar, which works, but it seems to add some massive overhead, a simple loop that just handles the files with no io_uring calls actually takes much longer than reading the data... I assume fetching the descriptors via io_uring operations is very lightweight?

Also, quick question, the logic you put together handles the user providing a range to read, but what if someone wanted to just provide a file and read the whole content? Looking more and what you did, it looks like a way to do that would be to retrieve the total file size from statx, which could then be used to set the buffer size for the read operation, do you think that's the right approach based on your experimentation?

Very interesting work btw, and thanks for getting back to me, this is (at the very least) a good source of information on how to use io_uring!

[JackKelly]

For the files descriptors to not go invalid in a loop, I either have to store them in a vec, or use mem::forget or something similar, which works, but it seems to add some massive overhead

LSIO uses a Tracker<Operation> to keep track of the ops in flight. The overhead should be too bad if you can pre-allocated the vec?

I assume fetching the descriptors via io_uring operations is very lightweight?

Yes (once the data is ready, of course!)

what if someone wanted to just provide a file and read the whole content? Looking more and what you did, it looks like a way to do that would be to retrieve the total file size from statx, which could then be used to set the buffer size for the read operation, do you think that's the right approach based on your experimentation?

Yes, exactly, if you want to be able to pre-allocate the buffer, and if the user doesn't know the filesize, then I think you have to use statx to get the filesize. (Which can be used to allocate the buffer and to give to the read op).

this is (at the very least) a good source of information on how to use io_uring!

Glad it's useful!

And yeah, as a general rule: I was quite surprised when writing LSIO at how much "effort" you have to put in to get top performance out of io_uring when reading from a fast PCIe 4 or PCIe 5 SSD. e.g. using multiple user-space threads etc.!

[maximedion2]

Hmm, interesting. I do remember seeing somewhere in the notes that you did get some nice performance in the end, but yeah it's quite involved.

I started off with something very simple (not even integrated in the zarr query engine, just a simple binary), my baseline is to read 100 20kb files with a simple loop using fs::read, and then I compared that with reading the same 100 files by submitting 100 openat/statx operations to an io uring submission queue, retrieving the results then submitting 100 read operations to the same queue, all on the main thread, no additional threads spawned or any threadpool.

The result is quite slow, about 1.5ms for the baseline that just uses fs::read, no io_uring, vs about 3ms with io_uring. I guess I need to put in more effort into this as you said, to get any improvement.

[maximedion2]

I'm curious, once you did set up all the logic, what kind of benchmarks did you run? Specifically, did you compare reading a big-ish file in multiple chunks to reading multiple smaller files (the latter probably having more overhead than the former, because of the need to open more files)?

And what did you compare against? fs::read(...), or maybe something like File::open(...).unwrap().read(...) (which appears to be much faster in some cases)?

[JackKelly]

I benchmarked against object_store::LocalFileSystem. The full details of how I benchmarked LSIO can be seen in the command I ran:

jack@jack-epyc-workstation:~/dev/rust/light-speed-io/crates/lsio_bench$ cargo run --release -- --filesize=41214400 --nrfiles=100 --blocksize=262144 --nr-worker-threads=8 --directory=/mnt/t700-2tb/lsio_bench

(copied from this comment)

Please see main.rs in lsio_bench for a description of the command line args.

[maximedion2]

Ah this is great, I was looking for that (a summary of performance, with incremental changes and thoughts on what worked and what didn't), thanks!