io_uring_buf_ring_cq_advance: ensure tail store is ordered with buffers

We skipped the store ordering for the bundled helper that updates both
the CQ and buffer rings, but this isn't always safe - if an in-kernel
consumer (such as an SQPOLL or io-wq thread) is simultaneously grabbing
a buffer that was just provided prior to calling this helper AND we're
on a weakly ordered architecture that can reorder stores, then it is
possible to have a tiny window of a few cycles where a consumer can find
the tail updated but the buffer not updated.

On x86/x86-64 this can never happen, for example, and for performant use
cases, it's not possible either as they would not use SQPOLL or io-wq
offload and hence the thread itself will always be both the producer and
the consumer. But it is theoretically possible, so we should cover all
the bases.

This should still work better than updating the CQ and buffer ring
separately, only because we can tightly bundle the two ordered stores.
And having them back-to-back will be faster than having other stores (or
loads) in-between.

Fixes: c4a1070 ("Add combined cq+buf ring advance helper")
Link: #1039
Signed-off-by: Jens Axboe <[email protected]>

src/include/liburing.h

@@ -1463,7 +1463,7 @@ IOURINGINLINE void __io_uring_buf_ring_cq_advance(struct io_uring *ring,
 						  struct io_uring_buf_ring *br,
 						  int cq_count, int buf_count)
 {
-	br->tail += buf_count;
+	io_uring_buf_ring_advance(br, buf_count);
 	io_uring_cq_advance(ring, cq_count);
 }