GH-43949: [C++] io::BufferedInput: Fixing the invalid state with SetBufferSize #44387
+32
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
@pitrou @felipecrv @wgtmac This might be a bit critical fixing Also cc @biljazovic sorry for so slow replying |
wgtmac
reviewed
Oct 12, 2024
| @@ -51,6 +51,7 @@ class BufferedBase { | |||
| return !is_open_; | |||
| } | |||
|
|
|||
| // Reset the `buffer_` to a new buffer of size `buffer_size_` | |||
There was a problem hiding this comment.
Suggested change
| // Reset the `buffer_` to a new buffer of size `buffer_size_` | |
| // Resize buffer_ to buffer_size_ |
Let's keep consistent variable names in the comment across this file.
| @@ -284,17 +285,30 @@ class BufferedInputStream::Impl : public BufferedBase { | |||
|
|
|||
| // Resize internal read buffer. Note that the internal buffer-size | |||
| // should be not larger than the raw_read_bound_. | |||
There was a problem hiding this comment.
Suggested change
| // should be not larger than the raw_read_bound_. | |
| // should not be larger than the raw_read_bound_. |
Comment on lines
+288
to
+289
| // | ||
| // SetBufferSize will not change the buffer_pos_ and bytes_buffered_. |
There was a problem hiding this comment.
Suggested change
| // | |
| // SetBufferSize will not change the buffer_pos_ and bytes_buffered_. | |
| // It will not change buffer states including buffer_pos_ and bytes_buffered_. |
Comment on lines
+309
to
+310
| new_buffer_size = | ||
| std::min(new_buffer_size, buffer_size_ + (raw_read_bound_ - raw_read_total_)); |
There was a problem hiding this comment.
Suggested change
| new_buffer_size = | |
| std::min(new_buffer_size, buffer_size_ + (raw_read_bound_ - raw_read_total_)); | |
| new_buffer_size = | |
| std::min(new_buffer_size, buffer_pos_ + buffer_size_ + (raw_read_bound_ - raw_read_total_)); |
Isn't buffer_pos_ required to be kept as well?
| @@ -350,7 +364,7 @@ class BufferedInputStream::Impl : public BufferedBase { | |||
| } | |||
|
|
|||
| Status DoBuffer() { | |||
| // Fill buffer | |||
| // Refill the buffer from the raw stream with `buffer_size_` bytes. | |||
There was a problem hiding this comment.
Suggested change
| // Refill the buffer from the raw stream with `buffer_size_` bytes. | |
| // Fill the buffer from the raw stream with at most `buffer_size_` bytes. |
github-actions
bot
added
awaiting committer review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
See #43949
The problem is
PeekandReadboth callsSetBufferSize, however:Readimplicit says that, whenSetBufferSizeor read, the previous buffer is not being required. In this scenerio,bytes_buffered_is always 0, sinceReadwill consume all data. Andnew_buffer_size == std::min(new_buffer_size, (raw_read_bound_ - raw_read_total_))Peekstill requires the buffer-size here. So,bytes_buffered_might not 0. WhenPeek, thenew_buffer_sizewould less than expected, which shrinks the bufferWhat changes are included in this PR?
Update the Logic of
SetBufferSizebytes_buffered_ == 0,SetBufferSizecan discard the current bufferSetBufferSizeshould resize minimal tobuffer_size_ + (raw_read_bound_ - raw_read_total_), since it should considering the current bufferAre these changes tested?
Yes
Are there any user-facing changes?
Bugfix