docs: produce message size rfc #4202
Conversation
fraidev
force-pushed
the
rfc_produce_msg_size
branch
3 times, most recently
from
8b495a7
to
7efe6bb
Compare
fraidev
force-pushed
the
rfc_produce_msg_size
branch
from
7efe6bb
to
f60994d
Compare
fraidev
force-pushed
the
rfc_produce_msg_size
branch
from
f60994d
to
eaab9a5
Compare
|
|
||
| 1. Handling Larger Messages than Batch Size | ||
|
|
||
| If a single record exceeds the defined `batch_size`, Fluvio will process the record as a standalone request, ensuring that larger messages are not discarded or delayed. If the record does not exceed the `batch_size`, Fluvio will process the record as part of an already existing batch or create a new one if the batch is full. |
There was a problem hiding this comment.
Hmmm, is this true? Customers are complaining of packets being dropped if they are larger than the batch size.
There was a problem hiding this comment.
I wrote the expectation, I'll rewrite it with what is doing now and what should be changed.
There was a problem hiding this comment.
not sure why this behavior is needed. should move to enhancement if this doesn't applies to original problem
There was a problem hiding this comment.
this can be subsumed into exiting batching behavior. if batch_size is exceed, then batch is flushed
|
|
||
| 3. Compression Behavior | ||
|
|
||
| Fluvio will ensure that configuration limits that use size constraints, such as `batch_size` and `max_request_size` will only use the uncompressed message size. |
There was a problem hiding this comment.
That's the problem we need to solve. The algorithm needs to compute the result after compression, as that's what's carried in the payload.
There was a problem hiding this comment.
As batch is a concept before the message, I really think that we should not use or try to guess the compression size for that.
Right now, Fluvio is just "trying to guess" the compression, multiplying every uncompressed size to half (0.5) when it's a compressed batch.
About max_request_size I don't see problem to use compression size for that, but Kafka does not use the compression size for max_request_size. So if we want that, it's better to use another name to not generate confusion for the clients.
|
Never mind previous comments. Here is clarifying semantics.
|
|
|
||
| ### Batch Size | ||
|
|
||
| `batch_size` will define the maximum size of a batch of records that can be sent by the producer. If a record exceeds this size, Fluvio will process the record as a standalone message. |
There was a problem hiding this comment.
batch_size should be ultimate raw limit. again not sure what this address
There was a problem hiding this comment.
I think this behavior will be confused, other products has the same parameter config batch_size. In other products, Its behavior is to just check if the record will be inside a batch or not.
I think that we should use the compressed size for
Sure, Kafka for example uses 16kb for |
|
|
||
| ### Compression | ||
|
|
||
| `batch_size` and `max_request_size` will only use the uncompressed message size. |
There was a problem hiding this comment.
see suggestion. Having batch_size applies to final raw batch should simplify calculation.
TL;DR
batch_sizeproducer config must not reject large records, just send them directly.Create a new
max_request_sizeproducer config that must reject large messages.I am using
max_request_sizebecause Kafka usesmax.request.sizebut we can change it to other config name.Compression sizes should not be used for these producer configs.
Related: