Fluent bit in_kafka not working properly

[dipendra-singh]

I am trying to fetch messages from kafka using the following config but the config does not print anything and at times throws out: decoder error : -4

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level debug

[INPUT]
    Name           kafka
    Brokers        <brokers>
    group_id       <consumer-group>
    Topics         <topic>


[OUTPUT]
    Name stdout
    Match *

Additionally, when I try to push these messages to azure adx, the fluent bit debug logs only prints that it is reading from kafka and the kusto simply doesnt emit any debug logs.

[dipendra-singh]

This also just simply prints this:

[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received
[2023/07/10 20:19:01] [debug] [input:kafka:kafka.0] kafka message received

and doesnt do anything

[MohamedEHJ]

I got this error too, if you try this:

[OUTPUT]
    Name        stdout
    Format      json_lines 

console print
{"date":1689168792.482278,"topic":"my-topic","partition":0,"offset":80,"error":null,"key":null}

Maybe it's related to this issue #7481.

[agup006]

We've connected offline and are going to look if the PR for 7481 can fix this. Will most likely have a nightly build for folks to try next week