SQL input plugin

[mudim]

Hi all, hope all well. Looking for some clarification around sql input plugin, in context of logs.

Documentation is clear on how the state is maintained so that we don't fetch same records on restarts. However, when deployed as daemonset, want to understand how the state is replicated across the plugins (one one each pod). Because, if the state is not replicated, wouldn't each of the input query same data there by duplicating log records?

Cheers

[patrick-stephens]

Do you mean the tail input plugin using the db parameter which maintains an SQL database of offsets for each inode?

This DB does indeed have to be persisted and specific to the node it is running on. There is no replication done by FB, this must be handled by mounting the right DB for a node into the pod running for that daemonset.

You could do this with a host mount, a PV and then separate directories/files per node, etc.

[mudim]

Hi Patrick, thanks for coming back so quickly. No, not for the tail plugin, it's for the sql input plugin, https://github.com/fluent/fluent-plugin-sql.

[patrick-stephens]

This is nothing to do with Fluent Bit, it's for Fluentd so I would raise it on that specific repo or ask in Slack.

[mudim]

Ah, my bad. Didn't check that clearly. Will raise it there, thank u