Kafka connection

[Kubo-SVK]

Hey guy,

I am having trouble to run ipfixcol2 JSON-kafka plugin.

Setup is like this:
Computer A:

Zookeeper with Kafka broker on standard port 9092.
IP address: 192.168.1.2

Computer B:

ipfixcol2 with json-kafka
softflowd - reporting data to localhost:4739
IP address 192.168.1.144

Here you can see that ipfixcol is keeping connection opened

But none of the data is being transfered.

Those are my settings. Almost 1:1 with example file.

<!--
  Receive flow data simultaneously over TCP and UDP and store them on a local
  drive in a nfdump compatible format (multiple instances of the same input
  plugin).
-->
<ipfixcol2>
  <!-- Input plugins -->
  <inputPlugins>
    <input>
      <name>TCP collector</name>
      <plugin>tcp</plugin>
      <params>
        <!-- List on port 4739 -->
        <localPort>4739</localPort>
        <!-- Bind to all local adresses -->
        <localIPAddress></localIPAddress>
      </params>
    </input>

    <input>
      <name>UDP collector</name>
      <plugin>udp</plugin>
      <params>
        <!-- List on port 4739 -->
        <localPort>4739</localPort>
        <!-- Bind to all local adresses -->
        <localIPAddress></localIPAddress>
      </params>
    </input>
  </inputPlugins>

  <!-- Output plugins -->
  <outputPlugins>
    <output>
      <name>JSON output</name>
      <plugin>json</plugin>
        <params>
          <tcpFlags>formatted</tcpFlags>
          <timestamp>formatted</timestamp>
          <protocol>formatted</protocol>
          <ignoreUnknown>true</ignoreUnknown>
          <ignoreOptions>true</ignoreOptions>
          <nonPrintableChar>true</nonPrintableChar>
          <octetArrayAsUint>true</octetArrayAsUint>
          <numericNames>false</numericNames>
          <splitBiflow>false</splitBiflow>
          <detailedInfo>false</detailedInfo>
          <templateInfo>false</templateInfo>

          <outputs>
              <kafka>
                  <name>Send to Kafka</name>
                  <brokers>192.168.1.2:9092</brokers>
                  <topic>ipfix</topic>
                  <blocking>false</blocking>
                  <partition>unassigned</partition>

                  <!-- Zero or more additional properties -->
                  <property>
                      <key>compression.codec</key>
                      <value>lz4</value>
                  </property>
              </kafka>
          </outputs>
      </params>
    </output>
  </outputPlugins>
</ipfixcol2>

This is an image after shuting down kafka.

It is fine because thess errors showed up just after shutting down kafka but its quiet strange that it is reporting connection problem to localhost all over the place.

EDIT: Forgot to mention, but its working when kafka is running on localhost

[Lukas955]

Hi,

can you try to increase verbosity of the JSON output plugin? By default, it prints only errors but there might be some hidden warnings. Just add <verbosity> parameter to the configuration of the plugin:

 <outputPlugins>
    <output>
      <name>JSON output</name>
      <verbosity>debug</verbosity>
      <plugin>json</plugin>
          ...
      </params>
    </output>
  </outputPlugins>

Lukas

[Lukas955]

Hi,

did you manage to resolve the issue?

Lukas

[Kubo-SVK]

Hi

Sorry I will try it this weekend because currently I am not able to test it

[Kubo-SVK]

Hello,

I am back with figuring out whats wrong.

There's a screenshot where you can see that parameter for bootstrap.server is set to 192.168.1.2:9092, but librdkafka is trying to connect to localhost. Not sure whats wrong

[Lukas955]

Hi,

so let's summarize, IPFIXcol2 is able to connect to the broker 192.168.1.2 (according to the Wireshark dump), but is not sending any data to it. Based on the log on the collector, it appears to be trying to connect to localhost, which it is failing to do.

I suspect the problem to be in the server configuration on the broker. The broker advertise localhost as the producer destination instead of its own address/hostname. I recommend yout to check parameters of the configuration file server.properties (especially advertised.listeners).

This article might help: https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

Lukas

[Lukas955]

Hi,

did you try to check the broker configuration? Was the article helpful in any way?

Lukas