Skip to content

This eccenca Corporate Memory plugin allows for sending and receiving messages from Apache Kafka.

License

Notifications You must be signed in to change notification settings

eccenca/cmem-plugin-kafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cmem-plugin-kafka

This eccenca Corporate Memory plugin allows for sending and receiving messages from Apache Kafka.

eccenca Corporate Memory workflow pypi version license

Installation

cmemc -c my-cmem admin workspace python install cmem-plugin-kafka

Development

Running Test

This plugin needs needs running Kafka and Corporate Memory orchestrations:

In order to setup access to your Corporate Memory uses cmemc's config eval command to fill environment variables:

$ eval $(cmemc -c my-cmem config eval)

In order to setup access to your Kafka, write the connection details to the .env file:

$ cat .env
KAFKA_BOOTSTRAP_SERVER=localhost:9093
KAFKA_SECURITY_PROTOCOL=PLAINTEXT

To run a Kafka orchestration locally, you can use task:

$ task custom:kafka:start
task: [custom:kafka:start] docker-compose -f docker/docker-compose.yml up --wait --no-color --force-recreate --renew-anon-volumes

[+] Running 2/2
 ⠿ Container docker-zookeeper-1  Healthy                                          1.1s
 ⠿ Container docker-kafka-1      Healthy                                          1.1s

Having Kafka as well as Corporate Memory in place, run the test suite with task check.

confluent-python installation

ARM based Macs

  • make sure brew doctor has no issues
  • install with brew install librdkafka
  • Provide this in your environment (based on this answer)
export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib
  • test build from source in separate environment with:
    • pip install https://files.pythonhosted.org/packages/fb/16/d04dded73439266a3dbcd585f1128483dcf509e039bacd93642ac5de97d4/confluent-kafka-1.8.2.tar.gz
  • then try poetry install

Kafka CLI Utility

kcat (formerly kafkacat) is a command-line utility that you can use to test and debug Apache Kafka® deployments. kcat is an open-source utility, available at https://github.com/edenhill/kcat.

To send messages

docker run -it --rm \
        edenhill/kcat \
                -b kafka-broker:9092 \
                -t test \
                -K: \
                -P 

To consume messages

ocker run -it --rm \
        edenhill/kcat \
           -b kafka-broker:9092 \
           -C \
           -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\tOffset: %o\n--\n' \
           -t test