connect-jdbc

Kafka Connect JDBC test environment

This Docker Compose can be used to spin up an environment in which to explore and test the Kafka Connect JDBC source connector.

The associated blog article can be found at https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector

Table 1. Current environment

Confluent Platform	5.1.0
MySQL	8.0.13
Postgres	11.1
MS SQL Server	2017 (RTM-CU13) (KB4466404) - 14.0.3048.4 (X64)
Oracle	12.2.0.1.0

Getting a CLI session

• KSQL

  docker-compose exec ksql-cli ksql http://ksql-server:8088

• MySQL

  docker-compose exec mysql bash -c 'mysql -u root -p$MYSQL_ROOT_PASSWORD'

• Postgres

  docker-compose exec postgres bash -c 'psql -U $POSTGRES_USER $POSTGRES_DB'

• Oracle

  docker-compose exec oracle bash -c 'sqlplus sys/$ORACLE_PWD@localhost:1521/ORCLCDB as sysdba'

• MS SQL Server

  docker-compose exec mssql bash -c '/opt/mssql-tools/bin/sqlcmd -l 30 -S localhost -U sa -P $SA_PASSWORD'

Example Connector Configuration

• MySQL

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
          "name": "jdbc_source_mysql_01",
          "config": {
                  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                  "connection.url": "jdbc:mysql://mysql:3306/demo",
                  "connection.user": "connect_user",
                  "connection.password": "asgard",
                  "topic.prefix": "mysql-01-",
                  "mode":"bulk",
                  "poll.interval.ms" : 3600000
                  }
          }'

• Postgres

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
          "name": "jdbc_source_postgres_01",
          "config": {
                  "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                    "connection.url": "jdbc:postgresql://postgres:5432/postgres",
                  "connection.user": "connect_user",
                  "connection.password": "asgard",
                  "topic.prefix": "postgres-01-",
                  "mode":"bulk",
                  "poll.interval.ms" : 3600000
                  }
          }'

• Oracle

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
                  "name": "jdbc_source_oracle_01",
                  "config": {
                          "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                          "connection.url": "jdbc:oracle:thin:@oracle:1521/ORCLPDB1",
                          "connection.user": "connect_user",
                          "connection.password": "asgard",
                          "topic.prefix": "oracle-01-",
                          "table.whitelist" : "NUM_TEST",
                          "mode":"bulk",
                          "poll.interval.ms" : 3600000
                          }
                  }'

• MS SQL Server

  curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
                  "name": "jdbc_source_mssql_01",
                  "config": {
                          "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
                          "connection.url": "jdbc:sqlserver://mssql:1433;databaseName=demo",
                          "connection.user": "connect_user",
                          "connection.password": "Asgard123",
                          "topic.prefix": "mssql-01-",
                          "table.whitelist" : "demo..num_test",
                          "mode":"bulk",
                          "poll.interval.ms" : 3600000
                          }
                  }'