Skip to content
This repository has been archived by the owner on Dec 7, 2021. It is now read-only.

Confluent Platform Ansible installation testing using Vagrant

License

Notifications You must be signed in to change notification settings

OneCricketeer/cp-ansible-vagrant

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

cp-ansible-vagrant

git clone [email protected]:cricket007/cp-ansible-vagrant.git
cd cp-ansible-vagrant
git submodule init && git submodule update --remote

Using Vagrant to get up and running.

  1. Install VirtualBox https://www.virtualbox.org/
  2. Install Vagrant http://www.vagrantup.com/
  3. Vagrant plugins
vagrant plugin install vagrant-hostmanager vagrant-vbguest
# Optional
vagrant plugin install vagrant-cachier # Caches & shares package downloads across VMs

Define the hosts

Several profiles have already been defined in the profiles/ directory.

By default, kafka-1 will be started. You can override this behavior by exporting the CP_PROFILE variable prior to starting Vagrant. This variable does not persist and will need re-defined for each new terminal session.

All profiles have at least one broker and one zookeeper unless otherwise mentioned.

Profile Description
kafka-1
kafka-cluster-3
  • 3 Zookeepers
  • 3 Brokers
kafka-registry-ksql-1
  • 1 Schema Registry (on broker)
  • 1 KSQL Server
kafka-registry-connect-1
  • 1 Schema Registry (on broker)
  • 1 Distributed Kafka Connect Worker
kafka-registry-connect-ksql-1
  • 1 Schema Registry (on broker)
  • 1 Distributed Kafka Connect Worker
  • 1 KSQL Server
kafka-registry-connect-ksql-c3-1

Requires at least 16 GB RAM

  • 1 Schema Registry (on broker)
  • 1 Distrubuted Kafka Connect Worker
  • 1 KSQL Server
  • 1 Confluent Control Center (C3)
kafka-rest-1
  • 1 Kafka REST Proxy
kafka-sslCA-1
  • 1 Certificate Authority
kafka-tools-1
  • 1 Kafka Tools Server

Example running a different profile

export CP_PROFILE=kafka-registry-connect-1
vagrant up

New profiles can be added following the YAML format of the other profiles. If not specified, the default memory is 1536 (MB) and default CPU is 1. If a port value mapping is not specified, it'll create a direct port forward to the host for the given key. The vars key will be assigned to the Ansible host variables.

Start Vagrant

By default, this starts up a bento/centos-7 Vagrant box with PLAINTEXT security between all services.

vagrant up

To use a different OS, set the VAGRANT_BOX variable. For example, to start Ubuntu 16.04

VAGRANT_BOX=ubuntu/xenial64 vagrant up

To set a different security mode, find the sibling folders next to roles/ in the parent directory that contain an all.yml and set the name of that directory to the CP_SECURITY_MODE variable. For example, to use the sasl_ssl/all.yml playbook file

CP_SECURITY_MODE=sasl_ssl vagrant up

Machine access

To log into one of the machines:

vagrant ssh <machineName>

Provisioning

If you need to update the running cluster, you can re-run the provisioner (the step that installs software and configures services):

vagrant provision

By default, all services are provisioned according to their machine groups and targets defined in the all.yml file for the chosen security mode playbook.
The ANSIBLE_LIMIT variable can be used override this behavior to target specific machines. Its syntax matches the --limit flag or host pattern options of ansible and ansible-playbook commands.

Provision a group

Group names can be found in the Ansible playbook directory set by CP_SECURITY_MODE. For example, if we wanted to re-provision all the Kafka brokers:

ANSIBLE_LIMIT=broker vagrant provision

Provision some hosts

The machines are labelled in the profiles/ directory. If we wanted to re-provision two machines in the zookeeper group, zk0 and zk1:

ANSIBLE_LIMIT='zk[01]' vagrant provision

Cleanup

If there is a change in the CP_PROFILE variable that uses different hostnames for the machines, or once we are done with the Vagrant environment, we can cleanup the cluster by destroying all VMs:

vagrant destroy -f

References

About

Confluent Platform Ansible installation testing using Vagrant

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published