Skip to content

Latest commit

 

History

History
executable file
·
154 lines (116 loc) · 6.64 KB

README.md

File metadata and controls

executable file
·
154 lines (116 loc) · 6.64 KB
Raw

Topcoder member account processor

Dependencies

Local Kafka setup

  • http://kafka.apache.org/quickstart contains details to setup and manage Kafka server, below provides details to setup Kafka server in Mac, Windows will use bat commands in bin/windows instead
  • download kafka at https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/kafka_2.11-1.1.0.tgz
  • extract out the downloaded tgz file
  • go to the extracted directory kafka_2.11-0.11.0.1
  • start ZooKeeper server: bin/zookeeper-server-start.sh config/zookeeper.properties
  • use another terminal, go to same directory, start the Kafka server: bin/kafka-server-start.sh config/server.properties
  • note that the zookeeper server is at localhost:2181, and Kafka server is at localhost:9092
  • use another terminal, go to same directory, create some topics:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic event.user.created
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic event.user.updated
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic member.action.profile.create
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic member.action.profile.update
  • verify that the topics are created:
bin/kafka-topics.sh --list --zookeeper localhost:2181

it should list out the created topics

  • run the producer and then write some message into the console to send to the topic event.user.created:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic event.user.created
  • In the console, write some message, one message per line: E.g.
{ "topic": "event.user.created", "originator": "some-originator", "timestamp": "2018-02-16T00:00:00", "mime-type": "application/json", "payload": { "handle": "test", "email": "[email protected]", "firstName": "first-name", "lastName": "last-name", "profiles": [{ "userId": "123" }], "status": "ACTIVE", "country": { "code": "US", "name": "USA" }, "roles": [{ "roleName": "copilot" }], "modifiedBy": "11111", "modifiedAt": "2018-02-18T11:11:22", "createdBy": "11111", "createdAt": "2018-02-17T11:11:11" } }
  • optionally, use another terminal, go to same directory, start a consumer to view the messages:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic event.user.created --from-beginning

Configuration

Configuration for the application is at config/default.js and config/production.js. The following parameters can be set in config files or in env variables:

  • LOG_LEVEL: the log level
  • PORT: Port for Health check dropin to run
  • KAFKA_URL: Kafka server URL
  • KAFKA_GROUP_ID: the Kafka group id
  • KAFKA_CLIENT_CERT: Kafka connection certificate, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to certificate file or certificate content
  • KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to private key file or private key content
  • AMAZON_AWS_REGION: the Amazon AWS region to access DynamoDB
  • AMAZON_AWS_DYNAMODB_MEMBER_PROFILE_TABLE: the DynamoDB member profile table name
  • COPILOT_ROLE_NAME: copilot role name used to check whether user is copilot
  • USER_CREATE_TOPIC: Kafka topic related to user creation
  • USER_UPDATE_TOPIC: Kafka topic related to user update
  • USER_CREATE_OUTPUT_TOPIC: Kafka topic to output member profile creation message
  • USER_UPDATE_OUTPUT_TOPIC: Kafka topic to output member profile update message
  • OUTPUT_MESSAGE_ORIGINATOR: Kafka output message originator

Due to using dotenv library, env variables may also be set via .env file in app root folder.

AWS Setup

  1. Download your AWS Credentials from AWS Console. Refer AWS Documentation

  2. Depending on your Operating System, create AWS credentials file in the path listed below

Linux, Unix, and macOS users: ~/.aws/credentials

Windows users: C:\Users\{USER_NAME}\.aws\credentials
  1. credentials file should look like below
[default]
aws_access_key_id = SOME_ACCESS_KEY_ID
aws_secret_access_key = SOME_SECRET_ACCESS_KEY

Local deployment

  1. From the project root directory, run the following command to install the dependencies
npm i
  1. To run linters if required
npm run lint

npm run lint:fix # To fix possible lint errors
  1. To delete DynamoDB table if needed
npm run delete-table
  1. To create DynamoDB table if needed
npm run create-table

A table must be created before the app starts.

  1. Properly configure the app via config file, env variables or .env file in app root folder

  2. Start the processor and health check dropin

npm start

Notes