Skip to content

Latest commit

 

History

History
139 lines (88 loc) · 5.17 KB

README.md

File metadata and controls

139 lines (88 loc) · 5.17 KB
Raw

Hot Deals

Description

Software as a service scraping the Hot Deals forums and delivering newsletter emails on trending deals from the Forum.

Project Status: Active Build Status Uptime Robot Healthchecks.io Go Report Card Last commit on develop License

Buy Me A Coffee

Objective / Motivation

Red Flag Deals does aggregate deals on their front page, but the Hot Deals Forums are community driven and crowdsourced. This is where the purpose of my project comes into play, this project scrapes the Hot Deals Forums several times per day.

With this project, I saved myself the chore of checking the (messy) forum a few times a day while still being aware of the good deals posted by the community.

Screenshot

Website

Website

Newsletter

How It Works

The project is split into several directories (services):

  • Scraper

    Scrapes the forums periodically and depends on the database to persist the data.

  • Mailer

    Sends the daily newsletter email.

  • Backend

    REST API, takes care of sending data to the frontend and managing subscribing and unsubscribing.

  • Frontend

    Displays the deals nicely, and provides the subscription form.

  • Proxy

    Forwards requests to the right service.

  • HTML templates

    Email templates built with mjml for the design of the newsletter emails and confirmation emails.

Built with / technologies used

  • Go programming language
  • SendGrid
  • Docker & Docker Swarm
  • PostgreSQL
  • Redis
  • Drone CI
  • Nginx
  • Next.js
  • React.js
  • Material-UI

Features

  • Automated scraping of the Forum
  • Insert and update a database
  • Automated newsletter email
  • Algorithm to filter the junk
  • Website to browse the deals and subscribe to the emails
  • Confirmation email when a user subscribes

Getting started

Prerequisites

  • Go(lang) v1.14+
  • Docker-compose v1.25.5+
  • Docker v19.03+
  • Node.js v14+

Configuration

Variable Description
SENDGRID_API_KEY SendGrid API key to use the service
SENDGRID_TEMPLATE SendGrid requires you to upload an html template to use for dynamic emails, this is the ID of the template
SENDGRID_TEMPLATE_CONFIRM SendGrid template ID for confirmation emails
POSTGRES_NONROOT_DB Database for the appuser
POSTGRES_DB Root database
POSTGRES_USER Root username to initialize PostgreSQL
POSTGRES_PASSWORD Root password to initialize PostgreSQL
POSTGRES_NONROOT_PASSWORD The user password to use the database, we don't want to use the root user for these tasks
POSTGRES_NONROOT_USER The user username to use the database
DEV Set this variable to anything to run as dev (testing purposes)
SCRAPER_HC_UUID UUID for Healthchecks.io
MAILER_HC_UUID UUID for Healthchecks.io

Installing

Build individual Go services into a binary

go build -o main .

Usage

Bring the services up using docker-compose.

Running the tests

go test

Roadmap

Check out the open issues for ideas and features I have planned!

Support

You may open an issue for discussion.

Authors

@gordonpn

License

MIT License