Skip to content

Commit

Permalink
Cleanup work + Issue #75
Browse files Browse the repository at this point in the history
  • Loading branch information
MuruganChandrasekar committed Oct 28, 2021
1 parent 78721af commit f3a48d3
Show file tree
Hide file tree
Showing 15 changed files with 104 additions and 97 deletions.
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ In `docker-compose.yml`, the following ports on the docker host are connected t

- Nginx runs on 80/tcp and 443/tcp. (All connections to port 80 are redirected to 443 using SSL).

- MQTTS(Mosquitto) runs on
- 443/tcp for MQTT over Nginx proxy
- 8883/tcp for MQTT over TLS/SSL
- 8083/tcp for WebSockets over TLS/SSL
- 1883/tcp for MQTT over TCP protocol (not secure);(Disabled by default)

*The below ports are exposed only for the inter-container communication; These ports can't be accessed by host system.*

- Grafana runs on 3000/tcp.
Expand All @@ -129,8 +135,6 @@ In `docker-compose.yml`, the following ports on the docker host are connected t

- Postfix runs on 25/tcp.

Remember, if the server is running on a cloud platform like Microsoft Azure or AWS, one needs to check the firewall and confirm that the ports are open to the outside world.

## Data Files

When designing this collection of services, there were two choices to store the
Expand Down
21 changes: 16 additions & 5 deletions SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,23 @@
- [Notes](#notes)
- [Cloud-Provider-Specific Setup](#cloud-provider-specific-setup)
- [On Digital Ocean](#on-digital-ocean)
- [Create droplet](#create-droplet)
- [Configure droplet](#configure-droplet)
- [Create droplet](#create-droplet)
- [Configure droplet](#configure-droplet)
- [List of required ports to be opened in the firewall](#list-of-required-ports-to-be-opened-in-the-firewall)
- [After server is set up](#after-server-is-set-up)
- [Create and edit the .env file](#Create-and-edit-the-.env-file)
- [Create and edit the .env file](#create-and-edit-the-env-file)
- [Set up the Node-RED and InfluxDB API logins](#set-up-the-node-red-and-influxdb-api-logins)
- [Migrating `htpasswd` from Apache to Nginx (if required)](#migrating-htpasswd-from-apache-to-nginx-if-required)
- [Creating new `htpasswd` files](#creating-new-htpasswd-files)
- [MQTT User Credentials setup](#mqtt-user-credentials-setup)
- [Set up the `MQTTs` User Credentials](#set-up-the-mqtts-user-credentials)
- [Start the server](#start-the-server)
- [Restart servers in the background](#restart-servers-in-the-background)
- [Initial testing](#initial-testing)
- [Set up first data source](#set-up-first-data-source)
- [Test Node-RED](#test-node-red)
- [Creating an InfluxDB database](#creating-an-influxdb-database)
- [Test Postfix Mail setup](#Test-Postfix-Mail-setup)
- [Test MQTT Channels](#Test-MQTT-Channels)
- [Test MQTT Channels](#test-mqtt-channels)


<!-- /TOC -->
Expand Down Expand Up @@ -156,6 +157,16 @@ As an initial step, a cloud provider is required and Docker and Docker-Compose m
sudo chmod g+w .
```
## List of required ports to be opened in the firewall
Remember, if the server is running on a cloud platform like Microsoft Azure or AWS, one needs to check the firewall and confirm that the below ports are open to the outside world.
- 80/tcp
- 443/tcp
- 8883/tcp
- 8083/tcp
- 1883/tcp (if needed; not secure)
## After server is set up
The following instructions are essentially independent of the cloud provider and the underlying distribution. But this was only tested on Ubuntu and (in 2019) on CentOS.
Expand Down
7 changes: 3 additions & 4 deletions cron-backup/grafana_backup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ if [ ! -d $grafana_src ]; then
echo "" >> /tmp/grafana.txt
echo "STATUS: Grafana backup failed" >> /tmp/grafana.txt
echo "" >> /tmp/grafana.txt
echo "The source backup directory: grafana_src is not available" >> /tmp/grafana.txt
echo "The source backup directory: grafana_src is not available" >> /tmp/grafana.txt
< /tmp/grafana.txt mail -s "${SOURCE_NAME}: Grafana Data Backup" "${CRON_BACKUP_MAIL}"
exit
exit
else
tar cvzf /var/lib/backup/grafana/"${SOURCE_NAME}"_grafana_data_backup_"${DATE1}".tgz ${grafana_src}/
tar cvzf /var/lib/backup/grafana/"${SOURCE_NAME}"_grafana_data_backup_"${DATE1}".tgz ${grafana_src}/
fi

# Moving the backup to S3 bucket
Expand All @@ -37,7 +37,6 @@ then
s3cmd ls --no-mime-magic s3://"${S3_BUCKET_GRAFANA}"/ --human-readable | grep -i "${SOURCE_NAME}"_grafana_data | cut -d' ' -f3- | tac | head -10 | sed "s,s3:\/\/${S3_BUCKET_GRAFANA}\/,,g" &>> /tmp/grafana.txt
echo "" >> /tmp/grafana.txt
echo "************** END **************************" >> /tmp/grafana.txt

else
echo "DATE:" "$DATE" > /tmp/grafana.txt
echo "" >> /tmp/grafana.txt
Expand Down
7 changes: 3 additions & 4 deletions cron-backup/mqtts_backup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ if [ ! -d $mqtts_src ]; then
echo "" >> /tmp/mqtts.txt
echo "STATUS: MQTTs backup failed." >> /tmp/mqtts.txt
echo "" >> /tmp/mqtts.txt
echo "The source backup directory: mqtts_src is not available" >> /tmp/mqtts.txt
echo "The source backup directory: mqtts_src is not available" >> /tmp/mqtts.txt
< /tmp/mqtts.txt mail -s "${SOURCE_NAME}: MQTTs Data Backup" "${CRON_BACKUP_MAIL}"
exit
exit
else
tar cvzf /var/lib/backup/mqtts/"${SOURCE_NAME}"_mqtts_data_backup_"${DATE1}".tgz ${mqtts_src}/
tar cvzf /var/lib/backup/mqtts/"${SOURCE_NAME}"_mqtts_data_backup_"${DATE1}".tgz ${mqtts_src}/
fi

# Moving the backup to S3 bucket
Expand All @@ -37,7 +37,6 @@ then
s3cmd ls --no-mime-magic s3://"${S3_BUCKET_MQTTS}"/ --human-readable | grep -i "${SOURCE_NAME}"_mqtts_data | cut -d' ' -f3- | tac | head -10 | sed "s,s3:\/\/${S3_BUCKET_MQTTS}\/,,g" &>> /tmp/mqtts.txt
echo "" >> /tmp/mqtts.txt
echo "************** END **************************" >> /tmp/mqtts.txt

else
echo "DATE:" "$DATE" > /tmp/mqtts.txt
echo "" >> /tmp/mqtts.txt
Expand Down
7 changes: 3 additions & 4 deletions cron-backup/nginx_backup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ if [ ! -d $nginx_src ]; then
echo "" >> /tmp/nginx.txt
echo "STATUS: Nginx backup failed." >> /tmp/nginx.txt
echo "" >> /tmp/nginx.txt
echo "The source backup directory: nginx_src is not available" >> /tmp/nginx.txt
echo "The source backup directory: nginx_src is not available" >> /tmp/nginx.txt
< /tmp/nginx.txt mail -s "${SOURCE_NAME}: Nginx Data Backup" "${CRON_BACKUP_MAIL}"
exit
exit
else
tar cvzf /var/lib/backup/nginx/"${SOURCE_NAME}"_nginx_data_backup_"${DATE1}".tgz ${nginx_src}/
tar cvzf /var/lib/backup/nginx/"${SOURCE_NAME}"_nginx_data_backup_"${DATE1}".tgz ${nginx_src}/
fi

# Moving the backup to S3 bucket
Expand All @@ -37,7 +37,6 @@ then
s3cmd ls --no-mime-magic s3://"${S3_BUCKET_NGINX}"/ --human-readable | grep -i "${SOURCE_NAME}"_nginx_data | cut -d' ' -f3- | tac | head -10 | sed "s,s3:\/\/${S3_BUCKET_NGINX}\/,,g" &>> /tmp/nginx.txt
echo "" >> /tmp/nginx.txt
echo "************** END **************************" >> /tmp/nginx.txt

else
echo "DATE:" "$DATE" > /tmp/nginx.txt
echo "" >> /tmp/nginx.txt
Expand Down
7 changes: 3 additions & 4 deletions cron-backup/nodered_backup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ if [ ! -d $nodered_src ]; then
echo "" >> /tmp/nodered.txt
echo "STATUS: Nodered backup failed." >> /tmp/nodered.txt
echo "" >> /tmp/nodered.txt
echo "The source backup directory: nodered_src is not available" >> /tmp/nodered.txt
echo "The source backup directory: nodered_src is not available" >> /tmp/nodered.txt
< /tmp/nodered.txt mail -s "${SOURCE_NAME}: Nodered Data Backup" "${CRON_BACKUP_MAIL}"
exit
exit
else
tar cvzf /var/lib/backup/nodered/"${SOURCE_NAME}"_nodered_data_backup_"${DATE1}".tgz ${nodered_src}/
tar cvzf /var/lib/backup/nodered/"${SOURCE_NAME}"_nodered_data_backup_"${DATE1}".tgz ${nodered_src}/
fi

# Moving the backup to S3 bucket
Expand All @@ -37,7 +37,6 @@ then
s3cmd ls --no-mime-magic s3://"${S3_BUCKET_NODERED}"/ --human-readable | grep -i "${SOURCE_NAME}"_nodered_data | cut -d' ' -f3- | tac | head -10 | sed "s,s3:\/\/${S3_BUCKET_NODERED}\/,,g" &>> /tmp/nodered.txt
echo "" >> /tmp/nodered.txt
echo "************** END **************************" >> /tmp/nodered.txt

else
echo "DATE:" "$DATE" > /tmp/nodered.txt
echo "" >> /tmp/nodered.txt
Expand Down
10 changes: 5 additions & 5 deletions htpasswd_migration.sh
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ case $var in
echo "It is done. Thanks!"
echo ""
echo ""
exit
;;
exit
;;

"creating htpasswd for each controlled service automatically")
while read -r line
Expand All @@ -69,9 +69,9 @@ case $var in
echo "create:" "${file}_htpasswd"
true > "${file}_htpasswd"
for k in $(echo "$line" | tr ' |,' '\n')
do
do
sed -n "/$k/p" "$htpasswd" >> "${file}_htpasswd"
done
done
done < "$htgroup"
echo ""
echo ""
Expand All @@ -82,7 +82,7 @@ case $var in
exit
;;

*)
*)
echo "Please enter correct number"
;;
esac
Expand Down
1 change: 0 additions & 1 deletion influxdb/backup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ then
s3cmd ls --no-mime-magic s3://"${S3_BUCKET_INFLUXDB}"/ --human-readable | grep -i "${SOURCE_NAME}"_influxdb_data | cut -d' ' -f3- | tac | head -10 | sed "s,s3:\/\/${S3_BUCKET_INFLUXDB}\/,,g" &>> /tmp/influxbackup.txt
echo "" >> /tmp/influxbackup.txt
echo "********************** END ********************* " >> /tmp/influxbackup.txt

else
echo "DATE:" "$DATE" > /tmp/influxbackup.txt
echo "" >> /tmp/influxbackup.txt
Expand Down
1 change: 0 additions & 1 deletion influxdb/influx.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
#!/bin/sh
#/etc/init.d/influxdb start
/usr/bin/influxd -config /etc/influxdb/influxdb.conf
29 changes: 14 additions & 15 deletions nginx/proxy-grafana.conf
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,19 @@
rewrite ^/index.html$ https://@{FQDN}/grafana/;
rewrite ^/grafana$ https://@{FQDN}/grafana/;
location /grafana/ {
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options nosniff always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline' blob:; img-src 'self' data: *.global.ssl.fastly.net" always;
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;

proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://grafana:3000/;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options nosniff always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline' blob:; img-src 'self' data: *.global.ssl.fastly.net" always;
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass http://grafana:3000/;

}
14 changes: 7 additions & 7 deletions nginx/proxy-influxdb.conf
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Content-Security-Policy "default-src 'self'" always;
add_header 'Referrer-Policy' 'origin';
add_header 'Referrer-Policy' 'origin';
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://influxdb:8086/;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://influxdb:8086/;
auth_basic "InfluxDB queries";
auth_basic_user_file /etc/nginx/authdata/influxdb/.htpasswd;
}
13 changes: 6 additions & 7 deletions nginx/proxy-mqtts.conf
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
rewrite ^/mqtts$ https://@{FQDN}/mqtts/;
location /mqtts/ {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass https://mqtts:8083/;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_pass https://mqtts:8083/;
}
60 changes: 30 additions & 30 deletions nginx/proxy-nodered.conf
Original file line number Diff line number Diff line change
@@ -1,38 +1,38 @@
rewrite ^/node-red$ https://@{FQDN}/node-red/;
location /node-red/ {
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:" always;
add_header 'Referrer-Policy' 'origin';
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://node-red:1880/;
auth_basic "Node-RED";
auth_basic_user_file /etc/nginx/authdata/nodered/.htpasswd;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:" always;
add_header 'Referrer-Policy' 'origin';
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://node-red:1880/;
auth_basic "Node-RED";
auth_basic_user_file /etc/nginx/authdata/nodered/.htpasswd;
}

# Enabling HTTP Endpoint using node-red

location /post/ {
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:" always;
add_header 'Referrer-Policy' 'origin';
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://node-red:1880/post;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Xss-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Content-Security-Policy "default-src 'self' https:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:" always;
add_header 'Referrer-Policy' 'origin';
add_header Feature-Policy: "accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'" always;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_pass http://node-red:1880/post;
}
14 changes: 7 additions & 7 deletions nginx/setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@ fi

# run cerbot to set up Nginx
if [ "$CERTBOT_TEST" != "test" ]; then
certbot --agree-tos --email "${CERTBOT_EMAIL}" --non-interactive --domains "$CERTBOT_DOMAINS" --nginx --agree-tos --rsa-key-size 4096 --redirect || exit 4
certbot --agree-tos --email "${CERTBOT_EMAIL}" --non-interactive --domains "$CERTBOT_DOMAINS" --nginx --agree-tos --rsa-key-size 4096 --redirect || exit 4

# certbot actually launched Nginx. The simple hack is to stop it; then launch
# it again after we've edited the config files.
/usr/sbin/nginx -s stop && echo "stopped successfully"
# certbot actually launched Nginx. The simple hack is to stop it; then launch
# it again after we've edited the config files.
/usr/sbin/nginx -s stop && echo "stopped successfully"
fi

# now, add the fields to the virtual host section for https.
Expand All @@ -66,7 +66,7 @@ if [ "$1" != "proxy-*.conf" ] ; then
sed -e '/listen 443 ssl;/r/tmp/proxyspecs.conf' /etc/nginx/sites-available/default > /tmp/000-default-le-ssl-local.conf || exit 6
mv /tmp/000-default-le-ssl-local.conf /etc/nginx/sites-available || exit 7
echo "enable the modified site, and disable the ssl defaults"
rm -rf /etc/nginx/sites-enabled/default || echo exit 8
rm -rf /etc/nginx/sites-enabled/000-default-le-ssl-local.conf || exit 9
ln -s /etc/nginx/sites-available/000-default-le-ssl-local.conf /etc/nginx/sites-enabled/000-default-le-ssl-local.conf || exit 10
rm -rf /etc/nginx/sites-enabled/default || echo exit 8
rm -rf /etc/nginx/sites-enabled/000-default-le-ssl-local.conf || exit 9
ln -s /etc/nginx/sites-available/000-default-le-ssl-local.conf /etc/nginx/sites-enabled/000-default-le-ssl-local.conf || exit 10
fi
Loading

0 comments on commit f3a48d3

Please sign in to comment.