Skip to content

Commit

Permalink
Merge pull request #611 from CS-SI/pr/docs
Browse files Browse the repository at this point in the history
Bugfixes and Doc updates
  • Loading branch information
oscarpicas authored Feb 1, 2023
2 parents fa8745c + 52382de commit 7b6a7c3
Show file tree
Hide file tree
Showing 25 changed files with 3,364 additions and 2,616 deletions.
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -79,13 +79,13 @@ allcover: logclean ground getdevdeps mod sdk generate lib cli minimock err vet s
version:
@printf "%b" "$(VERSION)-$$(git rev-parse --abbrev-ref HEAD | tr \"/\" \"_\")";

release: logclean ground getdevdeps mod releasetags sdk generate lib cli test minimock err vet semgrep style metalint releasearchive
release: logclean ground getdevdeps mod releasetags tunnel sdk generate lib cli test minimock err vet semgrep style metalint releasearchive
@printf "%b" "$(OK_COLOR)$(OK_STRING) Build for release, branch $$(git rev-parse --abbrev-ref HEAD) SUCCESSFUL $(NO_COLOR)\n";
@git ls-tree --full-tree --name-only -r HEAD | grep \.go | xargs $(MD5) 2>/dev/null > sums.log || true
@git ls-tree --full-tree --name-only -r HEAD | grep \.sh | xargs $(MD5) 2>/dev/null >> sums.log || true
@git ls-tree --full-tree --name-only -r HEAD | grep \.yml | xargs $(MD5) 2>/dev/null >> sums.log || true

releaserc: logclean ground getdevdeps mod releasetags sdk generate lib cli minimock err vet style metalint releasearchive
releaserc: logclean ground getdevdeps mod releasetags tunnel sdk generate lib cli minimock err vet style metalint releasearchive
@printf "%b" "$(OK_COLOR)$(OK_STRING) Build for rc, branch $$(git rev-parse --abbrev-ref HEAD) SUCCESSFUL $(NO_COLOR)\n";
@git ls-tree --full-tree --name-only -r HEAD | grep \.go | xargs $(MD5) 2>/dev/null > sums.log || true
@git ls-tree --full-tree --name-only -r HEAD | grep \.sh | xargs $(MD5) 2>/dev/null >> sums.log || true
Expand Down
45 changes: 28 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,15 @@ SafeScale is an Infrastructure and Platform as Code tool.
- [Description](#description)
- [SafeScale Infra](#safescale-infra)
- [SafeScale Platform](#safescale-platform)
- [SafeScale Security](#safescale-security)
- [Available features](#available-features)
- [Contributing](#contributing)
- [License](#license)

## Description
SafeScale offers an APIs and a CLI tools to deploy versatile computing clusters that span multiple Clouds. These APIs and CLIs are divided in 3 service layers:
SafeScale offers an APIs and a CLI tools to deploy versatile computing clusters that span multiple Clouds. These APIs and CLIs are divided in 2 service layers:

- SafeScale Infra to manage Cloud infrastructure (IaaS - Infrastructure as a Service)
- SafeScale Platform to manage Cloud computing platforms (PaaS - Platform as a Service)
- SafeScale Security to secure user environments

![SafeScale](doc/img/SafeScale.png "SafeScale")

Expand Down Expand Up @@ -70,15 +68,39 @@ For example the following command creates a Kubernetes cluster named `k8s-cluste
$ safescale cluster create --flavor k8s --complexity Normal k8s-cluster
```

Supplemental software and/or configurations can be installed in 2 ways on SafeScale Hosts or Clusters:
Supplemental software and/or configurations can be installed in 3 ways on SafeScale Hosts or Clusters:
- using ssh command (the old and manual way):
```
$ safescale ssh run -c "apt install nginx" my-host
```
- using "SafeScale `Feature`", that can be seen as the "ansible" for SafeScale:

```
$ safescale cluster feature add mycluster keycloak
$ safescale cluster feature add mycluster ntpclient
```
- and using ansible, which is the PREFERRED method to install your software in a SafeScale cluster:
installing a simple script:
```
$ safescale cluster ansible playbook my-cluster my-ansible-script.yml
```

where my-ansible-script.yml is something like:
```yml
---
- hosts: nodes
tasks:
- name: Install golang
become: yes
apt:
pkg:
- golang
- bison

```

or a more complex one (put all your files in a .zip)
```
$ safescale cluster ansible playbook my-cluster my-zipped-scripts.zip
```

A "SafeScale `Feature`" is a file in YAML format that describes the operations to check/add/remove software and/or configuration on a target (Host or Cluster).
Expand All @@ -91,17 +113,6 @@ A `Feature` can describe operations using different methods:

Additionally, a `Feature` is able to apply:
- reverse proxy rules
- Security Group rules

### SafeScale Security

SafeScale Security is a Web API and a Web Portal to create on-demand security gateways to protect Web services along 5 axes: Encryption, Authentication, Authorization, Auditability and Intrusion detection.
SafeScale Security relies on Kong, an open source generic proxy to be put in between user and service. Kong intercepts user requests and service responses and executes plugins to empower any API. To build a SafeScale Security gateway 3 plugins are used:
- Dynamic SSL plugin to encrypt traffic between the user and the service protected
- Open ID plugin to connect the Identity and Access Management server, KeyCloak
- UDP Log plugin to connect the Log management system, Logstash
The design of a SafeScale Security gateway can be depicted as below:
![SafeScale Security](doc/img/SafeScale_Security.png "SafeScale Security")

## Available features
SafeScale is currently under active development and does not yet offer all the abilities planned. However, we are already publishing it with the following ones:
Expand Down Expand Up @@ -143,7 +154,7 @@ As much as possible, try following these guides:
- [Go style guide](https://github.com/golang/go/wiki/CodeReviewComments)
- [Effective Go](https://golang.org/doc/effective_go)

For bugs and feature requests, [please create an issue](../../issues/new).
For bugs and feature requests, [please create an issue](https://github.com/CS-SI/SafeScale/issues/new/choose).

## Build
[See Build file](doc/build/BUILDING.md)
Expand Down
104 changes: 104 additions & 0 deletions build/Dockerfile2
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
FROM ubuntu:focal as base
LABEL maintainer="CS SI"
ARG http_proxy=""
ARG https_proxy=""
ARG LC_ALL=C.UTF-8
ARG LANG=C.UTF-8
ENV DEBIAN_FRONTEND noninteractive
ENV BUILD_ENV docker
ENV BRANCH_NAME $BRANCH_NAME
ENV GOVERSION $GOVERSION
ENV PROTOVERSION $PROTOVERSION
ENV COMMITSHA $COMMITSHA
ENV GOOSX $GOOSX
ENV GOARCHX $GOARCHX

RUN apt-get update -y \
&& apt-get install -y --allow-unauthenticated --no-install-recommends \
wget unzip apt-utils

WORKDIR /tmp

# ----------------------
# Install GO $GOVERSION
# ----------------------
RUN wget --no-check-certificate https://dl.google.com/go/go$GOVERSION.linux-amd64.tar.gz \
&& tar -C /usr/local -xzf go$GOVERSION.linux-amd64.tar.gz \
&& rm /tmp/go$GOVERSION.linux-amd64.tar.gz
ENV PATH $PATH:/usr/local/go/bin:/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# ----------------------
# Install Protoc $PROTOVERSION
# ----------------------
RUN wget --no-check-certificate https://github.com/google/protobuf/releases/download/v$PROTOVERSION/protoc-$PROTOVERSION-linux-x86_64.zip \
&& unzip -d /usr/local/protoc protoc-$PROTOVERSION-linux-x86_64.zip \
&& ln -s /usr/local/protoc/bin/protoc /usr/local/bin \
&& rm /tmp/protoc-$PROTOVERSION-linux-x86_64.zip

FROM base AS builder

WORKDIR /tmp

# -----------------
# Install Standard packages
# -----------------
RUN apt-get install -y --allow-unauthenticated --no-install-recommends \
locales \
sudo \
build-essential \
make \
curl \
git \
jq \
python3 \
python3-pip \
&& apt-get autoclean -y \
&& apt-get autoremove -y \
&& rm -rf /var/lib/apt/lists/*

# Set the locale
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
RUN python3 -c "print('testing UTF8: 👌')"

ENV SHELL /bin/bash
ENV GOPATH /go
COPY build-safescale2.sh /opt/build-safescale.sh

CMD sleep 5

COPY marker /dev/null

RUN cd /opt && COMMITSHA=$COMMITSHA GOOSX=$GOOSX GOARCHX=$GOARCHX ./build-safescale.sh

# --

FROM golang:$GOVERSION-alpine
LABEL maintainer="CS SI"
ARG http_proxy=""
ARG https_proxy=""
ARG LC_ALL=C.UTF-8
ARG LANG=C.UTF-8
ENV BUILD_ENV docker

RUN apk update && \
apk add --no-cache \
nano \
curl \
wget \
openssl \
ca-certificates \
iproute2 \
iperf

RUN ln -s /usr/lib/tc /lib/tc
RUN mkdir /exported-$GOOSX-$GOARCHX

RUN apk add --no-cache bash

COPY --from=builder /exported-$GOOSX-$GOARCHX/safescaled /exported-$GOOSX-$GOARCHX/safescaled
COPY --from=builder /exported-$GOOSX-$GOARCHX/safescale /exported-$GOOSX-$GOARCHX/safescale
COPY --from=builder /exported-$GOOSX-$GOARCHX/go.mod /exported-$GOOSX-$GOARCHX/go.mod
COPY --from=builder /exported-$GOOSX-$GOARCHX/go.sum /exported-$GOOSX-$GOARCHX/go.sum
2 changes: 1 addition & 1 deletion build/build-safescale.sh
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ make generate
sleep 4

echo "Make All"
make all
make release
[ $? -ne 0 ] && echo "Build failure" && exit 1

echo "Install"
Expand Down
97 changes: 97 additions & 0 deletions build/build-safescale2.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
#! /bin/bash

echo "Checks..."
if [[ ! -v BUILD_ENV ]]; then
echo "BUILD_ENV is not set, this script is intended to run inside a docker container"
[[ $SHLVL -gt 2 ]] && return 1 || exit 1
fi

# ----------------------
# Create working directory
# ----------------------
echo "Create working directory"
export WRKDIR=/opt
mkdir -p ${WRKDIR}
cd ${WRKDIR}
rm -rf SafeScale

if [ -z "$COMMITSHA" ]
then
# ----------------------
# Get source code
# ----------------------
echo "Get source code"
BRANCH_NAME=${BRANCH_NAME:="develop"}
GIT_REPO_URL=${GIT_REPO_URL:="https://github.com/CS-SI/SafeScale.git"}
echo "Cloning branch '${BRANCH_NAME}' from repo '${GIT_REPO_URL}'"

git clone ${GIT_REPO_URL} -b ${BRANCH_NAME} --depth=1

cd SafeScale
sed -i "s#\(.*\)develop#\1${BRANCH_NAME}#" common.mk
else
# ----------------------
# Get source code
# ----------------------
echo "Get source code, commit $COMMITSHA"
GIT_REPO_URL=${GIT_REPO_URL:="https://github.com/CS-SI/SafeScale.git"}

git clone ${GIT_REPO_URL}
cd SafeScale

git reset --hard $COMMITSHA
sed -i "s#\(.*\)develop#\1${BRANCH_NAME}#" common.mk
fi

# ----------------------
# Compile
# ----------------------

echo "deps"
make getdevdeps

sleep 4

echo "mod"
make mod

sleep 4

make sdk

sleep 4

make force_sdk_python

sleep 4

make force_sdk_js

sleep 4

make generate

sleep 4

export GOOS=$GOOSX
export GOARCH=$GOARCHX

echo "Make All"
make release
[ $? -ne 0 ] && echo "Build failure" && exit 1

echo "Install"
make install
[ $? -ne 0 ] && echo "Install failure" && exit 1

echo "Export"
export CIBIN=/exported-$GOOS-$GOARCH
mkdir -p /exported-$GOOS-$GOARCH

CIBIN=/exported-$GOOS-$GOARCH make installci
[ $? -ne 0 ] && echo "Export failure" && exit 1

cp ${WRKDIR}/SafeScale/go.mod /exported-$GOOS-$GOARCH
cp ${WRKDIR}/SafeScale/go.sum /exported-$GOOS-$GOARCH

exit 0
59 changes: 59 additions & 0 deletions build/create-docker2.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env bash

if [ "$(uname -s)" = "Darwin" ]; then
WRKDIR=$(readlink -n $(dirname "$0"))
[ -z "$WRKDIR" ] && WRKDIR=$(dirname "$0")
else
WRKDIR=$(readlink -f $(dirname "$0"))
fi

if [ ! -z "$1" ]; then
if [[ $1 == "-f" ]]; then
date >marker
fi
fi

if [ ! -f ./marker ]; then
curl https://api.github.com/repos/CS-SI/SafeScale/commits/$(git rev-parse --abbrev-ref HEAD) 2>&1 | grep '"date"' | tail -n 1 >./marker
else
curl https://api.github.com/repos/CS-SI/SafeScale/commits/$(git rev-parse --abbrev-ref HEAD) 2>&1 | grep '"date"' | tail -n 1 >./newMarker
diff ./marker ./newMarker 1>/dev/null && rm ./newMarker && echo "Nothing to do !, if you want to force a docker build launch with the -f flag" && exit 0
fi

stamp=$(date +"%s")

[ -z "$BRANCH_NAME" ] && BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
[ -z "$GOVERSION" ] && GOVERSION=1.18.4
[ -z "$PROTOVERSION" ] && PROTOVERSION=3.17.3

BRANCH_NAME=$BRANCH_NAME PROTOVERSION=$PROTOVERSION GOVERSION=$GOVERSION COMMITSHA=$COMMITSHA envsubst <Dockerfile2 >Dockerfile.$stamp
if [ -z "$COMMITSHA" ]
then
sed -i '/ENV COMMITSHA/d' Dockerfile.$stamp
fi

echo docker build --rm --network host --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg BRANCH_NAME=$BRANCH_NAME --build-arg GOVERSION=$GOVERSION -f ${WRKDIR}/Dockerfile.$stamp -t "safescale:${BRANCH_NAME/\//_}" $WRKDIR
docker build --rm --network host --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg BRANCH_NAME=$BRANCH_NAME --build-arg GOVERSION=$GOVERSION -f ${WRKDIR}/Dockerfile.$stamp -t "safescale:${BRANCH_NAME/\//_}" $WRKDIR
[ $? -ne 0 ] && echo "Docker build failed !!" && {
rm -f ./marker
rm -f ./Dockerfile.$stamp
exit 1
}

echo "Docker build OK"

docker create -ti --name dummy "safescale:${BRANCH_NAME/\//_}" bash
[ $? -ne 0 ] && echo "Failure extracting binaries 1/3" && exit 1
docker cp dummy:/exported-$GOOSX-$GOARCHX .
[ $? -ne 0 ] && echo "Failure extracting binaries 2/3" && exit 1
docker rm -f dummy
[ $? -ne 0 ] && echo "Failure extracting binaries 3/3" && exit 1

echo "Binaries extracted successfully"
if [ -f ./newMarker ]; then
mv ./newMarker ./marker
fi

rm -f ./Dockerfile.$stamp

exit 0
3 changes: 0 additions & 3 deletions cli/safescaled/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ import (
_ "github.com/CS-SI/SafeScale/v22/lib/backend"
"github.com/CS-SI/SafeScale/v22/lib/backend/iaas"
"github.com/CS-SI/SafeScale/v22/lib/backend/listeners"
"github.com/CS-SI/SafeScale/v22/lib/backend/resources/operations"
"github.com/CS-SI/SafeScale/v22/lib/protocol"
appwide "github.com/CS-SI/SafeScale/v22/lib/utils/app"
"github.com/CS-SI/SafeScale/v22/lib/utils/debug"
Expand Down Expand Up @@ -141,8 +140,6 @@ func work(c *cli.Context) {
// - /debug/fgprof
expose()

operations.StartFeatureFileWatcher()

version := Version + ", build " + Revision + " (" + BuildDate + ")"
if //goland:noinspection GoBoolExpressions
len(Tags) > 1 { // nolint
Expand Down
2 changes: 1 addition & 1 deletion common.mk
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
VERSION=22.11.2
VERSION=22.11.5
export VERSION

ifeq ($(MAKE_LEVEL),)
Expand Down
Loading

0 comments on commit 7b6a7c3

Please sign in to comment.