Skip to content

Latest commit

 

History

History
326 lines (244 loc) · 11 KB

README.md

File metadata and controls

326 lines (244 loc) · 11 KB
Raw

swift-browser-ui

Python Unit Tests Javascript ESLint check Python style check Chrome UI check Firefox UI check Coverage Status Documentation Status

Description

A web frontend for browsing and downloading objects saved in SWIFT compliant object storage, supporting SSO with SAML2 federated authentication.

Project documentation is hosted on readthedocs: https://swift-browser-ui.rtfd.io

Information on the additional APIs for container sharing, access requests, and better upload and download functionality are in their separate files.

Requirements

Python 3.12+ required.

  • The dependencies mentioned in requirements.txt.
  • A suitable storage backend supporting usage via OpenStack Object Storage API. (e.g. Ceph RGW, OpenStack Swift)
  • PostgreSQL
  • Redis

Usage – UI

At the current state the program configs can be specified either via environment variables or command line arguments. These usage directions assume envvars to be used.

Creating environment variable file for stand-alone use (no TLS proxy):

# Replace the example URLs with correct ones
echo '
export BROWSER_START_AUTH_ENDPOINT_URL="https://keystone-url.example.com:5001/v3"
export BROWSER_START_PORT="8080"'\
|tee -a envs.sh && chmod u+x envs.sh

Getting started:

git clone [email protected]:CSCfi/swift-browser-ui.git
cd swift-browser-ui
cd swift_browser_ui_frontend
npm install -g pnpm@8
pnpm install
pnpm run build
cd ..
pip install -r requirements.txt
pip install .[]

After install there should be swift-browser-ui command available:

➜ swift-browser-ui --help
Usage: swift-browser-ui [OPTIONS] COMMAND [ARGS]...

  Command line interface for managing swift-browser-ui.

Options:
  --version       Show the version and exit.
  -v, --verbose   Increase program verbosity.
  -D, --debug     Enable debug level logging.
  --logfile TEXT  Write program logs to a file.
  --help          Show this message and exit.

Commands:
  start    Start the browser backend and server

In order to start the server use swift-browser-ui start.

Additional options can be found with

swift-browser-ui --help
swift-browser-ui start --help

The current frontend can be found at: 127.0.0.1:8080.

Development

swift-browser-ui is composed of 4 components: request, sharing, ui, and upload. All of them must be run to have access to all features. They depend on a Redis instance for session cache, Postgres database for the sharing and request functionality, and the object storage backend. You will also need docker with Buildkit to build the keystone-swift docker image.

To start all required services, you can use the docker-compose files from https://github.com/CSCfi/swift-ui-deployment, or the provided Procfile, as shown bellow.

Please, read and adhere to the CONTRIBUTING guidelines for submitting changes.

Getting started:

git clone -b devel [email protected]:CSCfi/swift-browser-ui.git
cd swift-browser-ui

Install frontend dependencies, and build (without encryption or OIDC enabled).

pnpm --prefix swift_browser_ui_frontend install
pnpm --prefix swift_browser_ui_frontend run build

Install python dependencies, optionally in a virtual environment.

python3 -m venv venv --prompt swiftui  # Optional step, creates python virtual environment
source venv/bin/activate  # activates virtual environment
pip install -Ue .[docs,test,dev]
pre-commit install

Set up the environment variables

cp .github/config/.env.test .env  # Make any changes you need to the file

Open another terminal, and build the keystone-swift image

git clone [email protected]:CSCfi/docker-keystone-swift.git
cd docker-keystone-swift
docker buildx build -t keystone-swift .

Start the servers

honcho start

Now you should be able to access the development server at localhost:8081. The login and password are swift, and veryfast, respectively.

This configuration has both frontend and backend servers running with code reloading features, meaning that after code changes the servers reload.

Trusted TLS

Additionally, when testing with the encrypted upload features, browser features are used that require a trusted TLS connection. This can be achieved by using a development proxy server that can be built from files in the devproxy folder. The proxy has it's own instructions for building.

This guide assumes you're using devenv as the domain name. Replace this with the domain you're certificate sings, and if necessary, add it to /etc/hosts so it's resolvable both in docker, and locally.

Additional setup is required in your environment file. You'll need to configure the following keys to point to whatever hostname will be used to access the service. Additionally you should allow all hosts, assuming your machine is in a secure network when developing. In case you trust your network and want as easy of a setup as possible, you can use all to greenlight all hosts for access.

SWIFT_UI_FRONTEND_ALLOW_HOSTS=devenv
SWIFT_UI_TLS_PORT=8443
SWIFT_UI_TLS_HOST=hostname

Additionally you'll need to configure the endpoints to be correct, so that the backend APIs work as intended.

BROWSER_START_SHARING_ENDPOINT_URL=https://devenv:9443
BROWSER_START_SHARING_INT_ENDPOINT_URL=http://localhost:9090
BROWSER_START_REQUEST_ENDPOINT_URL=https://devenv:10443
BROWSER_START_REQUEST_INT_ENDPOINT_URL=http://localhost:9091
BROWSER_START_RUNNER_ENDPOINT=http://localhost:9092
BROWSER_START_RUNNER_EXT_ENDPOINT=https://devenv:11443

If your Docker network does not match the default, you'll need to change the network configuration to make the proxy aware of the backend services. The environment network defaults to the default Docker network, which is:

DOCKER_NETWORK_SEGMENT=172.17.0.0/24
DOCKER_NETWORK_GATEWAY=172.17.0.1

If you are using MacOS and Docker Desktop, the network can be defined as:

DOCKER_NETWORK_SEGMENT=host.docker.internal
DOCKER_NETWORK_GATEWAY=gateway.docker.internal

After this, comment out the commands to run without trusted TLS in the Procfile, and uncomment the commands to run with trusted TLS.

You should now be able to run the service with trusted TLS by running

honcho start
OIDC login provider

To run with OIDC support, set the OIDC_ environment variables in the .env file and restart the services. You'll also need to build the frontend again:

OIDC_ENABLED=True pnpm --prefix swift_browser_ui_frontend run build

CSC OIDC provider's certificate should be added to certifi's certificate store:

cd swift-browser-ui
source venv/bin/activate

curl -sLo oidc-cert.pem https://crt.sh/?d=2475254782
cert_path=$(python -c "import certifi;print(certifi.where())")
cat oidc-cert.pem >> ${cert_path}
rm oidc-cert.pem

Encrypted uploads

Encrypted uploads require that Hashicorp Vault is running and configured to use the crypt4gh transit encryption plugin.

With that done, set the policies and permissions to an account, so that they can use it, and set the account token to the environment variables VAULT_ROLE, VAULT_SECRET, and VAULT_URL.

During development, you can configure and run Vault locally

Follow instructions for starting vault local dev server https://gitlab.ci.csc.fi/sds-dev/c4gh-transit#usage

Setting up API authentication

Enable approle module

vault auth enable approle

Create policy to give access rights to the role

vault policy write swiftbrowser .github/config/vault_policy.hcl

Create a new role

vault write auth/approle/role/swiftbrowser \
    secret_id_ttl=0 \
    secret_id_num_uses=0 \
    token_ttl=5m \
    token_max_ttl=5m \
    token_num_uses=0 \
    token_policies=swiftbrowser \
    role_id=swiftbrowserui

Get role and secret (= username and password), set them to the environment variables mentioned above

vault read auth/approle/role/swiftbrowser/role-id
vault write -f auth/approle/role/swiftbrowser/custom-secret-id secret_id=swiftui

Testing

Backend

The backend python tests can be run with tox. Start the mock server in one terminal, and run tox in another.

cd swift-browser-ui
source venv/bin/activate
python tests/ui_unit/mock_server.py

and in another terminal

cd swift-browser-ui
source venv/bin/activate
pip install tox
tox

Frontend

The frontend tests are run with cypress, and you will need

  1. Full backend running, as shown above
  2. Building the wasm code for encryption support
  3. Having Openstack command-line client installed
  4. using a specific command for generating data (optional)

The wasm code is built automatically when using npm commands. It can also be triggered by running npm run build-wasm

Openstack command-line client can be installed with pip by running

pip install python-openstackclient

NOTE: Remember that the encrypted upload features cannot be used without having trusted TLS set up on all backend services.

The keystone-swift image comes with a script to generate data in the object storage server. With the services running, run these commands:

docker exec keystone-swift generate_data.py --keystone --username swift --password veryfast --containers 15
docker exec keystone-swift generate_data.py --keystone --username swift --password veryfast --project "swift-project"

After following the development steps above, cypress should already be installed.

cd swift_browser_ui_frontend

You can run the tests in headless mode

pnpm cypress run

Or you can use the interactive version

pnpm cypress open

It's possible to set the host to run against by using the environment variable CYPRESS_BASE_URL, so that it may run against the development frontend server, for e.g.

CYPRESS_BASE_URL=http://localhost:8081 pnpm cypress open

WebAssembly C Code Unit Tests

C code is tested using a unit tests collection built using ceedling. They can be ran by navigating to the swift_browser_ui_frontend/wasm directory and running the ceedling command. You'll have to install ceedling first, which can be found here

License

swift-browser-ui and all it sources are released under MIT License.