WebAppArchitecture
The purpose of the Pizeo web app is to allow users to submit Spark jobs on the Kubernetes cluster running on IRIS hardware. Any code and input data required for this job to run must already have been uploaded to ECHO, and the job outputs (logs and data) are written to ECHO rather than returned directly to the user. An instance of the Harbor web app must also be deployed (presumably also on IRIS) to provide access to the Spark Operator Docker image; it is sensible for the Harbor storage to be located on ECHO.
Not strictly part of the Piezo Web App, but crucial to user interaction with the system as it allows requests from outside the private subnet. Hosted on the kubernetes cluster.
The thin layer around the web API will be responsible for providing routes and directing requests to the appropriate services inside the web API.
The logging and monitoring service will be responsible for proving a generic logging mechanism to the rest of the web API as well as a way to gather metrics about the system.
Responsible for coordinating the actions of the components described below.
Responsible for checking that user inputs for a Spark job are valid, and for clearly explaining problems to the user if they are not.
Responsible for producing a manifest that can be submitted to the Spark Operator for an individual job. Fills a template with user-defined inputs and default values (where applicable), handling any special cases caused by other input choices (e.g. some inputs depend on the choice of programming language).
The Spark Job Manifest Populator is explicitly not responsible for checking that the inputs are sensible: this is the responsibility of the Spark Job Validator. Passing in appropriate inputs to the Spark Job Manifest Populator may lead to their inclusion in the manifest or to a system failure. It is for this reason that integration tests are especially important here.
A thin wrapper around the Python package for Kubernetes. Wrapping in this way allows unit testing of the Spark Job Service and integration testing of the entire web app.
Connects the Spark job service to the storage adapter. The default bucket name is kept here, and any storage business logic that is not implemented in Boto is handled here.
A thin wrapper around the Boto client for S3. Wrapping in this way allows unit testing of the Spark Job Service and integration testing of the entire web app.