- Implemented Key-Outputs and Interface for DSL 2.0 workflows
- Support setting the source to an application-dependency to a directory that's present in the Git repository of the workflow
- Improved validation of DSL 2.0 for command-line tools as well as the Build Canvas page of ST4SD Registry UI
- Miscellaneous optimizations to the ST4SD registry UI
- Registry UI now supports plotting histograms of properties
- Fixed bug in light validation of DSL 2.0 which would be triggered when the validated graphs contained templates for which some parameters were unset
- Fixed miscellaneous bugs in st4sd-registry-ui (most important are firefox compatibility and modals responsiveness)
- Improved light validation of DSL 2.0 helps catch more errors in Graphs before they are admitted into the Graph Library
- Verify the contents of Secrets containing S3 credentials for Internal Experiments (i.e. Build Canvas feature) and the Graph Library
- Support building experiments using an interactive Build Canvas
- Experiments created this way can also be Edited using the same Build Canvas
- Support for a Graph Library containing reusable Graph recipes
- (Beta) Support for Experiment Domain Specific Language 2.0 (DSL 2.0)
- Disable "View runs" link on ST4SD Registry for global instances
- Introduce View Canvas for viewing experiment graphs.
- (BETA) Introduce Edit Canvas for applying transformations to experiments.
- Miscellaneous fixes
- Enhanced support for Transformation Relationships: More powerful substitutions (e.g. replace references with variables and vice versa)
- Generate DSL 2.0 for Parameterised Virtual Experiment Packages (PVEP) (including synthesized ones).
- Generate a preview of the would-be DSL for a synthetic PVEP before applying a Relationship to synthesize it.
- Re-use default
gitsecret-oauth
that ST4SD admins manually register in st4sd-runtime-service ConfigMap without using Helm
- Miscellaneous fixes
- Upgrade to Vue 3
- UI can generate
stp login
command (similar to the OpenShift Web Console feature foroc login
) - Miscellaneous enhancements to improve User Interface and User Experience (UI/UX)
- Miscellaneous fixes
- Support deploying ST4SD in namespaces containing ResourceQuota objects
- Ensure properties endpoint produces valid JSON in st4sd-registry-backend
- Add endpoint for retrieving PVEP properties from all of its runs in st4sd-registry-backend
- Add property table and line chart for all runs of a PVEP in st4sd-registry-ui
- Allow searching experiments by properties in st4sd-registry-ui
- Allow fetching measured properties in st4sd-registry-backend
- Show measured properties for runs in st4sd-registry-ui
- Fix CVE-2022-25881 in st4sd-registry-ui
- Improved search functionality in the Experiment Registry UI
- Improved error-reporting in st4sd-runtime-service
- Surrogates: st4sd-runtime-service uses intra-dependencies to reduce the set of parameters which require an explicit
relationship.graphParameteers
mapping - Fix resolution of environment variables in Components which use a
DEFAULTS
instruction
- Runtime support for surrogates. The runtime registry supports defining relationships between graphs, using the relationships to auto-generate surrogate parameterised virtual experiment packages, and decide between starting a parameterised package and one of its surrogates based on a runtime policy.
- Local instance of ST4SD registry for parameterised virtual experiment packages. The registry supports viewing the parameterised packages in the ST4SD instance as well as runs of the parameterised packages and their logs. You can access your registry UI at
https://${your_domain}/registry-ui/
. - Documentation to install the full arsenal of ST4SD microservices locally.
- Public images! We will store our public images in https://quay.io/organization/st4sd/
- New API to upload relationships (
ExperimentRestAPI.api_relationship_push()
) - see our documentation for runtime support for surrogates documentation for more. - Updated the API to start an experiment so that users can also request to use a runtime policy. For more information see our documentation running virtual experiments on OpenShift and configuring runtime policies.
- Miscellaneous optimisations in error reporting
- Improved definition of virtual experiment entries in the registry. They are now self-contained. We refer to them with the term
parameterised virtual experiment packages
.- Documentation for creating parameterised virtual experiment packages here.
- Global ST4SD registry of parameterised virtual experiments. The packages in this registry are accessible by all ST4SD deployments.
- Updated the payload to
ExperimentRestAPI.api_experiment_push()
to support the new functionality that the ST4SD registry provides. Please read instructions for updating v1.5.3 virtual experiment entries to v1.6.0 parameterised virtual experiment packages entries.
- Miscellaneous optimisations in error reporting
- Support for virtual experiment interface (measure one or more characteristics of one or more input systems).
- Support defining namespace wide arguments to virtual experiment instances (e.g.
--executionMode="production"
) - Support executing specific commits of virtual experiment definitions hosted on Git
- Alpha: Support customising kubernetes tasks via
components.resourceManager.kubernetes.podSpec
- Miscellaneous fixes in the K8s backend and the Helm chart
Updated MongoDB backend to 5.0.9
- Miscellaneous fixes in the K8s backend.
This is the first release of the toolkit as the brand Simulation Toolkit for Scientific Discovery!
- Replaced custom deployment framework with helm charts
- Miscellaneous fixes and improvements
- Improved error handling and syntax error reports. The runtime will now even report what it thinks you meant to type!
- Improved RBAC for Kubernetes objects, received clearance by ETE to deploy on shared clusters.
- Improved reactive implementation leading to greater scalability.
- Support at least 4000 components per experiment.
- Garbage collection for kubernetes objects: Use it by providing the additionalOption
--executionMode=production
- For more information, run
elaunch.py --help
- For more information, run
- Security
- Improved authentication / authorization for services
- Support burstable and best-effort classes on k8s. Burstable pods start with 1/10th of their resource requirements and may grow.
- Please read the Kubernetes documentation regarding the
Quality of Service
and how that affects pod eviction. - You can configure the QoS of your kubernetes components by setting
component.resourceManager.kubernetes.qos
to [Burstable
,BestEffort
,Guaranteed
] (match is case-insensitive). - Default value is
Guaranteed
.
- Please read the Kubernetes documentation regarding the
- Miscellaneous fixes and improvements
Updated the internal structure of the experiment
python module. For example, experiment.db.ExperimentRestAPI
is now experiment.service.db.ExperimentRestAPI
.
All microservices now require authentication. Additionally, the microservices now reject OpenShift tokens if the associated account does not have permission to Get
the workflow-authentication
Service object inside the namespace that the workflow stack is deployed to.
- New MongoDB documents in the CDB are easier to digest
- All documents now contain a
uid
field. Component documents also contain aproducers
field that points to theuids
of their producers. - For more information see the api-quickstart notebooks.
- All documents now contain a
- Miscellaneous fixes and improvements
- Renamed nearly all
experiment.db.ExperimentRestAPI.api_instance_*
methods toexperiment.db.ExperimentRestAPI.api_rest_uid_*
. Exceptions:api_instance_create
is nowapi_experiment_start
api_instance_create_lambda
is nowapi_experiment_start_lambda
- Changed most
experiment.db.ExperimentRestAPI.cdb_*
methods to feature a consistent interface.- In the deprecated API, arguments to the
cdb_*
methods were automatically converted to regular strings. Now, this only happens to parameters other thanquery
. Thequery
argument is treated as a MongoDB query. - For example, to query for documents containing the
hello
key with a value that matches the regular expressionworld[0-9]+
you may use the$regex
operator of MongoDB insidequery
like so:api.cdb_get_document(query={"hello": {"$regex": "world[0-9]+"}})
. - Documentation for the
$regex
operator: https://docs.mongodb.com/manual/reference/operator/query/regex- You can find more information on MongoDB operators at the official documentation: https://docs.mongodb.com/manual/reference/operator/query/
- In the deprecated API, arguments to the
- We introduce a new class
experiment.db.ExperimentRestAPIDeprecated
that offers the interface of the now deprecated API. Users may find this new class useful to update their Python notebooks with minimum effort to make them compatible with the latest stable version of the workflow stack. Users may then work on a copy of their code to gradually replace the use ofexperiment.db.ExperimentRestAPIDeprecated
withexperiment.db.ExperimentRestAPI
. - Components that define neither
workflowAttributes.maxRestarts
norworkflowAttributes.restartHookFile
will no longer re-execute indefinitely on aResourceExhausted
exit-reason. They will restart up to 3 times.