Skip to content

Latest commit

 

History

History
417 lines (328 loc) · 21.3 KB

README.adoc

File metadata and controls

417 lines (328 loc) · 21.3 KB
Raw

Integration tests for Bayesian core API

This repository contains integration tests for the fabric8-analytics services.

The tests can be run against existing deployment, or locally by using docker-compose.

For Local System only, Set values of following environment variables in env.sh, to test specific deployments:

  • F8A_API_URL - API server URL

  • F8A_JOB_API_URL - Jobs service URL

  • AWS_ACCESS_KEY_ID - Access Key for Staging/Prod AWS (Optional)

  • AWS_SECRET_ACCESS_KEY - Secret Key for Staging/Prod AWS (Optional)

  • S3_REGION_NAME - S3 Region name (Optional)

  • THREE_SCALE_PREVIEW_USER_KEY - Staging/Prod 3 Scale User Key

  • OSIO_AUTH_SERVICE - Staging/Prod Auth Url

  • F8A_THREE_SCALE_PREVIEW_URL - Staging/Prod 3 Scale Authentication Url

  • F8A_SERVICE_ID - Service Id

  • F8A_GREMLIN_URL - Staging/Prod Gremlin Url for DB Queries

  • F8A_GEMINI_API_URL - Staging/Prod Gremlin API Url.

  • RECOMMENDER_REFRESH_TOKEN - Refer this

By default, the system running on the localhost is tested. The integration tests are not containerized currently hence they start and stop Fabric8 Analytics multiple times. Thus the following prerequisites need to be met:

  1. Configure a Python environment on the host system.

  2. Ensure that you are a member of the docker group to execute docker-compose without sudo. For information about setting up and using Docker as a non-root user, see here.

Creating new tests

Feature tests are written using behave.

To add new feature tests:

  1. Edit an existing name.feature file in the features/ folder (or create a new one) and save it as new-name.feature.

  2. Specify the missing steps in the common.py file in the features/steps/ folder (or create a new step file, where appropriate).

Currently defined feature tests

Older tests that have to be updated or deprecated

Adding new feature files

When you add a new feature file, you must add it to the feature_list.txt file, as it determines the set of features executed by the runtest.sh script.

Currently defined test steps

Documentation for the module with test steps is automatically generated into the common.html file. To know more about the available test steps see the existing scenario definitions for usage examples, or the step definitions in features/steps/common.py and the adjacent step files.

Adding new test step files

When you add a new test step file no additional changes are needed, as behave automatically checks all Python files in the steps directory for step definitions.

Note that a single step definition can be shared among multiple steps by stacking decorators. For example:

@when('I wait {num:d} seconds')
@then('I wait {num:d} seconds')
def pause_scenario_execution(context, num):
    time.sleep(num)

Thus it allows client pauses to be inserted into both Then and When clauses when defining a test scenario.

Writing new test steps

The behave hooks in features/environment.py and some of the common step definitions add a number of useful attributes and methods to the behave context.

The available methods include:

  • is_running(): Indicates whether the core API service is running.

  • start_system(): Starts the API service in its default configuration using Docker Compose.

  • teardown_system(): Shuts down the API service and removes all related container volumes.

  • restart_system(): Tears down and restarts the API service in its default configuration.

  • run_command_in_service: See features/environment.py for more information.

  • exec_command_in_container: See features/environment.py for more information.

The available attributes include:

  • response: A 'requests.Response' instance containing the most recent response retrieved from the server API. Ensure that, steps making requests to the API set this, steps checking responses from the server query it.

  • resource_manager: A contextlib.ExitStack instance for registering resources to be cleaned up at the end up of the current test scenario.

  • docker_compose_path: A list of Docker compose files defining the default configuration when running under Docker Compose.

The context life cycle policies defined by behave ensure that any changes to these attributes in step definitions remain in effect only until the end of the current scenario.

Host environment

The host environment must be configured with docker-compose, the behave behavior driven development testing framework, and a few other dependencies for particular behavioral checks.

You can configure the host environment in either of the following ways:

  • Install the following components:

    $ pip install --user -r requirements.txt

  • Set up a Python virtual environment (either Python 2 or 3) and install the necessary components:

    $ pip install -r requirements.txt

Test execution

The test suite is executed as follows:

$ ./runtest.sh <arguments>

Note that arguments passed to the test runner are passed through to the underlying behave invocation. See the behave docs for the full list of available flags.

The following custom configuration settings are available:

  • -D dump_logs=true (optional, default is not to print container logs): Requests display of container logs via docker-compose logs at the end of each test scenario

  • -D dump_errors=true (optional, default is not to print container logs): Provides dump_logs only for scenarios that fail.

  • -D tail_logs=50 (optional, default is to print 50 lines): Specifies the number of log lines to print for each container when dumping container logs. Implies dump_errors=true if neither dump_logs nor dump_errors is specified

  • -D coreapi_server_image=bayesian/bayesian-api (optional, default is bayesian/bayesian-api): Name of Bayesian core API server image

  • -D coreapi_worker_image=bayesian/cucos-worker (optional, default is bayesian/cucos-worker): Name of Bayesian Worker image

  • -D coreapi_url=http://1.2.3.4:32000 (optional, default is http://localhost:32000): Core API URL

  • -D breath_time=10 (optional, default is 5): Time to wait before testing

Important
 Running with non-default image settings will force-retag the given images as bayesian/bayesian-api and bayesian/worker so that docker-compose can find them. This may affect subsequent docker and docker-compose calls.

Some of the tests may be quite slow, you can skip them by passing --tags=-slow option to behave.

Packages that need to be imported into the database

The following packages need to be imported into the database for successful test run:

NPM ecosystem

sequence
array-differ
array-flatten
array-map
array-parallel
array-reduce
array-slice
array-union
array-uniq
array-unique
lodash
lodash.assign
lodash.assignin
lodash._baseuniq
lodash.bind
lodash.camelcase
lodash.clonedeep
lodash.create
lodash._createset
lodash.debounce
lodash.defaults
lodash.filter
lodash.findindex
lodash.flatten
lodash.foreach
lodash.isplainobject
lodash.mapvalues
lodash.memoize
lodash.mergewith
lodash.once
lodash.pick
lodash._reescape
lodash._reevaluate
lodash._reinterpolate
lodash.reject
lodash._root
lodash.some
lodash.tail
lodash.template
lodash.union
lodash.without
npm
underscore

PyPi ecosystem

clojure_py
requests
scrapy
Pillow
SQLAlchemy
Twisted
mechanize
pywinauto
click
scikit-learn
coverage
cycler
numpy
mock
nose
scipy
matplotlib
nltk
pandas
parsimonious
httpie
six
wheel
pygments
setuptools

Maven ecosystem

io.vertx:vertx-core
io.vertx:vertx-web
io.vertx:vertx-jdbc-client
io.vertx:vertx-rx-java
io.vertx:vertx-web-client
io.vertx:vertx-web-templ-freemarker
io.vertx:vertx-web-templ-handlebars
io.vertx:vertx-web
org.springframework:spring-websocket
org.springframework:spring-messaging
org.springframework.boot:spring-boot-starter-web
org.springframework.boot:spring-boot-starter
org.springframework:spring-websocket
org.springframework:spring-messaging

Resilient infrastructure tests

Run the resilient infrastructure tests as follows:

  1. Ensure that you have logged into OpenShift before the tests are run. These tests access OpenShift Console i.e.. the oc command.

  2. Switch to the right project.

    Important
    		 These tests restart different pods, so ensure that you do not run them against the production environment.

    To make sure you are switched to the right project in OpenShift use:

    $ oc projects

    The selected project is marked by *, for example:

    *  my-test-project
   bayesian-preview
   yet-another-project

    To switch to another project use the following command:

    $ oc project <project-name>

    For example:

    $ oc project bayesian-preview

  3. Start the resilient infrastructure tests using:

    $ ./runtest.sh --tags resilient.infrastructure

Security tokens for tests

A brief about setting up security tokens for end to end tests.

Currently we use the following user for test account: ptisnovs-preview-osiotest1

Caution
 As the offline token feature manifested in a point of vulnerability (where potential attackers may exploit a stolen token across an extensive period of time, without concern for the token expiring), we now recommend that standard access tokens, obtained using the standard OAuth flow are used instead.

The process looks like:

  1. Login to OSIO and acquire coded token

  2. Decode the refresh token

  3. Store the refresh token into Vault

  4. Setup CI jobs to put refresh token into environment variable with a known name

  5. Use this environment variable

Acquire token needed for most REST API tests

Important
 please choose the right system - production or pre-production!

To get the token for production system, open the following page:

To get the token for prod-preview, open the following page:

After logging in, you will be redirected to another URL.

Look at the URL of the new page.

Copy the <JSON> part from the URL, it will look like this:

%7B%22access_token%22%3A%22foobar22expires_in%22%3A2592000%2C%22not-before-policy%22%3Anull%2C%22refresh_expires_in%22%3A2592000%2C%22refresh_token%22foobar%22token_type%22%3A%22Bearer%22%7D

Use conversion function to convert these data into JSON format:

Conversion function:

urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }

Usage:

urldecode `cat url_part.txt` > url_part.json

Result should look like this:

"access_token":"foobar",
"expires_in":2592000,
"not-before-policy":null,
"refresh_expires_in":2592000,
"refresh_token":"foobar",
"token_type":"Bearer"

Get just the refresh_token part and store it into file named refresh_token.txt

Caution
 Make sure that the file don’t end with a new line. It will cause problems because the Vault CLI tool will use the whole content of a file, including newline, which is not correct.

TIP for VIM users: use the following settings to remove EOLN

:set binary
:set noendofline

For CI, Please Refer CI_README.adoc

Common issues

Please look into Standard operating procedures document for exlanation of most common issues.