ServiceRegistry to Manage migration

ServiceRegistry to Manage migration

Introduction

This page describes the steps needed for migrating ServiceRegistry to the new registry application called "Manage".

Serviceregistry is based on PHP with a MySQL storage backend: Manage is a Java application with a Mongo database backend. A lot of attention has been put into the migration process, to make this as smooth and predictable as possible. Nevertheless, it's strongly recommended to test the migration process.

The steps:

1. Ensure that your EngineBlock version uses the database backend (Janus PUSH-method)
2. Install MongoDB and Manage (< 3.0)
3. Verify data migration and make Manage leading
4. Remove metadata-exporter and serviceregistry

Use the PUSH mechanism to provision the entities to Engineblock

Currently, Engineblock supports two mechnisms to get information about SP and IdP entities. You have to make sure that you use the second mechanism (the 'PUSH' mechanism)

1) SR API access (PULL mechanism)

Engineblock can query the SR api directly during the authentication. That will look like this in /etc/openconext/engineblock.ini:

metadataRepository.database.type = JanusRestV1
metadataRepositories[0] = janus

With corresponds to the following in your Ansible group variables:

engine_block:
  metadataRepositories:
      - {
        index: 0,
        type: "janus",
        value: "JanusRestV1"
        }

Support for this method will be removed in EngineBlock 5.7.

2) PUSH mechanism

ServiceRegistry and Manage have a mechanism to push all the entities to the engineblock api. Engineblock can then query its own database for information on known entities. This option is more robust, and scales (a lot!) better than the previous mentioned method. This is configured like this in the engineblock.ini file:

metadataRepository.database.type = Doctrine
metadataRepositories[0] = database

And the representation in the Ansible group_vars:

engine_block:
  metadataRepositories:
        - {
        index: 0,
        type: "database",
        value: "Doctrine"
      }    

Migration to this PUSH mechnism is quite straightforward: Push the big "PUSH" button in ServiceRegistry, and change the engineblock.ini parameters:

metadataRepository.database.type = Doctrine
metadataRepositories[0] = database

Install the necessary components

You need a version of Manage < 3.0 to perform the migration; versions >= have dropped migration support. Of course you can upgrade your Manage to the current version after the migration is complete.

You need to install the role mongo, manage-server and manage-gui. Add these roles to your playbook and add the following variables:

group_vars:

manage_gui_version: 2.0.9
manage_server_version: 2.0.9

mongo_port: 27017
mongo_cluster: false
mongo_tls: false

mongo:
  manage_user: managerw
  manage_password: "{{ mongo_passwords.manage }}" 
  manage_database: manage

manage:
  db_user: "{{ janus_database_user }}"
  db_password: "{{ mysql_passwords.sr }}"
  db_name: "{{ janus_database_name }}"
  db_host: "{{ mysql_host }}"
  eb_db_user: "{{ engine_database_user }}"
  eb_db_password: "{{ mysql_passwords.eb }}"
  eb_db_name: "{{ engine_database_name }}"
  eb_db_host: "{{ mysql_host }}"
  mongo_port: "{{ mongo_port }}"
  mongo_user: "{{ mongo.manage_user }}"
  mongo_password: "{{ mongo.manage_password }}"
  mongo_database: "manage"
  backdoor_api_user: backdoor_api_user
  run_migrations: true
  push_after_migration: false
  features: migration, validation, push_preview, push
  apiUsers:
    - {
        name: "pdp",
        password: "{{ manage_pdp_secret }}",
        scopes: ["READ"]
      }

• Make sure it's added to the loadbalancing:, haproxy_applications: and shibboleth_apps: variables in the group_vars as well.

• Add manage to the apache_app_listen_address: in your host_vars.

Add the following secrets:

mysql_passwords.manage:
mongo_passwords.manage:
mongo_admin_password:
manage_pdp_password:
manage_internal_api_password: 
manage_backdoor_api_password:
manage_dashboard_secret: 
manage_pdp_secret: 
manage_sp_dashboard_secret: 

Please refer to the template environment directory (environments/template) for a complete overview.

For more information on the Mongo role, please refer to the README.md in the mongo role directory.

When the installation is finished, Manage will perform a migration from ServiceRegisty every 15 minutes, so that Manage stays up to date with all ServiceRegistry changes. A manual migration from the interface is supported as well.

The following three group_vars are used to make this happen:

manage.run_migrations: This variable controls whether an automatic import from the ServiceRegistry database is run every 15 minutes
manage.push_after_migration: Whether an automatic push to engineblock is done after the migration
features: migration, validation, push_preview, push: This controls which feautures are available in Manage: 
 - migration: This feautures allows an import from ServiceRegistry (by accessing its' database
 - validation: This feauture adds the option in the Manage gui to validate the migration against the metadata template
 - push_preview: This adds an option in the Manage gui to preview the JSON file that is pushed to the Engineblock API
 - push: This enables or disables the option to PUSH the metadata to engineblock

Verify data migration and make Manage leading

In the default installation Manage will migrate data from Serviceregistry every 15 minutes (so changes made in Manage only are lost). You can use the validation button to detect any problems with the data migration.

To start using Manage, press the Push button from Manage and Engineblock will be provisioned with the data in Manage. To revert back to Serviceregistry, press Push in Serviceregistry again. When satisfied with the Manage push, disable the migration feature. Manage will stop migrating data and you can start updating entities in Manage instead. Do not use Serviceregistry anymore from this point since the data in it will become stale.

Your migration is now essentially finished.

Remove metadata-exporter and serviceregistry

When you are satisfied with the migration, you can remove metadata-exporter (dependency of some Java apps) and serviceregistry itself from your installation.

Run the Ansible role remove-metadata-exporter to do this for metadata-exporter. For serviceregistry, run the role remove-janus.