📖 Synchronise Permissions between LakeFormation and DEDA for mojap-derived-tables bucket

[julialawrence]

User Story

DEDA -- Data engineering database access

I would like to be able to make contents of the CaDeT output bucket (COB) available for visualisation without compromising the security and running afoul of the existing permissions while keeping prod running. One part of this requires replication of the data into a separate account covered in this story and the other is synchronising bucket permissions between https://github.com/search?q=repo%3Amoj-analytical-services%2Fdata-engineering-database-access%20mojap-derived-tables&type=code and LakeFormation in the replicated account. The requirement is for these permissions to be updated synchronously or as near-real-time as possible.

Value / Purpose

Security is good. No security is bad.
Making QS available quicker while avoiding risks inherent in our current MVP setup is project goal and a user need.

Useful Contacts

Julia Lawrence Jacob Hamblin-Pyke

User Types

No response

Hypothesis

If we can manage the permissions in near-real-time, we will not be opening a security gap in our current setup via QS access.

Proposal

Identify all project files in DEDA that reference mojap-derived-tables bucket
Implement a bolt-on terraform or pulumi component in https://github.com/moj-analytical-services/data-engineering-database-access which is triggered by changes in config and project files that mention the COB and does the following

1. Map a user's github identity to their justice identity (the goal is to use an API for this going forward, but to speed development, using a lookup list for now is acceptable)
2. Infer their QS username from their justice identity
3. Use LakeFormation sharing role that exists in APC Production to assign the QS user required LF permissions in both London and Ireland on the replicated bucket paths
4. Undoes these changes when the configuration files change

In order to minimise refactoring, it might be worthwhile for the new component to run on all configuration-related changes in order to avoid changing the code when new projects are added and old one are removed.

Note: Specific permissions on each bucket paths are set up in iam_config.yaml files under each project. Specific user permissions are defined here

Additional Information

There will be a follow-on story raised to replicate the datbase/tables permissions as well. Buckets are split out to identify risks.
The replicated bucket will be managed in LF only which should simplify permissions management.

Definition of Done

• Implement proposal
• Follow-on stories raised
• Team reviewed.

[julialawrence]

For information: https://docs.google.com/document/d/1h6lft2IWB2eImFf6_Vd6DOFkLU-FNZIC0yJncmCimOc/edit