💄 Integrate visualisation MI/BI tool

[Ed-Bajo]

User Story

As a…
As an AP customer, I would like to publish visualisations based on AP data via the visualisation and dashboard tools, so that data can be viewed and disseminated.

Value / Purpose

The availability of a visualisation or dashboard tool will remove the need of building visualisation tools from scratch.
It will make it easier, faster and better for people to create, share and consume trusted data, metrics and insights

Useful Contacts

@julialawrence
@Ed-Bajo

Proposal

During our discovery and recent reprioritisation exercises, it was identified that one of present needs of the Analytical Platform users are dashboard and visualisation tools. In addition, it was also established that there was a need to expand our tooling offerings on the platform (e.g. PowerBI, Quicksight, etc).
The team's focus is to carry out discovery to ascertain users needs and business requirements; in respect of visualisation and dashboard tools. In parallel, the team will continue to explore options to expand our tooling offerings for the AP by conducting technical spikes and proof of concept to establish the technical feasibility of integrating new analytical tools onto the platform (e.g. Quicksight, etc).Additional Information

Additional Information

Calum's team would be leading on the strategy for visualisation and dashboard tools. We are expected to work collaboratively with his team.

AWS QuickSight already exists in the analytical-platform-data-production account. There are 40+ users in here.

13/03/2024 - MVP Scope:

Users can (without our interference):

• Authenticate with AP UI using their justice identity in front page
• Access and use QuickSight from within AP UI
• Have their permissions in AP/CP transfer into QS
• Make datasource available in QS
• Create dataset, analysis, dashboard
• Share dashboard with another QS user (Permissions)
• Use datasources in APDP seamlessly
• Traverse without reauthenticating between CP and AP
• Get some information about the use of QS. (API endpoint to get number of APUI users and QS datasources)
• Use some mechanism to share dashboards with users who are not AP users

Definition of Done

• Gather business requirements in respect of dashboard and visualisation tools
• Complete dashboard and visualisation Quicksight tool proof of concept
• Integrate identity management process and functionality for accessing the tools on the Analytical Platform
• Make recommendations on dashboard and visualisations tools to be integrated onto the Analytical Platform

Linked Stories

• 💷 QuickSight Licensing #2978
• ⚗️ Spike: Break down Dashboarding Epic #2996
• 🗺️ Create Wardley map to formalise AWS QuickSight usage on the Platform #3053
• 🧪 Test the following assumptions for Authors, Data Admins and AP Admins #3061
• 🔐 Spike: Investigate Row-Based and Tag-Based Security #3062
• 🎨 Create terraform to provision QuickSight in the new AP account #3064
• 🧵 Manage Quicksight S3 datasource permissions in the Analytical Platform Dashboard #3065
• 💡 Implement backend changes to allow the AP dashboard to grant access to Quicksight #3066
• 💡 Implement Control Panel changes to manage user permissions internal to QuickSight #3067
• 💬 Decide how we should operate AWS QuickSight #833
• ⚗️ Propose a way forward on QuickSight DataSources #3109
• 🎨 Service Design: Map QuickSight UI Journey for Authors, Data Admins and AP Admins #3121
• 🧪 End-to-End test of PowerBI setup #3130
• 🤔 Make decision on integrating QuickSight to the Analytical Platform #3296
• 🔒 Authentication for QuickSight #3311
• 🔑 IAM Policies Interaction with QuickSight #3312
• 🔧 Management of Current QuickSight Deployment #3313
• 🛠️ Designing QuickSight Production Architecture #3314
• ✨ Button to access QuickSight from Control Panel #3228
• 🛂 Placeholder - Alpha User QuickSight Permissions #3233
• 🏗️ Implement Control Panel Changes for S3 Policy Management #3693
• 📦 Capture the QuickSight 'As Is' and 'To Be' user journey #3696
• 🏗️ Finalise QuickSight Identity Approach and Architecture #3703
• 📖 Request new AP accounts from MP #3701
• 🛂 Spike - Create an IAM role that we can assign to users in Quicksight #3780
• 🔐 Add Azure app secrets to be read by Control Panel #3868
• 🔐 Template IAM Policies for QuickSight Users #3878
• 📖 Implement EntraID Identity Provider in MP Compute Account + Test Identity Federation #3879
• ⚗️ Investigate RAM Sharing of Glue Catalogue into MP Compute Account #3880
• 🔧 QuickSight Terraform #3881
• 🏗 Build Analytical Platform Compute EKS #3555
• 🔐 Implement Automated Rotation of AzureAD credentials #3990
• 🚀 Deploy the Analytical Platform UI in a development account #3698
• 📖 Implement terraform modules for sharing Athena resources via Lake Formation #4358
• 📖 Carry out tests on Lake Formation Hybrid Mode #4359
• 📖 Implement User Onboarding to QS #4360
• 📖 Register Assets Managed in database-access in Lake Formation & Share them into AP Compute #4361
• 📖 Design and Implement a mechanism for syncing permissions between AP Compute for QS and database-access #4362
• 📖 Update Documentation and DIagrams for QuickSight to reflect new approach #4364
• 📊 Bootstrap Lake Formation in Analytical Platform Compute and APDP Accounts #4461
• 📈 Document a Process for registering and sharing a Database/Table/Column from Ireland APDP to London APC using Lake Formation #4468
• 📝 Identify and document datasets to be used in in Quicksight MVP #4655
• 📖 Embed Quicksight within the AP UI #4658
• 📖 Determine IAM permissions required when using hybrid mode #4659
• 📖 AP UI users can grant access to their Glue resources #4748
• 💧 Work through data-engineering-database-access repo for projects to Lake Formation Tags #4749
• 🕵️ Investigate how existing Lake Formation is used in Data Production #4737
• 🔐 Amend IAM Roles to include lakeformation:GetDataAccess permission #4837
• 🪪 Complete Identity Center Integration With EntraID to Reach Transitional State #4838
• 🔒 Refactor AP Service Role Policy #4913
• 📖 Update LakeFormation sharing module to automate manual onboarding steps #5378
• 📖 Quicksight MVP next steps #5381
• 📖 Fix Lingering Issues in LF Module and EntraID script #5383
• 📖 Adjust language in the AP UI #5459
• 📖 Default to displaying "My access" on the AP UI database access page #5460
• 📖 Add links to Quicksight guidance to the AP UI #5461
• 📖 Add pagination to the AP UI #5462
• 📖 Grant multiple users access to database tables via the AP UI #5463
• 📖 Search for users to grant access to tables in the AP UI #5464
• 📖 Add basic guidance on using the AP UI #5465
• 📖 Additional filter options when viewing databases and tables in the AP UI #5466
• 📖 Implement Replication for mojap-derived-tables bucket into APC Ireland #5599