How to structure ERT, webviz-ert and ert-storage to best fit present and future needs

[oysteoh]

The discussions regarding the architecture, optimization of the current solution and make that work spawns a lot of questions which once and for all needs to be clarified in order for us to make decisions. What is important now is to identify if there is something someone knows today for sure will lead us to a dead-end 12 months down the road. What do we need to consider with regards to architecture, how to deploy and run. Is it possible making one piece of software working equally good on-prem as in azure or do we need to differentiate?

Definition of the discussion done is everyone boarded on the same train and agreeing to what is to be solved and what the boundaries is, what the architecture is to look like and what tools we are to use.

Store data

• Are we allowed / is it possible to utilize postgres for all of our data when moving to azure?
• Are there other time-series databases ( or equivalent databases ) we could use which could serve us our data in smarter ways other than files as azure-blobs?
• Will serving the data through azure-blobs be possible with one hosted api with regards to memory consumption? I'm thinking about many users visualizing their runs and hitting the same api instance pulling 1000's of blobs in various sizes could be a major bottleneck.
• Do we need to utilize blob-store for the actual data and only use postgres for relational data?
• Running on-prem - is the api to be hosted by dark-storage? We do not need to think about sqlite anymore?

Architecture

• Is webviz-ert going to be one instance hosted in Azure pulling data from different datasources? Or will it be multiple versions hosting different datasources? Or will it be ran as is today on RGS in azure one instance pr user on-demand?
• Is ert-storage going to be one instance hosted in Azure pulling data from different databases? Or will it be multiple versions hosting one database each? Or will it be ran as is today on RGS in azure one instance pr user on-demand connecting to a database?

Programming language for the ert-storage part of our software

• Is Python the best ( best as in good enough for us with our needs and boundaries we have to live with ) language for us hosting an high performing API serving a lot of data? Is it good enough? Does it has limitations we cannot live with?

Maybe not all of my questions make sense, but at least it's a good starting point. Please provide your thoughts / questions / concerns as well and let us agree to something 🙏 🚀

[dafeda]

In order to easier discuss different storage solutions, we should first have a sense of the requirements in terms of bandwidth and in terms of total capacity.

I've heard numbers in the hallways, but have we done a thorough analysis?

[dafeda]

This might be a bit silly, but I googled "HPC in Azure" and got this:

https://azure.microsoft.com/en-us/solutions/high-performance-computing/

I'm sure it's been considered, but since we are moving a HPC application to Azure it doesn't hurt to check...

Azure NetApp Files is perhaps particularly interesting as they write: "Azure NetApp Files is widely used as the underlying shared file-storage service in various scenarios."

[oysteoh]

We are going to utilize their compute cluster yes, but initially we have been thinking without the underlying disk assumption we have today. I`m not sure if this NetApp Files has been looked into - but if not we should i guess! That said, our discussion in this thread is more about how and where to store the data from the finished ERT runs and how to expose these for various applications - among others the webviz-ert instance. And - how is this to be deployed and runned?

[markusdregi]

ANF has been considered yes, but does not satisfy our needs...

[mortalisk]

Is it too expensive? Or are there technical limitations?