Store everest results in ERT storage

[yngve-sk]

Issue
Resolves #8811

Store it as some dataframes and dump it in storage, not entirely sure how much of it should be saved. Will use everest_data_api as reference for what is required.

Objectives, constraints and control names are pivoted so that each distinct value has its own column, which is more concise and better to serve via an API. It makes each row in the dataset represent the entire set of controls, objectives, and constraints for one batch | batch -> realization | batch -> realization -> perturbation.

objectives_for_batch.to_dicts()
Out[5]: 
[{'result_id': 0,
  'batch_id': 0,
  'total_objective_value': 1886.010009765625,
  'npv_function': 1886.010009765625}]
objectives_per_real.to_dicts()[:4]
Out[6]: 
[{'batch_id': 0,
  'realization': 0,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626},
 {'batch_id': 0,
  'realization': 1,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626}]

Objective results:

objectives_for_batch
Out[1]: 
shape: (1, 4)
┌───────────┬──────────┬───────────────────────┬──────────────┐
│ result_id ┆ batch_id ┆ total_objective_value ┆ npv_function │
│ ---       ┆ ---      ┆ ---                   ┆ ---          │
│ i64       ┆ i64      ┆ f64                   ┆ f64          │
╞═══════════╪══════════╪═══════════════════════╪══════════════╡
│ 0         ┆ 0        ┆ 1886.01001            ┆ 1886.01001   │
└───────────┴──────────┴───────────────────────┴──────────────┘
objectives_per_real
Out[2]: 
shape: (2, 19)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ npv_funct ┆ group_w14 ┆ … ┆ group_w11 ┆ group_w15 ┆ group_w06 ┆ group_w00 │
│ ---      ┆ on        ┆ ion       ┆ ---       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ f64       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆           ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
│ 0        ┆ 1         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_for_batch
Out[3]: 
shape: (1, 66)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---       ┆ ---      ┆ rate_000  ┆ rate_001  ┆   ┆ rate_012. ┆ rate_013. ┆ rate_014. ┆ rate_015. │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi │
│           ┆          ┆ f64       ┆ f64       ┆   ┆ ola…      ┆ ola…      ┆ ola…      ┆ ola…      │
│           ┆          ┆           ┆           ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│           ┆          ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0         ┆ 0        ┆ 1000.0    ┆ 1000.0    ┆ … ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_per_real
Out[4]: 
shape: (2, 50)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ rate_000  ┆ rate_001  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
│ 0        ┆ 1         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

Gradient results:

obj_gradient_for_batch
Out[4]: 
shape: (1, 4)
┌───────────┬──────────┬──────────────┬────────────────────┐
│ result_id ┆ batch_id ┆ npv_function ┆ npv_function.total │
│ ---       ┆ ---      ┆ ---          ┆ ---                │
│ i64       ┆ i64      ┆ f64          ┆ f64                │
╞═══════════╪══════════╪══════════════╪════════════════════╡
│ 1         ┆ 1        ┆ -0.0         ┆ -0.0               │
└───────────┴──────────┴──────────────┴────────────────────┘
obj_gradient_perturbations
Out[5]: 
shape: (4, 21)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ realizati ┆ perturbat ┆ … ┆ group_w08 ┆ group_w07 ┆ group_w01 ┆ group_w05 │
│ ---       ┆ ---      ┆ on        ┆ ion       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│           ┆          ┆ i64       ┆ i64       ┆   ┆           ┆           ┆           ┆           │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1         ┆ 1        ┆ 1         ┆ 0         ┆ … ┆ 0.059048  ┆ 0.068553  ┆ 0.062976  ┆ 0.062975  │
│ 1         ┆ 1        ┆ 0         ┆ 1         ┆ … ┆ 0.061752  ┆ 0.069882  ┆ 0.065536  ┆ 0.061705  │
│ 1         ┆ 1        ┆ 0         ┆ 0         ┆ … ┆ 0.057099  ┆ 0.070677  ┆ 0.055112  ┆ 0.071082  │
│ 1         ┆ 1        ┆ 1         ┆ 1         ┆ … ┆ 0.064173  ┆ 0.066663  ┆ 0.068711  ┆ 0.062269  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_for_batch
Out[6]: 
shape: (1, 33)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---      ┆ rate_004  ┆ rate_003  ┆ rate_006  ┆   ┆ rate_014. ┆ rate_005. ┆ rate_012. ┆ rate_010. │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ scaled    ┆ scaled    ┆ scaled    ┆ scaled    │
│          ┆ f64       ┆ f64       ┆ f64       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│          ┆           ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ … ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ 0.0       │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_perturbations
Out[7]: 
shape: (4, 51)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ perturbat ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ ion       ┆ rate_000  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ i64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066225  ┆ 0.067566  ┆ 0.05967   ┆ 0.062201  │
│ 1        ┆ 0         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.069046  ┆ 0.058626  ┆ 0.061307  ┆ 0.059147  │
│ 1        ┆ 1         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066712  ┆ 0.066355  ┆ 0.067376  ┆ 0.059584  │
│ 1        ┆ 1         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.065746  ┆ 0.069017  ┆ 0.060187  ┆ 0.06393   │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 90.72%. Comparing base (721f3c6) to head (f9c476a).

Files with missing lines	Patch %	Lines
src/ert/storage/local_ensemble.py	33.33%	2 Missing ⚠️
src/ert/run_models/everest_run_model.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9161      +/-   ##
==========================================
+ Coverage   90.70%   90.72%   +0.02%     
==========================================
  Files         351      351              
  Lines       21903    21908       +5     
==========================================
+ Hits        19867    19876       +9     
+ Misses       2036     2032       -4     

Flag	Coverage Δ
cli-tests	39.23% <20.00%> (+0.02%)	⬆️
gui-tests	71.73% <20.00%> (-0.04%)	⬇️
performance-tests	49.37% <20.00%> (-0.03%)	⬇️
unit-tests	79.63% <40.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yngve-sk]

(This is temporary/sanity check to be removed. If all tests pass with this, it means the storage is producing the same optimal results as the old. )