fix: gracefully handle missing test cases between two apps #40689

kirillzyusko · 2024-04-22T12:49:29Z

Details

Sometimes we may add new test case or rename a test. In this case the e2e pipeline will be broken, because we'll try to run not existing tests on baseline app version (which leads to timeout error now, because test is not getting executed and after 300s app gets killed by our test runner).

In this PR I fixed this problem by gracefully handling the case - when we receive non critical error (currently this is the only one error indicating that test is not found) we'll call forceTestCompletion on a server side to clearly indicate that test finished its execution and we can go further.

After these change in compareResults we'll have next test data:

Delta VS Main (report)

  {
  main: {
    'App start nativeLaunch': [
      51, 45, 48, 43,
      54, 45, 42, 53
    ],
    'App start nativeLaunchEnd_To_appCreationStart': [
      61, 38, 48, 41,
      37, 50, 36, 39
    ],
    'App start appCreation': [
      73, 56, 62, 66,
      51, 62, 68, 67
    ],
    'App start runJsBundle': [
      359, 309, 313,
      345, 313, 350,
      341, 321
    ],
    'App start appCreationEnd_To_contentAppeared': [
      455, 404, 422,
      445, 416, 451,
      438, 425
    ],
    'App start contentAppeared_To_screenTTI': [
      926.1352340001613,
      808.8870689999312,
      882.0101989991963,
      824.3700229991227,
      750.9790739994496,
      974.3169669993222,
      760.4869720004499,
      911.4533629994839
    ],
    'App start TTI': [
      1566.1352340001613,
      1351.8870689999312,
      1462.0101989991963,
      1419.3700229991227,
      1308.9790739994496,
      1582.3169669993222,
      1344.48697200045,
      1495.453362999484
    ],
    'App start regularAppStart': [
      0.018432000651955605,
      0.01855500042438507,
      0.016927000135183334,
      0.0174150001257658,
      0.018880000337958336,
      0.019245998933911324,
      0.020711999386548996,
      0.021362001076340675
    ],
    'Chat opening': [
      50.53926599957049,
      18.8972570002079,
      19.913980999961495,
      26.739176001399755,
      21.597574999555945,
      26.9226890001446,
      24.792521001771092,
      22.543579000979662,
      22.014240998774767
    ],
    'Chat TTI': [
      4417.695436000824,
      253.23368300125003,
      302.44303400069475,
      336.76261400058866,
      362.21338000148535,
      368.44978800043464,
      405.98950199969113,
      387.2469490002841,
      391.062663000077
    ]
  }
}
{
  delta: {
    'App start nativeLaunch': [
      40, 48, 47, 46,
      55, 50, 48, 49
    ],
    'App start nativeLaunchEnd_To_appCreationStart': [
      50, 43, 46, 45,
      55, 41, 38, 44
    ],
    'App start contentAppeared_To_screenTTI': [
      849.8249329999089,
      840.6559500005096,
      846.7410339992493,
      847.9091679994017,
      858.9504289999604,
      841.7195960003883,
      907.3378030005842,
      1017.2439729999751
    ],
    'App start appCreation': [
      58, 66, 55, 58,
      56, 63, 64, 60
    ],
    'App start runJsBundle': [
      308, 345, 350,
      307, 329, 357,
      341, 343
    ],
    'App start appCreationEnd_To_contentAppeared': [
      430, 452, 449,
      411, 427, 457,
      446, 449
    ],
    'App start TTI': [
      1427.824932999909,
      1449.6559500005096,
      1443.7410339992493,
      1407.9091679994017,
      1451.9504289999604,
      1452.7195960003883,
      1503.3378030005842,
      1619.243972999975
    ],
    'App start regularAppStart': [
      0.018636999651789665,
      0.017090000212192535,
      0.01757900044322014,
      0.017130998894572258,
      0.017740000039339066,
      0.020262999460101128,
      0.017211999744176865,
      0.020183000713586807
    ],
    'Open Chat Finder Page TTI': [
      788.0697019994259,
      1149.6198730003089,
      1014.984579000622,
      1017.9701340012252,
      1177.9677740000188,
      1162.9797769989818,
      1397.7467859983444,
      1065.965983999893,
      1155.0807300005108
    ],
    'Chat opening': [
      58.11433899961412,
      28.28991699963808,
      19.28430199995637,
      21.84948700107634,
      25.501506000757217,
      22.457032000645995,
      27.008464001119137,
      22.900961000472307,
      21.812541000545025
    ],
    'Chat TTI': [
      3671.6196299996227,
      308.547729998827,
      299.43794699944556,
      308.8984789997339,
      348.2585049998015,
      296.8203129991889,
      332.655232001096,
      352.6728520002216,
      322.54195199906826
    ]
  }
}

Fixed Issues

$ #40544
PROPOSAL: N/A

Tests

run e2e pipeline on CI (no new release is needed, it should work in current setup);

Offline tests

N/A

QA Steps

QA can skip testing this PR because it's e2e stuff.