Performance Regression or Improvement: pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs:mean_load_model_latency_milli_secs

[github-actions]

Performance change found in the
test: pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs for the metric: mean_load_model_latency_milli_secs.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: Pytorch image classification on 50k images of size 224 x 224 with resnet 101. Test link -

beam/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy

Line 34 in 42d0a6e

test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',

Test dashboard - http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

timestamp: Sun Jul  2 18:20:40 2023, metric_value: 85789.39
timestamp: Sat Jul  1 18:21:14 2023, metric_value: 92123.00
timestamp: Fri Jun 30 18:22:54 2023, metric_value: 79408.76 <---- Anomaly
timestamp: Wed Jun 28 18:22:35 2023, metric_value: 74666.87
timestamp: Tue Jun 27 18:39:19 2023, metric_value: 73562.27
timestamp: Mon Jun 26 18:30:29 2023, metric_value: 76724.94
timestamp: Sun Jun 25 18:19:10 2023, metric_value: 55545.37
timestamp: Sat Jun 24 18:20:13 2023, metric_value: 64281.98
timestamp: Fri Jun 23 18:21:09 2023, metric_value: 73625.82
timestamp: Thu Jun 22 18:28:34 2023, metric_value: 64631.02
timestamp: Wed Jun 21 18:24:43 2023, metric_value: 70528.39 
timestamp: Tue Jun 20 18:20:47 2023, metric_value: 62520.61
timestamp: Mon Jun 19 18:19:15 2023, metric_value: 68288.21

[AnandInguva]

This could be a regression but lets check the values after July 2nd as well to determine if this is a regression

[github-actions]

Performance change found in the
test: pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs for the metric: mean_load_model_latency_milli_secs.

For more information on how to triage the alerts, please look at
Triage performance alert issues section of the README.

Test description: Pytorch image classification on 50k images of size 224 x 224 with resnet 101. Test link -

beam/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy

Line 34 in 42d0a6e

test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',

Test dashboard - http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

timestamp: Sun Jul 23 18:18:40 2023, metric_value: 72692.07
timestamp: Sat Jul 22 18:18:24 2023, metric_value: 70775.53
timestamp: Fri Jul 21 18:20:07 2023, metric_value: 74081.73
timestamp: Thu Jul 20 18:19:59 2023, metric_value: 76973.05
timestamp: Wed Jul 19 18:19:20 2023, metric_value: 72380.61
timestamp: Tue Jul 18 18:18:53 2023, metric_value: 61089.28
timestamp: Mon Jul 17 18:17:51 2023, metric_value: 70040.71
timestamp: Sun Jul 16 18:18:37 2023, metric_value: 64607.32 <---- Anomaly
timestamp: Sat Jul 15 18:19:57 2023, metric_value: 77917.18
timestamp: Fri Jul 14 18:23:01 2023, metric_value: 112003.24
timestamp: Thu Jul 13 18:20:24 2023, metric_value: 77378.49
timestamp: Wed Jul 12 18:21:02 2023, metric_value: 83614.48
timestamp: Tue Jul 11 18:24:53 2023, metric_value: 82099.07
timestamp: Mon Jul 10 18:18:28 2023, metric_value: 78552.35
timestamp: Sun Jul  9 18:18:06 2023, metric_value: 72604.10
timestamp: Sat Jul  8 18:18:08 2023, metric_value: 70930.31
timestamp: Fri Jul  7 18:18:45 2023, metric_value: 72284.88
timestamp: Thu Jul  6 18:18:47 2023, metric_value: 87905.64

[AnandInguva]

Actual regression caused around June 1st week. It was due to the update of custom container image which increased the size of the image

[damccorm]

This doesn't look concerning from my end, it doesn't look like it was caused by any code changes so I'm going to close. @AnandInguva if you disagree please reopen