DevTools integration tests are broken on the CI

[kenzieschmoll]

This is blocking submit. Investigating failures and attempting fixes here: #8299

[kenzieschmoll]

I first noticed this failure on the CI testing for #8284.

Thursday 8/29: integration tests were passing: https://github.com/flutter/devtools/actions/runs/10623928422/job/29451334616

which flutter:  /Users/runner/work/devtools/devtools/tool/flutter-sdk/bin/flutter
which dart:  /Users/runner/work/devtools/devtools/tool/flutter-sdk/bin/cache/dart-sdk/bin/dart
...
++ flutter --version
Flutter 3.25.0-1.0.pre.151 • channel [user-branch] • unknown source
Framework • revision 3e4f1cdf49 (2 days ago) • 2024-08-27 09:25:38 -0700
Engine • revision 7d751acc81
Tools • Dart 3.6.0 (build 3.6.0-175.0.dev) • DevTools 2.39.0-dev.15
++ dart --version
Dart SDK version: 3.6.0-175.0.dev (dev) (Fri Aug 23 09:07:06 2024 -0700) on "macos_arm64"

Friday 8/30: integration tests were failing due to a timeout: https://github.com/flutter/devtools/actions/runs/10635908137/job/29486598941\
** Flutter & Dart version info is the same as the passing run **

which flutter:  /Users/runner/work/devtools/devtools/tool/flutter-sdk/bin/flutter
which dart:  /Users/runner/work/devtools/devtools/tool/flutter-sdk/bin/cache/dart-sdk/bin/dart
...
++ flutter --version
Flutter 3.25.0-1.0.pre.151 • channel [user-branch] • unknown source
Framework • revision 3e4f1cdf49 (3 days ago) • 2024-08-27 09:25:38 -0700
Engine • revision 7d751acc81
Tools • Dart 3.6.0 (build 3.6.0-175.0.dev) • DevTools 2.39.0-dev.15
++ dart --version
Dart SDK version: 3.6.0-175.0.dev (dev) (Fri Aug 23 09:07:06 2024 -0700) on "macos_arm64"

[DanTup]

Maybe related, Chrome 128 rolled out around this time, and the two results linked above are different versions:

ChromeDriver - Starting ChromeDriver 127.0.6533.119 (bdef6783a05f0b3f885591e7d2c7b2aec1a89dea-refs/branch-heads/6533@{#1999}) on port 4444

ChromeDriver - Starting ChromeDriver 128.0.6613.84 (606aa55c7d687518d34b55accc5a71ea0bd28727-refs/branch-heads/6613@{#1335}) on port 4444

---

Edit: If I remove --headless, the tests seem to work fine (got the idea from SeleniumHQ/selenium#14438 which may be the same issue.. there was some suggestion it might be a new search engine selection screen, although I don't see that appear in non-headless mode).

---

Edit2: Actually, it turns out they work fine for me locally (on Windows) in any case - I just wasn't waiting long enough the first time I tried to repro. They are very slow though - if I run without headless, I can see it pausing for a very long time (many many seconds) between actions interacting with DevTools. I don't know if that's normal (in which case maybe I'm not reproducing it), or if it's gotten slower in Chrome 128 and that might be the reason for the timeouts.

I'll see if I can easily get back to 127 locally to compare (edit: no luck...).

---

Edit3: I notice that my version of chromedriver is also slightly newer than the one above (ChromeDriver 128.0.6613.119)

---

Edit4: I re-ran the bots on my PR and it got the same newer chromedriver I have locally, but it didn't change the outcome - they still timed out.

[DanTup]

In the tool/ci/bots.sh section just before the test starts (the search on the logs seems to not work very well, it didn't find it until I'd already scrolled down!)

[kenzieschmoll]

I have the same chromedriver version as the bots, and the tests passes locally for me, with and without the --headless flag.

1. dart run integration_test/run_tests.dart --headless
2. dart run integration_test/run_tests.dart

Local version:

ChromeDriver 128.0.6613.119 (6e439cfca4deda5954b0c74cde9b521c03cb31ad-refs/branch-heads/6613@{#1464})

Same version on the CI: https://github.com/flutter/devtools/actions/runs/10724035525/job/29738829153?pr=8299

[DanTup]

Yeah, that's the state for me too (my earlier edit about --headless affecting it was corrected in the next edit - it always works, albeit very slowly).

I am suspicious that it's related to Chrome 128 though. While searching for "chromedriver 128 issue" I found quite a few people having issues like the above mostly blaming --headless changes (apparently there's now a --headless-old and --headless=new?) and a "search engine choice" screen (though I'd expect this to never appear in headless mode!).

I wonder if we can a) get more verbose logs or b) run without --headless on the bots and capture screenshots/video 🤔

[derekxu16]

I recently fixed some DWDS tests that were timing out on CI by passing --disable-gpu to Chrome (https://github.com/dart-lang/webdev/blob/5f30c560dc4e3df341356c43ec1a766ee6b74a7c/dwds/test/fixtures/context.dart#L387-L389).

[kenzieschmoll]

We don't actually start chrome ourselves here - I think flutter driver is responsible for that. When --headless is passed to the DevTools integration tests, it just changes what device we use for the flutter driver command (web-server or chrome): https://github.com/flutter/devtools/blob/master/packages/devtools_shared/lib/src/test/integration_test_runner.dart/#L51

I think this may be where chrome is started from a flutter drive command: https://github.com/flutter/flutter/blob/45ef8f36ff417550b1bd80dfb2add356be1bb8d1/packages/flutter_tools/lib/src/drive/web_driver_service.dart#L306

@yjbanov or @christopherfujino can you confirm? We are not passing as many flags as we are for the headless option in the ChromiumLanucher class from flutter tools: https://github.com/flutter/flutter/blob/45ef8f36ff417550b1bd80dfb2add356be1bb8d1/packages/flutter_tools/lib/src/web/chrome.dart#L225-L231

[kenzieschmoll]

I tried adding --disable-gpu via the --web-browser-flag option on the flutter drive command, and this did not work: https://github.com/flutter/devtools/actions/runs/10726828101/job/29747781465