Run tests also with Selenium 3.x #354

mvorisek · 2022-04-18T07:44:52Z

There were 3 errored and 2 failing tests, see post below for analysis and how they were fixed.

codecov · 2022-04-18T07:46:33Z

Codecov Report

Merging #354 (e84a258) into master (5d15043) will decrease coverage by 2.28%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #354      +/-   ##
============================================
- Coverage     90.02%   87.73%   -2.29%     
+ Complexity      150      149       -1     
============================================
  Files             1        1              
  Lines           421      367      -54     
============================================
- Hits            379      322      -57     
- Misses           42       45       +3

Impacted Files	Coverage Δ
src/Selenium2Driver.php	`87.73% <100.00%> (-2.29%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

.github/workflows/tests.yml

mvorisek · 2022-10-23T14:06:26Z

1) Behat\Mink\Tests\Driver\Basic\IFrameTest::testIFrame
WebDriver\Exception\InvalidArgument: data did not match any variant of untagged enum FrameId at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:286
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:426
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:321
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Basic/IFrameTest.php:17

test must be fixed, only integer indexes are accepted, when string id is passed, the following exception is shown:

name key tested, but it is not supported, not handle, nor element itself (might be supported when serialized correctly, but the latest driver does not support it)

WebDriver\Exception\InvalidArgument: invalid argument: 'id' can not be string
(Session info: chrome=84.0.4147.105)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DX6B48F375QD', ip: '10.8.136.130', os.name: 'Windows 10',
os.arch: 'amd64', os.version: '10.0', java.version: '12.0.2'

2) Behat\Mink\Tests\Driver\Js\ChangeEventTest::testSetValueChangeEvent with data set "file" ('the-file', '/home/runner/work/MinkSeleniu...e1.txt', '/home/runner/work/MinkSeleniu...e2.txt')
WebDriver\Exception\InvalidArgument: File not found: /home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/../../web-fixtures/file1.txt

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:234
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Container.php:241
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:679
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Element/NodeElement.php:105
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/ChangeEventTest.php:50

the uploaded paths are newly validated if exist and canonical, of course from browser, thus we must use some paths available within the browser container

3) Behat\Mink\Tests\Driver\Js\WindowTest::testWindow
WebDriver\Exception\InvalidArgument: missing field `handle` at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:269
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:418
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:311
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/WindowTest.php:17

Selenium 3.x + Chrome is working, Firefox is broken - and only handle must be passed, main window handle must be retrived from webdriver first, null/empty is not supported, nor name is supported (/w or /wo handle)

impossible to fix https://stackoverflow.com/questions/25760357/get-title-of-window-without-switching-in-selenium-webdriver, developer must pass window handle instead of window name (which can also be duplicate)

(only) handle support fixed in instaclick/php-webdriver#119

null handle fixed by retriving all handles and using the 1st one (the results is ordered, so the 1st one should always be main windows handle)

4) Behat\Mink\Tests\Driver\Custom\TimeoutTest::testInvalidTimeoutSettingThrowsException
Failed asserting that exception of type "\Behat\Mink\Exception\DriverException" is thrown.

/w Selenium 2x the following exception is thrown:

Behat\Mink\Exception\DriverException: Error setting timeout: Unknown wait type: invalid

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:336
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:322
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/tests/Custom/TimeoutTest.php:33

Caused by
WebDriver\Exception\UnknownError: Unknown wait type: invalid

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:319
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:333
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:322
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/tests/Custom/TimeoutTest.php:33

/w Selenium 3.x no exception is thrown with Firefox, in Chrome, exception is thrown, but a different one - WebDriver\Exception\InvalidArgument, test depending on the webdriver impl. removed

5) Behat\Mink\Tests\Driver\Js\EventsTest::testRightClick
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'right clicked'
+'single clicked'

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Js/EventsTest.php:45

natively via webdriver it seems impossible/broken, contextClick method is also not supported, can be probably fixed by using syn.js #358 for right click - https://github.com/bitovi/syn/blob/v0.15.0/test/mouse_button_test.js#L439 (non upgraded syn.js does not trigger any click $this->trigger($xpath, 'rightClick'); (but 'click' is delivered correctly via syn.js))

$this->wdSession->buttondown(array('button' => 2));
$this->wdSession->buttonup(array('button' => 2));

is not emitting right click either

the only fix seems to migrate to new webdriver actions - https://stackoverflow.com/questions/2171589/how-can-you-do-right-click-using-selenium/19123275#19123275

aik099 · 2022-11-18T10:48:48Z

1) Behat\Mink\Tests\Driver\Basic\IFrameTest::testIFrame
WebDriver\Exception\InvalidArgument: data did not match any variant of untagged enum FrameId at line 3 column 1

/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:198
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:167
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/instaclick/php-webdriver/lib/WebDriver/Session.php:286
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/src/Selenium2Driver.php:426
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/behat/mink/src/Session.php:321
/home/runner/work/MinkSelenium2Driver/MinkSelenium2Driver/vendor/mink/driver-testsuite/tests/Basic/IFrameTest.php:17

test must be fixed, only integer indexes are accepted, when string id is passed, the following exception is shown:

name key tested, but it is not supported, not handle, nor element itself (might be supported when serialized correctly, but the latest driver does not support it)

WebDriver\Exception\InvalidArgument: invalid argument: 'id' can not be string
(Session info: chrome=84.0.4147.105)
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:25:53'
System info: host: 'DX6B48F375QD', ip: '10.8.136.130', os.name: 'Windows 10',
os.arch: 'amd64', os.version: '10.0', java.version: '12.0.2'

Does Selenium 3 still follow the JsonWire protocol, that allows id parameter to be a string (see https://www.selenium.dev/documentation/legacy/json_wire_protocol/#sessionsessionidframe)?

Otherwise, an upgrade Selenium is only removing stuff instead of adding it :( .

Offtopic:
Some time ago, in the #298 , we're discussing another driver creation (like WebDriverDriver), that will use https://github.com/php-webdriver/php-webdriver (previously developed at Facebook) internally instead of https://github.com/instaclick/php-webdriver, which supports Selenium 2 only (as per the repo's README.md file).

There are also unmerged/closed PRs with attempts to use https://github.com/php-webdriver/php-webdriver in this driver instead.

Maybe we should move in that direction, @stof ?

aik099 approved these changes Apr 27, 2022

View reviewed changes

stof reviewed Apr 27, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

mvorisek force-pushed the selenium3_ci branch from e101450 to 6580799 Compare April 27, 2022 18:07

mvorisek force-pushed the selenium3_ci branch from 6580799 to d358fb1 Compare October 16, 2022 16:54

mvorisek marked this pull request as draft October 17, 2022 08:58

Run tests also with Selenium 3.x

87e6203

mvorisek force-pushed the selenium3_ci branch from d358fb1 to 87e6203 Compare October 23, 2022 12:39

This was referenced Oct 23, 2022

Test drop onto itself and hide when dragging minkphp/driver-testsuite#59

Merged

No WebDriver\Session::moveto() call when dragging onto itself #359

Open

mvorisek force-pushed the selenium3_ci branch from caf6a13 to 99f3677 Compare October 23, 2022 14:00

Fix switchToWindow for Selenium 3.x and Firefox (Chrome is always ok)

3d1c8d1

mvorisek force-pushed the selenium3_ci branch from 99f3677 to 3d1c8d1 Compare October 23, 2022 14:59

Drop invalid timeout test depending on specific WD impl

8c09245

mvorisek mentioned this pull request Oct 23, 2022

Fix tests for Selenium 3.x minkphp/driver-testsuite#60

Closed

Update GH Actions images to fix node 12 warnings

e84a258

This was referenced Feb 24, 2024

Also test on Selenium 3 #383

Open

The "EventsTest::testRightClick" test fails on Selenium 3.x #381

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests also with Selenium 3.x #354

Run tests also with Selenium 3.x #354

mvorisek commented Apr 18, 2022 •

edited

Loading

codecov bot commented Apr 18, 2022 •

edited

Loading

mvorisek commented Oct 23, 2022 •

edited

Loading

aik099 commented Nov 18, 2022 •

edited

Loading

Run tests also with Selenium 3.x #354

Are you sure you want to change the base?

Run tests also with Selenium 3.x #354

Conversation

mvorisek commented Apr 18, 2022 • edited Loading

codecov bot commented Apr 18, 2022 • edited Loading

Codecov Report

mvorisek commented Oct 23, 2022 • edited Loading

aik099 commented Nov 18, 2022 • edited Loading

mvorisek commented Apr 18, 2022 •

edited

Loading

codecov bot commented Apr 18, 2022 •

edited

Loading

mvorisek commented Oct 23, 2022 •

edited

Loading

aik099 commented Nov 18, 2022 •

edited

Loading