Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed test does not store output - or wrong error message #12

Open
JoseAlcerreca opened this issue Jun 17, 2022 · 8 comments
Open

Failed test does not store output - or wrong error message #12

JoseAlcerreca opened this issue Jun 17, 2022 · 8 comments

Comments

@JoseAlcerreca
Copy link
Contributor

java.lang.AssertionError: "MatchesActivityScreenshot" failed to match reference image. 18 pixels differ (8.6805556E-4 %)
Output written to: /storage/emulated/0/Download/screenshots/com.google.samples.apps.nowinandroid.feature.topic.test/MatchesActivityScreenshot.png

However the screenshots directory is empty, This might be a feature and the only bug is in the error message.

Using emulator API 30 and 28.

@sergio-sastre
Copy link

In my case ,the tests are failing.
I cloned this project, and tried to record the screenshot on an emulator running API 31, but got the following message:
Screenshot 2022-06-21 at 17 03 23

Unfortunately I could not get it working on any emulator

@rharter
Copy link
Collaborator

rharter commented Jun 21, 2022

It looks like that's still trying to compare. The flag to records recently changed from recordScreenshots to dropshots.record, so please make sure you're using the right one.

@sergio-sastre
Copy link

It looks like that's still trying to compare. The flag to records recently changed from recordScreenshots to dropshots.record, so please make sure you're using the right one.

It happens when executing ./gradlew :dropshots:connectedAndroidTest -Pdropshots.record. Looks like it ignores the flag when executing the task?

@rharter
Copy link
Collaborator

rharter commented Jun 30, 2022

Ahh, I see what's happening, @sergio-sastre. The Dropshots runtime library in this project doesn't use the dropshots gradle plugin since the plugin adds the dependency, and that would end up running in circles. I'm in the process of getting better documentation about how to update things for that project in #18.

@rharter
Copy link
Collaborator

rharter commented Jun 30, 2022

@JoseAlcerreca for the original issue, are you saying that the /storage/emulated/0/Download/screenshots/ directory on your test device is empty? I've seen a few issues with the Files app on the emulator not refreshing, but that error message is printed after the image is written to device, so I wouldn't expect to see that unless the write completed.

@Areyana
Copy link

Areyana commented Nov 11, 2022

actual issue, screenshots folder still empty

@JayShortway
Copy link
Contributor

I noticed this too. After running a failing test, the output / diff images are:

  1. pulled to the host machine into the configured resultsDir, which is <path_to_your_project>/<module_name>/build/outputs/test-results/ by default, and then
  2. cleared from the device by the ClearScreenshotsTask.

Step 2 is what makes the error message confusing, because if you go to look in that folder it's empty. So either the error message is incorrect or the fact that ClearScreenshotsTask runs is a bug.

Workaround
If you want to leave the diff images on the device, you can work around Step 2 by excluding the ClearScreenshotsTask like so:

./gradlew :<module_name>:connectedDebugAndroidTest --exclude-task :<module_name>:clearDebugAndroidTestScreenshots

(or equivalent for your build variant.)

@renetik
Copy link

renetik commented May 22, 2023

In my case I had to run:
./gradlew task

find name of task:
clearQaAndroidTestScreenshots

and run it like this:
./gradlew connectedCheck --exclude-task clearQaAndroidTestScreenshots

Its definitelly a bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants