WildCam Gorongosa - Map Explorer not showing camera images on Chrome 85-ish

[shaunanoordin]

Major Functionality Bug

Original report: https://zooniverse.freshdesk.com/a/tickets/5574

When using Chrome 85 to browse WildCam Gorongosa's /Darien's Map Explorer feature, wildcam camera images no longer appear on the "view selected camera" popup.

EDIT 2020.10.20: the problems ONLY affect WildCam Gorongosa Lab, not WildCam Darien Lab!

macOS10 + Chrome85 has notable issues

Testing

• Open Chrome (tested on 85)

• Go to https://classroom.zooniverse.org/#/wildcam-gorongosa-lab/explorers/map/

• Click on any of the yellow circles

• Expected: the Camera Viewer should show thumbnailed images (or at least any images) from the camera trap

• Actual: no images appear. Viewing the network tab, a whole host of errors appear

• Open Firefox

• Repeat the steps above.

• Actual results match expectations: images show normally.

macOS10 + Firefox81 has no issues

Dev Notes

• For the relevant code, check the CameraViewer component.
• The CameraViewer component uses the Thumbnail component from zooniverse-react-components
• Presumably, this Thumbnail component attempts to... fetch a thumbnail version from the Zooniverse thumbnail service? And defaults to the original image if that borks? Well...
  • thumbnails.zooniverse.org itself seems to have an issue. I'm seeing 415 errors from https://thumbnails.zooniverse.org/320x200/zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG
  • the thumbnail can't fall back to the original image because, goddammit, it seems that Chrome (at least circa version 85) has been strict about programmatically fetching cross-domain resources. See CORS issue with certain TESS Subjects front-end-monorepo#1845

Status

High impact for WildCam-type classrooms, might be medium or high urgency depending on when classrooms start using this.

EDIT 2020.10.20: high impact on WildCam Gorongosa Lab only.

[lcjohnso]

Reposting from Slack: I am not able to reproduce this behavior on my system (Mac OSX 10.14, Chrome 85) or my Chromebook. The non-consistent nature of this behavior seems worrisome and will make the diagnosis difficult.

[camallen]

Interesting - we don't have this bucket in our s3 domain (or i can't find it)...zooniverse-export was that owned by the originating research team?

If so it sounds like it's gone missing and broken the URLs

*   Trying 52.217.81.211...
* TCP_NODELAY set
* Connection failed
* connect to 52.217.81.211 port 443 failed: Operation timed out
* Failed to connect to zooniverse-export.s3-website-us-east-1.amazonaws.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to zooniverse-export.s3-website-us-east-1.amazonaws.com port 443: Operation timed out

Even still our thumbnail server can only work with data in the zooniverse static bucket, https://github.com/zooniverse/thumbnailer/blob/9502ec2eb608acd95a023ef9624c02b28585c456/nginx.conf#L51-L58

I assume the thumbnailer used to fail for these 'external' images and it used to fallback to the original s3 image in this case. Sadly if the upstream s3 data is missing the thumbnailer won't work at all as there is nothing to fall back on.

[lcjohnso]

@camallen -- Again, I'm not reproducing any of the behavior you and @shaunanoordin are seeing. I am currently able see the thumbnailer output and access the underlying linked images. Same as for the OceanEYEs case I mentioned in zooniverse/front-end-monorepo#1845, I have demonstrated clearly this is not due to broken links or missing files.

[eatyourgreens]

@lcjohnso this image URL times out for me. Does it work for you? https://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG

[camallen]

@lcjohnso you are entirely correct. i failed to account for the HTTP scheme in the URL when constructing the curl cmd, i added the https:// scheme component when copying from the original report:

thumbnails.zooniverse.org itself seems to have an issue. I'm seeing 415 errors from https://thumbnails.zooniverse.org/320x200/zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG

specifically

# 200 ok
$ curl -v -I http://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG
# timeout
$ curl -v -I https://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG

As i've stated above, our thumbnailer service will not work with URLs that aren't paths in our s3 static bucket, we do this to ensure our server isn't a universal thumbnails service for the wider internet.

Thus the thumbnail service will return 415 for images it can’t process, e.g. https://thumbnails.zooniverse.org/320x200/zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG

However the Thumbmailer component should default back to the original HTTP URL on error https://github.com/zooniverse/Zooniverse-React-Components/blob/3f6bfc0f1f556a0d68af6e57140ef284403c1c7e/src/components/thumbnail.jsx#L29

In the original report image, i see that the 'failing' request are falling back to the http://zooniverse-export.s3-website-us-east-1.amazonaws.com/* URLs however these report '(failed)'. However in the working example, those fallback images are being loaded from the browser cache (304 / cached).

If we can't reproduce this issue, perhaps there was an intermittent issue with the s3 origin web server where new pages didn't see the data but browsers with those images in the cache were ok?

[camallen]

I managed to reproduce this issue, you have to hit a specific part of the wildcam gorogosa map to hit the impacted subjects.

the plot thickens - thanks to an idea by @eatyourgreens

Chrome appears to be using the secure protocol scheme (see image) even though the original image element src is http - perhaps after the onLoad error it won’t allow the scheme to be ‘downgraded’ to http… 🤔

And the relevant HTML element.

[shaunanoordin]

Investigation Update

I made a terrible mistake in my earlier report - the issue only affects WildCam Gorongosa Lab, not WildCam Darien Lab.

@lcjohnso you mentioned you couldn't replicate the issue; can you please try comparing these two on Chrome?

• 👼 https://classroom.zooniverse.org/#/wildcam-darien-lab/explorers/map/ - Expected on Darien: clicking on an orange circle will load the camera view with images.
• 👿 https://classroom.zooniverse.org/#/wildcam-gorongosa-lab/explorers/map/ - Expected on Gorongosa: clicking on an orange circle will load the camera view with NO images.

Dev Notes

Q: why are 👼 Darien and 👿 Gorongosa different?
A: They're hosting images on two different kinds of AWS S3 buckets.

• 👼 Darien: https://s3.amazonaws.com/hhmi-glp-prod/Gorongosa%20Trail%20Camera/Soberania/Season%201/G01/Set%201/GAM_S01_G01_R01_IMAG0008.JPG - note 1: there's HTTPS, note 2: this domain is accessible to our thumbnailer.
• 👿 Gorongosa: http://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D60_Season%202_Set%201_EK000706.JPG - note 1: there's NO HTTPS, note 2: this domain is inaccessible to our thumbnailer

Q: what's the sequence of errors?
A: this applies only on 👿 WildCam Gorongosa Lab:

1. the <Thumbnail> component asks the Thumbnailer service to load a Gorongosa image, e.g. http://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D60_Season%202_Set%201_EK000706.JPG (HTTP)
2. ☠️ The Thumbnailer service fails with a 415 because the http://zooniverse-export.s3-website-us-east-1.amazonaws.com domain isn't owned by us, or isn't whitelisted, or etc. (See Cam's comments)
3. the <Thumbnail> component falls back to the original URL, which is found on the HTTP://zooniverse-export.s3-website-us-east-1.amazonaws.com domain.
4. Chrome says, "nuh uh", and throws a Mixed Content: The page at '<URL>' was loaded over HTTPS, but requested an insecure element '<URL>'. This request was automatically upgraded to HTTPS, For more information see <URL> warning
5. Chrome AUTOMATICALLY tries to upgrade to HTTPS, meaning it tries to fetch https://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D60_Season%202_Set%201_EK000706.JPG (HTTPS)...
6. ☠️ ...but that URL doesn't exist! The HTTP://zooniverse-export... URLs can't be upgraded to HTTPS.

Solutions?

Possibility 1: add http://zooniverse-export.s3-website-us-east-1.amazonaws.com domain to the thumbnailer service's whitelist?

See earlier comment...

As i've stated above, our thumbnailer service will not work with URLs that aren't paths in our s3 static bucket, we do this to ensure our server isn't a universal thumbnails service for the wider internet.
Thus the thumbnail service will return 415 for images it can’t process, e.g. https://thumbnails.zooniverse.org/320x200/zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D32_Season%202_Set%204_EK002250.JPG

Possibility 2: ask whoever owns the WildWatch Gorongosa's image bucket to enable https?

[lcjohnso]

Hi @shaunanoordin -- I'm now on Chrome 86, but I'm seeing the same behavior: both Darien and Gorongosa are working for me. However, thanks to your description I think I see why we're getting different behavior.

For Darien (works for both of us): thumbnailer requests are returning with 415 (due to disallowed outside-of-Zooniverse origin), and fallback loading of actual image is working because they are HTTPS enabled.

For Gorongosa (works for me, not for you): thumbnailer requests are returning with 415, but fallback loading of HTTP image IS WORKING, but where my browser reports the following "Issue": Mixed Content: The page at 'https://classroom.zooniverse.org/#/wildcam-gorongosa-lab/explorers/map/' was loaded over HTTPS, but requested an insecure image 'http://zooniverse-export.s3-website-us-east-1.amazonaws.com/21484_1000_D60_Season%202_Set%201_EK000706.JPG'. This content should also be served over HTTPS.

So it looks like your Chrome is more strict than mine: my Chrome notes the Mixed Content issue but allows the HTTP image to load.

But either way, to allow the images to load correctly for strict browsers, I agree that one of your two solutions is necessary.