Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stitchdata Integration 502 Errors #6

Open
katered8 opened this issue Mar 3, 2020 · 5 comments
Open

Stitchdata Integration 502 Errors #6

katered8 opened this issue Mar 3, 2020 · 5 comments

Comments

@katered8
Copy link

katered8 commented Mar 3, 2020

Hey! I'm using the stitchdata yotpo integration, and I've been getting these, or similar, 502 errors once or twice a day since we created the connection. It looks like all/most of the data is coming through properly, but I'm not sure why this error is occurring so frequently or if there's something I should be trying fix on my end?

502 Server Error: Bad Gateway for url: https://api.yotpo.com/v1/widget/......................................../products/<productid>/reviews.json?per_page=100&page=3&direction=asc&sort=date&sort=time

502 Server Error: Bad Gateway for url: https://api.yotpo.com/v1/widget/......................................../products/<productid>/reviews.json?direction=asc&sort=date&sort=time&per_page=100&page=7

@florenciacomuzzi
Copy link

florenciacomuzzi commented Jan 14, 2021

@katered8 The error Bad Request for URL is being caused by the fact that the URL being called is incorrect (it is missing the APP KEY and instead contains a series of dots)

When you test it directly it will return the same 404 error all the time: https://api.yotpo.com/v1/widget/......................................../products/2677202976853/reviews.json?direction=asc&per_page=100&sort=date&sort=time&page=83

The correct URL must be https://api.yotpo.com/v1/widget/{app_key}/products/{utoken}/reviews.json + any additional parameters that are in the API documentation.

@wesseljt
Copy link

I also have this problem (using stichdata platform) - I tried putting in my API info again - and this didn't seem to fix it. Any other recommendations?

@florenciacomuzzi
Copy link

@wesseljt After testing the tap locally, I am quite convinced that the ellipses are actually a result of Stitchdata obfuscating secrets in the error logs!

During local testing of the tap, I found that requests to the API often returned 502 Bad Gateway and 400 Client Error only to successfully return data when I manually re-sent the requests. The tap retries requests, using exponential backoff pattern, when 429, 503, and 504 HTTP error codes are received. This pattern is best practice for handling API rate limiting. No such handling occurs for other error codes; an exception is immediately raised for other error codes.

Long running processes increase the possibility of failure and the Yotpo tap is long running (several hours) even when the data date lookback range is relatively small, i.e. 30 days. The tap paginates through all records of each object and filters on the client side. After adding retry handling for 502 and 400 errors, the tap eventually did fail to create a new connection because authentication only occurs once at the beginning of a tap run. Yotpo docs state clearly that tokens expire when a connection is closed.

To make this tap more robust, the following changes are needed:

  • Tap retries 502 and 400 errors several times. Right now, only requests that receive 429, 503, and 504 errors are retried.
  • Tap requests a new token whenever a new connection is made. Right now, a token is requested only once at the beginning of a tap run. Yotpo docs state clearly that tokens expire when a connection is closed.
  • Tap only requests data for dates within lookback range. Right now, the tap paginates through all records of each object and filters on the client side.

@florenciacomuzzi
Copy link

@drewbanin @dmosorast @jacobrobertbaca @KAllan357 @cmerrick can I put in a PR with the changes above?

@dave-connors-3
Copy link

@flopotok I saw you opened and closed a PR - I am currently experiencing this issue - were those changes rejected for some reason?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants