Alert when jobs fail #569
Alert when jobs fail #569
Conversation
👋 Thanks for opening a pull request!If you are new, please check out the trimmed down summary of our deployment process below:
|
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
|
.deploy |
Deployment Triggered 🚀Razzmatazzz, started a branch deployment to production You can watch the progress here 🔗
|
Deployment Results ✅Razzmatazzz successfully deployed branch |
Working on troubleshooting exactly what causes jobs to fail to start running.
So far, have determined that jobs fail to start because the job is already running. I believe there are two likely culprits for this.
First, it's possible queries to the DB are stalling out and the job waiting for those queries therefore finishes. Currently, I've mocked up and deployed a timeout mechanism to guard against this.
Second, it's possible that the Cloudflare KV put calls are stalling. There is currently no timeout set for them. if the DB timeout fix doesn't work, this will be tried next. Upon further reflection, I think this is the more likely culprit. We occasionally used to get job error warnings about the put requests failing, but we no longer get those. For mysterious Cloudflare reasons, I would guess the requests just remain pending forever rather than throwing errors.