-
Notifications
You must be signed in to change notification settings - Fork 374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Crash tracker spawning long-running child processes? #3954
Comments
Hi @luc-financeit! Thanks for reporting this and sorry you've run into this issue. You are absolutely correct, the crashtracker starts an external process that sends out crash report: if Ruby VM crashes, it is unable to send the report to us, so we need an additional process to send out these reports. I think there is really no other way of doing this. Given that this is not critical for gem datadog or datadog-ci to work correctly, disabling crashtracker via |
Thanks for reporting @luc-financeit ! And to @anmarchenko for a quick response and work around. I'm sharing this with our team, see if we can diagnose/plan on what to do about this. We will share updates here as we have them. |
👋 @luc-financeit Thanks for reporting. I can reproduce it with require "bundler/inline"
gemfile { gem "datadog", "2.3.0", source: "https://rubygems.org" }
require "datadog"
Datadog.configure {}
Process.waitall Indeed, For example: pid = Process.fork do
puts Process.pid
end
Process.wait(pid) We will be looking into this issue and I suggest circumvent this by setting |
Crashtracking has been identified as causing issues with `ECHILD` and `waitpid`, and possibly anything involving `SIGCHLD` with obscure failure modes of hard to anticipate consequences. Disable by default as a cautionary measure. Mitigates #3954
Crashtracking has been identified as causing issues with `ECHILD` and `waitpid`, and possibly anything involving `SIGCHLD` with obscure failure modes of hard to anticipate consequences. Disable by default as a cautionary measure. Mitigates #3954
It is convenient to use but does make the assumption that there aren't any processes spawned in the background by dependent libraries or the core runtime. A more robust solution is to keep track of pids for processes spawned by the application and wait for them explicitly using one of the other wait* methods. |
Sure, I agree that our code doing "Don't use |
Hey @luc-financeit : we're going to disable this crash tracker feature by default until we can test and deploy a suitable alternative. Look for this in our next release. Sorry for the frustration and thanks for your patience! |
Crashtracking has been identified as causing issues with `ECHILD` and `waitpid`, and possibly anything involving `SIGCHLD` with obscure failure modes of hard to anticipate consequences. Disable by default as a cautionary measure. Mitigates #3954
Current behaviour
Upgrading the
datadog
gem from 2.2.0 to 2.3.0 causes a spec in our test suite to hang. The code being tested does something likeLooking at
ps
output, there's alibdatadog-crashtracking-receiver
child process which ourwaitall
is hung up on — manually killing the crashtracker process, or disabling the crashtracker entirely (viaDD_CRASHTRACKING_ENABLED
) allows the test to complete successfully.Expected behaviour
No hang.
Steps to reproduce
I'm afraid I don't have a stand-alone reproduction — the actual code is significantly more complicated, and I haven't tried slimming it down to see if there's additional triggers. I can dig in to it more if the above isn't enough to go on.
Environment
Datadog.configure ...
): See gist.The text was updated successfully, but these errors were encountered: