Try some more tuning #293
Try some more tuning #293
Conversation
* Log the state of apache processes when we restart so we can continue to understand the problem * Restart slightly less often * Update MaxConnectionsPerChild to 50, which is probably a bit more reasonable than 5 * Update server limit (see below) * Turn off KeepAlive (see below) From scale-infra, explanation on the last two above. ``` This is a confirmed bug in Apache with no configuration workaround: https://bz.apache.org/bugzilla/show_bug.cgi?id=53555 There's a pretty good description of the details, which match our behavior here: https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers One thing that some people seem to have had success with is turning keepalive off. ServerLimit, for the event MPM (what we use), the recommendation is: With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default). Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes. ``` All continuing socallinuxexpo/scale-drupal#79 Signed-off-by: Phil Dibowitz <[email protected]>
|
Interesting finds re the scoreboard bug. Should we move to pre-fork? Have we tried any of these settings out in prod yet? |
| @@ -10,13 +10,22 @@ | |||
| include_recipe 'scale_apache::common' | |||
| include_recipe 'fb_apache' | |||
|
|
|||
| apache_debug_log = '/var/log/apache_status.log' | |||
There was a problem hiding this comment.
Should we have datadog collect this as well?
There was a problem hiding this comment.
I don't htink it's necessary in this case...
| # to 25. So if I read that correctly (and I haven't tuned Apache for a living | ||
| # in a looonnnggg time), we want something like 50+25=75 plus some more for | ||
| # shutting down processes, so like... 80? | ||
| node.default['fb_apache']['extra_configs']['ServerLimit'] = 80 |
There was a problem hiding this comment.
One thing to keep in mind is our httpd/php processes are often at around 100Mb of memory, 256 or higher when speakers are uploading large slide decks, etc. At least one issue we were having this summer was that httpd was often having more prefork workers than we had memory on the system to handle. Reducing the number of allowed workers alleviated it quite a bit.
All this to say, I think these settings are reasonable - but without docs access right now I'm not sure what the resulting impact on memory usage might be. Will try to read them next stop I have.
|
|
||
| # see if this helps with hung processes... | ||
| node.default['fb_apache']['extra_configs']['KeepAlive'] = 'off' | ||
|
|
There was a problem hiding this comment.
seems reasonable based on the list posts you linked.
|
I think this is generally reasonable but I can't speak for the actual tunable settings here |
Part of socallinuxexpo#293 was supposed to drop restarts to hourly, but a rebase dropped that because those lines were changed by Ilan to fix a bug I had introduced. This drops us down to hourly. While we may run into issues, if we do at least now we have logging, and at worst will be down for less than an hour. Signed-off-by: Phil Dibowitz <[email protected]>
Part of #293 was supposed to drop restarts to hourly, but a rebase dropped that because those lines were changed by Ilan to fix a bug I had introduced. This drops us down to hourly. While we may run into issues, if we do at least now we have logging, and at worst will be down for less than an hour. Signed-off-by: Phil Dibowitz <[email protected]>
to understand the problem
reasonable than 5
From scale-infra, explanation on the last two above.
All continuing socallinuxexpo/scale-drupal#79
Signed-off-by: Phil Dibowitz [email protected]