Try some more tuning (#293)

* Log the state of apache processes when we restart so we can continue
  to understand the problem
* Restart slightly less often
* Update MaxConnectionsPerChild to 50, which is probably a bit more
  reasonable than 5
* Update server limit (see below)
* Turn off KeepAlive (see below)

From scale-infra, explanation on the last two above.

```
This is a confirmed bug in Apache with no configuration workaround:

https://bz.apache.org/bugzilla/show_bug.cgi?id=53555

There's a pretty good description of the details, which match our behavior here:

https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers

One thing that some people seem to have had success with is turning keepalive off.

ServerLimit, for the event MPM (what we use), the recommendation is:

With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).

Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes.
```

All continuing socallinuxexpo/scale-drupal#79

Signed-off-by: Phil Dibowitz <[email protected]>

cookbooks/scale_apache/recipes/default.rb

@@ -10,13 +10,22 @@
 include_recipe 'scale_apache::common'
 include_recipe 'fb_apache'
 
+apache_debug_log = '/var/log/apache_status.log'
 if node['hostname'] == 'scale-web2'
   cron 'ugly restarts' do
     minute '*/30'
-    command '/usr/bin/systemctl restart httpd'
+    command "
+       date >> #{apache_debug_log}
+       ps auxwww | grep http >> #{apache_debug_log}
+       /usr/bin/systemctl restart httpd
+    "
   end
 end
 
+node.default['fb_logrotate']['configs']['apache_status'] = {
+  'files' => [apache_debug_log],
+}
+
 common_config = {
   'ServerName' => 'www.socallinuxexpo.org',
   'ServerAdmin' => '[email protected]',
@@ -50,9 +59,22 @@
     'https://www.socallinuxexpo.org/',
 })
 
-node.default['fb_apache']['extra_configs']['MaxConnectionsPerChild'] = 5
+node.default['fb_apache']['extra_configs']['MaxConnectionsPerChild'] = 50
 node.default['fb_apache']['extra_configs']['MaxRequestWorkers'] = 50
 
+# With event, increase this directive if the process number defined by your
+# MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully
+# shutting down processes, is more than 16 server processes (default).
+#
+# Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults
+# to 25. So if I read that correctly (and I haven't tuned Apache for a living
+# in a looonnnggg time), we want something like 50+25=75 plus some more for
+# shutting down processes, so like... 80?
+node.default['fb_apache']['extra_configs']['ServerLimit'] = 80
+
+# see if this helps with hung processes...
+node.default['fb_apache']['extra_configs']['KeepAlive'] = 'off'
+
 base_config = common_config.merge({
   'Alias' => [
     '/past /home/webroot/past',