Developer dashboard - Logs

[thSoft]

• Log to HDFS per application
• Separate log per Akka actor / Spark job
• Auto-refresh
• Simultaneous text search

[thSoft]

I enabled YARN log aggregation in a11247d. Logs are aggregated on HDFS in /tmp/logs (by default), but unfortunately an application's logs are only aggregated after the application finished (see http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml). So it does not help us because the actor system and the Spark applications run forever.

[thSoft]

Alternatives for the implementation:

• mount HDFS and configure YARN log location to point to it
• reuse Hadoop's web interface - depends on Make tracking URLs work on EC2 #88
• integrate with Flume - seems overkill

[thSoft]

Log aggregation to HDFS implemented in 7736689.

[thSoft]

Please:

• merge into master
• test on EC2
• document running the dashboard on the wiki

[thSoft]

HDFS-FUSE is unreliable, Hadoop's logging mechanism should be intercepted instead by overriding its log4j.properties to use a HDFS appender.

[thSoft]

Currently, logs are aggregated to HDFS which is mounted properly, but all of them is empty.