How to Configure Periodic Upload of Running YARN Job Logs

0 votes
3 views
asked Aug 20, 2017 in Hadoop by admin (4,410 points)
SummaryYARN NodeManagers can be configured to upload the container logs for a running job periodically to HDFS.
Symptoms
Container logs for a running job is stored locally on each NodeManager, but not available from one central location.

The following warning in the NM logs confirms this behaviour:
WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: rollingMonitorInterval is set as -1. The log rolling monitoring interval is disabled. The logs will be aggregated after this application is finished.

 
Applies To
  • YARN
  • NodeManager
Cause
Prior to Hadoop-2.6.0, YARN there wasn't any way to get all the container logs for a running job in HDFS. Hadoop-2.6.0 adds the following YARN NodeManager property:
  •   yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds
Instructions
With YARN log aggregation turned on (see [1]), the property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds' causes the NM to upload logs periodically to 'yarn.nodemanager.remote-app-log-dir' (default: /tmp/logs) in HDFS. As otherwise, without the NM configured to periodically uploading the logs a user would need to access the NMs directly to get the logs for the current running job.

To have the container logs for a running job to be uploaded to HDFS every hour (3600 secs), please perform the following:
  1. Click on YARN service in CM.
  2. Click on Configuration tab.
  3. Click on 'NodeManager' under Scope.
  4. Type safety in the search/filter textbox.
  5. In the textbox for 'NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml', add the property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds' and a value of 3600.
    • NOTE: XML configuration might looks like this:

<property>
<name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
<value>3600</value>
</property>

 
  • IMPORTANT: Save changes and restart NodeManagers.

[1] - YARN Log Aggregation Property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds'' : http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

Please log in or register to answer this question.

...