|With YARN log aggregation turned on (see ), the property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds' causes the NM to upload logs periodically to 'yarn.nodemanager.remote-app-log-dir' (default: /tmp/logs) in HDFS. As otherwise, without the NM configured to periodically uploading the logs a user would need to access the NMs directly to get the logs for the current running job.|
To have the container logs for a running job to be uploaded to HDFS every hour (3600 secs), please perform the following:
- Click on YARN service in CM.
- Click on Configuration tab.
- Click on 'NodeManager' under Scope.
- Type safety in the search/filter textbox.
- In the textbox for 'NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml', add the property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds' and a value of 3600.
- NOTE: XML configuration might looks like this:
- IMPORTANT: Save changes and restart NodeManagers.
 - YARN Log Aggregation Property 'yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds'' : http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml