Health Messages Related To File Descriptor Monitoring Thresholds Against Datanodes

0 votes
0 views
asked Aug 20, 2017 in Hadoop by admin (4,410 points)
SummaryCloudera Manager begins showing health messages related to File Descriptor Monitoring Thresholds against some or all of the Datanodes.
Symptoms
Cloudera Manager begins showing health messages related to File Descriptor Monitoring Thresholds against some or all of the Datanodes.

Checking the files opened by the datanode process using the lsof utility shows the majority of these files are against data block meta files, eg:
 
$  lsof -p <datanode pid>
...
java 39634 hdfs 8456r REG 202,161 11 8664463 /data9/dfs/dn/current/BP-1923607587-10.101.1.113-1488993735678/current/finalized/subdir23/subdir252/blk_1075313827_1587886.meta (deleted)
java 39634 hdfs 8457r REG 202,17 11 10232550 /data/dfs/dn/current/BP-1923607587-10.101.1.113-1488993735678/current/finalized/subdir23/subdir252/blk_1075313892_1587888.meta (deleted)
...

The alert is generally display continuously and can only be resolved by a rolling restart on the datanodes.
 
Applies To

CDH5.9.1 and CDH5.5.6

Cause

HDFS-11056 was added to various CDH releases, and as a side effect introduced this problem.

Instructions

This issue can be resolved temporarily by a rolling restart of the Datanodes.

The fix for this issue is in HDFS-11229, and is resolved in the CDH 5.9.2. There is no fix in the CDH 5.5.x branch at this time. Releases on other version minor versions (eg 5.7.x, 5.8.x and 5.10.x) are not affected by this issue.

Please log in or register to answer this question.

...