HDFS Quota Management Not Available in CM Under HDFS File Browser After Quota is Set From Command Line.

0 votes
2 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
Summary

Symptoms

After setting the HDFS Quota on a NEW directory from command line, the HDFS File Browser does not show the Quota Management option 
in CM HDFS File Browser.  The following steps demonstrates the issue:

1.  Create a hdfs directory

sudo -u hdfs hdfs dfs -mkdir /tmp/new_directory


2.  Set quota in the new directory:

sudo -u hdfs hdfs dfsadmin -setSpaceQuota 4g   /tmp/new_directory


3.   Check the quota is set:

sudo -u hdfs hdfs dfs -count -q -h -v /tmp/new_directory

        
4.  Login to CM - HDFS - File Browser. Navigate to /tmp/new_directory
        Notice the "Quota Management" is not displayed (on the right side of the page)

Applies To
Cause


This is expected behavior as per documentation:
    http://www.cloudera.com/documentation/enterprise/latest/topics/admin_hdfs_quotas.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7a9b

     Section "Setting HDFS Quotas"  under "Points to note:" :
             "The Reports Manager must index a file or directory before you can set a quota for it."

Reports Manager will re-index HDFS data based on the configuration of "Reports Manager Update Frequency". The default is 1 hour.
Every hour, Reports Manager transfers the last Namenode Fsimage file and re-index that.

When Namenode namesystem modifications occur with regards to quota, they are written to the Edits file. 
This means the last fsimage file may not necessary contain all the modifications done on the filesystem since they may be in edits file. The Checkpoint process takes an fsimage and edit log and compacts them into a new fsimage.
  For more on this refer: http://blog.cloudera.com/blog/2014/03/a-guide-to-checkpointing-in-hadoop/
  
Once the Reports Managers gets this new fsimage file and indexes that, the "Quota Management" option will then be available.

This means the quota management can take up to 2 hours to display because:
  - first the checkpoint has to occur (default for this is also 1 hour or 1000000 transactions),
  - and then after the "Reports Manager Update Frequency" period which is another 1 hour.

Instructions


Wait for Data to be indexed by Reports Manager.

Please log in or register to answer this question.

...