HBase Upper and Lower Memstore limits changed, which can lead to write performance issues

0 votes
10 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummaryThe HBase Memstore Upper and Lower Memstore limits parameter changes were made to Cloudera Manager. This can lead to more frequent flushes of the Memstore when Cloudera Manager 5.8 or higher is being used to manage the cluster. This leads to the Memstore not being fully utilized.

Symptoms

Increased Memstore flushing and minor compactions, leading to potential write performance issues with HBase. You may observe hundreds of Memstore flushes an hour on a given Region Server which can also lead to Slow sync cost when flushing the WAL.

2017-02-25 22:19:50,059  INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~6.88 KB/7040, currentsize=0 B/0 for region settings,,1432846886499.48a445b3d9d1541c3b337033513132b0. in 376ms, sequenceid=749355, compaction requested=false

2017-02-25 22:20:14,759  INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog: Slow sync cost: 190 ms, current pipeline: [DatanodeInfoWithStorage[10.124.54.105:50010,DS-b67b3ad4-8a2f-4c51-b06e-7ab9543c31d6,DISK], 
DatanodeInfoWithStorage[10.124.54.124:50010,DS-6bd87b56-c0fc-4c00-9f6a-4d4e0850c44c,DISK], DatanodeInfoWithStorage[10.124.54.114:50010,DS-491079e5-a33e-4e7f-9064-72d06a5e0bb8,DISK]]

2017-02-25 22:21:51,609  INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Waited 90372ms on a compaction to clean up 'too many store files'; 
waited long enough... proceeding with flush of standard_log,80520_80520-20141116-BCK-151125015845840390.TXT_BCK_15715241_9219780098475657073-5,1488082529301.7715e94fa4cf242ef78f860ee87edee8.
Applies To

Cloudera Manager 5.8 and higher
CDH 5.8 and higher

Cause

The HBase Memstore parameter changes that were made to Cloudera Manager 5.8 and higher.  Starting with HBase 1.x (CDH 5.4) and higher there was a change to the Upper/LowerLimit Memstore parameters as part of HBASE-10392. Where hbase.regionserver.global.memstore.upperLimit changed to hbase.regionserver.global.memstore.size and hbase.regionserver.global.memstore.lowerLimit changed to hbase.regionserver.global.memstore.size.lower.limit.  In additional to the parameters changing the behavior of the new LowerLimit parameter also changed.

Old deprecated parameters and their definition:

hbase.regionserver.global.memstore.upperLimit = The upper limit amount of heap allocated to the memstore as a percentage.
hbase.regionserver.global.memstore.lowerLimit = The lower limit amount of heap allocated to the memstore as a percentage which is used to determine when flushes should start occurring.

New parameters and their new definition:

hbase.regionserver.global.memstore.size = The upper limit amount of heap allocated to the memstore as a percentage.
hbase.regionserver.global.memstore.size.lower.limit = The lower limit amount of heap allocated to the memstore as a percentage of the global memstore size which is used to determine when flushes should start occurring.

When these parameters were updated in Cloudera Manager 5.8 which manages CDH 5.8 and higher, the hbase.regionserver.global.memstore.size.lower.limit percentage value in Cloudera Manager was not updated from 0.38 to 0.95 so the HBase Memstore can potentially flush more frequently due to the Memstore not being fully utilized. This can lead to write performance issues.

Instructions

WARNING: The following procedure only applies if running Cloudera Manager 5.8 or higher and CDH 5.8 or higher. Do not make this change if running a lower version of Cloudera Manager and CDH.

  1.  Search for memstore under the HBase Service Configuration.
  2.  User-added image
  3. Change Low Watermark for Memstore Flush (i.e. hbase.regionserver.global.memstore.size.lower.limit) to 0.9.
  4. Save the changes.
  5. Restart the HBase Service. Rolling Restart is fine.
  6. Redeploy the client configuration, if needed.

Please log in or register to answer this question.

...