Region Split Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split

0 votes
2 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummaryBecause of the jira that introduced Region Split decision Jitter(HBASE-13412), may cause HBase Regions desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split.
Applies To

CDH 5.7.0 and higher
CDH 5.8.0 and higher
CDH 5.9.0

Symptoms

A table XXX has MAX_FILESIZE set to Long.Max = 9223372036854775807:

Table XXX is ENABLED
XXX, {TABLE_ATTRIBUTES => {MAX_FILESIZE => '9223372036854775807', M
ETADATA => {'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.ConstantSize
RegionSplitPolicy'}}

Now, the Jitter that was introduced for split logic decisions via Jira HBASE-13412 turned out to be a problem since  it will overflow when the MAX_FILESIZE is any value close to the the max
value of long. The problematic code line is :

this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() - 0.5D) * jitter);
50% of the time will be negative.

50% of the time RANDOM.nextFloat() will be above 0.5
which will cause the jitter to make desiredMaxFileSize negative

The above issue has been documented by Jira HBASE-15324.

Because of the above Jitter behavior, a Table with just few hundred regions can get automatically split into thousands of regions which is undesirable and cause administrative problems.

Cause

The introduction of the Region Split decision Jitter feature in CDH 5.7  (HBASE-13412)

Instructions

Upgrade to a version of CDH which includes the fix jira HBASE-15324:

CDH 5.9.1

Fixes are planned for CDH 5.7 and 5.8 but have not been released at the time of this article.

Workaround:

CM ----> HBase ----> Configuration -----> search for "Safety valve" ----> Identify the box for "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml"

Append/Insert the below property:

<property><name>hbase.hregion.max.filesize.jitter</name><value>0.0f</value></property>

Save changes

Restart the Region Servers.

##
Further, if the regions have already gone into thousands, to reduce the number of overall regions, make use the table region normalizer feature (HBASE-14867) introduced with CDH 5.7.0 and later.
For the applicable tables you would need to enable the NORMALIZATION_ENABLED table attribute and then run the normalizer:

> alter 'table_name', 'NORMALIZATION_ENABLED' => true
> normalizer_switch true
> normalizer_enabled
true

The only caveat is that you cannot specify the number of regions and the normalization will occur based on this logic:

  1. Get all regions of a given table.
  2. Get avg size S of each region (by total size of store files reported in RegionLoad).
  3. Seek every single region one by one. If a region R0 is bigger than S * 2, it is kindly requested to split. Thereon evaluate the next region R1.
  4. Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested to merge.
  5. Thereon evaluate the next region R2.
  6. Otherwise, R1 is evaluated.

Please log in or register to answer this question.

...