Upgrade to a version of CDH which includes the fix jira HBASE-15324:
Fixes are planned for CDH 5.7 and 5.8 but have not been released at the time of this article.
CM ----> HBase ----> Configuration -----> search for "Safety valve" ----> Identify the box for "RegionServer Advanced Configuration Snippet (Safety Valve) for hbase-site.xml"
Append/Insert the below property:
Restart the Region Servers.
Further, if the regions have already gone into thousands, to reduce the number of overall regions, make use the table region normalizer feature (HBASE-14867) introduced with CDH 5.7.0 and later.
For the applicable tables you would need to enable the NORMALIZATION_ENABLED table attribute and then run the normalizer:
> alter 'table_name', 'NORMALIZATION_ENABLED' => true
> normalizer_switch true
The only caveat is that you cannot specify the number of regions and the normalization will occur based on this logic:
- Get all regions of a given table.
- Get avg size S of each region (by total size of store files reported in RegionLoad).
- Seek every single region one by one. If a region R0 is bigger than S * 2, it is kindly requested to split. Thereon evaluate the next region R1.
- Otherwise, if R0 + R1 is smaller than S, R0 and R1 are kindly requested to merge.
- Thereon evaluate the next region R2.
- Otherwise, R1 is evaluated.