Region Server Decommissioned | HBase Master WebU Falsely reports Dead Region Servers

0 votes
0 views
asked Aug 28, 2017 in Hadoop by admin (4,410 points)
SummaryHMaster memory was not cleared causing the messages to appear after decommissioning a regain server. This article shows how to clear the message

Symptoms

After a Region Server is being decommissioned, it still appears in HMaster webui as Dead Region Servers.
The status command in the HBase shell also shows the Region Server as dead.

Applies To

CDH 5
HBase

Cause

There are several possible causes that lead this issue.

1) Obsolete information remains on the HMaster memory.
2) Obsolete WALs remain in the HDFS directory (e.g. /hbase/WALs/...).

 

Instructions

1) Restart HMaster processes in a rolling fashion
Warning: Make sure there are several HMasters deployed in the cluster.
The following steps assume there are two HMasters in the cluster.

  1. Stop active HMaster (for example, HMaster1).
  2. Confirm if the other HMaster (for example, HMaster2) is in active mode.
  3. Start HMaster1 and confirm it is in standby mode.
  4. Stop HMaster2 and confirm HMaster1 is in active mode.
  5. Start HMaster2 and confirm it is in standby mode.

2) Sideline the obsolete WALs
Warning: Before proceed the instruction, report to Cloudera support to make sure the WALs can be sidelined.

(Commands need to be ran as hbase user or principal )

  1. Make a temporary directory (for example, hdfs dfs -mkdir /tmp/old-wals)
  2. Move the obsolete WAL files (for example, hdfs dfs -mv /hbase/WALs/<OBSOLETE-WALs> /tmp/old-wals/)
  3. Restart both HMasters in a rolling fashion

Please log in or register to answer this question.

...