DataNode Fails To Start "Unable to obtain password from user"

0 votes
0 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummaryOn a cluster with Kerberos enabled a service or role may fail to start if the encryption type used is AES256 and the wrong JCE Policy files are in place

Symptoms


After adding a new node to the cluster and adding a Datanode role to it the Datanode failed to start.
The following was seen in the Role Log,
 

Exception in secureMain
java.io.IOException: Login failure for hdfs/host008.example.com@EXAMPLE.COM from keytab hdfs.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:275)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:239)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2323)

Applies To

CM all versions, with Kerberos AES256 encryption keys only.

Cause

The encryption type used were only AES256, there were no fallback encryption types.

Comparing the JCE Policy files from a working node the new non-working node it was noticed that the file sizes were different.

For example, compare the output of the following on the working node and non-working node.

# ls -l $JAVA_HOME/jre/lib/security/*jar
-rw-r--r-- 1 root root 3035 Aug  2  2016 /usr/java/jdk1.8.0_60/jre/lib/security/local_policy.jar
-rw-r--r-- 1 root root 3023 Aug  2  2016 /usr/java/jdk1.8.0_60/jre/lib/security/US_export_policy.jar

Instructions
  1. Confirm the version of Java being used by the node.
    # ps -ef | grep java  
  2. From another node were things are working correctly copy the JCE Policy files from that JDK to this host.
    # scp local_policy.jar root@<node>:$JAVA_HOME/jre/lib/security/
    # scp US_export_policy.jar root@:<node>:$JAVA_HOME/jre/lib/security/
    
    
  3. ​Restart the DataNode or service in question.

Please log in or register to answer this question.

...