DataNode Fails To Start "Unable to obtain password from user"

0 votes
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummaryOn a cluster with Kerberos enabled a service or role may fail to start if the encryption type used is AES256 and the wrong JCE Policy files are in place


After adding a new node to the cluster and adding a Datanode role to it the Datanode failed to start.
The following was seen in the Role Log,

Exception in secureMain Login failure for hdfs/ from keytab hdfs.keytab: Unable to obtain password from user
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(

Applies To

CM all versions, with Kerberos AES256 encryption keys only.


The encryption type used were only AES256, there were no fallback encryption types.

Comparing the JCE Policy files from a working node the new non-working node it was noticed that the file sizes were different.

For example, compare the output of the following on the working node and non-working node.

# ls -l $JAVA_HOME/jre/lib/security/*jar
-rw-r--r-- 1 root root 3035 Aug  2  2016 /usr/java/jdk1.8.0_60/jre/lib/security/local_policy.jar
-rw-r--r-- 1 root root 3023 Aug  2  2016 /usr/java/jdk1.8.0_60/jre/lib/security/US_export_policy.jar

  1. Confirm the version of Java being used by the node.
    # ps -ef | grep java  
  2. From another node were things are working correctly copy the JCE Policy files from that JDK to this host.
    # scp local_policy.jar root@<node>:$JAVA_HOME/jre/lib/security/
    # scp US_export_policy.jar root@:<node>:$JAVA_HOME/jre/lib/security/
  3. ​Restart the DataNode or service in question.

Please log in or register to answer this question.