HiveServer2 Crash with javax.jdo.JDODataStoreException: No current connection.

0 votes
0 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
Summary
Symptoms

HiveServer2 crashes with below error,

2017-02-25 05:56:02,773 ERROR org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager: [Thread[Thread-25,5,main]]: ExpiredTokenRemover thread received unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: javax.jdo.JDODataStoreException: No current connection.
NestedThrowables:
java.sql.SQLNonTransientConnectionException: No current connection.
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: javax.jdo.JDODataStoreException: No current connection.
NestedThrowables:
java.sql.SQLNonTransientConnectionException: No current connection.
at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:155)
at org.apache.hadoop.hive.thrift.DBTokenStore.getToken(DBTokenStore.java:88)
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.removeExpiredTokens(TokenStoreDelegationTokenSecretManager.java:256)
at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:319)
at java.lang.Thread.run(Thread.java:745)
...
Caused by: java.sql.SQLNonTransientConnectionException: No current connection.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.noCurrentConnection(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.checkIfClosed(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.isReadOnly(Unknown Source)


 

Applies To

HiveServer2

Cause

When HMS is configured to use DBTokenStore for delegation tokens, Cloudera Manager also applies the same configuration for HS2.  Since HS2 does not support DBTokenStore (HIVE-12270 is not back ported into CDH), current behavior is to start a local HiveMetaStore handler to connect to an embedded Derby Database for storing delegation tokens. Thus, any issues from Derby will crash HS2. 

There is an internal JIRA (OPSAPS-32305) to track the CM issue which generates DBTokenStore configuration for HS2 incorrectly.

The fix of HIVE-13090 prevents crashing HS2 even when Derby throws exceptions. However it stops the DelegationToken removing thread.  HIVE-15090 recovered ExpireTokenRemoval threads, thus it's the final fix for this issue.

https://issues.apache.org/jira/browse/HIVE-13090
https://issues.apache.org/jira/browse/HIVE-15090
 

Instructions

Both fix for HIVE-13090 and HIVE-15090 are in CDH5.7.6 , so we can upgrade to this version to avoid this issue.

Alternatively, we have below workarounds if upgrade cannot be scheduled at this time.

As far as you don't use load balancer before the HS2 instances, we could change the token store to MemoryTokenStore just for HiveServer2, but leave DBTokenStore for HMS.

To achieve this, please do the following:
1) Navigate to CM > Hive > Configuration > Hive MetaStore Server > Hive Metastore Delegation Token Store
2) Select DBTokenStore (should already be done)
3) Navigate to CM > Hive > Configuration > HiveServer2 > HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml
4) enter:

<property>
<name>hive.cluster.delegation.token.store.class</name>
<value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value>
<final>true</final>
</property>

5) Save
6) Restart Hive.

In case of Hive Available load balanced HS2 setup, we could change the token store to  org.apache.hadoop.hive.thrift.ZooKeeperTokenStore instead.

Note: Cloudera is testing the use of org.apache.hadoop.hive.thrift.ZooKeeperTokenStore in the use of Hive Server 2 load balanced environments. If you experience problems with this method, Cloudera suggests that you not target the load balancer in the jdbc-url for the Hive2 action, but target a specific HiveServer2 instance behind the load balancer instead

Please log in or register to answer this question.

...