Hive Queries Fail | HiveServer2 Stops Working | Sentry enabled | Expired TGT

0 votes
5 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummarySENTRY-1265 if Sentry fails to renew its Ticket Granting Ticket (TGT), it will go into a failed state when it can't accept any requests from HS2 or HMS. 
TGT is only needed for renewal, however it is running in the renewThread (Sentry is never a Kerberos Client to other Kerberos Services).

Symptoms

All queries fail in HiveServer2 with error message:

Error while compiling statement: FAILED: InvocationTargetException null

HiveServer2 and Hive MetaStore logs indicate Sentry communication issues.

HiveServer2:

2016-07-22 09:39:48,988 ERROR org.apache.hadoop.hive.ql.Driver: FAILED: InvocationTargetException null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor14.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getAuthProvider(HiveAuthzBinding.java:205)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:87)
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:79)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.<init>(HiveAuthzBindingHook.java:97)
[...]
Caused by: sentry.org.apache.thrift.transport.TTransportException: Peer indicated failure: Failure to initialize security context
at sentry.org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at sentry.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

Hive MetaStore:

2016-07-22 09:31:12,353 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive/_HOST@REALM.COM (auth:KERBEROS) cause:sentry.org.apache.thrift.transport.TTransportException: Peer indicated failure: Failure to initialize security context
2016-07-22 09:31:12,354 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: MetaException(message:Failed to connect to Sentry service null)
[...]
2016-07-22 14:22:53,360 ERROR org.apache.sentry.hdfs.MetastorePlugin: Error talking to Sentry HDFS Service !!
java.lang.reflect.UndeclaredThrowableException
[...]
Caused by: sentry.org.apache.thrift.transport.TTransportException: Peer indicated failure: Failure to initialize security context
at sentry.org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at sentry.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

Sentry Logs:

Corresponding stack traces in Sentry logs, repeated several times:

2016-07-21 23:36:30,439 ERROR sentry.org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
[...]
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
[...]
2016-07-21 23:36:30,439 ERROR sentry.org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: sentry.org.apache.thrift.transport.TTransportException: Failure to initialize security context

Applies To
Cause

Due to a known issue, if Sentry fails to renew its Ticket Granting Ticket (TGT), it will go into a failed state when it can't accept any requests from HS2 or HMS.
This is also indicated in the logs by an error message like:

2016-07-21 17:39:54,479 WARN org.apache.sentry.service.thrift.SentryKerberosContext: Failed to renew ticket
javax.security.auth.login.LoginException: Connection reset
[...]
2016-07-21 17:39:54,479 INFO org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer thread finished


See also the following upstream jira:
SENTRY-1265 - Sentry service should not require a TGT as it is not talking to other kerberos services as a client

Please log in or register to answer this question.

...