Hive on Spark | Jobs fail with AccessDeniedException | Insufficient permissions for hive user on scope hbase:meta

0 votes
14 views
asked Aug 28, 2017 in Hadoop by admin (4,410 points)
Summary

 

Applies To
  • CDH 5.10.1
  • Hive on Spark
  • HBase service not configured for Hive
Symptoms

After upgrading to CDH 5.10.1 all Hive on Spark queries / jobs fail with the following message:

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=hive/hiveserver2host.realm.com@REALM.COM, scope=hbase:meta, params=[table=hbase:meta],action=EXEC)
at org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:447)
at org.apache.hadoop.hbase.security.access.AccessController.preEndpointInvocation(AccessController.java:2184)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$67.call(RegionCoprocessorHost.java:1628)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$EndpointOperation.call(RegionCoprocessorHost.java:1693)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1732)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preEndpointInvocation(RegionCoprocessorHost.java:1623)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7840)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1988)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1970)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
... 30 more

The same query with Hive on MapReduce (set hive.execution.engine=mr) works, impala is not impacted. The query does not touch any Hive table backed by HBase, moreover the HBase service dependency is not selected in Hive service.
The query fails before starting a YARN application.

 

Cause

CDH 5.10.1 includes SPARK-12523 "Support long-running of the Spark On HBase and hive meta store."
This changed the default behavior of spark-submit (which is used when submitting Hive on Spark queries) and tries to get a token for HBase even if HBase service disabled for Hive.

Instructions

The workaround is to:
1. Disable getting tokens for HBase before running the query with

set spark.yarn.security.tokens.hbase.enabled=false;

2. or set the same globally in Hive > Configuration > "HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml"
set name to

spark.yarn.security.tokens.hbase.enabled

set value to 

false

3. or alternatively set up the HBase dependency for Hive service and allow the EXEC permission to "hive" user on the hbase:meta table

Please log in or register to answer this question.

...