Hive implements a global table locking mechanism between all instances of HiveServer2. The most common locking mechanism depends on Apache ZooKeeper. The following lock modes are defined in Hive: Implicit locks are automatically acquired by a query depending on the target table and the operation being requests (SELECT, INSERT, etc). These locks are normally released as part of the cleanup process of every query.
Explicit locks are created manually by a user. These locks must be manually removed by a user.
HiveServer2 is responsible for creating and removing these locks. In certain Hive failure scenarios, these locks may not be removed after a query has completed or a user simply may have forgotten to remove their explicit lock. If a lock is left on a table, certain queries will block and eventually fail waiting for the lock to be removed. It will be required to manually remove these locks for operation to continue. Implicit LocksWhen global locks are enabled, Hive uses ZooKeeper ephemeral nodes to represent implicit locks. These nodes are normally removed after a query has completed. These nodes will also be removed if the HiveServer2 which created the nodes fails or is restarted. In a HiveServer2 failure/restart scenario, these nodes are removed 60 seconds after the HiveServer2 looses connectivity with ZooKeeper. Explicit LocksWhen global locks are enabled, Hive uses ZooKeeper persistent nodes to represent explicit locks. These nodes can only be manually removed from Zookeeper directly or using Hive's UNLOCK statement. They will not be removed if the HiveServer2 instance that created the node fails or is restarted. Lock TimeoutsWhen a lock exists on a table, any attempts to issue a DELETE or INSERT statement against the table will block waiting for the lock to be removed. In certain other scenarios, SELECT statements may block as well. If the lock is not removed, the query will fail after a specified timeout. This timeout is controlled by two configurations: hive.lock.numretries and hive.lock.sleep.between.retries. If the environment has long duration queries, these configurations must be increased or queries that are blocked by legitimate locks will fail while waiting. Manually Removing a Lock From ZooKeeper- Ensure no active queries are running against the target table
- Launch the zookeeper-client on one of ZooKeeper nodes
- Use "ls" to browse into the hive zookeeper directories, ex. "hive_zookeeper_namespace_hive"
- Once the desired locked table is found, Run the following command in the zookeeper-client shell:
> ls /
[zookeeper, hive_zookeeper_namespace_hive]
> rmr /hive_zookeeper_namespace_hive/<DATABASE>/<BROKEN_TABLE> - Try the failing query again
Automatically Removing a Lock From ZooKeeperThis only applies to implicit locks. This requires a service interruption during restart. - Restart all instances of HiveServer2
- Wait 60 seconds
- Try the failing query again
|