Wednesday, February 25, 2015

Reading table blocks creating tables in the same database when hive.support.concurrency=true.

Symptom:

Reading table blocks creating tables in the same database when hive.support.concurrency=true.
Creating table will fail with below errors due to database level lock.
hive> create table test(id int);
conflicting lock present for default mode EXCLUSIVE

Env:

Hive 0.13

Root Cause:

By default in Hive 0.13 when hive.support.concurrency=true, it is using
hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
hive.lock.manager=org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
Some DDLs needs Exclusive lock on database using this transaction manager.
This issue is reported in HIVE-9199 which is fixed in Hive 1.1.

Minimum reproduce:

1. Set hive.support.concurrency=true in hive-site.xml.
2. Open 2 hive sessions:
--Session A:

create table passwords (col0 string, col1 string, col2 string, col3 string, col4 string,
col5 string, col6 string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ":";
load data local inpath "/etc/passwd" into table passwords;

select count(*) from passwords a,passwords b, passwords c, passwords d
where a.col0=b.col0
and a.col0=c.col0
and a.col0=d.col0;

--Session B:

hive> create table test(id int);
conflicting lock present for default mode EXCLUSIVE

Solution:

This is current lock behavior in Hive 0.13. You can disable this feature by setting hive.support.concurrency=false in hive-site.xml.
Or upgrade to Hive 0.14 or above in which new lock manager DbLockManager and new transaction manager DbTxnManager are introduced.

No comments:

Post a Comment

Popular Posts