Hive 2.x queries got stuck when waiting for "tryAcquireCompileLock" in HS2 stacktrace

Env:

Hive 2.x

Symptom:

Hive 2.x queries got stuck when waiting for "tryAcquireCompileLock" in HS2 stacktrace.
When the issue happens, connecting using beeline to HS2 works.
However any queries, eg, "show databases" will hung.

Below is one example of the jstack output on HS2 process:

"7d88a5ad-cd2c-4c37-9025-8372164524fd HiveServer2-Handler-Pool: Thread-214" #214 prio=5 os_prio=0 tid=0x00007fdacc2d1800 nid=0x7af0 waiting on condition [0x00007fda9b6fa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005c1fb7f28> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at org.apache.hadoop.hive.ql.Driver.tryAcquireCompileLock(Driver.java:1324)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1236)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1230)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:191)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:276)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:324)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:499)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:486)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
        at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
        at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
        at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
        at com.sun.proxy.$Proxy25.executeStatementAsync(Unknown Source)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:294)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

Root Cause:

Starting from Hive 2.0, by default all sessions will share the same compilation lock.
More details are inside HIVE-4239.
As a result, 2 Hive parameters are introduced:

• hive.server2.compile.lock.timeout -- Number of seconds a request will wait to acquire the compile lock before giving up.
• hive.driver.parallel.compilation -- Enable parallel compilation between sessions on HiveServer2. The default is false.

By default, hive.driver.parallel.compilation is set to false in Hive 2.1.

Solution:

1. Change hive.driver.parallel.compilation=true in hive-site.xml.
2. Restart HS2.