How to modify max heap size for Impalad embedded JVM

Goal:

The impalad is a 'native' process that has an embedded JVM. The JVM is started from within c++ code.
 The -mem_limit startup option sets an overall limit for the impalad process (which handles multiple queries concurrently).
However the max heap size of impalad embedded JVM is much smaller than that limit.
Some impala queries may use up all embedded JVM heap size before reaching the limit set by "-mem_limit" startup option, so that it may cause impalad errors "OutOfMemoryError: Java heap space" or just get hung. In that situation, we need to increase the JVM max heap size.

This article shows how to check and modify the max heap size for impalad embedded JVM.

Env:

Impala 2.x

Solution:

1. Identify the impalad process and its current "-mem_limit" startup option setting

# ps -ef|grep -i impalad
mapr     25700     1  0 Aug24 ?        00:38:34 /opt/mapr/impala/impala-2.5.0/sbin/impalad -log_dir=/opt/mapr/impala/impala-2.5.0/logs -state_store_port=24000 -use_statestore -authorized_proxy_user_config=mapr=* -state_store_host=localhost -catalog_service_host=localhost -be_port=22000 -mem_limit=100%

Here "-mem_limit=100%" means impalad can use up to 100% of total OS memory.
In this example, the total OS memory is 7GB:

$ free -g
             total       used       free     shared    buffers     cached
Mem:             7          7          0          0          0          1
-/+ buffers/cache:          6          1
Swap:            7          0          7

2. Check current max heap size of impalad embedded JVM

$ jcmd 25700 VM.flags
25700:
-XX:CICompilerCount=3 -XX:InitialHeapSize=130023424 -XX:MaxHeapSize=2065694720 -XX:MaxNewSize=688390144 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=42991616 -XX:OldSize=87031808 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC

Here "-XX:MaxHeapSize=2065694720" means the max heap size is 2GB.

3. Increase max heap size of impalad embedded JVM to 5GB

Put below line into /opt/mapr/impala/impala-<version>/conf/env.sh

export JAVA_TOOL_OPTIONS="-Xmx5g"

4. Restart impalad

maprcli node services -name impalaserver -action restart -nodes <node_host_name>

5. Verify the current max heap size after restarting impalad

$ jcmd 8789 VM.flags
8789:
-XX:CICompilerCount=3 -XX:InitialHeapSize=130023424 -XX:MaxHeapSize=5368709120 -XX:MaxNewSize=1789394944 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=42991616 -XX:OldSize=87031808 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC

Here it is "-XX:MaxHeapSize=5368709120".

Note: Of course, you can also change other JVM options for this embedded JVM in the same way, but you need to be very careful and understand what you are trying to change.