Symptom:
Drill query fails with error message like below:[<SQL_ID>:foreman] WARN o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune partition. org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 256 due to memory limit. Current allocation: 268435456This error always shows in the foreman drillbit.log during planning phase.
In above example, it fails when doing partition pruning in planning phase.
Root Cause:
Starting in Drill 1.2, parameter "planner.memory_limit" defines the maximum amount of direct memory allocated to a query for planning. By default it is set to 256MB(268435456 bytes).During partition pruning, if the query needs to filter more partitions, it needs more memory during planning.
Solution:
Increase planner.memory_limit to large enough per query.For example, set it to 512MB:
alter session set `planner.memory_limit`=536870912;
No comments:
Post a Comment