Refer to
Determine YARN and MapReduce Memory Configuration Settings.
As a general recommendation, allowing for two Containers per disk and per core gives the best balance for cluster utilization.
1. Determine reserved memory.
Reserved Memory = Reserved for stack memory + Reserved for HBase memory (If HBase is on the same node).
| Total Memory per Node | Recommended Reserved System Memory | Recommended Reserved HBase Memory |
| 4 GB | 1 GB | 1 GB |
| 8 GB | 2 GB | 1 GB |
| 16 GB | 2 GB | 2 GB |
| 24 GB | 4 GB | 4 GB |
| 48 GB | 6 GB | 8 GB |
| 64 GB | 8 GB | 8 GB |
| 72 GB | 8 GB | 8 GB |
| 96 GB | 12 GB | 16 GB |
| 128 GB | 24 GB | 24 GB |
| 256 GB | 32 GB | 32 GB |
| 512 GB | 64 GB | 64 GB |
2. Determine max number of Containers per node.
# of Containers = minimum of (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)
where MIN_CONTAINER_SIZE=
| Total RAM per Node | Recommended Minimum Container Size |
| Less than 4 GB | 256 MB |
| Between 4 GB and 8 GB | 512 MB |
| Between 8 GB and 24 GB | 1024 MB |
| Above 24 GB | 2048 MB |
3. Determine RAM per Container.
RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))
4. YARN and MapReduce configuration.
| Configuration File | Configuration Setting | Value Calculation |
| yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
| yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
| yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = Containers * RAM-per-Container |
| mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
| mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
No comments:
Post a Comment