How to limit the size for each zookeeper transaction log

Goal:

How to limit the size for each zookeeper transaction log.

Env:

Zookeeper 3.4.5 on MapR 5.1 

Solution:

1. Limit number of transactions in one Zookeeper Transaction Log

By default, the Zookeeper transaction log switch will happen after "snapCount(100000)" transactions are written.
If the Zookeeper data directory is not enough to hold 3 versions of Zookeeper Transaction Logs, we may need to reduce the size for each log.
For example, after we reduced "snapCount" to 20 in zoo.cfg and restarted zookeeper service, each log in my test environment had below many transactions:

[root@v7 version-2]# ls -altr log*
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.40000000d
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.40000001f
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.400000030

[root@v7 version-2]#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.40000000d|tail -1
EOF reached after 18 txns.
[root@v7 version-2]#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.40000001f|tail -1
EOF reached after 17 txns.
[root@v7 version-2]#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.400000030|tail -1
EOF reached after 8 txns.

2. Limit the block size of Zookeeper Transaction Log

Even after #1 is done to limit the number of transactions, from above output we can see that each log is still 64MB. For sure, space is wasted because the minimum block size is set to preAllocSize kilobytes.

For example, we can set the block size to 1MB instead in zoo.cfg and restart zookeeper:

[root@v5 conf]# cat zoo.cfg |grep pre
preAllocSize=1000

After that, minimum block size of Zookeeper Transaction Log  will become 1MB:

[root@v7 version-2]# ls -altr log*
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:38 log.400000001
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.40000000d
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.40000001f
-rw-r--r-- 1 mapr mapr 67108880 Jan 11 18:40 log.400000030
-rw-r--r-- 1 mapr mapr  1024016 Jan 11 19:43 log.500000001
-rw-r--r-- 1 mapr mapr  1024016 Jan 11 19:43 log.500000013
-rw-r--r-- 1 mapr mapr  1024016 Jan 11 19:43 log.500000022
-rw-r--r-- 1 mapr mapr  1024016 Jan 11 19:43 log.50000002e
-rw-r--r-- 1 mapr mapr  1024016 Jan 11 19:43 log.50000003c