MapR Stream Workshop 1: streams.buffer.max.time.ms

Theory:

Per Documentation for parameter streams.buffer.max.time.ms
   Messages are buffered in the producer for at most the specified time. A thread will flush all the messages that have been buffered for more than the time specified.
   Default: 3 * 1000 msec

Experiment:

By monitoring the MapR Stream internal metrics using command "/opt/mapr/bin/guts streams:all", we are proving above theory.

1. SampleProducer keeps writing messages without stop.

mapr openkb.stream.SampleProducer

The performance metrics from "guts" output is:

  mpr     mpm mpMB
    0       0    0
    0       0    0
    0       0    0
    0       0    0
    4   59584    1
   11  148416    3
    7   80000    1
    7  103680    2
    9  108320    2
    0       0    0
    0       0    0

To understand above metrics, here is the documentation:

Name   Description
mpr    The number of RPCs from MapR Streams producers to the server.
mpm    The number of messages that have been published to the server.
mpMB   The total size in MB of the messages that have been published to the server.

The speed is normally more than 100k messages/sec.

2. SampleProducer_1sec writes 1 message and then sleep for 1 second.

mapr openkb.stream.SampleProducer_1sec

The performance metrics from "guts" output is:

mpr     mpm mpMB
  0       0    0
  0       0    0
  1       3    0
  0       0    0
  0       0    0
  1       3    0
  0       0    0
  0       0    0
  1       3    0
  0       0    0
  0       0    0
  1       3    0

The speed is 1 RPC call to write 3 messages every 3 seconds.
This clearly shows the role of parameter streams.buffer.max.time.ms