Also, batch size is related with number of brokers and number of partitions.
For example, if you have 2 brokers and 3 partitions for each broker that means the total count of partition is 6.
y this way, the batch size equals to tridentKafkaConfig.fetchSizeBytes X total partition count.
if we assume that the tridentKafkaConfig.fetchSizeBytes is 1024X1024, the batch size equals to 6 MB.(3x2x1024x1024)bytes
相当于从 kafka 里面是批量读取的, 然后形成 storm 中的一个 batch
而且 spout 里面有一个 buffer, 应该也算是一个缓存, 将正在处理的一部分数据, 存在内存中, 直到处理完
来源: http://www.bubuko.com/infodetail-2619017.html