Kafka: Message Consumer

Kafka: Message Consumer

$ cd ~/kafka_2.11-1.1.0/bin

• All Messages: Fetch all the messages present in the topic.
  Example 1:

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_test_topic --from-beginning
  
  Output:
   Kafka Message 1   
   Kafka Message 2
   Kafka Message 3
   Kafka Message 4
   Kafka Message 5
   
   
  For Older versions: 
  ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka_test_topic --from-beginning
  

  Example 2:

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_test_topic_withpartition --from-beginning
  
  Output:
   Kafka Topic 2 Message 1   
   Kafka Topic 2 Message 2
   Kafka Topic 2 Message 3
   Kafka Topic 2 Message 4
   Kafka Topic 2 Message 5
  
  
  For Older versions: 
  ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka_test_topic_withpartition --from-beginning
  

  Example 3:

  ./kafka-simple-consumer-shell.sh --broker-list localhost:9092 --topic kafka_test_topic
   
  Output:
   Kafka Message 1   
   Kafka Message 2
   Kafka Message 3
   Kafka Message 4
   Kafka Message 5
   
   Note: kafka-simple-consumer-shell.sh will be deprecated in the future release.
  

• Limiting Messages: Limit the number of messages to be pulled from topic using "max-messages" attribute.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_test_topic_withpartition --from-beginning --max-messages 2
  
  Output:
  Kafka Topic 2 Message 1   
  Kafka Topic 2 Message 2
  
  
  For Older versions: 
  ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafka_test_topic_withpartition --from-beginning --max-messages 2
  

• Fetching Messages from particular offset: Messages can be pulled specifically from particular partition & offset using below commands.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092  --topic kafka_test_topic --offset 2 --partition 0
  
  Output:
  Kafka Message 3
  Kafka Message 4
  Kafka Message 5
  
  
  
  For Older versions: 
  ./kafka-simple-consumer-shell.sh --broker-list localhost:9092 --topic kafka_test_topic --offset 2 --partition 0
  

• Minimum offset: Checking the minimum offset for the topic

  ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic kafka_test_topic_withpartition --time -2
  
  Output:
  kafka_test_topic_withpartition:0:0
  kafka_test_topic_withpartition:1:0
  kafka_test_topic_withpartition:2:0
  
  
  --Using Consumer group
  ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cg_name_4 --reset-offsets --to-earliest --topic kafka_test_topic --dry-run
  
  Output:
  TOPIC                          PARTITION  NEW-OFFSET     
  kafka_test_topic               0          4  
  

• Maximum offset: Checking the maximum offset for the topic.

  ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic kafka_test_topic_withpartition --time -1
  
  Output:
  kafka_test_topic_withpartition:0:0
  kafka_test_topic_withpartition:1:5
  kafka_test_topic_withpartition:2:0
  
  
  --Using Consumer group
  ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cg_name_4 --reset-offsets --to-latest --topic kafka_test_topic --dry-run
  
  Output:
  TOPIC                          PARTITION  NEW-OFFSET     
  kafka_test_topic               0          13 
  

  Note: Difference between Maximum & Minimum offset can be defined as the number of messages present in Kafka topic.