Kafka Secure: Message Consumer

Kafka Secure: Message Consumer

Kafka provide a way to implement SSL security for both producers and consumer applications to connect the server. Kafka administrator who setup the secure Kafka, will also generate information/files for connecting to the server. This tutorial will show how to read from Kafka with SSL security.

Following information and files are required to connect to the Kafka secure server.

• Keystore file (kafka_client_keystore.jks)
• Keystore password
• Key password
• Truststore file (kafka_client_truststore.jks)
• Truststore password

All the information required for connecting Secure Kafka need to be stored in the properties file. Click here to download sample SSL properties file.

*****************ssl_detail.properties**************
security.protocol=SSL
ssl.keystore.location=/full_path_to_file/kafka_client_keystore.jks
ssl.keystore.password=keystore_password
ssl.key.password=key_password
ssl.truststore.location=/full_path_to_file/kafka_client_truststore.jks
ssl.truststore.password=truststore_password
*************************************************

➠ Getting Messages: '--consumer.config' attribute can be used to specify SSL properties to the consumer command.

• All Messages: Fetch all the messages present in the topic.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_ssl_test_topic --from-beginning --consumer.config ~/files/ssl_detail.properties
  

• Limiting Messages: Limit the number of messages to be pulled from topic using "max-messages" attribute.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_ssl_test_topic --max-messages 2 --consumer.config ~/files/ssl_detail.properties
  

• Fetching Messages from particular offset: Messages can be pulled specifically from particular partition & offset using below commands.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092  --topic kafka_test_topic --offset 5 --partition 0 --consumer.config ~/files/ssl_detail.properties
  

• Fetching Message for the particular offset: Messages can be pulled specifically for a particular partition & offset using below commands.

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092  --topic kafka_test_topic --offset 5 --partition 0 --max-messages 1 --consumer.config ~/files/ssl_detail.properties
  

• Fetch Messages using Consumer Group: New messages can be pulled using below command, click here to know more about consumer group

  ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka_ssl_test_topic --consumer-property group.id=cg_name_1 --consumer.config ~/files/ssl_detail.properties
  

➠ Topic offsets: Checking Topic offsets in Secure Kafka

• Minimum offset: Checking the minimum offsets for all the partitions of topic.

  ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cg_name_4 --reset-offsets --to-earliest --topic kafka_test_topic_withpartition --dry-run --command-config ~/files/ssl_detail.properties
  
  Output:
  TOPIC                          PARTITION  NEW-OFFSET
  kafka_test_topic_withpartition 0          2         
  kafka_test_topic_withpartition 1          2         
  

• Maximum offset: Checking the maximum offsets for all the partitions of topic.

  ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group cg_name_4 --reset-offsets --to-latest --topic kafka_test_topic_withpartition --dry-run --command-config ~/files/ssl_detail.properties
  
  Output:
  TOPIC                          PARTITION  NEW-OFFSET
  kafka_test_topic_withpartition 0          4         
  kafka_test_topic_withpartition 1          3