Apache Kafka
There are different types of messaging systems that are available today. Broadly those can be categorized into 2 categories.- Point to Point Messaging System
- Publish Subscribe Messaging System
➠ Point to Point Messagaing System: In this type of messaging system, publisher pushes the message in the queue (called topic) and single subscriber can consume messages from the queue. Message will be removed/expired from the queue once the subscriber consumes the data from queue.
➠ Publish Subscribe Messagaing System: In this type of messaging system, publisher pushes the messages in the queue (called topic) and multiple subscribers can consume messages from the queue. Message will remain in the queue until the retention period specified for the queue.
➠ Lets Discuss Kafka
- Kafka Messaging is a distributed streaming platform which generally fall under the category of Publish Subscribe Messaging System but can be defined as Point To Point Messaging System using consumer groups.
- Kafka require 2 different types of servers for running, one is Zookeeper and other one is Broker (Bootstrap Server).
- Same messages can be read from the topic multiple times.
- Replication factor (how many copies of each message to be kept on other buddy node/server) can be set for Kafka queues (topic) as it is distributed system. This makes Kafka fault tolerant & durable.
- Messages can be consumed as soon as they are published to the Kafka queue.
- Message can be published in any text format like plain text, JSON or XML.
➠ Common Kafka Terms
- Topic : Topic is an entity (queue) where messages are pushed by producer and stored for consumption by consumer.
- Producer : Producer can be seen as application that will publish message/data into the topic(queue).
- Consumer : Consumer can be seen as application that will consume message/data from the topic(queue).
- Zookeeper : Zookeeper is highly reliable coordination server which store the state of the kafka broker server. Zookeeper provide coordination between broker & consumer.
- Broker : Broker is the kafka server where topics are created and messages are stored in it. Brokers can be configured to auto-create topics when a non-existent topic is published.
- Offset : Each partition maintain the sequential order of messages it received. This sequential order of messages still present in topic is identified as offset.