Newbie’s intro to kafka
I recently started learning about kafka and through this article I will try to answer the questions I faced during my learning.
First question that I faced was when i read definition of kafka “Event streaming platform used to collect, store and process real time data streams”. After spending some time on this what I understood is this. Lets say you have a website and you want to track how many times a certain link is been clicked. Here clicking a link is an event. This event contains data which can be later used to cater the website to users preferences. For that you need a system which will catch this event, process it and store it for later purpose. Kafka comes handy in such use cases.
Website here plays role of producer. After Website generated the event. event will be sent to kafka broker. Broker is the kafka server which holds multiple topics. Each topic holds messages of particular type. This help segregate messages for processing.
Each producer sends its events to one of the topics, broker then stores that message in its logs. Each message is allocated a unique offset id. Offset id is like index used to access data in array. It gives unique position to the message so that consumers can start consuming messages from their intended position.
Consumers are applications which consume the messages. Each consumer subscribes to a topic and then it can pull messages from that topic from whatever offset it has. One of the important you need to observe is that consumer has no knowledge of which producer produced certain message. They have no connection at all. Such loosely coupled components are big plus in distributed computing. Once consumer received the message it can process it the way it likes. In our initial example, consumer might use this data to feed into ML model to generate recommendations.
This is very top level introduction to kafka. There are lot of intricacies that are left untouched. For example partitions, zookeeper, consumer groups etc. but this article should give you initial head start in kafka.
