At CoolaData, we analyze behavioral events as they occur in real-time, sent by our customers’ platforms when their users perform various actions. This data is processed and prepared so that customers can understand and obtain insights about how their products and services are used. We handle hundreds of millions of these events every day and it’s crucial that data is available for analysis quickly, reliably, efficiently and accurately. We’re always mindful of our customers’ peaks so that we can manage changing amounts of data every day, without interruptions or data loss.
With Kafka, CoolaData ensures that our resources consume and process billions of messages without losing any data along the way. We’re able to monitor our system for peaks, as well as increase the amount of resources that we provide for data processing. It also enables us to safely deploy new features and fixes without losing or delaying customer data. In this post, we will dive into how Kafka works and describe what we use it for.
How Kafka Works
Apache Kafka is a non-commercial, open source log service that receives messages sent from its clients to its system, holds onto them and then distributes them soon as it gets a retrieval request. Kafka is built differently from other messaging systems in that it doesn’t follow common messaging queue practices, while still providing the same needed features. It saves messages so that they can be retrieved at any time; configured to guarantee that messages are consumed only once; support multi-subscribers; and automatically balances data consumption. Kafka’s elasticity is what makes it truly unique, expanding as a cluster of one or more servers to deliver higher scales of data.
How CoolaData Uses Kafka
Here at CoolaData we handles terabytes of data every month and Kafka helps us maintain our services and offerings. CoolaData’s web servers receive messages from our customers and pass them on to Kafka, which stores them until enough resources are available. Sometimes our engines receive more incoming messages than our allocated resources can handle. In such cases, Kafka acts as a buffer by accumulating messages until the normal handling rate is reached. Data is then forwarded to CoolaData’s engines when resources become available.
To demonstrate how CoolaData allows customers to track user behavior trends, let’s imagine a mobile application that aggregates and distributes news items and articles for its users, as an example. This mobile application notifies users about new articles that are published, based on their users’ preferences. Some users may read the suggested articles, others will ignore them, or adjust their preferences after receiving suggestions. This application is popular, so suggestions are sent to millions of users each day. The app’s developers integrate CoolaData’s tracker to send messages about actions that users perform according to their interests. These messages are received by our web servers, which in turn pass them on to Kafka. Kafka then sends the messages through a process that enriches and prepares them for analysis. This flow happens on a daily basis, hour by hour, minute by minute. It can happen at an average rate of 60,000 messages per minute, but, depending on how successful the app is and how good its suggestions are, it can reach a peak of up to 100,000, or even 600,000 per minute.
CoolaData monitors each customer’s allocated resources and makes adjustments accordingly. However, when we process these events according to a customer’s allocated resources, at a steady rate of 60,000 messages per minute, adjustments might not occur fast enough. Kafka comes to the rescue by accumulating messages regardless of the rate that our servers send them out. It holds them until the peak drops, and then releases them for us to pull and process. This enables us to reliably process messages quickly and accurately. Our customers then perform user behaviour analysis using CoolaSQL or dedicated widgets. They visualize and present insights with our Analytical Document.
Kafka is essential to the service that CoolaData provides due to enormous amounts of events our API receives from customers every day. With Kafka’s robust queueing system, we can adjust our resources according to demand, while protecting our customers’ data and ensuring that it is ready for analysis quickly and reliably.