Getting Started with Landoop’s Kafka on Docker for Windows

Here’s a quick guide to running Kafka on Windows with Docker. I’ll show you how to pull Landoop’s Kafka image from Docker Hub, run it and how you can get started with Kafka. We’ll also be building an ASP.NET C# console app for this demonstration. I learned most of this from the Kafka Udemy videos mentioned below and someone showed me a Kafka C# demonstration. I’ve modified everything I’ve been taught to a more simpler version. This article is for people who are brand new to Kafka and want to work with ASP.NET.

Source Code: Kafka C# Sample

Requirements

Make sure you installed and are familiar with Docker for Windows, ConEmu, Visual Studio and C#.

Docker for Windows uses the localhost address of 127.0.0.1 where Docker Toolbox / Docker Machine will use 192.168.99.100. Landoop’s fast-data-dev Github readme currently displays the steps for docker-machine.

Getting Setup with Docker

After you’ve installed Docker,you will need to pull down the Landoop Kafka (fast-data-dev) image from Docker Hub. The commands below will pull the image.

Running Landoop’s Kafka Container in Docker

We will start this container in a detached state (“-d”) and then we will access the logs so we can see what’s happening.

To see the logs of the container and have them persist to the screen pass the docker attribute “-f” for follow. This screen is important for troubleshooting errors. Some common errors you may see are memory issues. The URL for the Kafka UI is also in the logs.

Landoop’s Kafka UI

Landoop's Kafka UI
Landoop’s Kafka UI

Once the Docker image for fast-data-dev is running you will be able to access theLandoop’s Kafka UI. This gives developers the ability to see in real time what Kafka is doing, how it creates and manages topics. You can visually see configuration and topic data in the UI.

Landoop’s Kafka UI: http://127.0.0.1:3030/

Working with Kafka via Command Line

It will benefit you to learn how Kafka works from a command line basis. Users can create topics, publish and subscribe; all from the bash terminal. This is a great starting place because you can see exactly how it works under the hood.

Accessing Docker Container with Bash

Executing the command below will connect you to the container with bash. Doing so will allow you to interact with Kafka via command line.

Kafka CLI Producer

We’re going to create our first topic, “first_topic” and push a single string of data to it.

If you receive an error message something similar to ” Error: {new_topic={LEADER_NOT_AVAILABLE}”, this is normal because it takes time to create the topic.

Kafka CLI Consumer

The consumer will immediately listen for publishes from the producer.

However, if you want to pull all of the data from the beginning you can run this command:

Confluent Kafka with DotNet Core

The source code for this application is maintained in GitHub. This demo is very simple, we’ll use Kafka as a pub/sub for messages. We’ll serialize a model class, Msg and publish it to Kafka and allow the consumer to use C# to read that data.

Source Code: Kafka C# Sample

Confluent Kafka: https://github.com/confluentinc/confluent-kafka-dotnet

Message Data Model

This data model will be shared between stored in its own project (Kafka.Data) and shared between the Producer and the Consumer.

Kafka C# Producer Source Code

This is a simple sample that creates a console application DotNet core that captures the username and message then publishes that data to Kafka.

Kafka C# Consumer

This is a sample of how to read the data from the Kafka stream.

Troubleshooting

Some people have issues with getting this up and running because they aren’t famililar with the surrounding technologies like Docker.

winpty – if you are running Windows and run into an issue prepend the command with winpty so it reads “winpty docker run -d….”

Closing

We recently started using Kafka at my current job and I thought it would be beneficial to share a quick tutorial of how to use C#, Confluent Kafka and Landoop’s fast-data-dev. I couldn’t find a quick all-in-one tutorial for the basics so I created one.

Confluent Kafka Git Repo

Udemy Videos

If you’d like to learn more about Kafka I highly recommend watching these videos by Stephane Maarek at Udemy.com

https://www.udemy.com/apache-kafka/

https://www.udemy.com/kafka-streams/

https://www.udemy.com/kafka-cluster-setup/

https://www.udemy.com/kafka-connect/

https://www.udemy.com/confluent-schema-registry/

https://www.udemy.com/apache-kafka-security/