Debugging Kafka Connect Connectors/SMTs in Java

Debugging Kafka Connect Connectors/SMTs in Java

Are you developing a custom Kafka Connect Connector or Single Message Transform (SMT) and need to debug this? This blog post will give you ideas on how to output information to standard output and how to attach a remote debugger to a custom SMT running in a Docker container. This article can save you enormous amounts of time while developing a custom Connector or SMT.

GitHub

This is part of a blog series on Kafka Connect.
GitHub: Kafka Connect Custom SMT

Expectations

In order for this blog post to make sense make sure you fit the expectations below. Otherwise you may want to brush up on some of these technologies.

• Capable of programming in Java
• Comfortable using Docker and creating Dockerfiles
• Familiarity with IntelliJ
• Kafka Connect

Debugging Using Standard Out

Debugging the custom Connector or SMT can be done using standard out. Within the Kafka Connect library, is a class that supports logging. The LoggerFactory has a getLogger() method for retrieving a logger that you can use.

Enabling Tracing in Kafka Connect

For this to work you will also have to enable TRACE in the log4j properties file in the Docker Container. It is located at /etc/kafka/docker/connect-log4j.properties. The prefered way is to use Docker Compose but I’ll demonstrate how to do this with a manual update which will allow you to inspect the values which might be useful.

Manual Update

To access the Docker container, run these commands against your container. In the sample below the container is named cc-connect. Yours may be something else.

Make change in configuration file. Should be last line.

docker restart cc-connect`

Docker Compose

Another alternative is if you’re using Docker Compose to update the variable that sets this property.

Code

This usually goes as a property on the SMT/Connector class.

private static final Logger log = LoggerFactory.getLogger(SmtEftDb.class);

Outputting data can be done using either info, warn, and error.

log.info("[+] Configuring Custom SMT");

Debugging Kafka Connect with Docker & Java

I’ll demonstrate how to debug a Kafka Connect Single Message Transform (SMT) running in a Docker container. For me, the easiest way to develop an SMT was to create a custom Docker image that extended the Confluent Cloud’s Kafka Connect Docker image. By doing so, I could also copy in my SMT Plugin through the Docker build process and set environmental variables that are necessary for debugging. The key thing here is to bind the debugger to listen on all adapters by using “*:”.

Dockerfile

Setting the JAVA_TOOL_OPTIONS environmental variable is all you have to do to enable debugging within the Kafka Connect Docker container.

Binding *:5005

I can’t stress how important this small detail is but in order for the debugger in IntelliJ to connect to the running Docker container you absolutely must bind all adapters by using “*:”.

IntelliJ Configuration

This follows basic remote debugging configuration.

• 

•