Quick & simple yaml for kafka on kubernetes

Want to evaluate the kafka stack on your kubernetes cluster?

One of the barriers to quickly checking out the kafka ecosystem on your k8s cluster can be needing to generate your yaml from a helm chart (most likely, the confluent helm charts or cobbling the components together from the  incubator/kafka chart plus the incubator/zookeeper chart, etc)... If you are not already using helm, then the adoption cost can be suddenly steeper than you'd hoped;  I know I wrestled for a good little while getting it how I wanted, mostly because - despite my utmost google-fu, I could not find  simple, working yaml files to compare/easily grok.

So... Here's a pretty basic and definitely not production-ready set of yaml files - read the notes before using!

https://github.com/PaulieMac/kstack-yaml

These should stand up a working kafka stack based on the https://www.confluent.io/ community suite (version 5.5.0 at time of writing). This is comprised of:

• zookeper
• kafka
• avro schema registry
• connect
• kafka-rest-api
• ksqldb (version 0.9.0)

I have also included 

• the excellent kafdrop by obsidian dynamics
• Schema Registry UI and Connect UI, both by https://github.com/lensesio/

External Ingress

Once you have it deployed and healthy - which may take a few minutes - you will be able to connect to kafka itself from external to the cluster (for example, from VSCode on your desktop) using something like the following, with 192.168.164.01 replaced by the IP you end up with. Ports will always be 32400, 32401, 32402 :)

192.168.164.01:32400
192.168.164.01:32401
192.168.164.01:32402

You will also have (obviously switch out your-domain.local with your real value)

schema-reg.your-domain.local
ks-rest.your-domain.local
kafdrop.your-domain.local
connect-ui.your-domain.local
schema-reg-ui.your-domain.local

Internal Links

Internally to the k8 namespace, you will have:

• kafka bootstrap:  PLAINTEXT://ks-kafka-headless:9092
• schema registry:  http://schema-reg:8081
• connect:  http://ks-connect:8083
• zookeeper:  zk-headless:2181

And that should be that; hopefully it is now a little bit easier for you to take this stack for a spin!