If you're using Kafka, kcat is an essential tool to have in your
toolbox. As the name suggests, it's cat for Kafka. It gives you
fast, command-line access to simple tasks. Like, say you want to tail
a topic called orders:
kcat -Ct orders(-C is 'consume'. -t sets the topic name. This is unix, so you can collapse those flags into -Ct.)
Easy huh? Well, not quite. Because there's usually more to type than that. For a local Kafka instance you need to set the broker:
kcat -Ct orders -b localhost:9092That's a pain. And if you want to access a remote machine it gets much worse:
kcat -Ct orders -b localhost:9092 -X security.protocol=sasl_ssl -X sasl.mechanism=PLAIN ...ugh...But you don't have to do that. You can put it all in a kcat
config file. It's a little bit of a tricky one to
get right though, so for your reference and mine, here's how you set
that up once and for all.
The file you need to edit is ~/.config/kcat.conf.
For a local instance, it should contain something like:
metadata.broker.list=localhost:9092For a remote server, you'll need to include the authentication details, so your kcat.conf will look more like:
metadata.broker.list=<Bootstrap Server>
security.protocol=sasl_ssl
sasl.mechanism=PLAIN
sasl.username=<API Key>
sasl.password=<API Secret>With a kcat.conf correctly set up, all the connection information
gets picked up every time you call kcat, so that first, simpler
command kcat -Ct orders should Just Work.
Update: After I wrote this up I found myself wishing I could do the same trick to connect to a Schema Registry. I'd like to be able decode some Avro values by simply saying:
kcat -Ct orders -s value=avroAgain, that's not too hard. You "just" need to cram the location of
the schema registry, and your authentication details, in the right
format under the right key. Here's the magic incantation to add to
your ~/.config/kcat.conf:
schema.registry.url=https://<SR API Key>:<SR API Key>@<SR URL>With that, kcat will attempt to fetch schemas from the registry
whenever it needs a codec.