If you're using Kafka,
kcat is an essential tool to have in your
toolbox. As the name suggests, it's
cat for Kafka. It gives you
fast, command-line access to simple tasks. Like, say you want to tail
a topic called
kcat -Ct orders
-C is 'consume'.
-t sets the topic name. This is unix, so you can collapse those flags into
Easy huh? Well, not quite. Because there's usually more to type than that. For a local Kafka instance you need to set the broker:
kcat -Ct orders -b localhost:9092
That's a pain. And if you want to access a remote machine it gets much worse:
kcat -Ct orders -b localhost:9092 -X security.protocol=sasl_ssl -X sasl.mechanism=PLAIN ...ugh...
But you don't have to do that. You can put it all in a
config file. It's a little bit of a tricky one to
get right though, so for your reference and mine, here's how you set
that up once and for all.
The file you need to edit is
For a local instance, it should contain something like:
For a remote server, you'll need to include the authentication details, so your
kcat.conf will look more like:
metadata.broker.list=<Bootstrap Server> security.protocol=sasl_ssl sasl.mechanism=PLAIN sasl.username=<API Key> sasl.password=<API Secret>
kcat.conf correctly set up, all the connection information
gets picked up every time you call
kcat, so that first, simpler
kcat -Ct orders should Just Work.
Update: After I wrote this up I found myself wishing I could do the same trick to connect to a Schema Registry. I'd like to be able decode some Avro values by simply saying:
kcat -Ct orders -s value=avro
Again, that's not too hard. You "just" need to cram the location of
the schema registry, and your authentication details, in the right
format under the right key. Here's the magic incantation to add to
schema.registry.url=https://<SR API Key>:<SR API Key>@<SR URL>
kcat will attempt to fetch schemas from the registry
whenever it needs a codec.