So, I bumped into this thing called “shovel shape”, and I was like, “What the heck is that?” I gotta move some data between these message queues, RabbitMQ and Kafka, ’cause reasons. I did some digging, and it turned out to be a pretty neat solution. Here’s how I got it working:
Figuring Out the Setup
First, I needed to get my hands dirty with the RabbitMQ side of things. I already had RabbitMQ and Kafka up and running, I made sure that. Then I checked the RabbitMQ management UI. It’s usually on port 15672, you know?
I poked around and found that shovel thingy. It’s in the “Admin” section, under “Shovel Management”. Clicked on that, and it’s pretty straightforward from there.
Making the Shovel
To create a shovel, I clicked “Add a new shovel”. Gave it a name – something simple like “rabbit-to-kafka”.
- Source: Picked “AMQP 0.9.1” (that’s RabbitMQ’s protocol). Typed in the URI of my RabbitMQ queue. Something like
amqp://user:password@rabbitmq-host:5672. Don’t forget to change the details of your actual setup! Then entered the name of my source queue in the “queue name” field. - Destination: This is where Kafka comes in. Selected “AMQP 1.0” (because Kafka plays nice with that). The URI looked like this:
amqp://user:password@kafka-host:9092. Again, Used my actual Kafka broker details, and put the Kafka topic name under “Address”.
The Moment of Truth
Once I had all that filled in, I double-checked everything and clicked “Add Shovel”. Boom! It should have started working immediately. I was a little surprised, thought it would be more complicated than it had.
Watching It Work
Now, to make sure it’s not just sitting there, I went back to the “Shovel Status” page. It showed my newly created shovel, and hopefully, it says “running”. If it does, that means messages are flowing from RabbitMQ to Kafka!
Testing and Tweaking
Of course, I had to test it. I sent some messages to my RabbitMQ queue and then checked my Kafka topic to see if they arrived. Sure enough, they were there! Pretty smooth.
I played around with a few settings, I’ll be frank here. Things like prefetch count (how many messages the shovel grabs at once) and acknowledgement modes. Just to see if I could make it even faster. Because you never know what the needs of tomorrow will be, right?
And that’s pretty much it! It wasn’t rocket science, thankfully. Now I can move data between RabbitMQ and Kafka whenever I need to, all thanks to that shovel thingy. A small feature but helped me achieve what I needed to get done!
