Debugging puppet queueing

Today we ran into a problem where the data put in ActiveMQ by the puppetmaster seemed corrupted in some way. When running the puppet queue daemon on the foreground (with –debug –verbose –no-daemonize), we noticed messages like these:

info: Loaded queued catalog in 22.16 seconds
debug: Searched for resources in 0.31 seconds
err: Could not save queued catalog for syntax error on line 68, col 34: `  serverversion: 2.7.6  sshdsakey: [long string]'
notice: Processing queued catalog for in 0.41 seconds

It seemed like for some reason there’s a newline missing there, but what exactly is it trying to do? It would be helpful if we could check the message in total, to see which resource is doing this. Python to the rescue!

On the machine that’s running the ActiveMQ, install python-stompy (we’re on Debian Squeeze). Open a python interactive shell and do this:

>>> from stompy.simple import Client
>>> stomp = Client()
>>> stomp.connect()
>>> stomp.subscribe("/queue/catalog")
>>> message = stomp.get_nowait()
>>> f = open("message","w")
>>> f.write(message.body)
>>> f.close()
>>> stomp.unsubscribe("/queue/catalog")
>>> stomp.disconnect()

You know have a file called message that contains the message. You might want to make the file a little bit easier to read by executing the following: sed -i 's/{/\n{/g' message, which adds a newline in front of each opening accolade. Now to search for the problem and the resource that causes it.

I hope this helps someone!

Tags: , , , , ,

Leave a Reply

Kumina helps companies innovate with the power of open source software. As specialists in managed IT operations since 2007, our mission is simple: building and managing the perfectly tailored technical infrastructures that allow our clients to thrive.

With fully managed solutions, we help our customers unlock the full potential of the cloud and Kubernetes. Our team also supports organisations with IT consulting and Kubernetes training courses. Learn more about our services or get in touch, we would love to hear about your business and projects.