On the systems we configure and maintain through Puppet, we sometimes see processes running for an excessive amount of time, causing the actual synchronization of system configuration to stall. To mitigate this, we have developed a script called checkpuppet.
The checkpuppet script obtains the process IDs from both agent.pid and puppetdlock. If the lock file is older than one hour, it kills the associated process. In addition, it scans the global list of processes to kill any Puppet instances not corresponding to the process IDs. Finally, if needed, it restarts the Puppet daemon in case it was not running. Summarized, it acts as a safety belt to reduce the likelihood of Puppet getting stuck, causing the system configuration to get out of sync.