Auto-restart supervisorctl

Sometimes queue jobs fail to restart for various reasons. We don't get notifications from Forge when this happens, so we need to ensure that queues are running without any intervention.

This script should run on a cron every minute, and will restart any supervisorctl processes that have failed and not restarted.

Resources used

Steps

  1. Add the script below to the repository under the scripts dir, and call it restart-supervisorctl.sh
  2. Ensure it is executable: chmod +x scripts/restart-supervisorctl.sh
  3. Navigate to the "Processes" tab for the relevant server in Laravel Forge
  4. Click the "Scheduler" menu item
  5. Click "+ Add Scheduled Job"
  6. Enter these details:
    • Name: Restart Supervisorctl
    • Command: export SITE_NAME="<site.host.name>" && /home/forge/<site.host.name>/scripts/restart-supervisorctl.sh
      • Example: export SITE_NAME="staging.kooimaag.com" && /home/forge/staging.kooimaag.com/scripts/restart-supervisorctl.sh
    • User: forge
    • Frequency: Every minute
  7. Click "Create Scheduled Job"
  8. After a minute or so, confirm that the job has run. It should output "No failed processes".

Script

#!/bin/bash
set -e

failed=$(sudo supervisorctl status | awk '$2=="BACKOFF" || $2=="FATAL" {print $1}')

if [[ -n $failed ]]; then
  echo "Restarting failed processes"
  echo "$failed" | xargs -r -n1 sudo supervisorctl restart
  /usr/bin/php /home/forge/${SITE_NAME}/craft queue/retry all
else
  echo "No failed processes"
fi

Note that if you have custom queue names, you will need to update the queue/retry part, or add additional retry commands, for example:

...
  /usr/bin/php /home/forge/${SITE_NAME}/craft queue/retry all
  /usr/bin/php /home/forge/${SITE_NAME}/craft my-custom-queue/retry all
...