Managing Processes with Supervisor - In Depth Tutorial | CS Journey

In this post I’ll show you how to setup Supervisor on Linux and become familiar with a few handy options and settings. If you haven’t used Supervisor in the past this post should give you a decent start.

Supervisor is a Python program that makes managing other programs (processes) a breeze. It can ease the process of starting, killing and restarting applications that you develop. I’ll be explaining various aspects that you need to keep in mind while using Supervisor.

Basics

Supervisor can help you administrate and automate the process of starting applications on your web server. It can run the steps specified by you and start multiple applications. Maybe you want somebody to automatically start the application after you’ve restarted your server. Supervisor can do all that and more.
Supervisor starts your application or script as it’s own child process. This way it can kill them when you want to. It daemonizes something that runs infinitely say a web server. You should not use it to control a process which has a definite end (say a task which takes 3 minutes and exits after that).
The command to start a particular process and various options are given via a .conf file.
Supervisor looks for files ending with .conf in certain directories. It then follows the instructions that you’ve given in those files.
Once a proper .conf file is written, starting any program is as simple as
supervisorctl start appname.
It daemonizes the process that you want to run. You can save logs that your application spits out in files.
Say your server runs 50 processes. It would be a pain to start every one of them whenever you restart the server. Supervisor has an autostart option which will start the process when Supervisor is initialised.
You could make Supervisor run a program as a particular user. There are loads of such options that can help make your day to day life easier.

Installing Supervisor

On Ubuntu or debian you can
```
sudo apt-get install supervisor
```
You can also install it using pip. Supervisor on Pypi.

Writing conf Files

Supervisor can be informed of any program that you wish to run via a .conf file.

I found this example from Digital Ocean to be very helpful. So I’m adapting a similar approach but with a different script. For the sake of this example and brevity we’ll write a small script that displays the amount of RAM that the machine sports.

Go to your home directory (~/) and save the following as displayram.sh

#!/bin/bash

counter=0
while true
do
    ((counter++))
    echo "Count is now $counter."
    cat /proc/meminfo | grep MemTotal
    sleep 1
done

Do note this is a never ending process. It runs forever because of the while loop. I repeat Supervisor can only manage and monitor never ending processes.

Make the file you just created executable.

chmod +x displayram.sh

Save the following as displayram.conf. Supervisor will use this file to control the process. Substitute abhirath with your username.

[program:displayram]
command=/home/abhirath/displayram.sh
autostart=false
autorestart=false
stderr_logfile=NONE
stdout_logfile=/home/abhirath/displayram.out.log

Move the file that you just created to /etc/supervisor/conf.d/displayram.conf. This is the default directory that Supervisor looks for programs.

Then on the terminal

supervisorctl reread

followed by

supervisorctl update

Finally run

supervisorctl start displayram

Our script starts instantly. You can verify that it works by opening /home/yourusername/displayram.out.log. It should be filled with lines displaying the RAM installed on the machine.

A better way would be to run the following on the terminal.

tail -f /home/yourusername/displayram.out.log
# or
tail -f ~/displayram.out.log

It will keep printing out new entries on the terminal.

You will also need to familiarise yourself to spawn more than one process - An example which requires running multiple processes to start an app.

To stop the script we just started run

supervisorctl stop displayram

A Few Options Explained

Command

The way via which Supervisor starts a particular program. Give the command / path that you normally use to start your program.
Commands are not equivalent to bash or your typical terminal commands by default. They are fairly similar but a command that you use on your terminal may not yield the same effect here.
Commands can contain absolute paths or relative paths. The displayram example uses absolute paths. If you want to use relative paths you need to set the path. You can read more about it here.
You can ensure that your command is treated as a bash command by using bash -c "command here". Our earlier example can be rewritten as

[program=displayram]
directory=/home/abhirath
command=bash -c "./displayram"
autostart=false
autorestart=false
stderr_logfile=NONE
stdout_logfile=/home/abhirath/displayram.out.log

Stop and Kill as Group

Sometimes your program doesn’t really get killed. They might have children that they don’t kill. So to actually stop them Supervisor has to be mean and kill every process that might have spawned. One possible reason is that they expect a different kill signal.
You don’t have to use them unless you notice that your program is still running even after you explicitly ran supervisorctl stop programname
This helped me a lot - Really killing a program.

Logs

stderr_logfile - the file to which it shall redirect stderr to. If set to NONE it will not log output from stderr.
stdout_logfile - the file to which it redirect stdout to. This can also be set to NONE.
stderr_logfile_maxbytes - maximum space that the log file will take before Supervisor uses a new file to store logs. You can input a numerical followed by GB, KB etc. By default it’s set to 50MB.
stdout_logfile_maxbytes - same as above but for stdout.
stdout_logfile_backups - the number of backup files it stores. Say you’ve set it to 3. After the first files exceeds the value of stdout_logfile_maxbytes it creates a second file then a third. If the third file exceeds the size it proceeds to delete the first file and continue logging. In all it would use a maximum of 3 * 50MB in our case. By default it has a value of 10.
stderr_logfile_backups - same as above but for stderr.
An example which uses a variety of log settings.

Supervisor saves it’s own logs as well. They are different from the logs you set for your program. These logs contain information about Supervisor. They are stored in /var/log/supervisor/supervisord.log To see the last few logs you can tail /var/log/supervisor/supervisord.log

A useful read - Options available under program.

Some Common Errors

1. You get a message similar to unix:///var/run...sock no such file?

It means Supervisor isn’t running. You will have to start Supervisor. sudo service supervisor restart

2. Spawn errors. Supervisord is unable to start your program?

Check the error log and output logs. Supervisor keeps appending stuff to your program’s log file. So it’s a good idea to use tail -f /path/to/program/log/file to keep track of changes happening.
Check the environment and directory options under your program. Give absolute paths like /home/abhirath/so/and/so.

3. Supervisor doesn’t start automatically after bootup?

On Ubuntu 16 you can enable it by sudo systemctl enable supervisor. Also read this stackexchange question. - init scripts for various operating systems.
Frequently new .conf files that you write are not recognised. If that is the case do the following:- Check the syntax of everything in the file. If there’s an error Supervisor doesn’t even show the file. Also the files must have an extension of .conf.

4. Any signs of the process / app that you want to start is already running?

They might already be running. Check the status using supervisorctl appname status.
I’ve seen .NET apps that don’t kill their child processes by themselves. So even after you stop a program their child process might be running. For instance you might get errors like port is already in use.
Read about stopasgroup, killasgroup and stopsignal over here
stopasgroup, killasgroup work on individual programs. They are not to be confused with group applications
Really killing a program.

Managing Processes with Supervisor - In Depth Tutorial