Debug issues with supervisor not keeping service up

jeherve

Hi there!

I'm hoping some of you can help me with what seems to be me not understanding supervisor well enough. 🙂

Following some of the tutorials here, I rely on supervisor to run an instance of GoToSocial with Opalstack. It has worked really well up until recently. However, the service will now stop every other day, and I cannot seem to figure out why; I do not see anything suspicious in the logs, and do not understand why the service won't restart on its own.

Here is my supervisor.conf file:

[unix_http_server]
file=/home/my_user/logs/apps/my_app/supervisor.sock

[supervisord]
logfile=/home/my_user/logs/apps/my_app/supervisord.log
logfile_maxbytes=50MB
logfile_backups=10
loglevel=info
pidfile=/home/my_user/logs/apps/my_app/supervisord.pid

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///home/my_user/logs/apps/my_app/supervisor.sock
[program:gotosocial]
directory=/home/my_user/apps/my_app
command=/home/my_user/apps/my_app/gotosocial --config-path config.yaml server start
stdout_logfile=/home/my_user/logs/apps/my_app/my_app_gotosocial.log
stderr_logfile=/home/my_user/logs/apps/my_app/my_app_gotosocial.log
autorestart=true

Am I missing something obvious with my configuration, or is there some place other than /home/my_user/logs/apps/my_app/my_app_gotosocial.log where I could look for information?

Thank you!

sean

jeherve the problem is that your account is exceeding your RAM allowance. When this happens our quota monitor terminates all of your processes, including your supervisord instance.

I recommend running a cron job every minute which checks to see if supervisor is running and starts it if it is not. You can use a shell script like the following for this:

#!/bin/bash

pgrep -f supervisord > /dev/null || ~/.local/bin/supervisord -c /home/my_user/supervisor.conf

Save that in your bin directory, like ~/bin/cron_supervisor, then schedule it in your crontab like:

* * * * * ~/bin/cron_supervisor

Hope this helps 🙂

jeherve

Ah, thank you, that makes sense!

I suspect GoToSocial itself is the app causing the spikes. I'll look into it some more.

Until then, I've implemented your recommendation, as well as this one to get warnings when I'm about to blow past my RAM quota. 🙂