tl dr: I've fixed this problem for now by restarting nginx. Posting here in case anyone has any advice about how to automatically detect this problem so I can trigger a restart through CRON. Or should I just force a restart every so often regardless?
full version:
I set up a legacy ruby web app a couple of years ago using these instructions: https://community.opalstack.com/d/273-howto-run-old-or-new-rails-apps-with-rvm-nginx-and-passenger
I visited the app this week and it was 502.
My crontab autostarts the app if the TCP listener is absent
$ crontab -l
MAILTO=me@my.email
*/15 * * * * if [[ -z $(/usr/sbin/lsof -i TCP:<my_app_port>) ]] ; then /home/myuser/apps/myapp/nginx/sbin/nginx; fi
but listeners were present
$ lsof -i TCP:<my_app_port>
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 92861 myuser 7u IPv4 416546 0t0 TCP *:<my_app_port> (LISTEN)
nginx 92863 myuser 7u IPv4 416546 0t0 TCP *:<my_app_port> (LISTEN)
and the relevant processess were there
myuser 92861 0.0 0.0 54484 132 ? Ss Feb16 0:00 nginx: master process /home/myuser/apps/myapp/nginx/sbin/nginx
myuser 92863 0.0 0.0 54892 716 ? S Feb16 0:00 nginx: worker process
myuser 124946 0.0 0.0 112812 988 pts/30 S+ 15:01 0:00 grep -E --color=auto (myapp|nginx)
myuser 208564 0.0 0.0 583528 8436 ? Sl Jul06 0:00 Passenger RackApp: /home/myuser/apps/myapp/therubyapp
I looked up the the creation dates of the those processes in the nginx log:
[ 2023-02-16 11:45:03.6893 92817/7f4bc436f780 agents/HelperAgent/Main.cpp:554 ]: PassengerHelperAgent online, listening at unix:/tmp/passenger.1.0.92784/generation-0/request.socket
[ 2023-02-16 11:45:03.7271 92837/7f6e96333880 agents/LoggingAgent/Main.cpp:272 ]: PassengerLoggingAgent online, listening at unix:/tmp/passenger.1.0.92784/generation-0/logging.socket
...
[ 2023-07-06 13:02:45.6510 92817/7f4bc4248700 Pool2/Spawner.h:739 ]: [App 208401 stdout]
[ 2023-07-06 13:02:46.5430 92817/7f4bc4248700 Pool2/SmartSpawner.h:300 ]: Preloader for /home/myuser/apps/myapp/therubyapp started on PID 208401, listening on unix:/tmp/passenger.1.0.92784/generation-0/backends/preloader.208401
[ 2023-07-06 13:17:04.3972 92817/7f4bc4289700 Pool2/Spawner.h:739 ]: [App 255558 stdout]
[ 2023-07-06 13:17:05.2142 92817/7f4bc4289700 Pool2/SmartSpawner.h:300 ]: Preloader for /home/myuser/apps/myapp/therubyapp started on PID 255558, listening on unix:/tmp/passenger.1.0.92784/generation-0/backends/preloader.255558
The end of the log was a page access a few weeks before, followed by many undated lines like the one in the post title above:
[ 2023-08-20 22:31:44.0907 92817/7f4bc1907700 Pool2/Implementation.cpp:1156 ]: [App 208401 stderr] 207.46.13.231 - - [20/Aug/2023 22:31:44] "GET /robots.txt " 200 52890 0.0021
chdir("/tmp/passenger.1.0.92784") failed: No such file or directory (2)
chdir("/tmp/passenger.1.0.92784") failed: No such file or directory (2)
chdir("/tmp/passenger.1.0.92784") failed: No such file or directory (2)
...
...
Anyone have any advice? The app was running for a couple of years before this hiccough. It's legacy code so I don't think I could update any of the parts (nginx, passenger, ruby).
