uwsgi with Django: bind(): Address already in use

stephan0h

Having an issue here with Django and uwsgi: it seems the service is restarting from time to time. Got this snippet from the log right when we were seeing an error 502:

[pid: 7783|app: 0|req: 187/363] 127.0.0.1 () {48 vars in 1250 bytes} [Mon Dec 20 10:10:14 2021] GET /newsletter/media/png/COM_MO-M9-2-9-AT10-C0-R0-P0N0-2021-12-01-2022-11-30-385310.png => generated 34655 bytes in 158 msecs via sendfile() (HTTP/1.1 200) 6 headers in 248 bytes (0 switches on core 0)
*** Starting uWSGI 2.0.20 (64bit) on [Mon Dec 20 10:13:02 2021] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-44) on 21 November 2021 19:33:17
os: Linux-3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
nodename: opal5.opalstack.com
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/flylittlebird
writing pidfile to /home/flylittlebird/apps/fly_uwsgi/tmp/fly_uwsgi.pid
detected binary path: /home/flylittlebird/apps/fly_uwsgi/env/bin/uwsgi
chdir() to /home/flylittlebird/apps/fly_uwsgi/theApp
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 4096
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
probably another instance of uWSGI is running on the same address (127.0.0.1:42786).
bind(): Address already in use [core/socket.c line 769]
*** Starting uWSGI 2.0.20 (64bit) on [Mon Dec 20 10:23:02 2021] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-44) on 21 November 2021 19:33:17
os: Linux-3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020
nodename: opal5.opalstack.com
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/flylittlebird
writing pidfile to /home/flylittlebird/apps/fly_uwsgi/tmp/fly_uwsgi.pid
detected binary path: /home/flylittlebird/apps/fly_uwsgi/env/bin/uwsgi
chdir() to /home/flylittlebird/apps/fly_uwsgi/theApp
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 4096
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:42786 fd 3
Python version: 3.9.7 (default, Nov 21 2021, 17:57:12)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
PEP 405 virtualenv detected: /home/flylittlebird/apps/fly_uwsgi/env/
Set PythonHome to /home/flylittlebird/apps/fly_uwsgi/env/
Python main interpreter initialized at 0x1b8f900
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 274704 bytes (268 KB) for 4 cores
*** Operational MODE: preforking+threaded ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1b8f900 pid: 4956 (default app)
mountpoint  already configured. skip.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 4956)
spawned uWSGI worker 1 (pid: 5087, cores: 2)
spawned uWSGI worker 2 (pid: 5096, cores: 2)
[pid: 5096|app: 0|req: 1/1] 127.0.0.1 () {62 vars in 1384 bytes} [Mon Dec 20 10:27:40 2021] GET /newsletter/chart-list/ => generated 176370 bytes in 4939 msecs (HTTP/1.1 200) 6 headers in 183 bytes (1 switches on core 0)
[pid: 5096|app: 0|req: 3/2] 127.0.0.1 () {60 vars in 1505 bytes} [Mon Dec 20 10:27:45 2021] GET /newsletter/media/png/CAL_SY-2-10-AT10-C0-R0-P0N0-2021-12-15-2022-12-15-141821.png => generated 20379 bytes in 179 msecs via sendfile() (HTTP/1.1 200) 6 headers in 246 bytes (0 switches on core 1)
[pid: 5096|app: 0|req: 4/3] 127.0.0.1 () {60 vars in 1514 bytes} [Mon Dec 20 10:27:45 2021] GET /newsletter/media/png/CAL_MO-M1-2-10-AT10-C0-R0-P0N0-2021-12-15-2022-12-15-141821.png => generated 49626 bytes in 157 msecs via sendfile() (HTTP/1.1 200) 6 headers in 249 bytes (0 switches on core 0)
[pid: 5096|app: 0|req: 5/4] 127.0.0.1 () {60 vars in 1514 bytes} [Mon Dec 20 10:27:45 2021] GET /newsletter/media/png/CAL_MO-M3-2-10-AT10-C0-R0-P0N0-2021-12-15-2022-12-15-141821.png => generated 64298 bytes in 152 msecs via sendfile() (HTTP/1.1 200) 6 headers in 249 bytes (0 switches on core 1)

uwsgi.ini looks like that:

[uwsgi]
chdir=/home/flylittlebird/apps/fly_uwsgi/theApp
master = True
http-socket = 127.0.0.1:42786
virtualenv = /home/flylittlebird/apps/fly_uwsgi/env/
daemonize = /home/flylittlebird/logs/apps/fly_uwsgi/uwsgi.log
pidfile = /home/flylittlebird/apps/fly_uwsgi/tmp/fly_uwsgi.pid
workers = 2
threads = 2

buffer-size=8192
ignore-sigpipe=true
ignore-write-errors=true
disable-write-exception=true
socked-timeout=200

# adjust the following to point to your project
wsgi-file = /home/flylittlebird/apps/fly_uwsgi/theApp/theApp/wsgi.py
module=theApp.wsgi:application
touch-reload = /home/flylittlebird/apps/fly_uwsgi/theApp/theApp/wsgi.py

It seems to be an open files issue - or is it something with the uwsgi configuration?

Thanks,
Stephan

sean

stephan0h installed via the dashboard include a cron job that runs the app's "start" command every 10 minutes.

It's not RE-starting the app, it's only running the start command.

The "address already in use" message is logged when the start command is run while the app is already running. It's not related to a 502 error.

Your 502s are happening because you're exceeding your RAM allowance. When this happens our quota monitor will terminate your processes automatically.

stephan0h

sean Oh, this I didn't check! Thanks for the hint.
Actually I couldn't find my current settings (Ram, bandwith etc.) in the dashboard - where is this information hiding? Furhermore a short notice that I ran out of RAM would be helpful - of did I overlook it in the log? But these would be minor enhancements - still very content with your service!
Stephan

sean

stephan0h Your RAM and disk usage are available on the front page of the dashboard at https://my.opalstack.com/. Bandwidth statistics are not available at this time.

We're not sending notifications for RAM overages at this time but we plan to add that in the future.