502 Bad Gateway in Opal6

elena

Working fine for me too, thank you Sean 🙂

tinygod

Also working on my end. 4 hours of downtime though...

erodri

Once everything is over, it would be nice to have an explanation from Opalstack about how this 4 hours long outage (during regular business hours) has occurred and if/which measures have been implemented so that it doesn't happen again.

sean

Hey gang, we're back. The Opal6 Apache httpd and PHP-FPM services were hitting an open files limit. We've raised the limit and the services are back online. Sorry for the disruption!

wweb

sean IMHO the main issue is not the technical one, but the human response time. I know Opalstack is young and still relatively small, and also we are in complicated times due to COVID. Still I think having 24/7 availability of an Opalstack sysadmin is crucial. I don't know how WF shifts where structured, but if I may suggest having at least two or three sysadmin living and working in different timezones would really make the difference in customer satisfaction and services reliability.

sean

wweb 24/7 coverage is the goal, we hope to get there soon.

wweb

sean Great, thank you for all the time and energies you are putting into Opalstack, you are doing a great job for a lot of people and I wish Opalstack will become bigger and greater than WF!

pjrobertson

Thanks Sean for fixing!

Really hope 24/7 coverage can come soon. Or at least better server monitoring - I'd say working out why this wasn't this reflected on the status page, and how the team can better be alerted should be a priority!

Keep up the good work, I know you guys are busy!

DavidUrban

Thanks for fixing it! @sean would it be possible to tell us what actions you took so the issue does not reoccur? Or happen on another server? Raising the limit sounds more like firefighting, than a real solution 🙂. I have utmost respect for your skills, I could not do what you do. But my clients are breathing down my neck and I would like to be sure the issues has been investigated and addressed 🙂. Hope you understand…

sean

DavidUrban Raising the open files limit for the httpd service was the real solution here. It was set too low for a multi-tenant service and raising it was something we should have done long ago.

The slow response from our side was a monitoring failure that we're working to address.

leehinde

re-upping this. I have an email into support.

I have two 'sites' pointing to the same django app/code. One was a domain I controlled that I used for testing, before cutting over to the live site. The 'real' site is currently down with a 502 error, but the 'testing' site is still working. Both point to the same code base.

Not sure what to do other than wait for support.

Googling the errors implies an nginx issue, which I don't think I have any influence over.

upstream prematurely closed connection while reading response header from upstream,
no live upstreams while connecting to upstream, client:

leehinde

leehinde sort of might be expected, now both sites are down.

sean

leehinde sorry for the support response times, I'm looking into your issue now.

Sleckie

I'm getting 502 Bad Gateway on Opal3. Yesterday, tuesday 12/8 allday as well.

sean

Sleckie please email support, thank you.

sean

The subject of this post was an incident on opal6 that happened three weeks go. The incident was resolved a few hours later so this topic is now closed.

« Previous Page