502 Bad Gateway in Opal6

mattez

pjrobertson yeah same here, worst thing is that how you mentioned it is not reflected on status page! So bad. I just start to migrate from Webfaction and this is new experience for me. Webfaction was solid stable from my point of view.

dragan

mattez Same here. I have been using Webfaction for the last 6 years without a minute of downtime, now I have half of the day outage. (for now, who knows how long this will last)

Ariel

mikey3d opla6 is DE location, server is working (ssh, php+fpm+nginx, static sites), however Apache+PHP applications do not work. So, it's not an issue with Linode. It's something related to the Apache instance on this server as mentioned by @dolomiti.

mikey3d

I know why their status doesn't show the problem because it hosting on DigitalOcean.
https://status.opalstack.com/

Ariel

dragan @mattez I also moved from webfaction. In the past months with webfaction, I had multiple downtimes, at least one per month. They were always shorter than this one around 10 to 30 minutes, they had downtimes too 😰

I noticed them because I use Uptime Robot to monitor the sites, so each time one of them goes down I have an alert. If all of them are down it's a server issue.

mikey3d

dragan Not true, Webfaction does have downtimes you wouldn't even notice it because I use an Uptime Robot that notify me by email. I have been on Webfaction for over 6 years. It was down for 8 minutes and 39 seconds on Tue, Nov 10, 10:46 PM (6 days ago).

roadworn

please let us know, i have a dozen of clients are calling me and they are disappointed 🙁
this is so frustating
thanks from italy

sean

Sorry all, looking into this now.

roadworn

i started seeing the light! 🙂 anyone else?

erodri

Yep, back on track

elena

Working fine for me too, thank you Sean 🙂

tinygod

Also working on my end. 4 hours of downtime though...

erodri

Once everything is over, it would be nice to have an explanation from Opalstack about how this 4 hours long outage (during regular business hours) has occurred and if/which measures have been implemented so that it doesn't happen again.

sean

Hey gang, we're back. The Opal6 Apache httpd and PHP-FPM services were hitting an open files limit. We've raised the limit and the services are back online. Sorry for the disruption!

wweb

sean IMHO the main issue is not the technical one, but the human response time. I know Opalstack is young and still relatively small, and also we are in complicated times due to COVID. Still I think having 24/7 availability of an Opalstack sysadmin is crucial. I don't know how WF shifts where structured, but if I may suggest having at least two or three sysadmin living and working in different timezones would really make the difference in customer satisfaction and services reliability.

sean

wweb 24/7 coverage is the goal, we hope to get there soon.

wweb

sean Great, thank you for all the time and energies you are putting into Opalstack, you are doing a great job for a lot of people and I wish Opalstack will become bigger and greater than WF!

pjrobertson

Thanks Sean for fixing!

Really hope 24/7 coverage can come soon. Or at least better server monitoring - I'd say working out why this wasn't this reflected on the status page, and how the team can better be alerted should be a priority!

Keep up the good work, I know you guys are busy!

DavidUrban

Thanks for fixing it! @sean would it be possible to tell us what actions you took so the issue does not reoccur? Or happen on another server? Raising the limit sounds more like firefighting, than a real solution 🙂. I have utmost respect for your skills, I could not do what you do. But my clients are breathing down my neck and I would like to be sure the issues has been investigated and addressed 🙂. Hope you understand…

sean

DavidUrban Raising the open files limit for the httpd service was the real solution here. It was set too low for a multi-tenant service and raising it was something we should have done long ago.

The slow response from our side was a monitoring failure that we're working to address.

« Previous Page Next Page »