Too many connections errors after switch to PHP-CGI

sean

igor hmm, will pass that along to the sysadmin.

sean

etienneh great suggestions, thanks! and I hear on you the frustration.

igor

@sean Are you maybe creating backups or some other DB maintenance at about 22:00-22:20 UTC? 🙂

I just witnessed active connections skyrocket from 180 to 250 (and too many connections) in two seconds, after that it plummeted to around 20 connections and stayed there. I assume it started at around 20-40 and pushed to 250 really fast. Seems to happen at approximately the same time every day.

etienneh

igor can corroborate that my monitoring alert fires at 2PM PST (UTC-8) consistently.

sean

igor etienneh yep, earlier today we determined that the backups seem to line up with most of these problems. We're looking into ways to reduce the impact that the backups have.

etienneh

sean any news? :-)
Still getting daily crash reports and downtime alerts

sean

etienneh still working on it. It's likely we'll have to stop backing up the largest DBs, will notify the affected customers if it comes to that.

etienneh

sean would it be possible to backup with a jitter? Each new dB gets a random timeslot in which the backups occur?

etienneh

sean Any news? 🙂

sean

etienneh I don't think the current backup setup can do that, but I'll pass the suggestion along.

etienneh

And thoughts on providing a quick installer for a private instance or a migration script from/to a private instance?

Still getting daily crashes and downtime alerts 😉

etienneh

Actually @sean I tried to do the migration myself and I'm still getting errors on the private instance. So something must be wrong that's not adding up?

sean

etienneh sorry, no migration script or installer available yet. We're still working on improving the backups.

System load was not high during the backups today (a few minutes ago) so your own private instance should not have been affected by that. Feel free to email support with the details about your private instance whatever apps are using it so we can look into it further.

etienneh

sean Ah looks like I forgot to swap my django connection from 'localhost' to my socket 😅 - will post on the other forum the issue I had.

sean

etienneh we're working on a new backup routine which we hope will mitigate this issue.

etienneh

How is the script? :-)

sean

etienneh still in progress, sorry for the delays with this.

etienneh

Any news? 🙂

sean

etienneh Not yet, will be sure to post here when there is.

sean

etienneh We've rolled out a new mariadb backup system on opal6 earlier today.

When the backups ran at 22:00, we saw no lock errors in the system mariadb logs, whereas before there would usually be a couple hundred of them. System load stayed normal during the backup as well.

Please let me know if you notice an improvement on your end.

kierenp

sean Many thanks for this update. The last errors I saw were the 25th Feb so the changes on the 26th seem to have done the trick for me on opal6. Many thanks!

« Previous Page