Access Logs: Rotation Schedule?

aa11

Do logs (sites and apps) follow the same rules and scheduling as frontend and user logs did at WebFaction?

To prevent log files from becoming too large, logs are rotated daily at 3:00 a.m. server time. Each existing log is renamed by appending a number which indicates how many days old the log is. For example, error_website_name.log is named error_website_name.log.1 initially. Each subsequent day the log is incremented by one, until the log has aged seven days, upon which it is deleted.

https://docs.webfaction.com/software/general.html?highlight=logs#accessing-logs

I wasn't able to find any "log" specific help documentation at: https://help.opalstack.com

Thanks!

sean

aa11 Our log rotation works the same way except:

We compress the rotated logs
The rotated log files have dated filenames like nginx_access.txt-20201027.gz

etienneh

sean Does this apply blindly to any .log file in the logs directory? Or Only ones which opalstack has created.

Thinking for convenience sake of logging in that directory, but want to know If should set up my own rotation/compression jobs.

aa11

Ohhh, that sounds much better.

On WF I had put together a few shell scripts, and scheduled them via cron, to catenate and archive these so I didn't lose logs older than 7 days. It was a whole thing to setup (largely because of the file naming), but it seems to have been working well ever since. You (@sean) even helped create one of the scripts! (Thanks again!)

A few questions (just to make sure):

So these nginx_access.txt-20201027.gz files still get deleted after 7 days?
If I wanted to retain ALL log files (long term archiving), I guess it would be pretty simple to setup a cron job to copy each to another directory?
- ex: Run once per/day: cp *_access.txt-*.gz /archive and cp *_error.txt-*.gz /archive
Hum... I might need to get a little cleverer to account for files that have already been copied. Or I guess just letting cp overwrite into /archive would be ok??

I know (and will) try some of this myself. But I don't have enough logs yet, and it'll take at least 7 days to see complete a cycle. Which is why I'm asking now, instead of just doing it :-) Also, it might help others. Speaking of:

+1 Request for all of these details to be added to: http://help.opalstack.com (No rush :-)

Thank you!

sean

aa11 You (@sean) even helped create one of the scripts! (Thanks again!)

Heh, I totally remember that 😆

Yes, rotated logs get deleted after 7 days.
You can't copy/move stuff into /archive but maybe you meant ~/archive?
If you use rsync that will deal with existing files automatically, I think rsync -Pa ~/logs/ ~/archive should be sufficient to catch everything. Letting cp overwrite would be OK too I guess.

aa11

sean

Heh, I totally remember that 😆

Aww man. Bad times 😂

Yes, rotated logs get deleted after 7 days.

👍

You can't copy/move stuff into /archive but maybe you meant ~/archive?

Yes. (+100 Duh Points for me)

If you use rsync that will deal with existing files automatically, I think rsync -Pa ~/logs/ ~/archive should be sufficient to catch everything. Letting cp overwrite would be OK too I guess.

Ohhhhh. rsync is cool idea too.

Thanks!!

aa11

sean If you use rsync that will deal with existing files automatically, I think rsync -Pa ~/logs/ ~/archive should be sufficient to catch everything.

Follow-up:

I setup cron to run rsync once per/day.

It works great! .gz log files accrue in the archive directory as expected.

Thanks Sean!