Using local log analysis tools to avoid Google Analytics export headaches?

aa11

Hi,

tl;dr

I (any many others) need to archive legacy Google Analytics Universal data because it's shutting down on July 1, 2024. But doing so is a nightmare.

I've been wanting to switch to locally hosted access_log analyzer tool anyway.

I realized that I could maybe skip the whole "exporting data from GA and then figuring out a way to visualize it" nightmare and instead use a locally hosted stats tool that would both use the current access_logs and import the legacy access_log data that I have.

Can anyone help?

Overview

I started using Google Analytics because it was easy to setup and provided a lot of information. But I never loved using it for all of the obvious reasons (Google controls it, somewhat locked-in data, opaque, etc.). It was helpful to have an off-site tool during the the transition from WebFaction to Opalstack, because it continued to provide analytics without any extra work (since there isn't yet any built-in log analyzers at OS).

Starting on July 1, 2024, legacy Google Analytics Universal data (IOW: Pre-GA4) will become inaccessible and then purged.

Apparently there's no straightforward way of exporting ALL of the GA data, let alone visualize that exported information in any meaningful way. (You can export PDFs of the Dashboard visualizations, but you can only do that one view at a time.)

There's lots of blog posts about this and various companies selling their services to facilitate different styles of exports, imports, and visualizations of this data. I can post some of those links in a follow-up post if necessary.

Some methods and services seem better than others and most lock you into their own monthly subscription fees*. But I'm not confident in any of them without doing a lot of research into each.

This is one of the exceptions: Analytics Canvas - Backup Universal Analytics Data with our UA Backup Utility

And then I realized:

I think one of the factors that a lot of these services are relying on is many people (most?) who use GA do not have access to their raw server access_logs* (error_logs would be good too).

But we (Opalstack customers) do!

By default we have the server logs for the past year and I have a script running that archives each year's data. It even stretches back to logs generated at WebFaction.

(*I realize that the server's access_logs do not capture some of the data that GA does. Namely all of the frontend stuff that their Javascript tracker can.)

So my questions are:

Does that sound correct?
- That I can essentially forgo all of this craziness of trying to download historical GA data because I/we already have it (for the most part) via the server's access_logs.
- Or am I missing something very important? (No, I don't care much about Google Ad Sense and how it correlates with Google Analytics.)
Does anyone here have experience with any of the various server log analysis tools?

But mostly: Importing legacy apache_access.log style file into them, in order to create a local GA replacement that would include both the historical AND current data?

Some of the tools:

https://www.goatcounter.com/help/logfile
https://goaccess.io/
Matomo (web analytics) - Opalstack Community Forum (👋 @scp)
https://github.com/hyc/webalizer (Old standard)
https://www.awstats.org (Old standard)
(These have been mentioned by others in this forum before.)

Thanks!!

These OS Community posts were helpful in my research:

sean

I'm partial to goaccess myself. It's already installed on our servers and doesn't require you to "import" anything. You can just invoke it with whatever log file (or even multiple files simultaneously) and then view the stats. You can also use to monitor live logs to see the stats moving around in real time.

The command to use is (assuming your site name is "mysite", modify as needed):

goaccess --log-format=COMBINED ~/logs/sites/whatever/access.log

... or if you want to view an older compressed log:

zcat ~/logs/sites/whatever/access.log-20240615.gz | goaccess --log-format=COMBINED -f -

You'll see your stats in your terminal.

If you want to generate a static HTML report for your analytics the command is (assuming you have a static app named "static" attached to a site, modify the app name as needed):

goaccess --log-format=COMBINED -o ~/apps/static/report.html ~/logs/sites/whatever/access.log

You can then go to /report.html on your site to see the results.

If you wanted a weekly or monthly report the command is essentially the same, except you'll change the file argument at the end of the command to include all of the logs that you want to parse simultaneously.

This won't have the breadth of info that GA gives you but IMO it's a solid choice for basic analytics.

davidmiles

sean Thanks for this Sean! 😁 I was actually just about to go through the process of building GoAccess from source and trying to figure out a way to easily set it up for all of my sites (which would be a bit of a headache since they're all under different shell users, and I only wanted to build from source once). Then thought I'd search to see if anyone else had set up GoAccess on Opalstack...

So this should help immensely!

FYI, you can also use the following to generate a report based on all of the logs present (access.log as well as any access.log-YYYYMMDD.gz files):

zcat -f ~/logs/sites/{site-name}/access.log* | goaccess --log-format=COMBINED

zcat -f ~/logs/sites/{site-name}/access.log* | goaccess --log-format=COMBINED -o ~/apps/{app-name}/report.html

Also, if you're using Apache then you will want to use your respective app's apache_access.log, which would be located at:

~/logs/apps/{app-name}/apache_access.log
~/logs/apps/{app-name}/apache_access.log-YYYYMMDD.gz

aa11

Thank you @sean

I'll look into this and post questions here.

For anyone else that's in need of a relatively painless, automated, and ok-priced GA 3 data exporter (in the next week), these are the two best I've found:

Export your GA3 / Universal Analytics Data with GA3-Exporter
- ~$80/property
Backup Universal Analytics Data with our UA Backup Utility
- ~$100-200 for up to 10million rows (but seemingly unlimited properties)