I've noticed for quite some time now that the access logs of my site is full of a certain type of requests that have a few things in common:
- The user-agent is always "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0", which is a very old browser from about 10 years ago.
- None of the request have a referrer, so they are always direct requests.
- All requests are to a certain section of my website, which happens to be an index page, requesting many different items from there.
- Often the same item is requested multiple times, over a period of time (unknown) sometimes millions of times.
- All requests come from different IPs, as far as I can see. Many appear to be Russian, but also French or German IPs. I can't make out a pattern here.
- The requests happen in rapid succession, about 200.000 times per day, over 8000x per hour, about 140x per minute (as shown by running this command:
less access.txt-20220403.gz | grep "Firefox/45.0" | wc -l
, or simply by tailing the active log)
A few seconds of "grepping the log's tail", looks like this:
tail -f access.txt | grep "Firefox/45.0"
51.38.169.208 - - [08/Apr/2022:20:19:02 +0000] "GET /url/1/ HTTP/2.0" 200 7052 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
5.135.209.59 - - [08/Apr/2022:20:19:02 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.255.146.8 - - [08/Apr/2022:20:19:02 +0000] "GET /url/3/ HTTP/2.0" 200 8135 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.38.169.225 - - [08/Apr/2022:20:19:03 +0000] "GET /url/4/ HTTP/2.0" 200 7040 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
213.32.94.187 - - [08/Apr/2022:20:19:03 +0000] "GET /url/4/ HTTP/2.0" 200 7040 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
77.222.103.13 - - [08/Apr/2022:20:19:03 +0000] "GET /url/5/ HTTP/2.0" 200 8052 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.77.237.254 - - [08/Apr/2022:20:19:03 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.33.19.60 - - [08/Apr/2022:20:19:03 +0000] "GET /url/6/ HTTP/2.0" 200 8452 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
54.36.140.130 - - [08/Apr/2022:20:19:04 +0000] "GET /url/1/ HTTP/2.0" 200 7051 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.65.176.81 - - [08/Apr/2022:20:19:04 +0000] "GET /url/3/ HTTP/2.0" 200 8135 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
176.116.186.151 - - [08/Apr/2022:20:19:05 +0000] "GET /url/7/ HTTP/2.0" 200 6106 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
5.196.154.196 - - [08/Apr/2022:20:19:05 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.33.17.187 - - [08/Apr/2022:20:19:05 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
37.187.189.30 - - [08/Apr/2022:20:19:06 +0000] "GET /url/4/ HTTP/2.0" 200 7042 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
46.105.224.159 - - [08/Apr/2022:20:19:06 +0000] "GET /url/6/ HTTP/2.0" 200 8452 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.33.217.184 - - [08/Apr/2022:20:19:06 +0000] "GET /url/8/ HTTP/2.0" 200 7061 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.255.0.66 - - [08/Apr/2022:20:19:06 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.38.169.241 - - [08/Apr/2022:20:19:06 +0000] "GET /url/3/ HTTP/2.0" 200 8135 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
46.105.224.153 - - [08/Apr/2022:20:19:07 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
151.80.92.217 - - [08/Apr/2022:20:19:07 +0000] "GET /url/8/ HTTP/2.0" 200 7061 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
137.74.101.170 - - [08/Apr/2022:20:19:07 +0000] "GET /url/1/ HTTP/2.0" 200 7051 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.33.19.61 - - [08/Apr/2022:20:19:08 +0000] "GET /url/4/ HTTP/2.0" 200 7042 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
151.80.170.195 - - [08/Apr/2022:20:19:08 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.255.146.8 - - [08/Apr/2022:20:19:08 +0000] "GET /url/8/ HTTP/2.0" 200 7061 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
164.132.9.105 - - [08/Apr/2022:20:19:09 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
54.38.114.251 - - [08/Apr/2022:20:19:10 +0000] "GET /url/2/ HTTP/2.0" 200 8136 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.38.169.240 - - [08/Apr/2022:20:19:10 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.255.99.254 - - [08/Apr/2022:20:19:10 +0000] "GET /url/6/ HTTP/2.0" 200 8453 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
5.135.209.59 - - [08/Apr/2022:20:19:11 +0000] "GET /url/1/ HTTP/2.0" 200 7052 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
151.80.179.179 - - [08/Apr/2022:20:19:12 +0000] "GET /url/6/ HTTP/2.0" 200 8456 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
151.80.92.22 - - [08/Apr/2022:20:19:12 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
95.174.118.225 - - [08/Apr/2022:20:19:13 +0000] "GET /url/2/ HTTP/2.0" 200 8134 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
54.36.138.235 - - [08/Apr/2022:20:19:13 +0000] "GET /url/8/ HTTP/2.0" 200 7060 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.38.169.244 - - [08/Apr/2022:20:19:13 +0000] "GET /url/3/ HTTP/2.0" 200 8135 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
54.36.138.234 - - [08/Apr/2022:20:19:13 +0000] "GET /url/9/ HTTP/2.0" 200 8485 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.77.240.172 - - [08/Apr/2022:20:19:14 +0000] "GET /url/1/ HTTP/2.0" 200 7051 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
91.227.189.151 - - [08/Apr/2022:20:19:14 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
164.132.221.30 - - [08/Apr/2022:20:19:14 +0000] "GET /url/4/ HTTP/2.0" 200 7041 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.33.217.190 - - [08/Apr/2022:20:19:14 +0000] "GET /url/4/ HTTP/2.0" 200 7041 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
51.38.169.226 - - [08/Apr/2022:20:19:15 +0000] "GET /url/3/ HTTP/2.0" 200 8135 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
217.107.124.171 - - [08/Apr/2022:20:19:15 +0000] "GET /url/6/ HTTP/2.0" 200 8453 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
217.182.38.239 - - [08/Apr/2022:20:19:15 +0000] "GET /url/9/ HTTP/2.0" 200 8486 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0"
(For privacy reasons I've obscured the url with a find/replace, but you can still see how often the same GET is performed in a time-frame of less than 15 seconds.)
The fact that it's always direct requests from the same user-agent makes me thinks it's some type of crawler, but usually the user-agents of those are more descriptive. It's definitely some bot, but I cannot find anything useful on this particular user-agent.
I'm not sure what to think about this or what to do about it, if anything.
Any idea what this could be?