Forum Moderators: open

Message Too Old, No Replies

Possibly malicious traffic of impatient .bots? humans?

         

1script

11:55 pm on Dec 30, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Seasons Greeting, fellow netizens!
I am seeing thousands of requests, most of which look exactly the same in the logs:
It comes in to the homepage of the site, always without a referrer, always some version of iPhone, and it almost always breaks connection before Nginx can send anything back (hence the Nginx specific 499 HTTP error code).

Has anyone seen this or can offer some sort of an explanation for what this traffic can be? These are IPs that belong to various ISPs, so they can potentially be real users (tho can't tell if their PCs are hijacked or something). The requests are a nuisance, but they are not massive enough to be a DDoS attack. If it were a bot, it does not get any content to chew on (size of response - 0). Could it be a legit use, some sort of prefetch or some such? I don't have an iPhone to try that theory on ...

Anyhow, will appreciate any help with making sense of this. Sample logs below.

And Happy New Year 2021. Can't wait to get rid of this one!


94.4.208.89 - - [30/Dec/2020:23:35:23 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"
199.188.255.28 - - [30/Dec/2020:23:35:29 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"
108.249.18.37 - - [30/Dec/2020:23:35:42 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"
98.229.72.244 - - [30/Dec/2020:23:36:21 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"
98.229.72.244 - - [30/Dec/2020:23:36:25 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"
24.114.93.143 - - [30/Dec/2020:23:36:36 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Mobile/15E148 Safari/604.1"
174.226.7.114 - - [30/Dec/2020:23:36:47 +0000] "GET / HTTP/1.1" 499 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Mobile/15E148 Safari/604.1"

phranque

1:04 am on Dec 31, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



i would look for and analyze other visits by some of those IP/UA combinations to see if there are other clues.

for example, it may be that the home page was slow to load and the visitor bailed or refreshed the page before the response was sent.
in other words, the client error status code (an "impatient" 499) may be hiding an underlying issue with the server (a potential slow-responding 5XX).

graeme_p

2:34 pm on Dec 31, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Successive requests from different IPs in different locations and belonging to different ISPs but with the same UA seems suspicious to me.

I once got a lot of traffic that looked malicious. It was in intent, but the attacker had misconfigured their bot so badly it just made pointless malformed requests. I wonder whether this could be something similar

1script

7:51 pm on Dec 31, 2020 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thank you for your input, guys. These IPs do not seem to generate any other activity, so they do seem malicious in intent.

In fact, I started looking into this issue when I discovered a strange spike in AdSense earnings coming specifically from the home page. This may be a part of it. Unfortunately, it is hard to say if these are simply misconfigured bots that give up faster than my server can return a response or if they get some HTML but intentionally do no acknowledge, and confuse Nginx. Unfortunately I can't find anything in Nginx docs to confirm that the zero size of the response means that the client didn't get anything or if it can also mean that Nginx does not know if it did, and perhaps it already has the full HTML of the page. The former case would be a simple nuisance, the latter may be pretty darn dangerous.

Can anyone suggest a strategy for dealing with this kind of a situation?

Thank you for all your input and ideas!

phranque

1:20 am on Jan 1, 2021 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Can anyone suggest a strategy for dealing with this kind of a situation?

a 4XX response (the client error class of status codes essentially means "go away and fix your request") and zero content returned...
what else could you ask for?

tangor

4:07 am on Jan 5, 2021 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



what else could you ask for?

Agreed--just as rude or bad guests aren't allowed in my living room, these kind of visitors (bot or not) are shown the door.

Either one I do not lose any sleep over it.

On the other hand, if the activity is consistent, abusive, and truly annoying they get a place of honor in my .htaccess DENY and ignored from then on.

(my 403 return is only 228 BYTES)

1script

5:34 am on Jan 14, 2021 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought more info for the context may be useful because this issue has not really fixed itself, despite being just a nuisance and on the surface of it simply clogging the logs with unnecessary things.

I have started to investigate things when I noticed huge spike in AdSense ads revenue +300%, all invalid obviously, confirmed by Google. Google being Google could not disclose any useful info about the bad traffic but I was able to confirm that a lot of my UK traffic is invalid. Like, 75% of it. So, I did some log digging, banned some obvious bots coming from UK/France/Germany hosting services and the first wave subsided.

Couple days ago it started again. UK once again. I have realized that there are more hosting providers with lax use policies in Europe than grains of sand on the beaches of Normandy, and started looking for patterns harder. And I think there is a pattern that these pesky requests ending up with 499 errors are a part of:

I believe they are designed to drown out in logs traces of the actual nefarious activities, i.e. clicking on ads on my site for the only reason that I can see - have me banned on AdSense(?) . Perhaps something else, who knows what. They are probably someone's PCs taken over (99% of them have iPhone user agent but that's easy enough to fake) and they are, in fact being impatient by design, I think - to generate a hit and a log file line, but not create too much network activity by actually receiving content. Sort of like DDoS but for logs. They would have been 200 HTTP responses on Apache (please correct me if I'm wrong) but since I am running Nginx, this 499 response makes them easy to find.

But back to the actual nefarious activity that these bots are trying to mask (my conjecture): every once in a while (every 20-30 minutes) I get a hit like this:

47.29.180.173 - - [14/Jan/2021:04:15:49 +0000] "GET / HTTP/1.1" 200 17714 "https://www.office.jocial.com/Account/RewardProgram/Promotional/WebSurf" "Mozilla/5.0 (Linux; Android 10; Mi A2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.96 Mobile Safari/537.36"

As the referrer URL suggests, someone is probably having a reward for people to come to my site and click on some (any?) ad. I cannot see without logging into the system, but I'm kinda assuming that's what this is for. So, they only download the homepage and I presume start clicking on ads to their heart's content. Again, I am not sure why this is done, but I sure hope Google does not think I am the one paying the reward. Communicated that to their support people, and they are obviously being cagey about it, but I got a sense - not really surprised.

So, I guess, to clarify my earlier question: had this not been a part of a large scheme, I would not have bothered with these pesky but small requests. But I think it is, and I am looking for some kind of a WAF that I can implement, hopefully something that can sit on a server in front of my HTTP server so that these requests don't make it there and don't clog up my logs. Or perhaps there is an IP ban list for networks involved in it that I can get somewhere? Anyhow, looking for suggestions on how to deal with larger situation as a whole, I guess, not just the 499s

Also, hope this can help someone in a similar situation

lucy24

7:55 am on Jan 14, 2021 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



As the referrer URL suggests
Is it always the same referer, or too many to count? You could certainly start by 403'ing any requests with known malign referers. (Check the list every year or so. Most bogus referers don't last long, and then you can remove them from your access controls.)