Forum Moderators: open

Message Too Old, No Replies

I'm working on killing the bots that are eating our bandwidth

         

stoveboltgeek

1:48 am on Oct 3, 2023 (gmt 0)

Top Contributors Of The Month



I've been working on killing the bots that are generating millions of hits and Gigs of bandwidth use, and I've made some progress.

I'm using the following in .htacess.
RewriteEngine On

# Block Bots that are running up the traffic counts
RewriteCond %{HTTP_USER_AGENT} ^.*(python-requests*|Sogou*|ALittle*|Go-http-client*|curl|wp_is_mobile|Wordpress*|DotBot|feedi|bot\\*i|bytespider).*$ [NC]
RewriteRule .* - [F,L]

# Block via UserAgent
# Block via IP Address
Allow from 127.0.0.1
Deny from 212.235.15.153
Deny from 212.29.233

# Block by domain
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*bytedance\.com [NC]
RewriteRule .* - [F]
</IfModule>


I run this to find the log entries.

awk -F\" '{print $6}' /var/log/httpd/stovebolt/ssl_access_log | grep -v -E -i "mozilla|chrome|safari|opera|google|Konqueror" | sort | uniq -c | sort


I have a question about log entires for Wordpress. Here's an example.
45.233.170.157 - - [01/Oct/2023:18:46:19 -0400] "GET /ubbthreads/ubbthreads.php?ubb=changeprefs&what=style&value=11&curl=https%3A%2F%2Fwww.google.by%2Furl%3Fsa%3DX%26q%3Dhttp%253A%252F%252FYs.A@srv5.cineteck.net%252Fphpinfo%252F%253Fa%25255B%25255D%253D%25253Ca+href%25253Dhttps%25253A%25252F%25252FPORcu.PineoYs.A%252540Srv5.Cineteck.net%25252Fphpinfo%25252F%25253Fa%2525255B%2525255D%25253D%2525253Ca%25252Bhref%2525253Dhttp%2525253A%2525252F%2525252FHaedongacademy.org%2525252Fphpinfo.php%2525253Fa%252525255B%252525255D%2525253D%252525253Ca%2525252Bhref%252525253Dhttps%252525253A%252525252F%252525252Fretizen.republika.co.id%252525252Fposts%252525252F159390%252525252Fcara-membuat-laporan-keuangan-praktis-dengan-software-akuntansi%252525253Esoftware%2525252Bakuntansi%2525252Bcustom%252525253C%252525252Fa%252525253E%2525253Esoftware%25252Bakuntansi%2525253C%2525252Fa%2525253E%2525253Cmeta%25252Bhttp-equiv%2525253Drefresh%25252Bcontent%2525253D0%2525253Burl%2525253Dhttp%2525253A%2525252F%2525252Fkm10805.Keymachine.de%2525252Fphp.php%2525253Fa%252525255B%252525255D%2525253D%252525253Ca%2525252Bhref%252525253Dhttps%252525253A%252525252F%252525252Fblog.momitsubo.jp%252525252Fpromotion%252525252Fcomment-page-2014%252525252F%252525253Esoftware%2525252Baccounting%252525253C%252525252Fa%252525253E%25252B%2525252F%2525253E%25253EWordpress%25253C%25252Fa%25253E HTTP/1.1" 302 20 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:114.0) Gecko/20100101 Firefox/114.0"

What in the world is being attempted here? Is there an online resource that can convert this charset to readable Enlgish?

lucy24

5:16 am on Oct 3, 2023 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



^.*

.*$
Neither of these elements is necessary, and in the long run will add a few picoseconds to processing time.

What in the world is being attempted here? Is there an online resource that can convert this charset to readable Enlgish?
Probably. I made one years ago in javascript that I keep on my HD; this sample says
GET /ubbthreads/ubbthreads.php?ubb=changeprefs&what=style&value=11&curl=https://www.google.by/url?sa=X&q=http://Ys.A@srv5.cineteck.net/phpinfo/?a[]=<a+href=https://PORcu.PineoYs.A@Srv5.Cineteck.net/phpinfo/?a[]=<a+href=http://Haedongacademy.org/phpinfo.php?a[]=<a+href=https://retizen.republika.co.id/posts/159390/cara-membuat-laporan-keuangan-praktis-dengan-software-akuntansi>software+akuntansi+custom</a>>software+akuntansi</a><meta+http-equiv=refresh+content=0;url=http://km10805.Keymachine.de/php.php?a[]=<a+href=https://blog.momitsubo.jp/promotion/comment-page-2014/>software+accounting</a>+/>>Wordpress</a>
(and if any of that is truly sinister, I will get a 403 when I click Submit). Looks like someone in Indonesia is misbehaving.

stoveboltgeek

3:22 pm on Oct 3, 2023 (gmt 0)

Top Contributors Of The Month



Thank you, lucy24, for desiphering that gobbledegook. I still have no clue what this person is attempting to do.

I appreciate the comment on the code. I'll remove those.

stoveboltgeek

3:32 pm on Oct 3, 2023 (gmt 0)

Top Contributors Of The Month



While I've got your attention, I'm seeing this in the USER_AGENT strings in my logs.
><script >alert(String.fromCharCode(88,83,83))</script>&Number=509953
1 ><script >alert(String.fromCharCode(88,83,83))</script>&Number=885268
1 ><script >alert(String.fromCharCode(88,83,83))</script>&Number=912802
1 ><script >alert(String.fromCharCode(88,83,83))</script>&what=style&value=11&curl=http://\xd0\xbb\xd0\xb5\xd1\x81\xd1\x82\xd0\xbd\xd0\xb8\xd1\x86\xd1\x8b-\xd0\xbf\xd1\x80\xd0\xb0\xd0\xba\xd1\x82\xd0\xb8\xd0\xba.\xd1\x80\xd1\x84/aFbpsept26wwwstoveboltcomtp3
I have tried a number of different combinations in the .htacess file to catch this crap, but everything I try cause Apache to barf. I've tried using *fromCharCode*, *\<script*, and several others, but every one throws errors. Is there a way to blcok this crap that I haven't tried yet?

lucy24

5:01 pm on Oct 3, 2023 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is the angle-bracket the very first thing in the UA string? My access-control rules (using a combination of mod_setenvif and mod_auth-I-forget-what-it's-called) include
BrowserMatch ^\W bad_agent=nonword
....
Require env bad_agent
(the latter of course inside a RequireNone envelope), meaning "if the first character in the UA string is a non-word, set the bad_agent user variable". You could do the same thing, mutatis mutandis, in mod_rewrite, though it's a bit more server-intensive, especially in htaccess:
RewriteCond %{HTTP_USER_AGENT} ^\W

Since no legitimate user-agent* contains the string "script" you could also say something like
BrowserMatch script bad_agent=script
(Environmental variables don't have to have a value--they default to 1--but I like to keep a record.)
or
RewriteCond %{HTTP_USER_AGENT} script
meaning if the element "script" occurs anywhere in the UA string, et cetera.

Any time you can do something based on the very first character in the UA, it's going to be more efficient, because the server only has to look at one character instead of scanning the whole string. Whether you use mod_setenvif or a RewriteCond, this would of course have to be separate from the rules that do look at the whole string. In mod_rewrite, you could have two or more Conditions separated by [OR] instead of the default AND.

Edit: Although it does no harm, the [L] in [F,L] isn't needed. Non-300-class responses carry an implied [L].

* And precious few illegitimate ones; I only found a couple involving Google-Apps-Script.

stoveboltgeek

6:49 pm on Oct 3, 2023 (gmt 0)

Top Contributors Of The Month



Thank you again, Lucy24. Your explanations are clear and concise and extremely helpful. Bless you.

blend27

12:51 pm on Oct 8, 2023 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



#script xss insert
RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (\<|%3C).*script.*(\>|%3E) [NC,OR]
RewriteCond %{HTTP_REFERER} (\<|%3C).*script.*(\>|%3E)

then
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|\.\.\.).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\*|;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|%00).* [NC]


food for though....

..one has to understand what one's site URIs are, if one not expecting any special characters on one's own site in URIs, do not serve the content, tell web-server of what it can dish out for.

Dashes, Underscores, ampersands - what else is there?

Same goes for GET & POST data, in headers.

lucy24

5:06 pm on Oct 8, 2023 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



(,|;|:|<|>|">|"<|\.\.\.)
Or, if you prefer,
([,;:<>]|\.\.\.)
The ">|"< part is redundant in any case, since < and > by themselves have already been excluded. I also block "   " (three consecutive spaces): some legitimate UAs might accidentally have two "  " but never more.

stoveboltgeek

6:37 pm on Oct 8, 2023 (gmt 0)

Top Contributors Of The Month



I found a different way to block all of these.
RewriteCond %{QUERY_STRING} fromCharCode
RewriteRule ^(.*)$ - [F,L]

94.131.112.179 - - [08/Oct/2023:07:59:59 -0400] "GET /ubbthreads/ubbthreads.php?ubb=\"><script%20>alert(String.fromCharCode(88,83,83))</script>&Number=735293 HTTP/1.1" 403 227 "https://www.example.com/ubbthreads/ubbthreads.php?ubb=\"><script >alert(String.fromCharCode(88,83,83))</script>&Number=735293" "Mozilla/5.0 (Windows NT 10.0; WOW64; Rv:50.0) Gecko/20100101 Firefox/50.0"




[edited by: not2easy at 7:04 pm (utc) on Oct 8, 2023]
[edit reason] Reminder: use example.com [/edit]