Forum Moderators: phranque

Message Too Old, No Replies

Mysterious commands added to htaccess file

someone, not me, added 3 commands

         

Martin Potter

2:25 am on Sep 2, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



I have just noticed that someone has added commands to my htaccess file. In any section where there are RewriteCond rules, the following 3 commands have been added :
RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/[0-9a-zA-Z_-]+$
RewriteCond %{REQUEST_URI} !^/\.well-known/cpanel-dcv/[0-9a-zA-Z_-]+$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$

My .well-known directory contains none of these files except the pki-validation sub-directory which is empty. I'm no expert on Apache rules (I know just enough to be cautiously dangerous) but I think these added commands give permission where none existed before. Regardless, I don't like them.

Unlikely, they might have been added by staff at my hosting company out of some concern for security but it is a long weekend here in Canada and the after-hours staff are not often helpful, so I won't ask them. More likely, the new rules might have been added by Attracta with whom I recently opened an account to see what they might offer in SEO. (Big mistake maybe -- I have heard stories.)

Could someone advise me whether there is some increased threat posed by these added rules, beyond my discomfort with someone having mucked about with my htaccess file? Thanks for any advice.

TorontoBoy

2:54 am on Sep 2, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



Your host provider is running vulnerability checks on your site. My host provider runs 71 such tests daily. They simply scan your site for a slew of vulnerabilities, which hopefully they will not find. If they do find something, your site is compromized and they will freeze your account until the virus or hack is cleaned up.

In your raw access log will probably see something like:
GET /.well-known/acme-challenge/SDV9AWZOOOUQUCCK67ETOFMCTRS4X4FO HTTP/1.1

I believe this is set up within C-Panel and run on a schedule by your host. This is pretty common, and is not indicative of something gone wrong. On the contrary, this test is an early warning system that is beneficial to you, because you will catch the hack early. On Tuesday you can contact your host provider and verify.

keyplyr

3:37 am on Sep 2, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



When a security certificate is added to a domain, it creates the /.well-known/acme-challenge/ folder, or in some cases, just the /well-known/ folder.

In order for the certificate to automatically update either by your host or the issuing agency, permission is needed via htaccess if on a shared server.

lucy24

4:31 am on Sep 2, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The “commands” you quote are Conditions. Are they attached to an existing RewriteRule, or is there also a brand-new RewriteRule that got lost in editing?

The conditions look like exemptions to some broader rule that presumably begins
RewriteRule ^\.well-known
... but then what? What’s the rule itself?

Just how many people have access to your site files? NOBODY edits my htaccess. Yes I'm on shared hosting, and yes I've got security certificates that auto-update. Any permissions are handed out in some entirely different location; it's got nothing to do with HTTP requests (which is what htaccess rules are for).

keyplyr

6:06 am on Sep 2, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



NOBODY edits my htaccess
Same basic thing is taking place at your server just up a level out of visibility, it's how the cert updates.

lucy24

5:59 pm on Sep 2, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Exactly. They're not touching my htaccess; they're setting rules in the config file. But I still don't see why updating a file requires an http request.

Martin Potter

12:54 am on Sep 3, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



VERY interesting and informative comments! Thank you to everyone for your thoughts.

@TorontoBoy:
Your host provider is running vulnerability checks on your site... On Tuesday you can contact your host provider and verify.
I certainly will.

In your raw access log will probably see something like:
And I did (from 20 August, I had missed it). Significantly the GET request apparently originated from my site's own IP address, pointing (I think) to my hosting provider as the culprit.

@keyplyr:
In order for the certificate to automatically update either by your host or the issuing agency, permission is needed via htaccess if on a shared server.
I guess that would now apply to me as I recently upgraded my service which now includes auto-updates for the certificate.

@Lucy24:
The “commands” you quote are Conditions. Are they attached to an existing RewriteRule
Yes, for each RewriteRule that I had in my file, the three new conditions were added immediately preceding the rule.

Just how many people have access to your site files?
Just me, that I know of, and of course the folks at my hosting provider.

So it seems the mystery is solved, and there is nothing to worry about after all. It would have been nice, though, if my provider had told they were going to make changes to a file that I have always considered to belong to me. Or, maybe I should have expected it. Another learning experience! Thanks again for all your help. The expertise here always amazes me.

keyplyr

1:36 am on Sep 3, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



In order for the certificate to automatically update either by your host or the issuing agency, permission is needed via htaccess if on a shared server.

I guess that would now apply to me as I recently upgraded my service which now includes auto-updates for the certificate.
My work is done.

Martin Potter

1:23 pm on Sep 3, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



Indeed, @keyplyr, and take a well-deserved break. :-)

Still, I am puzzled by the necessity of inserting the same three new conditions for *every* occurrence of a RewriteRule in my htaccess file. It has increased the size of the file from just over 7 kB to 12, nearly doubled. Wouldn't once have been sufficient, if associated with an appropriate RewriteRule? Or, is this part of the security test scenario, looking for any rule that is vulnerable?

lucy24

5:16 pm on Sep 3, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Still, I am puzzled by the necessity of inserting the same three new conditions for *every* occurrence of a RewriteRule in my htaccess file.

It's possible that the whole thing is automated. So instead of finding those specific rules that do, in fact, have ".well-known" as part of the pattern, and adding the conditions to only those rules, the conditions are added globally to all rules.

In which case, frankly, you should feel free to delete the irrelevant ones. I mean, if you've got a RewriteRule redirecting /foo.html to /bar.html, there is no earthly reason to say “Oh! wait! unless the request was concurrently for {filename which has already been excluded by the body of the rule}”.

Whether 7k or 12k, that's still a very modestly-sized htaccess. Mine's currently 16k--and that's not counting the higher-level htaccess shared by all sites in the userspace, which is 11k on its own. Sure, evaluating all those irrelevant conditions will add a few picoseconds to processing time, but on a normal site that would be the least of your worries.

keyplyr

8:25 pm on Sep 3, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I have a site with an htaccess file weighing in at 212k and Google Page Speed says its pages are very fast.

IMO I would *not* remove any rules or conditions generated by your host.

Martin Potter

12:48 am on Sep 4, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



I would *not* remove any rules or conditions generated by your host.


I don't know yet that it was them. Waiting for (not holiday) tomorrow to ask. But it does seem likely. And given the mindless way in which the conditions were added, I am sure @Lucy24 is right about it having been some automated process.

Yes, I know, I know. My htaccess file is paltry and meager compared to most people's and I'm beginning to suffer from HSES (htaccess size envy syndrome). Maybe I should pad it with something ... (Wanders off to look for the first 10,000 digits of an asymptotic estimate of one-half iota.)

phranque

1:35 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Still, I am puzzled by the necessity of inserting the same three new conditions for *every* occurrence of a RewriteRule in my htaccess file.

they want to avoid your configuration responding with any internal rewrites or external redirects for any requests that match the patterns in the conditional directives.

lucy24

2:20 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



they want to avoid
Still, it's overkill. They're saying over and over again
RewriteCond %[REQUEST_URI} !^/dir1/filename.html
RewriteRule ^dir2/othername.html /do-stuff-here [L]
At least 95% of the time, the body of a RewriteRule--the pattern that is first tested--will already exclude the condition.

Martin, one reason I changed my access-control structure (from IP-based to header-based) was that I wanted to keep my htaccess as svelte and trim as possible. Sure, it's an exceedingly trivial aspect of site speed--but keeping it small makes it easier for you yourself to find things in it, which potentially includes pinpointing the source of problems when they arise. As with so many programming-related things, one factor you have to take into consideration is how easy is it for a human--either yourself or someone else--to see how the code works.

If you are afraid your htaccess is malnourished, add an explanatory comment to each RewriteRule. That should plump it up nicely.

phranque

3:08 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Still, it's overkill. They're saying over and over again

only the RewriteCond directives immediately preceding the RewriteRule apply, so that send of conditions must be applied to each of the RewriteRules unless they are being applied more intelligently (vs automatically)
At least 95% of the time, the body of a RewriteRule--the pattern that is first tested--will already exclude the condition.

no problem then - the preceding RewriteCond directives won't fire unless the pattern in the following RewriteRule matches.

lucy24

4:31 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



My point is that the RewriteConds shouldn't be there at all, precisely because you know ahead of time that they will always succeed. So the server has to waste time and effort evaluating superfluous conditions.

[edited by: lucy24 at 4:39 am (utc) on Sep 4, 2018]

phranque

4:38 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



the server has to waste time and effort evaluating superfluous conditions.

the preceding RewriteCond directives won't fire unless the pattern in the following RewriteRule matches.

lucy24

4:44 am on Sep 4, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Dang! Five minutes after shutting down the computer for the night, I realized I’d said “fail” when I meant to say “succeed”. I was hoping I’d get to the edit before anyone saw it. Oh well.

Other than that, what I said still stands: every single RewriteRule has acquired additional Conditions--even if the rule previously had no conditions at all. It isn’t about the same condition being evaluated dozens of times, it's about the condition being evaluated at all. OP did note above that the addition almost doubled the size of his htaccess.

Martin Potter

1:48 am on Sep 5, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



Yeah, it was my hosting provider all right. Here is what they said :
Please note that cPanel now automatically updates .htaccess files to exclude DCV checks from HTTP redirections so that the DCV checks complete successfully. This resolves issues that some users reported with AutoSSL and cPanel Market purchases.


I had to look up DCV (domain control validation). My provider later implied that cPanel would remove any redundant code in a later review so I have asked how often these reviews take place.

So I don't own my htaccess file. Good thing to learn as I have never paid much attention to the version of htaccess residing on the server, and any time I wanted changes I simply edited the current copy on my own computer and uploaded it to the server. This time I noticed (by chance) the discrepancy in sizes.

Lucy, I only recently learned about header-based control, thanks to @TorontoBoy. Must read more. (Much more!) I do include short explanatory comments before each section of htaccess, just to remind myself. Sorry that you don't think estimated digits of one-half iota would make good filler.

keyplyr

1:50 am on Sep 5, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks for that update Martin Potter.

More htaccess control info here: Blocking Methods [webmasterworld.com]

Martin Potter

2:06 am on Sep 6, 2018 (gmt 0)

5+ Year Member Top Contributors Of The Month



As @keyplyr had already suggested, my hosting provider strongly advised that I *not* delete any of the Rewrite conditions that cPanel has added to my htaccess file. Although they didn't answer my question about how often cPanel checks the file, they did confirm that reviews are "frequent" and that it is AutoSSL that uses this code for domain control validation. Guess I will have to change the way I edit htaccess and do it to their copy instead of my own. (And I won't need any "filler" material after all.)

If others out there are using providers who offer AutoSSL, maybe this situation will become common.

Thanks to @lucy24, @keyplyr and @TorontoBoy for info on header-based control.