Heh!

Betcha this magical file will be a quick reference for llm bots.

If you have one: gaming the system while separating wheat and chaff.

If you DON'T have one: "That's the stuff we want!"

:: quick run to raw logs ::

Back in April I find six requests--from six different IPs, all the same humanoid UA with the same uninspiring headers--for llms.txt on two different sites, all https. Two 301 (missing www, would have to be), four 404. That means two different robots--or the same robot wearing two different masks--got a 301 and then came back about thirty seconds later in a new mask, rather than immediately with the same mask. Oh, and every last one was a HEAD. Is this file, when present, so vast that it’s worth the robot's trouble to start with a HEAD and only come back with a GET if it really is present? Seems a bit pointless, like when there's a HEAD for robots.txt.

I cross-checked. None of those six IPs came back later to do further snooping.

But I digress.

[Sorry. Started to say something else. ]

[edited by: Martin_Potter at 2:46 am (utc) on Jul 9, 2025]

If I were cynical, I would begin to wonder why we bother to post anything on the Internet.

Yesterday I was looking for media icons via search > images and found a logo scraped from one of our websites, offered in various colours, but nevertheless ripped from us. It was being displayed in a catalogue by an online stock image library and offered for sale.

They do have a DMCA form so lets see how that pans out. Just another vein of bot scraping.

"AND THAT IS WHEN the creator hires a lawyer to get things done. Filing a DMCA these days has little teeth WITHOUT"

Can you suggest some lawyers who deal with this?

From my "research", the only thing that happens every time I try to take action against this is that I waste tons of time and energy. The "lawyers" who say they can help, basically want to take money.

It usually shows face after 15 minutes or so in speaking to the "lawyer" and realizing their not fully understanding what you are talking about and want an enormous amount to get started, Like $5,000 for what he just spoke about that you know is only the tip of the iceberg.

You need $1 million dollars to defend yourself?

"I suggested that he report it to Google, but they came back saying that the copycat was the owner!"

Yes that's the problem! They have so many websites they are changing their "Post Meta Time" to be before you. How can you prove it was you first ? It's like trying to get all this "proof" is a waste of time and will put you out of business even faster.

"Tangor: Theft is theft. We have laws that can take care of this."

We do! But it's not working! Look at all the people stealing in New York and California... are they just saying "SEO! IT's SEO!" and that's why tehy don't get in trouble or what?

NOPE SEO! It's not illegal to pick something up from the ground and it's not my fault the window was broken!

-------------

"In my case, I'm aware of the risks, but being exploited is something else."

That's how I feel. I am being deliveredly exploited.

People say "Don't worry about it etc".

Ya, you try doing that for years while they say "The algorithms will take care of that" while they just pump more and more sites out and are making xxx,xxx / month while you can barely pay the bills and all of your work!

Go up in traffic? They can now box you in with all of their websites and say "I haven't even copied them for a whole 1 week"!

---------- I don't understand why this is a "thing", other than people are profiting HUGE off "SEO" - --- but aren't people profiting huge off theft from the stores? //endrant

Like.. you know what's happening.. you can wait 3 days to post about something... but the moment you do, there's 5 websites echoing you and Facebook groups posting 10 times and tons of fake social media accounts...

It doesn't matter how much proof you get.

Can you show me some "lawyers" that fixed problems like this? Usually it's "HUGE" websites that are making millions of dollars a month that actually get help. No? Please show me that i'm wrong and make me believe that the world isn't as messed up as I see it.

I don't want to get too far off track, but does anybody else think that this could be the "AI" ?

What I mean is...

These websites seem to have some like "Whitelisting" on the searches, they can do whatever they want. Things that would get others penalized, they seem to have free access to do over and over and over.

I could get into specifics, but i'm sure you know what i'm talking about if your a webmaster.

Does nobody else see that?

Who's really behind these websites?

When you finally get your bot detection 100%, who do you see coming through ? <=---------------------------

I'm kindove afraid to say this, but i'm going to say it without saying it.

Start monitoring your website for traffic with

if(window.outerWidth = 1 && window.outerHeight = 1){
Log


You will notice this is a very, very, very unique pattern. I do and it's how I got my "AI" attacks to stop against me, but i'm in a huge different battle now.

Somebody please monitor that and post the logs after a few weeks from users and bots.

The only thing I can't figure out is if 1 bot is replicating another or if they are the same.. but.. that is the fingerprint of what I believe to be my "AI" clone and I wonder if it's the same for anybody else.

Sorry I have to say one more thing. Because the 1x1 requests are replicating a legitimate bot from ranges that are unpublished, it makes it impossible to 403 or captcha, but I was able to create a "softblock" type thing in my system.

When I say that I got the clones to "stop" after doing that, it stopped for a few days but then somebody purchased over 8 different internet services providers to their home (including phones) and I was able to link them all from stealing my stuff immediatly and block the entire city ranges.

That was just 2 weeks ago, maybe 3 now. And they are responding by using all of the clone websites to copy the best of everything I do more times (instead of everything i'm doing as quickly as possible), because they can't outrank for everything if they don't post in a timely manner (a few hours after me maximum).

Er, WanttoPost, the site allows you an hour to edit posts. (Hm, now, is that an hour from original posting, or an hour from most recent edit? Someone will know.)

Someone will know.

Not me.

I know there is some time to edit but I've never had any exact info there. AFAIK it is at least half an hour. But I really don't know.

Back in the day it was 30 minutes, as many times as needed, but don't know if any edit reset the timer back to 30. What the edit interval might be now I do not know. By the time I see an error I should have caught while posting and didn't edit in 5 minutes 24 hours has passed. Sigh.

HOWEVER, in those cases where an edit would be important for clarity, the mods have kindly made those. Just ask (and supply the full edit to copy and paste!). Ala Joe Walsh: "Life's Been Good To Me So Far".

I'm not complaining, just saying... after years of being around this forum, there are days when I absolutely don't want to deal with text-code posting for quoting, I would love to have here what the rest of the world has had for years: rich text editors. But well..

Martin Potter: If I were cynical, I would begin to wonder why we bother to post anything on the Internet.

It's natural, in fact I've been thinking about opening a thread about that. I guess it makes sense when people create stuff and post it on the web for profit, at least... IT KIND of makes more sense than doing it for free, just for passion (to me, it's the opposite, it makes more sense this way), and naturally, nice things get some ROI. As far as I can tell, many of us around here (writers, content creators) have some artist kind-of-wood, we create, creating and spreading knowledge and stuff is part of our dna. My conversations with people around here reveal many enjoy painting, photography, writing, woodworking, etc.

@ IJustWanttoPost: yes, Google can sometimes (many times) tag the thief as the original creator and owner, it sucks, and even when this doesn't happen, the processes are now too slow and dumb, you won't get to talk or deal with a human, just algorithms. In fact, it's been a while since people do not care about creating quality stuff, but instead, trying to fool the algos.

other than people are profiting HUGE off "SEO"

Yeah, and we don't see any of them weighing in on the discussions here.

I wonder if any of them have actually done any real research like setting up a dozen websites, each utilising a different recommended SEO factor to compare the results over a period of time. I have and what Google has been saying all along about optimisation is nothing more than a smokescreen to disguise the BS that really goes on..

Honestly, it can be frustrating when others copy your work without credit especially when you’ve put real effort and creativity into it. But I try to see it as a sign that my content has value. It pushes me to keep evolving, staying original, and creating things that can’t easily be replicated.

@CheapSpyder ... Welcome to Webmasterworld! Positive attitudes will go far!

@CheapSpyder, welcome.

I agree with your words, it's just, after several years seeing this happen, it really feels different. You can do hard work on content and profit during a conference perhaps way more than over 6 months running your website. I don't feel comfortable writing the word "profit" because it was never my goal with my hobby websites, but the loss of value and work for free push me to question myself and rethink the value of what I do, it's not healthy just throwing away things for free and watching it dissolve.

I remember debating on some forums putting effort on the concepts I explained, others did the same (sharing my views and data, or not), but then I noticed how some people are just provocateurs, they push wrong ideas for others to correct them, and while you debate: you are educating them for free, and later they post your own opinions as theirs...

Mmmm that's not ok.
What they do is not ok
And what we do training people for free that way, is also not ok

There are differences between sharing data and hard earned conclusions, knowledge is ok (like in a Star Trek way), but one has to become aware of effort vs benefit, and not just become tools for others. In that sense, I'm still tracing my strategies to keep producing content, but not falling on traps and old mistakes.

I try to see it as a sign that my content has value.

Isn't this like telling a {crime} victim that they should take it as a compliment?

lucy24: Isn't this like telling a {crime} victim that they should take it as a compliment?

that one hurt ha ha, in ways that can only be explained efficiently with the insertion of a very simple emoticon (that we don't have here).

I'm on a journey of re evaluating authorship and copyright, I worked on writing for years and I'm familiar with the concepts, both in practical terms and legally, but recently... it feels like I have to learn this again.

The value of what we do (like content) can and must have a direct link to/with respect, people use our content and provide certain credit respect us (profit or not), but people who steal our content show no respect at all. By no means I'm trying to debate the original post above, not at all, all the contrary, I Want to bring back some respect tot he group, anyone who creates content. People who steal it may see value on our content, but they don't respect us at-all.

Perhaps, some of them may see the whole thing as an innocent way to earn money, but most see it as no-face no-name, just "here's stuff I can grab without consequences and make money out of it". Some are willing to risk copy right hits, but most think "nah, they aren't going to do anything about it, 5 out of 10 don't move a finger because the process are too slow and long". And it get's work, as some people focus on ways to deceive SE's so YOU don't appear as the author, and thus, you will loose the DMCA, or you will just not get anything in return except wasting your time.

It really is kind of simple. If you can't bear having your stuff stolen, then don't post it on the WEB!

If you wish to PROTECT what you publish, put it behind a PAYWALL and LOGIN.

If you have an attorney on staff, set loose the dogs of war.

THEFT is not going to stop by talking about it. YOU have to do the deterrence UP FRONT, pursue EVERY infraction, and keep a bottle of TUMS on the bedside.

The SOLUTION is an Internet Copyright Office that ALL NATIONS acknowledge and empowered to provide real penalties for bad actors. That solution requires political will and international agreements and going through a third party (as noted above) to FILE you copyright, likely enduring a FEE, before the content is ever displayed. The roadblock will be AI and the on-going necessity of LLM and updating same.

What will not work is begging the existing tech giants (g, Bing, etc) to self-police TRUE COPYRIGHT. That would be cost prohibitive and contrary to their business model.

Returning to the OP, do you get tired?

Not any more. I don't publish anything that I would miss if stolen. Also don't publish anything many would want to steal (and if they did, it would make me smile for sharing my hobby!).

HOWEVER, my case is different in there is no profit profile for what I do. Those working the 'net for a LIVING are faced with extraordinary challenges during these days and times.

One last thought (for tonight at least!) this theft and copy crisis is not just one-sided. USERS are also to blame. They don't seek the original. They take what is low-hanging---lazy, insatiable, and yet very easily satisfied. How else can you explain that recently the #1 on Billboard's Top 100 Country is an AI generated "artist" complete with fake video to go with the fake music?

This, too, is theft---but far more insidious.

Well, yes, but... NO.

Different opinions and angles are fine, but there is a central context. I don't believe there is a single forum member who enjoys having their content copied or stolen, worse if others profit from it. We can take it to the next level: it doesn't matter if you create content for business, residual income or hobby, nobody wants to be taken for a fool, as that person who will not react when others are having a party on his/her backyard, and not only nobody gives you credit: you are not even invited.

What's not the topic? complaining without seeking solutions, no, that's a dead end. What's the topic? people stealing your content, profiting from it, getting credit for it, and the mechanisms to deal with this fail or take too long to come to a conclusion while on top of that, your website does not rank as in the past. It's nothing new that the old SEO (white hat) stuff doesn't really work anymore, so, what are you going to do?

Yeah, --do, doing--

The problem is, before doing, there is a point where you get hit and you don't want to do this anymore. Can you solve the critical points being mentioned? no, let's be honest, the answer is no, regardless of how someone tells someone else "deal with it, get up and fight", there is no clear path to a solution, at least to my knowledge is not clear.

Another problem: you (generic reader) may not care, because just like some of us (if not all of us), you don't care as long as your profit and expenses are not hurt. As long as your ship stays afloat, it's ok, I mean, it's not ok but the damage is marginal, you can deal with it. But then you wake up one day and someone is profiting seriously, while your ship is sinking.

And... there is an extra problem. Retired. Yeah, retiring is ok, it's fine, people are no longer an active part of the constantly changing ecosystem, they live by other rules, other system of values probably, and they don't care so much about certain battles, that's fine. But one cannot compare the approaches of someone retired versus someone still in battle. In many ways, just like a young guy experiencing a break up after discussing marriage, it doesn't really click when someone retired gives advice, like "yeah, won't be your first and only heart break, nothing matters, nothing is original, everyone is copying someone else, and... anyway we are all going to die"

Take it as a funny remark. And yes, even the big ones say it's painful the way the web has changed, we can complain about it too, at least a bit (before going back to our strategies), in the meantime, talking can also lead to conclusions: this is not worthy anymore, let's dream another dream.

All this fuss about AI and content theft.

Yes, they are borrowing our information and selling it as their own. But how accurate is it?

The other day I received an email from a client who I have been providing assistance for getting the most out of one of our software solutions. Most of his clients are at sea without Internet and he has 100s of videos on USB - while we do provide offline DRM tokens, he cannot cope with registering 100s of videos into the system. So we are working on a work around for that.

But in the meantime he has come back with advice to solve his problem provided by Chat GPT.

At first I was impressed that Chat GPT had dived so far into our archives to be able to recommend alternatives that even I had forgotten about. When reading again I realised that while it seemed a good answer, its best advice was to contact our support team for custom scripting.

Kudos to Chat GPT.

Then I started thinking about all the information that we do provide online - including all of the information that we do not provide online.

Yes, the web is inundated with social media, blogs and wikis that provide knowledge, opinions and gossip. Chat GPT and other search organisms will find those easily enough. But what about all the information that is not disclosed?

For example, info that we do not disclose is:
1. how some things really work in technical terms - to prevent plagiarism
2. instructions for clients that could be abused by competitors and other malevolents
3. anything that could bring our product/service into disrepute

AI can only be as informed as we allow it to be.

Content copying can be restricted by using coding methods such as disabling right-click, text selection, and copy actions through JavaScript and CSS. These techniques help prevent users from easily copying text from the website.

Content copying can be restricted by using coding methods such as disabling right-click, text selection, and copy actions through JavaScript and CSS.

Actually, JavaScript and CSS is totally useless for preventing copy - all JS can do is disable right-click menu actions. It cannot prevent:
- page save
- web scraping
- screenshots and recording

Serious content thieves scrape web pages. Some simply replace header and footer and post online as their own. Images are scraped and get offered as image stock. PDF and video is copied and rebranded for training courses.

The only way to prevent copy is by using a proven solution that actions at system level - accessible via proprietary reader or secure web browser. Normal web browsers are useless for copy protection.

If you want search friendly pages, you need to decide what to use as a doormat and what to protect for subscribers.

@tangor wrote :

The SOLUTION is an Internet Copyright Office that ALL NATIONS acknowledge and empowered to provide real penalties for bad actors.

This seems like a light at the end of the tunnel. Is it possible to make this happen? Is it possible to get our respective governments to accept this idea and to push for it? Should this role be adopted by the ITU (International Telecommunications Union, an organ of the United Nations)? Especially given their advisory and voluntary, concensus-driven business model. What can be done to further this idea?

More than 90% of all web content is derived from plagiarism, piracy and content theft.

The damage is already done and in spite of Copyright. Hoping to reverse that would be akin to asking the mafia to give back the billions they stole from us.

What can be done to further this idea?

Look someplace other than the UN! That body politic can't even 100% agree that the sky is blue.

Even the Berne Convention (International Copyrights) first signed in 1886, most recently renewed in 2022 with 181 countries out of 195 has so many ambiguities that asserting copyright AND WINNING regarding internet publication is problematic. The internet is not "published in a fixed physical medium" which is the KEY difference---and until that is hashed out we're in the mess we are right now.

Do I think my suggestion of an Internet Copyright Registry (ICR, perhaps?) ever happen? Re-read the first paragraph of this post.

(sigh)