Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Server Farms 2026 Continuing discussion of hosting and data center IP ranges |
6 | blend27 | 3:27 pm Feb 21, 2026 | ||||
| Server Farms 2025 Continuing discussion of hosting and data center IP ranges |
5 | not2easy | 5:09 pm Mar 1, 2025 | ||||
| AI UAs - Identifying the crawlers AI uses |
1 | not2easy | 5:38 pm Oct 18, 2023 | ||||
| At Home Without the Robots: 2023 edition |
5 | lucy24 | 11:39 am Jul 29, 2023 | ||||
| At Home with the Law-Abiding Robots (2021) |
2 | lucy24 | 8:32 pm Sep 6, 2021 | ||||
| At Home with the Robots: 2020 edition |
11 | lucy24 | 2:48 pm Aug 13, 2020 | ||||
| At Home with the Robots: 2019 edition |
3 | lucy24 | 3:52 am Mar 13, 2019 | ||||
| At Home with the Robots: 2017 edition Critique of the Year's Active User Agents |
15 | lucy24 | 3:52 pm May 18, 2017 | ||||
| User Agents of Programming Libraries and Command Line Tools Common user agents |
14 | incrediBILL | 8:54 pm Apr 29, 2008 | ||||
| Hacker spoofing as GoogleBot 136.234.222.64 |
3 | Bewenched | 8:55 pm May 19, 2026 | ||||
| 0 | 12:00 am Jan 1, 1970 | ||||||
| Bad actor claiming to be OpenAi 74.7.242.63 |
2 | Bewenched | 2:16 am May 14, 2026 | ||||
| Microsoft, AS8075. Google, AS396982. |
8 | shawnb61 | 2:09 am May 14, 2026 | ||||
| reverse() -- ColdFusion(or your language)/SQL + JavaScript Antiscrape Trick - What is stopping me from doing it this way? |
3 | blend27 | 5:19 pm May 11, 2026 | ||||
| what do they want? [3] ( 1 2 3 ) |
88 | lucy24 | 9:40 pm Apr 29, 2026 | ||||
| My website was vacuumed up by OpenAI |
3 | SumGuy | 6:27 pm Apr 13, 2026 | ||||
| Fixed Line ISP- gtt.net Are they Hu-men, AS3257? |
13 | blend27 | 3:25 am Apr 9, 2026 | ||||
| SleepBot |
6 | Bewenched | 3:59 pm Apr 7, 2026 | ||||
| "traceparent" header ...part of OpenTelemetry SDK |
1 | blend27 | 8:07 pm Apr 1, 2026 | ||||
| PerplexityBot What's the verdict? |
3 | SumGuy | 12:16 am Mar 30, 2026 | ||||
| Youbot (new crawler?) and google-inspection tool |
1 | SumGuy | 11:50 pm Mar 13, 2026 | ||||
| FeedFetcher-Google running amok |
7 | Pfui | 5:12 pm Mar 9, 2026 | ||||
| Procter & Gamble bot-running (not just soap 'n' stuff) |
9 | Pfui | 11:57 pm Feb 19, 2026 | ||||
| A useful resource for bot / proxy detectives |
2 | SumGuy | 7:26 pm Jan 6, 2026 | ||||
| Softblocking 35.0.0.0/8 |
22 | IJustWanttoPost | 11:26 pm Dec 21, 2025 |
