Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| FBI Seeks To Unmask Anonymous Web Archiving Service Owner Archive.today |
4 | SumGuy | 11:46 pm Dec 17, 2025 | ||||
| Is anyone getting hits from 121.99.4.0/24 ? |
15 | SumGuy | 3:28 am Dec 16, 2025 | ||||
| Where is grok bot / xAI? Does grok use anonymous VPNs to crawl sites? |
18 | SumGuy | 9:03 pm Dec 9, 2025 | ||||
| Complete mess all over 43.0.0.0/8 Concidering nuking the whole /8 for US/Canada sites |
15 | blend27 | 5:34 am Dec 5, 2025 | ||||
| ShapBot |
3 | SumGuy | 3:27 am Nov 14, 2025 | ||||
| Bad actors from 140.235.0.0/22 140.235.0.0/22 |
2 | Bewenched | 3:46 pm Nov 11, 2025 | ||||
| the robots ye will always have with you |
6 | lucy24 | 5:50 pm Nov 9, 2025 | ||||
| Evolving and Cloaking Your Robots.txt into an Active Bot Gatekeeper |
9 | Brett_Tabke | 5:37 pm Nov 9, 2025 | ||||
| Archive.org now sometimes using generic user-agent? |
3 | SumGuy | 5:28 pm Oct 28, 2025 | ||||
| blocking amazon networks |
24 | markRg | 10:04 pm Oct 20, 2025 | ||||
| Web host not seen before - AI hosting |
5 | Bewenched | 5:10 pm Oct 16, 2025 | ||||
| xAI / grok likely getting web search from Microsoft |
1 | SumGuy | 1:48 pm Oct 15, 2025 | ||||
| TikTokSpider |
7 | lucy24 | 5:52 pm Oct 14, 2025 | ||||
| Why does ClaudeBot / Anthropic do this? Repeatedly ask for robots and sitemap |
2 | SumGuy | 6:10 am Oct 13, 2025 | ||||
| ACEVILLEPTELTD Singapore Scraper Farm |
8 | not2easy | 11:11 pm Oct 12, 2025 | ||||
| Found some bad actors from this ISP cheapy.host LLC 196.251.81.0 |
4 | Bewenched | 2:18 am Oct 12, 2025 | ||||
| msnbot / bingbot interacting with my site's search script started recently |
2 | SumGuy | 6:18 am Oct 8, 2025 | ||||
| ChatGPT-User Does it indicate bot or user (human) engagement? |
4 | SumGuy | 5:00 pm Oct 4, 2025 | ||||
| Web hit from Residential ISP or international VPN? |
8 | SumGuy | 4:46 pm Oct 1, 2025 | ||||
| A new UA string? At least to me it's new |
6 | mcneely | 4:59 am Sep 24, 2025 | ||||
| IbouBot Something new |
7 | SumGuy | 4:11 pm Sep 22, 2025 | ||||
| AI crawlers overwhelming sites |
2 | londrum | 11:05 pm Sep 17, 2025 | ||||
| GoogleOther ? 66.249.65.xx |
5 | Bewenched | 3:33 pm Sep 17, 2025 | ||||
| Why doesn't Windows NT 11 show up in user-agent string? |
8 | SumGuy | 12:32 am Sep 17, 2025 | ||||
| CloudFlare Releases AI Crawl Control |
10 | Brett_Tabke | 11:16 am Sep 10, 2025 |
