Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Bingbot, Robots.txt and Port 80 or When Will Bingbot Grow Up? |
17 | dstiles | 10:32 am May 12, 2021 | ||||
| TkBot |
3 | lucy24 | 4:40 am May 12, 2021 | ||||
| Seekport Crawler I think first time seeing this? |
2 | SumGuy | 12:35 am May 8, 2021 | ||||
| LinuxGetUrl/2.0 |
11 | WebOpz | 4:50 pm Apr 28, 2021 | ||||
| Is there a UA setting that allows all Google Bots? |
13 | born2run | 11:15 am Apr 24, 2021 | ||||
| The Knowledge AI The Knowledge AI |
4 | WebOpz | 2:30 pm Apr 23, 2021 | ||||
| Search bot operating on OpenDNS IP address? Anyone seen this before? |
8 | SumGuy | 2:27 pm Apr 20, 2021 | ||||
| M247 - Part II Unfortunately here we go again. |
3 | blend27 | 1:04 am Apr 17, 2021 | ||||
| Wise to block or challenge site visitors with empty UA string? |
6 | born2run | 11:52 pm Apr 13, 2021 | ||||
| LanaiBot Agressive Scraping |
11 | WebOpz | 10:17 pm Apr 9, 2021 | ||||
| Linespider - do I allow, or block? Who uses it? |
4 | SumGuy | 3:14 am Apr 2, 2021 | ||||
| Googlebot and HTTP/2 - still stuck in HTTP/1.1 |
2 | 1script | 6:39 pm Mar 28, 2021 | ||||
| gowikibot - again changing UA - again |
3 | blend27 | 3:58 am Mar 11, 2021 | ||||
| How to deal with persistent cookies and tracking |
14 | RMKoRn | 11:55 am Mar 9, 2021 | ||||
| More Bot Fun fron MSFT New Headers are in use. |
4 | blend27 | 11:38 pm Feb 27, 2021 | ||||
| IDBTE4M CODE87 in UA field? |
5 | not2easy | 4:01 pm Feb 16, 2021 | ||||
| Bad Behavior by AmazonAWS and what to do about it? |
14 | JesterMagic | 8:24 pm Feb 7, 2021 | ||||
| Verifying SemrushBot Learning to understand what's a spoof and what's not |
24 | Solution2 | 11:35 am Jan 29, 2021 | ||||
| MTRobot |
2 | lucy24 | 6:06 pm Jan 26, 2021 | ||||
| INTERNETBOLAGET-SE = No More internetvikings.com |
8 | blend27 | 2:45 am Jan 26, 2021 | ||||
| Googlebot will start crawling over HTTP/2 in November 2020 |
20 | phranque | 10:28 pm Jan 22, 2021 | ||||
| Bots slamming my site [2] ( 1 2 ) |
33 | timchuma | 6:09 pm Jan 12, 2021 | ||||
| Facebot / Twitterbot hit from AS6730 (Sunrise Communications AG) |
7 | SumGuy | 2:52 pm Jan 10, 2021 | ||||
| Server Farms June 2020 Continuing discussion of hosting company IP ranges |
14 | not2easy | 3:09 am Jan 3, 2021 | ||||
| Why would bingbot be requesting atom.xml? What is the purpose of that file anyways? |
6 | SumGuy | 5:37 pm Dec 28, 2020 |