Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
ResearchBot
ResearchBot; Alpha 0.01.07
2 w3bmastine 8:28 pm May 25, 2016
rank-checker.online
block?
10 smallcompany 9:05 pm May 21, 2016
Springshare Link Checker
2 aristotle 1:37 pm May 21, 2016
Just-Crawling
3 keyplyr 11:38 pm May 20, 2016
Maroc Telecom
7 w3bmastine 10:01 pm May 18, 2016
GoogleDocs
3 keyplyr 11:04 am May 14, 2016
Google Web Preview Analytics
5 Andy_Langton 2:58 pm May 12, 2016
Dataprovider
6 keyplyr 10:03 am May 12, 2016
Same UA, different IPs, a lot of 404s
6 smallcompany 8:37 am May 12, 2016
facebookexternalhit/1.1;kakaotalk-scrap/1.0;
3 keyplyr 8:21 am May 11, 2016
human curl.async
4 keyplyr 9:48 pm May 10, 2016
CISPA Vulnerability Notification
9 keyplyr 7:49 pm May 10, 2016
Convincing IP spoofing / user simulation
Pointers to detection of IP spoofing[3] ( 1 2 3 )
62 Simon_H 1:46 pm May 10, 2016
Golden Telecom
2 w3bmastine 10:31 am May 10, 2016
SeznamBot's new UA
4 lucy24 12:04 am May 9, 2016
foobar2000
3 keyplyr 2:08 am May 5, 2016
DeuSu
10 keyplyr 5:12 am May 4, 2016
Searchincognito.com
12 aristotle 9:24 am May 3, 2016
America Online Browser
6 w3bmastine 6:27 am Apr 30, 2016
Baiduspider weird hits
8 winexec 8:42 am Apr 29, 2016
Safari Technology Preview
7 lucy24 7:57 am Apr 29, 2016
BLEXBotTest
3 keyplyr 11:08 pm Apr 28, 2016
Internet Memory
5 keyplyr 1:12 pm Apr 28, 2016
Melvil Rawi
7 keyplyr 9:39 pm Apr 27, 2016
Clickbots and other bad unknown crawlers
bad bot blocking issues
7 AlexB77 8:43 am Apr 25, 2016