Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Google bots are visiting like crazy? |
3 | myowndistrict | 6:08 pm May 22, 2006 | ||||
| Google Wireless Transcoder (again) from Google, but does not obey robots.txt |
11 | simplicity | 2:51 pm May 22, 2006 | ||||
| Bots Coming Into My Site How do I know if they are crawling? |
2 | F_Rose | 8:38 pm May 17, 2006 | ||||
| msnbot-media/1.0 .microsoft.com and (MS's) .phx.gbl |
3 | Pfui | 4:07 pm May 17, 2006 | ||||
| hundreds of hits per day on robots.txt from FireFox UAs Appear to be through "planetlab" |
6 | abates | 10:48 pm May 16, 2006 | ||||
| Shopping.com sdcresearchlabs-testbot |
2 | fiestagirl | 5:36 am May 11, 2006 | ||||
| Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0) YES, this is -not- a human visitor |
4 | JAB_Creations | 8:40 pm May 8, 2006 | ||||
| bot/1.0 |
3 | bobothecat | 6:48 pm May 8, 2006 | ||||
| spider or scraper? based on IP, how can you get the REAL STORY? |
12 | arnarn | 4:58 pm May 8, 2006 | ||||
| Mediabot coming through Yahoo! Proxy |
9 | volatilegx | 11:59 pm May 7, 2006 | ||||
| Microsoft-WebDAV - What was this trying to do? |
7 | JAB_Creations | 7:33 am May 7, 2006 | ||||
| Gt::www/1.026 a PHP/Perl class? |
2 | hakre | 4:33 pm May 5, 2006 | ||||
| KakleBot |
2 | bobothecat | 7:24 pm May 4, 2006 | ||||
| Explore XY |
11 | wilderness | 4:16 pm May 4, 2006 | ||||
| OutfoxBot /0.1? Anyone know who what where? |
2 | fusion5 | 3:41 pm May 4, 2006 | ||||
| Tarantula/0.1 |
3 | bumpaw | 8:33 pm May 1, 2006 | ||||
| TAIS Pinger (A SQL thing? An MSN thing?) Looked for (unlinked) referring site's files on my site. Loose w/ privacy. |
3 | Pfui | 6:23 pm Apr 26, 2006 | ||||
| NameProtect.com's NPBot bot hides in plain sight via IP 12.175.0.44 Rarely asks for, never heeds robots.txt |
2 | Pfui | 4:53 pm Apr 26, 2006 | ||||
| crawl-66-249-66-39.googlebot.com never seen this one in the host logs |
3 | hu12 | 4:04 pm Apr 26, 2006 | ||||
| Patwebbot |
3 | bobothecat | 12:23 am Apr 26, 2006 | ||||
| zedzo |
2 | wilderness | 7:49 am Apr 25, 2006 | ||||
| Bacon Media Bot Bacon Media Bot |
3 | ByteEnable | 3:57 am Apr 25, 2006 | ||||
| Was my site crawled? |
2 | TuanLa1972 | 2:21 am Apr 25, 2006 | ||||
| lanshanbot/1.0 |
2 | bobothecat | 2:33 am Apr 21, 2006 | ||||
| msnbot visiting in pairs? |
2 | bobothecat | 11:41 pm Apr 20, 2006 |