Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Globalspec engineering search engine |
5 | volatilegx | 11:13 pm Oct 20, 2004 | ||||
| Jetbot/1.0 NOT obeying robots.txt |
7 | pendanticist | 10:03 pm Oct 20, 2004 | ||||
| Google Spidering With No UA |
2 | volatilegx | 4:16 pm Oct 18, 2004 | ||||
| tpiol.com new spider? Whois? |
3 | guarez | 3:17 pm Oct 18, 2004 | ||||
| Google Newsbot User Agent Does it have its own, or come from specific IP range? |
2 | projectphp | 1:28 pm Oct 18, 2004 | ||||
| ProloCrawler |
2 | volatilegx | 1:37 am Oct 13, 2004 | ||||
| Bond, James Bond |
2 | volatilegx | 3:26 pm Oct 12, 2004 | ||||
| Crawlzilla/1.0 Checked robots. |
10 | pendanticist | 10:40 am Oct 12, 2004 | ||||
| Yahoo Crawlers Which crawlers are from Yahoo? |
3 | jimh009 | 7:47 pm Oct 11, 2004 | ||||
| Megite no robots.txt |
2 | pendanticist | 3:44 pm Oct 11, 2004 | ||||
| Is Yahoo's Spider Inktomi? |
2 | sirkei | 8:53 pm Oct 10, 2004 | ||||
| Wotbox/alpha0.6 |
5 | wilderness | 2:00 am Oct 8, 2004 | ||||
| stat statcrawler@gmail.com just noticed this today... |
8 | BillyS | 11:50 pm Oct 7, 2004 | ||||
| New Spider? statcrawler (at) gmail.com no url given |
2 | uncle_bob | 12:57 am Oct 3, 2004 | ||||
| Strange UA and Activity from Google has the gbot become sentient? |
2 | uncle_bob | 12:55 am Oct 3, 2004 | ||||
| Ocelli/globalspec.com Who is this? |
3 | PCInk | 9:56 am Oct 2, 2004 | ||||
| Echelon? NSA? What?! Is anyone else seeing this, and being creeped out? |
3 | Enhasa | 9:45 pm Sep 26, 2004 | ||||
| How do I track the spiders? Novice here... |
2 | Phil_AM | 7:39 pm Sep 24, 2004 | ||||
| Seekon new spider |
2 | volatilegx | 5:14 pm Sep 24, 2004 | ||||
| findlinks/0.87 |
3 | wilderness | 6:22 pm Sep 22, 2004 | ||||
| Rogue Yahoo-MMAudVid crawler Yahoo crawler hammers website, ignores robots.txt |
22 | tangent | 9:07 am Sep 21, 2004 | ||||
| Microsoft Data Access Internet Publishing Provider DAV 1.1 |
2 | wilderness | 3:22 pm Sep 18, 2004 | ||||
| JetEye Spider? |
2 | joeychgo | 4:25 am Sep 18, 2004 | ||||
| Googlebot IP 66.249.64.n |
6 | Just_Guessing | 9:35 pm Sep 16, 2004 | ||||
| Is it Google or not? I would expect any Ip used by Google would DNS resolve?! |
4 | MaxGrenk | 11:32 pm Sep 15, 2004 |