Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| I need help building a list I am building a list of spiders that we should let in and would like help |
6 | anoryu | 6:10 am Dec 29, 2002 | ||||
| Teradex Mapper violating robots.txt in fact, they never bothered to check |
2 | Finder | 6:08 am Dec 29, 2002 | ||||
| Fast PartnerSite Crawler Question |
4 | Jon_King | 2:49 pm Dec 24, 2002 | ||||
| Microsoft URL Control A warning and a request for information |
29 | Crazy_Fool | 12:45 am Dec 23, 2002 | ||||
| Universal Payment Gateway UP.Browser/3.2.9.1-UPG1 UP.Link/3.2 |
4 | wilderness | 7:50 pm Dec 22, 2002 | ||||
| New Spam Bot? Hit by "Lite Bot 0816b" |
5 | 121focus | 10:03 am Dec 21, 2002 | ||||
| 5 | lazyz | 10:14 am Dec 20, 2002 | |||||
| libwww-perl/5.63 bot from tracerlock.com libwww-perl/5.63 not just from Inktomi or FAST |
2 | Falcon | 1:05 am Dec 20, 2002 | ||||
| im4u oh, really? |
2 | bull | 1:00 am Dec 20, 2002 | ||||
| Why does AV crawl with Scooter AND Mercator please explain |
2 | jec2002 | 4:26 am Dec 19, 2002 | ||||
| Indy Library Explained Indy Library is a programming toolset that can be anything. |
2 | WarDead | 4:10 am Dec 19, 2002 | ||||
| Rumours-Agent Site downloader? |
6 | pendanticist | 7:19 pm Dec 17, 2002 | ||||
| DA5.0 come to my site and downloal stuff. I don mind traffic, but .. |
2 | Penguinsnob | 7:17 pm Dec 17, 2002 | ||||
| Page Beamer Web Accelorator |
2 | carfac | 8:09 pm Dec 16, 2002 | ||||
| Hungry Bot |
16 | menyak | 1:35 pm Dec 16, 2002 | ||||
| ArcNet - over 1000 requests? |
7 | billdaly | 8:14 pm Dec 15, 2002 | ||||
| Summation Script |
2 | DavidT | 7:40 pm Dec 15, 2002 | ||||
| links manager robot Identity |
3 | Need_More_Hits | 3:18 pm Dec 15, 2002 | ||||
| Mysterious User Agents [3] ( 1 2 3 ) |
76 | ratman | 4:17 pm Dec 14, 2002 | ||||
| FreeFind.com viral marketing banned at the firewall |
10 | Finder | 7:52 pm Dec 13, 2002 | ||||
| lcabotAccept who does this belong to? |
7 | Finder | 7:40 pm Dec 13, 2002 | ||||
| Mercator and Altavista? |
4 | HuhuFruFru | 7:12 pm Dec 13, 2002 | ||||
| Szukacz violating robots.txt Polish search engine |
2 | Finder | 7:07 pm Dec 13, 2002 | ||||
| Coincidence? |
3 | wilderness | 6:39 pm Dec 13, 2002 | ||||
| 4.38.68.69 unknown crawler |
3 | Orange_XL | 8:25 am Dec 13, 2002 |
