Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| iCCrawler - any info? claims to come from iccenter.net - purpose? |
2 | thetrasher | 7:42 pm Sep 24, 2005 | ||||
| zzrobots.txt .yeah human error my [bleep]! |
6 | JAB_Creations | 8:38 pm Sep 21, 2005 | ||||
| Anyone read Finnish? Virgo/0.7 |
9 | keyplyr | 8:03 pm Sep 21, 2005 | ||||
| PreviewGenerator.org |
2 | MLHmptn | 2:01 pm Sep 20, 2005 | ||||
| Project Kolinka Forum Search Another bot that ignores robots.txt |
8 | Moparx | 4:18 am Sep 20, 2005 | ||||
| SBIder/0.7 What do you think about this bot? |
8 | GaryK | 2:49 am Sep 20, 2005 | ||||
| Clushbot Again This time in the Laomedon flavor |
3 | GaryK | 12:47 am Sep 20, 2005 | ||||
| Reputable Spider lists? Trying to find a reputable spider list... what do you recommend? |
3 | profitpuppy | 12:41 am Sep 20, 2005 | ||||
| Omnipelagos.com RSS Spider 0.3 Claims to find shortest path between any two things. |
2 | GaryK | 12:39 am Sep 20, 2005 | ||||
| Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) Just noticed this one today... |
2 | BillyS | 11:32 pm Sep 19, 2005 | ||||
| Dynamic+ Does anyone know what this crawler/company does? |
3 | GaryK | 4:24 pm Sep 19, 2005 | ||||
| MPFv7.0.0.153 To script or not to script? |
2 | JAB_Creations | 11:51 pm Sep 18, 2005 | ||||
| D:\\@\\5\\5.exe |
2 | JAB_Creations | 1:06 am Sep 15, 2005 | ||||
| Lycos_Spider_(modspider) |
2 | WarmGlow | 8:43 pm Sep 14, 2005 | ||||
| CFNetwork/1.1 useragent |
2 | JAB_Creations | 4:07 pm Sep 14, 2005 | ||||
| Weirdly formated UA requested robots.txt |
2 | Dijkgraaf | 4:06 am Sep 14, 2005 | ||||
| Vision Research Lab |
8 | wilderness | 5:46 pm Sep 8, 2005 | ||||
| Help me with my .htaccess file please |
2 | a_angova | 3:07 am Sep 8, 2005 | ||||
| 200 (return error code 0) - What does it means? |
2 | Shaundm | 1:56 pm Sep 7, 2005 | ||||
| envolk[ITS]spider |
4 | wilderness | 1:51 pm Sep 7, 2005 | ||||
| JRSocketEngine? |
4 | teasers | 3:33 am Sep 7, 2005 | ||||
| SlurpConfirm404 Is this really Slurp? |
4 | fish_eye | 6:34 pm Sep 6, 2005 | ||||
| Strange UA |
4 | wilderness | 1:31 am Sep 5, 2005 | ||||
| Mozilla/4.0 (compatible; prgrabber 1.0) Is this a Google PageRank checker? |
2 | GaryK | 6:17 pm Sep 4, 2005 | ||||
| Amazon Crawling aws-alexa libwww-perl/5.65 |
9 | The_Contractor | 5:37 pm Sep 4, 2005 |