Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| semanticdiscovery/0.2 has problems with robots.txt...or a forged UA |
3 | WebJoe | 8:04 pm Oct 13, 2003 | ||||
| Mozilla/5.0 (Version: XXXX Type:XXXX) |
2 | BlueSky | 4:32 pm Oct 12, 2003 | ||||
| InetURL:/1.0 Doesn't read robots.txt ... |
4 | bobothecat | 10:48 pm Oct 11, 2003 | ||||
| Brief UA's |
6 | wilderness | 3:43 pm Oct 11, 2003 | ||||
| What do you watch the spiders with? Any good tools? |
6 | stuntdubl | 2:24 pm Oct 11, 2003 | ||||
| puf/0.91beta6a (Linux 2.4.19-metacarta-appliance-2; i686) Savagely attacked a site of mine |
3 | seindal | 6:02 pm Oct 9, 2003 | ||||
| 207.213.165.104 204K hits |
2 | hanuman | 11:10 pm Oct 8, 2003 | ||||
| googlebot ips |
6 | anand | 10:07 pm Oct 8, 2003 | ||||
| conducent ip? |
5 | coffeebean | 9:29 am Oct 8, 2003 | ||||
| Detecting spoofed user agents Best method? |
3 | Demeter | 7:09 am Oct 8, 2003 | ||||
| AHHH help |
3 | ganderla | 7:39 pm Oct 7, 2003 | ||||
| Never seen this from Google before... What the heck is going on? |
3 | guillermo5000 | 5:14 am Oct 6, 2003 | ||||
| Fake AV log spammer? |
3 | wilderness | 9:41 pm Oct 5, 2003 | ||||
| PersonaPilot/1.00 |
2 | BergtheRed | 12:15 pm Oct 5, 2003 | ||||
| Tide Is this a new msn spider? |
9 | quiet | 5:10 am Oct 5, 2003 | ||||
| Wavepluz what is this? |
9 | coyote | 4:17 am Oct 5, 2003 | ||||
| possible scanning agent |
2 | wkitty42 | 1:37 am Oct 4, 2003 | ||||
| 210.192.120.74 |
3 | wilderness | 9:39 pm Oct 3, 2003 | ||||
| Formmail exploit hunter(s) out again Mozilla/4.06 (Win95; I) ++ Mozilla/4.0 (compatible; MSIE 5.5; Windows N[2] ( 1 2 ) |
39 | claus | 3:10 pm Oct 3, 2003 | ||||
| 212.98.46.104 Java/1.4.1-p3 fell in trap |
3 | WebJoe | 6:21 am Oct 3, 2003 | ||||
| DTS Agent error_log shows 86MB of Warning: Truncating oversized username |
19 | nancyb | 3:04 am Oct 3, 2003 | ||||
| Mozilla UA only Mozilla |
2 | coyote | 2:56 am Oct 3, 2003 | ||||
| Robot.txt - Please help me to make one Robot.txt . I need to make one |
2 | kteoj | 2:54 am Oct 3, 2003 | ||||
| telia.net telia.net sends unsolicited packet to server and slow it down |
4 | Penguinsnob | 10:43 pm Oct 2, 2003 | ||||
| Traffic-Power / 66.209.77.194 Spidering Ignores robots.txt / says its IE |
2 | ccwaterz | 10:23 am Oct 2, 2003 |