How many times do browers send requests to non-responding servers?

Forum Moderators: phranque

Message Too Old, No Replies

How many times do browers send requests to non-responding servers?

Before giving up?

SumGuy

3:24 pm on Nov 1, 2021 (gmt 0)

I'm looking at router logs that are rejecting IP's (all ports) and noting that some IP's bang on port 80 or 443 for upwards of 20 - 40 times over a period of 30-60 seconds before going away. I'm particularly wondering when the IP seems to be residential (not an obvious server/hoster).

So I'm curious how the typical web browser is set up in terms of sending repeat requests to non-responding servers. Is there anything solid on this?

Do any of them send multiple requests (as frequently as 1 per second) over 10 to 30 seconds?

Are these retry-attempts something that is user configurable in a browser setting?

lucy24

6:49 pm on Nov 1, 2021 (gmt 0)

Are we talking about a firewall-type router which simply sends no response at all, as opposed to a 500-class response? (If you sent a 503 with a Retry-After header set for at least an hour in the future, that should identify the robots.)

Think about the rare case when you try to get to an unreachable site in your own human browser. Typically the error message will include a “try again” option, which strongly implies that the browser won’t repeat the request unless you explicitly tell it to. (I checked Firefox prefs. There doesn’t seem to be a setting for how many times to try, unlike email clients which typically do have a “retry” option. But those tries would be spread over several hours or even days, not over and over again in a few seconds.)

One recent site (2020) says the browser waits 30 seconds for a response.

Memo to self: Next time I happen to meet a non-response, turn on Live Headers before trying again. That would show me if the browser is actually sending more than one request in those 30 seconds.

phranque

9:01 pm on Nov 1, 2021 (gmt 0)

there is a http.response.timeout setting available on FF in about:config

mine is currently set to 300 which i assume means 5 minutes.

lucy24

10:16 pm on Nov 1, 2021 (gmt 0)

I always forget about (haha) about:config

:: detour ::

Criminy, FF really really doesn’t want you to use this feature which they have intentionally created and made available, do they.

Mozilla docs confirm that it's 300 seconds. And they do make it easy to see which ones are the never-modified defaults. There's also a network.http.connection-timeout which defaults to 90, so now to figure out what each one does ... (It is probably futile to guess why some of these settings have a . dot as word separator while others have a - hyphen.)

SumGuy

4:24 am on Nov 2, 2021 (gmt 0)

Yea, I'm talking about the router dropping the requests silently, giving no response. This is a breakdown of how 96% of the rejected port-80 hits look:

3 / 33.37%
1 / 14.63%
4 / 8.27%
5 / 8.05%
6 / 7.77%
2 / 7.48%
10 / 3.66%
7 / 2.98%
8 / 2.98%
12 / 1.97%
9 / 0.96%
20 / 0.96%
14 / 0.84%
16 / 0.62%
18 / 0.62%
15 / 0.45%
28 / 0.39%
30 / 0.34%

So 1/3 of the blocked IP's hit me with 3 requests before going away. The next most common hit frequency is 1 (ie a single hit). The rejected port 443 hits are very similar, the most common hit frequency is also 3 (44%) followed by 1 (13%) followed by 2, 5, 6, 4 and 10. What I haven't looked at here is the time between hits.