Forum Moderators: phranque

Processes serving WebSocket connections hang after graceful reload

         

gargleblaster

7:37 pm on Aug 4, 2025 (gmt 0)



Hi all!
We are running Apache as a reverse proxy for WebSocket connections (in addition to serving a simple PHP website). We are using event MPM.

Every time Apache does a graceful restart (e.g. due to logrotate) we and up with at least one fully defunct Apache process, because it sets all the workers that are serving the WebSocket connections into 'G' state – which is how they will stay forever of course, as the WebSocket connections are meant to persist (that is what WebSocket are for!) and normally will not terminate. That means that the whole process these workers belong to becomes useless (also all other workers stop accepting connections during graceful restart) and sooner or later we end up with a fully defunct Apache server with error "AH00485: scoreboard is full, not at MaxRequestWorkers"

Question:
How can I tell Apache to do the "normal" graceful termination of "normal" connections (= wait for normal HTTP connections to finish serving the current request and close), but simply kill WebSocket connections (because they won't close/finish by themselves and have a re-connect logic anyway) so that I can perform graceful reload without "zombie-ing" processes.

If this is not possible Apache seems to be unusable for reverse-proxying WebSocket connections...:(

tangor

11:06 am on Aug 5, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



@gargleblaster ... Welcome to Webmasterworld.

Have you checked this source?

[serverlab.ca...]

gargleblaster

10:14 am on Aug 11, 2025 (gmt 0)



Hey @tangor - thx for your response! Unfortunately this source does not (from what I can tell) go into the issue I described. Also, the setup described there uses proxy_wstunnel, which is not recommended anymore.

See
[httpd.apache.org...]
"Since Apache HTTP Server 2.4.47, protocol Upgrade (tunneling) can be better handled by mod_proxy_http."

Brett_Tabke

12:49 pm on Aug 11, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Have you explored GracefulShutdownTimeout directive? I think it was 2.4 that introduced that? From what I read on apache docs, I think it kills websockets after waiting for all workers to finish their current requests.

Just add to your .conf
GracefulShutdownTimeout 30

gargleblaster

1:54 pm on Aug 11, 2025 (gmt 0)



Hey Brett, thx for the reply. Yes, in the meantime I have and am in discussion about it over at ApacheLounge [apachelounge.com...]
Unfortunately it does not do what the documentation (at least to me) suggests it should. At least not with the event MPM (which I must use). I am setting it to a few seconds but Apache still never terminates the effected workers and the whole process stays in "Stopping = yes (old gen)" mode forever.

gargleblaster

8:13 am on Sep 27, 2025 (gmt 0)



A remark about GracefulShutdownTimeout: It turns out that this option is solely for a graceful-stop, not the normal graceful (restart) apache2ctl command! During a graceful restart it does nothing…

Brett_Tabke

11:22 am on Sep 27, 2025 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Interesting - I did not understand tath..