libURL hangs after getting response with keep-alive, header

[Dave Cragg]

On 24 Apr 2011, at 18:19, David Beck wrote:
> If I send a "Connection: close" with the httpHeaders, everything works as expected with no hang. In this case, by using using the logging feature in libURL, I could see that the response header have "connection: close" instead of "connection: keep-alive".

This is at it should be. Under HTTP 1.1, both sides of the connection must respect the "Connection: close" header. So if the server receives a "Connection: close" header, it is required to close the connection after completing the request, and send Connection: close in its response.


> It looks like there are several places were libURL tests for the "Connection: close" header in the response, but I don't see it testing for a "keep-alive" header anywhere.
> 
> Should I log this as a bug?

I don't think so, at least not specifically with regard to the keep-alive thing. There is no requirement to check for "keep-alive" headers, and as libUrl will try to re-use connections by default (following the HTTP 1.1 spec) it would serve no purpose.

The reason I suggested trying the "Connection: close" header in your request is that there appears to be a bug (in past versions of the engine anyway) that affects the way the openSockets list is maintained.  I filed a bug report in the past about a reproducible bug affecting Windows where closed sockets are not always removed from the openSockets list. It is a little complicated to reproduce, and at the time I only saw it on Windows. But I guess it's possible that there may be circumstances where it occurs on other platforms. (I can't access the bug database right now to check the status.)

In this case, the problem would occur when your application, after receiving a response from the server, does some intensive activity in a repeat loop, and during that activity, the server closes the connection. On a subsequent request to the server, the connection may still be listed in the openSockets and libUrl will try to use it. 

I've heard of some other cases where using  Connection: close helps, but I was never able to reproduce the problem.

It may be that libUrl is remiss in the way it handles these situations. But my memory is that it is difficult for libUrl to immediately detect an error when writing to a socket that the engine believes is open.

If it works, using "Connection: close" is probably an acceptable solution for most cases. I think that many virtual servers these days close the connection after each request anyway. I guess someone has calculated the trade-off between having to open a new connection each time and keeping connections open but unused.

Cheers
Dave