[OT] More on false timeouts and headers

[J. Landman Gay]

On 5/21/2015 3:42 AM, Dave Cragg wrote:
> Is there really a "Content-Transfer-Encoding" line among the headers?

Yup, there is, and it happens to occur before the Transfer-Encoding header.

> If so, that may be part of the problem. As far as I know,
> "Content-Transfer-Encoding" is not a valid http header. (It’s used in
> email transfers I think.) libUrl parses the headers looking for
> certain header types. It contains the line:
>
> put lineOffset("Transfer-Encoding:",laRhHeader[laUrl[x]]) into
> tCodeLine
>
> I think that will pick up any Content-Transfer-Encoding header if it
> exists before a Transfer-Encoding header, and therefore miss the
> "chunked" value, and so not try to de-chunk the data. (I guess that
> might be considered a bug in libUrl. Even if
> Content-Transfer-Encoding is not valid, something like
> X-Special-Transfer-Encoding is valid, and would probably trip up
> libUrl)

That's exactly what's happening, you've nailed it. I did a quick check 
yesterday and found over 1900 insertions of the chunk markers inside a 
550K file, so it's being sent to my script as a raw data stream. We 
still don't know what's inserting some of the header lines, it isn't 
anything our own scripts are doing. Something from AWS maybe.

So is this something you think I should report? Or is it just a 
side-effect of working with different servers? There is a workaround; 
specifying a Content-Length header appears to eliminate both the 
"Content-Transfer-Encoding" and "Transfer-Encoding: chunked" headers.

-- 
Jacqueline Landman Gay         |     jacque at hyperactivesw.com
HyperActive Software           |     http://www.hyperactivesw.com