My blog has moved! Redirecting...

You should be automatically redirected. If not, visit and update your bookmarks.

02 July 2008

HTTP/1.0 Is Dead

Update - it seems the pesky problem we were having for HTTP/1.1 is actually still relevant to us. The problem is we cannot force .NET framework to properly close a response stream within the HttpWebResponse - the thread just waits forever. Sucks, but it appears we have to use HTTP/1.0 for the time being, at least until we manage to hack it someway (understand how the class works internally and hack it away).

Working for the Diggers team in a search engine, a large chunk of our tasks is writing sophisticated web crawlers.

We were using HTTP/1.0 because we didn’t need any features in 1.0, and to circumvent a problem we thought we had with 1.1. On such a crawl mission, I discovered today that MSN’s site does not support HTTP/1.0.

After another check we found that the above problem does not affect us, because it is only relevant the Microsoft Compact Framework. But it’s still a rather shock to find a large site (even though it’s Microsoft) that utterly does not support HTTP/1.0.

Try it yourself – go to about:config in Firefox and set network.http.version to 1.0, and then surf to MSN. You’ll get a blank page. What actually happens is MSN assumes you’re using HTTP/1.1, and sends you a chunked response. Your web browser will not parse a chunked response, and rightly so.

(Needless to say that HTTP is supposed to be backward compatible).

No comments: