Coding HTTP interaction is a common task in mobile app development. It can be one of the trickiest if not done right. There’s a lot of edge cases to consider in the HTTP protocols –  different versions of the protocol, different server implementations, firewalls and routers to consider, etc.

Blackberry developers had been hamstrung for years by the more rudimentary net.rim.device.api.io.http package, or even plain old java URLConnection, and have had to deal with coding more of the HTTP protocol implementation themselves or else bloating the size of their apps by including a third-party HTTP library. (There’s also the issues of routing through BES/BIS and MDS to contend with!).

The Apache HttpComponents suite is a great toolset for handling all the common http interactions in Java apps, and a lot of the more specialized ones too. It can even tackle WebDAV, Authentication extensions, and HTTPS connections. Thankfully, Android had the Apache HttpComponents included in the SDK by default since day one, so there’s no need to bloat out the size of your app with extra jars if you want the convenience of HttpClient.

The classic tradeoff in using any library is that, while it may save time and cost, it can also be a ‘black box’ that obscures some of the details and makes it harder to track down and fix problems. No such problems with HttpClient though because it is well-designed (i.e. extensible) and is open source.

Here’s an example. I had an Android app interacting perfectly well with a test server, then switched it over to access Yahoo’s web server on the internet. Instant problem: HTTP 502 Proxy Error.

So why didn’t it work? In the end, the solution is simple (but finding it was a little tricky). This is where the design of the HttpComponents library really helped. HttpClient allows logging of all HTTP chatter through the standard Java logging api. Here’s an example of setting-up logging to the standard Java error console.

System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.SimpleLog");
System.setProperty("org.apache.commons.logging.simplelog.defaultlog","trace"); 
System.setProperty("org.apache.commons.logging.simplelog.showdatetime", "true"); 
System.setProperty("org.apache.commons.logging.simplelog.log.httpclient.wire.header", "trace"); 
System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.commons.httpclient", "trace");

Note: I had to do this with a java console app because commons logging won’t configure in Android via the system properties – that’s a job for another day!
Here’s the trace it produced:

[DEBUG] headers - >> REPORT /dav/user/Calendar/MyCal/ HTTP/1.1
[DEBUG] headers - >> Depth: 1
[DEBUG] headers - >> Authorization: Basic [redacted]
[DEBUG] headers - >> Content-Type: application/xml; charset="UTF-8"
[DEBUG] headers - >> Content-Length: 296
[DEBUG] headers - >> Host: caldav.calendar.yahoo.com
[DEBUG] headers - >> Connection: Keep-Alive
[DEBUG] headers - >> Expect: 100-Continue
[DEBUG] wire - >> "<C:calendar-query xmlns:D="DAV:" ...
[DEBUG] wire - << HTTP/1.1 502 Proxy Error[EOL]"

HttpClient was sending the request ok, but the server was immediately returning the 502 Proxy Error and no data. The request headers show that HttpClient was sending an Expect: 100-Continue header. (The Expect-Continue mechanism was introduced for HTTP 1.1 [RFC 2616] to make HTTP interaction more efficient).

Proxy Servers are very common on the internet. They buffer and insulate hosted environments from attack, balance loads, etc. The thing is – a proxy server uses the lowest common denominator of the HTTP protocols (HTTP 1.0) because it can never assume the client understands HTTP 1.1. Since Expect-Continue is a mechanism of the 1.1 HTTP protocol, a proxy server rejects it, even if the web server behind it can handle HTTP 1.1. It’s to do with authentication (which I won’t get into here). For this, it was enough to realise that the third-party server must have been using a proxy server.

One solution to this is to force HttpClient to use HTTP 1.0, but then you lose all the performance benefits of HTTP 1.1, which are especially important on a mobile device.

Yet again the excellent design of HttpClient comes to the rescue though. There is a configuration that you can set to prevent HttpClient from sending the Expect-Continue header over a HTTP 1.1 connection:

HttpClient httpclient = new HttpClient();
httpclient.getParams().setParameter("http.protocol.expect-continue", false);

It worked! HttpClient is great.