While writing a java net application I've encountered an interesting problem, which solution was not very trivial. The task was to connect to one website, parse the output, and write extracted information to a file. At first glance the task is "
a piece of cake", but the problem appeared at the stage of connecting to the web site. The site uses the
https protocol, and I had to use something more complex than basic java's
HttpURLConnection. I've tried to use java's
HttpsURLConnection, but while connecting to the site I received the following exception, which got me stuck me for a while:
javax.net.ssl.SSLException: bad_record_mac. Eventually the problem was solved and the solution is described below.
Firstly, I will describe classes that were used to perform the connection. I've decided not to use internal java api directly and to look for some 3-rd party library, because besides connecting to the site, I had to execute some post and get queries to get required information and I wanted to do that with less pain. I chose to use the
HttpClient library from Apache commons project. Here is the basic connection example that uses that library:
HttpClient httpclient = new DefaultHttpClient();
try {
HttpGet httpget = new HttpGet("http://www.google.com/");
ResponseHandler<String> responseHandler = new BasicResponseHandler();
String responseBody = httpclient.execute(httpget, responseHandler);
} finally {
httpclient.getConnectionManager().shutdown();
}
However, it won't work with some https websites. It does work with many of them, but some of them fail with different ssl exceptions. The site I was connecting to firstly failed with the following exception:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
That means that the certificates were not verified correctly. Therefore, I had to implement more complex solution to handle different SSL cases. Here you might look for some other 3-rd party library that will handle secure socket connections by itself, however you might not get the full control. You can use this
advice from the HttpClient manual and implement everything on your own. Eventually, everything can work fine OR you might get the exception mentioned at the beginning. If you're lucky one - my congratulations. If not - keep reading :)
In order to get the better understanding I decided to implement everything based on HttpClient and then debug it to see where it fails. I didn't care about checking the certificates and trusting the connection and just wanted to get the page contents. I've used examples from
this blog post that I found pretty interesting. Therefore, I've created
MyConnectionFactory that extends
SSLSocketFactory. It initialized the SSL context with my own
TrustManager and then uses it to create
secure sockets.
public static class MySSLSocketFactory extends SSLSocketFactory {
private SSLContext sslContext = SSLContext.getInstance("SSLv3");
public MySSLSocketFactory() throws Exception {
super((TrustStrategy) null, new AllowAllHostnameVerifier());
sslContext.init(null, new TrustManager[] { myTrustManager }, null);
}
public Socket createSocket(Socket socket, String host, int port, boolean autoClose)
throws IOException, UnknownHostException {
return sslContext.getSocketFactory().createSocket(socket, host, port, autoClose);
}
public Socket createSocket() throws IOException {
return sslContext.getSocketFactory().createSocket();
}
}
I created the implementation of TrustManager that basically does nothing in its methods - so it skipped any security check. In other circumstances its probably not a wise thing to do, but for my task that was enough, so I didn't want to complicate things.
TrustManager myTrustManager = new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain, String authType)
throws CertificateException { }
public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException { }
public X509Certificate[] getAcceptedIssuers() {
return null;
}
}
Then I put everything together by registering my socket factory for the port 443 connections and then creating the HttpClient with my connection manager. The code to setup HttpClient looks like this:
private static HttpClient setupHttpClient() throws Exception {
SSLSocketFactory sf = new MySSLSocketFactory();
SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
registry.register(new Scheme("https", sf, 443));
ClientConnectionManager ccm = new ThreadSafeClientConnManager(registry);
return new DefaultHttpClient(ccm);
}
With this code I was able to connect to my https website, but I started to get the exception javax.net.ssl.SSLException: bad_record_mac. And that's where all the fun started - looking into Internet and debugging to get the solution.
First of all, I've downloaded source code for HttpClient and started to debug to see where exactly the problem occurs. It was interesting to find out that the problem occurs when the
getSession() method is called on
SSLSocket. Java performs the
SSL handshake and it fails. So, I went to the Internet to find the solution, because very likely the problem was solved before.
Quite interesting looks the following
stackoverflow answer to similar question, which says that the problem could be due to the
load balancing, when the MAC address changes during negotiation. It could be the reason that the problem occurs sometimes. However, in my case the exception appeared every time, so I continued to search.
Eventually, I found two different sources that pointed out for the reason and how to overcome it:
one is the knowledge base article for
JIRA client connecting to server, and
another is last coderanch answer. The problem appears because Java tries all kinds of
secure socket protocols (TLS, SSLv2, SSLv3, etc.) even if only one is supported by the server. This results in the aborted connection. Trapping this error can be hard if you don't have access to the server. But if not the load balancing, then the reason could be either unsupported protocol, or unsupported
cipher algorithm.
The solution is to configure Java in such a way that it will use only one secure socket protocol for communication. Therefore, in both createSocket methods of MySSLSocketFactory I've added the following code after creating sockets:
((SSLSocket) socket).setEnabledProtocols(new String[] { "SSLv3" });
((SSLSocket) socket).setUseClientMode(true);
And it worked! That solved my problem. The handshake went fine, the connection went fine, and I was able to do my task. In conclusion, I can say, that while writing this blog post, I could not reproduce the problem. :) So, maybe, it really has something to do with load balancing - I didn't dig into it. In any case, debugging of SSL problems can be tough, so enabling Java net log by calling
System.setProperty("javax.net.debug", "all"); can be very helpful.
The testing program with working example with SSL connection can be downloaded here:
http://goo.gl/JFdw5
