javax.net.ssl.SSLException: bad_record_mac

While writing a java net application I've encountered an interesting problem, which solution was not very trivial. The task was to connect to one website, parse the output, and write extracted information to a file. At first glance the task is "a piece of cake", but the problem appeared at the stage of connecting to the web site. The site uses the https protocol, and I had to use something more complex than basic java's HttpURLConnection. I've tried to use java's HttpsURLConnection, but while connecting to the site I received the following exception, which got me stuck me for a while: javax.net.ssl.SSLException: bad_record_mac. Eventually the problem was solved and the solution is described below.

Firstly, I will describe classes that were used to perform the connection. I've decided not to use internal java api directly and to look for some 3-rd party library, because besides connecting to the site, I had to execute some post and get queries to get required information and I wanted to do that with less pain. I chose to use the HttpClient library from Apache commons project. Here is the basic connection example that uses that library:

HttpClient httpclient = new DefaultHttpClient();
try {
    HttpGet httpget = new HttpGet("http://www.google.com/");
    ResponseHandler<String> responseHandler = new BasicResponseHandler();
    String responseBody = httpclient.execute(httpget, responseHandler);
} finally {
    httpclient.getConnectionManager().shutdown();
}

However, it won't work with some https websites. It does work with many of them, but some of them fail with different ssl exceptions. The site I was connecting to firstly failed with the following exception:

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

That means that the certificates were not verified correctly. Therefore, I had to implement more complex solution to handle different SSL cases. Here you might look for some other 3-rd party library that will handle secure socket connections by itself, however you might not get the full control. You can use this advice from the HttpClient manual and implement everything on your own. Eventually, everything can work fine OR you might get the exception mentioned at the beginning. If you're lucky one - my congratulations. If not - keep reading :)

In order to get the better understanding I decided to implement everything based on HttpClient and then debug it to see where it fails. I didn't care about checking the certificates and trusting the connection and just wanted to get the page contents. I've used examples from this blog post that I found pretty interesting. Therefore, I've created MyConnectionFactory that extends SSLSocketFactory. It initialized the SSL context with my own TrustManager and then uses it to create secure sockets.

public static class MySSLSocketFactory extends SSLSocketFactory {
    private SSLContext sslContext = SSLContext.getInstance("SSLv3");
    public MySSLSocketFactory() throws Exception {
        super((TrustStrategy) null, new AllowAllHostnameVerifier());
        sslContext.init(null, new TrustManager[] { myTrustManager }, null);
    }
    public Socket createSocket(Socket socket, String host, int port, boolean autoClose) 
            throws IOException, UnknownHostException {
        return sslContext.getSocketFactory().createSocket(socket, host, port, autoClose);
    }
    public Socket createSocket() throws IOException {
        return sslContext.getSocketFactory().createSocket();
    }
}

I created the implementation of TrustManager that basically does nothing in its methods - so it skipped any security check. In other circumstances its probably not a wise thing to do, but for my task that was enough, so I didn't want to complicate things.

TrustManager myTrustManager = new X509TrustManager() {
    public void checkClientTrusted(X509Certificate[] chain, String authType) 
        throws CertificateException {    }
    public void checkServerTrusted(X509Certificate[] chain, String authType) 
        throws CertificateException {    }
    public X509Certificate[] getAcceptedIssuers() {
        return null;
    }
}

Then I put everything together by registering my socket factory for the port 443 connections and then creating the HttpClient with my connection manager. The code to setup HttpClient looks like this:

private static HttpClient setupHttpClient() throws Exception {
    SSLSocketFactory sf = new MySSLSocketFactory();
    SchemeRegistry registry = new SchemeRegistry();
    registry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
    registry.register(new Scheme("https", sf, 443));
    ClientConnectionManager ccm = new ThreadSafeClientConnManager(registry);
    return new DefaultHttpClient(ccm);
}

With this code I was able to connect to my https website, but I started to get the exception javax.net.ssl.SSLException: bad_record_mac. And that's where all the fun started - looking into Internet and debugging to get the solution.

First of all, I've downloaded source code for HttpClient and started to debug to see where exactly the problem occurs. It was interesting to find out that the problem occurs when the getSession() method is called on SSLSocket. Java performs the SSL handshake and it fails. So, I went to the Internet to find the solution, because very likely the problem was solved before.

Quite interesting looks the following stackoverflow answer to similar question, which says that the problem could be due to the load balancing, when the MAC address changes during negotiation. It could be the reason that the problem occurs sometimes. However, in my case the exception appeared every time, so I continued to search.

Eventually, I found two different sources that pointed out for the reason and how to overcome it: one is the knowledge base article for JIRA client connecting to server, and another is last coderanch answer. The problem appears because Java tries all kinds of secure socket protocols (TLS, SSLv2, SSLv3, etc.) even if only one is supported by the server. This results in the aborted connection. Trapping this error can be hard if you don't have access to the server. But if not the load balancing, then the reason could be either unsupported protocol, or unsupported cipher algorithm.

The solution is to configure Java in such a way that it will use only one secure socket protocol for communication. Therefore, in both createSocket methods of MySSLSocketFactory I've added the following code after creating sockets:

((SSLSocket) socket).setEnabledProtocols(new String[] { "SSLv3" });
((SSLSocket) socket).setUseClientMode(true);

And it worked! That solved my problem. The handshake went fine, the connection went fine, and I was able to do my task. In conclusion, I can say, that while writing this blog post, I could not reproduce the problem. :) So, maybe, it really has something to do with load balancing - I didn't dig into it. In any case, debugging of SSL problems can be tough, so enabling Java net log by calling System.setProperty("javax.net.debug", "all"); can be very helpful.

The testing program with working example with SSL connection can be downloaded here: http://goo.gl/JFdw5

Read More...

Installing X.509 certificates on Nokia S60 3rd edition device

Sometimes you need to install your own X.509 certificate on the phone in order to authorize some web site, Exchange server, or installed software, or in general to use TLS (SSL) connection. According to the guide, provided on Nokia forums – it a piece of cake! However, sometimes problems may arise, and you start wondering about what can go wrong. The next thing you do is try to find the answer somewhere on the Internet. Well, you'll get a whole bunch of different posts about positive or negative experience, you'll find different solutions that are sometimes weird, sometimes don't work, and sometimes work but not on your phone. In general, most solutions are correct, you just need to pay attention to one nobody-ever-mentioned thing.

At first, a small amount of theory, so you'll know what I am talking about. X.509 certificate files have several extensions. Here is the description from Wikipedia for the most common:

• .pem - Base64 encoded DER certificate, enclosed between "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----"
• .cer, .crt, .der - usually in binary DER form, but Base64-encoded certificates are common too
• .p12 - PKCS#12, may contain certificate(s) (public) and private keys (password protected)
• .pfx - PFX, predecessor of PKCS#12 (but most of the time the actual data is PKCS#12, only the extension is kept)

If you are not into technical very much, you just need to know that .pem files are text files. So you can open them in any text editor (like GEdit or Notepad) and see bunch of letters and numbers after the first line, which says “-----BEGIN CERTIFICATE-----”. Those letters and numbers is actually a specially encoded binary certificate. Now, the files with extensions .cer, .crt, .der are usually (but not always!) binary files. That means that you cannot view their contents with usual text editor. But pay attention to what is being said at the end for this kind of files: “... but Base64-encoded certificates are common too”! That means, that you can have a certificate file with, for example, .cer extension, but it will be a text file actually encoded as .pem file! And, what's the difference, you ask, why should I care?

Well, here we came to the most interesting part. Nokia phones on S60 3^rd edition platform cannot import .pem certificates! They cannot import certificates that are represented as text files. You need a binary DER encoded file. And what's really exciting – you can have a .cer, .crt, or even .der file, but it will be a text file that was renamed from .pem! Well, your computer web browser will cope with it just fine. You computer mailing program will cope with it just fine, too. But not your Nokia phone!

If you are still wondering on “how do I know what kind of certificate file I have?”, then try to do a simple thing. Open the file with any text editor and see: if it opens correctly and has the first line as “-----BEGIN CERTIFICATE-----”, then this is a text .pem file. If it doesn't open correctly or has something really crazy inside – then it is probably a binary .der file.

If you have a binary certificate file, then just transfer it to the phone and open with phone's File Manager. It will ask you to import the certificate. Just like the mentioned tutorial says. What if you have a text certificate file? Then you should convert it to binary representation. There is an open source and free program that can do it for you, called OpenSSL. A compiled version of OpenSSL for Windows can be found here. Ubuntu users just type in terminal: sudo apt-get install openssl.

So, you've got the program! Next thing you do is type in terminal the following command:

openssl x509 -outform der -in certificate.pem -out certificate.der

In place of 'certificate.pem' you should put your text certificate file (it can have extensions .pem, .cer, .crt, .der but it should be text file) and in place of 'certificate.der' you just put any name you want. After executing this command (pressing Enter in terminal after typing it there) you should get in your directory a file named 'certificate.der' (or whatever you've put there), which will be a binary DER encoded certificate file. After that you do the same thing that I wrote before – transfer file to the phone and open it with File Manager. And you're done!

Now, lets mention the .p12 files. As I said before that are the binary files containing public certificate and password-protected private key. According to the already beloved Nokia guide this kind of files should be supported for installing on the phone. Well, not necessarily. In fact, unfortunately my Nokia N73 phone with the latest firmware does not import .p12 certificate files. I don't know why. Nokia support does not know either. I didn't dig it deeper into the problem, because I actually didn't need to install the certificate with private key – I probably will not decrypt any content with my phone that was signed or encrypted with public certificate (that is why you actually need a private key). But if you still need to import just the certificate from the .p12 file, you need to extract it first. The following command can be used:

openssl pkcs12 -in keyStore.p12 -out certificate.pem -nodes -nokeys

In place of 'keyStore.p12' you put the name of your .p12 file and in place of 'certificate.pem' you put any name. After executing this command you will get a text certificate file. Please, pay attention – it will be a text certificate file. You will need to convert it to binary format with the command mentioned before.

Well, that's about it! Now you should have no problems with installing your own X.509 certificate files on the Nokia phone. I didn't test the other versions of Symbian OS for the support of text certificate files, because I do not have many Nokia phones, except my own N73 :) But I think that S60 4^th and 5^th editions do not support then either.

Read More...