It's 3am, Do You Know Where Your Certs Are?

So, things are trundling along on your Rails project. You’re doing things properly and have lots of tests, which get kicked off by a Jenkins CI server every time a developer pushes to the repository. Then, it comes time to replace your SSL certificate on the domain you use for testing. Suddenly, your tests start breaking hard with this error when they try doing fun things with your SSL-enabled API:

OpenSSL::SSL::SSLError
SSL_connect returned=1 errno=0 state=SSLv3 
read server certificate B: certificate verify failed

And yet, when you open a pry or irb session, you can connect to the URL in question fine and dandy. What’s going on?

Welcome to the wonderful world of OpenSSL::SSL::VERIFY_NONE. The Net/HTTP code in Ruby can be passed an SSLContext for setting up SSL connections. Unfortunately, the default verification step in a new SSLContext is OpenSSL::SSL::VERIFY_NONE, which as you can imagine from the name, does no checking whatsoever. So that explains why your pry or irb session is connecting fine.

But why is it failing in the tests? Well, chances are that you’re using something like HTTPClient for your REST calls, aren’t you? And these gems, being somewhat better behaved, set their SSL context to be OpenSSL::SSL::VERIFY_PEER, to at least check the chain of certificates. And that’s what’s failing - Ruby can’t complete the certificate chain.

Which leads to another question - you’ve paid out the money for the SSL cert, it comes from a reputable provider (ours was from RapidSSL), so why isn’t it verifying? Well, it turns out that as well as setting the default check to VERIFY_NONE, Ruby also doesn’t load any of the cert files in the OpenSSL gem; instead it just relies on the OS’s certs. Which is probably fine in most cases, but as it turns out, not if you’re running Ubuntu 12.04 and using a RapidSSL cert.

How do we fix this? Well, you could monkey patch either OpenSSL and/or Net/HTTP to ignore any options and always use OpenSSL::SSL::VERIFY_NONE. Please don’t do this. Ruby using it by default is a bad thing, leaving you open to MITM attack; HTTPClient is doing the right thing. What we really want to is tell Ruby to load its cert store up with with the certs both in the OS and the supplied OpenSSL gem (which includes the latest root GeoTrust cert for RapidSSL). Luckily, this is fairly simple once you know how:

http_client = HTTPClient.new
http_client.ssl_config.clear_cert_store
http_client.ssl_config.cert_store.set_default_paths

Magically, the RapidSSL cert now gets accepted and the Jenkins tests can go off and go their merry thing, hopefully preventing panic as the builds return to passing.