Update SSL Certs in ESXi Server

I try to use the vSphere client as much as possible when managing my home ESXi server, but sometimes the UI breaks or otherwise isn't useful. Such is the case with updating SSL certificates. I installed a commercial certificate on the server a year or two ago which subsequently expired, but when I tried to update it, the web UI wasn't cooperating with what I was supplying, plus wasn't giving any useful feedback on what the actual problem was.

It shouldn't be this hard, but here's the hassle-free way of doing it via SSH:

First, make sure that your new certificate is broken out into three different parts:

1. The private key
2. The certificate
3. The CA intermediate certificate chain

All three of these items MUST be in PEM format. To convert a p12 certificate into its disparate private key and public certificate parts, use these two commands:

openssl pkcs12 -in certificate.p12 -out newfile.crt.pem -clcerts -nokeys
openssl pkcs12 -in certificate.p12 -out newfile.key.pem -nocerts -nodes

Getting the correct certificate chain for your certificate can be a little tricky, but I found this great resource for obtaining it. Make sure that you paste in the certificate, NOT your private key. Put all of the text from that service into a file with a .crt extension and save it. You can then convert to PEM format by running the following command:

openssl x509 -in mycert.crt -out mycert.pem -outform PEM

Now that you have all of your necessary certificate items in place, we should turn our attention to logging into your ESXi server.

You'll need to (temporarily) enable SSH access to your server and remote into it. Take a look at the official VMWare KB article on how to enable SSH on your server for more details.

Once remoted into your server, go ahead and cd into /etc/vmware/ssl. Create a backup of your existing SSL files by copying/moving them:

mv castore.pem castore.pem.bak  
mv rui.key rui.key.bak  
mv rui.crt rui.crt.bak

Next, use your favorite CLI text editor to create these files anew. In case it isn't obvious: castore.pem is your certificate chain PEM, rui.key is your private key, and rui.crt is your certificate.

Once you have that done, you can restart your management services. The process for doing this can be found in this KB article: Restarting the Management agents in ESXi.

Now, if for some reason you didn't follow these procedures and the SSL certs were not installed correctly, your vpxa service process is going to throw a fit and not want to show the web interface. DO NOT panic. Just roll back the files that you changed from the originals:

mv castore.pem.bak castore.pem  
mv rui.key.bak rui.key  
mv rui.crt.bak rui.crt

Restart the management services and now start this article over again.

If everything went perfectly, make sure to turn off SSH access (security first!) and now you can enjoy another year or two of SSL protection for your server.