vSphere 6 makes managing SSL certificates a lot easier than previous releases. It ships with its own Certificate Authority, (VMCA) that issues certificates for all components on your behalf, rather than having to replace each service certificate manually, or relying on self-signed certificates. This new VMCA comes with the Platform Services Controller (PSC) that can be installed as a separate appliance, or embedded within the vCenter Server installation or Appliance.
By default, the VMCA will self-sign its own certificate to be used as a CA certificate that will sign all requests for certificates. This self-signed CA certificate can be replaced by a certificate that is signed by a 3rd party root CA or your own root CA. Any certificate signed by the VMCA, which is an intermediate CA to your root CA, can then be validated by clients with the root CA and VMCA certificates installed.
Replacing the VMCA Root Certificate is very simple. The basic steps required are:
1. On the PSC node run the certificate-manager tool (on the VCSA, it's located under /usr/lib/vmware-vmca/bin/certificate-manager) to generate a new SSL certificate request key and csr file.
2. Submit the CSR to your enterprise certificate authority or a commercial certificate authority for signing
3. Combine the newly signed certificate and all other CA certificates in the chain into a single file
4. Run the certificate-manager tool again to import the signed certificate chain file and key
IMPORTANT!
If you are looking to replace the VMCA root certificate on a newly deployed vCenter server with an empty inventory (i.e. no ESXi hosts have been added to the vCenter inventory as yet), the VMware documentation suggests that you replace your VMCA certificate first before adding your ESXi hosts to the vCenter Server Inventory. Although this suggestion by the official documentation is understandable and makes sense, I would not recommend it unless the new certificate that you will be using to replace the VMCA certificate with was signed AT LEAST 24-HOURS before attempting this procedure.
Here is why: The VMCA will replace vCenter certificates as well as ESXi certificates. When adding a new ESXi host with a self-signed certificate to the vCenter Server Inventory, the VMCA will sign a new SSL certificate for the "new" ESXi host as part of the process. The problem is, that VMware in their infinite wisdom have decided to backdate all new ESXi certificates by 24-hours to "avoid time-sync" issues. "24-hours?" I hear you say? Yes, that is what I said. 24 hours. What does this mean?
Well, if you generate a new CSR using the certificate-manager tool and then get it signed immediately by your root CA and install it straight away to the VMCA, the VMCA will use that certificate that was only signed a few minutes ago, to sign a new SSL certificate for ESXi, however, the ESXi certificate will be backdated by 24 hours, which means that it would seem to have been signed by a root CA at a date and time at which the signing VMCA certificate itself didn't even exist. It will error with a time error and you will not be able to add a new ESXi host to your vCenter Inventory for at least 24 hours. If you are deploying a new environment on a customer site, a 24 hour delay in building out your environment could be costly.
The workaround is to add your ESXi hosts to the new vCenter Server Inventory first. Once added, you can replace the VMCA root certificate without a problem.
I ran into this issue when I was rebuilding my lab. I read the VMware documentation, which stated I should replace my VMCA certificate first and then add ESXi hosts. In the end, I could probably have rolled back the SSL certificate change on the VMCA, but I opted to destroy the empty vCenter server and redeploy.
Back to the Certificate Replacement Procedure
1. Enable Bash and SSH on the VCSA
2. Log into the VCSA using Puty via SSH
3. This is optional, but will save time later on. Edit the file /usr/lib/vmware-vmca/share/config/certool.cfg
# # Template file for a CSR request # # Country is needed and has to be 2 characters Country = GB Name = VirtualvCP VMware Certificate Authority Intermediate CA Organization = VirtualvCP OrgUnit = VirtualvCP Labs State = Lincolnshire Locality = Spalding IPAddress = 127.0.0.1 Email = Hostname = vcenter.spiesr.com
1. Run /usr/lib/vmware-vmca/bin/certificate-manager
2. Select Option 2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates
3. Enter the administrator@vsphere.local SSO account password
4. Select option 1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
5. Type in an Output directory path: /var/tmp/
6. Select Option 2. Exit certificate-manager
7. Back at the Bash shell, type cat /var/tmp/root_signing_cert.csr
8. Copy the entire block of text, including the
-----BEGIN CERTIFICATE REQUEST-----
and
-----END CERTIFICATE REQUEST-----
lines.
Get the request signed by your root CA. Once received back from your CA, form a full certificate chain that includes the new certificate as well as the CA certificate in one file and save the file to /var/tmp/root_signing_chain.cer
NOTE: The first Certificate in the file is the new VMCA root CA certificate that was signed by my own CA. The second certificate in the file is my own root CA certificate that was used to sign the new VMCA root certificate. This completes the entire certificate chain in a single file.
-----BEGIN CERTIFICATE----- MIIFPjCCBCagAwIBAgIKYQ/hGwAAAAAASjANBgkqhkiG9w0BAQUFADBMMRMwEQYK CZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPyLGQBGRYGc3BpZXNyMR0wGwYDVQQD ExRWaXJ0dWFsVkNQIFNwaWVzUiBDQTAeFw0xNTA3MDQxOTQ4MzNaFw0yMDA3MDQx OTU4MzNaMGcxCzAJBgNVBAYTAlVTMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEV MBMGCgmSJomT8ixkARkWBWxvY2FsMRswGQYDVQQKExJ2Y2VudGVyLnNwaWVzci5j b20xCzAJBgNVBAMTAkNBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA +/jA4l2pggvNEb6lMnOL3fH4cckT9dqSXtzCaTwXyb2paHzl5CYg8qNMqTozonUJ 8RC+vZSwlGq3ydDvF2cDFiLwgzMZkAld0ethhW25sGqNu1Q8ZNJ9ny1BD3YsZiGH IfDOo7k/J7uoXu7VtewCjEH8sVRJEUqGO3IG0SzHWW8EXqkn9qeQTZ2lk2k88xW2 h+MPMtuwzoCiRMcpNo6uE7Yn4Zdu0DJUysEVrJhOUyz6U6RMrd9UVlctKFWGjz8j X/VH5vZSzVCLTYqG3oZpctKju3qbY1v/3ATEqDHGmq7M+22OeW426f8t0FNsXAcB piN3GLve7JJKipjGHuZB/QIDAQABo4ICBTCCAgEwCwYDVR0PBAQDAgGGMA8GA1Ud EwEB/wQFMAMBAf8wHQYDVR0OBBYEFL/1eKkojcR7OGcjkkZu4RIfgwqAMB8GA1Ud IwQYMBaAFLqfRbJo3OZIIUzuVo4xaPD0tj0SMIHUBgNVHR8EgcwwgckwgcaggcOg gcCGgb1sZGFwOi8vL0NOPVZpcnR1YWxWQ1AlMjBTcGllc1IlMjBDQSxDTj1SU0RD MDEsQ049Q0RQLENOPVB1YmxpYyUyMEtleSUyMFNlcnZpY2VzLENOPVNlcnZpY2Vz LENOPUNvbmZpZ3VyYXRpb24sREM9c3BpZXNyLERDPWNvbT9jZXJ0aWZpY2F0ZVJl dm9jYXRpb25MaXN0P2Jhc2U/b2JqZWN0Q2xhc3M9Y1JMRGlzdHJpYnV0aW9uUG9p bnQwgckGCCsGAQUFBwEBBIG8MIG5MIG2BggrBgEFBQcwAoaBqWxkYXA6Ly8vQ049 VmlydHVhbFZDUCUyMFNwaWVzUiUyMENBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXkl MjBTZXJ2aWNlcyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPXNwaWVz cixEQz1jb20/Y0FDZXJ0aWZpY2F0ZT9iYXNlP29iamVjdENsYXNzPWNlcnRpZmlj YXRpb25BdXRob3JpdHkwDQYJKoZIhvcNAQEFBQADggEBAB8BpQ49zYojs5UNeb1m AsQnAhGq6Ef4jqzFTBnt7fWsDwNHVoDqaRiYEUlXEH6/wO6OFplNdw+SLLjVXBN1 nJp+MpKruaTy10+rbFlcqFSHpozQCEBdLLeB8LP7ESlmNqMj+qJDosAZmE7oDFUR 6sXU1vbaV7UnwzYVDT3Svl8shZSuZwQ7UhO6Zq0Nrn5HoEDAdvSLEDd+myz/w32R 5HPNQ04AKt/4xNzg9xgu4NSuk5QBTrrMKQM3OFCXMv7m4X2v8MxXREHgr5HjQHJm gl28y8a0MePMnKO/im/je1cp9GwkB+RX618tLByndc4V8UDB8jLh0LPndx/7fNOx NHU= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDiDCCAnCgAwIBAgIQa+TfxvTkyLRO+ZDRHNeWpzANBgkqhkiG9w0BAQUFADBM MRMwEQYKCZImiZPyLGQBGRYDY29tMRYwFAYKCZImiZPyLGQBGRYGc3BpZXNyMR0w GwYDVQQDExRWaXJ0dWFsVkNQIFNwaWVzUiBDQTAeFw0xNDAzMjAyMjM1NDRaFw0z NDAzMjAyMjQ1NDJaMEwxEzARBgoJkiaJk/IsZAEZFgNjb20xFjAUBgoJkiaJk/Is ZAEZFgZzcGllc3IxHTAbBgNVBAMTFFZpcnR1YWxWQ1AgU3BpZXNSIENBMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwY2axOWg1euyQjUakLzOgxKDh1xH DNWt944p/ATJg51LhjgNSmXcpaJ50+04L/kHOPbrsJ7KwXRQ1HFnR610+a3k1rNT R8B6dkPuhaucOcF6tehnsdkBbSETwmUfVhe2xexFZYaxtpJ/NPWYoNIfca1fQ62m Wic5BT6wuKvdIwgVJpW+G7EfEHvbNd5krecPW53TQeXtqKxDyarkQPIvaBQIusly lycDH6pFC2WXoptvbeMrP36X35MuV7n9udNztnJix1jQmlcTzV6c04YPcAnAZY7y x/XwcopDi65r2q646wlEbYX+EBu03zMZkPD+DccTeUY+m3sCW+R/W+rZxwIDAQAB o2YwZDATBgkrBgEEAYI3FAIEBh4EAEMAQTALBgNVHQ8EBAMCAYYwDwYDVR0TAQH/ BAUwAwEB/zAdBgNVHQ4EFgQUup9Fsmjc5kghTO5WjjFo8PS2PRIwEAYJKwYBBAGC NxUBBAMCAQAwDQYJKoZIhvcNAQEFBQADggEBAAjTrqsNzlwiNhdL2jP/wCfCiDip t7ibDq0zbNLnUDUSCS/+GwErmy1NQfnrDjFeJ7mVYYiL6PZYSAlqyuImKeqYA517 aHIJxNMHM37s5MGcftG3WcoMxbHOtjC2tHhRr1I1DTrKbA7kmXhact937FThG59Q +lo5pLRAhJrqKwCw6Ur1BxGWR7hxu+/5zNr0L+ZqGed51fTf/dqzC2FDksrvdURo roPQhCs/bqg0HVG2srwjX+qgBAPuiHAIL/Lho2n/JbhPgngsGTj7EcgKoXspEkDD ZOY3nWVQsa3ec9Hu+xe3v+sB6fpoBoVfZ18xFZS0yvO1o9ics1cND3uEbFc= -----END CERTIFICATE-----
Replacing the VMCA Root Certificate:
1. run /usr/lib/vmware-vmca/bin/certificate-manager
2. Select option 2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates
3. Provide valid SSO password to perform certificate operations.
4. Select option 2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate
5. Please provide valid custom certificate for Root.
File : /var/tmp/root_signing_chain.cer
6. Please provide valid custom key for Root.
File : /var/tmo/root_signing_cert.key
7. Please provide valid file location, couldn't find file : /var/tmo/root_signing_cert.key
File : /var/tmp/root_signing_cert.key
8. You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : Y
9. The VMCA Root Certificate will now be replaced. Once replaced the tool will prompt you for some properties. This is so that it can generate a new machine certificate. If you are running the PCS embedded, the properties required are:
Name: Name of the entity who the cert is being issued to . This can be anything
Organization: Your Org Name
OrgUnit: Your Org Unit
State: Your state
Locality: Your town/city
IPAddress: Optional
Emai: blah@blah.com
Hostname: This is the FQDN of the service that will be using the certificate. In the case of this post, this certificate was issued for vcenter.spiesr.com, which is my vCenter server FQDN.