Hey folks, can't find much on this error after scouring the web and many blogs. Basically the Certificate replacement process (Option #2) looks like it's completing successfully then hangs at 85% during the " Starting Services" phase, which then gives a message about services failing to start due to a timeout and proceeds to roll back this process however also fails and I'm left with a semi-bricked appliance unless I restart.
Here's what I'm doing:
I have a small lab on VMware Workstation 12 running ESXi 6.5d and vCenter appliance 6.5d (I've also tried A, B patches to see if this is a new bug introduced in version D) along with Horizon View 7. I want to have the vCenter appliance act as a "Subordinate CA", replace the root cert with the appliance using a Certificate generated by my CA server, and Automate the replacement of all Certificates, including ESXi hosts done by VMCA with Signed CA certs. I have a single tier PKI using SHA 384 and 4096 bit key.
I've spent several days/nights looking over many articles and videos and I don't appear to be missing any Critical step with the setup of my Certificate Template for vSphere 6.0 VMCA, configuration of my CA, or the Certificate Signing process itself, but something is wrong somewhere.
What I've done so far. vSphere 6.0 Environment with Custom Certificates (External PSC) - YouTube
1. I've installed a Server 2012 R2 Root CA in Enterprise mode with Certificate Web services and have created the template per guidance of this article and this video.
2. I've patched the vCenter to the latest build which is Version D Build #
3. I've duplicated the Subordinate CA Certificate template and have customized it per VMware guidance
4. I'm using VMCA Cert Tool to generate the CSR
5. I am able to successfully generate a certificate based on this CSR
6. I'm able to upload the cert chain and key file provided by the vCenter appliance into the cert-tool during the process for Option # 2 from the main menu
7. the Process executes and looks to be updating and replacing all of the certs using the Certificate i've generated for VMCA
8. the process fails @ 85% when attempting to start the services again
9. I've exhausted most of my troubleshooting and knowledge in this area
However, I'm running into this weird error when attempting to run through the process. As described in this Article.
Replacing a vSphere 6.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate (2112277) | VMwar…
As soon as I get to 85% starting services, it hangs for several minutes and then errors out and rolls back everything. Upon examining the logs, I can find no clear indication of what is failing outside of services not starting, which does not make sense is the fact that the certificate replacement was successful per the logs, why would a failure to start these services cause the entire process to roll back?
The thing that boggles me is that I can confirm this in the /storage/log/vmware/vmcad/certificate-manager.log, I receiving messages that would lead one to believe that the certificates were successfully replaced along the way.
2017-05-26T22:51:09.381Z INFO certificate-manager []
2017-05-26T22:51:09.382Z INFO certificate-manager Create a entry using Key and File generated earlier
2017-05-26T22:51:09.382Z INFO certificate-manager Running command :- ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'create', '--store', u'vpxd', '--alias', u'vpxd', '--cert', u'/storage/certmanager/rollback/vpxd_bkp.crt', '--key', u'/storage/certmanager/rollback/vpxd_bkp.priv']
2017-05-26T22:51:09.413Z INFO certificate-manager Command output :-
Entry with alias [vpxd] in store [vpxd] was created successfully
If i do a search for error the only items that show up are
Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start vapi-endpoint, vpxd-svcs services. Error: Operation timed out
there's also mention of this during the rollback, but I don't find it usefull at all...
2017-05-26T22:51:09.871Z ERROR certificate-manager 2017-05-26T22:51:09.833Z Updating certificate for "com.vmware.vim.eam" extension
2017-05-26T22:51:09.871Z INFO certificate-manager Command executed successfully
2017-05-26T22:51:09.871Z INFO certificate-manager Running command : ['/usr/bin/python', '/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py', '-e', 'com.vmware.rbd', '-s', 'vc1.lab.local', '-c', u'/storage/certmanager/rollback/vpxd-extension_bkp.crt', '-k', u'/storage/certmanager/rollback/vpxd-extension_bkp.priv', '-u', 'administrator@vsphere.local', '-p', '*****']
2017-05-26T22:51:10.109Z INFO certificate-manager Command output :-
2017-05-26T22:51:10.071Z Updating certificate for "com.vmware.rbd" extension
2017-05-26T22:51:10.109Z ERROR certificate-manager 2017-05-26T22:51:10.071Z Updating certificate for "com.vmware.rbd" extensio
Any thoughts folks?