Site Recovery Manager – vCenter Server SSL Replace

https://ssbkang.com/2014/10/23/site-recovery-manager-vcenter-server-ssl-replace/

ssbkang Virtualisation October 23, 2014 3 Minutes

Introduction

vCenter was rebuilt few weeks back, which replaced SSL certificate. Due to this, existing Site Recovery Manager (SRM) couldn’t communicate with vCenter servers anymore (The previous work I’ve done could be found here). To resolve this problem, I had to re-connect vCenter servers from SRM to accept new SSL certificate.

In this blog post, I will be going through how tackled this issue.

Environment

The following products were in place for this work:

vCenter 5.5
- Windows 2008 R2 server
SRM 5.5
- Windows 2008 R2 servers
External Database
- Microsoft SQL 2008 R2

Symptom

The symptom was, whenever I start Site Recovery Manager Service, it starts but within a few seconds it stops.

First attempt was made on investigating log files, located under C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs and found out that SRM wasn’t able to get the SSL certificate properly. The reason was because of the vCenter rebuild work which replaced existing SSL certificate to a new one. Log is attached below:

 2014-10-15T14:57:30.711+13:00 [02712 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000005590b50, TCP:vcenter.test.com:80>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters:
 --> PeerThumbprint: AA:BB:CC:DD:EE:FF:GG:HH:II:JJ:KK:LL:MM:NN:OO:PP:QQ:RR:SS:TT
 --> ExpectedThumbprint: TT:SS:RR:QQ:PP:OO:NN:MM:LL:KK:JJ:II:HH:GG:FF:EE:DD:CC:BB:AA
 --> ExpectedPeerName: vcenter.test.com
 --> The remote host certificate has these problems:
 -->
 --> * The host certificate chain is incomplete.
 -->
 --> * unable to get local issuer certificate)

Now what?

While looking at executable files under C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin, I found a script called srm-config.exe. Running this script, it had an option of updating vCenter server with the following arguments:

-u
- The user to communicate to vCenter servers
-vc
- vCenter server FQDN
-thumbprint
- New thumbprint
-cfg
- Configuration file, which is located under “C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config\vmware-dr.xml”
-sitename
- FQDN of SRM server

Ran the command as attached below and it was successful.

C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin>srm-config.exe
-cmd updatevc -u srm_administrator -vc vcenter.test.com:80 -thumbprint TT:SS:RR:QQ:PP:OO:NN:MM:LL:KK:JJ:II:HH:GG:FF:EE:DD:CC:BB:AA -cfg “C:\Program Files\
VMware\VMware vCenter Site Recovery Manager\config\vmware-dr.xml” -sitename srm.test.com

Result

2014-10-15T18:41:26.172+13:00 [03324 info 'Default'] Logging uses fast path: false
2014-10-15T18:41:26.172+13:00 [03324 info 'Default'] Handling bora/lib logs with VmaCore facilities
2014-10-15T18:41:26.172+13:00 [03324 info 'Default'] Initialized channel manager
2014-10-15T18:41:26.188+13:00 [03324 info 'Default'] Current working directory:C:\Program Files\VMware\VMware vCenter Site Recovery Manager\bin
2014-10-15T18:41:26.188+13:00 [03324 verbose 'Default'] Setting COM threading model to MTA
2014-10-15T18:41:26.188+13:00 [03324 info 'Default'] ThreadPool windowsStackImme diateCommit = true
2014-10-15T18:41:26.188+13:00 [03324 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 401, 2, 200, 2147483647
2014-10-15T18:41:26.188+13:00 [03324 info 'ThreadPool'] Thread enlisted
2014-10-15T18:41:26.188+13:00 [02400 info 'ThreadPool'] Thread enlisted
2014-10-15T18:41:26.188+13:00 [03400 info 'ThreadPool'] Thread enlisted
2014-10-15T18:41:26.188+13:00 [02124 info 'ThreadPool'] Thread enlisted
2014-10-15T18:41:26.188+13:00 [04000 info 'ThreadPool'] Thread enlisted
Enter password for username srm_administrator:
2014-10-15T18:41:28.672+13:00 [03324 info 'Default'] Set dump dir to 'C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\DumpFiles'
2014-10-15T18:41:28.703+13:00 [03324 info 'Default'] Vmacore::InitSSL: handshake TimeoutUs = 20000000
2014-10-15T18:41:28.735+13:00 [03324 warning 'Default'] Ignoring bad DNS vcenter.test.com because of correct thumbprints
2014-10-15T18:41:28.735+13:00 [03324 verbose 'HttpConnectionPool-000000'] HttpConnectionPoolImpl created. maxPoolConnections = 200; idleT
meout = 900000000; max OpenConnections = 50; maxConnectionAge = 0
2014-10-15T18:41:28.750+13:00 [04000 verbose 'Default'] Local and remote versions are the same.  Talking with version vim.version.version9
2014-10-15T18:41:28.782+13:00 [02400 verbose 'Default'] Local and remote versions are the same.  Talking with version vim.version.version9
2014-10-15T18:41:28.782+13:00 [03324 info 'Default'] VC Connection: Authenticating unprivileged user 'srm_administrator'
2014-10-15T18:41:28.860+13:00 [03324 info 'Default'] VC Connection: Logged in session 5255d <vcversion>5.5.0<vcversion>2014-10-15T18:41:2
.860+13:00 [03324 info 'Default'] vCenter Server version is: 5.5.0
2014-10-15T18:41:28.860+13:00 [03324 verbose 'Default'] VC Connection: Logging out session 5255d
2014-10-15T18:41:28.860+13:00 [03324 verbose 'Default'] VC Connection: Logged out session 5255d
2014-10-15T18:41:28.860+13:00 [03324 info 'vmomi.soapStub[1]'] Resetting stub adapter for server <cs p:00000000040c0640, TCP: : Closed
2014-10-15T18:41:28.860+13:00 [03324 verbose 'CredentialsStore'] Stored credentials, key='', username=''
Command executed successfully.
2014-10-15T18:41:28.875+13:00 [02400 info 'ThreadPool'] Thread delisted
2014-10-15T18:41:28.875+13:00 [02124 info 'ThreadPool'] Thread delisted
2014-10-15T18:41:28.875+13:00 [04000 info 'ThreadPool'] Thread delisted
2014-10-15T18:41:28.875+13:00 [03400 info 'ThreadPool'] Thread delisted

Even the command was executed successfully, the Site Recovery Manager service didn’t start.