We are currently on vCloud Director 8.20 and vCenter 6.0 Update 3. (Yes, we are in the process of upgrading..)
Every few days vCloud director loses the connection to the vCenter server but, it will also reconnect automatically with in 1-3 minutes. Below is the snippet log from VCD - vcloud-contailer-info.log
-------------------------------
2019-11-03 17:03:36,718 | ERROR | pool-jetty-47446941 | JDBCExceptionReporter | The query has timed out. | requestId=78c7564f-dba9-4ae7-9f29-0318bf976374,request=GET https://vcd.vmcloud.ca/api/admin/vdc/96e4d057-4a74-4ec9-9116-6959f83a0b5a,requestTime=1572818616429,remoteAddress=x.x.x.x:44621,userAgent=Apache-HttpClient/4.5 (Java/1.8.0_162),accept=application/*+xml;version 20.0
2019-11-03 17:03:36,718 | ERROR | pool-jetty-47423267 | JDBCExceptionReporter | The query has timed out. | requestId=33717197-4bc5-4e08-b1db-2f983f8c61ed,request=GET https://vcd.vmcloud.ca/api/vAppTemplate/vm-ADAD4C2C-05BF-43D1-A879-E30D4B2818A7,requestTime=1572818616277,remoteAddress=x.x.x.x:49666,userAgent=<none>,accept=application/*+xml; version 5.5
2019-11-03 17:03:36,724 | ERROR | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VcUpdateListenerImpl | VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a: Break on Unrecoverable error in Outer WFU Loop |
org.hibernate.exception.GenericJDBCException: could not execute query
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:126)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:114)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
at org.hibernate.loader.Loader.doList(Loader.java:2231)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
at org.hibernate.loader.Loader.list(Loader.java:2120)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
at com.vmware.vcloud.common.persist.CriteriaProxy.uniqueResult(CriteriaProxy.java:211)
at com.vmware.vcloud.common.persist.dao.HibernateVirtualCenterDaoImpl.getVcRuntime(HibernateVirtualCenterDaoImpl.java:58)
at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.dispatchVcConnectionUpEvent(VcUpdateListenerImpl.java:1274)
at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.innerWaitForUpdatesLoop(VcUpdateListenerImpl.java:976)
at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.outerWaitForUpdatesLoop(VcUpdateListenerImpl.java:657)
at com.vmware.vcloud.vimproxy.internal.impl.VcUpdateListenerImpl.run(VcUpdateListenerImpl.java:367)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: The query has timed out.
at net.sourceforge.jtds.jdbc.TdsCore.wait(TdsCore.java:3936)
at net.sourceforge.jtds.jdbc.TdsCore.executeSQL(TdsCore.java:1046)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:465)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:776)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 17 more
2019-11-03 17:03:36,732 | INFO | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VirtualCenterListener | VC connection state event 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a. Changing state to DISCONNECTED |
2019-11-03 17:03:36,732 | INFO | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | VirtualCenterListener | Disconnected from virtual center 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a |
2019-11-03 17:03:36,981 | ERROR | pool-jetty-47423262 | JDBCExceptionReporter | The query has timed out. | requestId=6286ce62-9641-48d0-bf69-99267699488a,request=GET https://vcd.vmcloud.ca/api/query,requestTime=1572818616366,remoteAddress=x.x.x.x:63345,userAgent=<none>,accept=application/*+xml; version 5.5
2019-11-03 17:03:37,026 | INFO | VC-546e9c6e-47e0-4cdd-8570-bbdf86e3f17aListener (34921915) | InventoryServiceImpl | Received listener stopped for VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a |
2019-11-03 17:03:37,061 | INFO | pool-async-email-sender-17-thread-26 | Emailer | Sent email to 3 recipients. No failures |
2019-11-03 17:03:37,064 | INFO | VimConnectionCleanup-546e9c6e-47e0-4cdd-8570-bbdf86e3f17a | VcUpdateListenerImpl | VC 546e9c6e-47e0-4cdd-8570-bbdf86e3f17a: Ignoring exception null while unregistering filters |
-------------------------------
Looks like the DB query is timing out but, not sure why its causing a disconnect from vCloud. Has anyone experienced the above?
Thanks for your help.