Gang,
I'm trying to resolve this issue for a long time. My VCSA come up with "vmonapi" service down, always after first LAB boot (all service VMs are starting up - PSC, DNS, NSXM, VRLI, NSXC, etc.).
Issue can be fixed by second reboot VCSA appliance. It is annoying. Here are the logs, pictures and debug infos. Any idea is welcomed!
How to ensure that vmonapi will start over boot storm? Any idea how to troubleshoot? Any recommendations for resource pools for VCSA?
Error from GUI Home->System Config->Summary
HTTP response with status code 503 (enable debug logging for details): <HTML><BODY><H1>503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fdba4003a70] _serverNamespace = /vmonapi action = Allow _port = 8900)</H1></BODY></HTML>
root@vcsa-01a [ /var/log/vmware/vmon ]# service-control --status
Running:
applmgmt lwsmd vmafdd vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
vmcam vmonapi vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha
root@vcsa-01a [ /var/log/vmware/vmon ]# vi vmon-syslog.log
17-12-18T11:28:15.825616+01:00 notice vmon Executing op START on service statsmonitor...
17-12-18T11:28:15.825715+01:00 notice vmon Constructed command: /usr/lib/vmware-statsmonitor/statsMonitor.sh /etc/vmware/statsmonitor/statsMonitor.xml
17-12-18T11:28:16.814873+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-rhttpproxy/rhttpproxy-vmon-apihealth.py
17-12-18T11:28:16.815353+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-statsmonitor -f /var/vmware/applmgmt/statsmonitor_health.xml
17-12-18T11:28:18.274459+01:00 warning vmon Service vmonapi pre-start command's stderr: urlopen() failed! Trying force_refresh=True...
17-12-18T11:28:18.274777+01:00 warning vmon
17-12-18T11:28:18.308797+01:00 warning vmon Service vmonapi pre-start command's stderr: Failed to start vmonapi service. Exception : <urlopen error [Errno 111] Connection refused>
17-12-18T11:28:18.320921+01:00 err vmon Service vmonapi pre-start command failed with exit code 255.
17-12-18T11:28:19.256395+01:00 warning vmon Service rhttpproxy api-health command's stderr: Health URL: https://localhost:443/rhttpproxyhealth
17-12-18T11:28:19.256527+01:00 warning vmon <urlopen error [Errno 111] Connection refused>
17-12-18T11:28:19.256641+01:00 notice vmon Re-check service rhttpproxy health since it is still initializing.
17-12-18T11:28:19.274526+01:00 notice vmon Service applmgmt pre-start command completed successfully.
17-12-18T11:28:19.274696+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/applmgmt/base/bin/vherdrunner /usr/lib/applmgmt/transport/bin/multiserve --config /etc/applmgmt/applmgmt.conf
17-12-18T11:28:20.245715+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-rhttpproxy/rhttpproxy-vmon-apihealth.py
17-12-18T11:28:20.393563+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/applmgmt/applmgmt_vmonhealth.py
17-12-18T11:28:20.393805+01:00 warning vmon Service rhttpproxy api-health command's stderr: Health URL: https://localhost:443/rhttpproxyhealth
17-12-18T11:28:20.393942+01:00 warning vmon <urlopen error [Errno 111] Connection refused>
17-12-18T11:28:20.394053+01:00 notice vmon Re-check service rhttpproxy health since it is still initializing.
17-12-18T11:28:20.825672+01:00 warning vmon Service applmgmt api-health command's stderr: error(111, 'Connection refused')
17-12-18T11:28:20.832896+01:00 notice vmon Re-check service applmgmt health since it is still initializing.
17-12-18T11:28:21.322502+01:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-rhttpproxy/rhttpproxy-vmon-apihealth.py