I just ran across an issue that has me scratching my head. I have a feeling this is something with the switches and not ESXi, but I figured I'd start here first and see if anyone has any thoughts.
Last weekend, I was decommissioning an old host in our internal environment and replacing it with newer hardware.
During the switch, the host that was remaining and all the guests on it vanished from the network - nothing was responding. Nothing had been touched or changed on that host or on the switches - it just vanished out of the blue. What brought everything back was unplugging one of the two uplinks from the host to the switches. The way it had been configured was one uplink to one switch and the other to the other switch, with the uplinks configured (standard switch) so that both are set to active for the virtual switch that services the hosts management network and the guest networking, and the switches (Avaya ERS3500) are stacked via the stacking ports on the back and are aware of each other, so links to different switches shouldn't cause any loops or other issues. This had been working fine for a long time and overall is similar to every other environment is configured, the only real difference being the two stacked switches here and other setups it's generally one switch, although our datacenter environment has two non-stacked MXL switches in our M1000e chassis. In this case, the existing host is running 6.0 and the new host is running 6.7, with vCenter upgraded to the latest revision of 6.7 as of a few weeks ago.
What got it working again with both uplinks connected was to reconfigure a few ports on one switch (in trunk mode as I have a few tagged VLANs being provided to the hosts) and have them all connected to the same physical switch. This seemed fine until this weekend, one of the hosts and all guests on it dropped off. Disabling one of the ports for it on the switch brought everything back up. None of the machines actually went down - they continued to hum along happily, just lost communication with everything outside that host.
There is nothing that I can find that's different between this and other environments that could account for this other than the switches - my home sandbox has two uplinks connected in the same active/active config to a Procurve 5406zl, like I said, our DC environment has the hosts with an uplink to each of the two MXLs, and our satellite office has a host with two active uplinks to the ERS3500 that is in that office and it hasn't had any issues to date.
Does anyone have any thoughts as to why having the two uplinks is causing these hosts to just drop off like this? In my years of working with VMWare and almost always having it set up with two or more active physical uplinks, I can't say I ever recall having this kind of issue come up.