As part of a new hybrid cloud installation, I’ve chosen to implement some Fortinet Fortigate firewalls running on Esxi 5.1 in our on-prem data centers. It would have been nice to have chosen a vendor that had capability both in Azure (our chosen cloud provider) and on-prem however due to the lack of any real decent offerings in Azure, we needed two solutions.
Several days were spent trying to get a High Availability a-p (active/passive) cluster working in a single HP blade chasis across several hosts. The symptom appeared to be that when the two firewall guests were on a single host, clustering worked fine. When a VM was migrated to a new host they couldn’t see each other resulting in a split brain scenario.
Long story short, If you need to establish a HA heartbeat between different hosts then the standard switch created in Esxi must support both MAC address changes (default) and also promiscuous mode (non default). The Fortinet documentation only specifies this as a requirement when you are deploying the firewalls in a transparent mode not NAT. It turns out you need to enable it for both implementations. If you are also having Fortigate VM64 High Availability problems it would pay to check this!