In order to use O365 over Express route you need to get the subscriptions the circuits belong to authorised. This is effectively a vetting process Microsoft require in order to satisfy themselves that there is a real use case for their customers. Using O365 over the express route brings some design concerns, mostly around asymmetrical routing and resiliency. In order to consume O365 / Skype for business over express route you now need to attach a route filter to the express route circuit and select the relevent service communities.

Adding rules to route filter

 

Attempting to do this on a non authorised express route will result in  the following error:

The subscription with id ‘xxxxxxx-xxxx-xxxx-xxxx-xxxx’ is not authorized for creating route filters for O365 services. Read more about requesting approval.

Once you have been approved, attempting to add these to a non premium express route will give the following slightly different error:

Failed to modify route filter rule ‘AcceptedRoutes’. Error: The remote server returned an unexpected response: (400) xxxx-xxxx is not compatible with the route filter. The route filter includes Office 365 communities. The circuit does not have access to Office365..

The first non authorised message will take a couple of weeks to sort, as it needs to go to the Microsoft product group. The second error is just a matter of changing your circuit from standard to premium.

Migrating Classic Azure VNets to an ARM express route

We are in the process of migrating all of our classic (ASM) resources to ARM in Azure. The first thing we moved is our express route connections, it was relatively painless to do this.

In order to move our classic VNets over it turns out that they cannot contain webroles (the pre validation checks fail if the VNet contains these). Basically all of our Azure applications run on webroles so this creates a bit of an issue. So, to migrate the VNets we need to isolate the webroles. To do this we needed to create some webrole specific VNets, a temporary holding place until the applications have been redesigned to work either in containers or app service.

The Problem

After creating the VNets (which you are fine to do through the gui) you need to create a gateway, which is then in turn connected to Express Route. It turns out that the gateway can only be created using powershell. If you create the gateway through the web gui and then attempt to link these to your express route you will get either of these error messages:

New-AzureDedicatedCircuitLink : BadRequest: The current provisioning status of the gateway prevents this operation.

New-AzureDedicatedCircuitLink : BadRequest: This operation is enabled only for the following gateway mode(s): DedicatedCircuit.

The command to create the VNet gateway is:

New-AzureVNetGateway -VNetName “VNetName” -GatewayType “DynamicRouting” -GatewaySKU “Standard”

Running this will result in an error as well!

New-AzureVNetGateway : BadRequest: A gateway is not specified in the network configuration file for the specified virtual network

This is where it gets a bit messy. Although the error suggests a gateway is not specified in the network configure file (which is true) this is not the reason we see this. In the old portal you used to be able to create a “LocalNetworkSite” which specifies which on-prem networks you will be connecting to over express route. As the old portal doesn’t exist, you can’t create this LocalNetworkSite in the GUI or using powershell. Without this, you see the errors above.

The Solution

In order to fix this you must create it manually (a bit of a hack really) using resource explorer, https://resources.azure.com/.

Navigate to the Subscription, then the resource group, and finally the VNet you are wishing to create the VNet gateway on. Click on “Read/Write” and then Edit.

After the subnets, you need to add the section name “GatewayProfile”.

    "subnets": [
      {
        "name": "GatewaySubnet",
        "addressPrefix": "x.x.255.240/28"
      },
      {
        "name": "Subnet1",
        "addressPrefix": "x.x.x.0/24"
      },
      {
        "name": "Subnet2",
        "addressPrefix": "x.x.x.0/24"
      }
    ],
    "gatewayProfile": {
      "size": "Small",
      "localNetworkSites": [
        {
          "localNetworkSiteName": "OnPremNetworks",
          "addressSpace": [
            "10.0.0.0/16",
            "192.168.0.0/16"
          ],
          "connectionTypes": [
            "Dedicated"
          ]
        }
      ]
    }

Click Put and this will save the config.

Go back to powershell, when you attempt to run the New-AzureVNetGateway command, it should will now work.

I’ve been involved in a few discussion with people at Microsoft and it seems there has been a swing in sentiment with regards to whether customers should use Microsoft Peering over Express route at all. The long and short of Microsoft Peering is that it enables several Azure services to be available over Express route rather than the general internet. This includes Exchange Online, SharePoint Online, Skype for Business, and Dynamics 36. It excludes services such as Azure websites and Azure storage as these are available over ‘Public Peering’ instead.

When we first deployed Office365 (probably a good 4 years ago now I think?) we saw a perceivable increase in performance when running over Express Route rather than the internet – Our PA’s were running Outlook in the non-cached mode due to the number of mailboxes and calendars they were looking after. Express route made a big difference here.

It’s hard to say at this point why the recommendation would have changed from a reliable path (Express route) to big bad internet routing. To be fair, Microsoft would probably peer with most major ISPs in some way or another so maybe that’s where they are coming from (that it’ll effectively be the same number of hops there anyway?). It’s still a contended, and non QOS supporting path.

I see some murmurings on the internet that they are looking to combine the Public and Microsoft peering into a single routing domain. I’d welcome this as it means a few less BGP peering points. All of the prefixes they advertise now have BGP communities so it’s easy enough to distinguish different services being advertise over the same peering.

The benefits of Express route are well documented, and from the horses mouth, “ExpressRoute connections offer higher security, reliability, and speeds, with lower and consistent latencies than typical connections over the Internet”. A point of contention has been this so called reliability.

The only way to guarantee reliability in packet switched networks is QOS. Sure, through capacity management and or ‘massive pipes’ this can be avoided to a certain extent, but there is always a risk that one device/user/application could consume all of that bandwidth leaving other systems with a smaller piece of the pie.

We have recently adopted Skype for Business PSTN Calling and through this have developed a need for a priority queue on EF (voice traffic effectively) marked packets. Microsoft’s documentation is quite clear on how it thinks QOS should be deployed.

It needs to be deployed end-to-end.

A QoS capable connection must be configured end-to-end (PC, network switches and routers to the cloud) as any part in the path that fails to support QoS could degrade the quality of the entire call

Your express route provider must provide a class of service for EF packets.

Each ExpressRoute network service provider will have a class of service (QoS) that is appropriate for real-time voice and video. This COS is called ‘Expedited Forwarding’ (EF) for voice and ‘Assured Forwarding’ (AF) for video

The problem is that our network provider and Microsoft doesn’t mandate this requirement despite it’s clear stance on it. I’ve labored through support tickets both with our provider and Microsoft – The provider has no interest as there is no pressure (from Microsoft, or other customers at this point) and Microsoft won’t force them to implement it.

The post serves little more purpose than to say, ‘be careful’ when expecting your express route provider to support QOS. They may not, and they are not required to.

We are looking at alternative network providers.

A while ago Azure announced a new feature allowing you to connect VNets together without express route or VPN.

VNet peering is a mechanism that connects two virtual networks (VNets) in the same region through the Azure backbone network. Once peered, the two virtual networks appear as one for all connectivity purposes. They are still managed as separate resources, but virtual machines in these virtual networks can communicate with each other directly by using private IP addresses.

The product page can be found here.

The Existing Architecture

Originally, your architecture may have consisted of several VNets and all of these would have had a gateway (a billable item) and a form of connectivity between them – either express route or VPN. There are some inefficiencies here as all traffic between VNets needs to be routed via your express route circuit. Bandwidth and the actual location of your express route termination points may be problems here. A 1000mb/s link may be plenty between your on-prem networks and Azure, but is it enough for your traffic between all of your VNets? With the below architecture we have 4 gateways for 4 VNets.

Legacy Azure Architecture - A gateway in each VNET

Legacy Azure Architecture – A gateway in each VNet

The New Architecture

VNet peering does away with the requirement to have a gateway on every VNet. We can create a VNet which serves as the link between your on-prem networks and have the rest connected via VNet peering. This is even supported between VNets that are in different subscriptions. To use the same reference architecture but changed to adopt VNet peering, we will have something like the below.

VNet Peering Azure Architecture - One more VNet but 3 fewer gateways. Profit!

VNet Peering Azure Architecture – One more VNet but 3 fewer gateways. Profit!

So we have one extra VNet, but 3 fewer gateways. The gateways are what cost money in this setup, so this will not only be a cheaper setup, but a more efficient one. All traffic between VNets will be via the (blazing fast!!!) azure backbone and not your express route link. At the moment VNet peering traffic is free of charge, so you will save some money on your ingress/egress express route charges.

As with everything Azure, please do your own research as pricing and features change regularily.