In order to use O365 over Express route you need to get the subscriptions the circuits belong to authorised. This is effectively a vetting process Microsoft require in order to satisfy themselves that there is a real use case for their customers. Using O365 over the express route brings some design concerns, mostly around asymmetrical routing and resiliency. In order to consume O365 / Skype for business over express route you now need to attach a route filter to the express route circuit and select the relevent service communities.

Adding rules to route filter

 

Attempting to do this on a non authorised express route will result in  the following error:

The subscription with id ‘xxxxxxx-xxxx-xxxx-xxxx-xxxx’ is not authorized for creating route filters for O365 services. Read more about requesting approval.

Once you have been approved, attempting to add these to a non premium express route will give the following slightly different error:

Failed to modify route filter rule ‘AcceptedRoutes’. Error: The remote server returned an unexpected response: (400) xxxx-xxxx is not compatible with the route filter. The route filter includes Office 365 communities. The circuit does not have access to Office365..

The first non authorised message will take a couple of weeks to sort, as it needs to go to the Microsoft product group. The second error is just a matter of changing your circuit from standard to premium.

Migrating Classic Azure VNets to an ARM express route

We are in the process of migrating all of our classic (ASM) resources to ARM in Azure. The first thing we moved is our express route connections, it was relatively painless to do this.

In order to move our classic VNets over it turns out that they cannot contain webroles (the pre validation checks fail if the VNet contains these). Basically all of our Azure applications run on webroles so this creates a bit of an issue. So, to migrate the VNets we need to isolate the webroles. To do this we needed to create some webrole specific VNets, a temporary holding place until the applications have been redesigned to work either in containers or app service.

The Problem

After creating the VNets (which you are fine to do through the gui) you need to create a gateway, which is then in turn connected to Express Route. It turns out that the gateway can only be created using powershell. If you create the gateway through the web gui and then attempt to link these to your express route you will get either of these error messages:

New-AzureDedicatedCircuitLink : BadRequest: The current provisioning status of the gateway prevents this operation.

New-AzureDedicatedCircuitLink : BadRequest: This operation is enabled only for the following gateway mode(s): DedicatedCircuit.

The command to create the VNet gateway is:

New-AzureVNetGateway -VNetName “VNetName” -GatewayType “DynamicRouting” -GatewaySKU “Standard”

Running this will result in an error as well!

New-AzureVNetGateway : BadRequest: A gateway is not specified in the network configuration file for the specified virtual network

This is where it gets a bit messy. Although the error suggests a gateway is not specified in the network configure file (which is true) this is not the reason we see this. In the old portal you used to be able to create a “LocalNetworkSite” which specifies which on-prem networks you will be connecting to over express route. As the old portal doesn’t exist, you can’t create this LocalNetworkSite in the GUI or using powershell. Without this, you see the errors above.

The Solution

In order to fix this you must create it manually (a bit of a hack really) using resource explorer, https://resources.azure.com/.

Navigate to the Subscription, then the resource group, and finally the VNet you are wishing to create the VNet gateway on. Click on “Read/Write” and then Edit.

After the subnets, you need to add the section name “GatewayProfile”.

    "subnets": [
      {
        "name": "GatewaySubnet",
        "addressPrefix": "x.x.255.240/28"
      },
      {
        "name": "Subnet1",
        "addressPrefix": "x.x.x.0/24"
      },
      {
        "name": "Subnet2",
        "addressPrefix": "x.x.x.0/24"
      }
    ],
    "gatewayProfile": {
      "size": "Small",
      "localNetworkSites": [
        {
          "localNetworkSiteName": "OnPremNetworks",
          "addressSpace": [
            "10.0.0.0/16",
            "192.168.0.0/16"
          ],
          "connectionTypes": [
            "Dedicated"
          ]
        }
      ]
    }

Click Put and this will save the config.

Go back to powershell, when you attempt to run the New-AzureVNetGateway command, it should will now work.

I’ve been involved in a few discussion with people at Microsoft and it seems there has been a swing in sentiment with regards to whether customers should use Microsoft Peering over Express route at all. The long and short of Microsoft Peering is that it enables several Azure services to be available over Express route rather than the general internet. This includes Exchange Online, SharePoint Online, Skype for Business, and Dynamics 36. It excludes services such as Azure websites and Azure storage as these are available over ‘Public Peering’ instead.

When we first deployed Office365 (probably a good 4 years ago now I think?) we saw a perceivable increase in performance when running over Express Route rather than the internet – Our PA’s were running Outlook in the non-cached mode due to the number of mailboxes and calendars they were looking after. Express route made a big difference here.

It’s hard to say at this point why the recommendation would have changed from a reliable path (Express route) to big bad internet routing. To be fair, Microsoft would probably peer with most major ISPs in some way or another so maybe that’s where they are coming from (that it’ll effectively be the same number of hops there anyway?). It’s still a contended, and non QOS supporting path.

I see some murmurings on the internet that they are looking to combine the Public and Microsoft peering into a single routing domain. I’d welcome this as it means a few less BGP peering points. All of the prefixes they advertise now have BGP communities so it’s easy enough to distinguish different services being advertise over the same peering.

The benefits of Express route are well documented, and from the horses mouth, “ExpressRoute connections offer higher security, reliability, and speeds, with lower and consistent latencies than typical connections over the Internet”. A point of contention has been this so called reliability.

The only way to guarantee reliability in packet switched networks is QOS. Sure, through capacity management and or ‘massive pipes’ this can be avoided to a certain extent, but there is always a risk that one device/user/application could consume all of that bandwidth leaving other systems with a smaller piece of the pie.

We have recently adopted Skype for Business PSTN Calling and through this have developed a need for a priority queue on EF (voice traffic effectively) marked packets. Microsoft’s documentation is quite clear on how it thinks QOS should be deployed.

It needs to be deployed end-to-end.

A QoS capable connection must be configured end-to-end (PC, network switches and routers to the cloud) as any part in the path that fails to support QoS could degrade the quality of the entire call

Your express route provider must provide a class of service for EF packets.

Each ExpressRoute network service provider will have a class of service (QoS) that is appropriate for real-time voice and video. This COS is called ‘Expedited Forwarding’ (EF) for voice and ‘Assured Forwarding’ (AF) for video

The problem is that our network provider and Microsoft doesn’t mandate this requirement despite it’s clear stance on it. I’ve labored through support tickets both with our provider and Microsoft – The provider has no interest as there is no pressure (from Microsoft, or other customers at this point) and Microsoft won’t force them to implement it.

The post serves little more purpose than to say, ‘be careful’ when expecting your express route provider to support QOS. They may not, and they are not required to.

We are looking at alternative network providers.

As users of Azure RemoteApp I was tasked with finding a replacement due to the product being retired.

We used the service primarily as a remote access solution, allowing users to RDP to their desktops (assuming they were turned on).

The preferred solution recommended by Azure is Citrix Xenapp Express, a cloud only version of XenApp. This was trialed internally and vetoed almost straight away after deployment as it lacked two factor authentication – this is a huge oversight in my opinion. While views on this will vary company to company, I personally feel that ALL external applications should be protected by two factor authentication. It’s simply not good enough to rely on a username and password.

I decided to take a look at the Netscaler VPX appliance. It is available in the Azure Microsoft and can be deployed to any ARM environments.

Pros

  • Clientless access (although you need to have the Microsoft terminal services client installed – native in Windows, you need to install it on MacOSX, Android etc)
  • Works on every device I’ve tried – Includes iPhone, Android Tablets, MacOSX etc
  • Much cheaper than RemoteApp. A Netscaler VPX enterprise license (which you need for RDP Proxy) and 200mb throughput will be ~£8000 per year + Azure VM costs. We were spending circa £15,000 for 200 or so users.
  • Good user experience – web portal is fairly straight forward.
  • Performant – test users report their desktops are much more responsive than publishing RDP over Remote App.
  • Feature Rich – supports web bookmarks, load balancing, DDoS protection

Cons

  • I wouldn’t say Netscaler VPX is very intuitive to administer/configure. I think it’s more than just a lack of familiarity with the system.
  • IaaS – you need to maintain the system, patch it, and make sure it’s configured correctly
  • Supports any authentication methods you can think of. We are using radius in conjunction with the Microsoft multi factor authentication server.

The system is being trialed by our I.T department but it’s so far proved to be really good.