In order to use O365 over Express route you need to get the subscriptions the circuits belong to authorised. This is effectively a vetting process Microsoft require in order to satisfy themselves that there is a real use case for their customers. Using O365 over the express route brings some design concerns, mostly around asymmetrical routing and resiliency. In order to consume O365 / Skype for business over express route you now need to attach a route filter to the express route circuit and select the relevent service communities.

Adding rules to route filter

 

Attempting to do this on a non authorised express route will result in  the following error:

The subscription with id ‘xxxxxxx-xxxx-xxxx-xxxx-xxxx’ is not authorized for creating route filters for O365 services. Read more about requesting approval.

Once you have been approved, attempting to add these to a non premium express route will give the following slightly different error:

Failed to modify route filter rule ‘AcceptedRoutes’. Error: The remote server returned an unexpected response: (400) xxxx-xxxx is not compatible with the route filter. The route filter includes Office 365 communities. The circuit does not have access to Office365..

The first non authorised message will take a couple of weeks to sort, as it needs to go to the Microsoft product group. The second error is just a matter of changing your circuit from standard to premium.

Migrating Classic Azure VNets to an ARM express route

We are in the process of migrating all of our classic (ASM) resources to ARM in Azure. The first thing we moved is our express route connections, it was relatively painless to do this.

In order to move our classic VNets over it turns out that they cannot contain webroles (the pre validation checks fail if the VNet contains these). Basically all of our Azure applications run on webroles so this creates a bit of an issue. So, to migrate the VNets we need to isolate the webroles. To do this we needed to create some webrole specific VNets, a temporary holding place until the applications have been redesigned to work either in containers or app service.

The Problem

After creating the VNets (which you are fine to do through the gui) you need to create a gateway, which is then in turn connected to Express Route. It turns out that the gateway can only be created using powershell. If you create the gateway through the web gui and then attempt to link these to your express route you will get either of these error messages:

New-AzureDedicatedCircuitLink : BadRequest: The current provisioning status of the gateway prevents this operation.

New-AzureDedicatedCircuitLink : BadRequest: This operation is enabled only for the following gateway mode(s): DedicatedCircuit.

The command to create the VNet gateway is:

New-AzureVNetGateway -VNetName “VNetName” -GatewayType “DynamicRouting” -GatewaySKU “Standard”

Running this will result in an error as well!

New-AzureVNetGateway : BadRequest: A gateway is not specified in the network configuration file for the specified virtual network

This is where it gets a bit messy. Although the error suggests a gateway is not specified in the network configure file (which is true) this is not the reason we see this. In the old portal you used to be able to create a “LocalNetworkSite” which specifies which on-prem networks you will be connecting to over express route. As the old portal doesn’t exist, you can’t create this LocalNetworkSite in the GUI or using powershell. Without this, you see the errors above.

The Solution

In order to fix this you must create it manually (a bit of a hack really) using resource explorer, https://resources.azure.com/.

Navigate to the Subscription, then the resource group, and finally the VNet you are wishing to create the VNet gateway on. Click on “Read/Write” and then Edit.

After the subnets, you need to add the section name “GatewayProfile”.

    "subnets": [
      {
        "name": "GatewaySubnet",
        "addressPrefix": "x.x.255.240/28"
      },
      {
        "name": "Subnet1",
        "addressPrefix": "x.x.x.0/24"
      },
      {
        "name": "Subnet2",
        "addressPrefix": "x.x.x.0/24"
      }
    ],
    "gatewayProfile": {
      "size": "Small",
      "localNetworkSites": [
        {
          "localNetworkSiteName": "OnPremNetworks",
          "addressSpace": [
            "10.0.0.0/16",
            "192.168.0.0/16"
          ],
          "connectionTypes": [
            "Dedicated"
          ]
        }
      ]
    }

Click Put and this will save the config.

Go back to powershell, when you attempt to run the New-AzureVNetGateway command, it should will now work.

A while ago Azure announced a new feature allowing you to connect VNets together without express route or VPN.

VNet peering is a mechanism that connects two virtual networks (VNets) in the same region through the Azure backbone network. Once peered, the two virtual networks appear as one for all connectivity purposes. They are still managed as separate resources, but virtual machines in these virtual networks can communicate with each other directly by using private IP addresses.

The product page can be found here.

The Existing Architecture

Originally, your architecture may have consisted of several VNets and all of these would have had a gateway (a billable item) and a form of connectivity between them – either express route or VPN. There are some inefficiencies here as all traffic between VNets needs to be routed via your express route circuit. Bandwidth and the actual location of your express route termination points may be problems here. A 1000mb/s link may be plenty between your on-prem networks and Azure, but is it enough for your traffic between all of your VNets? With the below architecture we have 4 gateways for 4 VNets.

Legacy Azure Architecture - A gateway in each VNET

Legacy Azure Architecture – A gateway in each VNet

The New Architecture

VNet peering does away with the requirement to have a gateway on every VNet. We can create a VNet which serves as the link between your on-prem networks and have the rest connected via VNet peering. This is even supported between VNets that are in different subscriptions. To use the same reference architecture but changed to adopt VNet peering, we will have something like the below.

VNet Peering Azure Architecture - One more VNet but 3 fewer gateways. Profit!

VNet Peering Azure Architecture – One more VNet but 3 fewer gateways. Profit!

So we have one extra VNet, but 3 fewer gateways. The gateways are what cost money in this setup, so this will not only be a cheaper setup, but a more efficient one. All traffic between VNets will be via the (blazing fast!!!) azure backbone and not your express route link. At the moment VNet peering traffic is free of charge, so you will save some money on your ingress/egress express route charges.

As with everything Azure, please do your own research as pricing and features change regularily.

We’ve been steadily moving more and more of our servers into the azure cloud and have now started to look at moving servers from our DMZ environments there too. Traditional network security says put any internet accessible servers in a multi-tiered DMZ environment, where your web servers, application servers, and databases are in separate network segments. A security event would be localised to just that area of your network reducing the risk to the rest of your network.

Trying to replicate this way of thinking in Azure proves rather difficult currently. There is no concept of a network layer firewall (except for their Network Security Groups which I will get to) and the only security players in this field at the moment are Barracuda. With this solution (their NG firewall) you essentially create a big VPN mesh where each server is VPN’d to a firewall and you can write your enforcement policy there. A big problem with this approach is that you are reliant on a piece of software running on the server OS, any malicious user could simply disable the software and circumvent that entirely.

Using host OS firewalls such as Windows Firewall or Linux’s iptables carry the same risks as above. Anyone with access to your server can simply disable, or change the firewall rules to suit their needs.

Network Security Groups which were implemented only recently offer some form of protection, and these ACL’s can be applied to subnets or individual VM’s within a VNET. I have several gripes with these at the moment.

  • Visibility – Traffic accepted or blocked is done so silently. Traditional firewalls offer an interface which allows you to filter logs to show you whether any particular connection was allowed or blocked, and why. Without this, troubleshooting traffic is difficult, particularly with big policies.
  • No Object Groups – You cannot create an object that represents multiple hosts, or networks. For example, you cannot reference an object name ‘ActiveDirectory’ which contains all your AD servers. With most traditional firewalls you simply update that object and wherever it is referenced in your rulesets gets updated too.
  • Administration – The only way to update NSG’s at the moment is through powershell. This is administratively very laborious. Each change takes anywhere between 5 and 30 seconds to apply.

Examples shown on the Azure website show a very simplified view of this. Port 80 to the webserver, and a database call to your database. The reality is there’s so much more to it than this. Each server of ours has a requirement to connect to a lot of infrastructure services. Some of these include, Active Directory, Patch Management, AntiVirus etc. Many of these services require bidirectional rules to allow the client / server connections. In a company like ours where we are moving servers to the cloud aggressively, IP addresses are frequently changing. This change simply cannot be managed with the NSG’s with this amount of change.

I’m very much hoping Cisco, Juniper, Fortigate, Checkpoint and other firewall vendors will arrive in this space as I’m worried there is no existing solution to designing a proper DMZ architecture in Azure yet.

By default any Azure virtual machines that have a cloud service with an endpoint will have full outbound internet access. To prevent internet access from Azure virtual machines you can either trust a host level firewall (ie Windows Firewall or Iptables) or you can simply remove the endpoints (which will also remove the ability to get to the machine externally).

One issue i’ve found when connecting an Azure VM to an express route network is that all machines have full outbound access even when the cloud endpoints have been removed. This is a different behaviour than when they’re not on express route (as mentioned in the previous paragraph).

In order to restrict the outbound access in this scenario you seemingly have two options. One option which is documented by Microsoft is to advertise a default route (ie 0.0.0.0/0) from your on premesis networks so all traffic which doesn’t match specific Azure VNET’s heads towards your on prem routers (advertised with BGP). This will often not be preferable as there will be a lot of junk traffic your router/onprem firewalls will need to filter – it’ll also cost you at the standard express route data charges for traffic between your cloud network and your on prem network.

Another solution is by using the recently implemented feature, Network Security Groups. By creating a network security group and applying it to particular VM’s, VNET’s, or Subnets you are able to restrict the access to a specific set of rules defined within the security group. In my scenario, I wished to create a group where access is permitted only to RFC1918 addresses. Another subnet which contains systems with internet access can then use a modified security group to permit not only the RFC1918 traffic but also the traffic to the internet. This is where our web and mail proxies, and web servers will live.

The below powershell commands will create a new security group named “OnlyInternalNetworks” and should be used in situations where you wish for a VM not to have internet access (unless via a proxy or relay on another local subnet):

New-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" -Location "West Europe" -Label "Only allow traffic to RFC1918 Addresses"
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-10 -Type "Inbound" -Priority 100 -Action Allow -SourceAddressPrefix '10.0.0.0/8' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-172 -Type "Inbound" -Priority 110 -Action Allow -SourceAddressPrefix '172.16.0.0/12' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-192 -Type "Inbound" -Priority 120 -Action Allow -SourceAddressPrefix '192.168.0.0/16' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-10 -Type "Outbound" -Priority 100 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '10.0.0.0/8' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-172 -Type "Outbound" -Priority 110 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '172.16.0.0/12' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-192 -Type "Outbound" -Priority 120 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '192.168.0.0/16' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name NoInternet -Type "Outbound" -Priority 130 -Action Deny -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix 'INTERNET' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Remove-AzureNetworkSecurityRule -Name ALLOW INTERNET
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" -Detailed

The resulting security group will look like the following:

azure network security groups

It can then be applied to a Subnet with the following syntax:

Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityGroupToSubnet -VirtualNetworkName 'VnetX' -SubnetName 'SubnetX'

Or to a VM:

Get-AzureVM -ServiceName "MyWebsite" -Name "Instance1" | Set-AzureNetworkSecurityGroupConfig -NetworkSecurityGroupName "OnlyInternalNetworks"