By default any Azure virtual machines that have a cloud service with an endpoint will have full outbound internet access. To prevent internet access from Azure virtual machines you can either trust a host level firewall (ie Windows Firewall or Iptables) or you can simply remove the endpoints (which will also remove the ability to get to the machine externally).

One issue i’ve found when connecting an Azure VM to an express route network is that all machines have full outbound access even when the cloud endpoints have been removed. This is a different behaviour than when they’re not on express route (as mentioned in the previous paragraph).

In order to restrict the outbound access in this scenario you seemingly have two options. One option which is documented by Microsoft is to advertise a default route (ie 0.0.0.0/0) from your on premesis networks so all traffic which doesn’t match specific Azure VNET’s heads towards your on prem routers (advertised with BGP). This will often not be preferable as there will be a lot of junk traffic your router/onprem firewalls will need to filter – it’ll also cost you at the standard express route data charges for traffic between your cloud network and your on prem network.

Another solution is by using the recently implemented feature, Network Security Groups. By creating a network security group and applying it to particular VM’s, VNET’s, or Subnets you are able to restrict the access to a specific set of rules defined within the security group. In my scenario, I wished to create a group where access is permitted only to RFC1918 addresses. Another subnet which contains systems with internet access can then use a modified security group to permit not only the RFC1918 traffic but also the traffic to the internet. This is where our web and mail proxies, and web servers will live.

The below powershell commands will create a new security group named “OnlyInternalNetworks” and should be used in situations where you wish for a VM not to have internet access (unless via a proxy or relay on another local subnet):

New-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" -Location "West Europe" -Label "Only allow traffic to RFC1918 Addresses"
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-10 -Type "Inbound" -Priority 100 -Action Allow -SourceAddressPrefix '10.0.0.0/8' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-172 -Type "Inbound" -Priority 110 -Action Allow -SourceAddressPrefix '172.16.0.0/12' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Inbound-192 -Type "Inbound" -Priority 120 -Action Allow -SourceAddressPrefix '192.168.0.0/16' -SourcePortRange '*' -DestinationAddressPrefix '*' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-10 -Type "Outbound" -Priority 100 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '10.0.0.0/8' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-172 -Type "Outbound" -Priority 110 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '172.16.0.0/12' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name Outbound-192 -Type "Outbound" -Priority 120 -Action Allow -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix '192.168.0.0/16' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityRule -Name NoInternet -Type "Outbound" -Priority 130 -Action Deny -SourceAddressPrefix '*' -SourcePortRange '*' -DestinationAddressPrefix 'INTERNET' -DestinationPortRange '*' -Protocol '*'
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Remove-AzureNetworkSecurityRule -Name ALLOW INTERNET
Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" -Detailed

The resulting security group will look like the following:

azure network security groups

It can then be applied to a Subnet with the following syntax:

Get-AzureNetworkSecurityGroup -Name "OnlyInternalNetworks" | Set-AzureNetworkSecurityGroupToSubnet -VirtualNetworkName 'VnetX' -SubnetName 'SubnetX'

Or to a VM:

Get-AzureVM -ServiceName "MyWebsite" -Name "Instance1" | Set-AzureNetworkSecurityGroupConfig -NetworkSecurityGroupName "OnlyInternalNetworks"

I’ve used Cisco firewalls in their various forms for the last 10 years or so, from baby PIX 501’s through to the new ASA 5500X’s. While I’ve always been a big fan of the platform, one area which has always been deficient is their logging and reporting capability. There really is no comparison when you line up other vendors such as Pallo and Checkpoint along side Cisco when it comes to this.

I’ve recently started playing around with what people call the ELK stack, and have found it to be excellent at visualising cisco ASA logs. The ELK stack consists of three applications, ElasticSearch, Logstash, and Kibana.

ElasticSearch:

Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents

Logstash:

logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching).

Kibana:

Kibana is a great tool for real time data analytics.

The end result is a system which is able to turn simple syslog messages into a screen which looks like my example below. Broken down are graphs to represent the top protocols, actions (ie accept, deny), destination ports, origin countries, and source and destination IP’s. Within each of these views all of the content is dynamic, say you’re only interested in dropped traffic, you can click this in the graph and the whole filter will change to only represent this data.

ASA

Click on “Deny” as the action, and “TCP” and as the protocol shows the top sources of this traffic. It immediately becomes easy to see the usual volumes of this type of traffic over any period of time you specify. Anomalies are easy to see and can be drilled down into to see more closely.

While setting up an ELK stack is outside what I’d planned on mentioning here, digital ocean have a good guide on setting this all up. The pertinent logstash config I have used for the above is as follows:

input {
udp {
port => 10514
type => syslog
}
}

filter {
if [type] == "syslog" {
grok {
patterns_dir => "./patterns"
match => { "message" => "%{CISCOFW106014}" }
match => { "message" => "%{CISCOFW106001}" }
match => { "message" => "%{CISCOFW106023}" }
match => { "message" => "%{CISCOFW313005}" }
match => { "message" => "%{CISCOFW302020_302021}" }
match => { "message" => "%{CISCOFW302013_302014_302015_302016}" }
match => { "message" => "%{CISCOFW106015}" }
match => { "message" => "%{CISCOFW106006_106007_106010}" }
match => { "message" => "%{CISCOFW106100}" }
add_tag => [ "firewall" ]
}

geoip {
source => "src_ip"
target => "geoip"
database =>"/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}

mutate {
convert => [ "[geoip][coordinates]", "float" ]
lowercase => [ "protocol" ]
}
}
}

output {
redis { host => "127.0.0.1"
data_type => "list"
key => "logstash" }
}


The above system will listen for syslog messages on udp 10514, run them through grok to extract the pertinent parts of the message, add geo-ip data, and forward it onto a local redis cache. Another system (the ELK receiver/indexer) polls this system and retrieves the logs and finally displays them to the user through the Kibana interface.

I recently got the second express route connection up and running at work. Out of our primary data center, London,  we have a 1Gb/s connection to Azure through their London point of presence.  The second of the two links is to the Amsterdam POP and is from our Manchester data center. As per Microsoft’s documentation, a resilient path is required from a different peering location in order for their SLA’s to be valid. This must also be an active/active configuration.

I have tested failover between these two links by altering some of the prefixes we advertise and this seems to work in an expected manner. Failover (on default BGP configuration) is between 15-20 seconds from one data centre to the other. The architecture looks like this:

Azure Express Route Toplogy

The express route topology I have implemented

I’ve chosen to use AS prepend for our advertised routes (which will control inbound traffic) and weight, which is a local value to each router, for the outbound routes. This way we should have a deterministic failover – The egress and ingress paths should always be London POP1 from ‘Datacenter A’ under normal circumstances. There are of course many ways with BGP to modify the best path, in order achieve the same or different routes for specific prefixes.

Under this same topology I run iBGP between our two data centres. Through this design a route map can be used to manipulate the path to azure that each data centers have. As we have two 1GB/s links, one at each data centre, it may be desirable to route some traffic out one link and the rest out another (an example might be sending all the backup traffic our the 2nd link).

I’m impressed with how flexible Azure allows this to be.

PS: When you have provisioned your second dedicated link, be sure to connect both to your VNET!