With a large collection of traditional SQL servers, our business has started to look into the feasibility of moving our databases over to the Microsofts ‘SQL Azure’ service. The current databases are run on IaaS machines and have their own internal IP joined to our VNET’s. They have static IP’s so providing access to them from a machine protected by a NSG is as simply as adding a new rule:
Set-AzureNetworkSecurityRule -Action Allow -DestinationAddressPrefix 220.127.116.11/32 -DestinationPortRange 1433 -Name “SQL_Database” -Priority 100 -Protocol TCP -SourceAddressPrefix 10.10.10.10/32 -SourcePortRange * -Type Outbound
The PaaS SQL server is a different kettle of fish altogether. Their dynamic nature means they will not have a static IP address – If you download a list of the subnets here you will see that maintaining a list of accurate NSG’s rules would be an exercise in futility. As of writing, there are 216 subnets in the North Europe datacenter alone. These are likely to change fairly regularly as a datacenter grows in size.
I can see no other way of allowing access to these PaaS SQL instances other than adding a rule allowing access to all of the Internet.
Set-AzureNetworkSecurityRule -Action Allow -DestinationAddressPrefix INTERNET -DestinationPortRange 1433 -Name “SQL_PaaS_Database” -Priority 100 -Protocol TCP -SourceAddressPrefix 10.10.10.10/32 -SourcePortRange * -Type Outbound
With free access to any INTERNET SQL server, shouldn’t these apps now all be in a DMZ?
If you’ve been using Azure for a while there’s a good chance you’re using the classic deployment model. ARM is the new model which will eventually replace classic, and already there is an amount of feature disparity where new features are only being added to ARM. I started looking into the logistics of moving the networking components over to ARM recently and was worried about cost, where you would potentially need to run two environments while moving from the old to the new. One of the big drivers for the change (from a networking perspective) is that NSG’s in ARM support logging:
- Event logs: You can use this log to view what NSG rules are applied to VMs and instance roles based on MAC address. The status for these rules is collected every 60 seconds.
- Counter logs: You can use this log to view how many times each NSG rule was applied to deny or allow traffic.
Great news for people that have been running blind out there!
Until recently an express route circuit created in the classic model could not coexist with one created in ARM – it meant no easy way of connecting these two environments. Microsoft have announced that this month you are able to upgrade ExpressRoute to ARM from Classic, and have VNET’s connected together that have been provisioned in both deployment models.
The documentation is here. I’ll be looking to upgrade our secondary express route connection in the comings weeks, so we can start exploring how easy the rest of the infrastructure will be to move.
Busy as ever, I’ve not had much opportunity to spend any time writing. I’m still heavily involved in cloud and these are some of my current thoughts.
- We need to start looking at ARM (ie Azure v2). When we deployed apps in Azure there was only one way of doing it – configuring each aspect of an application either by powershell or with the web gui. Configuring a web app, for example, required you to create a subscription, a vnet, a subnet, the storage account etc. Azure Resource Manager brings a simplified way of deploying apps – in Microsoft’s own words, “You can deploy, manage, and monitor all of the resources for your solution as a group, rather than handling these resources individually.” Many of the old and new azure components cannot coexist so in order to change to ARM we’ll basically need to start from scratch.
- There is still a need for proper firewalls in Azure. NSG’s are still really laborious and cumbersome to administer. A simple change often requires changing many NSG policies as there is still no concept of object groups. With vendors such as Barracuda, Checkpoint, and Fortinet all offering products in Azure, I think it’s time to deploy these so DMZ environments can be more easily managed.
- Billing is important – understand what services will cost, tag and size resources appropriately.
- Don’t get too crazy with VNET’s. Each VNET will incur charges when the gateway is provisioned. Do you really need one for Test, Dev, and Prod or could you role all of these into one, and separate them by subnets and NSG’s ?
I’ll try and post a little more regularly.
So, it’s been a little while since I last made an update, lack of time as a result of being so busy at work mostly. I’m still heavily involved in a cloud migration and up against some pretty tight deadlines on leaving our on prem data centers; we have been furiously moving our applications (around 70 of them!) to Azure.
There’s a whole heap of stuff happening out in Azure networking land, probably the biggest (and most interesting) change has been the introduction of User Defined Routes – Essentially your own routing table for a subnet. This creates the possibility of routing traffic to a gateway, such as a firewall – I have played with this using a linux box running iptables and it seems to work as advertised. With the introduction of this functionality, I see checkpoint were quick to get a virtual appliance in the marketplace. Barracuda who have offered their NG firewall in Azure for a long time no longer require you to run VPN software on each of the VM’s – Your malicious VM administrator can no longer simply disable the VPN software in order to circumvent your security policies (they’ll just use powershell instead to change the route – I joke, kind of). It’s routing, but different – your default gateway can be in a completely different subnet which boggles the mind somewhat. ARP? The only limitation with this is that the default gateway needs to be in the same VNET as the VM routing traffic to it – this could be inconvenient (and expensive) if you have lots of subscriptions. It would require you to deploy Firewall HA pairs in each VNET.
We’re using Network Security Groups extensively and have these applied to not only our DMZ subnets, but also our LAN ones. In order to prevent someone accidentally adding an internet endpoint, by default only RFC1918 addresses are permitted in and out of every subnet. Internet access is added by exception. I stumbled upon this awesome script which allows you to manage these NSG’s via a CSV file. It’s been an absolute life saver as NSG’s via powershell would be completely unmanageable. The CSV process makes this a whole lot more efficient and much easier – we can also wrap some change / version control around these CSV files to see what and when things are changing.
We’re still finding it hugely frustrating that there is no logging (or SIEM integration) with the NSG’s. Comments on the Azure forums suggest we could know more in July – it may be addressed by the Azure Resource Manager (ARM) stack. There have been plenty of scenarios where we’re attempting to troubleshoot access issues and are completely blind ‘out there’. It’s refreshing when these azure machines are accessing on-prem networks and we get this presented in our lovely ELK stack.
I’ve also started with the Azure Application Proxy as a potential replacement for our on-prem reverse proxies. First impressions are that it’s prettty good – very easy to setup. As you’d expect, it’s tightly integrated with ADFS and this makes authentication and authorization a breeze. Single sign on is easily achieved too.
That’s about it for now. More to come!
We’ve been steadily moving more and more of our servers into the azure cloud and have now started to look at moving servers from our DMZ environments there too. Traditional network security says put any internet accessible servers in a multi-tiered DMZ environment, where your web servers, application servers, and databases are in separate network segments. A security event would be localised to just that area of your network reducing the risk to the rest of your network.
Trying to replicate this way of thinking in Azure proves rather difficult currently. There is no concept of a network layer firewall (except for their Network Security Groups which I will get to) and the only security players in this field at the moment are Barracuda. With this solution (their NG firewall) you essentially create a big VPN mesh where each server is VPN’d to a firewall and you can write your enforcement policy there. A big problem with this approach is that you are reliant on a piece of software running on the server OS, any malicious user could simply disable the software and circumvent that entirely.
Using host OS firewalls such as Windows Firewall or Linux’s iptables carry the same risks as above. Anyone with access to your server can simply disable, or change the firewall rules to suit their needs.
Network Security Groups which were implemented only recently offer some form of protection, and these ACL’s can be applied to subnets or individual VM’s within a VNET. I have several gripes with these at the moment.
- Visibility – Traffic accepted or blocked is done so silently. Traditional firewalls offer an interface which allows you to filter logs to show you whether any particular connection was allowed or blocked, and why. Without this, troubleshooting traffic is difficult, particularly with big policies.
- No Object Groups – You cannot create an object that represents multiple hosts, or networks. For example, you cannot reference an object name ‘ActiveDirectory’ which contains all your AD servers. With most traditional firewalls you simply update that object and wherever it is referenced in your rulesets gets updated too.
- Administration – The only way to update NSG’s at the moment is through powershell. This is administratively very laborious. Each change takes anywhere between 5 and 30 seconds to apply.
Examples shown on the Azure website show a very simplified view of this. Port 80 to the webserver, and a database call to your database. The reality is there’s so much more to it than this. Each server of ours has a requirement to connect to a lot of infrastructure services. Some of these include, Active Directory, Patch Management, AntiVirus etc. Many of these services require bidirectional rules to allow the client / server connections. In a company like ours where we are moving servers to the cloud aggressively, IP addresses are frequently changing. This change simply cannot be managed with the NSG’s with this amount of change.
I’m very much hoping Cisco, Juniper, Fortigate, Checkpoint and other firewall vendors will arrive in this space as I’m worried there is no existing solution to designing a proper DMZ architecture in Azure yet.