by ·0 Comments

In order to create an Azure Express Route connection I found myself running through the “Configure an ExpressRoute Connection” guide on Microsoft’s MSDN library.

Attempting to create a new dedicated circuit with a speed of ‘1024’ as the document suggests results in the following error:

New-AzureDedicatedCircuit -CircuitName “ExpressRoute” -ServiceProviderName “ProviderA” -Bandwidth “1024” -Location “Amsterdam”

New-AzureDedicatedCircuit : ResourceNotFound: The specified bandwidth 1024 does not exist for the provider ProviderA
At line:1 char:1
+ New-AzureDedicatedCircuit -CircuitName “ExpressRoute” -ServiceProviderName ” …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [New-AzureDedicatedCircuit], CloudException
+ FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.ExpressRoute.NewAzureDedicatedCircuitCommand

It turns out you need to choose a bandwidth of 1000 rather than 1024. I’m not sure if you’re getting screwed out of 24mb/s here or whether this is purely cosmetic. I guess we will need to put it to the test when it’s all up and running!

The express route may be created without any linked VNets so this can be performed to get the ball rolling with your exchange provider even before you have decided upon an Azure architecture, IP addressing structure, or subnet layout. You will be given a service key which you can give to your provider.

by ·0 Comments

I’ve finally got around to provisioning two Ubuntu machines in two separate VNETs to test azure VNet to VNet throughput. They are connected with the native azure VPN. The machines are type “Standard_A1” which is 1 core and 1.75 GB memory although I don’t think should have too much of a bearing on the throughput. They are running ubuntu Linux abd iperf on both ends. The following represents transfers between the two geographic regions (Dublin and Amsterdam).

First up, latency:

Vnet to Vnet Latency

Latency between two VNET’s – Dublin and Amsterdam

 

Next up, Throughput on a single thread.

Test 1 :

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50855 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 12.6 MBytes 10.5 Mbits/sec

Test 2:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50852 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.4 MBytes 20.4 Mbits/sec

Test 3:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50860 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 14.6 MBytes 12.2 Mbits/sec

Test 4:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50861 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.2 sec 34.8 MBytes 28.7 Mbits/sec

Next up, Throughput on 10 threads:

Test 1:

[ ID] Interval Transfer Bandwidth
[ 12] 0.0-10.3 sec 5.25 MBytes 4.29 Mbits/sec
[ 9] 0.0-10.3 sec 7.25 MBytes 5.92 Mbits/sec
[ 3] 0.0-10.3 sec 7.62 MBytes 6.21 Mbits/sec
[ 5] 0.0-10.4 sec 4.50 MBytes 3.64 Mbits/sec
[ 4] 0.0-10.4 sec 6.00 MBytes 4.84 Mbits/sec
[ 8] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 10] 0.0-10.4 sec 5.50 MBytes 4.44 Mbits/sec
[ 6] 0.0-10.5 sec 4.75 MBytes 3.80 Mbits/sec
[ 7] 0.0-10.5 sec 4.38 MBytes 3.50 Mbits/sec
[ 11] 0.0-10.5 sec 5.12 MBytes 4.10 Mbits/sec
[SUM] 0.0-10.5 sec 55.4 MBytes 44.3 Mbits/sec

Test 2:

[ ID] Interval Transfer Bandwidth
[ 7] 0.0-10.2 sec 6.62 MBytes 5.43 Mbits/sec
[ 12] 0.0-10.3 sec 5.62 MBytes 4.58 Mbits/sec
[ 10] 0.0-10.3 sec 6.88 MBytes 5.61 Mbits/sec
[ 3] 0.0-10.3 sec 7.38 MBytes 5.99 Mbits/sec
[ 11] 0.0-10.4 sec 6.88 MBytes 5.54 Mbits/sec
[ 5] 0.0-10.5 sec 7.00 MBytes 5.62 Mbits/sec
[ 4] 0.0-10.5 sec 7.50 MBytes 6.01 Mbits/sec
[ 6] 0.0-10.5 sec 5.75 MBytes 4.59 Mbits/sec
[ 9] 0.0-10.6 sec 6.62 MBytes 5.25 Mbits/sec
[ 8] 0.0-11.0 sec 6.50 MBytes 4.94 Mbits/sec
[SUM] 0.0-11.0 sec 66.8 MBytes 50.7 Mbits/sec

Test 3:

[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.2 sec 8.25 MBytes 6.77 Mbits/sec
[ 11] 0.0-10.2 sec 7.25 MBytes 5.94 Mbits/sec
[ 10] 0.0-10.3 sec 6.50 MBytes 5.30 Mbits/sec
[ 7] 0.0-10.4 sec 7.62 MBytes 6.17 Mbits/sec
[ 12] 0.0-10.4 sec 7.88 MBytes 6.37 Mbits/sec
[ 4] 0.0-10.4 sec 7.12 MBytes 5.74 Mbits/sec
[ 6] 0.0-10.5 sec 7.88 MBytes 6.28 Mbits/sec
[ 3] 0.0-10.6 sec 6.38 MBytes 5.04 Mbits/sec
[ 9] 0.0-10.7 sec 7.75 MBytes 6.09 Mbits/sec
[ 8] 0.0-10.8 sec 5.75 MBytes 4.46 Mbits/sec
[SUM] 0.0-10.8 sec 72.4 MBytes 56.1 Mbits/sec

Test 4:

[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.1 sec 10.8 MBytes 8.90 Mbits/sec
[ 11] 0.0-10.1 sec 5.75 MBytes 4.76 Mbits/sec
[ 6] 0.0-10.2 sec 11.4 MBytes 9.40 Mbits/sec
[ 8] 0.0-10.2 sec 8.50 MBytes 7.02 Mbits/sec
[ 4] 0.0-10.2 sec 7.25 MBytes 5.97 Mbits/sec
[ 9] 0.0-10.2 sec 7.00 MBytes 5.75 Mbits/sec
[ 7] 0.0-10.2 sec 6.75 MBytes 5.54 Mbits/sec
[ 12] 0.0-10.2 sec 7.38 MBytes 6.04 Mbits/sec
[ 3] 0.0-10.3 sec 6.00 MBytes 4.90 Mbits/sec
[ 10] 0.0-10.3 sec 7.12 MBytes 5.81 Mbits/sec
[SUM] 0.0-10.3 sec 77.9 MBytes 63.6 Mbits/sec

Throughput on 20 threads, starting to look like diminishing returns.

Test 1:

[ 12] 0.0-10.3 sec 3.25 MBytes 2.65 Mbits/sec
[ 14] 0.0-10.4 sec 4.12 MBytes 3.31 Mbits/sec
[ 10] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 16] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 9] 0.0-10.5 sec 3.75 MBytes 2.98 Mbits/sec
[ 13] 0.0-10.6 sec 3.25 MBytes 2.58 Mbits/sec
[ 15] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 19] 0.0-10.5 sec 3.75 MBytes 2.99 Mbits/sec
[ 18] 0.0-10.6 sec 4.62 MBytes 3.67 Mbits/sec
[ 17] 0.0-10.6 sec 3.62 MBytes 2.87 Mbits/sec
[ 3] 0.0-10.6 sec 3.25 MBytes 2.57 Mbits/sec
[ 21] 0.0-10.6 sec 3.25 MBytes 2.56 Mbits/sec
[ 11] 0.0-10.7 sec 3.25 MBytes 2.54 Mbits/sec
[ 7] 0.0-10.8 sec 2.88 MBytes 2.24 Mbits/sec
[ 6] 0.0-10.8 sec 3.88 MBytes 3.02 Mbits/sec
[ 8] 0.0-10.8 sec 3.62 MBytes 2.81 Mbits/sec
[ 5] 0.0-10.9 sec 4.50 MBytes 3.48 Mbits/sec
[ 22] 0.0-10.9 sec 2.88 MBytes 2.22 Mbits/sec
[ 20] 0.0-10.9 sec 2.75 MBytes 2.11 Mbits/sec
[ 4] 0.0-11.1 sec 3.00 MBytes 2.27 Mbits/sec
[SUM] 0.0-11.1 sec 68.6 MBytes 51.8 Mbits/sec

Test 2:

[ 10] 0.0-10.5 sec 3.38 MBytes 2.69 Mbits/sec
[ 20] 0.0-10.5 sec 3.75 MBytes 2.98 Mbits/sec
[ 12] 0.0-10.6 sec 3.00 MBytes 2.38 Mbits/sec
[ 16] 0.0-10.6 sec 3.50 MBytes 2.76 Mbits/sec
[ 5] 0.0-10.7 sec 3.62 MBytes 2.85 Mbits/sec
[ 11] 0.0-10.7 sec 3.25 MBytes 2.55 Mbits/sec
[ 4] 0.0-10.7 sec 3.75 MBytes 2.93 Mbits/sec
[ 9] 0.0-10.7 sec 3.12 MBytes 2.44 Mbits/sec
[ 3] 0.0-10.8 sec 3.00 MBytes 2.33 Mbits/sec
[ 8] 0.0-10.8 sec 3.50 MBytes 2.72 Mbits/sec
[ 22] 0.0-10.9 sec 3.88 MBytes 2.99 Mbits/sec
[ 14] 0.0-10.9 sec 3.12 MBytes 2.40 Mbits/sec
[ 19] 0.0-11.0 sec 3.62 MBytes 2.76 Mbits/sec
[ 15] 0.0-11.1 sec 2.62 MBytes 1.98 Mbits/sec
[ 6] 0.0-11.2 sec 3.25 MBytes 2.44 Mbits/sec
[ 21] 0.0-11.3 sec 3.50 MBytes 2.60 Mbits/sec
[ 18] 0.0-11.3 sec 3.25 MBytes 2.41 Mbits/sec
[ 17] 0.0-11.4 sec 4.25 MBytes 3.14 Mbits/sec
[ 13] 0.0-11.4 sec 3.25 MBytes 2.39 Mbits/sec
[ 7] 0.0-11.5 sec 3.62 MBytes 2.64 Mbits/sec
[SUM] 0.0-11.5 sec 68.2 MBytes 49.7 Mbits/sec

Test 3:

[ 9] 0.0-10.4 sec 4.75 MBytes 3.85 Mbits/sec
[ 12] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 13] 0.0-10.4 sec 3.62 MBytes 2.93 Mbits/sec
[ 10] 0.0-10.4 sec 4.75 MBytes 3.83 Mbits/sec
[ 14] 0.0-10.4 sec 5.38 MBytes 4.32 Mbits/sec
[ 18] 0.0-10.4 sec 3.25 MBytes 2.61 Mbits/sec
[ 19] 0.0-10.5 sec 5.38 MBytes 4.31 Mbits/sec
[ 20] 0.0-10.5 sec 4.38 MBytes 3.50 Mbits/sec
[ 8] 0.0-10.5 sec 4.12 MBytes 3.28 Mbits/sec
[ 16] 0.0-10.6 sec 3.50 MBytes 2.78 Mbits/sec
[ 11] 0.0-10.6 sec 4.50 MBytes 3.57 Mbits/sec
[ 6] 0.0-10.7 sec 5.62 MBytes 4.43 Mbits/sec
[ 3] 0.0-10.7 sec 3.50 MBytes 2.75 Mbits/sec
[ 17] 0.0-10.7 sec 5.88 MBytes 4.62 Mbits/sec
[ 21] 0.0-10.8 sec 4.00 MBytes 3.11 Mbits/sec
[ 5] 0.0-10.8 sec 3.88 MBytes 3.01 Mbits/sec
[ 15] 0.0-10.8 sec 5.25 MBytes 4.07 Mbits/sec
[ 7] 0.0-10.8 sec 3.00 MBytes 2.32 Mbits/sec
[ 4] 0.0-10.9 sec 3.50 MBytes 2.70 Mbits/sec
[ 22] 0.0-11.2 sec 6.00 MBytes 4.49 Mbits/sec
[SUM] 0.0-11.2 sec 89.2 MBytes 66.8 Mbits/sec

Test 4:

[ ID] Interval Transfer Bandwidth
[ 8] 0.0-10.2 sec 3.62 MBytes 2.99 Mbits/sec
[ 18] 0.0-10.3 sec 3.75 MBytes 3.07 Mbits/sec
[ 20] 0.0-10.3 sec 4.50 MBytes 3.68 Mbits/sec
[ 6] 0.0-10.3 sec 4.75 MBytes 3.87 Mbits/sec
[ 5] 0.0-10.3 sec 4.75 MBytes 3.87 Mbits/sec
[ 11] 0.0-10.3 sec 4.50 MBytes 3.65 Mbits/sec
[ 4] 0.0-10.4 sec 3.12 MBytes 2.53 Mbits/sec
[ 22] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 13] 0.0-10.4 sec 5.62 MBytes 4.53 Mbits/sec
[ 19] 0.0-10.4 sec 3.50 MBytes 2.82 Mbits/sec
[ 14] 0.0-10.4 sec 3.75 MBytes 3.02 Mbits/sec
[ 21] 0.0-10.4 sec 3.25 MBytes 2.61 Mbits/sec
[ 16] 0.0-10.5 sec 3.75 MBytes 3.01 Mbits/sec
[ 15] 0.0-10.5 sec 4.25 MBytes 3.41 Mbits/sec
[ 9] 0.0-10.5 sec 3.62 MBytes 2.90 Mbits/sec
[ 17] 0.0-10.5 sec 4.12 MBytes 3.30 Mbits/sec
[ 12] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 10] 0.0-10.6 sec 4.25 MBytes 3.36 Mbits/sec
[ 3] 0.0-10.8 sec 4.12 MBytes 3.20 Mbits/sec
[ 7] 0.0-10.9 sec 4.25 MBytes 3.27 Mbits/sec
[SUM] 0.0-10.9 sec 81.5 MBytes 62.8 Mbits/sec

To conclude, the highest throughput I saw on a single thread was 28mb/s, it’s pretty slow when you might be used to 1gb or evern 10gb links between your on premises data center. Replication of data between geographic regions could be very troublesome on systems where a lot of data is being written. The best throughput I saw was 66mb/s on 20 threads, although there were comparable speeds on 10 threads. There’s no huge advantage to using more than about 10-15 threads it seems.

In another update I will test intra DC vnet to vnet (ie in the same data center) and also to other regions.

by ·2 Comments

The client I’m currently working for is starting a fairly large migration of it’s applications to the azure cloud platform. One aspect of this that I’ve been interested in exploring is the performance of the network stack, both throughput and VNet latency. A logical way of separating environments is by creating VNET’s and joining them with VNET to VNET VPN’s which are built in natively.

I’m a little surprised about the VNet’s latency, even in the same datacenter. I have deployed two machines in two separate VNET’s in the same geographic region (North Europe – Dublin) and the standard latency is 2-3ms but it frequently jumps to 6-9ms. While this is just an observation using a standard windows ping, I plan to look into this more closely, a little more scientifically. iperf and throughput stats to come!

Are there any applications that are so latency dependent that this alone would prevent you from moving said applications into cloud infrastructure? Low latency on an on premises network is almost so this might be important to you when considering a move away from on-prem. ‘Noisy Neighbours’ are more easily dealt with when you look after the network end to end of course, troubleshooting this latency variation could be difficult.

 

by ·0 Comments

I’ve spent a lot of time on F5’s over the last few weeks as we’re implementing them in a new data center design. Route domain functionality is being used in order to provide load balancing services to both our Extranet/DMZ environments as well as on the LAN. While good practice would usually dictate that these are on separate devices, route domains allow you to cut up a single appliance into virtual areas where traffic is completely isolated from one another. One aspect of this scenario that has trouble me a little has been getting dynamic routing on each of these route domains working – there isn’t a great deal of example configurations and f5’s own documentation doesn’t seem to offer much in the way of how you actually get it all set up. Dynamic routing on multiple routing domains is new as of LTM and TMOS 11.2.0.

In reality the config is very simple but knowing where and what to add proved difficult to find.

Assuming you have a partition called LAN and within it a route domain with instance id ‘1’, the following is required.

VERY IMPORTANT: On the Self-IP of the VLAN/Interface that you wish to have participate in OSPF make sure the port lockdown allows OSPF. I ended up making a custom profile which allows this or you can select the ‘ALL’ or ‘Default’ settings which will permit this. Weirdly, the default state which allow nothing will permit FULL OSPF adjacencies to form (sometimes) but weird behavior – route tables showing in the OSPF database but those routes not being added to the route table of the routing domain. On the Cisco end I frequently saw EXSTART or EXCHANGE states which wouldn’t move.

To edit route domain one, from the CLI run “imish -r 1” (where 1 is the route domain).

Enter configuration mode by typing ‘conf t’

The following is an example that will form an adjacency with a Cisco switch.

interface /LAN/LANTrunk
ip ospf priority 0
!
router ospf 1
redistribute kernel
network 192.168.0.0 0.0.0.255 area 10

The priority of 0 will just mean your F5 doesn’t become the DR, this may or may not be preferable on your network.

Redistribute Kernel will advertise any live VIP’s from the active f5 node (if they’re in a cluster).