I’ve finally got around to provisioning two Ubuntu machines in two separate VNETs to test azure VNet to VNet throughput. They are connected with the native azure VPN. The machines are type “Standard_A1” which is 1 core and 1.75 GB memory although I don’t think should have too much of a bearing on the throughput. They are running ubuntu Linux abd iperf on both ends. The following represents transfers between the two geographic regions (Dublin and Amsterdam).

First up, latency:

Vnet to Vnet Latency

Latency between two VNET’s – Dublin and Amsterdam

 

Next up, Throughput on a single thread.

Test 1 :

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50855 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 12.6 MBytes 10.5 Mbits/sec

Test 2:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50852 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 24.4 MBytes 20.4 Mbits/sec

Test 3:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50860 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 14.6 MBytes 12.2 Mbits/sec

Test 4:

————————————————————
Client connecting to 10.3.125.6, TCP port 5001
TCP window size: 22.7 KByte (default)
————————————————————
[ 3] local 10.3.160.4 port 50861 connected with 10.3.125.6 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.2 sec 34.8 MBytes 28.7 Mbits/sec

Next up, Throughput on 10 threads:

Test 1:

[ ID] Interval Transfer Bandwidth
[ 12] 0.0-10.3 sec 5.25 MBytes 4.29 Mbits/sec
[ 9] 0.0-10.3 sec 7.25 MBytes 5.92 Mbits/sec
[ 3] 0.0-10.3 sec 7.62 MBytes 6.21 Mbits/sec
[ 5] 0.0-10.4 sec 4.50 MBytes 3.64 Mbits/sec
[ 4] 0.0-10.4 sec 6.00 MBytes 4.84 Mbits/sec
[ 8] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 10] 0.0-10.4 sec 5.50 MBytes 4.44 Mbits/sec
[ 6] 0.0-10.5 sec 4.75 MBytes 3.80 Mbits/sec
[ 7] 0.0-10.5 sec 4.38 MBytes 3.50 Mbits/sec
[ 11] 0.0-10.5 sec 5.12 MBytes 4.10 Mbits/sec
[SUM] 0.0-10.5 sec 55.4 MBytes 44.3 Mbits/sec

Test 2:

[ ID] Interval Transfer Bandwidth
[ 7] 0.0-10.2 sec 6.62 MBytes 5.43 Mbits/sec
[ 12] 0.0-10.3 sec 5.62 MBytes 4.58 Mbits/sec
[ 10] 0.0-10.3 sec 6.88 MBytes 5.61 Mbits/sec
[ 3] 0.0-10.3 sec 7.38 MBytes 5.99 Mbits/sec
[ 11] 0.0-10.4 sec 6.88 MBytes 5.54 Mbits/sec
[ 5] 0.0-10.5 sec 7.00 MBytes 5.62 Mbits/sec
[ 4] 0.0-10.5 sec 7.50 MBytes 6.01 Mbits/sec
[ 6] 0.0-10.5 sec 5.75 MBytes 4.59 Mbits/sec
[ 9] 0.0-10.6 sec 6.62 MBytes 5.25 Mbits/sec
[ 8] 0.0-11.0 sec 6.50 MBytes 4.94 Mbits/sec
[SUM] 0.0-11.0 sec 66.8 MBytes 50.7 Mbits/sec

Test 3:

[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.2 sec 8.25 MBytes 6.77 Mbits/sec
[ 11] 0.0-10.2 sec 7.25 MBytes 5.94 Mbits/sec
[ 10] 0.0-10.3 sec 6.50 MBytes 5.30 Mbits/sec
[ 7] 0.0-10.4 sec 7.62 MBytes 6.17 Mbits/sec
[ 12] 0.0-10.4 sec 7.88 MBytes 6.37 Mbits/sec
[ 4] 0.0-10.4 sec 7.12 MBytes 5.74 Mbits/sec
[ 6] 0.0-10.5 sec 7.88 MBytes 6.28 Mbits/sec
[ 3] 0.0-10.6 sec 6.38 MBytes 5.04 Mbits/sec
[ 9] 0.0-10.7 sec 7.75 MBytes 6.09 Mbits/sec
[ 8] 0.0-10.8 sec 5.75 MBytes 4.46 Mbits/sec
[SUM] 0.0-10.8 sec 72.4 MBytes 56.1 Mbits/sec

Test 4:

[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.1 sec 10.8 MBytes 8.90 Mbits/sec
[ 11] 0.0-10.1 sec 5.75 MBytes 4.76 Mbits/sec
[ 6] 0.0-10.2 sec 11.4 MBytes 9.40 Mbits/sec
[ 8] 0.0-10.2 sec 8.50 MBytes 7.02 Mbits/sec
[ 4] 0.0-10.2 sec 7.25 MBytes 5.97 Mbits/sec
[ 9] 0.0-10.2 sec 7.00 MBytes 5.75 Mbits/sec
[ 7] 0.0-10.2 sec 6.75 MBytes 5.54 Mbits/sec
[ 12] 0.0-10.2 sec 7.38 MBytes 6.04 Mbits/sec
[ 3] 0.0-10.3 sec 6.00 MBytes 4.90 Mbits/sec
[ 10] 0.0-10.3 sec 7.12 MBytes 5.81 Mbits/sec
[SUM] 0.0-10.3 sec 77.9 MBytes 63.6 Mbits/sec

Throughput on 20 threads, starting to look like diminishing returns.

Test 1:

[ 12] 0.0-10.3 sec 3.25 MBytes 2.65 Mbits/sec
[ 14] 0.0-10.4 sec 4.12 MBytes 3.31 Mbits/sec
[ 10] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 16] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 9] 0.0-10.5 sec 3.75 MBytes 2.98 Mbits/sec
[ 13] 0.0-10.6 sec 3.25 MBytes 2.58 Mbits/sec
[ 15] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 19] 0.0-10.5 sec 3.75 MBytes 2.99 Mbits/sec
[ 18] 0.0-10.6 sec 4.62 MBytes 3.67 Mbits/sec
[ 17] 0.0-10.6 sec 3.62 MBytes 2.87 Mbits/sec
[ 3] 0.0-10.6 sec 3.25 MBytes 2.57 Mbits/sec
[ 21] 0.0-10.6 sec 3.25 MBytes 2.56 Mbits/sec
[ 11] 0.0-10.7 sec 3.25 MBytes 2.54 Mbits/sec
[ 7] 0.0-10.8 sec 2.88 MBytes 2.24 Mbits/sec
[ 6] 0.0-10.8 sec 3.88 MBytes 3.02 Mbits/sec
[ 8] 0.0-10.8 sec 3.62 MBytes 2.81 Mbits/sec
[ 5] 0.0-10.9 sec 4.50 MBytes 3.48 Mbits/sec
[ 22] 0.0-10.9 sec 2.88 MBytes 2.22 Mbits/sec
[ 20] 0.0-10.9 sec 2.75 MBytes 2.11 Mbits/sec
[ 4] 0.0-11.1 sec 3.00 MBytes 2.27 Mbits/sec
[SUM] 0.0-11.1 sec 68.6 MBytes 51.8 Mbits/sec

Test 2:

[ 10] 0.0-10.5 sec 3.38 MBytes 2.69 Mbits/sec
[ 20] 0.0-10.5 sec 3.75 MBytes 2.98 Mbits/sec
[ 12] 0.0-10.6 sec 3.00 MBytes 2.38 Mbits/sec
[ 16] 0.0-10.6 sec 3.50 MBytes 2.76 Mbits/sec
[ 5] 0.0-10.7 sec 3.62 MBytes 2.85 Mbits/sec
[ 11] 0.0-10.7 sec 3.25 MBytes 2.55 Mbits/sec
[ 4] 0.0-10.7 sec 3.75 MBytes 2.93 Mbits/sec
[ 9] 0.0-10.7 sec 3.12 MBytes 2.44 Mbits/sec
[ 3] 0.0-10.8 sec 3.00 MBytes 2.33 Mbits/sec
[ 8] 0.0-10.8 sec 3.50 MBytes 2.72 Mbits/sec
[ 22] 0.0-10.9 sec 3.88 MBytes 2.99 Mbits/sec
[ 14] 0.0-10.9 sec 3.12 MBytes 2.40 Mbits/sec
[ 19] 0.0-11.0 sec 3.62 MBytes 2.76 Mbits/sec
[ 15] 0.0-11.1 sec 2.62 MBytes 1.98 Mbits/sec
[ 6] 0.0-11.2 sec 3.25 MBytes 2.44 Mbits/sec
[ 21] 0.0-11.3 sec 3.50 MBytes 2.60 Mbits/sec
[ 18] 0.0-11.3 sec 3.25 MBytes 2.41 Mbits/sec
[ 17] 0.0-11.4 sec 4.25 MBytes 3.14 Mbits/sec
[ 13] 0.0-11.4 sec 3.25 MBytes 2.39 Mbits/sec
[ 7] 0.0-11.5 sec 3.62 MBytes 2.64 Mbits/sec
[SUM] 0.0-11.5 sec 68.2 MBytes 49.7 Mbits/sec

Test 3:

[ 9] 0.0-10.4 sec 4.75 MBytes 3.85 Mbits/sec
[ 12] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 13] 0.0-10.4 sec 3.62 MBytes 2.93 Mbits/sec
[ 10] 0.0-10.4 sec 4.75 MBytes 3.83 Mbits/sec
[ 14] 0.0-10.4 sec 5.38 MBytes 4.32 Mbits/sec
[ 18] 0.0-10.4 sec 3.25 MBytes 2.61 Mbits/sec
[ 19] 0.0-10.5 sec 5.38 MBytes 4.31 Mbits/sec
[ 20] 0.0-10.5 sec 4.38 MBytes 3.50 Mbits/sec
[ 8] 0.0-10.5 sec 4.12 MBytes 3.28 Mbits/sec
[ 16] 0.0-10.6 sec 3.50 MBytes 2.78 Mbits/sec
[ 11] 0.0-10.6 sec 4.50 MBytes 3.57 Mbits/sec
[ 6] 0.0-10.7 sec 5.62 MBytes 4.43 Mbits/sec
[ 3] 0.0-10.7 sec 3.50 MBytes 2.75 Mbits/sec
[ 17] 0.0-10.7 sec 5.88 MBytes 4.62 Mbits/sec
[ 21] 0.0-10.8 sec 4.00 MBytes 3.11 Mbits/sec
[ 5] 0.0-10.8 sec 3.88 MBytes 3.01 Mbits/sec
[ 15] 0.0-10.8 sec 5.25 MBytes 4.07 Mbits/sec
[ 7] 0.0-10.8 sec 3.00 MBytes 2.32 Mbits/sec
[ 4] 0.0-10.9 sec 3.50 MBytes 2.70 Mbits/sec
[ 22] 0.0-11.2 sec 6.00 MBytes 4.49 Mbits/sec
[SUM] 0.0-11.2 sec 89.2 MBytes 66.8 Mbits/sec

Test 4:

[ ID] Interval Transfer Bandwidth
[ 8] 0.0-10.2 sec 3.62 MBytes 2.99 Mbits/sec
[ 18] 0.0-10.3 sec 3.75 MBytes 3.07 Mbits/sec
[ 20] 0.0-10.3 sec 4.50 MBytes 3.68 Mbits/sec
[ 6] 0.0-10.3 sec 4.75 MBytes 3.87 Mbits/sec
[ 5] 0.0-10.3 sec 4.75 MBytes 3.87 Mbits/sec
[ 11] 0.0-10.3 sec 4.50 MBytes 3.65 Mbits/sec
[ 4] 0.0-10.4 sec 3.12 MBytes 2.53 Mbits/sec
[ 22] 0.0-10.4 sec 5.00 MBytes 4.03 Mbits/sec
[ 13] 0.0-10.4 sec 5.62 MBytes 4.53 Mbits/sec
[ 19] 0.0-10.4 sec 3.50 MBytes 2.82 Mbits/sec
[ 14] 0.0-10.4 sec 3.75 MBytes 3.02 Mbits/sec
[ 21] 0.0-10.4 sec 3.25 MBytes 2.61 Mbits/sec
[ 16] 0.0-10.5 sec 3.75 MBytes 3.01 Mbits/sec
[ 15] 0.0-10.5 sec 4.25 MBytes 3.41 Mbits/sec
[ 9] 0.0-10.5 sec 3.62 MBytes 2.90 Mbits/sec
[ 17] 0.0-10.5 sec 4.12 MBytes 3.30 Mbits/sec
[ 12] 0.0-10.5 sec 3.00 MBytes 2.39 Mbits/sec
[ 10] 0.0-10.6 sec 4.25 MBytes 3.36 Mbits/sec
[ 3] 0.0-10.8 sec 4.12 MBytes 3.20 Mbits/sec
[ 7] 0.0-10.9 sec 4.25 MBytes 3.27 Mbits/sec
[SUM] 0.0-10.9 sec 81.5 MBytes 62.8 Mbits/sec

To conclude, the highest throughput I saw on a single thread was 28mb/s, it’s pretty slow when you might be used to 1gb or evern 10gb links between your on premises data center. Replication of data between geographic regions could be very troublesome on systems where a lot of data is being written. The best throughput I saw was 66mb/s on 20 threads, although there were comparable speeds on 10 threads. There’s no huge advantage to using more than about 10-15 threads it seems.

In another update I will test intra DC vnet to vnet (ie in the same data center) and also to other regions.