Teams Network Assessment Tool (July 2021 Update)

In July of 2021 Microsoft released a new version of their Network Assessment Tool specifically for Microsoft Teams. This supersedes the old version that was originally released for Skype for Business which we were all still using for Teams deployments (until now). I have a front-end GUI that allows you to graph the results in real time using the Skype for Business version of the tool (still available here). The latest version of the tool appears to have had its core code rewritten and updated with Teams in mind. In this post I’ll do a deep dive into the new tool and compare it to its predecessor.

Here's the link for the new Teams Network Assessment Tool: https://www.microsoft.com/en-us/download/details.aspx?id=103017

At this time the Skype for Business Network Assessment Tool is still also available here: https://www.microsoft.com/en-us/download/details.aspx?id=53885

Note: The version discussed in this post is the Teams Network Assessment Tool 1.4.0.0.

The first change from the old tool is that the commands have slightly changed. By default, the previous tool would run a media quality test and report back latency, packet loss, jitter and reorder ratio. In the new tool, its default is to run a connectivity test to all of the Teams relay servers in Office 365, and the media quality check now requires you use the "qualitycheck" flag when you execute the tool. The output of both of the commands is also quite different than it was in the previous Skype for Business version, which will be demonstrated in the following sections. 

 

Relay Check

The relay check is used to ensure that you can successfully reach the Teams relay services within Office 365 from your current location. It’s important that you run this from a subnet that your Teams clients will be connecting from. There’s not a lot of value in running this from your home when the clients are going to be connecting from an office location that will have different networking and firewall configuration.

No command line flags are required when running the relay check. You simply execute the tool from the command line as shown below:

Command Example:

C:\Program Files (x86)\Microsoft Teams Network Assessment Tool>NetworkAssessmentTool.exe

 

Unlike the previous version of the tool, you do not get to see all the addresses that the tool attempts to connect to when using the verbose flag (as there is no verbose flag anymore). What you get now is a port negotiation check with two servers that will confirm connectivity on all of the 3478-3481 and 443 ports. After this, in the background, it does a check to the full list of relays. However it doesn’t print these tests to the command line:

Something that is new here is that it checks all the TURN redirect ports (3478-3481) for the various workloads (audio, video and screensharing). This matters when you have turned on the “Insert QoS Markets for real-time traffic” setting in the Teams Admin Centre. because this setting will make the relays use the full range of ports when enabled. When it is disabled, the Teams clients only connects to port 3478 on relays and doesn't use the extra 3479-3481 ports.

 Teams Admin Centre > Meeting Settings > Insert QoS Markets for real-time traffic:

 

The previous Skype for Business Network Assessment Tool only checked for connectivity to 3478 which means there may well have been issues with the 3479-3481 ports being blocked on the firewall - you couldn’t establish this using this tool.

If you do have a connectivity issue to one of the servers, then you will receive a specific error for that endpoint. Here is an example of what happened when I blocked connectivity to one of the relay servers:

 

You will see in the results here that “Service verification failed” and it tells me that it could not reach the relay server at 52.114.159.101. So - when you run the tool pay careful attention to the output because it doesn’t highlight these errors in red.

Note: If you are testing for a DoD or GCC domain they will have different Relays than the enterprise tenants. In this case you must include a specific FQDN in your config file (NetworkAssessmentTool.exe.config):

Office 365 Endpoint	Recommended Relay FQDN
Worldwide (including GCC)	worldaz.tr.teams.microsoft.com
U.S. Government DoD	tr.dod.teams.microsoft.us
U.S. Government GCC High	tr.gov.teams.microsoft.us

The setting is in the config file under this key:

<!-- Custom FQDN of the relay (VIP) to be used in the relay connectivity and quality checker  -->

<!-- Note that if this value is kept empty, then a default FQDN (from ECS) is used  -->

      <add key="Relay.FQDN" value=""/>

 

Quality Check

The Quality Check feature of the new version of the tool works slightly differently than the previous version of the tool. In the previous version the tool would send 17 seconds worth of media to the cloud and then report back on the result. In the new version the tool will constantly send media to the cloud and report the results in the command line every 5 seconds. By default, it will do this for 5 minutes and then stop. The command for doing a quality check is as follows:

Command Example:

C:\Program Files (x86)\Microsoft Teams Network Assessment Tool>NetworkAssessmentTool.exe /qualitycheck

  

You can see in the output above that the tool doesn't report reorder ratio like it did in the Skype for Business version. You will only get Latency, Jitter and Packet Loss statistics. The tool with each test will also write the results out to a CSV file so that you can review this data over the running period. Using the CSV file you can create your own graphs with Excel or any other graphing application you might prefer.

Something I have noticed with the new Teams tool compared to the previous Skype for Business tool is that it seems to give consistently higher Jitter values than the old version of the tool. I have found these values to be about 30%-50% higher. This may be a result of it calculating the jitter in a different way or because the length of the media sample it is averaging is about one third the length of the previous tool (5 seconds instead of 17 seconds).

When I ran the new and old tool side-by-side on the same machine on a relatively clean network I was seeing Jitter values of about 7ms reported on the old tool and 10-14ms on the new tool. Then when I injected some jitter using a network emulator, I was getting about 30ms of jitter being reported on the new Teams version of the Network Assessment Tool when the old tool was reporting 15ms of jitter. Bear this in mind when using the new tool as the Jitter values will be higher than you might expect. Based on this, I would say it would be difficult to hit Microsoft’s recommendation of under 15ms average for jitter using the new tool. It might be wise to be a little more lenient with the jitter thresholds and say 30ms as the value from the new Teams version of the tool.

 

Media Duration Setting

By default the Quality Checker will run for 5 minutes and output results every 5 seconds. To change the length of time the quality checker will run for you, need to change the MediaDuration setting in the config file (NetworkAssessmentTool.exe.config):

 

<!-- Duration of media flow for the quality checker, in seconds -->

<!-- Note that Ctrl+C can be pressed at any time to stop the quality check -->

      <add key="MediaDuration" value="300"/>

 

 Port Range Settings

An important thing to note here is that you are using non-default port in the Teams Admin Centre > Meeting Settings > Media Ports section (i.e not 50000-50019 for audio) then it’s best that you update the config file (NetworkAssessmentTool.exe.config) to match your configured range. By default, the tool is only checking connections from the Audio range, so if you really want to do testing from your Video and Screensharing range (e.g. you’re having quality issue specifically with Video or Screensharing) then you can expand or change the range to fall into your specific ranges.

<!--default source port ranges (inclusive) for the relay connectivity and quality checker     -->

      <!-- Audio: 50000-50019

           Video: 50020-50039

           VBSS:  50040-50059 -->

      <add key="MinimumSourcePort" value="50000"/>

      <add key="MaximumSourcePort" value="50019"/>

 

QoS Marker Setting

There is a setting in the config file for QoS Marking settings with and Audio or Video setting. This doesn’t appear to do anything in this version of the tool. I've found that in the current version of the tool I get CS5 (DSCP 40) which is a layer 2 precedence of 5 for both "Audio" and "Video" values. This is not what I would have expected, so I'm not sure if this is fully implemented yet.

<!-- UDP QoS Marking is used just for the quality checker -->

<!-- QoS Marking values should be chosen between 'None', 'Audio' or 'Video' -->

<!-- Note that this field does not apply to the relay connectivity checker -->

        <add key="QoSUDPMarking" value="none"/>

 

 

Excel Example Chart

My Network Assessor Tool GUI, that I made for the Skype for Business version of the tool, would create a line graph of each of the values for Latency, Packet Loss, Average Jitter and Reorder Ratio. A similar kind of graph can be achieved with the CSV output of the new tool using Excel. 

Microsoft's recommended values for maximum Latency, Jitter and Packet loss are as follows:  

Metric	Target
Latency (one way)	< 30ms
Latency (RTT)	< 60ms
Burst packet loss	<1% during any 200 ms interval
Packet loss	<0.1% during any 15s interval
Packet inter-arrival Jitter	<15ms during any 15s interval
Packet reorder	<0.01% out-of-order packets

Note: As I mentioned earlier, the new Teams Network Assessment Tool seems to report Jitter as being greater than the previous Skype for Business version of the tool. I would recommend using 30ms as the threshold for Jitter values in this tool.

Using the above settings as threshold values, you can create a chart that automatically highlights all the values that were out of these ranges. Here is an example of a chart created in Excel with this data:

 

Note: With this graph I used the Microsoft recommended 15ms for average Jitter, and as you can see it was quite consistently reporting a value close to or over that value. This is due to the Jitter reporting issue that I mentioned earlier. I believe that you should use a threshold around 30ms with the values reported from the new version of the tool.

There is a little bit of work required to tweak the graph to look like this. To display the red circles around the values that are outside of the threshold, you need to create a few extra columns in the CSV. Here is an example of formula I used to create these columns:

Column Header	Formula (Filled Down)
LossRate-Over	=IF(B2 > 10, B2,NA())
AverageLatency-Over	=IF(C2 > 60, C2,NA())
AverageJitter-Over	=IF(D2 > 30, D2,NA())

 

You then graph all the columns and 3 new columns listed above together, and change the newly calculated columns above to have markers that look like red circles instead of the original coloured dots. There is a little bit of fiddling required to make it look nice, but I’ll leave that your creative side. Here's an example of what I used for the Format of the Data Series:

 

Pass or Fail

In the previous Skype for Business version of the tool shipped with a results analyzer tool that would check your results file to see if it considered the tests to be a PASS or FAIL. It did this by checking if 10% or more of the tests had failed. We can reproduce this quite easily in Excel using the following formula example: 

=IF((((COUNTA(E2:E68) - COUNTIF(E2:E68,"=#N/A")) / COUNTA(E2:E68)) * 100) > 10,"FAIL","PASS" )

 

You need to run this at the bottom of each of the LossRate-Over, AverageLatency-Over, AverageJitter-Over columns that we calculated earlier. If there was over 10% of the attempts failed then it will  display FAIL and if not then it will present PASS. 

 

HTTP Infra Check

There is a new flag in the Teams Network Assessment Tool named "infraconnectivitytest". This offers a HTTP Infrastructure Check, which can be used to check from proxy issues when connecting to Teams web services. When you run the tool with this flag there is very detailed debug level logging that is outputted to the command line:

Amongst all of the debug messages you will see a summary of the connection attempts which looks like this:

==== Summary ====

 

Successful connections to URLs

        CONFIGS         URL

        9/9             'https://go.trouter.teams.microsoft.com/'

        9/9             'https://ic3.events.data.microsoft.com/Collector/3.0/'

        9/9             'https://config.teams.microsoft.com/config/'

        9/9             'https://api.flightproxy.teams.microsoft.com/api/v1/health'

 

Successful connections using ECS configuration

        URLs            CONFIG

        4/4             '{}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":1}}'

 

Presumably if connections don't work you will see some errors show up in this list. So keep an eye on the Summary output for any errors.

The Wrap Up

There have been a few nice improvements with this version of the Network Assessment tool from Microsoft. Hopefully, this post has shed some light on what it can do and what has changed from the previous version of the tool. If you have any questions feel free to drop me a comment below.

Read more →