Teams Network Assessment Tool (July 2021 Update)

UPDATE: I have released a new version of the Network Assessor for Microsoft Teams. Get it here: https://www.myteamslab.com/2024/11/network-assessor-for-microsoft-teams.html

In July of 2021 Microsoft released a new version of their Network Assessment Tool specifically for Microsoft Teams. This supersedes the old version that was originally released for Skype for Business which we were all still using for Teams deployments (until now). I have a front-end GUI that allows you to graph the results in real time using the Skype for Business version of the tool (still available here). The latest version of the tool appears to have had its core code rewritten and updated with Teams in mind. In this post I’ll do a deep dive into the new tool and compare it to its predecessor.

Here's the link for the new Teams Network Assessment Tool: https://www.microsoft.com/en-us/download/details.aspx?id=103017

At this time the Skype for Business Network Assessment Tool is still also available here: https://www.microsoft.com/en-us/download/details.aspx?id=53885

Note: The version discussed in this post is the Teams Network Assessment Tool 1.4.0.0.

The first change from the old tool is that the commands have slightly changed. By default, the previous tool would run a media quality test and report back latency, packet loss, jitter and reorder ratio. In the new tool, its default is to run a connectivity test to all of the Teams relay servers in Office 365, and the media quality check now requires you use the "qualitycheck" flag when you execute the tool. The output of both of the commands is also quite different than it was in the previous Skype for Business version, which will be demonstrated in the following sections. 

 

Relay Check

The relay check is used to ensure that you can successfully reach the Teams relay services within Office 365 from your current location. It’s important that you run this from a subnet that your Teams clients will be connecting from. There’s not a lot of value in running this from your home when the clients are going to be connecting from an office location that will have different networking and firewall configuration.

No command line flags are required when running the relay check. You simply execute the tool from the command line as shown below:

Command Example:

C:\Program Files (x86)\Microsoft Teams Network Assessment Tool>NetworkAssessmentTool.exe

 

Unlike the previous version of the tool, you do not get to see all the addresses that the tool attempts to connect to when using the verbose flag (as there is no verbose flag anymore). What you get now is a port negotiation check with two servers that will confirm connectivity on all of the 3478-3481 and 443 ports. After this, in the background, it does a check to the full list of relays. However it doesn’t print these tests to the command line:

Something that is new here is that it checks all the TURN redirect ports (3478-3481) for the various workloads (audio, video and screensharing). This matters when you have turned on the “Insert QoS Markets for real-time traffic” setting in the Teams Admin Centre. because this setting will make the relays use the full range of ports when enabled. When it is disabled, the Teams clients only connects to port 3478 on relays and doesn't use the extra 3479-3481 ports.

 Teams Admin Centre > Meeting Settings > Insert QoS Markets for real-time traffic:

 

The previous Skype for Business Network Assessment Tool only checked for connectivity to 3478 which means there may well have been issues with the 3479-3481 ports being blocked on the firewall - you couldn’t establish this using this tool.

If you do have a connectivity issue to one of the servers, then you will receive a specific error for that endpoint. Here is an example of what happened when I blocked connectivity to one of the relay servers:

 

You will see in the results here that “Service verification failed” and it tells me that it could not reach the relay server at 52.114.159.101. So - when you run the tool pay careful attention to the output because it doesn’t highlight these errors in red.

Note: If you are testing for a DoD or GCC domain they will have different Relays than the enterprise tenants. In this case you must include a specific FQDN in your config file (NetworkAssessmentTool.exe.config):

Office 365 Endpoint	Recommended Relay FQDN
Worldwide (including GCC)	worldaz.tr.teams.microsoft.com
U.S. Government DoD	tr.dod.teams.microsoft.us
U.S. Government GCC High	tr.gov.teams.microsoft.us

The setting is in the config file under this key:

<!-- Custom FQDN of the relay (VIP) to be used in the relay connectivity and quality checker  -->

<!-- Note that if this value is kept empty, then a default FQDN (from ECS) is used  -->

      <add key="Relay.FQDN" value=""/>

 

Quality Check

The Quality Check feature of the new version of the tool works slightly differently than the previous version of the tool. In the previous version the tool would send 17 seconds worth of media to the cloud and then report back on the result. In the new version the tool will constantly send media to the cloud and report the results in the command line every 5 seconds. By default, it will do this for 5 minutes and then stop. The command for doing a quality check is as follows:

Command Example:

C:\Program Files (x86)\Microsoft Teams Network Assessment Tool>NetworkAssessmentTool.exe /qualitycheck

  

You can see in the output above that the tool doesn't report reorder ratio like it did in the Skype for Business version. You will only get Latency, Jitter and Packet Loss statistics. The tool with each test will also write the results out to a CSV file so that you can review this data over the running period. Using the CSV file you can create your own graphs with Excel or any other graphing application you might prefer.

Something I have noticed with the new Teams tool compared to the previous Skype for Business tool is that it seems to give consistently higher Jitter values than the old version of the tool. I have found these values to be about 30%-50% higher. This may be a result of it calculating the jitter in a different way or because the length of the media sample it is averaging is about one third the length of the previous tool (5 seconds instead of 17 seconds).

When I ran the new and old tool side-by-side on the same machine on a relatively clean network I was seeing Jitter values of about 7ms reported on the old tool and 10-14ms on the new tool. Then when I injected some jitter using a network emulator, I was getting about 30ms of jitter being reported on the new Teams version of the Network Assessment Tool when the old tool was reporting 15ms of jitter. Bear this in mind when using the new tool as the Jitter values will be higher than you might expect. Based on this, I would say it would be difficult to hit Microsoft’s recommendation of under 15ms average for jitter using the new tool. It might be wise to be a little more lenient with the jitter thresholds and say 30ms as the value from the new Teams version of the tool.

 

Media Duration Setting

By default the Quality Checker will run for 5 minutes and output results every 5 seconds. To change the length of time the quality checker will run for you, need to change the MediaDuration setting in the config file (NetworkAssessmentTool.exe.config):

 

<!-- Duration of media flow for the quality checker, in seconds -->

<!-- Note that Ctrl+C can be pressed at any time to stop the quality check -->

      <add key="MediaDuration" value="300"/>

 

 Port Range Settings

An important thing to note here is that you are using non-default port in the Teams Admin Centre > Meeting Settings > Media Ports section (i.e not 50000-50019 for audio) then it’s best that you update the config file (NetworkAssessmentTool.exe.config) to match your configured range. By default, the tool is only checking connections from the Audio range, so if you really want to do testing from your Video and Screensharing range (e.g. you’re having quality issue specifically with Video or Screensharing) then you can expand or change the range to fall into your specific ranges.

<!--default source port ranges (inclusive) for the relay connectivity and quality checker     -->

      <!-- Audio: 50000-50019

           Video: 50020-50039

           VBSS:  50040-50059 -->

      <add key="MinimumSourcePort" value="50000"/>

      <add key="MaximumSourcePort" value="50019"/>

 

QoS Marker Setting

There is a setting in the config file for QoS Marking settings with and Audio or Video setting. This doesn’t appear to do anything in this version of the tool. I've found that in the current version of the tool I get CS5 (DSCP 40) which is a layer 2 precedence of 5 for both "Audio" and "Video" values. This is not what I would have expected, so I'm not sure if this is fully implemented yet.

<!-- UDP QoS Marking is used just for the quality checker -->

<!-- QoS Marking values should be chosen between 'None', 'Audio' or 'Video' -->

<!-- Note that this field does not apply to the relay connectivity checker -->

        <add key="QoSUDPMarking" value="none"/>

 

 

Excel Example Chart

My Network Assessor Tool GUI, that I made for the Skype for Business version of the tool, would create a line graph of each of the values for Latency, Packet Loss, Average Jitter and Reorder Ratio. A similar kind of graph can be achieved with the CSV output of the new tool using Excel. 

Microsoft's recommended values for maximum Latency, Jitter and Packet loss are as follows:  

Metric	Target
Latency (one way)	< 30ms
Latency (RTT)	< 60ms
Burst packet loss	<1% during any 200 ms interval
Packet loss	<0.1% during any 15s interval
Packet inter-arrival Jitter	<15ms during any 15s interval
Packet reorder	<0.01% out-of-order packets

Note: As I mentioned earlier, the new Teams Network Assessment Tool seems to report Jitter as being greater than the previous Skype for Business version of the tool. I would recommend using 30ms as the threshold for Jitter values in this tool.

Using the above settings as threshold values, you can create a chart that automatically highlights all the values that were out of these ranges. Here is an example of a chart created in Excel with this data:

 

Note: With this graph I used the Microsoft recommended 15ms for average Jitter, and as you can see it was quite consistently reporting a value close to or over that value. This is due to the Jitter reporting issue that I mentioned earlier. I believe that you should use a threshold around 30ms with the values reported from the new version of the tool.

There is a little bit of work required to tweak the graph to look like this. To display the red circles around the values that are outside of the threshold, you need to create a few extra columns in the CSV. Here is an example of formula I used to create these columns:

Column Header	Formula (Filled Down)
LossRate-Over	=IF(B2 > 10, B2,NA())
AverageLatency-Over	=IF(C2 > 60, C2,NA())
AverageJitter-Over	=IF(D2 > 30, D2,NA())

 

You then graph all the columns and 3 new columns listed above together, and change the newly calculated columns above to have markers that look like red circles instead of the original coloured dots. There is a little bit of fiddling required to make it look nice, but I’ll leave that your creative side. Here's an example of what I used for the Format of the Data Series:

 

Pass or Fail

In the previous Skype for Business version of the tool shipped with a results analyzer tool that would check your results file to see if it considered the tests to be a PASS or FAIL. It did this by checking if 10% or more of the tests had failed. We can reproduce this quite easily in Excel using the following formula example: 

=IF((((COUNTA(E2:E68) - COUNTIF(E2:E68,"=#N/A")) / COUNTA(E2:E68)) * 100) > 10,"FAIL","PASS" )

 

You need to run this at the bottom of each of the LossRate-Over, AverageLatency-Over, AverageJitter-Over columns that we calculated earlier. If there was over 10% of the attempts failed then it will  display FAIL and if not then it will present PASS. 

 

HTTP Infra Check

There is a new flag in the Teams Network Assessment Tool named "infraconnectivitytest". This offers a HTTP Infrastructure Check, which can be used to check from proxy issues when connecting to Teams web services. When you run the tool with this flag there is very detailed debug level logging that is outputted to the command line:

Amongst all of the debug messages you will see a summary of the connection attempts which looks like this:

==== Summary ====

 

Successful connections to URLs

        CONFIGS         URL

        9/9             'https://go.trouter.teams.microsoft.com/'

        9/9             'https://ic3.events.data.microsoft.com/Collector/3.0/'

        9/9             'https://config.teams.microsoft.com/config/'

        9/9             'https://api.flightproxy.teams.microsoft.com/api/v1/health'

 

Successful connections using ECS configuration

        URLs            CONFIG

        4/4             '{}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":0, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":1}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":0}}'

        4/4             '{"Regular":{"TLS_Force_Full_Handshake":1, "GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":1}}'

 

Presumably if connections don't work you will see some errors show up in this list. So keep an eye on the Summary output for any errors.

The Wrap Up

There have been a few nice improvements with this version of the Network Assessment tool from Microsoft. Hopefully, this post has shed some light on what it can do and what has changed from the previous version of the tool. If you have any questions feel free to drop me a comment below.

Read more →

Network Assessor for Skype for Business Online and Microsoft Teams

UPDATE: I have released a new version of the Network Assessor for Microsoft Teams. Get it here: https://www.myteamslab.com/2024/11/network-assessor-for-microsoft-teams.html

Skype for Business Online and Skype Teams are both real time communications tools that rely heavily on the networking infrastructure that they run on top of. This has always been, and will remain, a challenge in cloud deployments for the foreseeable future. In order to prepare for deploying Skype for Business Online or Microsoft Teams, you need to first ensure that your networking infrastructure will be able to handle the real-time traffic that these applications produce.

To help with this, Microsoft released a tool call the Skype for Business Network Assessment Tool. This tool runs via the command line and uses the same media stack that is used by the Skype for Business client to test Packet Loss, Jitter, Round Trip Latency and Reorder Packet Percentage on connections from your network into Microsoft’s Office 365. The audio stream created by the tool gets sent to Office 365 and then mirrored back from the closest Edge server location to your current physical location. Based on the return stream the tool is able to give accurate statistics on Packet Loss, Jitter, Round Trip Latency and Reorder Packet Percentage. This will give you an idea of the how congested the current network environment is and how well you might expect Skype for Business or Skype Teams conferences to function in your environment.

The Microsoft Network Assessment Tool is useful because it allows you to test a network before an Office 365 tenant has even been deployed and it’s also free! What the tool does lack though is the ability to easily monitor the network over a period of time and produce nicely formatted output that you can show to a manager or customer. So, I thought I would take some time and try to rectify these limitations and build a new front end for the tool…

Network Assessor

Introducing Network Assessor for Skype for Business and Microsoft Teams:

Version 1.00 Features

• Graphs Packet Loss, Average Jitter, Latency and Reorder Ratio.
• Zoom graphs in / out / forwards / backwards to view the data clearly.
• Start / Stop and Pause the network tests at the click of a button.
• Keep an eye on the status of testing using the tray icon colour by setting breach percentage thresholds. This will easily let you know that things are within your desired operating levels without having to open the interface. There are two levels of threshold breaches; one that changes the icon to orange in colour and the other that changes the icon to red in colour. These percentages are calculated on a per graph basis and if any graph breaches the percentage the colour will change.
• Graphs will automatically highlight in red points in each of the graphs which are outside of Microsoft’s recommended bounds of operation. There are two levels for these thresholds: Client thresholds and Edge Thresholds. For more details see the Usage section below.
• The status bar will display PASS/FAIL results for each graph. This calculation follows Microsoft's "ResultsAnalyzer.exe" PASS/FAIL calculation which is based on any graph having more than 10% of test attempts resulting in (Client or Edge) threshold breaches will equal a FAIL result.
• You can select the frequency at which the tool will run tests. This ranges between 1 and 120 minutes (ie. Run a 17 second long test every 1 minute or up to every 2 hours).
• Allows for graphs to be saved as PNG images for use in documentation/reports.
• All the graphs can be shown at once or individual graphs can be selected using the “Window” menu item. Graphs are saved at the resolution that they are displayed, so opening individual graphs and then saving them can offer higher resolutions.
• Every session that is created by the tool gets recorded in a CSV file for future reference. Sessions will “roll” to a new file based on the “Roll Time (hours)” setting in the Settings dialog. Logs can be rolled between 1 and 12 hours.
• CSV files can be imported back in to the tool by using the File > Import CSV File menu or dragging and dropping the file onto the interface directly from Windows Explorer.
• Supports automatic download (note: Internet connection is required for this to work) and installation of the Microsoft Network Assessment Tool and VC++ Redistributable pre-requisite by using the Help > Install Microsoft Tool menu item.

Version 1.10 Features

• Works with Microsoft tool version 1.1.0.0 released on 5/3/2018
• Support Connectivity Check. Select "File > Connectivity Check"
• Supports new Microsoft tool installation procedure
• The Microsoft Tool in its latest incarnation doesn’t handle all exceptions that it throws. These seem to be mostly around when it does the initial lookup REST call where it gets a JSON list of the servers to connect to for testing. If DNS fails or there is no network interface available on the machine the Microsoft tool throws an unhandled exception. If this happens the Network Assessor GUI will report an error and stop running tests. 
• Be careful when trying to manually install the new 5/3/2018 version of the Microsoft Command Line tool because it does not install properly over the top of previous version (i.e. it doesn't install over it and still says that installation was successful).

Note: This tool is not intended for load/stress testing!

DOWNLOAD HERE

Prerequisites

Tested with version 1.1.0.0 released on 5/3/2018 of the Microsoft Skype for Business Network Assessment Tool (LINK: https://www.microsoft.com/en-us/download/details.aspx?id=53885).

Network Assessor requires the following:

• At least .NET 4.5 must be installed on the PC to run the tool. (Windows 8 / Windows Server 2012 and above will support this by default. If you are using earlier operating systems the installer will download .net for you)
• Do not run the tool on a machine with multiple interfaces. Microsoft’s Network Assessment Tool may fail due to sending traffic out the wrong interface.
• When run on Windows Server, Microsoft’s command line tool needs “Media Foundation” (for Windows 2012, 2012 R2) or “Desktop Experience” (Windows 2008 R2) installed when running on Windows Server. These can be installed via the Server Manager interface or through Powershell:

Windows Server 2008 R2 (Desktop Experience install):

Import-Module ServerManager

Add-WindowsFeature Desktop-Experience

Windows Server 2012 or 2012 R2 (Media Foundation install):

Add-WindowsFeature Server-Media-Foundation

Note: This is not documented in Microsoft’s documentation but is required.

Microsoft’s command line tool and pre-reqs can be can automatically downloaded and installed using the Help > Install Microsoft Tool… menu directly in the Network Assessor interface. For manual offline installation of Microsoft’s Tool, install the following:

• Microsoft’s command line tool
• You must uninstall any old version of the tool before installing a newer version. The installer does not warn about this and fails to install over the top of existing versions.

Note: Microsoft’s tool should be left with all of the default settings in its config file.

Firewall Requirements:

The tool needs to be able to send traffic to Office 365 in order to work. The following ports are required to be allowed on firewalls between the Network Assessor tool and Office 365:

• TCP Source Ports: 50000 - 50019
• TCP Destination Port: 443
• UDP Source Ports: 50000 - 50019
• UDP Destination Port: 3478

Network Assessment Procedures:

For more procedural details on how to do Network Assessments, please refer to the Skype Operations Framework. There is a lot of useful documentation in the Determine Network Readiness section of Planning stage within this framework and should give you a good starting point for running Network Readiness assessments.

Usage Details

Walkthrough Video:

Main Window Settings

The “Run Every x Mins” setting controls how often the network assessment tool will be run and log results. This value can be set between 1 minute and 2 hours.

The Thresolds setting refers to two levels of thresholds that are recommend by Microsoft for real time audio for the statistics that are generated by the Network Assessment Tool. The first is the “Edge” values which are recommended for when you are testing from your perimeter network into Office 365. The second is recommended thresholds when testing from a “Client” subnet. The Edge values are lower and more restrictive than the client subnet because is expected that the client subnet will have more hops to get to Office 365 than the Edge.

Customer Edge to Microsoft Edge Thresholds:

Metric	Target
Latency (RTT)	< 60ms
Packet loss	<0.1% during any 15s interval Note: The value output by the Microsoft tool is not a percentage.
Packet inter-arrival Jitter	<15ms during any 15s interval
Packet reorder	<0.01% out-of-order packets Note: The value output by the Microsoft tool is not a percentage.

Client to Microsoft Edge Thresholds:

Metric	Target
Latency (RTT or Round-trip Time)	< 100ms
Packet loss	<1% during any 15s interval Note: The value output by the Microsoft tool is not a percentage.
Packet inter-arrival Jitter	<30ms during any 15s interval
Packet reorder	<0.05% out-of-order packets Note: The value output by the Microsoft tool is not a percentage.

Settings Window:

The tool has the following settings:

Network Assessment Tool Location: This is the location of the Skype for Business Network Assessment Command Line Tool on the system. If you have manually downloaded the tool and unzipped it to a folder on the system you can enter the location via the Browse button.

Session Log Folder: The Session Logs by default are placed on the system in the “C:\Users\<Username>\AppData\Roaming\NetworkAssessor\SessionLogs” folder. If for some reason you would like to change this folder (perhaps for centralizing the logs on a file share) you can select a new folder here.

Red Level Breach Percentage: This is the percentage of breaches (on a per graph basis) that has to occur for the task bar icon to change to red colour.

Orange Level Breach Percentage: This is the number of breaches (on a per graph basis) that has to occur for the task bar icon to change to orange colour.

Turn Off Minimise Notification: When this check box is ticked it will turn off the task bar notifications.

Auto Scroll Graph: When this check box is ticked it means that the graphs will automatically scroll when new points are added to it graph.

Roll Time (Hours): This is the number of hours of testing that will be logged to CSV until a new session log file is created. When the file rolls the graphs will also be cleared. This is to try and maintain the performance of the graphs. Also, when the graph is cleared the breach counters will also be cleared.

The Wrap Up

Hopefully this post will give you another tool in the toolbox to help you in your preparation for deploying Skype for Business Online or Microsoft Teams in your environment. If you have any comments or issues with the tool please provide feedback below.

Read more →