In July of 2021 Microsoft released a new version of
their Network Assessment Tool specifically for Microsoft Teams. This supersedes the old version that was originally released for Skype for Business which we were all still using for Teams deployments (until now). I have a front-end GUI that allows you to graph the results in real time using the Skype for Business version of the tool (still available here). The latest version of the tool appears to have had its core code rewritten and updated with Teams in mind. In this post I’ll do a
deep dive into the new tool and compare it to its predecessor.
The first change from the old tool is that the commands
have slightly changed. By default, the previous tool would run a media quality test and report back latency, packet loss, jitter and reorder ratio. In the new tool, its default is to run a connectivity test to all of the Teams relay servers in Office 365, and the media quality
check now requires you use the "qualitycheck" flag when you execute the tool. The
output of both of the commands is also quite different than it was in the previous Skype for Business version, which will be demonstrated in the following sections.
The relay check is used to ensure that you can successfully reach the Teams
relay services within Office 365 from your current location. It’s important that
you run this from a subnet that your Teams clients will be connecting from. There’s not a lot of value in running this from your home when
the clients are going to be connecting from an office location that will have different networking and firewall configuration.
No command line flags are required when running the relay check. You
simply execute the tool from the command line as shown below:
Unlike the previous version of the tool, you do not get to
see all the addresses that the tool attempts to connect to when using the verbose flag (as there is
no verbose flag anymore). What you get now is a port negotiation check with two
servers that will confirm connectivity on all of the 3478-3481 and 443 ports.
After this, in the background, it does a check to the full list of relays. However it doesn’t print these tests to the command line:
Something that is new here is that it checks all the TURN
redirect ports (3478-3481) for the various workloads (audio, video and screensharing). This matters when you have turned on the “Insert QoS Markets for
real-time traffic” setting in the Teams Admin Centre. because this setting will make the
relays use the full range of ports when enabled. When it is disabled, the
Teams clients only connects to port 3478 on relays and doesn't use the extra 3479-3481
ports.
Teams Admin Centre > Meeting Settings > Insert QoS
Markets for real-time traffic:
The previous Skype for Business Network Assessment Tool
only checked for connectivity to 3478 which means there may well have been
issues with the 3479-3481 ports being blocked on the firewall - you couldn’t
establish this using this tool.
If you do have a connectivity issue to one of the servers, then you will receive a specific error for that endpoint. Here is an example of
what happened when I blocked connectivity to one of the relay servers:
You will see in the results here that “Service
verification failed” and it tells me that it could not reach the relay server
at 52.114.159.101. So - when you run the tool pay careful attention to the output
because it doesn’t highlight these errors in red.
Note: If you are testing for a DoD or GCC domain
they will have different Relays than the enterprise tenants. In this case you
must include a specific FQDN in your config file (NetworkAssessmentTool.exe.config):
|
Office 365 Endpoint
|
Recommended Relay FQDN
|
|
Worldwide (including GCC)
|
worldaz.tr.teams.microsoft.com
|
|
U.S. Government DoD
|
tr.dod.teams.microsoft.us
|
|
U.S. Government GCC High
|
tr.gov.teams.microsoft.us
|
The setting is in the config file under this key:
<!-- Custom FQDN of the relay (VIP) to be used in the relay
connectivity and quality checker -->
<!-- Note that if
this value is kept empty, then a default FQDN (from ECS) is used -->
<add
key="Relay.FQDN" value=""/>
Quality Check
The Quality Check feature of the new version of the tool works
slightly differently than the previous version of the tool. In the previous
version the tool would send 17 seconds worth of media to the cloud and then
report back on the result. In the new version the tool will constantly send
media to the cloud and report the results in the command line every 5 seconds. By default, it will do this for 5 minutes and then stop. The command for doing a quality check is as follows:
Command Example:
C:\Program Files (x86)\Microsoft Teams Network Assessment
Tool>NetworkAssessmentTool.exe /qualitycheck
You can see in the output above that the tool doesn't report reorder ratio like it did in the Skype for Business version. You will only get Latency, Jitter and Packet Loss statistics. The tool with each test will also write the results out to a CSV file so that you can review this data over the running period. Using the CSV
file you can create your own graphs with Excel or any other graphing application you might prefer.
Something I have noticed with the new Teams tool compared
to the previous Skype for Business tool is that it seems to give consistently
higher Jitter values than the old version of the tool. I have found these values to be about 30%-50%
higher. This may be a result of it calculating the jitter in a
different way or because the length of the media sample it is averaging is about one
third the length of the previous tool (5 seconds instead of 17 seconds).
When I ran the new and old tool side-by-side on the same
machine on a relatively clean network I was seeing Jitter values of about 7ms reported on the old tool and
10-14ms on the new tool. Then when I injected some jitter using a network emulator,
I was getting about 30ms of jitter being reported on the new Teams version of the
Network Assessment Tool when the old tool was reporting 15ms of jitter. Bear this in mind when using the new tool as the Jitter values will be higher than you might
expect. Based on this, I would say it would be difficult to hit Microsoft’s
recommendation of under 15ms average for jitter using the new tool. It might be
wise to be a little more lenient with the jitter thresholds and say 30ms as the
value from the new Teams version of the tool.
Media Duration Setting
By default the Quality Checker will run for 5 minutes and
output results every 5 seconds. To change the length of time the quality
checker will run for you, need to change the MediaDuration setting in the config
file (NetworkAssessmentTool.exe.config):
<!-- Duration of media flow for the quality checker, in
seconds -->
<!-- Note that Ctrl+C can be pressed at any time to stop the
quality check -->
<add
key="MediaDuration" value="300"/>
Port Range Settings
An important thing to note here is that you are using
non-default port in the Teams Admin Centre > Meeting Settings > Media
Ports section (i.e not 50000-50019 for audio) then it’s best that you update
the config file (NetworkAssessmentTool.exe.config) to match your configured range. By
default, the tool is only checking connections from the Audio range, so if you
really want to do testing from your Video and Screensharing range (e.g. you’re
having quality issue specifically with Video or Screensharing) then you can
expand or change the range to fall into your specific ranges.
<!--default source port ranges (inclusive) for the relay
connectivity and quality checker
-->
<!-- Audio:
50000-50019
Video:
50020-50039
VBSS: 50040-50059 -->
<add
key="MinimumSourcePort" value="50000"/>
<add key="MaximumSourcePort"
value="50019"/>
QoS Marker Setting
There is a setting in the config file for QoS Marking
settings with and Audio or Video setting. This doesn’t appear to do anything in
this version of the tool. I've found that in the current version of the tool I get CS5 (DSCP 40) which is a layer 2 precedence of 5 for both "Audio" and "Video" values. This is not what I would have expected, so I'm not sure if this is fully implemented yet.
<!-- UDP QoS Marking is used just for the quality checker
-->
<!-- QoS Marking values should be chosen between 'None',
'Audio' or 'Video' -->
<!-- Note that this field does not apply to the relay
connectivity checker -->
<add key="QoSUDPMarking"
value="none"/>
Excel Example Chart
My Network Assessor Tool GUI, that I made for the Skype for Business version of the tool, would
create a line graph of each of the values for Latency, Packet Loss, Average
Jitter and Reorder Ratio. A similar kind of graph can be achieved with the CSV output of the new tool using Excel.
Microsoft's recommended values for maximum Latency, Jitter and Packet loss are as follows:
|
Metric
|
Target
|
|
Latency (one way)
|
<
30ms
|
|
Latency (RTT)
|
<
60ms
|
|
Burst packet loss
|
<1%
during any 200 ms interval
|
|
Packet loss
|
<0.1%
during any 15s interval
|
|
Packet inter-arrival Jitter
|
<15ms
during any 15s interval
|
|
Packet reorder
|
<0.01%
out-of-order packets
|
Note: As I mentioned earlier, the new Teams Network Assessment Tool seems to report
Jitter as being greater than the previous Skype for Business version of the tool. I would
recommend using 30ms as the threshold for Jitter values in this tool.
Using the above settings as threshold values, you can
create a chart that automatically highlights all the values that were out of
these ranges. Here is an example of a chart created in Excel with this data:
Note: With this graph I used the Microsoft recommended 15ms for average Jitter, and as you can see it was quite consistently reporting
a value close to or over that value. This is due to the Jitter reporting issue that I mentioned earlier. I believe that you should use a threshold around 30ms with the values reported from the new version of the tool.
There is a little bit of work required to tweak the graph to look
like this. To display the red circles around the values that are outside of the
threshold, you need to create a few extra columns in the CSV. Here is an example
of formula I used to create these columns:
|
Column Header
|
Formula (Filled Down)
|
|
LossRate-Over
|
=IF(B2
> 10, B2,NA())
|
|
AverageLatency-Over
|
=IF(C2
> 60, C2,NA())
|
|
AverageJitter-Over
|
=IF(D2
> 30, D2,NA())
|
You then graph all the columns and 3 new columns listed above together, and change the newly calculated columns above to have markers that look like red circles instead of the original
coloured dots. There is a little bit of fiddling required to make it look nice,
but I’ll leave that your creative side. Here's an example of what I used for the Format of the Data Series:
Pass or Fail
In the previous Skype for Business version of the tool shipped
with a results analyzer tool that would check your results file to see if it
considered the tests to be a PASS or FAIL. It did this by checking if 10% or
more of the tests had failed. We can reproduce this quite easily in Excel using
the following formula example:
=IF((((COUNTA(E2:E68) - COUNTIF(E2:E68,"=#N/A")) /
COUNTA(E2:E68)) * 100) > 10,"FAIL","PASS" )
You need to run this at the bottom of each of the LossRate-Over,
AverageLatency-Over, AverageJitter-Over columns that we calculated earlier. If
there was over 10% of the attempts failed then it will display FAIL and if not then it will present
PASS.
HTTP Infra Check
There is a new flag in the Teams Network Assessment Tool named "infraconnectivitytest". This offers a HTTP Infrastructure Check, which can be used to check from proxy issues when connecting to Teams web services. When you run the tool with this flag there is very detailed debug level logging that is outputted to the command line:
Amongst all of the debug messages you will see a summary of the connection attempts which looks like this:
==== Summary
====
Successful
connections to URLs
CONFIGS URL
9/9
'https://go.trouter.teams.microsoft.com/'
9/9
'https://ic3.events.data.microsoft.com/Collector/3.0/'
9/9
'https://config.teams.microsoft.com/config/'
9/9
'https://api.flightproxy.teams.microsoft.com/api/v1/health'
Successful
connections using ECS configuration
URLs CONFIG
4/4 '{}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":0,
"GenericTcpConnect_Version":1,
"IPv6_Killswitch_Enabled":0}}'
4/4 '{"Regular":{"TLS_Force_Full_Handshake":0,
"GenericTcpConnect_Version":1,
"IPv6_Killswitch_Enabled":1}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":0,
"GenericTcpConnect_Version":2,
"Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":0}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":0,
"GenericTcpConnect_Version":2,
"Proxy_RespectSystemProxy":1,
"IPv6_Killswitch_Enabled":1}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":1,
"GenericTcpConnect_Version":1, "IPv6_Killswitch_Enabled":0}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":1,
"GenericTcpConnect_Version":1,
"IPv6_Killswitch_Enabled":1}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":1,
"GenericTcpConnect_Version":2, "Proxy_RespectSystemProxy":1,
"IPv6_Killswitch_Enabled":0}}'
4/4
'{"Regular":{"TLS_Force_Full_Handshake":1,
"GenericTcpConnect_Version":2,
"Proxy_RespectSystemProxy":1, "IPv6_Killswitch_Enabled":1}}'
Presumably if connections don't work you will see some errors show up in this list. So keep an eye on the Summary output for any errors.
The Wrap Up
There have been a few nice improvements with this version
of the Network Assessment tool from Microsoft. Hopefully, this post has shed some light on what it can do and what has changed from the previous version of the tool. If you have any questions feel free to drop me a
comment below.
0
comments
