Microsoft Teams Survivable Branch Appliance (SBA) is “410 Gone”

When configuring a Microsoft Teams Direct Routing Survivable Branch Appliance the other day I ran into an error. It was an interesting one that I think others will likely run into at some point too, so here is a blog post to save you wasting any more time than you need to thinking about it. After setting up the Teams SBA, I found that I could get users connected to it to send calls outbound to the PSTN, however, when I tried to send calls to the users in the other direction they would fail. Looking closer at the logging from the SBC I could see the error response was “410 Gone”… Gone Baby Gone…  

The 410 Gone error from a Teams SBA looked like this:

SIP/2.0 410 Gone

FROM: <sip:+61399992000@10.0.0.25>;tag=1c1327591735

TO: <sip:+61388886201@sbc01.mym365lab.us>;tag=703d3577eb1a468bbcdb34a0a78c690f

CSEQ: 1 INVITE

CALL-ID: 530796863134202353123@sbc01.domain.com

VIA: SIP/2.0/TLS sbc01.mym365lab.us:5067;branch=z9hG4bKac1432917422

REASON: Q.850;cause=22;text="c586066f-ceb5-4d83-8803-400791d033de;MediaOfferError"

CONTACT: <sip:teamssba01.domain.com:5061;transport=tls;x-i=c586066f-ceb5-4d83-8803-400791d033de;x-c=9baa3cbe22ec46ffb8ac39fffce08f20>

CONTENT-LENGTH: 0

ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY

SERVER: Microsoft.Teams.SIPSBA v.2022.6.14.1

 

"Gone" is not a SIP error that I have seen very often in the wild and kind of sounds like it might be related to the user not being connected to the SBA properly. However, after looking more closely at the error, I saw that there was a REASON attribute included in the message that said “MediaOfferError” which then made me further consider what was happening on the media side of things… From most traditional SIP Stacks, I would usually expect a “488 Not Acceptable Here” message response for an SDP refusal scenario, but this is the Teams SBA, so why expect anything that makes sense :)

When I looked at the INVITE that was being sent to the SBA from the SBC, it looked legit:

INVITE sip:+61388886201@sbc01.mym365lab.us SIP/2.0

Via: SIP/2.0/TLS sbc01.domain.com:5067;alias;branch=z9hG4bKac1432917422

Max-Forwards: 69

From: <sip:+61399992000@10.0.0.25>;tag=1c1327591735

To: <sip:+61388886201@sbc01.domain.com>

Call-ID: 530796863134202353123@sbc01.mym365lab.us

CSeq: 1 INVITE

Contact: <sip:+61399992000@sbc01.domain.com:5067;transport=tls;ob>

Supported: norefersub,100rel,timer,replaces,sdp-anat

Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS

Session-Expires: 1800

Min-SE: 90

User-Agent: Mediant SW/v.7.40A.250.265

Content-Type: application/sdp

Content-Length: 287

 

v=0

o=- 419722796 1831657774 IN IP4 10.0.0.25

s=media

b=AS:84

t=0 0

a=X-nat:0

m=audio 50012 RTP/AVP 0 8 96

c=IN IP4 10.0.0.25

b=TIAS:64000

a=rtcp:50013 IN IP4 10.0.0.25

a=sendrecv

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:96 telephone-event/8000

a=fmtp:96 0-16

 

The keen eyed folk in the audience may have noticed though that the SDP portion of the SIP message is in the regular format and not the fancy ICE format that includes a candidate list. The even more keen eyed may be thinking, "yeah, but the Teams Direct Routing service supports this format for SDP, so what's the problem?". Well, unfortunately, the problem is that the SBA only supports ICE formatted SDP lists (which may be because calls are technically media bypassing the SBA). I could see this catching out production deployments where the traditional SDP format is being used for calls to Teams Direct Routing (because this does work). Just keep in mind that the same settings pointing to the SBA will fail dismally.

The Fix

In order to fix this on an AudioCodes SBC you need to make sure that the IP Profile assigned to the SBA has ICE Mode set to “Lite”:

After enabling ICE Lite Mode, the INVITE messages that go to the SBA will be formatted with candidate information, which looks like this:

INVITE sip:+61398736201@sbc01.domain.com SIP/2.0

Via: SIP/2.0/TLS sbc01.domain.com:5067;alias;branch=z9hG4bKac11041617

Max-Forwards: 69

From: <sip:+61399992000@10.0.0.25>;tag=1c200843402

To: <sip:+61388886201@sbc01.domain.com>

Call-ID: 431229997134202354040@sbc01.mym365lab.us

CSeq: 1 INVITE

Contact: <sip:+61399992000@sbc01.domain.com:5067;transport=tls;ob>

Supported: norefersub,100rel,timer,replaces,sdp-anat

Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS

Session-Expires: 1800

Min-SE: 90

User-Agent: Mediant SW/v.7.40A.250.265

Content-Type: application/sdp

Content-Length: 900

 

v=0

o=- 617366211 1969606384 IN IP4 10.0.0.25

s=media

b=AS:84

t=0 0

a=X-nat:0

a=ice-lite

m=audio 50004 RTP/SAVP 0 8 96

c=IN IP4 10.0.0.25

b=TIAS:64000

a=rtcp:50005 IN IP4 10.0.0.25

a=sendrecv

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:96 telephone-event/8000

a=fmtp:96 0-16

a=ice-ufrag:IE15BCMtIoIFC4S8

a=ice-pwd:VYGaTzCfX9gdvnfd9CuDiHuw

a=candidate:128566170 1 udp 2130706431 10.0.0.25 50004 typ host

a=candidate:128566170 2 udp 2130706430 10.0.0.25 50005 typ host

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:UKAS7+0XGSZeXGFreZWWQU0vuf0D1D/F97dVtVLs|2^31

a=crypto:2 AES_CM_128_HMAC_SHA1_32 inline:2kuctHvNudt69PR8gxX/SPcX8LLWGsqdD+uecRxe|2^31

a=crypto:3 AES_256_CM_HMAC_SHA1_80 inline:L0E/e24PTtJ9mUwwWdb3QlCQBtd4WdSuSZvVi45ZkkyhqhnwGTuZGbXjcWJ6dc==|2^31

a=crypto:4 AES_256_CM_HMAC_SHA1_32 inline:z2WjgpGD1HC2HR0+3NZVEfuwEVJ8u0CyHAKuEZAIbMtnHCNw0ieSE22XVCQ39s==|2^31

 

Once you’ve done this, calls from the PSTN to the Teams Client via the SBA should work.

 

The Wrap Up

 

There you have it, more madcapped craziness from the world of Microsoft Teams. Till next time, I’m “410 Gone”. Catch ya later!

Read more →

Microsoft Teams Location Based Bandwidth Control (Network Roaming Policy)

Microsoft Teams now has the ability to limit the network bandwidth used by calls/meetings based on the network location of the user. This uses a feature called Network Roaming Policy. I have found the documentation relating to this feature to be a bit lacking from Microsoft, so I’ve put together this post to go into some more detail about how the policy works.

This feature is an extension of the existing Meeting Policy settings that have always been available for Teams. I wrote an extensive post about how Meeting Policy bandwidth control works  over at this post (https://www.myteamslab.com/2019/10/microsoft-teams-bandwidth-usage-deep.html). The Network Roaming policy has the same effect on the Teams client. However, it's now dynamically implemented by the client based on its location, rather than it always being on in the previous Meeting Policy implementation. This means that if you have a specific site that you know has low bandwidth constraints, then you can limit the maximum bandwidth per call and also restrict video usage for this site location only.

 

How Does the Policy Work?

The Network Roaming Policy is based on network IP Addressing of the client machine and the NATed IP Address of the client as it access the Internet. This relies on the configuration of both the Trusted IP Address ranges and Network Site subnets within the Teams Admin Centre. The client will compare both its local subnet and its public IP address (NATed address that it accesses the Internet through, e.g. Type “What's my IP” into Google) in order to know if it will implement the Network Roaming policy. The diagram below shows two different sites with different LAN Address ranges as well as different internet egress IP Addresses through the Internet facing firewall:

Note: The policy is not just based on the IP Address of the user because you could have multiple sites with the same internal private IP Address range. The Trusted IP (Public IP) must also match for the policy to be implemented by the client.

In the configuration example we will configure a Network Roaming policy for the "Low Bandwidth Site" on the left hand side of the diagram. The site on the right hand side will not be configured and will fall back to having the default bandwidth settings used for Teams.

Importantly, the only clients that currently support the Network Roaming policy at the moment are the Windows and MacOS desktop clients. So don’t expect this to work with Teams Phones, MTRs, or Linux clients. 

 

Configuration of Network Roaming Bandwidth Policy

Note: In classic cloud style, you will usually need to wait about 24 hours before this policy takes effect. As a result, make sure you're not in any kind of rush when setting this up.

Network Roaming Policy is configured under the Locations > Network Topology section of the Teams Admin Centre: 

The Network Topology section consists of 3 tabs - Network Sites, Trusted Sites and Roaming Policy. You will need to configure all of these areas in order for Network Roaming Policy to work.  For this example we will configure a policy that will limit the Teams client to only use a maximum of 300kbps worth of bandwidth for its Audio and Video streams.

Step 1. Start by creating a Network Roaming Policy from the Roaming Policy tab, select Add:

Step 2. Configure the policy with the required bandwidth per call and whether or not video will be supported (for more details about how much bandwidth is used for video calls, see my previous post here: https://www.myteamslab.com/2019/10/microsoft-teams-bandwidth-usage-deep.html):

Step 3. Select the Trusted IPs tab and click the Add button:

Step 4. The trusted IP Address is the external facing NATed IP address that Office 365 will see as the source address coming from your client connection. If you search for “What's my IP” on Google from the location it will tell you what this IP Address is. In the case of connecting to Office 365 there are likely a range of IP Addresses used here, so you need to get the Network Mask correct.  

 Step 5. In the network sites tab you need to create a new site, click the Add button:

 In the new Site policy you select the Network Roaming Policy that was created in Step 1:

Each subnet that is used internally at the site should be added to the Site by clicking the Add Subnets button:

Step 6: In addition to the Network Location configuration above, you also need to turn on Network Configuration Lookup in Meeting Policy:

Within the Meeting policy you need to ensure that the following setting is enabled:

The Microsoft Docs (https://learn.microsoft.com/en-us/microsoftteams/network-roaming-policy) say the following:

“To enable the network roaming policy for users who are not enterprise voice enabled, you must also enable the AllowNetworkConfigurationSettingsLookup setting in TeamsMeetingPolicy. This setting is off by default.”

So really, it’s best that you always turn this on if you want the Network Roaming Policy to take effect for everyone.

 

 

How do you know if the policy is working?

 

The client doesn’t display anything to the user to inform them that this policy is in use. The only way you can really tell is by looking into the Teams Client logs. You can get the client to output the logs by pressing the Ctrl + Alt + Shift + 1 keys on your keyboard. When this is done the client will output log files to your Downloads folder. From here you open up the following file:

Downloads\MSTeams Diagnostics Log <Date>\web\ MSTeams Diagnostics Log <Date>_calling.txt

This file contains information about if the client has matched any of the existing policies.

 

When the policy is not being applied you will find something like this indicating that the default policy is in use:

" networkRoamingPolicy": {

" allowIPVideo " : true ,

"mediaBitRateKb" : 50000,

"policyDocument ": "Default "

 

When the Network Roaming policy has been successfully deployed you should see the Network Roaming Policy section of the file display information about the policy that the client is implementing.  Importantly, the trustedIpMatchInfo and siteMatchInfo sections must say that they have "Matched" one of the policies.

 

Current MT location response:

{

  "emergencyCallingPolicy": {

    "policyDocument": "Default"

  },

  "emergencyCallRoutingPolicy": {

    "emergencyNumbers": [],

    "policyDocument": "Default"

  },

  "networkRoamingPolicy": {

    "allowIPVideo": true,

    "mediaBitRateKb": 300,

    "policyDocument": "TeamsNetworkRoamingPolicy=Tenant:300kbps"

  },

  "endpointNetwork": "Trusted",

  "networkSiteId": "Low Bandwidth Site",

  "enableLocationBasedRouting": false,

  "siteAddress": "Low Bandwidth Site",

  "subnetId": "10.1.0.0",

  "debugInfo": {

    "ncsDebugInfo": {

      "trustedIpMatchInfo": {

        "publicIp": "50.1.2.100",

        "trustedIpAddress": "50.1.2.100",

        "maskBits": 24,

        "reason": "Matched",

        "_comment": "Match Client Public IP to Tenant Trusted IP"

      },

      "siteMatchInfo": {

        "ipv4": "10.1.0.180",

        "subnetLengthIPv4": "24",

        "subnetId": "10.1.0.0",

        "maskBits": 24,

        "networkSiteId": "Low Bandwidth Site",

        "enableLocationBasedRouting": false,

        "reason": "Matched",

        "_comment": "Used to match endpoint subnet to Tenant site if trustedIpMatchInfo matches"

      },

      "networkLocationMatchInfo": {

        "bssid": "74-ac-b9-2e-f3-b3",

        "ipv4": "10.1.0.180",

        "reason": "NotMatched",

        "_comment": "Used to find emergency address,against Tenant Location Network Information (LIS), otherwise against Client Geo Location Information (CLS) if available"

      }

    },

    "mtDebugInfo": {

      "isDirectRoutingOnlyUser": true,

      "emergencyCallingPolicyTag": "Default",

      "emergencyCallRoutingPolicyTag": "Default",

      "networkRoamingPolicyTag": "TeamsNetworkRoamingPolicy=Tenant:300kbps",

      "emergencyCallingPolicyAssignedTo": "Tenant or Host Global",

      "emergencyCallRoutingPolicyAssignedTo": "Tenant or Host Global",

      "networkRoamingPolicyAssignedTo": "Network Site",

      "ncsResponseReceived": true,

      "correlationId": "1DEF3C2C47D64C1EB7060657876ECE95"

    }

  }

}

   

The Wrap Up

Awesome - now you can walk up to random people on the street and tell them about how you know all about Teams Network Roaming Policy. Believe me, they will be thrilled to hear all the details. Especially the bit about the AllowNetworkConfigurationSettingsLookup setting. That one really cracks them up. Cheers, Enjoy!

Read more →

Teams PowerShell Module - Certificate Authentication

There are several ways to get the Teams PowerShell module to authenticate against Azure in order to get access to running Teams PowerShell in your Tenancy. Most people will be used to using the interactive method, where you just run the basic Connect-MicrosoftTeams command and get an interactive Azure auth screen that pops up and you enter your user account details into. This is fine if you are manually connecting and doing this by hand. However, what if you want to run an Azure Function that needs to authenticate automatically each time it executes? Well, for that you want to use an application authentication method.

Currently as of the module version 4.9.1, Microsoft officially supports two methods for application authentication when connecting to the Teams PowerShell module:

• Certificate based authentication – In this method you have a certificate with private and public keys. The PC connecting to Azure needs to have a copy of the private key and Azure needs to have a stored copy of the public key. As part of the connection the private key will be used to sign the connection and if Azure can decrypt the information with the public key then the PC is trusted to connect. This essentially makes the private key like a password that you need to ensure that no one else has access to.
• Token based authentication – The token-based method requires that you set up a Client Secret in Azure. When you connect to Azure using the PowerShell module, you authenticate against the Token Service using the Client Secret and get Tokens back that you can use to connect to the service. In the case of the Teams module, you need to get two tokens to be able to run all the commands. You can find out more about this option here: Token Auth Post

In this post we are going to focus on the certificate-based authentication method. Here are the steps for setting up and connecting using certificate-based authentication:

 

Step 1: Generate a Certificate:

The good news here is that the certificate you need does not need to be signed by an internet-based Certificate Authority. You can simply create a self-signed certificate on a PC and use this for connection. Below is a PowerShell command you can run (you must run the PowerShell window as Administrator to execute it). The most important flag in the command is the KeySpec flag which tells it to generate a certificate that can be used for Key Exchange.

New-SelfSignedCertificate -certstorelocation cert:\localmachine\my -Subject TeamsAppTestCertificate01 -KeySpec KeyExchange

 

Step 2: Export the Certificate:

In order to export the certificate you can either run PowerShell or do it by hand with the Certificate Snap-in. Both options will be detailed below. Choose the one that you’re most comfortable with.

Export the Private Key (This step is only needed if you want to run PowerShell from a different machine that you have created the certificate on, for example, a Function App, etc):

PowerShell:

$password = ConvertTo-SecureString -String "SpecialPassword123!" -Force –AsPlainText

Export-PfxCertificate -Cert "cert:\LocalMachine\My\8E4CKSHDUSG873F66D99AC7935F53" -FilePath "C:\temp\AuthPrivateKey.pfx" -Password $password

Note: The thumbprint to be used here will be output from the New-SelfSignedCertificate command you previously ran.

 

Or Windows UI:

Once again, this is is only needed if you want to run PowerShell from a different machine that you have created the certificate on, for example, a Function App, etc.

Search PC for mmc.exe > File Menu > Add or Remove Snap-in > Certificates > Add > Computer Account > Personal > Certificates Folder > Right Click on the Certificate > Export…

Export Certificate Wizard:

Click Next...

Select: "Yes, export the private key":

Go with the default export options:

Enter a password for the PFX file and ensure that you select TripleDES-SHA1 (I have found that importing on some platforms is not supported with AES):

Enter a name and location for the pfx file to be output to:

Export the Public Key (to be uploaded to the App Registration in Azure)

PowerShell:

Export-Certificate -Cert "Cert:\LocalMachine\My\8E4CKSHDUSG873F66D99AC7935F53" -FilePath "C:\temp\AuthPublicKey.cer"

Or Windows UI:

Click Next... 

Do not export private key:

Select the format of the file (DER is okay in this case) and click next:

Select the name of the file and click next:

Step 3: Configure the App Registration:

Open the Azure AD Portal and select the Azure Active Directory > App Registration section:

 Click the "New Registration" Button:

 

Fill in a name for the application and click the Register button:

The App Registration will now be created, however, there is still more config to do:

Go to the “Certificates & secrets” blade:

Select the location of the public key certificate that you exported earlier. Add a description and click OK:

The certificate should now show up in the certificates area:

Now open the API Permissions tab within the App Registration, Click "Add a permission":

You will need to add the following Graph API permissions to the App Registration:

User.Read.All

Group.ReadWrite.All

AppCatalog.ReadWrite.All

TeamSettings.ReadWrite.All

Channel.Delete.All

ChannelSettings.ReadWrite.All

ChannelMember.ReadWrite.All

These permissions are documented by Microsoft here, so you may want to check to see if there have been any updates for the PowerShell version you're using: https://learn.microsoft.com/en-us/microsoftteams/teams-powershell-application-authentication

Click the "Microsoft Graph" option:

Select "Application Permissions":

Select the permissions that were listed above:

Ensure that you have added all the permissions. If you missed any then repeat the above steps for all the permissions in the list…

 

After adding all the permissions, you need to click the “Grant admin consent for <tenant id>” button on the main Permissions screen:

At the end of this procedure, you should have the following permissions all assigned with admin consent granted:

In a final twist in this adventure, you also need to make sure you assign Teams Administrator privileges to the App Registration in order for it to be able to run CS commands. You do this by going to the Active Directory tab > Roles and Administrators > Teams Administrator Role:

 

Assign the Service Permission that is named the same as your App Registration to the RBAC Role:

Connecting To Teams PowerShell

 

Now that all the backend work has been done we can get down to doing some connecting. When you connect it needs to be from a machine that has access to the certificate. On a PC this means it has to be in the certificate store. If it’s from a machine in Azure there are various methods for doing this too (which I may well get to in another post…).

You will need to know the ApplicationId that was given to the App Registration in Azure. You can find this out by looking going to the Overview tab and looking for the GUID that has the title “Application (client) ID”. The TenantId is the base domain name that was first given to your tenancy when it was created or alternatively can be the GUID that’s in the Overview tab named as “Directory (tenant) ID” (both options will work).

 

Teams Module Connect Command:

Connect-MicrosoftTeams -CertificateThumbprint "3ab0e057bc3278ecb2a33123042e5a7a8001ff8c" -ApplicationId "319d0a47-9a48-45b0-b416-14aca00e7ece" -TenantId contoso.onmicrosoft.com

 

When connection is successful, you’ll get back an object displayed in the PowerShell window that tells you the Account, Environment, Tenant and TenantId values. From here you should be able to run almost all of the commands from the module, with the following exclusions:

As of 4.9.1; All cmdlets are supported now, except for the cmdlets mentioned below:

• New-Team
• [Get|Set|New|Sync]-CsOnlineApplicationInstance
• *-CsUserCallingSettings
• *-CsUserCallingDelegate
• *PolicyPackage*
• *-CsTeamsShiftsConnection*
• *-CsBatchTeamsDeployment*

  

The Wrap Up

 

Congratulations, you’re now an expert at certificate-based authentication with the Teams PowerShell module. If you are interested in Token Based Authentication using the Teams Module, then you can check out my post on that over here: Token Auth Post.

Read more →