Microsoft eCDN Deep Dive

Microsoft Teams Town Hall events can support up to 20,000 participants with Teams Premium, and probably more than this in the future. If the majority of those participants are at sites within your organization, you could easily see how individual unicast streams of 1.5Mbps per participant will generate a very large amount of traffic. This could create a perfect storm for a denial of service on your organization's Internet connection...

This is where eCDN (Enterprise Content Delivery Network) solutions come in. If you are unfamiliar with eCDN, you might envision deploying on-premises servers to manage traffic distribution to on-site participants. However, eCDN offers an innovative approach by utilizing peer-to-peer networking among clients, eliminating the need for local infrastructure. This is achieved through a sophisticated software layer that orchestrates streaming from the cloud to a select number of participants. These participant machines then act as hubs, efficiently streaming content to nearby clients via peer-to-peer streaming of media.

Several eCDN providers are available in the market, including Hive, Kollective, and Ramp. Following Microsoft’s acquisition of Peer5 in 2021, the company now offers its own in-house eCDN service, known as Microsoft eCDN. This service currently supports the following streaming video platforms:

• Teams Town Hall
• Viva Engage
• Teams Live Events (now deprecated)

 

What Does eCDN do?

eCDN optimizes the number of streams required from the cloud to individual participants in large meetings. It achieves this by initially sending the stream from the cloud to a subset of participants. Other participants on the same network segment as those already receiving the stream can then access it directly through peer-to-peer connectivity.

The diagram below illustrates a traditional unicast model in the “Before” image and an eCDN solution that reduces the three streams from the cloud to a single stream in the “After” image:

In practice, the Microsoft eCDN solution can support one cloud streaming peer servicing up to 30 other peer-to-peer participants. This configuration can potentially achieve a 30:1 reduction in bandwidth usage. However, this optimal scenario may not always be realized due to participants being distributed across various internal networks and physical locations. The comparison between the bandwidth used in a unicast model and that used with the eCDN model is referred to as the peer-to-peer efficiency of the solution. Below is an example illustrating how to calculate the efficiency for a specific case:

An example from Microsoft:

A 2 Mbps stream with 1,000 concurrent viewers would effectively require a 2 Gbps connection to accommodate that stream. If Microsoft eCDN provided, for example, a 90% P2P efficiency, only 10% of the internal bandwidth, that is, 200 Mbps, would be required.

 

How Does eCDN do it?

The Teams application and web app incorporate an additional software layer atop the media stack. This client software layer collaborates with Microsoft’s eCDN cloud services to orchestrate media flows, thereby minimizing the number of clients that need to retrieve the stream from the Internet. Within the Microsoft cloud, the server-side components of the eCDN service include the following elements:

• Peering discovery service: Responsible for peer discovery.
• Switchboard: Responsible for creating the initial P2P connections between viewers.
• Data pipeline: Consumes all service telemetry and stores it in a data warehouse for analytics consumption.

  

Within the Teams application there are the following components that talk to the eCDN services and relay media to local peers:

• Player Plugin: Responsible for intercepting and forwarding video-related requests to the Client SDK.
• Client SDK: Responsible for intelligently requesting video resources from HTTP / P2P and stitching the data buffers in real-time.
• The Client SDK connects with the backend (Peering discovery service, Switchboard, Data pipeline).
• The discovery service sends the Client SDK a set of peers that it believes will benefit this particular viewer. Peers are selected based on network proximity, cache allocation, stream relevance, among other parameters.
• The Client SDK establishes WebRTC data channel connections with the specified set of peers with the help of the Switchboard.
• HTTP requests that are generated by the video player are intercepted by the Player Plugin and forwarded to the Microsoft eCDN Client SDK, which decides, based on real-time measurements, whether to fetch the desired resource from the P2P network or from HTTP or from both concurrently in order to provide that resource back to the player in the most efficient and timely manner.
• The manifest requests, DRM license and encryption are always retrieved from the HTTP edge server in order to get the most current copy and to adhere to authorization mechanisms.
• Independently, the Client SDK requests authorization to create peer connections from the Microsoft eCDN backend. Once authorized, the Client SDK begins to download resources from HTTP and P2P.

 

A high-level diagram of these services look like this:

What are the Management Capabilities of Microsoft eCDN?

In addition to the bandwidth advantages offered by the Microsoft eCDN service, an analytics portal is also provided. The analytics portal gives you access to deep insights into each of the Townhall meetings that take place on your network.  

The most critical metric within the portal is the Peer-to-Peer (P2P) efficiency of the network. A higher P2P efficiency percentage indicates that more participants are receiving their media via local Peer-to-Peer connections rather than directly from the Internet.

Note: I have added the Microsoft eCDN portal to the latest version of Portals for Office 365 software for your administrating pleasure (available shortly). Portals for O365

In addition to the high-level aggregated view, detailed drilldowns are available for each meeting, providing information on a per-user basis: 

 

Below is an example of four separate Teams clients operating with the Portals for Office 365 application (Portals for O365). A notable feature of the eCDN plugin integrated into Teams is the ability to access a debug screen, which provides detailed information about the ongoing processes. By opening the debug screen (Alt+Shift+P) on each of these windows, the Teams client in the bottom right corner is the only one receiving its stream directly from the Internet (indicated by the blue graph). The other three clients are receiving their streams from the bottom right client (indicated by the orange graphs).

Zooming in on the stats, you can see the extensive information that's available to you to troubleshoot what is going on with any specific client:

The statistics in the client you can see that the following information is provided:

FPS (Actual/Video): Frames per second of the received video content.

Frames (Drop/Total): Frames that have been dropped and total frames received.

HTTP Speed: Indicating the data speed coming from the cloud to the client.

Peer to Peer mode: Indicating that eCDN is enabled for peering mode.

Peer to Peer Speed: The amount of data per second being downloaded from a local peer.

Peers: The number of peers that are peering off a local client.

Total HTTP Downloaded: The amount of data that this client downloaded from the cloud (this may occur for a while at the start of a call while the peering information is being learned and negotiated.)

Total P2P Downloaded: The amount of data that this client downloaded from a local peer

Total P2P: Percentage between media streamed from the cloud verses from a local peer.

Peer to Peer Speed: The current streaming speed per second that’s coming from a local peer to this client.

Local IP: This allows you to see that the client was able to see the local machines IP to determine which location it’s in. If the client is having issues with WebRTC Browser Disable mDNS, this value will show up as N/A. This is a good troubleshooting indicator. See the WebRTC Browser Disable mDNS section for more details.

Group ID: The group that the client was associated with based on Subnet details.

 

What Are The Prerequisites to Use Microsoft eCDN?

To utilize Microsoft eCDN, appropriate licensing is required. The eCDN license can be purchased separately as an add-on. According to Microsoft, a license is necessary for each benefiting user, defined as a user who participates in an eCDN-enabled event at least once per year.

Note: Above pricing is USD.

The eCDN license is also included as part of the Microsoft Teams Premium license. The licensing mechanism operates slightly differently than one might expect. Instead of assigning licenses to specific users, the number of Teams Premium licenses within the tenancy offsets the number of eCDN licenses required for a meeting. For instance, if you have 2,000 users needing eCDN functionality and already possess 500 Teams Premium licenses, you would need to purchase an additional 1,500 supplemental Microsoft eCDN licenses. (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#what-if-i-have-teams-premium-licenses)

Here’s a useful table provided by Microsoft to describe other scenarios for licensing eCDN (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#example-license-scenarios):

Scenario	Calculation	Number of licenses
A Technology company with 10,000 users with 10% of those users working from home permanently	9,000 licenses for 9,000 benefiting users	9,000
A grocery store chain with 200,000 users, 120,000 of which work in stores and never watch live events	80,000 licenses for the 80,000 corporate users/benefiting users	80,000
A telecommunication company with 50,000 users that plans to buy 30,000 Teams Premium seats	30,000 Teams Premium licenses plus 20,000 standalone Microsoft eCDN licenses	50,000
A pharmaceutical company with 70,000 users spread across three regions including North America (50,000), EMEA (10,000), APAC (10,000). IT wants to enable Microsoft eCDN in North America	50,000 licenses for the 50,000 benefiting users in North America.   Note: Microsoft eCDN needs to be disabled for EMEA & APAC via subnet mapping.	50,000

 

What Do You Need To Configure For eCDN?

To turn on eCDN you simply need to go to the Teams Admin Centre > Meetings > Live Event Settings > Video Distribution Provider > ON, and select Microsoft eCDN as the provider. In order to be able to select Microsoft you will need to have some Premium licences in your tenancy (Message: “You need Teams Premium or a Microsoft eCDN license to use some settings. Upgrade to a supporting license in the Microsoft 365 admin center. Learn more about Teams Premium”).

Once you have turned on the Microsoft eCDN you will then have access to the eCDN Portal:

https://admin.ecdn.teams.microsoft.com/

Subnet mapping

In the eCDN Portal, the primary configuration task is to set up your subnets, enabling the software to accurately determine the clients’ locations within your network environment.

This configuration allows you to specify how clients in particular subnets should behave and be grouped. Grouping subnets enables media peering between them. Whether a subnet permits media peering depends on the P2P (Peer-to-Peer) setting assigned to it. Additionally, mapping subnets facilitates per-site analytics within the eCDN portal.

The P2P setting can be ON, OFF, or LEECH. The ON and OFF settings are straightforward, indicating whether peers in these subnets will allow or disallow peering. The LEECH setting means that the subnet can only receive data from another subnet within its group that has peering enabled.

Another more complex setting is WAN. This setting allows clients to use the STUN protocol to discover their public addresses, which are then published to the eCDN as potential peers. This is an optional setting, applicable to specific use cases.

Please note that it is suboptimal for VPN users to participate in peering. It is recommended to disable peering for VPN users.

Below is a table provided by Microsoft for these settings:

Column name	Data type	Description	Guidance
Group Id	String	Your site name is displayed in the analytics. Only peers with the same Group ID can peer with each other.	Required
Subnets	CIDR	Space or comma (,) delimited network specifiers in the CIDR format (e.g. 10.0.0.0/24).	Required
P2P	Enum	Configure how users who fall under this group behave. Possible values: ·        On - Peering is enabled. ·        Off - Peering is disabled. ·        Leech - Peering is enabled in consume-only mode. Note: When formatting the CSV this entry can be in the format “p2p-on” or it can just be “on”. When you export the CSV from the system it will use the “p2p-on” formatting of this config item.	Required
WAN	Enum	Enabling WAN toggles on the STUN protocol, which allows us to create peering connections between internet users. For example, in regions with poor internet connectivity but good local bandwidth it may be better to peer between sites over the internet than sourcing directly from the cloud. Possible values: ·        On - Peering via public IP is enabled. ·        (default) Off - Peering via public IP is disabled. Note: When formatting the CSV this entry can be in the format “wan-on” or it can just be “on”. When you export the CSV from the system it will use the “wan-on” formatting of this config item.	Optional
Label	String	Labels help you identify sites more precisely. For example, you have a Group ID called "Melbourne" and within that group there's a subnet that's a WiFi network, so you could label it "WiFi-Only."	Recommended
Country	String	Manually override the country/region as determined based on the viewer's public IP address. This option is useful for viewers whose internet breakout point doesn't represent their true location.	Optional
City	String	Same behaviour and guidance as Country column.	Optional

 

Default Group Mappings:

Criteria matrix	Subnet mapping present		No subnet mapping
	Group name	p2p config	Group name	p2p config
No IP / Failed IP	Ungrouped	off	Ungrouped	off
IP supplied (no match)	Default Group	off	all	on

 

One of two significant groups could apply to your viewers even when a subnet mapping is present.

·        Ungrouped - Viewers are assigned to this P2P-off group when their eCDN client does not receive a valid IP address from the operating system. This typically occurs due to mDNS issues with the browser. For more details, please refer to the mDNS section of this post.

·        Default Group - This P2P-off group is applied to viewers who were not accounted for in the subnet mapping at the time of the event.

·        all - This group is applied when subnet mapping is not present. It includes all viewers whose eCDN client received a valid IP address from the operating system. These viewers are set to P2P-on.

 

Below is an example of the CSV file format for importing Subnet information:

GroupId,Subnets,P2P,WAN,Label,Country,City

Site A,10.0.0.0/24,p2p-on,wan-on,Melb Office,AU,Melbourne

Site B,10.1.0.0/24,p2p-on,wan-on,Syd Office,AU,Sydney

Note: You can have a maximum of 50,000 subnet entries.

 

WebRTC Browser Disable mDNS

The Microsoft documentation outlines the requirement to disable mDNS in browsers using registry keys. The initial question that arises is: what is mDNS in this context? It turns out that with WebRTC frameworks in the browser, it was possible to run a background check to discover a machine’s private IP address. This capability could potentially be exploited for monitoring or hacking purposes. Consequently, browser vendors implemented a feature that prevents JavaScript from accessing the private IP address of a machine. This is not desirable in this case, as a critical component of how eCDN functions is its need to access private IP address information to determine subnet mapping.

Upon further investigation, it appears that if users have granted microphone and camera access to the browser during the meeting setup, the mDNS IP address hiding functionality will be disabled by default. Therefore, if users enable their microphone and camera, they will be able to utilize the eCDN functionality without any issues. Below are examples of optimal settings that allow camera and microphone access:

If you run into a situation where clients are not getting linked to their group due to their IP Addresses being blocked then you can use Windows Registry settings that are specific for the browser being used: https://learn.microsoft.com/en-us/ecdn/how-to/disable-mdns

Networking

You will need to ensure that the Teams client have access to the following cloud-based web services to orchestrate this feature:

External Network Connections

Microsoft eCDN setup has some networking requirements for connecting to the eCDN services in Microsoft’s cloud:

1.      When a user browses to the event page, the client needs to download the Microsoft eCDN script - that requires an HTTPS connection to *.ecdn.teams.microsoft.com.

2.      It then creates a secure WebSocket connection to Microsoft’s eCDN backend.

3.      The peer-to-peer connection itself is a UDP connection over the port range 1025-65535, chosen randomly by the browser.

 

External Network Connections  

Hostname	Ports	Protocol	Description
*.ecdn.teams.microsoft.com	443	HTTPS over TCP	Microsoft eCDN scripts
*.ecdn.teams.microsoft.com	443	WebSocket over TCP	Microsoft eCDN backend

Internal Network Connections  

Connections that remain inside the corporate network. Usually packets in these connections don't go through a firewall and wouldn't need any configuration to allow them.

 

Hostname	Ports	Protocol	Description
n/a	1025-65535	SCTP over DTLS over UDP	P2P communication

 

Limit The Media Port Range in Browsers

The Media port range can be limited for both Microsoft Edge and Google Chrome browsers. Note: The Teams client requires ports 1025-65535 to be allowed on internal networks between peers.

·        https://chromeenterprise.google/policies/##WebRtcUdpPortRange

·        https://admx.help/?Category=EdgeChromium&Policy=Microsoft.Policies.Edge::WebRtcUdpPortRange

 

Limiting Access to the Service

To restrict eCDN access, you must specify the external public IP addresses from which clients will be accessing the Internet (i.e. the external NATed address). You do this by specifying Public IP Addresses under the Configuration > Security section. If no IPs are specified, then all users are allowed to connect from all public IPs.

Supported input formats:

•  Single IP - A single 4 octet IP. Example: 13.107.231.100
•  CIDR - A standard Classless inter-domain routing (CIDR) consisted of a start address and a subnet mask. Example: 13.107.231.0/24 (equivalent to: 13.107.231.0 - 13.107.231.255)
• JSON Format - A custom JSON format which matches the export format. Example: [ "13.107.231.0/24", "51.12.41.7/32" ]

Limitations

• IPv6 is not support at this time
• Up to 256 ranges

 

Testing eCDN

The Microsoft eCDN also provides the capability to conduct network tests to evaluate performance at scale before hosting a significant meeting on the platform. The silent test operates by running a script in the background on selected users’ machines, initiating a muted video session on the end-user’s device. This allows users to continue their work uninterrupted and unaware of the test. However, as the test imposes actual load on your networks, it should be used carefully.

To keep this post from getting too long, I will not go into all the details of setting up silent testing. If you want to know more you can find the docs here: https://learn.microsoft.com/en-us/ecdn/technical-documentation/silent-testing-framework

 

The Wrap Up

Apart from the Microsoft documentation, there is limited information available on the Microsoft eCDN service. This prompted me to conduct an in-depth exploration of the service. It offers clear advantages for organizations planning large internal Townhall meetings. Additionally, eCDN is a valuable component included with Teams Premium, especially if it is widely deployed within your organization.

For full eCDN documentation is available here for your reference: https://learn.microsoft.com/en-us/ecdn/intro

Read more →

Bulk Sign in for Microsoft Teams SIP Gateway

In 2024 we have received a new feature for Standard SIP phones on Teams which has had no fanfare whatsoever. Despite the lack of fanfare, this enhancement promises to alleviate some of the challenges associated with deploying a large number of devices on the Teams SIP Gateway. Microsoft’s official documentation doesn’t provide an exhaustive explanation of this feature, which may leave you scratching your head about exactly what it does and how it works. In this article, I’ll delve into the details, shedding light on what this feature entails. Additionally, I’ll introduce a practical tool that streamlines bulk sign-ins, making the deployment process even easier.

What is Bulk Sign-in anyway?

If you’ve ever connected a SIP Phone to Teams SIP Gateway you will know that the phone has two states:

The Signed out state: This is when your phone has been configured to connect to the provisioning servers within Microsoft’s infrastructure, but has not yet been signed into an account. In this state you’ll usually see something like this on the phone interface:

The Signed in state: In the signed in state you will see the phone number of the user on the screen and there will be whatever buttons are supported by the device. Here’s an example:

 

To transition Standard SIP phones between these states you usually have to do a sign in dance that involves a web sign-in flow, where you have to know the individual passwords for every phone account. This is both tedious and, in a lot of cases, impractical.

Are you ready to hear the cool part? What if I was to tell you that you can now bulk sign in 100 phones at a time without ever entering a password?? Well, that’s what we are talking about with the new Bulk Sign in process.

 

Prerequisites

As you can imagine, if you’re going to allow someone to sign in a bunch of phones without ever entering a password there are going to have to be some guardrails to ensure that people aren’t able to randomly sign phones in. The prerequisites are as follows:

• In addition to Teams Administrator, you must use an account that has the Global Administrator, Privileged Authentication Administrator or the Authentication Administrator role to run the cmdlets (and the Tool I've provided below). Note: You will get an “Access Denied” error when you try to run the bulk sign-in cmdlets.
• You must apply CommonAreaPhone policy to the accounts that are part of a bulk sign-in request.
• The accounts must not have Multi Factor Authentication (MFA) enabled.
• The accounts must have a phone number assigned.
• The accounts must have the SIP device Calling Policy assigned with AllowSIPDevicesCalling set to True.
• The BulkSignIn attribute must be set to Enabled in TeamsSipDevicesConfiguration policy.
• You must have Trusted IP Addresses applied for any external IP Address that Standard SIP phones will be using to access the internet.
• Account settings: The user account that is used for the device must not have User must change password at next login, or User's password can't be changed selected, or have the minimum password age set to a value more than 0.
• The phones provisioning URL must include the Azure Entra Tenant ID at the end. This is an additional “/tenantid/<your-tenant-ID-guid>” that is put at the end of the basic provisioning URL. Note: I have found if you leave this out, the error that you will receive is that the Trusted IP address is not assigned. I believe this is because it is unable to associate the phone to a tenant and by extension, the trusted IP will not be found.
• Phone accounts need a licence that includes Teams Phone capability. In most cases this will be a Microsoft Teams Shared Device Licence, but also can be standard user licensing too.

Like I said, if you expect to sign in devices without passwords you’re going to have to jump through some hoops. Here's the post from Microsoft that you can also check out: https://learn.microsoft.com/en-us/microsoftteams/sip-gateway-configure

 

Prerequisite Configuration Steps

Add the Trusted IP Address and Network Addresses in Teams Admin Centre (Locations > Network Topology):

Network Sites:

It’s not entirely clear if you need this configuration; however, you normally do for other Trusted IP configuration so I’m including it. You enter your Network Region and internal subnets:

Trusted IP Addresses:

Trusted IP Addresses are the external IP Addresses that your organisation uses to access the Internet (i.e. the external NATed IP Address of your network). This specifically needs to be the IP Address (or range) that your SIP Phones will be using to access the Internet. When they connect to the SIP Servers on the Microsoft side, the source IP Address will be noted by the service. When you request a specific Phone MAC Address to be logged in as a user account, the service will confirm that the source IP Address of the known MAC address is coming from a Trusted IP Address range.

Note: It may to 24 hours for these settings to take effect. If they haven’t, you will get an error message when you try to sign in the device that looks something like this: “Public IP IPv4 - <IP>, IPv6 -  for the device is not trusted. Please whitelist the public IP in TAC…”

 

Set the phone number:

Your phones' accounts will all need phone numbers associated with them:

Set-CsPhoneNumberAssignment -Identity CommonAreaPhone@domain.com -PhoneNumber +61399995555 -PhoneNumberType DirectRouting

 

Create a Teams IP Phone Policy for Common Area Devices and Assign it to all phones that need to be signed in:

New-CsTeamsIPPhonePolicy -Identity CommonAreaPhone -SignInMode CommonAreaPhoneSignin

Grant-CsTeamsIPPhonePolicy -Identity CommonAreaPhone@domain.com -PolicyName CommonAreaPhone

 

Create a Teams SIP Devices Configuration:

This is a new commandlet that was created specifically for the Bulk Sign-in feature. You can either assign it as a Global setting or create a User based policy that you assign directly to accounts.

#Global Setting Example

Set-CsTeamsSipDevicesConfiguration -identity Global -BulkSignIn "Enabled"

 

 

Ensure that you have a Teams Calling Policy that allows for SIP Devices to sign-in:

New-CsTeamsCallingPolicy -Identity SIPDevices -AllowSIPDevicesCalling $true -AllowCallRedirect Enabled

Grant-CsTeamsCallingPolicy -Identity CommonAreaPhone@domain.com -PolicyName SIPDevices

 

SIP Phone Provisioning:

When you prepare your SIP Phones for connection to the Teams SIP Gateway, you need to load in a provisioning URL that is slightly different than the default one. By this I mean it needs to include your Entra Tenant ID in the URL.

EMEA: http://emea.ipp.sdg.teams.microsoft.com/tenantid/<your-tenant-ID-guid>

Americas: http://noam.ipp.sdg.teams.microsoft.com/tenantid/<your-tenant-ID-guid>

APAC: http://apac.ipp.sdg.teams.microsoft.com/tenantid/<your-tenant-ID-guid>

Note: When the Tool boots it will print out what your provisioning URLs should be for your tenant. 

Bulk Sign in

Microsoft has given us some PowerShell commandlets to do the Bulk Sign in Process. The commands require that you import a CSV with the account and the MAC address of the device in it. The format for this file is as follows:

Username, HardwareId

CommonAreaPhone1@domain.com,00-04-f2-81-24-b3

CommonAreaPhone2@domain.com,00-04-f2-81-24-b4

 

The file can only contain 100 phones for each import process. So build up your CSV files with 100 devices per file. Once you have your CSV files you need to get them into the system. I found the Microsoft PowerShell here to be a bit clunky to use and I thought that I could improve the experience by building a GUI for it, so that’s what I did. Behold the Teams SIP Gateway Bulk Signin Tool:

Note: In addition to Teams Administrator, you will need to have Global Administrator, Privileged Authentication Administrator or the Authentication Administrator roles assigned to the account you use to sign the tool into PowerShell. If not, you will get an "Access Denied" error.

DOWNLOAD FROM GITHUB

 

Using the tool is dead simple - just click the Browse button and find your CSV file (as described earlier) and open it. When it’s opened, the users will be shown in the tool (the tool will also check the format of the file for you). Now all you need to do is click the “Bulk Sign In” button and the tool will handle the rest. If the phone gets signed in correctly it will change colour to green and if it fails then it will turn red.

The tool will run the New-CsSdgBulkSignInRequest command and Get-CsSdgBulkSignInRequestStatus commands for you and keep track of the batch for you until it completes.

When the tool first loads it will print out a list of the provisioning URLs for you, so you don't have to go searching for your Tenant ID and potentially make any mistakes with the formatting of the URL.

You can also Export a CSV of the results if you would like to keep that for future reference.

Note: When you download PowerShell scripts from the Internet, Microsoft Windows now requires that you open the properties and check the “Unblock” checkbox to allow it to run:

  

Error Messages Provided by Microsoft 365

Error messages that you may run into when signing your Standard SIP Phones into the system:

Error message	Potential solution
User not found in tenant.	Check the username or User Principal Name (UPN) is correct.
User missing phone number assignment.	Verify the user has a phone number assigned.
User missing AllowSIPDevicesCalling policy assignment	Verify that AllowSIPDevicesCalling policy is set to Enabled. See prerequisite 7.
User missing CAP policy assignment.	Verify that the account has CommonAreaPhone policy assigned. See prerequisite 4.
Device not found in records.	Check if the device was correctly provisioned to SIP Gateway, and the region parameter in bulk sign in request is correct.
BulkSignIn Tag missing for the device	Check to see if the device provisioning URL has the correct tenant ID.
Device is offline.	The device can't be found because it's powered off or disconnected from network. Reconnect the device and try it again.
Public IP not configured as Trusted IP.	The tenant ID listed in the provisioning URL isn't correct or the public IP address of the device isn't listed as a trusted IP address in Teams admin center. See prerequisite 1.
Bulk Sign-in deadline expired.	The device hasn't been signed in to within 72 hours of provisioning (or 168 hours).
Duplicate devices found for bulk sign-in.	Verify the MAC addresses you included in the CSV file are correct and there aren't duplicated addresses. IP addresses of the duplicate devices are returned.
Input hardware-ID is of wrong format	Verify the hardware-ID format. See How to create a bulk sign in request.
On-premises AD configuration failure.	Contact your on-premises Active Directory team.
On-premises AD throttling detected	Try it again but with a smaller number of devices in the batch. Depending on network connectivity, large batches will take more time to complete and may get stuck.
The Password writeback service failed to set a password on the tenant's local directory.	The user account that is used for the device must not have User must change password at next login or User's password can't be changed selected, OR have the minimum password age set to a value more than 0. Verify the password options aren't selected and the minimum password age is set to 0 and try again.

 

The Wrap Up

There you have it - all you should need to know about using Bulk Sign-in with Standard SIP phones on the Microsoft Teams Gateway. Enjoy the time savings!

 

Read more →

How to Fix Microsoft Teams Screen Sharing!

In the context of Teams Meetings, a recurring concern has been the perceived jerky video quality when sharing content via screen sharing. However, there’s good news: Microsoft recently rolled out an update to the Teams client in May 2024. This update introduces an optimization specifically designed for video playback during screen sharing.

Typically, screen sharing in Microsoft Teams prioritizes a low frame rate, assuming that users primarily display static slides or non-moving content. Consequently, the default frame rate hovers around 2-3 frames per second, which suffices for most slide presentations.

The new screenshare optimization feature addresses this limitation by dynamically increasing the frame rate during screen sharing. Specifically, it boosts the frame rate to a smooth 30 frames per second, ensuring a more fluid and seamless experience for meeting participants.

This enhancement should significantly improve the video playback quality during screen sharing sessions, enhancing the overall collaboration experience in Teams Meetings.

Here's a video I put together to show the new feature:

Read more →

AudioCodes Microsoft Teams Phone Manual Update

The other day I pulled out an AudioCodes C450HD that hadn’t been plugged in for a while. When I attempted to sign it into Microsoft Teams I was greeted with an error that said the Company Portal software was out of date and needed updating:

 

The phone was recommending that I should update it through the Google Play store. Given that Microsoft Teams phones don’t have the Google Play store on them, I could see that this was going to be a problem. The phone also was unable to be logged into Teams, so I wasn’t going to be able to update it using the Teams service via the regular update method. I figured that a manual process was going to be required.

After doing some searching on the web I came to realise that the manual update process for these phones was not documented anywhere that I could find. I did find a random PDF that talked about a tool called the “Teams Phone Utility” which I hadn’t come across before. Unfortunately, no amount of googling seemed like it was going to allow me to find or download this tool.

After hunting around on the AudioCodes software download site and looking through every folder on there, I was able to find a couple of different versions of firmware for the C450HD phone. These were not named in such a way that I could use to understand which software would be used for the upgrade process. In the search, I also stumbled upon a folder called the “Teams IPP Utility”, that contained a tool called the “Android Phone Tool” which did sound promising.

I downloaded a copy of the tool and after opening it, it looked just like the tool that I had seen in the PDF, it was just named differently. Now it was time to randomly guess how to use the tool. I put in the IP Address of the phone and went with the Username and Password of “admin” and clicked the "SSH Connect" button. Low and behold, it was connected:

 

As mentioned earlier, there was a couple of different types of firmware that I had found on the AudioCodes software site. Some of them were ZIP files and others were IMG files. I noticed that the tool seemed to only accept ZIP or APK file types for upgrade. I went with the ZIP file. I also noticed that there was a couple of different kinds of ZIP files. One named C450HD_AN and one there named C450HD_TEAMS.

If you open the zip file it has the following item inside:

I figured that I had better go with the TEAMS named file as the other ones may be a generic android load. So I downloaded the TEAMS version and selected it as the “Firmware file (zip)” file:

The firmware version was shown in the tool so it appeared that it could read the ZIP file and didn’t immediately fail. Now the moment of truth, I clicked the Submit button. The tool then popped a message saying “Processing the update package. This may take a few minutes”

So I waited. After a few minutes it told me that the process had completed successfully:

Nothing up until this point had happened on the screen of the phone which was a bit disconcerting. After a few more seconds a popup showed up on the screen:

After this the phone took a while and then rebooted and came up with the latest software version! Success!

Note: After upgrading the phone I noticed that there was a setting in newer versions of software for turning on SSH in the Debugging menu of the phone. You will need to turn this on in order for the tool to connect:

 

The Wrap Up

There you have it, simply keep looking and guessing and you too can find the answer to almost every problem. Hopefully this post saves you all the searching and guessing part 😊 Adios!

Read more →

What’s the Difference Between Microsoft Copilot and ChatGPT?

Introduction

In this post I go into some detail of how the different Copilots in Microsoft 365 operate in practice and show that not all the Copilots are created equal. This information could be both useful from a technical perspective but also useful from a staff training perspective. When you rollout Microsoft Copilot, people within the organisation need to understand that all the Copilots within the Office applications are not the same and are all tuned in different ways.

 

A Copilot is a Copilot is a Copilot?

When I first heard about Microsoft Copilot and saw the similar looking Copilot frame on the right side of the screen, I figured it was probably just a common interface that could access the data from the application you had open at the time. However, after actually getting the opportunity of playing with Microsoft Copilot in the various apps it has becomes clear that it is actually a lot more complex than that. Each of the Copilots within the apps has been tailored to respond in a context that makes sense for the type of application that you’re using. This has been achieved by the engineers at Microsoft, using various methods of prompt engineering and orchestration in the background.

I thought it would be useful to demonstrate the differences in the way the various Copilots in different apps respond to the exact same prompt. For this demo I have chosen an innocuous query that is not explicit and could be interpreted in different ways to see what happens. The query I chose was “Tell me about the weather in Melbourne”. This is not the kind of prompt you would really use in practice but is instead something that I’ve chosen to highlight the differences in the way each Copilot responds to the prompt.

Let's start by querying the OpenAI ChatGPT 3.5 model and see how this foundation model interprets this request. This will offer a comparison to see the difference that the exact same prompt will give when asking it of the various Copilots.

 

1. ChatGPT 3.5

You will see here that the ChatGPT foundational model has interpreted this question as a request to know the specific temperature in Melbourne right now. This is because I wasn’t explicit enough in what I had asked the model and so I didn’t get back any general information about expected temperature ranges in Melbourne. 

In setting up the ChatGPT model the OpenAI team appear to have created the system to fail gracefully in these cases where it thinks it's getting asked for data that's more current than it knows about. This is an unfortunate trait of the foundation models, they only know information up to when they were finished being trained. It is interesting that it did not respond with some more generic information about what the expected temperatures are throughout the year or historical information about the weather though (keep this in mind when we get to the Word Copilot example).

 

2. Bing Chat

Bing Chat is geared to behave much more like a web search engine. You can see in the example above that it reached out to the web and pulled back information from various websites about what the current temperature, and upcoming temperatures, will be in Melbourne. It also gave references to websites that it got this information from.

The method used here is called a Retrieval Augmented Generation (RAG) framework, where it doesn't ask the foundation model for the answer to the question directly. Instead, Bing will first retrieve some reputable sources for the kind of information being requested and provide that data as part of the prompt to the foundation model (also often referred to as Grounding the model with data). The foundation model here has been used to interpret the retrieved data instead of using its own “knowledge” from the data it was trained on. In this case, Bing is functioning as an orchestration engine that retrieves data which it compiles into an expanded prompt that will be sent to the ChatGPT mode in addition to your original query.

 

3. M365 Chat

When I asked the M365 Chat interface within Teams this question, it responded that it couldn’t find the answer to the question and recommended that I use a web search. This is because the M365 Chat Copilot uses a similar Retrieval Augmented Generation (RAG) framework to Bing. Rather than searching the Internet for information on the weather in Melbourne, it attempted a Semantic Index search (Reference: https://learn.microsoft.com/en-us/microsoftsearch/semantic-index-for-copilot) across the documents, emails, chats and other data within my Office 365 tenant. I didn’t actually have any information within my tenancy on this topic at the time. As a result, M365 Chat was unable to get any information to the pass onto the foundation model to provide an answer. What is interesting to me here, is that it didn’t just ask the foundation model to have a go at telling me about the weather in Melbourne, but instead apologised for not being able to find any documents about this.

Note: In this case, the Microsoft 365 Chat Copilot was configured to only have access to internal documents and was not enabled for searching the Internet for data. This is a setting that administrators have control over: https://learn.microsoft.com/en-us/microsoft-365-copilot/manage-public-web-access

Of course, had I have had documents that contained information on the weather in Melbourne it would have been able to answer me. Below is an example of the output when there is a document containing information about the weather in Melbourne. You will see here that the RAG model has been used to retrieve the data and the document is referenced below the response:

What is also interesting about the previous response is that this information was actually generated in Word from a later example that I ran for this blog post. The data being displayed here is actually an interpretation of information previously generated by the model. I find this to be an interesting, because when data like this keeps getting recycled through these models over time, will there start to be degradation of the quality of the information? Like a photocopy of a photocopy. Here’s an interesting article that goes into some more detail on what could be the result of this in the long term: (reference: https://cosmosmagazine.com/technology/ai/training-ai-models-on-machine-generated-data-leads-to-model-collapse/). Always take care to check the information the Copilot outputs before using the information.

 

4. Microsoft Word

Microsoft Word is usually used to create longer form documents, as a result, Microsoft has tuned the way the foundation model is prompted when you ask it questions in Word. In the example of asking it about the weather in Melbourne, the model responded with more of a Wikipedia style response, where it attempts to go into depth about what the climate is like in Melbourne throughout the year.

This is a stark difference to the way the ChatGPT foundation model tried to answer this question. This happens by design, as Microsoft realises that this is more likely what you want in a Word document rather than the wanting to know the temperature right now. The way they do this is by taking the original query and then adding additional (“system prompt”) information to it before sending it to the foundation model. This allows them to change the output to be more like what you might want in a Word document. It’s not clear exactly what Microsoft is including in the prompt that it sends to the foundation model, as you never get to see this additional information. If you play around enough with ChatGPT you can see that adding additional text like “provide an extended response similar to a reference encyclopaedia” will cause the model to give outputs more like this. I don’t believe it’s documented anywhere exactly what Microsoft add to the prompts to get these responses as the prompt engineering is a bit of secret sauce.

  

5. PowerPoint

The PowerPoint Copilot is an even more interesting topic as it doesn’t just produce text, it will also add pictures and make design choices when producing its output. You can see that for our example weather query it produced a nice picture of Melbourne’s botanical gardens and skyline, creates a meaningful heading and some dot points about the weather in Melbourne. It looks pretty impressive as an output to such a basic query:

This is all the more impressive when you have some understanding of what’s going on in the background for the PowerPoint Copilot. There is an interesting paper that I found which is produced by some of the research staff at Microsoft about how this works. It can be found here: https://arxiv.org/abs/2306.03460

TLDR: For apps like PowerPoint the Copilot needs to be able to tell the application itself how to style the page in addition to just generating text. This kind of thing can be done with scripting languages which the foundation model could be used to produce (like Github Copilot), however, this method is prone to syntax errors. The researchers at Microsoft found that it was safer to create a specialised domain specific language for describing the layout of a document (more like a declarative language like is used for Terraform or PowerShell Desired State Configuration). The language, in this case, is called Office Domain Specific Language (ODSL) and is designed to use a minimal number of tokens (words) and be easily describable as an input to a foundation model. Here’s an example of the language:

1 # Inserts new "Title and Content" slides after provided ones.

2 slides = insert_slides(precededBy=slides, layout="Title and Content")

When the prompt is sent to the model it will include schema information about what the ODSL language and what the format of the desired response. The model will then respond with a description of what each slide should look like in the desired ODSL format. The response is thoroughly checked and validated to have the right format and then translated into a lower-level language by an interpreter program which then gets executed by PowerPoint. This is both very cool and crazy that the foundation models are powerful enough to do these kinds of things.

 

6. Outlook

When you write an email your colleagues you don’t really want to be known as the person that writes the dreaded War and Peace novel length emails. Fortunately, Microsoft are aware of this and when designing the Outlook Copilot, they took this into account. The output of this Copilot is designed to produce output that looks like, in both format and content, like an email. You can see below that the simple weather in Melbourne prompt actually created what looks and reads like an email. I must admit it did take a bit of artistic licence and go on a bit more of a ramble than I would have liked in this case though:

 

7. Excel

The Excel Copilot is once again quite different than the other Copilots. Asking it the weather is not exactly what it’s supposed to be used for, but I asked it anyway, because, why not?:

In excel, the Copilot is more for creating formulas and reasoning over the data that is in your spreadsheets. In the current preview version, the Copilot will only work on data that is in a defined table. This is likely to do with the fact that the data needs to be ordered in such a way to be sent as a prompt to the foundation model. In doing this the data needs to retain all the column and row information but also keeps the token count low enough to be processed. I’m not sure if it’s clear how Microsoft could process an entire very large spreadsheet (with the potential complexity of multiple pages, and scattered data, etc) through the foundation models give their token limits currently. Until they figure this out, we may be stuck with only processing data that is in defined smaller tables for the time being.

If you are wondering what the Excel Copilot can actually do though, here’s an example of how you could ask the Excel Copilot to reason over the data in a table and give you an answer:

Also, here’s an example of how you can ask the Excel Copilot for a formula for producing a Fahrenheit column from a Celsius column:

 

 

8. Microsoft Whiteboard

The Microsoft Whiteboard Copilot has another take on what it produces based on our modest weather question. It produced a bunch of sticky notes for various things that the weather could be in Melbourne. This is more contextualized toward a brainstorming type of session which is be common when using a Whiteboard:

 

This is once again, a fun and different take on how a foundation model can be used to produce a more context aware output for the application at hand.

 

The Wrap Up

As you can see, all these Copilots across the Microsoft Office apps are all very different beasts, and this is something that people within your organisation should understand in order to get the most out of Copilot product set. This is certainly something to keep in mind when training staff on the potential use cases and determining which Copilot is right for the task at hand. Cheers!

Read more →