Microsoft eCDN Deep Dive

Microsoft Teams Town Hall events can support up to 20,000 participants with Teams Premium, and probably more than this in the future. If the majority of those participants are at sites within your organization, you could easily see how individual unicast streams of 1.5Mbps per participant will generate a very large amount of traffic. This could create a perfect storm for a denial of service on your organization's Internet connection...

This is where eCDN (Enterprise Content Delivery Network) solutions come in. If you are unfamiliar with eCDN, you might envision deploying on-premises servers to manage traffic distribution to on-site participants. However, eCDN offers an innovative approach by utilizing peer-to-peer networking among clients, eliminating the need for local infrastructure. This is achieved through a sophisticated software layer that orchestrates streaming from the cloud to a select number of participants. These participant machines then act as hubs, efficiently streaming content to nearby clients via peer-to-peer streaming of media.

Several eCDN providers are available in the market, including Hive, Kollective, and Ramp. Following Microsoft’s acquisition of Peer5 in 2021, the company now offers its own in-house eCDN service, known as Microsoft eCDN. This service currently supports the following streaming video platforms:

• Teams Town Hall
• Viva Engage
• Teams Live Events (now deprecated)

 

What Does eCDN do?

eCDN optimizes the number of streams required from the cloud to individual participants in large meetings. It achieves this by initially sending the stream from the cloud to a subset of participants. Other participants on the same network segment as those already receiving the stream can then access it directly through peer-to-peer connectivity.

The diagram below illustrates a traditional unicast model in the “Before” image and an eCDN solution that reduces the three streams from the cloud to a single stream in the “After” image:

In practice, the Microsoft eCDN solution can support one cloud streaming peer servicing up to 30 other peer-to-peer participants. This configuration can potentially achieve a 30:1 reduction in bandwidth usage. However, this optimal scenario may not always be realized due to participants being distributed across various internal networks and physical locations. The comparison between the bandwidth used in a unicast model and that used with the eCDN model is referred to as the peer-to-peer efficiency of the solution. Below is an example illustrating how to calculate the efficiency for a specific case:

An example from Microsoft:

A 2 Mbps stream with 1,000 concurrent viewers would effectively require a 2 Gbps connection to accommodate that stream. If Microsoft eCDN provided, for example, a 90% P2P efficiency, only 10% of the internal bandwidth, that is, 200 Mbps, would be required.

 

How Does eCDN do it?

The Teams application and web app incorporate an additional software layer atop the media stack. This client software layer collaborates with Microsoft’s eCDN cloud services to orchestrate media flows, thereby minimizing the number of clients that need to retrieve the stream from the Internet. Within the Microsoft cloud, the server-side components of the eCDN service include the following elements:

• Peering discovery service: Responsible for peer discovery.
• Switchboard: Responsible for creating the initial P2P connections between viewers.
• Data pipeline: Consumes all service telemetry and stores it in a data warehouse for analytics consumption.

  

Within the Teams application there are the following components that talk to the eCDN services and relay media to local peers:

• Player Plugin: Responsible for intercepting and forwarding video-related requests to the Client SDK.
• Client SDK: Responsible for intelligently requesting video resources from HTTP / P2P and stitching the data buffers in real-time.
• The Client SDK connects with the backend (Peering discovery service, Switchboard, Data pipeline).
• The discovery service sends the Client SDK a set of peers that it believes will benefit this particular viewer. Peers are selected based on network proximity, cache allocation, stream relevance, among other parameters.
• The Client SDK establishes WebRTC data channel connections with the specified set of peers with the help of the Switchboard.
• HTTP requests that are generated by the video player are intercepted by the Player Plugin and forwarded to the Microsoft eCDN Client SDK, which decides, based on real-time measurements, whether to fetch the desired resource from the P2P network or from HTTP or from both concurrently in order to provide that resource back to the player in the most efficient and timely manner.
• The manifest requests, DRM license and encryption are always retrieved from the HTTP edge server in order to get the most current copy and to adhere to authorization mechanisms.
• Independently, the Client SDK requests authorization to create peer connections from the Microsoft eCDN backend. Once authorized, the Client SDK begins to download resources from HTTP and P2P.

 

A high-level diagram of these services look like this:

What are the Management Capabilities of Microsoft eCDN?

In addition to the bandwidth advantages offered by the Microsoft eCDN service, an analytics portal is also provided. The analytics portal gives you access to deep insights into each of the Townhall meetings that take place on your network.  

The most critical metric within the portal is the Peer-to-Peer (P2P) efficiency of the network. A higher P2P efficiency percentage indicates that more participants are receiving their media via local Peer-to-Peer connections rather than directly from the Internet.

Note: I have added the Microsoft eCDN portal to the latest version of Portals for Office 365 software for your administrating pleasure (available shortly). Portals for O365

In addition to the high-level aggregated view, detailed drilldowns are available for each meeting, providing information on a per-user basis: 

 

Below is an example of four separate Teams clients operating with the Portals for Office 365 application (Portals for O365). A notable feature of the eCDN plugin integrated into Teams is the ability to access a debug screen, which provides detailed information about the ongoing processes. By opening the debug screen (Alt+Shift+P) on each of these windows, the Teams client in the bottom right corner is the only one receiving its stream directly from the Internet (indicated by the blue graph). The other three clients are receiving their streams from the bottom right client (indicated by the orange graphs).

Zooming in on the stats, you can see the extensive information that's available to you to troubleshoot what is going on with any specific client:

The statistics in the client you can see that the following information is provided:

FPS (Actual/Video): Frames per second of the received video content.

Frames (Drop/Total): Frames that have been dropped and total frames received.

HTTP Speed: Indicating the data speed coming from the cloud to the client.

Peer to Peer mode: Indicating that eCDN is enabled for peering mode.

Peer to Peer Speed: The amount of data per second being downloaded from a local peer.

Peers: The number of peers that are peering off a local client.

Total HTTP Downloaded: The amount of data that this client downloaded from the cloud (this may occur for a while at the start of a call while the peering information is being learned and negotiated.)

Total P2P Downloaded: The amount of data that this client downloaded from a local peer

Total P2P: Percentage between media streamed from the cloud verses from a local peer.

Peer to Peer Speed: The current streaming speed per second that’s coming from a local peer to this client.

Local IP: This allows you to see that the client was able to see the local machines IP to determine which location it’s in. If the client is having issues with WebRTC Browser Disable mDNS, this value will show up as N/A. This is a good troubleshooting indicator. See the WebRTC Browser Disable mDNS section for more details.

Group ID: The group that the client was associated with based on Subnet details.

 

What Are The Prerequisites to Use Microsoft eCDN?

To utilize Microsoft eCDN, appropriate licensing is required. The eCDN license can be purchased separately as an add-on. According to Microsoft, a license is necessary for each benefiting user, defined as a user who participates in an eCDN-enabled event at least once per year.

Note: Above pricing is USD.

The eCDN license is also included as part of the Microsoft Teams Premium license. The licensing mechanism operates slightly differently than one might expect. Instead of assigning licenses to specific users, the number of Teams Premium licenses within the tenancy offsets the number of eCDN licenses required for a meeting. For instance, if you have 2,000 users needing eCDN functionality and already possess 500 Teams Premium licenses, you would need to purchase an additional 1,500 supplemental Microsoft eCDN licenses. (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#what-if-i-have-teams-premium-licenses)

Here’s a useful table provided by Microsoft to describe other scenarios for licensing eCDN (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#example-license-scenarios):

Scenario	Calculation	Number of licenses
A Technology company with 10,000 users with 10% of those users working from home permanently	9,000 licenses for 9,000 benefiting users	9,000
A grocery store chain with 200,000 users, 120,000 of which work in stores and never watch live events	80,000 licenses for the 80,000 corporate users/benefiting users	80,000
A telecommunication company with 50,000 users that plans to buy 30,000 Teams Premium seats	30,000 Teams Premium licenses plus 20,000 standalone Microsoft eCDN licenses	50,000
A pharmaceutical company with 70,000 users spread across three regions including North America (50,000), EMEA (10,000), APAC (10,000). IT wants to enable Microsoft eCDN in North America	50,000 licenses for the 50,000 benefiting users in North America.   Note: Microsoft eCDN needs to be disabled for EMEA & APAC via subnet mapping.	50,000

 

What Do You Need To Configure For eCDN?

To turn on eCDN you simply need to go to the Teams Admin Centre > Meetings > Live Event Settings > Video Distribution Provider > ON, and select Microsoft eCDN as the provider. In order to be able to select Microsoft you will need to have some Premium licences in your tenancy (Message: “You need Teams Premium or a Microsoft eCDN license to use some settings. Upgrade to a supporting license in the Microsoft 365 admin center. Learn more about Teams Premium”).

Once you have turned on the Microsoft eCDN you will then have access to the eCDN Portal:

https://admin.ecdn.teams.microsoft.com/

Subnet mapping

In the eCDN Portal, the primary configuration task is to set up your subnets, enabling the software to accurately determine the clients’ locations within your network environment.

This configuration allows you to specify how clients in particular subnets should behave and be grouped. Grouping subnets enables media peering between them. Whether a subnet permits media peering depends on the P2P (Peer-to-Peer) setting assigned to it. Additionally, mapping subnets facilitates per-site analytics within the eCDN portal.

The P2P setting can be ON, OFF, or LEECH. The ON and OFF settings are straightforward, indicating whether peers in these subnets will allow or disallow peering. The LEECH setting means that the subnet can only receive data from another subnet within its group that has peering enabled.

Another more complex setting is WAN. This setting allows clients to use the STUN protocol to discover their public addresses, which are then published to the eCDN as potential peers. This is an optional setting, applicable to specific use cases.

Please note that it is suboptimal for VPN users to participate in peering. It is recommended to disable peering for VPN users.

Below is a table provided by Microsoft for these settings:

Column name	Data type	Description	Guidance
Group Id	String	Your site name is displayed in the analytics. Only peers with the same Group ID can peer with each other.	Required
Subnets	CIDR	Space or comma (,) delimited network specifiers in the CIDR format (e.g. 10.0.0.0/24).	Required
P2P	Enum	Configure how users who fall under this group behave. Possible values: ·        On - Peering is enabled. ·        Off - Peering is disabled. ·        Leech - Peering is enabled in consume-only mode. Note: When formatting the CSV this entry can be in the format “p2p-on” or it can just be “on”. When you export the CSV from the system it will use the “p2p-on” formatting of this config item.	Required
WAN	Enum	Enabling WAN toggles on the STUN protocol, which allows us to create peering connections between internet users. For example, in regions with poor internet connectivity but good local bandwidth it may be better to peer between sites over the internet than sourcing directly from the cloud. Possible values: ·        On - Peering via public IP is enabled. ·        (default) Off - Peering via public IP is disabled. Note: When formatting the CSV this entry can be in the format “wan-on” or it can just be “on”. When you export the CSV from the system it will use the “wan-on” formatting of this config item.	Optional
Label	String	Labels help you identify sites more precisely. For example, you have a Group ID called "Melbourne" and within that group there's a subnet that's a WiFi network, so you could label it "WiFi-Only."	Recommended
Country	String	Manually override the country/region as determined based on the viewer's public IP address. This option is useful for viewers whose internet breakout point doesn't represent their true location.	Optional
City	String	Same behaviour and guidance as Country column.	Optional

 

Default Group Mappings:

Criteria matrix	Subnet mapping present		No subnet mapping
	Group name	p2p config	Group name	p2p config
No IP / Failed IP	Ungrouped	off	Ungrouped	off
IP supplied (no match)	Default Group	off	all	on

 

One of two significant groups could apply to your viewers even when a subnet mapping is present.

·        Ungrouped - Viewers are assigned to this P2P-off group when their eCDN client does not receive a valid IP address from the operating system. This typically occurs due to mDNS issues with the browser. For more details, please refer to the mDNS section of this post.

·        Default Group - This P2P-off group is applied to viewers who were not accounted for in the subnet mapping at the time of the event.

·        all - This group is applied when subnet mapping is not present. It includes all viewers whose eCDN client received a valid IP address from the operating system. These viewers are set to P2P-on.

 

Below is an example of the CSV file format for importing Subnet information:

GroupId,Subnets,P2P,WAN,Label,Country,City

Site A,10.0.0.0/24,p2p-on,wan-on,Melb Office,AU,Melbourne

Site B,10.1.0.0/24,p2p-on,wan-on,Syd Office,AU,Sydney

Note: You can have a maximum of 50,000 subnet entries.

 

WebRTC Browser Disable mDNS

The Microsoft documentation outlines the requirement to disable mDNS in browsers using registry keys. The initial question that arises is: what is mDNS in this context? It turns out that with WebRTC frameworks in the browser, it was possible to run a background check to discover a machine’s private IP address. This capability could potentially be exploited for monitoring or hacking purposes. Consequently, browser vendors implemented a feature that prevents JavaScript from accessing the private IP address of a machine. This is not desirable in this case, as a critical component of how eCDN functions is its need to access private IP address information to determine subnet mapping.

Upon further investigation, it appears that if users have granted microphone and camera access to the browser during the meeting setup, the mDNS IP address hiding functionality will be disabled by default. Therefore, if users enable their microphone and camera, they will be able to utilize the eCDN functionality without any issues. Below are examples of optimal settings that allow camera and microphone access:

If you run into a situation where clients are not getting linked to their group due to their IP Addresses being blocked then you can use Windows Registry settings that are specific for the browser being used: https://learn.microsoft.com/en-us/ecdn/how-to/disable-mdns

Networking

You will need to ensure that the Teams client have access to the following cloud-based web services to orchestrate this feature:

External Network Connections

Microsoft eCDN setup has some networking requirements for connecting to the eCDN services in Microsoft’s cloud:

1.      When a user browses to the event page, the client needs to download the Microsoft eCDN script - that requires an HTTPS connection to *.ecdn.teams.microsoft.com.

2.      It then creates a secure WebSocket connection to Microsoft’s eCDN backend.

3.      The peer-to-peer connection itself is a UDP connection over the port range 1025-65535, chosen randomly by the browser.

 

External Network Connections  

Hostname	Ports	Protocol	Description
*.ecdn.teams.microsoft.com	443	HTTPS over TCP	Microsoft eCDN scripts
*.ecdn.teams.microsoft.com	443	WebSocket over TCP	Microsoft eCDN backend

Internal Network Connections  

Connections that remain inside the corporate network. Usually packets in these connections don't go through a firewall and wouldn't need any configuration to allow them.

 

Hostname	Ports	Protocol	Description
n/a	1025-65535	SCTP over DTLS over UDP	P2P communication

 

Limit The Media Port Range in Browsers

The Media port range can be limited for both Microsoft Edge and Google Chrome browsers. Note: The Teams client requires ports 1025-65535 to be allowed on internal networks between peers.

·        https://chromeenterprise.google/policies/##WebRtcUdpPortRange

·        https://admx.help/?Category=EdgeChromium&Policy=Microsoft.Policies.Edge::WebRtcUdpPortRange

 

Limiting Access to the Service

To restrict eCDN access, you must specify the external public IP addresses from which clients will be accessing the Internet (i.e. the external NATed address). You do this by specifying Public IP Addresses under the Configuration > Security section. If no IPs are specified, then all users are allowed to connect from all public IPs.

Supported input formats:

•  Single IP - A single 4 octet IP. Example: 13.107.231.100
•  CIDR - A standard Classless inter-domain routing (CIDR) consisted of a start address and a subnet mask. Example: 13.107.231.0/24 (equivalent to: 13.107.231.0 - 13.107.231.255)
• JSON Format - A custom JSON format which matches the export format. Example: [ "13.107.231.0/24", "51.12.41.7/32" ]

Limitations

• IPv6 is not support at this time
• Up to 256 ranges

 

Testing eCDN

The Microsoft eCDN also provides the capability to conduct network tests to evaluate performance at scale before hosting a significant meeting on the platform. The silent test operates by running a script in the background on selected users’ machines, initiating a muted video session on the end-user’s device. This allows users to continue their work uninterrupted and unaware of the test. However, as the test imposes actual load on your networks, it should be used carefully.

To keep this post from getting too long, I will not go into all the details of setting up silent testing. If you want to know more you can find the docs here: https://learn.microsoft.com/en-us/ecdn/technical-documentation/silent-testing-framework

 

The Wrap Up

Apart from the Microsoft documentation, there is limited information available on the Microsoft eCDN service. This prompted me to conduct an in-depth exploration of the service. It offers clear advantages for organizations planning large internal Townhall meetings. Additionally, eCDN is a valuable component included with Teams Premium, especially if it is widely deployed within your organization.

For full eCDN documentation is available here for your reference: https://learn.microsoft.com/en-us/ecdn/intro

Read more →