Microsoft
Teams Town Hall events can support up to 20,000 participants with Teams
Premium, and probably more than this in the future. If the majority of those
participants are at sites within your organization, you could easily see how
individual unicast streams of 1.5Mbps per participant will generate a very
large amount of traffic. This could create a perfect storm for a denial of
service on your organization's Internet connection...
This is where eCDN (Enterprise Content Delivery Network) solutions come in. If you are unfamiliar with eCDN, you might envision deploying on-premises servers to manage traffic distribution to on-site participants. However, eCDN offers an innovative approach by utilizing peer-to-peer networking among clients, eliminating the need for local infrastructure. This is achieved through a sophisticated software layer that orchestrates streaming from the cloud to a select number of participants. These participant machines then act as hubs, efficiently streaming content to nearby clients via peer-to-peer streaming of media.
Several eCDN providers are available in the market, including Hive, Kollective, and Ramp. Following Microsoft’s acquisition of Peer5 in 2021, the company now offers its own in-house eCDN service, known as Microsoft eCDN. This service currently supports the following streaming video platforms:
- Teams Town Hall
- Viva Engage
- Teams Live Events (now deprecated)
What Does eCDN do?
eCDN optimizes the number of streams required from the cloud to individual participants in large meetings. It achieves this by initially sending the stream from the cloud to a subset of participants. Other participants on the same network segment as those already receiving the stream can then access it directly through peer-to-peer connectivity.
The diagram below illustrates a traditional unicast model in the “Before” image and an eCDN solution that reduces the three streams from the cloud to a single stream in the “After” image:
In practice, the Microsoft eCDN solution can support one cloud streaming peer servicing up to 30 other peer-to-peer participants. This configuration can potentially achieve a 30:1 reduction in bandwidth usage. However, this optimal scenario may not always be realized due to participants being distributed across various internal networks and physical locations. The comparison between the bandwidth used in a unicast model and that used with the eCDN model is referred to as the peer-to-peer efficiency of the solution. Below is an example illustrating how to calculate the efficiency for a specific case:
An
example from Microsoft:
A 2 Mbps
stream with 1,000 concurrent viewers would effectively require a 2 Gbps
connection to accommodate that stream. If Microsoft eCDN provided, for example,
a 90% P2P efficiency, only 10% of the internal bandwidth, that is, 200 Mbps,
would be required.
How Does eCDN do it?
The Teams application and web app incorporate an additional software layer atop the media stack. This client software layer collaborates with Microsoft’s eCDN cloud services to orchestrate media flows, thereby minimizing the number of clients that need to retrieve the stream from the Internet. Within the Microsoft cloud, the server-side components of the eCDN service include the following elements:
- Peering discovery service: Responsible for peer discovery.
- Switchboard: Responsible for creating the initial P2P connections between viewers.
- Data pipeline: Consumes all service telemetry and stores it in a data warehouse for analytics consumption.
Within the Teams application there are the following
components that talk to the eCDN services and relay media to local peers:
- Player Plugin: Responsible for intercepting and forwarding video-related requests to the Client SDK.
- Client SDK: Responsible for intelligently requesting video resources from HTTP / P2P and stitching the data buffers in real-time.
- The Client SDK connects with the backend (Peering discovery service, Switchboard, Data pipeline).
- The discovery service sends the Client SDK a set of peers that it believes will benefit this particular viewer. Peers are selected based on network proximity, cache allocation, stream relevance, among other parameters.
- The Client SDK establishes WebRTC data channel connections with the specified set of peers with the help of the Switchboard.
- HTTP requests that are generated by the video player are intercepted by the Player Plugin and forwarded to the Microsoft eCDN Client SDK, which decides, based on real-time measurements, whether to fetch the desired resource from the P2P network or from HTTP or from both concurrently in order to provide that resource back to the player in the most efficient and timely manner.
- The manifest requests, DRM license and encryption are always retrieved from the HTTP edge server in order to get the most current copy and to adhere to authorization mechanisms.
- Independently, the Client SDK requests authorization to create peer connections from the Microsoft eCDN backend. Once authorized, the Client SDK begins to download resources from HTTP and P2P.
A high-level diagram of these services look like this:
What are the Management
Capabilities of Microsoft eCDN?
In addition to the bandwidth advantages offered by the Microsoft eCDN service, an analytics portal is also provided. The analytics portal gives you access
to deep insights into each of the Townhall meetings that take place on your
network.
The most critical metric within the portal is the Peer-to-Peer (P2P) efficiency of the network. A higher P2P efficiency percentage indicates that more participants are receiving their media via local Peer-to-Peer connections rather than directly from the Internet.
Note: I have added the Microsoft eCDN portal to the latest version of Portals for Office 365 software for your administrating pleasure (available shortly). Portals for O365
In addition to the high-level aggregated view, detailed drilldowns are available for each meeting, providing information on a per-user basis:
Below is an example of four separate Teams clients operating with the Portals for Office 365 application (Portals for O365). A notable feature of the eCDN plugin integrated into Teams is the ability to access a debug screen, which provides detailed information about the ongoing processes. By opening the debug screen (Alt+Shift+P) on each of these windows, the Teams client in the bottom right corner is the only one receiving its stream directly from the Internet (indicated by the blue graph). The other three clients are receiving their streams from the bottom right client (indicated by the orange graphs).
Zooming in on the stats, you can see the extensive information that's available to you to troubleshoot what is going on with any specific client:
The statistics in the client you can see that the following information is provided:
FPS
(Actual/Video): Frames per second of the received video content.
Frames (Drop/Total): Frames that have been dropped and total frames received.
HTTP
Speed: Indicating
the data speed coming from the cloud to the client.
Peer
to Peer mode:
Indicating that eCDN is enabled for peering mode.
Peer
to Peer Speed: The
amount of data per second being downloaded from a local peer.
Peers: The number of peers that are
peering off a local client.
Total
HTTP Downloaded: The
amount of data that this client downloaded from the cloud (this may occur for a
while at the start of a call while the peering information is being learned and
negotiated.)
Total
P2P Downloaded: The
amount of data that this client downloaded from a local peer
Total
P2P: Percentage
between media streamed from the cloud verses from a local peer.
Peer
to Peer Speed: The
current streaming speed per second that’s coming from a local peer to this
client.
Local
IP: This allows you
to see that the client was able to see the local machines IP to determine which
location it’s in. If the client is having issues with WebRTC Browser Disable
mDNS, this value will show up as N/A. This is a good troubleshooting indicator.
See the WebRTC Browser Disable mDNS section for more details.
Group
ID: The group that
the client was associated with based on Subnet details.
What Are The Prerequisites
to Use Microsoft eCDN?
To utilize Microsoft eCDN, appropriate licensing is required. The eCDN license can be purchased separately as an add-on. According to Microsoft, a license is necessary for each benefiting user, defined as a user who participates in an eCDN-enabled event at least once per year.
Note: Above pricing is USD.
The eCDN license is also included as part of the Microsoft Teams Premium license. The licensing mechanism operates slightly differently than one might expect. Instead of assigning licenses to specific users, the number of Teams Premium licenses within the tenancy offsets the number of eCDN licenses required for a meeting. For instance, if you have 2,000 users needing eCDN functionality and already possess 500 Teams Premium licenses, you would need to purchase an additional 1,500 supplemental Microsoft eCDN licenses. (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#what-if-i-have-teams-premium-licenses)
Here’s a useful table provided by Microsoft to describe other scenarios for licensing eCDN (Reference: https://learn.microsoft.com/en-us/ecdn/how-to/purchasing#example-license-scenarios):
Scenario |
Calculation |
Number
of licenses |
A Technology company with 10,000 users with 10% of those
users working from home permanently |
9,000 licenses for 9,000 benefiting users |
9,000 |
A grocery store chain with 200,000 users, 120,000 of
which work in stores and never watch live events |
80,000 licenses for the 80,000 corporate
users/benefiting users |
80,000 |
A telecommunication company with 50,000 users that plans
to buy 30,000 Teams Premium seats |
30,000 Teams Premium licenses plus 20,000 standalone
Microsoft eCDN licenses |
50,000 |
A pharmaceutical company with 70,000 users spread across
three regions including North America (50,000), EMEA (10,000), APAC (10,000).
IT wants to enable Microsoft eCDN in North America |
50,000 licenses for the 50,000 benefiting users in North
America. Note: Microsoft eCDN needs to be disabled for EMEA
& APAC via subnet mapping. |
50,000 |
What Do You Need To
Configure For eCDN?
To turn
on eCDN you simply need to go to the Teams Admin Centre > Meetings
> Live Event Settings > Video Distribution Provider > ON, and
select Microsoft eCDN as the provider. In order to be able to select Microsoft
you will need to have some Premium licences in your tenancy (Message: “You need
Teams Premium or a Microsoft eCDN license to use some settings. Upgrade to a
supporting license in the Microsoft 365 admin center. Learn more about Teams
Premium”).
Once you have turned on the Microsoft eCDN you will then have access to the eCDN Portal:
https://admin.ecdn.teams.microsoft.com/
Subnet mapping
In the eCDN Portal, the primary configuration task is to set up your subnets, enabling the software to accurately determine the clients’ locations within your network environment.
This configuration allows you to specify how clients in particular subnets should behave and be grouped. Grouping subnets enables media peering between them. Whether a subnet permits media peering depends on the P2P (Peer-to-Peer) setting assigned to it. Additionally, mapping subnets facilitates per-site analytics within the eCDN portal.
The P2P setting can be ON, OFF, or LEECH. The ON and OFF settings are straightforward, indicating whether peers in these subnets will allow or disallow peering. The LEECH setting means that the subnet can only receive data from another subnet within its group that has peering enabled.
Another more complex setting is WAN. This setting allows clients to use the STUN protocol to discover their public addresses, which are then published to the eCDN as potential peers. This is an optional setting, applicable to specific use cases.
Please note that it is suboptimal for VPN users to participate in peering. It is recommended to disable peering for VPN users.
Below is
a table provided by Microsoft for these settings:
Column
name |
Data
type |
Description |
Guidance |
Group Id |
String |
Your site name is displayed in the analytics. Only peers
with the same Group ID can peer with each other. |
Required |
Subnets |
CIDR |
Space or comma (,) delimited network specifiers in
the CIDR format
(e.g. 10.0.0.0/24). |
Required |
P2P |
Enum |
Configure how users who fall under this group behave.
Possible values: ·
On - Peering is enabled. ·
Off - Peering is disabled. ·
Leech - Peering is enabled in consume-only
mode. Note: When formatting the CSV this
entry can be in the format “p2p-on” or it can just be “on”. When you export
the CSV from the system it will use the “p2p-on” formatting of this config
item. |
Required |
WAN |
Enum |
Enabling WAN toggles on the STUN protocol, which allows
us to create peering connections between internet users. ·
On - Peering via public IP is
enabled. ·
(default) Off - Peering via public
IP is disabled. Note: When formatting the CSV this
entry can be in the format “wan-on” or it can just be “on”. When you export
the CSV from the system it will use the “wan-on” formatting of this config
item. |
Optional |
Label |
String |
Labels help you identify sites more precisely. |
Recommended |
Country |
String |
Manually override the country/region as determined based
on the viewer's public IP address. This option is useful for viewers whose
internet breakout point doesn't represent their true location. |
Optional |
City |
String |
Same behaviour and guidance as Country column. |
Optional |
Default
Group Mappings:
Criteria
matrix |
Subnet
mapping present |
|
No
subnet mapping |
|
Group name |
p2p config |
Group name |
p2p config |
|
No IP / Failed IP |
Ungrouped |
off |
Ungrouped |
off |
IP supplied (no match) |
Default Group |
off |
all |
on |
One of two significant groups could apply to your viewers
even when a subnet mapping is present.
·
Ungrouped - Viewers are assigned to this P2P-off group when their eCDN client does not receive a valid IP address from the operating system. This typically occurs due to mDNS issues with the browser. For more details, please refer to the mDNS section of this post.
·
Default Group - This P2P-off group is applied to viewers who were not accounted for in the subnet mapping at the time of the event.
·
all - This group is applied when subnet mapping is not present. It includes all viewers whose eCDN client received a valid IP address from the operating system. These viewers are set to P2P-on.
Below is
an example of the CSV file format for importing Subnet information:
GroupId,Subnets,P2P,WAN,Label,Country,City
Site
A,10.0.0.0/24,p2p-on,wan-on,Melb Office,AU,Melbourne
Site B,10.1.0.0/24,p2p-on,wan-on,Syd
Office,AU,Sydney
Note: You can have a maximum of 50,000
subnet entries.
WebRTC Browser Disable mDNS
The Microsoft documentation outlines the requirement to disable mDNS in browsers using registry keys. The initial question that arises is: what is mDNS in this context? It turns out that with WebRTC frameworks in the browser, it was possible to run a background check to discover a machine’s private IP address. This capability could potentially be exploited for monitoring or hacking purposes. Consequently, browser vendors implemented a feature that prevents JavaScript from accessing the private IP address of a machine. This is not desirable in this case, as a critical component of how eCDN functions is its need to access private IP address information to determine subnet mapping.
Upon further investigation, it appears that if users have granted microphone and camera access to the browser during the meeting setup, the mDNS IP address hiding functionality will be disabled by default. Therefore, if users enable their microphone and camera, they will be able to utilize the eCDN functionality without any issues. Below are examples of optimal settings that allow camera and microphone access:
If you run into a situation where clients are not getting linked to their group due to their IP Addresses being blocked then you can use Windows Registry settings that are specific for the browser being used: https://learn.microsoft.com/en-us/ecdn/how-to/disable-mdns
Networking
You will
need to ensure that the Teams client have access to the following cloud-based
web services to orchestrate this feature:
External
Network Connections
Microsoft eCDN setup has some networking
requirements for connecting to the eCDN services in Microsoft’s cloud:
1.
When a user browses to the event page, the
client needs to download the Microsoft eCDN script - that requires
an HTTPS connection to *.ecdn.teams.microsoft.com.
2.
It then creates a secure WebSocket connection to
Microsoft’s eCDN backend.
3.
The peer-to-peer connection itself is a UDP
connection over the port range 1025-65535, chosen randomly by the browser.
External Network Connections
Hostname |
Ports |
Protocol |
Description |
*.ecdn.teams.microsoft.com |
443 |
HTTPS over TCP |
Microsoft eCDN scripts |
*.ecdn.teams.microsoft.com |
443 |
WebSocket over TCP |
Microsoft eCDN backend |
Internal Network Connections
Connections that remain inside the corporate network.
Usually packets in these connections don't go through a firewall and wouldn't
need any configuration to allow them.
Hostname |
Ports |
Protocol |
Description |
n/a |
1025-65535 |
SCTP over DTLS over UDP |
P2P communication |
Limit The Media Port Range in Browsers
The Media port range can be limited for both Microsoft
Edge and Google Chrome browsers. Note: The Teams client requires ports
1025-65535 to be allowed on internal networks between peers.
·
https://chromeenterprise.google/policies/##WebRtcUdpPortRange
·
https://admx.help/?Category=EdgeChromium&Policy=Microsoft.Policies.Edge::WebRtcUdpPortRange
Limiting Access to the Service
To restrict eCDN access, you must specify the external public IP addresses from which clients will be accessing the Internet (i.e. the external NATed address). You do this by specifying
Public IP Addresses under the Configuration > Security section. If no
IPs are specified, then all users are allowed to connect from all public IPs.
Supported input formats:
- Single IP - A single 4 octet IP. Example: 13.107.231.100
- CIDR - A standard Classless inter-domain routing (CIDR) consisted of a start address and a subnet mask. Example: 13.107.231.0/24 (equivalent to: 13.107.231.0 - 13.107.231.255)
- JSON Format - A custom JSON format which matches the export format. Example: [ "13.107.231.0/24", "51.12.41.7/32" ]
Limitations
- IPv6 is not support at this time
- Up to 256 ranges
Testing eCDN
The Microsoft eCDN also provides the capability to conduct network tests to evaluate performance at scale before hosting a significant meeting on the platform. The silent test operates by running a script in the background on selected users’ machines, initiating a muted video session on the end-user’s device. This allows users to continue their work uninterrupted and unaware of the test. However, as the test imposes actual load on your networks, it should be used carefully.
To keep this post from getting too long, I will not go into all the details of setting up silent testing. If you want to know more you can find the docs here: https://learn.microsoft.com/en-us/ecdn/technical-documentation/silent-testing-framework
The Wrap Up
Apart from the Microsoft documentation, there is limited information available on the Microsoft eCDN service. This prompted me to conduct an in-depth exploration of the service. It offers clear advantages for organizations planning large internal Townhall meetings. Additionally, eCDN is a valuable component included with Teams Premium, especially if it is widely deployed within your organization.
For full eCDN documentation is available here for your reference: https://learn.microsoft.com/en-us/ecdn/intro