This blog post covers an interesting issue that I ran into the other day with AudioCodes Mediant
Virtual Edition and getting transcoding capabilities working on VMWare. The AudioCodes Virtual Edition SBCs are
very capable and are great for SIP Trunk deployments and routing calls between various SIP platforms. They can support a large number of non-transcoded calls
as well as supporting transcoding for audio and DTMF streams. In a hardware SBC, transcoding is usually done with a DSP chip that is used to convert one codec
to another codec. However, in a Virtual SBC this is done using the CPU cores provided to the virtual machine.
The requirement for transcoding in a deployment can come up when connecting
to carriers with mandatory codec requirements like G.729 (a low bandwidth codec) that are not supported by Skype for Business. In these cases, if the carrier decides to send a call inbound with only G.729 (or some other unsupported codec) as
the only media type available in the INVITE SDP, rather than reject the call, we can have
the SBC transcode between the two different codecs.
So what is required for transcoding to work on the
AudioCodes Virtual Edition SBC? Just CPU cores right? Well, not exactly…
Continue on, intrepid reader…
CPU Specifications
The AudioCodes manuals are quite clear with their requirements for how many CPU Cores are required to support different numbers of sessions and
transcoding capabilities. These range from low spec VMs with only 2 cores up
to 8 cores (as of version 7.2). For details of the number of CPU Cores required for
each scenario, I suggest you check out the latest Mediant Virtual Edition User
Guide (the Channel Capacity section).
In addition to the number of CPU Cores required, the 7.0 and
7.2 manuals for the AudioCodes Virtual Edition both state the following about CPU
requirements:
Specifications
Resource
|
Specifications
|
Processor type
|
64-bit Intel® CPU of at least 2.5 GHz, with support for hardware
virtualization (Intel VT-x) enabled with Advanced Vector Extensions (AVX) and
AES-NI support (Sandy-Bridge architecture or newer)
|
Whilst usually the exact specifications of a CPU that an application vendor
lists can be taken as a generic baseline for the systems that they may have
done capacity testing on, in this case the CPU capabilities absolutely matter! It
seems that when the Virtual Edition was designed, the code used for transcoding
was specifically targeted for CPUs that support the AVX feature set introduced
in Sandy Bridge CPUs. When the Virtual SBC boots, it checks the CPU capacities
and if it is not at least Sandy Bridge level (with AVX support) then the SBC
transcoding capabilities will not be available!
This limitation has some additional flow-on effects
depending on the hypervisor on which it is being deployed. See the following section for more
details of these limitations.
VMWare and CPU Capabilities
In addition to the raw CPU capabilities in the host machines (ie. Sandy Bridge generation of processors with AVX support), a VMWare environment using vMotion also can be configured with a specific level of processor support across all hosts in a vMotion cluster. This is a feature within VMWare that is called Enhanced vMotion Compatibility (EVC) and controls the level of CPU capabilities being emulated at the VM level across multiple machines. This ensures that the same CPU capabilities are presented for all machines in a cluster, and protects against having differing CPU capabilites when VMs are migrated between hosts. This can mean that even if the host machine has a Sandy Bridge CPU, this capability set may not be passing through to the VM on which the SBC is running.
The VMWare vMotion EVC level will be used when the host machines in a cluster have different levels of CPU capabilities. The lowest CPU capability will be the one
that is emulated for all machines in the cluster. So, whilst many of the hosts
may have Sandy Bridge or above CPUs, there may be lower capability machines
included in the cluster that force the EVC baseline in VMWare to be configured
to a lower CPU feature set. This setting can directly affect the Virtual Edition SBC's ability to support transcoding, because even if the host has Sandy Bridge
capabilities the VMWare environment might be masking these capabilities and
presenting a lower CPU level than is actually in the host.
Below is a table that describes the various EVC levels that
can be configured within VMWare:
VMWare Description of Intel EVC Baselines:
EVC
Level
|
EVC
Baseline
|
Description
|
L0
|
Intel® "Merom" Gen. (Intel® Xeon® Core™ 2)
|
Applies baseline feature set of Intel®
"Merom" Generation (Intel® Xeon® Core™ 2) processors to all hosts
in the cluster.
|
L1
|
Intel® "Penryn" Gen. (formerly Intel® Xeon® 45nm Core™ 2)
|
Applies baseline feature set of Intel® "Penryn" Generation
(Intel® Xeon® 45nm Core™ 2) processors to all hosts in the cluster.
Compared to the Intel® "Merom" Generation EVC mode, this EVC mode exposes additional CPU features including SSE4.1. |
L2
|
Intel® "Nehalem" Gen. (formerly Intel®
Xeon® Core™ i7)
|
Applies baseline feature set of Intel®
"Nehalem" Generation (Intel® Xeon® Core™ i7) processors to all
hosts in the cluster.
Compared to the Intel® "Penryn" Generation EVC mode, this EVC mode exposes additional CPU features including SSE4.2 and POPCOUNT. |
L3
|
Intel® "Westmere" Gen. (formerly Intel® Xeon® 32nm Core™
i7)
|
Applies baseline feature set of Intel® "Westmere"
Generation (Intel® Xeon® 32nm Core™ i7) processors to all hosts in the
cluster. Compared to the Intel® "Nehalem" Generation mode, this EVC
mode exposes additional CPU features including AES and PCLMULQDQ.
|
L4
|
Intel® "Sandy Bridge" Generation
|
Applies baseline feature set of Intel® "Sandy
Bridge" Generation processors to all hosts in the cluster. Compared to
the Intel® "Westmere" Generation mode, this EVC mode exposes
additional CPU features including AVX and XSAVE.
|
L5
|
Intel® "Ivy Bridge" Generation
|
Applies baseline feature set of Intel® "Ivy Bridge"
Generation processors to all hosts in the cluster. Compared to the Intel®
"Sandy Bridge" Generation EVC mode, this EVC mode exposes
additional CPU features including RDRAND, ENFSTRG, FSGSBASE, SMEP, and F16C.
|
L6
|
Intel® "Haswell" Generation
|
Applies baseline feature set of Intel®
"Haswell" Generation processors to all hosts in the cluster.
Compared to the Intel® "Ivy Bridge" Generation EVC mode, this EVC
mode exposes additional CPU features including ABMX2, MOVBE, FMA, PERMD,
RORX/MULX, INVPCID, VMFUNC.
|
In order for AVX to be supported in a large VMWare
environment, the Processor Support needs to be configured at a minimum of L4 Sandy Bridge level. If EVC mode is not being used the EVC mode may display as "N/A" or "disabled", both of these modes will allow for the raw processor capabilities to be passed through to the SBC, which will also allow for transcoding to work if the CPU is Sandy Bridge or above.
VMWare reference: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003212
The host EVC level can be viewed in the host summary page within vSphere. In the example below the
current EVC level is set to Merom Generation or L0 level which would not support transcoding:
Hyper-V Processor Compatibility Mode
I haven’t actually tested this on Hyper-V because transcoding
on Hyper-V has only been supported as of 7.2 software. However, Hyper-V also
has a CPU Compatibility mode feature which allows for various features that
involve moving VMs between machines (eg. Live Migration, Quick Migration and
Save/Restore features). This feature is similar to VMWare EVC mode, however,
the implementation is slightly different than VMWare. When the Processor
Compatibility mode is enabled in Hyper-V the CPU (for Intel and AMD CPUs)
feature set gets “normalised” by removing extended feature sets of the each CPU
type to lower the capabilities to a baseline.
For the Microsoft documentation, when a VM
in processor compatibility mode is started, the following processor features are
hidden from the VM:
Host running Intel based processor
|
SSSE3, SSE4.1, SSE4.2,
POPCNT, Misaligned SSE, XSAVE, AVX
|
You will note that AVX is one of the hidden features, which
is one of the features required for the AudioCodes Virtual Edition SBC to work!
Uh-oh…
When this feature was released, the following Q&A was
documented about this functionality:
From the Hyper-V Compatibility
Mode Q and A:
Q: Can I
individually select which features should be hidden from VM when put in
processor compatibility mode?
A: No, once you put VM in a processor
compatibility mode, Hyper-V will hide the necessary set of features from the VM
to ensure a successful migration between Hyper-V enabled processors.
Q: What happens
if a vendor has written an application that uses one of these features that
isn’t visible with processor compatibility enabled?
A: Since the feature
isn’t exposed to the virtual machine, the application won’t “see it” and it’s
up to the application to determine how to proceed; however, there are two
likely paths.
So presumably the CPU compatibility feature within Hyper-V
is going to cause the AudioCodes Virtual Edition SBC to not be able to see the
AVX capabilities of the CPU. As a result, transcoding will likely not work on
Hyper-V if processor compatibility mode is enabled.
Hyper-V Processor Compatibility
References:
Checking CPU Capabilities after Installation
If the Virtual Edition SBC has been installed within a virtual environment, you can tell if the hardware will support transcoding by checking the following CLI commands:
Working Transcoding example:
Mediant VE SBC> show system hardware
CPU:
Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz, total 4 cores avx support(avx)
Memory: total RAM: 4096 MB
Chassis: VMware Virtual Platform
Network:
VMware VMXNET3 Ethernet Controller (rev 01)
VMware VMXNET3 Ethernet Controller (rev 01)
Virtual env: vmware
Mediant VE SBC>
In the example above you can see that
the CPU has “avx support (avx)”. Transcoding will work on this server.
Non-supported hardware example:
Mediant VE SBC# show system hardware
CPU:
Intel(R) Xeon(R) CPU X5650 @ 2.67GHz, total 4 cores avx support(none)
Memory: total RAM: 8192 MB
Chassis: VMware Virtual Platform
Network:
VMware VMXNET3 Ethernet Controller (rev 01)
VMware VMXNET3 Ethernet Controller (rev 01)
VMware VMXNET3 Ethernet Controller (rev 01)
Virtual env: vmware
Mediant VE SBC#
In the example above you can see that
the CPU has “avx support (none)”. Transcoding will not work on this server.
Web Interface Check
The only way to tell from the web interface whether or not the current server hardware will support transcoding is to download the configuration file and check the following:
Transcoding Working Example:
;Board: Mediant VE SBC
;HW Board Type: 73 FK Board Type:
79
;Product Key:
;Slot Number: 1
;Software Version: 7.00A.095.004
;DSP Software Version: SOFTDSP => 700.53
;Board IP Address: 10.20.1.168
;Board Subnet Mask: 255.255.255.0
;Board Default Gateway: 10.20.1.1
;Ram size: 3831M Flash size: 0M
;Num of DSP Cores:
3 Num DSP Channels: 100
;Profile: NONE
;Client defaults file is being used
(file length=352)
In this working example the
configuration file displays that it has discovered 3 DSP Cores and has 100 DSP
channels available for transcoding.
In this example, the configuration file displays that it has discovered 0 DSP Cores and has 0 DSP channels available for transcoding. In this case the SBC has detected that the CPU does not have AVX support and as a result will not support transcoding.
;Board: Mediant VE SBC
;HW Board Type: 73 FK Board Type:
79
;Product Key:
;Slot Number: 1
;Software Version: 7.00A.095.004
;DSP Software Version: SOFTDSP => 700.53
;Board IP Address: 10.20.1.165
;Board Subnet Mask: 255.255.255.0
;Board Default Gateway: 10.20.1.1
;Ram size: 7871M Flash size: 0M
;Num of DSP Cores:
0 Num DSP Channels: 0
;Profile: NONE
;Client defaults file is being used
(file length=352)
Example of Transcoding
Failure Errors
When the transcoding resources are not
available due to a lack of CPU support for AVX, you will see errors complaining about a lack of resources in the log
at the time of the SBC trying to apply extension coders or setting up the call:
22:41:25.812
147.76.26.84 local0.notice [S=4742] [SID=a64ab9:33:279] (
lgr_flow)( 4645)
#MediaResourcesConnector::AllocateMediaResources
22:41:25.812
147.76.26.84 local0.notice [S=4743] [SID=a64ab9:33:279] ( media_connect)( 4646)
ConnectionData::CalculateResourcesForExtTranscoding Leading:DSP
Opposite:CODERTRANSCODING MediationLevel:RTP
22:41:25.812
147.76.26.84 local0.notice [S=4744] [SID=a64ab9:33:279] ( media_service)( 4647)
ServicesMngr: Allocate Coder Transcoding session. current active: 0 and
max is: 250
22:41:25.812
147.76.26.84 local0.notice [S=4745] [SID=a64ab9:33:279] ( media_service)( 4648)
ServicesMngr: Allocate Media channel. current active: 0 and max is: 0
22:41:25.812
147.76.26.84 local0.warn [S=4746] [SID=a64ab9:33:279] ( media_service)( 4649) !! [ERROR] ServicesMngr: Cannot
allocate more Media channel. current active: 0 and max is: 0
22:41:25.812 147.76.26.84 local0.warn [S=4747] [SID=a64ab9:33:279] (
lgr_flow)( 4650) !! [ERROR]
(#2198)RTPStreamResource::AllocateResource Allocate Resource - cannot allocate
DSP. probably lack of resources
22:41:25.812
147.76.26.84 local0.notice [S=4748] [SID=a64ab9:33:279] ( media_service)( 4651)
ServicesMngr: Deallocate Coder Transcoding session. current active: 1
and max is: 250
22:41:25.812
147.76.26.84 local0.notice [S=4749] [SID=a64ab9:33:279] ( media_service)( 4652)
ServicesMngr: Allocate Coder Transcoding session. current active: 0 and
max is: 250
22:41:25.812
147.76.26.84 local0.notice [S=4750] [SID=a64ab9:33:279] ( media_service)( 4653)
ServicesMngr: Allocate Media channel. current active: 0 and max is: 0
Licensing
In addition to the CPU requirements, you will also need licensing for transcoding to work. The CODER-TRANSCODING, DspCh and IPMediaDspCh licences must be available in the licence installed on the SBC. The CODER-TRANSCODING, DspCh and IPMediaDspCh licences will appear in the licence list as shown below:
Key features:
Board Type: Mediant VE SBC
IP Media: VXML
Coders: G723 G729 G728 NETCODER GSM-FR GSM-EFR AMR EVRC-QCELP
G727 ILBC EVRC-B AMR-WB G722 EG711 MS_RTA_NB MS_RTA_WB SILK_NB SILK_WB SPEEX_NB
SPEEX_WB OPUS_NB OPUS_WB
Channel Type: DspCh=100 IPMediaDspCh=100
HA
Security: IPSEC MediaEncryption StrongEncryption
EncryptControlProtocol
DSP Voice features:
DATA features:
Control Protocols: MGCP SIP SBC=10 MSFT FEU=50 CODER-TRANSCODING=50
Default features:
Coders: G711 G726
If you do not have the CODER-TRANSCODING, DspCh and IPMediaDspCh licences then the
SBC will still run, however transcoding functionality will not be available (config will show "Num of DSP Cores: 0 Num DSP Channels: 0" as explained earlier).
The Wrap Up
So before jumping in to the Virtual Edition SBC world with
your customers, be sure to understand the server and Hypervisor environment that you
are deploying the SBC into. Some useful questions to ask up front are:
- What hypervisor will the SBC be installed on?
- Have you purchased CODER-TRANSCODING licences for the SBC?
- Is the hypervisor server host CPU at least Sandy Bridge level and does it support AVX?
- If vMotion is being used within VMWare: What is the VMWare vMotion EVC baseline of the host that the SBC is deployed on?
- If Hyper-V is being used, is Processor Capability mode disabled?
- Have you ensured that the hypervisor is going to have enough dedicated CPU Cores available for the SBC? Also, note that the CPU resource must be reserved for the SBC and not shared.
Armed with this information, you will be able to make an
informed assessment on whether or not transcoding is going to be supported on the
Virtual Edition SBC! Hope this was helpful. Cheers!
