Polycom VVX Not Displaying PIN Authentication Option

I had an interesting issue with a Polycom VVX deployment recently that I thought I would share in case others run into the same issue.

The Issue

The symptom of the problem was that after the Polycom VVX had completed booting, including getting an IP Address and downloading software/config files, the PIN Authentication option did not appear on the sign-in options screen. This meant that I was unable to use PIN Authentication at all for signing in the devices which was a problem because we planned on using it for all the phones. Below is an example of what the screen looked like:

Sign-in Screen without PIN Auth option

Troubleshooting

There was a series of steps that I went through in troubleshooting this issue. I will take you through all of them so you too can check whether your issue might be solved with some of the earlier steps that I tried before reaching a resolution.

STEP 1

I first confirmed that PIN Authentication was in fact turned on in the configuration file(s) of the phone. To do this I checked that the following setting was not in the configuration files:

<!-- Disable PIN Auth by setting "0" -->

<reg reg.1.auth.usePinCredentials="0" />

Note: The phone can have multiple configuration files that are both manually added by administrators and automatically created by the phone (ie. <MAC>-phone.cfg, <MAC>-web.cfg, etc). You need to check all of the files associated with the phone's MAC address to ensure it’s not being overridden by another file.

I also checked the setting directly in the phone using my VVX Phone Manager Tool to get the active setting out of the phone using the REST interface. In my case this setting was not configured in the config file and it defaults to being on (ie. set to "1"). So this wasn't the problem.

STEP 2

I checked that PIN Authentication was actually enabled on the Skype for Business server. This can be done in the Control Panel > Security > Web Services > Pin Authentication Enabled:

This was also enabled - so in this case it wasn't the problem.

STEP 3

I tested the PIN Authentication process on the server by running Test-CsPhoneBootstrap PowerShell command on the system. This worked just fine:

PS C:\ > Test-CsPhoneBootstrap -PhoneOrExtension 4500 -PIN 12345 -TargetFqdn 2015ENTFE004.myskypelab.com -TargetUri https://2015ENTFE004.myskypelab.com:443/CertProv/CertProvisioningService.svc

Target Fqdn   : 2015ENTFE004.myskypelab.com

Target Uri    : https://2015ENTFE004.myskypelab.com:443/CertProv/CertProvisioningService.svc

Result        : Success

Latency       : 00:00:01.2333041

Error Message :

Diagnosis     :

STEP 4

In this deployment there was a centralised Windows Server that was serving DHCP to all the client subnets. On the central DHCP server I confirmed that all of the DHCP options were correct using my Skype4B/Lync DHCP Config Tool. This tool parses the byte format Vendor Options and displays them as readable text, and if it is unable to parse the byte format it will display an error:

This is an example image from my lab

In this case, all settings were displayed and no encoding issues were detected by the tool, which means this wasn’t the issue. So I checked that there was no DHCP server on a closer subnet (ie. a switch or router) that was responding to DHCP before the central Window DHCP server. This also wasn’t the case as I could see that the central DHCP server had logged the address lease for the Polycom VVX with the particular MAC Address of the test device.

STEP 5

At this point this was starting to look like a more complex problem so I took to the lab to see if I could reproduce such behaviour.  I noticed that after a factory default the phone initially didn’t display the Pin Authentication option for a couple of seconds - it appeared belatedly. This indicated to me that there was some additional check that was being done by the VVX before it would display this option. So this begged the question: what is required for PIN Authentication to function on the VVX? The most important thing that is required is that the phone gets the DHCP Options which tell it where the Cert Provisioning services resides, so it can communicate with the web services required for PIN Authentication.

Given that the VVX phones were being issued IP Addresses via DHCP, it didn’t seem likely to be a connectivity issue between the VVX and the DHCP server. However, I looked into the traffic flows to confirm this and found something interesting. In this Wireshark capture, you can see that the DHCP Options get sent out in response to an INFORM message that the VVX sends. The INFORM message is a special DHCP message that is outside of the initial DHCP IP Address discover process (DISCOVER > OFFER > REQUEST > ACK). The interesting thing about the INFORM message is that the ACK for this message from the DHCP server gets sent as a Unicast response directly back VVX itself rather than to the DHCP Relay IP Address, unlike all the other messages. The screen shot below also shows this from the DHCP server perspective - you can see the final ACK message has a Destination IP Address of the VVX instead of the DHCP relay IP Address:

The highlighted INFORM ACK message in Wireshark shows that it contains all the additional Microsoft specific Vendor Class Options (Certificate Provisioning Service details). It’s the packet that has the information that the VVX needs to get PIN Authentication working.

In this case, because a centralised DHCP was being used, the broadcast DHCP messages on the local subnet were being changed into unicast messages by the local router and sent over to the central DHCP server. There was also a firewall in between this local router and the centralised DHCP server. This meant that because the returning INFORM ACK message was sent directly back to the phone (which is part of the DHCP specification and is correct operation) the firewall had not created a UDP flow for it and the packet gets blocked. The diagram below shows how the DHCP traffic flow works with a DHCP Relay in place and where the issue resides:

As you can see from the diagram above, the firewall appears to be allowing traffic from the DCHP relay through to the DHCP server. After transiting the DCHP relay, the DHCP traffic flows from source port 67 to destination port 67. Then the INFORM ACK message then gets sent back from source port 67 on the DHCP server to destination port 68 on the VVX -  which the firewall did not have an existing flow for and it dropped the packet. As a result, the VVX didn’t receive its required Cert provisioning service URL and because of this didn't display the PIN Authentication sign-in button.

The Solution

So the solution here, as it often is, is firewall related. In this case we had to allow port 68 from the DHCP server IP Address to all the VVX phone subnets. After this was done the INFORM ACK messages could flow as required for the VVX to get its Vendor Class options.

If you don’t have access to the firewall or you need a quick solution to the problem, you can hard code the data contained in the Vendor Class options into the phone. This was added as a config option in software version 5.3. The configuration item is shown below:

<dhcp dhcp.option43.override.stsUri="https://s4bwebint.domain.com:443/CertProv/CertProvisioningService.svc" />

This can also be set in the web interface of the phone in the Settings > Provisioning Server > DHCP Menu > DHCP Option 43 Override STS-URI:

The Wrap Up

For all the old-school UC people out there, let's finish with a Haiku in the style of the old Lync 2010 powershell blog:

Firewalls drop packets,

This causes many issues,

Switch off all firewalls.

Till next time, see ya! 

Read more →