Thursday 6 February 2014

Lync Edge Testing Suite (Part 1) – Lync Edge Port Tester Tool

Okay, let's start 2014 with a bang, and turn this thing up to 11. This is the first post in a series that I will be doing about Lync Edge servers. More specifically, this series will be about some tools that I have written to help you with pre-deployment testing and troubleshooting issues with your Edge Server deployments.

For Part 2 of the Lync Edge Suite series go here.

Lync Edge Deployment


Deploying an Edge server within an organisation can often be quite challenging. The process involves topics like public certificates, private and public DNS records, routing, firewall configuration, DMZs, firewalls, NATing, etc… This often means that several people, or even several teams of people, will need to be involved in the process. As a general rule, the more people involved in a process, the more opportunities there are for things to go wrong. This is especially the case when only a small error like typing “_sip._tcp”, instead of “_sip._tls” can mean the difference between things working, and things not working.

When deploying an Edge server it is very important that you check that all the pieces of the puzzle are correctly in place before starting the installation. Otherwise, you will be “Nexting” your way through the deployment wizard towards a nightmare-hell-ride of live production troubleshooting. So it’s always advisable to test that everything is configured correctly on your network before even thinking about opening the topology wizard.

The area that we are going to focus on in this post is the testing of a DMZ’s firewall configuration. Microsoft has done a very good job of documenting the firewall requirements when deploying Lync Edge servers. The Technet pages here go in to quite a bit of detail about the different topologies and their port requirements. The Lync workloads poster is also a good place to look for more information. So, after reading all this information and configuring your firewall, how do you actually test that it is working? HINT: keep reading…

  Lync Edge Port Tester Tool




     Features:
  • One tool for testing both ends of the connection. Run the Lync Edge Port Tester on your Edge server and on an internal and external network machine.
  • Test both UDP and TCP ports are open in both directions.
  • Select a profile from the list (Edge Outside, Edge Inside, Front End, Client Outside, Client Inside) to check all the relevant ports for that connection type.
  • Open all the required firewall ports temporarily with the press of a button.
  • Can be used to test connections to a live Edge (or Front End) server. If you run the tool on a live edge server it will automatically learn which ports are currently in use and only allow you to Listen on unused ports.
  • Can be used on a Consolidated Edge configuration or on an Edge running with a single external IP address (ie. all 443 services running on different ports). The Single External IP check box allows for this configuration.
  • Tool Tips to provide you with contextual help.
  • All in one interface. (Over 100 interface objects! Yep, that was a lot of typing...)





 The Interface:



  1. These drop-down text boxes need to be populated with the IP Addresses used on the inside and outside of the Edge server. The drop down menu for each of these gets populated with IP addresses that exist on the machine. So if you are running the tool on the edge server you can fill these in with the drop down menu selections. If you are running the tool on a machine that isn't the Edge you will have to type these IP addresses in yourself.
  2. These profiles will set the port check boxes to the appropriate settings for testing each scenario. For example, if you are running the tool on the Edge server to test the outside ports, you will select “Edge Outside”.
  3. This is the IP Address of the machine running the “Front End”, “Client Outside” or "Client Inside" profiles, that you are testing your connections to. If you are running the tool on one of these machines you can use the drop down box to select the IP. Or if you are running it on the Edge you will need to type the IP address into the field.
  4. These are the ports/interfaces that you would like the tool to listen on. When you select the appropriate profile, the necessary ports will be automatically selected here. If you would like to test specific ports you can set these check boxes yourself. The naming convention used for these check boxes are <Port number to listen on><Interface to listen on>. The interface naming convention is, AC (Access Edge IP), AV (Audio Video Edge), CON (Conferencing Edge), and IN (Internal Interface of the Edge). The labels on the listening ports will change colour depending on which ports may already be used on the machine. If the machine already has a service listening on the port the label will turn a gray colour (and the check box will be disabled), so you will not be able to open a listener on these ports. The label will turn red if a port is being used by an existing service AND the port is used by the selected profile (this is informational, to allow you to know that you should be able to test connection to this port). 
  5. These are the settings for making outbound connections from the tool. The naming convention for these check boxes are the same as above <Port number to connect to><Interface to connect to>. You should always be able to make outbound connections on these ports, so these port labels don't change colours like the listening port column.
  6. These are Misc Options. “Clear All Boxes” will clear all the Check Boxes to allow you to make your own settings. “Close Listeners” allows you to close all the listeners that have been spawned in different Powershell windows. “Open and Close Firewall” button allows you to automatically open all the required ports within Windows Firewall to use the tool. This is necessary in order to be able to accept inbound connections to listening ports. When you first use the tool on a machine you should open the ports, and when you are finished testing you should press the Close Firewall button to remove the Windows firewall settings.
  7. This is a window that will log information about what the tool is doing.


The 50000 Range


The 50000AV range is special because it can use TCP and UDP for the entire range of ports between 50000 and 59999. It's also special because, although the Edge will listen on a random selection of these ports, the ports themselves will only accept connections when media has been formally set up by client SIP signalling. This is a security measure that was implemented to reduce the attack surface of the external Edge AV interface.

The Lync Edge Port Tester tool will test the first port (50000) and the last port (59999) in this range. This is by design for two reasons, the first is that the Media Relay service on the Edge only listens on a random selection of these ports (run "netstat -a" to check this) and most of the time this random selection doesn't include port 50000 or 59999. This allows these ports to be happily opened by the tool the majority of the time. The second thing is that firewall rules are always set on a range of ports so if we test the first port in the range (50000) and the end port in the range (59999). Testing these two ports is usually enough to allow you to assume that the ports in between these values will also be allowed by same firewall rule (if not, then you have bigger problems with your firewall).

For Lync environments that don’t need to federate to partners with OCS, you only need to open outbound TCP 50000-59999 range, so failures of the inbound connections can usually be ignored (ie. unless OCS federation is required). By default, the "Edge Server Outside" profile (and "Client Outside" profile) will test the 50000 range in both directions for both TCP and UDP. You can simply untick the tests in the inbound direction if you do not want to run tests on these ports.

Note: The purpose of this post is not to teach you how Edge servers work. If you would like to know more about how Edge servers work, Bryan Nyce and Thomas Binder have some good Tech Ed videos about it.


Using the Tool


In order to be able to use the tool you will need to run two copies of the tool in different locations on your network. The diagram below shows how you would test the connections in the internal interface of your Internal Edge interface:

Diagram – Testing Internal Connections

In the diagram there is a copy of the tool running on a server on the internal network that will behave as the front end server (it may actually be the front end server, but doesn’t have to be). This copy of the tool needs to be configured as the “Front End” profile. On the Edge server the tool is also running and it will be configured as being the “Edge Inside” profile. Configure the IP addresses at the top of both tools to match the IP Addresses used on the edge server. Configure the Front End/Client IP address on both tools to be the Front End IP address. Example:


To test the ports in both directions you now do the following:

To test connections from the Edge to the Front End (which as you can see in the diagram is just TCP Port 5061):
  • Press the Listen button on the Front End side tool. The tool will spawn new Powershell windows for each listening port. Note, if you're running this on a live Front End server you will not be able to listen on port 5061 because it's already in use. However, there is a service on the machine already listening on the port, so when you test the connection from the other side the machine service should still respond to the socket connection.
  • Press the Connect button on the Edge Server side. The Edge side tool will now send TCP (and UDP) connections to the tool at the other side as required and report the result. When the listening end receives the connections the spawned Powershell windows will disappear. The connecting side tool will give you feedback as to if the connections were successful or not.

Note: Make sure you have opened the firewall on the listening side of the connection using the "Open Firewall" button. Once you have finished using the tool you should always use the "Close Firewall" button to clean up and close the Windows firewall ports.

To test connections from the Front End to the Edge Server (which, as you can see in the diagram, is ALL the ports):
  • Press the Listen button on the Edge Server side tool. The tool will spawn new Powershell windows for each listening port. Note, if this is a live Edge server you will not be able to listen on ports that are already in use. However, if there is a service on the machine already listening on a port, you will still be able to test the connection to this listening socket.
  • Press the Connect button on the Front End Server side. The Front End tool will now send TCP and UDP connections to the tool at the other side as required. When the other end receives the connections the spawned Powershell windows will disappear.

Diagram – Testing External Connections

In the diagram there is a copy of the tool running on a server on the external network that will behave as an external Client/Federation Partner. This copy of the tool needs to be configured as the “External Client” profile. On the Edge server the tool is also running and it will be configured as being the “Edge Outside” profile. Configure the IP addresses at the top of both tools to match the IP Addresses used on the Edge server. Configure the Front End/Client IP address on both tools to be the IP address of the Outside Client machine. To test the ports in both directions you now do the following:

To test connections from the Outside Client to the Edge Server:
  • Press the Listen button on the Edge Server side tool. The tool will spawn new Powershell windows for each listening port. Note, if this is a live Edge server you will not be able to listen on ports that are already in use. However, if there is a service on the machine already listening on a port, you will still be able to test the connection to this listening socket.
  • Press the Connect button on the Outside Client side. The Outside Client tool will now send TCP and UDP connections to the tool at the other side as required. When the other end receives the connections the spawned Powershell windows will disappear.
To test connections from the Edge to the Outside Client:

Important Note: The outside machine will need to be directly on the internet in order for this to work. Most external connections to the internet are through Network Address Translating routers which will block unsolicited inbound connections. If you do not have a direct internet connection to test to you could configure port forwarding on the NAT router to allow each specific port into the outside client machine)

  • Press the Listen button on the Outside Client side tool. The tool will spawn new Powershell windows for each listening port.
  • Press the Connect button on the Edge Server side. The Edge side tool will now send TCP and UDP connections to the tool at the other side as required. When the other end receives the connections the spawned Powershell windows will disappear.

Tips, Tricks and FAQ

  1. For a NATed Edge configuration you will need to set the Edge IP addresses on the tool running on the outside machine to be the Public IP Addresses for the Edge (ie. the external post NATed IP addresses). On the tool running on the Edge server you need to set the IP Addresses of the Edge server to the private addresses that are actually configured on the interfaces of the Edge server. If you don't do this then all the external tested connections will fail.
  2. The ideal time to use the Lync Edge Port Tester is prior to installing the Edge software. This way you can be sure the network is right before installing the Edge server.
  3. The Lync Edge Port Tester tool can actually be used to test connections to all TCP ports on a live Edge server. Also, it can be used to check UDP 3478 (STUN) on a live Lync Edge server, because it sends and accepts real STUN messages! However, the Edge does not respond to TCP and UDP for 50000 range of ports, as these only get opened when media has been negotiated. So you can only test the 50000 range between servers by using the Edge Port Tester at both ends of the connection (see the section about the 50000 range earlier in the post).

The Wrap Up



    Tell me what you think about your sit-u-a-tion,
    Complication – aggravation,
    Is getting to you,

        - The immortal words of Steven Tyler of Aerosmith in their 1993 hit song “Living on the Edge

Hopefully, this new tool will help you to avoid lyrics such as these in any Lync Edge Server related songs that you may write in the future. If you find any bugs or issues with the tool, please report them back to me and if I can reproduce them I will fix them. Part 1 over-and-out. I will be back with Part 2 in this series shortly!


11 comments:

  1. Excellent write up James, I'll be putting your testing tool through its paces very soon

    ReplyDelete
  2. Very good and useful tool.
    Kudos to you!

    ReplyDelete
  3. Very cool, and very useful. thanks for the time spent making this.

    ReplyDelete
  4. Thanks man, you are my hero!

    ReplyDelete
  5. Just had a some issue on Lync Edge and your tool really helps! Thank you!

    Omar

    ReplyDelete
  6. Great tool. I see it is a PS1. Does it run under Lync Powershell?

    ReplyDelete
    Replies
    1. Hi Christian,

      Yes, you are correct. Most of the tools that you will find on this site are Powershell tools with .NET GUI front ends. You run them from from the Powershell command line like you would any other Powershell command.

      Thanks for your interest.

      Delete
  7. Am I missing something? The ports tested on the tool don't seem to line up with the required ports on the "protocol workloads poster". For example: No where on the poster does it mention port 23456, but when I run a test from my front-end to the inside edge, I get: "ERROR: Connecting to IP: Port: 23456, From IP: , port: 58355. Any ideas?

    ReplyDelete
    Replies
    1. Hi David,

      Port 23456 is on the Protocol Workload Poster as "XMPP/MTLS:23456". This port is used for internal XMPP communications between the Front End server and the Edge server. This port is only required to be open when you have enabled XMPP on your system.

      Thanks for the question.

      Delete
  8. Hi James,

    Great tool and very useful. I tested it on Edge server for Skype 2015, and using client Skype and Lync. It help me to discover some firewall bad settings on one of my Edge servers. Is just matter of usage of the tool.

    best regards,
    Javier

    ReplyDelete

Popular Posts