Power Syslog Server

“My Kingdom for a free and simple syslog server!” – Anonymous System Administrator

So I don’t know about you, but I can’t remember how many times I have got to the point of having to troubleshoot an issue with a Sonus gateway and suddenly remembering I need a Syslog server to get logging out of the box. At this point I usually go and ask Google politely “Google, can you please point me in the direction of a free, and simple, syslog server that I can run without installing a bunch of malware and other rubbish on this nice customer’s server?” At this point Google usually responds “No, I cannot. However, here is a syslog server that requires you install SQL, IIS, and fifteen other dependent services as well as being crippled unless you pay $14.99 per month to a Russian guy name Vlad via this popup window that displays in the middle of the screen every 5 minutes. Also, here’s a Yahoo browser search bar for your trouble.”

This is not an ideal situation… So as usual, I just decided to build it myself. In doing this I sat down and thought about the things I wanted in a simple syslog server, and came up with this list:

• It needs to have no installation process, and leave no trace once removed from a server, as it will be run on customers' servers in a lot of cases.
• It needs to have a display where I can see the messages coming in in real time.
• The messages being displayed must be able to be paused and reviewed, so I can check if a specific event has happened yet.
• The messages window must be able to be cleared so that I can start fresh when trying to troubleshoot a fault.
• The syslog server needs to be able to log to file. Ideally the files should be able to be opened in Sonus LX tool so that further message debugging can be done easily.
• The syslog server needs to be able to roll the log files once they get to a specific size (so they can be emailed, etc).
• The syslog server should only keep a specific number of these log files so that the server’s hard disk does not get filled with log files.
• Both the display and log files should be able to be filtered to display only information that I want to see. For example, only show lines with a specific phone number in them, or only show me SIP messages. These filters should be independent so that you can view the filtered information on screen whilst more detailed information is getting logged to file for further review and troubleshooting later.

Based on these requirements I figured it would be very cool to write the server in Powershell, as this allows for absolutely no installation and can be run on any Windows machine you are likely to run into. How hard could it be?

<Insert training montage>

SMASH CUT:

EXT. TRAINING MONTAGE - THE STAIRS AT THE FRONT OF THE PHILADELPHIA MUSEUM OF ART- DAY

A man in a sweaty hoody runs to the top of a large set of stairs carrying a tablet based productivity device that he is furiously typing on. A large group of the town’s population is also running after him in a large pack for no apparent reason. Upon reaching the top of the steps he punches the air and launches the tablet into the sky. The tablet hits the concrete and smashes into a million pieces. He falls to the ground and screams towards the sky.

MAN

Nooooooo! I should have backed up to the cloud, the cloud I tells ya.

Okay okay, let’s cut to the chase. I did it, and now you too can syslog with me into the sunset.

Power Syslog Server

Version 1.0 Features:

• Zero installation.
• Real time log display (Approximately 1000 lines).
• Copy the displayed text with the Copy Text button. This is useful for more in depth analysis in your favourite notepad software.
• Rolling log files based on file size and number of files to keep.
• Clear display and Pause display functions.
• Filter real-time display logging with regular expression.
• Filter logging to file with regular expression.
• Open firewall for Syslog Server port with the click of a button. If you are not seeing any syslog output in the Power Syslog Server display log then try pressing the Open Firewall button.
• Server listening port can be changed by creating a config file (PowerSyslogServerSettings.cfg) in the same directory as the script. The config file needs to have text in it in the following format "SysLogPort=514". This allows you to maintain the integrity of the code signing by not directly editing the script file.

Version 2.0 Update:

• Added output formatting options to work with Sonus LX tool and AudioCodes Syslog Viewer tool (Commonly used Skype for Business syslog tools used with SBC devices).
• In version 2 if you create a config file named "PowerSyslogServerSettings.cfg" in the same directory as the tool it will use the config file to save all of its settings. The SyslogPort="514" setting remains a hidden setting that can still be used in the config file to change the listening port number.
• UDP socket code has been made more robust to deal with errors when the listening port is being used by another app.
• Changed the font to Courier New for fixed width goodness.
• Fixed issue with rolling files in folders including "." in name and faster processing.

2.01 Bug Fixes (9/8/2017):

• Fixed Sonus LX output formatting to only have LF and not CRLF.
• Increased socket buffer and tuned threading to fix dropped packet issues and double writing of some lines.
• Added disable display checkbox to increase performance when display is not required.

DOWNLOAD HERE

Version 2.0 – Output formats

Version 2.0 of Power Syslog Server now gives you the option to add additional prefix formatting to the start of each line of syslog output. From the research I have done the format of output from each syslog server varies greatly and contains items such as date/time, text based priority field interpretation (ie. The <135> value at the start of syslog messages sent on the wire) and IP Address of the server that sent the message.

The reason that these prefixes are important is that if you want to import the file output back into a tool like Sonus LX or AudioCodes Syslog Viewer to generate call flow diagrams or other features the file needs to be in a format that these tools can interpret. So in order to achieve this, the Format dropdown box has been added in version 2. The Format setting will alter the outputs into the required format for Sonus LX or AudioCodes syslog tools. In addition to these specific tool formats, some other generic prefix formats have been added which will make the output files easier for humans to read.

Output Formats

Format	Example Prefix Format	Comment
None	<No Prefix>	Output syslog in the exact format that it was sent from the device in.
AudioCodes	"17:50:17.588  10.20.2.170     local0.notice"	Output syslog in AudioCodes Syslog Viewer tool format.
SonusLX	"10.20.1.150:53434 <==>"	Output syslog in the same format as the Sonus LX tool.
Level	"Local0.Debug"	Prefix the syslog with the Facility and Severity levels.
DateTime	"2011-10-11 15:00:02.123"	Prefix the syslog with the date and time.
DateTimeLevel	"2011-10-11 15:00:02.123 Local0.Debug"	Prefix the syslog with Date/Time and Facility/Severity.
DateTimeLevelIP	"2011-10-11 15:00:02.123 Local0.Debug    192.168.0.100"	Prefix the syslog with Date/Time, Facility/Severity, and IP Address of the device.

Note: Sonus LX tool cannot open AudioCodes files and AudioCodes syslog tool cannot open LX files. This is because there are special lines of output generated by each brand of SBC that the specific syslog tools use for generating call flow diagrams. So you need to select the correct format for the device and tool you are using if you want to be able to import the files at a later date.

Config File Example

Version 2 can use a configuration file to retain settings that will be applied when the tool boots. When settings are changed within the tool the values will be saved out to the config file. It is important to note that the config file needs to be manually created in order for the tool to start using it. This is deliberate as the config file is for advanced usage scenarios. To create the config file, simply create a text file in the same directory as the script is located and rename the file to "PowerSyslogServerSettings.cfg". Once the file exists the tool will start writing settings to the file. Below is an example of the file format:

SyslogPort="514"

Format="AudioCodes"

LogFile="C:\PowerSyslogFile.cfg"

KeepFiles="2000"

RollFile="20"

Note: Setting values must be surrounded in quote marks. 

How to configure a Sonus Gateway for Syslog Output

Sonus makes some of the most popular Lync Gateways on the market, so I have chosen to use them as an example of how to set up a device to output syslog. Power Syslog Server will work with any other UDP based syslog client as well though, so feel free to use it with other devices too.

Remote Log Servers:

Setup your device to output syslog to the server you are running Power Syslog Server on.  

Global Log Level: If your subsystems are set to “Default” logging level then this setting will be applied to them. This is also the level it will log for all services that are not specified in Subsystems. You will usually set this to a low value like “Error” or “Warning” to avoid log flooding.

Log Destination: The server with the Power Syslog Server running on it.

Port: 514                 

Protocol: UDP

Log Facility: local0

Enabled: Yes

Important Note: When you're finished debugging remember to Disable the syslog output. Otherwise the device will continue to output syslog data over the network, which can be a significant amount of unnecessary overhead for your device, network and server. 

Subsystems:

Then enable the Subsystems as required:

Subsystem: Set the specific Subsystem that you would like to have logged to the syslog output. For troubleshooting call flows and SIP messaging the “SIP Stack Service”, “Common Call Control” (for ISDN translation tables), “Call Routing Service” (for SIP translation tables), and "ISDN Protocol" (for E1 integrations) are useful subsystems to configure here.

Log Level: Set the required Log Level.

Log Destination: The Remote Log Server we created in the first step.

Debugging Log Files in LX Tool

Once you have captured your syslog files using the Power Syslog Server on the server on site you may want to do further call flow debugging using the Sonus LX tool (which can offer you decoded call flows for both SIP and ISDN calls providing your syslog contrains "ISDN Protocol" DEBUG and "SIP Stack Service" DEBUG logging).

To import the file into the LX tool, simply take one of the log files that the Power Syslog Server created and drag it into the LX tool window (or use File->Open). When you do this the LX tool will break the syslog file down into the individual call flows that were captured in the log. Here is an example:

Sonus LX Tool

By double clicking on a call in the "Calls" tab at the bottom of the screen you can get further details on each call flow (including ISDN decoding!):

Sonus LX Tool - Call Flow

Note: The LX Tool is a tool orginally created by NET (which was subsequently acquired by Sonus). To get a copy of the software go to the Sonus Salesforce portal and select "Software Downloads" then select "LX" from the Products list. If you don't have access to the Portal, speak to your Sonus representative to get a copy of the software.

AudioCodes Syslog Viewer

AudioCodes also have a nice Syslog Viewer Tool that can be used with the AudioCodes range of SBCs. The tool has a very nice call flow viewer which gives you a ladder diagram of SIP messages per call which allows you to click on the SIP message to see its contents.

I have found this tool to be much quicker and easier to use in comparison with the Sonus LX tool for troubleshooting SIP related call flow issues. The tool also can accept inputs from multiple devices at once and will put each syslog input into different tabs on the main screen. Using version 2 of Power Syslog Server you can output files into a format that the AudioCodes Syslog Viewer can import and display call flows and multiple device tab windows.

Example Display/Log Filters

Power Syslog Server includes a feature that allows you to filter (using regular expressions) what lines of syslog get displayed on the screen and logged to file. The reason for allowing for having a separate Display Filter and Log Filter is to help you when troubleshooting in real time. By this I mean that you can configure a very specific Display Filter to allow you to see only the messages you want to see for a specific issue and a more general Log File Filter so you can capture more detailed logs to review later in order to pinpoint the exact cause of the issue. Below are some examples of how you can use these filters when troubleshooting issues:

Show Only SIP Messaging

When you are running SIP Stack Service logging at a DEBUG level the Sonus gateway will output all of the SIP messaging that is traversing it. This can be very useful when you need to know what error messages are being sent by the Carrier SIP network or Lync when a call fails.

Example Filter (without quote marks): “sip:”

Example Output:

192.168.0.20 <135>[2014-09-16 00:57:02,709]  287 0002

OPTIONS sip:ux1000lab.mylynclab.com SIP/2.0

FROM: <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab

TO: <sip:ux1000lab.mylynclab.com>

CSEQ: 9993 OPTIONS

CALL-ID: 87a0bbd93e7f4e33a2c87ff8bbccd3d7

MAX-FORWARDS: 70

VIA: SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa

CONTACT: <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;maddr=192.168.0.96>

CONTENT-LENGTH: 0

USER-AGENT: RTCC/5.0.0.0 MediationServer

192.168.0.20 <135>[2014-09-16 00:57:02,718]  322 0001

SIP/2.0 200 OK

Allow: INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER

Call-ID: 87a0bbd93e7f4e33a2c87ff8bbccd3d7

Content-Length: 0

CSeq: 9993 OPTIONS

From:  <sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab

Server: SONUS SBC1000 3.0.2v270 Sonus SBC

Supported: replaces,update,100rel

To:  <sip:ux1000lab.mylynclab.com>;tag=aedb006-3ef64

Via: SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa

192.168.0.20 <135>[2014-09-16 00:57:04,827]  393 0003

OPTIONS sip:siptrunk.aapt.com.au:5060 SIP/2.0

Allow: INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER

Call-ID: call-71280200-0000-0010-1101-0@10.237.176.6

Content-Length: 0

CSeq: 132654 OPTIONS

From:  <sip:Anonymous@10.237.176.6:5060>;tag=aedb006-1

Max-Forwards: 70

Supported: replaces,update,100rel

To:  <sip:Anonymous@siptrunk.aapt.com.au:5060>

User-Agent: SONUS SBC1000 3.0.2v270 Sonus SBC

Via: SIP/2.0/UDP 10.237.176.6:5060;branch=z9hG4bK-UX-0aed-b006-40c88

Show Output Relating to Transformation and Route Rules

This can be extremely useful for troubleshooting what transformation rules a call is using and what routing rule it has chosen.

Example Filter (without quote marks): “regex match|transformation|route request”

Note: You need to be logging at DEBUG level for “Common Call Control” (for ISDN translation tables) and the “Call Routing Service” (for SIP translation tables) for this to work.

Example Output:

192.168.0.20 <134>[2014-09-16 00:51:13,126] 1160 0097 com.sonus.sbc.route INFO (callrouter.cpp:2193) - Handling route request.

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1163 0094 com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Testing Calling Party Rule (13.1(4)).

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1164 0093 com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCallingSubNumber" field for "^(9999113\d{2})$" (updated "^(9999113\d{2})$") with input of ""

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1165 0092 com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry 4 digit to E.164 (13.2(1)).

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1166 0091 com.sonus.sbc.route DEBUG (translation.cpp:653) - Successful regex match of "tfCalledNumber" field for "^(45\d{2})$" (updated "^(45\d{2})$") with input of "4501"

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1168 008f com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Full National to Lync (13.3(2)).

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1169 008e com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCalledNumber" field for "^0(3958245\d{2})$" (updated "^0(3958245\d{2})$") with input of "+61395824501"

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1170 008d com.sonus.sbc.route DEBUG (translation.cpp:1332) - Performing OPTIONAL transformation using entry Local to Lync (13.4(3)).

192.168.0.20 <135>[2014-09-16 00:51:13,127] 1171 008c com.sonus.sbc.route DEBUG (translation.cpp:649) - Failed regex match of "tfCalledNumber" field for "^(958245\d{2})$" (updated "^(958245\d{2})$") with input of "+61395824501"

192.168.0.20 <134>[2014-09-16 00:51:13,127] 1172 008b com.sonus.sbc.route INFO (callrouter.cpp:2396) - Successful route request with entry Analog to Lync (5.1(3))

Show Only Syslog Lines Related to a Specific Phone Number

This can be useful if you know a users telephone number and you only want to see messages that relate to them.

Example Filter (without quote marks): “+61399995555”

The Wrap Up

So there you have it, another tool for the kit bag. I hope you like it and find it useful, I know it’s already got me out of a few close calls. If you find any bugs or have any feature requests feel free to drop me a line.