“My Kingdom for a free and simple syslog server!” –
Anonymous System Administrator
So I don’t know about you, but I can’t remember how many
times I have got to the point of having to troubleshoot an issue with a Sonus
gateway and suddenly remembering I need a Syslog server to get logging out of
the box. At this point I usually go and ask Google politely “Google, can you
please point me in the direction of a free, and simple, syslog server that I
can run without installing a bunch of malware and other rubbish on this nice
customer’s server?” At this point Google usually responds “No, I cannot.
However, here is a syslog server that requires you install SQL, IIS, and
fifteen other dependent services as well as being crippled unless you pay $14.99
per month to a Russian guy name Vlad via this popup window that displays in the
middle of the screen every 5 minutes. Also, here’s a Yahoo browser search bar
for your trouble.”
This is not an ideal situation… So as usual, I just decided
to build it myself. In doing this I sat down and thought about the things I
wanted in a simple syslog server, and came up with this list:
- It needs to have no installation process, and leave no trace once removed from a server, as it will be run on customers' servers in a lot of cases.
- It needs to have a display where I can see the messages coming in in real time.
- The messages being displayed must be able to be paused and reviewed, so I can check if a specific event has happened yet.
- The messages window must be able to be cleared so that I can start fresh when trying to troubleshoot a fault.
- The syslog server needs to be able to log to file. Ideally the files should be able to be opened in Sonus LX tool so that further message debugging can be done easily.
- The syslog server needs to be able to roll the log files once they get to a specific size (so they can be emailed, etc).
- The syslog server should only keep a specific number of these log files so that the server’s hard disk does not get filled with log files.
- Both the display and log files should be able to be filtered to display only information that I want to see. For example, only show lines with a specific phone number in them, or only show me SIP messages. These filters should be independent so that you can view the filtered information on screen whilst more detailed information is getting logged to file for further review and troubleshooting later.
Based on these requirements I figured it would be very cool
to write the server in Powershell, as this allows for absolutely no
installation and can be run on any Windows machine you are likely to run into. How
hard could it be?
<Insert training montage>
SMASH CUT:
EXT. TRAINING MONTAGE - THE
STAIRS AT THE FRONT OF THE PHILADELPHIA MUSEUM OF ART- DAY
A man in a sweaty hoody
runs to the top of a large set of stairs carrying a tablet based productivity
device that he is furiously typing on. A large group of the town’s population
is also running after him in a large pack for no apparent reason. Upon reaching
the top of the steps he punches the air and launches the tablet into the sky.
The tablet hits the concrete and smashes into a million pieces. He falls to the
ground and screams towards the sky.
MAN
Nooooooo!
I should have backed up to the cloud, the cloud I tells ya.
Okay okay, let’s cut to the chase. I did it, and now you too
can syslog with me into the sunset.
Power Syslog Server
Version 1.0 Features:
- Zero installation.
- Real time log display (Approximately 1000 lines).
- Copy the displayed text with the Copy Text button. This is useful for more in depth analysis in your favourite notepad software.
- Rolling log files based on file size and number of files to keep.
- Clear display and Pause display functions.
- Filter real-time display logging with regular expression.
- Filter logging to file with regular expression.
- Open firewall for Syslog Server port with the click of a button. If you are not seeing any syslog output in the Power Syslog Server display log then try pressing the Open Firewall button.
- Server listening port can be changed by creating a config file (PowerSyslogServerSettings.cfg) in the same directory as the script. The config file needs to have text in it in the following format "SysLogPort=514". This allows you to maintain the integrity of the code signing by not directly editing the script file.
Version 2.0 Update:
- Added output formatting options to work with Sonus LX tool and AudioCodes Syslog Viewer tool (Commonly used Skype for Business syslog tools used with SBC devices).
- In version 2 if you create a config file named "PowerSyslogServerSettings.cfg" in the same directory as the tool it will use the config file to save all of its settings. The SyslogPort="514" setting remains a hidden setting that can still be used in the config file to change the listening port number.
- UDP socket code has been made more robust to deal with errors when the listening port is being used by another app.
- Changed the font to Courier New for fixed width goodness.
- Fixed issue with rolling files in folders including "." in name and faster processing.
- Fixed Sonus LX output formatting to only have LF and not CRLF.
- Increased socket buffer and tuned threading to fix dropped packet issues and double writing of some lines.
- Added disable display checkbox to increase performance when display is not required.
Version 2.0 – Output formats
Version 2.0 of Power Syslog Server now gives you the option
to add additional prefix formatting to the start of each line of syslog output. From the
research I have done the format of output from each syslog server varies
greatly and contains items such as date/time, text based priority field
interpretation (ie. The <135> value at the start of syslog messages sent
on the wire) and IP Address of the server that sent the message.
The reason that these prefixes are important is that if you want to
import the file output back into a tool like Sonus LX or AudioCodes Syslog
Viewer to generate call flow diagrams or other features the file needs to be in
a format that these tools can interpret. So in order to achieve this, the Format dropdown box
has been added in version 2. The Format setting will alter the outputs into the required
format for Sonus LX or AudioCodes syslog tools. In addition to these specific tool formats, some other generic prefix formats have been added which will make the output files easier for
humans to read.
Output Formats
Format
|
Example
Prefix Format
|
Comment
|
None
|
<No Prefix>
|
Output syslog in the exact format that it was sent
from the device in.
|
AudioCodes
|
"17:50:17.588 10.20.2.170 local0.notice"
|
Output syslog in AudioCodes Syslog Viewer tool format.
|
SonusLX
|
"10.20.1.150:53434
<==>"
|
Output syslog in the same format as the Sonus LX
tool.
|
Level
|
"Local0.Debug"
|
Prefix the syslog with the Facility and Severity levels.
|
DateTime
|
"2011-10-11
15:00:02.123"
|
Prefix the syslog with the date and time.
|
DateTimeLevel
|
"2011-10-11 15:00:02.123
Local0.Debug"
|
Prefix the syslog with Date/Time and Facility/Severity.
|
DateTimeLevelIP
|
"2011-10-11
15:00:02.123 Local0.Debug
192.168.0.100"
|
Prefix the syslog with Date/Time,
Facility/Severity, and IP Address of the device.
|
Note: Sonus LX tool
cannot open AudioCodes files and AudioCodes syslog tool cannot open LX files. This is because there are special lines of output generated by each brand of SBC that the specific syslog tools use for generating call flow diagrams. So you need to select the correct format for the device and tool you are
using if you want to be able to import the files at a later date.
Config File Example
Version 2 can use a configuration file to retain settings that will be applied when the tool boots. When settings are changed within the tool the values will be saved out to the config file. It is important to note that the config file needs to be manually created in order for the tool to start using it. This is deliberate as the config file is for advanced usage scenarios. To create the config file, simply create a text file in the same directory as the script is located and rename the file to "PowerSyslogServerSettings.cfg". Once the file exists the tool will start writing settings to the file. Below is an example of the file format:
SyslogPort="514"
Format="AudioCodes"
LogFile="C:\PowerSyslogFile.cfg"
KeepFiles="2000"
RollFile="20"
Note: Setting values must be
surrounded in quote marks.
How to configure a Sonus Gateway for Syslog Output
Sonus makes some of the most popular Lync Gateways on the market, so I have chosen to use them as an example of how to set up a device to output syslog. Power Syslog Server will work with any other UDP based syslog client as well though, so feel free to use it with other devices too.
Remote Log Servers:
Setup your device to output syslog to the server you are running Power Syslog Server on.
Setup your device to output syslog to the server you are running Power Syslog Server on.
Global Log Level: If your subsystems
are set to “Default” logging level then this setting will be applied to them. This
is also the level it will log for all services that are not specified in
Subsystems. You will usually set this to a low value like “Error” or “Warning”
to avoid log flooding.
Log Destination: The server with the Power Syslog Server running on it.
Port: 514
Protocol: UDP
Log Facility: local0
Enabled: Yes
Important Note: When you're finished debugging remember to Disable the syslog output. Otherwise the device will continue to output syslog data over the network, which can be a significant amount of unnecessary overhead for your device, network and server.
Important Note: When you're finished debugging remember to Disable the syslog output. Otherwise the device will continue to output syslog data over the network, which can be a significant amount of unnecessary overhead for your device, network and server.
Subsystems:
Then enable the Subsystems as required:
Then enable the Subsystems as required:
Subsystem: Set
the specific Subsystem that you would like to have logged to the syslog output.
For troubleshooting call flows and SIP messaging the “SIP Stack Service”, “Common Call Control” (for ISDN translation tables), “Call Routing Service” (for SIP translation tables), and "ISDN Protocol" (for E1 integrations) are useful subsystems to configure here.
Log Level: Set
the required Log Level.
Log Destination:
The Remote Log Server we created in the first step.
Debugging Log Files in LX Tool
Once you have captured your syslog files using the Power Syslog Server on the server on site you may want to do further call flow debugging using the Sonus LX tool (which can offer you decoded call flows for both SIP and ISDN calls providing your syslog contrains "ISDN Protocol" DEBUG and "SIP Stack Service" DEBUG logging).
To import the file into the LX tool, simply take one of the log files that the Power Syslog Server created and drag it into the LX tool window (or use File->Open). When you do this the LX tool will break the syslog file down into the individual call flows that were captured in the log. Here is an example:
Sonus LX Tool |
By double clicking on a call in the "Calls" tab at the bottom of the screen you can get further details on each call flow (including ISDN decoding!):
Sonus LX Tool - Call Flow |
Note: The LX Tool is a tool orginally created by NET (which was subsequently acquired by Sonus). To get a copy of the software go to the Sonus Salesforce portal and select "Software Downloads" then select "LX" from the Products list. If you don't have access to the Portal, speak to your Sonus representative to get a copy of the software.
AudioCodes Syslog Viewer
AudioCodes also have a nice Syslog Viewer Tool that can be used
with the AudioCodes range of SBCs. The tool has a very nice call flow viewer which
gives you a ladder diagram of SIP messages per call which allows you to click
on the SIP message to see its contents.
I have found this tool to be much quicker and easier to use
in comparison with the Sonus LX tool for troubleshooting SIP related call flow
issues. The tool also can accept inputs from multiple devices at once and will
put each syslog input into different tabs on the main screen. Using version 2
of Power Syslog Server you can output files into a format that the AudioCodes
Syslog Viewer can import and display call flows and multiple device tab
windows.
Example Display/Log Filters
Power Syslog Server includes a feature that allows you
to filter (using regular expressions) what lines of syslog get displayed on the
screen and logged to file. The reason for allowing for having a separate
Display Filter and Log Filter is to help you when troubleshooting in real time.
By this I mean that you can configure a very specific Display Filter to allow
you to see only the messages you want to see for a specific issue and a more
general Log File Filter so you can capture more detailed logs to review later in order to pinpoint the
exact cause of the issue. Below are some examples of how you can use these
filters when troubleshooting issues:
Show Only SIP Messaging
When you are running SIP Stack Service logging at a DEBUG level the Sonus gateway will output all of the SIP messaging that is traversing it. This can be very useful when you need to know what error messages are being sent by the Carrier SIP network or Lync when a call fails.
Example Filter (without quote
marks): “sip:”
Example Output:
192.168.0.20
<135>[2014-09-16 00:57:02,709] 287
0002
OPTIONS
sip:ux1000lab.mylynclab.com SIP/2.0
FROM:
<sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab
TO: <sip:ux1000lab.mylynclab.com>
CSEQ:
9993 OPTIONS
CALL-ID:
87a0bbd93e7f4e33a2c87ff8bbccd3d7
MAX-FORWARDS:
70
VIA:
SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa
CONTACT:
<sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;maddr=192.168.0.96>
CONTENT-LENGTH:
0
USER-AGENT:
RTCC/5.0.0.0 MediationServer
192.168.0.20
<135>[2014-09-16 00:57:02,718] 322
0001
SIP/2.0
200 OK
Allow:
INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER
Call-ID:
87a0bbd93e7f4e33a2c87ff8bbccd3d7
Content-Length:
0
CSeq:
9993 OPTIONS
From:
<sip:2013ENTFE003.mylynclab.com:5068;transport=Tcp;ms-opaque=152721d992435f69>;epid=B3F80C5FC7;tag=fb568a1fab
Server:
SONUS SBC1000 3.0.2v270 Sonus SBC
Supported:
replaces,update,100rel
To: <sip:ux1000lab.mylynclab.com>;tag=aedb006-3ef64
Via:
SIP/2.0/TCP 192.168.0.96:51823;branch=z9hG4bK96df5daa
192.168.0.20
<135>[2014-09-16 00:57:04,827] 393
0003
OPTIONS
sip:siptrunk.aapt.com.au:5060 SIP/2.0
Allow:
INVITE, ACK, CANCEL, BYE, UPDATE, NOTIFY, OPTIONS, REFER, REGISTER
Call-ID:
call-71280200-0000-0010-1101-0@10.237.176.6
Content-Length:
0
CSeq:
132654 OPTIONS
From:
<sip:Anonymous@10.237.176.6:5060>;tag=aedb006-1
Max-Forwards:
70
Supported:
replaces,update,100rel
To:
<sip:Anonymous@siptrunk.aapt.com.au:5060>
User-Agent:
SONUS SBC1000 3.0.2v270 Sonus SBC
Via:
SIP/2.0/UDP 10.237.176.6:5060;branch=z9hG4bK-UX-0aed-b006-40c88
Show Output Relating to Transformation and Route Rules
This can be extremely useful for troubleshooting what transformation rules a call is using and what routing rule it has chosen.
Example Filter (without quote marks): “regex match|transformation|route request”
Note: You need to be logging at DEBUG level for “Common Call Control” (for ISDN translation tables) and the “Call Routing Service” (for SIP translation tables) for this to work.
Example Output:
192.168.0.20
<134>[2014-09-16 00:51:13,126] 1160 0097 com.sonus.sbc.route INFO
(callrouter.cpp:2193) - Handling route request.
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1163 0094 com.sonus.sbc.route DEBUG
(translation.cpp:1332) - Performing OPTIONAL transformation using entry Testing
Calling Party Rule (13.1(4)).
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1164 0093 com.sonus.sbc.route DEBUG
(translation.cpp:649) - Failed regex match of "tfCallingSubNumber"
field for "^(9999113\d{2})$" (updated "^(9999113\d{2})$")
with input of ""
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1165 0092 com.sonus.sbc.route DEBUG
(translation.cpp:1332) - Performing OPTIONAL transformation using entry 4 digit
to E.164 (13.2(1)).
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1166 0091 com.sonus.sbc.route DEBUG
(translation.cpp:653) - Successful regex match of "tfCalledNumber"
field for "^(45\d{2})$" (updated "^(45\d{2})$") with input
of "4501"
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1168 008f com.sonus.sbc.route DEBUG
(translation.cpp:1332) - Performing OPTIONAL transformation using entry Full
National to Lync (13.3(2)).
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1169 008e com.sonus.sbc.route DEBUG
(translation.cpp:649) - Failed regex match of "tfCalledNumber" field
for "^0(3958245\d{2})$" (updated "^0(3958245\d{2})$") with
input of "+61395824501"
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1170 008d com.sonus.sbc.route DEBUG
(translation.cpp:1332) - Performing OPTIONAL transformation using entry Local
to Lync (13.4(3)).
192.168.0.20
<135>[2014-09-16 00:51:13,127] 1171 008c com.sonus.sbc.route DEBUG
(translation.cpp:649) - Failed regex match of "tfCalledNumber" field
for "^(958245\d{2})$" (updated "^(958245\d{2})$") with
input of "+61395824501"
192.168.0.20
<134>[2014-09-16 00:51:13,127] 1172 008b com.sonus.sbc.route INFO
(callrouter.cpp:2396) - Successful route request with entry Analog to Lync
(5.1(3))
Show Only Syslog Lines Related to a Specific Phone Number
This can be useful if you know a users telephone number and you only want to see messages that relate to them.
Example Filter (without quote marks): “+61399995555”
The Wrap Up
So there you have it, another tool for the kit bag. I hope you like it and find it useful, I know it’s already got me out of a few close calls. If you find any bugs or have any feature requests feel free to drop me a line.