Lync 2013 Centralised Logging Tool

Centralised logging is a very powerful service. However, because it’s only accessible through Powershell commands, it can be a rather daunting alternative to the old Lync Logging Tool, especially when you’re trying to troubleshoot an issue.

So what can be done about this? Well, Randy Wintle over at the UC Made Easy blog had the great idea of making a GUI interface for the Lync 2013 Preview release, which was based on the ‘ClsController.exe’ command line tool. I like this idea a lot, so now that 2013 is released, I decided to give Randy’s tool a major “Backend Lift” (different than that offered by Kim Kardashian’s plastic surgeon…) and add a bunch of new capabilities.

Centralised Logging Tool Features

Centralised Logging Tool features:

• The tool's backend uses Powershell commands, instead of ClsController.exe commands.
• The tool has the ability to manage logging on multiple Pools at once. When the tool first loads it will learn the current state of all the Pools. The UI can show the state of each pool as they are clicked on in the Pools listbox and only allow the user to execute functions that are available in that state (eg. Start and Stop buttons will be disabled if the current state of the pool doesn’t allow you to use them).
• When the tool loads it will query the Lync servers for the status of running logging scenarios, and the tool will display the current state of each pool as it is selected in the pools list. This means that you can use the tool to learn what logging is currently running on each pool.
• The scenarios list is dynamic, so Custom Scenarios will also be included in the scenarios list along with the default scenarios.
• All pool types, including Persistent Chat pools are listed. If Mediation server is co-located with the front end server it will not be added to the list, but if it’s on a separate physical server it will be listed.
• The tool has the ability to have AlwaysOn and Scenario logging enabled at the same time. AlwaysOn logging has its own button and status messages.
• The components filter list can either show all the components available for logging on the system, or list only the components for the specific scenario that is currently being logged (or just logged) on a pool.
• The Logging duration time can be specified when you Start logging, and running scenarios can have their Duration updated with the new Update button. By default, logging will run for 4 hours and then automatically stop. So if you want to do a long running log, or a much shorter log you can specify this when you start logging, or you have the option to Change the duration of a log whilst it’s running. The format of Logging duration is as follows: <days.hours:minutes> (eg. “0.04:00”) .
• You can now select either MatchAll, or MatchAny settings. These settings affect the way multiple filters are applied when Searching/Exporting logs. If MatchAll is used, then the filters execute in a logical ‘AND’ fashion, where all the filters must match to return results. If MatchAny is selected, then the filters apply in a logical ‘OR’, which means if any of the filters match the result will be returned.
• The Call-Id, IP Address, Phone and URI filters have separate text boxes, so these filters can be used in conjunction with each other (and the MatchAll/MatchAny setting) for more precise filtering.
• The Components listbox is Multi-Selectable, offering export filtering by one or many component types at once.
• Start and End time filter is now included as a filter option. By default the Search command will filter through only the last 30 minutes of logs to return a result. If you need to capture more than this then you can used the Start and End time settings. I have made the defaults in these boxes display the range of 1 hour as a starting point. It’s best that you keep the date/time format in the format shown.
• The logging folder can be selected using a Folder Browse dialog to having to type the location.
• Analyse Log button allows you to choose a log file to open directly in Snooper, to save you having to manually open Snooper and import the logs.
• Interface is now written in Australian/British English :)

The Details

The Centralised Logging Tool is Stateful on a per pool basis. By that I mean that it will learn the status of all pools when it loads and it will maintain the state as you Start and Stop services. The process of querying the server (Show-CsClsLogging) for the status of the Centralised Logging Service can take several seconds to execute. This means the Controller has to query all of the Agents and then wait for them to respond before it can report back about the status of the pools. As a result, I tried to minimise the number of times the tool had to re-sync its state with the server. So, it will only request updates on the status of the Agents upon receiving an error response from a command execution (ie. A Start/Stop request). When this happens, it will fully refresh all of the Pools logging status.

I give you the new Centralised Logging Tool UI:

 Update ( 6/5/2015 ) 

1.01 Enhancements:

• Fixed the format of the Start/Stop export date format (text fields filled when the script loads) to the region setting of the machine. This was previously hardcoded to Australian format in v1.00. Lync expects the date format in the format of the region of the computer (as set in Control Panel) for the Search command. Thanks to John Crouch for reporting this not working in the US in v1.00.

1.02 Enhancements:

• Added SBA's to the Pool list. They also have the centralised logging service. - Nice pickup by John Crouch.
• Collocated Persistent Chat servers now don't get added to the pool list.

1.03 Enhancements:

• Pools listbox now is in alphabetical order.
• Scenarios listbox is now in alphabetical order.
• Components listbox is now in alphabetical order.
• Annoying counting in Powershell window when the script loads has now been removed.
• Added prerequisite check for Snooper and debugging tools when script loads.
• Script is now signed.
• Using UP and DOWN arrow keys on pools listbox now updates GUI correctly.
• If pool does not have a status then an error ("ERROR: Pool status not found. Check that CLS is running correctly on pool.") will be displayed in the GUI.
• A status message will now display to indicate to select a Pool when no pool is selected in the Pools listbox.
• Fixed error that is displayed when Cancel button on browse dialog is clicked.
• Fixed issue with load files in Snooper that have spaces in them.

1.04 Enhancements:

• Now supports Skype for Business!

1.05 Enhancements:

• Now checks for Skype for Business Snooper location.
• Changed the default trace saving location to C:\Tracing\

1.06 Enhancements:

• Now supports Skype for Business 2019. I got requests to release a update for 2019 as an alternative to the existing ClsLogger tool provided by Microsoft. So here it is!

DOWNLOAD HERE

When starting out with the tool you might easily get confused with how the filtering of captures works. As I detailed in my last post about Centralised Logging, logging captures are filtered in the following way:

The scenario filter section shown above will either be a default scenario (capturing only the pre-defined components) that’s are already defined as part of Lync 2013, or you can build your own custom scenarios that log only the components that you are interested in.

The Agent Search filter shown above allows you to run post-capture filters on the data, so you only end up with the log information you want. This is done in the Centralised Logging Tool using the Export filters, and the Export button.

The Centralised Logging Tool has these two sections broken up as follows:

This basically means that the settings outlined in red work in conjunction with the Export button, and the Scenario list works in conjunction with the Start tracing button.

 

Export Filters

The export filters function as detailed below:

Start and End Times: The Start and End filter will export only items logged between the times specified.

Components Filter: Component filter can be used to select one or more components and the CLS service will export only logs relating to the components specified.

Logging Level: Logging level filters will return all levels up to the level you select; for example ‘Warning’ will retrieve Fatal, Error, and Warning results. The hierarchy is:

1. Fatal
2. Error
3. Warning
4. Info
5. Verbose
6. Debug
7. All

URI Filter: The URI Filter setting works as long as you use a full SIP URI, for example john.woods@domain.com, or sip:john.woods@domain.com

IP Filter: Doesn’t appear to work. Lync returns no results for IP addresses. Last checked on version 5.0.8308.872.

CallID Filter: Enter a SIP CallID and any SIP message that matches that CallID will be captured by the filter. In addition to this; any messages with Trace-Correlation-Id's that match that of the SIP message will also be exported.

Example Filter: a1ffa6a5ef7949079ae4b34e7c67293f

...

To: <sip:holly.hunt@mylynclab.com>;tag=DB120080

Call-ID: a1ffa6a5ef7949079ae4b34e7c67293f

CSeq: 2 SUBSCRIBE

...

Phone Filter: The Phone Export Filter matches against the TO and the FROM fields of SIP messages. So this filter is useful for finding PSTN routed calls to specific numbers. example:

Example Filter:  +61395829999

..

<<<<<<<<<<<<Incoming SipMessage c=[<SipTcpConnection_210F5C0>], 192.168.0.97:5068<-192.168.0.20:43904

INVITE sip:+61395829999@2013ENTFE004.mylynclab.com:5068;user=phone SIP/2.0

FROM: <sip:11300@192.168.0.20:5066;user=phone>;tag=aedb006-a46ce

TO: <sip:+61395829999@2013ENTFE004.mylynclab.com:5068;user=phone>

..

Note: The number doesn’t have to be in E.164 format, it just needs to match the number in the TO or FROM field.

SIP Contents: Enter a piece of text from a SIP or SDP Message and any SIP message that matches contain that text will be captured by the filter. In addition to this; any messages with Trace-Correlation-Id's that match that of the SIP message will also be exported.

Example Filter: rtcp:53115

v=0

o=- 29 1 IN IP4 192.168.0.97

s=session

c=IN IP4 192.168.0.97

b=CT:1000

t=0 0

m=audio 53114 RTP/AVP 8 101 13

c=IN IP4 192.168.0.97

a=rtcp:53115

a=label:Audio

a=sendrecv

a=rtpmap:8 PCMA/8000

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-16

a=rtpmap:13 CN/8000

Known Issue: Errors when Stopping Logging Service

Whilst testing the tool I’ve had some issues from time to time when calling Stop on a running scenario (both AlwaysOn and other Scenarios). The error received looks like this:

Failed on 1 agents

Agent - 2013ENTFE004.domain.com, Reason - Error code - 20003, Message - Unknown error returned by Wpp native assembly - 1168. Please refer CLS logs for details.

Unfortunately, the CLS log didn’t shine any light on why this is occurring (the logs report the same error as shown above with no additional info). I have made the tool so it’s smart enough to recognise these errors and query the server for the state of the pool again, to re-sync the status of all the pools. I have found that if you wait a few minutes (maybe 5 minutes) and then try to stop the logging scenario again it tends to work (or just keep trying till it does). I should stress that this isn’t a problem with the tool, as the issue happens when using Powershell directly as well. So I assume it’s a bug in the Agent side of the code.

If anyone knows how to avoid this error, let me know!

 

Notes on 2013 Snooper

Snooper is a tool that was available in Lync 2010, which provides the ability to analyse log files that are generated by the Lync Logging Tool. Snooper 2013 now also provides the ability to analyse Lync Centralised Logging Service log files.

Whilst you can also open these log files in a text editor, Snooper makes reading logs far easier. So, to make life easier I decided to put a button in the Centralised Logging Tool that will launch Snooper directly. Here it is:

When you click this button you will be shown a File Dialog that allows you to select which log file you want to open. Select the file you want to open and Snooper will open the file.

Note: If your Snooper tool is in a non-standard location you can edit a global variable in the script to point to whatever location you like:

$script:snooperLocation = "C:\Program Files\Microsoft Lync Server 2013\Debugging Tools\Snooper.exe"

If you want to change the location that the log files are saved to, edit this global variable:

$script:pathDialog = "C:\Lync2013Tracing\"

Some other good news about Snooper for Lync 2013 is that it has some cool new features, one of which is the Call Diagram button:

When you have messages from a SIP dialog selected in Snooper, and press this button, it will launch a diagrammatic representation of that particular SIP Dialog, as shown here:

Note: You have to be on the Messages tab of Snooper to launch the call flow analyser. If you are on the Trace tab you will receive a “This message is not eligible for callflow” error.

Pretty cool, huh!

 

Wrap Up

I have spent quite a bit of time working on the tool and tweaking it to work for most functions that I can see people wanting to use it for. If I’ve missed something and you have an awesome idea for additional functionality, let me know. Other than that, happy logging!

Read more →

Lync 2013 Centralised Logging Deep Dive

Overview

Lync 2013 brings with it a new form of logging that differs greatly from the logging tools previously supplied in Lync 2010. In Lync 2010 the Lync Logging Tool offered the ability to log directly on a single server in a pool. As a result, in Lync 2010 logging across a pool of ten servers requires running the logging tool on multiple servers at once, which can be a cumbersome undertaking.   

Lync 2013 has new capability called the Centralised Logging Service. The new logging service consists of two main components, a Logging Controller, and a Logging Agent. The Logging Agent runs on each of the Lync Front End servers, under the service name “Lync Server Centralised Logging Service Agent”.

The service runs the ClsAgent.exe executable and listens for commands on the following ports: TCP 50001, TCP 50002, and TCP 50003. This can be seen using netstat:

When the agent receives a command, it sends messages to the defined components for tracing, and writes the trace logs to disk. It also reads the trace logs for its computer and sends the trace data back to the controller when requested.

The Second component used for centralised logging is the Centralised Logging Controller. The Centralised Logging Controller sends Start, Stop, Flush, and Search commands to the ClsAgents on all the servers in a specified pool. The Lync 2013 Preview release contained a command line tool called ‘ClsController.exe’ (which still exists in the RTM release) that could be used to control the Centralised Logging Agents on the Lync Front End servers. The release of Lync 2013 RTM has taken this a step forward offering Powershell commands that send commands through a Dynamic Link Library called “ClsControllerLib.dll”.

When search commands are sent, the resulting logs are returned to the ClsControllerLib.dll and aggregated. The controller is responsible for sending commands to the agent, receiving the status of those commands and managing the search log file data as it is returned from the agents, and aggregating the log data into a meaningful and ordered output set.

Here is a diagram of the relationship between the two Controller and Agents:

The documentation now available on TechNet states that two scenarios can be run on a given computer at any one point in time. What this actually means in practice is that you can run one default or custom scenario, in addition to AlwaysOn logging. Even if you have AlwaysOn logging turned off, you can’t run multiple other scenarios at once. If you are in a situation where you need to run multiple default scenarios at once, then you can create your own custom scenario that includes all of the logging providers used by the default scenarios you are trying to emulate (see the Custom Scenarios section of this post).

 

How Does Centralised Logging compare to Lync 2010 Logging Tool?

So, what are all these weird scenario names and how do they relate to my favourite Lync Logging Tool filters (SIPStack, S4, etc)? I’m glad you asked, and I made this table to help you understand what the new Scenarios are doing in the background. The following table shows the default scenario names used for centralised logging, and the components that are logged for each of these scenarios.

Lync 2010 Logging Tool

The third column in this table relates to the Components from the old Lync Logging Tool (which is incidentally still available for download for Lync 2013 in the Debugging tools install). The components list can be seen on the left hand side of the tool.

Logging Comparison Table

The table explains which components get used for each default logging scenario in Lync 2013 centralised logging.

Description	Centralised Logging Tool Scenario	Lync 2010 Logging Tool Components
AlwaysOn is a special scenario that can be enabled to log all the time. The AlwaysOn scenatio can run in conjunction with one other scenario.	AlwaysOn	AsMcu AcpMcu AVMCU AVMP BICommon BICOSMOS BIDATACOLLECTOR CAAServer Collaboration DataMCU DataMcuRuntime ExumRouting IMMCU InboundRouting InterClusterRouting McuInfra MediationServer OutboundRouting Routing_Data_Sync_Agent S4 ServerTransportAdaptor Sipstack LDM AppShareOoty RDPApiTrace RDPEncComTrace RdpApiTrace UserServices UDCAgent PNCHService TenantAdminUI HostedMigration Powershell UCWA ChatCommon ChatEndpoint ChatServer ChatCompliance ChatWebService XmppTGW XmppCommonLibrary XmppListener XmppRouting XmppTGWProxy Lyss StoreWeb TranslationApplication RgsClientsLib RgsCommonLibrary RgsDatastores RgsDiagnostics RgsHostingFramework RgsMatchMakingService CpsDiagnostics CpsHostingFramework JoinLauncher WebInfrastructure Infrastructure InternalCommon McxService UserPinService CertProvisioning BackupService RtcDbSyncAgent WebRelay ServerAgent
Media related logging	MediaConnectivity	MediaStack_AUDIO_AGC MediaStack_AUDIO_DRC MediaStack_AUDIO_ECHODT MediaStack_AUDIO_FAXDT MediaStack_AUDIO_HEALER MediaStack_AUDIO_NOISEDT MediaStack_AUDIO_VAD MediaStack_AUDIO_VSP MediaStack_AudioCodecs MediaStack_AudioEngine MediaStack_COMAPI MediaStack_COMMON MediaStack_Crossbar MediaStack_Crypto MediaStack_DebugUI MediaStack_DebugUI_AEC MediaStack_DEVICE MediaStack_MassConvertedTraces1 MediaStack_MediaManager MediaStack_PerFrame MediaStack_PerPacket MediaStack_QualityController MediaStack_RTCP MediaStack_RTP MediaStack_StreamingEngine MediaStack_TLS MediaStack_Transport MediaStack_VIDEO MediaStack_VOICEENHANCE
Application Sharing	ApplicationSharing	AsMcu AppShareOoty ServerTransportAdaptor RDPApiTrace RDPEncComTrace MCUInfra Collaboration S4
Audio and Video Conferencing Logging	AudioVideoConferencingIssue	AvMcu AvMP MCUInfra Collaboration S4
Hybrid Cloud voice deployments	HybridVoice	MediationServer S4 Sipstack OutboundRouting TranslationApplication
General Call logging	IncomingAndOutgoingCall	MediationServer S4 Sipstack TranslationApplication OutboundRouting InboundRouting UserServices
Voice Mail	VoiceMail	Sipstack ExumRouting InboundRouting
IM and Presence Logging	IMAndPresence	Sipstack UserServices
Address Book service logging	AddressBook	ABCommon ABServer ABServerIISModule Dlx
Device Update service logging	DeviceUpdate	DeviceUpdate DeviceUpdateHttpHandler
Lync Storage Service and Unified Contact Store logging	LYSSAndUCS	UserServices Lyss McuInfra
Centralised Logging Service logging	CLS	CLSAgent CLSCommon CLSController CLSControllerLib
Support Portal logging	SP	SupportPortal SPHelper SPModule SPSearchTool SPCLSTool CLSCommon CLSControllerLib
Web Access Server logging	WAC	DataMCURunTime DataMCU LDM Infrastructure WebInfrastructure InternalCommon
User Replicator Service (synchronization between AD DS and Lync Server)	UserReplicator	UserServices
Lync Online Migration	HostedMigration	HostedMigration Powershell WebInfrastructure
Monitoring and Archiving logging	MonitoringAndArchiving	UDCAgent
LILRLegacy	LILRLegacy	LogRetention LegalIntercept
LILRLYSS	LILRLYSS	LogRetention UDCAgent
Meeting logging	MeetingJoin	Collaboration S4 UserServices McuInfra JoinLauncher WebInfrastructure Infrastructure InternalCommon UCWA WebRelay
Response Group Service	RGS	RgsClientsLib RgsCommonLibrary RgsDatastores RgsDeploymentApi RgsDeploymentLibrary RgsDiagnostics RgsHostingFramework RgsMatchMakingService UserServices Collaboration S4 Sipstack
Call Park Service	CPS	CpsDiagnostics CpsHostingFramework UserServices Collaboration S4 Sipstack
XMPP service logging	XMPP	XmppTGW XmppCommonLibrary XmppListener XmppRouting XmppTGWProxy Collaboration S4 Sipstack
Conference Auto Attendant	CAA	CAAServer Collaboration S4 Sipstack UserServices
Dial in conferencing MCU	ACPMCU	AcpMcu Collaboration S4 SipStack
Authentication Logging	Authentication	SipStack UserServices WebInfrastructure UserPinService CertProvisioning
High Availability Disaster Recovery	HADR	UserServices PowerShell BackupService RtcDbSyncAgent
Powershell	Powershell	Powershell
FilterApps	FilterApps	IIMFilter ClientVersionFilter ServerAgent SipStack

What is AlwaysOn Logging?

AlwaysOn is a setting that can be made on a pool that forces the pool to be logging all the time. When you run a Show-CsClsLogging on your server you can see that AlwaysOn gets its own separate status shown in the output:

> Show-CsClsLogging

Success Code - 0, Successful on 1 agents

Tracing Status:

2013STDFE001.mylynclab.com (2013STDFE001 v5.0.8308.0) (AlwaysOn=No,Scenario=HostedMigration,Started=

4/6/2013 11:25:47 PM,By=2013STDPC\Administrator,Duration=0.04:00)

    2013STDFE001.mylynclab.com (2013STDFE001 v5.0.8308.0) (Same as pool)

This is because AlwaysOn functions as an everyday log that can be used to troubleshoot issues that might have happened during the day. The AlwaysOn logging scenario logs for all of the components available, at an Info level, as shown below:

> Get-CsClsScenario | Where-Object {$_.identity -like "Global/AlwaysOn"} | Select-Object provider | Select-Object -ExpandProperty provider

Name  : AsMcu

Type  : WPP

Level : Info

Flags : TF_COMPONENT,TF_PROTOCOL

Name  : AcpMcu

Type  : WPP

Level : Info

Flags : TF_COMPONENT

Name  : AVMCU

Type  : WPP

Level : Info

Flags : TF_COMPONENT,TF_PROTOCOL

Continued...

So, if you are running AlwaysOn already on a pool, then the only reason you would need to run another scenario at the same time would be to log at a higher level of detail (eg. Debug). Note, only one additional scenario can be logging at the same time as AlwaysOn is running.

If you run a Search-CsClsLogging (with no filters) on a scenario, at the same time as AlwaysOn was logging, you will receive the AlwaysOn logging in the output file as well. As a result, if you need to do very specific logging, you might want to turn off the AlwaysOn logging before you start the specific scenario you want to log for (or remember to use an appropriate filter when running Search-CsClsLogging to remove the unnecessary logs).

Logging Files

Temporary log files are written to the folder on each of the Lync servers with logging enabled. All log files will be placed in the folder specified in the Get-CsClsConfiguration command, as shown below:

> Get-CsClsConfiguration

Identity                      : Global

Scenarios                     : {Name=AlwaysOn, Name=MediaConnectivity, Name=ApplicationSharing, Name=AudioVideoConferencingIssue...}

SearchTerms                   : {Type=Phone;Inserts=ItemE164,ItemURI,ItemSIP,ItemPII, Type=URI;Inserts=ItemURI,ItemSIP,ItemPII, Type=CallId;Inserts=ItemCALLID,ItemURI,ItemSIP,ItemPII,

                                Type=ConfId;Inserts=ItemCONFID,ItemURI,ItemSIP,ItemPII...}

SecurityGroups                : {}

Regions                       : {}

EtlFileFolder                 : %TEMP%\Tracing

EtlFileRolloverSizeMB         : 20

EtlFileRolloverMinutes        : 60

TmfFileSearchPath             :

CacheFileLocalFolders         : %TEMP%\Tracing

CacheFileNetworkFolder        :

CacheFileLocalRetentionPeriod : 14

CacheFileLocalMaxDiskUsage    : 80

ComponentThrottleLimit        : 5000

ComponentThrottleSample       : 3

MinimumClsAgentServiceVersion : 6

Note: If you cut and paste this address into the file browser you will end up in a location like this:

C:\Users\Administrator.DOMAIN\AppData\Local\Temp\2\Tracing

Most likely this location is void of any logging files, which might leave you scratching you head. However, there is good reason for this: you’re logged in as a user on the server and as a result you’re getting taken to the temp folder of the user you’re logged in as.

Meanwhile, the Logging Service is logged in as the NetworkService (as are all Lync services), so when it accesses this address it ends up at the following folder:

C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\Tracing

The logs folder will have three different types of files in it: ETL, HDR and CACHE files. Currently running logging services are streaming to ETL files (eg. CLS_WPP_04-05-2013-13-34-46_1.etl). When the logging is stopped the ETL file content gets converted into an HDR and a CACHE file.

The HDR file contains information about the capture, eg:

AgentVersion=2.0

Starttime=5/04/2013 2:27:11 AM

Endtime=5/04/2013 2:30:26 AM

MostSevereLevel=3

ComponentNames=(Shared),SIPStack,UserServices

CorrelationIds=0,2446939738,2446939785,2899899976,2970877857,1910046407,2765731706,1926842570,947404259,3734898740,3285769719,2895070652,2630969344,1910046457,3397217402,3308872234,2765731776,1484520309,1137574355

The CACHE file is an actual logging file that contains the information that was logged. Sections of this file are in binary, and sections in clear text. The CACHE file cannot be directly opened in Snooper. In order to do this you must run the Search-CsClsLogging command to retrieve the correct format.

 Other Useful Centralised Logging Settings

There are a number of other useful settings within Get-CsClsConfiguration that might be of use to you when it comes to finely tuning your logging solution. Here are some settings that you might consider tweaking:

EtlFileRolloverSizeMB         : 20

EtlFileRolloverMinutes        : 60

TmfFileSearchPath             :

CacheFileLocalFolders         : %TEMP%\Tracing

CacheFileNetworkFolder        :

CacheFileLocalRetentionPeriod : 14

CacheFileLocalMaxDiskUsage    : 80

ComponentThrottleLimit        : 5000

ComponentThrottleSample       : 3

EtlFileRolloverSizeMB: Maximum size (in megabytes) that at event trace log file can reach before a new file is created. The default value is 20.

EtlFileRolloverMinutes: Maximum amount of time (in minutes) that can elapse before a new event log trace file is created. (This new file will be created even if the existing trace file has not reached the specified rollover size.) The default value is 60.

CacheFileLocalRetentionPeriod: Maximum number of days that cache files are retained locally. The default value is 14.

CacheFileLocalMaxDiskUsage: Maximum amount of disk space (percentage) that can be used for the cache files. The default value is 80.

Creating Custom Logging Scenarios

Let me first start by explaining in a little more depth what a scenario actually is. A scenario is basically a collection of one, or many, logging streams (in Microsoft speak; ‘components’). For example, if we were to take AddressBook scenario, which logs things to do with the Address Book Service. From the table earlier we can see that the components being logged for the AddressBook scenario include ABCommon, ABServer, ABServerIISModule, and Dlx. As might be obvious by their names, these services are all directly relating to the Address Book, whereas components like MediaStack_MediaManager have nothing to do with the AddressBook so we don’t include them in the scenario.

Put in a different way, a Scenario functions as a Pre-Capture filter that determines what is written to disk on the server running the Centralised Logging Agent Service. After this information has been logged to disk on the Agent server, the search-cslcslogging command can be used in conjunction with additional (Post-Capture) filter parameters to determine the logging output that is written to the final log file. Below is a diagram that describes the process:

 

Creating a custom Centralised Logging Scenario is quite simple: it’s a matter of first creating provider(s) and then creating a scenario from the providers:

Create a provider:

$provider = New-CsClsProvider -Name "AvMcu" -Type "WPP" -Level "Info" -Flags "All"

The New-CsClsProvider page on TechNet currently (April 2013) states that the provider Name parameter is a “Unique Name”. After reading this and looking at the command, I was left a little confused, because there are no other parameters that tell the provider what components it should be logging for. So, whilst the New-CsClsProvider command will let you create a provider with a Name of “MegaLoggingSuperLogLogger”, this isn’t a component name that means anything to the logging service. If you use this provider (with a unique name) and try and run it on an Agent it will Start successfully:

> Start-CsClsLogging -Scenario CUSTOMSCENARIO -pools 2013ENTFE004.mylynclab.com

Success Code - 0, Successful on 1 agents

Tracing Status:

2013ENTFE004.mylynclab.com (2013ENTFE004 v5.0.8308.0) (AlwaysOn=No,Scenario=CUSTOMSCENARIO,Started=4/04/2013 5:43:07 PM,By=2013ENT\Administrator,Duration=0.04:00)

2013ENTFE004.mylynclab.com (2013ENTFE004 v5.0.8308.0) (Same as pool)

It will say it’s logging when you run the Show command:

> Show-CsClsLogging

Success Code - 0, Successful on 6 agents

Tracing Status:

2013ENTFE004.mylynclab.com (2013ENTFE004 v5.0.8308.0) (AlwaysOn=No,Scenario=CUSTOMSCENARIO,Started=4/04/2013 5:43:07 PM,By=2013ENT\Administrator,Duration=0.04:00)

...

However, it will fail when you stop correctly run the Stop command:

> Stop-CsClsLogging -scenario CUSTOMSCENARIO -pools 2013ENTFE004.mylynclab.com

Failed on 1 agents

Agent - 2013ENTFE004.mylynclab.com, Reason - Error code - 20005, Message - Scenario 'CUSTOMSCENARIO' not currently running.

So choosing a “Unique Name” doesn’t actually appear to be the way to go… The Name parameter should actually be the name of a component. Here is a list of Component names to choose from:

Component Names
ABCommon	InternalCommon	PNCHService
ABServer	JoinLauncher	Powershell
ABServerIISModule	LDM	RdpApiTrace
AcpMcu	LegalIntercept	RDPEncComTrace
AppShareOoty	LogRetention	RgsClientsLib
AsMcu	Lyss	RgsCommonLibrary
AvMcu	McuInfra	RgsDatastores
AVMP	McxService	RgsDeploymentApi
BackupService	MediaStack_AudioCodecs	RgsDeploymentLibrary
BICommon	MediaStack_AudioEngine	RgsDiagnostics
BICOSMOS	MediaStack_AUDIO_AGC	RgsHostingFramework
BIDATACOLLECTOR	MediaStack_AUDIO_DRC	RgsMatchMakingService
CAAServer	MediaStack_AUDIO_ECHODT	Routing_Data_Sync_Agent
CertProvisioning	MediaStack_AUDIO_FAXDT	RtcDbSyncAgent
ChatCommon	MediaStack_AUDIO_HEALER	S4
ChatCompliance	MediaStack_AUDIO_NOISEDT	ServerAgent
ChatEndpoint	MediaStack_AUDIO_VAD	ServerTransportAdaptor
ChatServer	MediaStack_AUDIO_VSP	Sipstack
ChatWebService	MediaStack_COMAPI	SPCLSTool
ClientVersionFilter	MediaStack_COMMON	SPHelper
CLSAgent	MediaStack_Crossbar	SPModule
CLSCommon	MediaStack_Crypto	SPSearchTool
CLSController	MediaStack_DebugUI	StoreWeb
CLSControllerLib	MediaStack_DebugUI_AEC	SupportPortal
Collaboration	MediaStack_DEVICE	TenantAdminUI
CpsDiagnostics	MediaStack_MassConvertedTraces1	TranslationApplication
CpsHostingFramework	MediaStack_MediaManager	UCWA
DataMCU	MediaStack_PerFrame	UDCAgent
DataMCURunTime	MediaStack_PerPacket	UserPinService
DeviceUpdate	MediaStack_QualityController	UserServices
DeviceUpdateHttpHandler	MediaStack_RTCP	WebInfrastructure
Dlx	MediaStack_RTP	WebRelay
ExumRouting	MediaStack_StreamingEngine	XmppCommonLibrary
HostedMigration	MediaStack_TLS	XmppListener
IIMFilter	MediaStack_Transport	XmppRouting
IMMCU	MediaStack_VIDEO	XmppTGW
InboundRouting	MediaStack_VOICEENHANCE	XmppTGWProxy
Infrastructure	MediationServer
InterClusterRouting	OutboundRouting

Now that you’ve created a provider and written it to a variable ($provider), it now needs to be attached to a Scenario. You do this by running New-CsClsScenario and passing it the provider as a parameter.

Attaching the provider to a new custom global scenario:

New-CsClsScenario -Identity "global/CUSTOMSCENARIO" -Provider $provider

If you would like to have a scenario with multiple providers (components) logging at once, then you can do so by passing the New-CsClsLogging command a plist with multiple providers in it.

Scenarios can be created with multiple providers, here’s how:

$LyssProvider = New-CsClsProvider -Name "Lyss" -Type "WPP" -Level "Info" -Flags "All"

$ABServerProvider = New-CsClsProvider -Name "ABSever" -Type "WPP" -Level "Info" -Flags "All"

$SIPStackProvider = New-CsClsProvider -Name "SIPStack" -Type "WPP" -Level "Info" -Flags "All"

New-CsClsScenario -Identity "global/MULTICUSTOMSCENARIO" -Provider @{Add=$LyssProvider, $ABServerProvider,  $SIPStackProvider}

What about if you want to only allow administrators at a specific site to use the logging scenario? Then you can create the scenario with a site scope.

You can attach the provider to a new site specific custom scenario:

New-CsClsScenario -Identity "site:Melbourne Site/SITECUSTOMSCENARIO" -Provider $provider

Note: After creating a Custom Site based scenario (as shown above), and you try and start any of the Default Global scenarios, you will get an error like the one below telling you that the scenario is “Unknown”:

Start-CsClsLogging : Cannot validate argument on parameter 'Scenario'. Unknown scenario name 'VoiceMail'

At C:\Users\Administrator.2013ENT\Desktop\myCentralisedLoggingUI.ps1:43 char:50

+         [string]$clscmd = Start-CsClsLogging -Scenario $scenariotype -pools $pooltotra ...

+                                                        ~~~~~~~~~~~~~

    + CategoryInfo          : InvalidData: (:) [Start-CsClsLogging], ParameterBindingValidationException

    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.Rtc.Management.Cls.StartOcsLoggingCmdlet

But hang on a minute, of course the “VoiceMail” scenario is a valid scenario! In fact, it’s one of the default scenarios… So what is going on here?

Well, the reason for this is that when you create a Site based Custom scenario, in the background, Lync will also create a new site based Configuration Identity (eg. In the background it runs New-Cs-ClsConfiguration), for example, if you run a Get-CsClsConfiguration you will see:

Identity                      : Global

Scenarios                     : {Name=AlwaysOn, Name=MediaConnectivity, Name=ApplicationSharing, Name=AudioVideoConferencingIssue...}

SearchTerms                   : {Type=Phone;Inserts=ItemE164,ItemURI,ItemSIP,ItemPII, Type=URI;Inserts=ItemURI,ItemSIP,ItemPII, Type=CallId;Inserts=ItemCALLID,ItemURI,ItemSIP,ItemPII,

                                Type=ConfId;Inserts=ItemCONFID,ItemURI,ItemSIP,ItemPII...}

SecurityGroups                : {}

Regions                       : {}

EtlFileFolder                 : %TEMP%\Tracing

EtlFileRolloverSizeMB         : 20

EtlFileRolloverMinutes        : 60

TmfFileSearchPath             :

CacheFileLocalFolders         : %TEMP%\Tracing

CacheFileNetworkFolder        :

CacheFileLocalRetentionPeriod : 14

CacheFileLocalMaxDiskUsage    : 80

ComponentThrottleLimit        : 5000

ComponentThrottleSample       : 3

MinimumClsAgentServiceVersion : 6

Identity                      : Site:Melbourne Site

Scenarios                     : {Name=SITECUSTOMSCENARIO}

SearchTerms                   : {Type=Phone;Inserts=ItemE164,ItemURI,ItemSIP,ItemPII, Type=URI;Inserts=ItemURI,ItemSIP,ItemPII, Type=CallId;Inserts=ItemCALLID,ItemURI,ItemSIP,ItemPII,

                                Type=ConfId;Inserts=ItemCONFID,ItemURI,ItemSIP,ItemPII...}

SecurityGroups                : {}

Regions                       : {}

EtlFileFolder                 : %TEMP%\Tracing

EtlFileRolloverSizeMB         : 20

EtlFileRolloverMinutes        : 60

TmfFileSearchPath             :

CacheFileLocalFolders         : %TEMP%\Tracing

CacheFileNetworkFolder        :

CacheFileLocalRetentionPeriod : 14

CacheFileLocalMaxDiskUsage    : 80

ComponentThrottleLimit        : 5000

ComponentThrottleSample       : 3

MinimumClsAgentServiceVersion : 6

See the new Site Identity (Site:Melbourne Site) that has appeared out of nowhere... It’s now taking precedence over the top of the Global policy for that pool. Also, notice that the only scenario that is under this Site is the SITECUSTOMSCENARIO, and as a result, this is the only scenario that can now be run from this site.

At this point, you may think that removing the scenario with a “Remove-CsClsScenario”, may fix this, however, this is not the case. To revert back using the Global configuration you need to remove the site configuration with the following command:

Remove-CsClsConfiguration –Identity “Site:Melbourne Site”

Now, you may need to restart the Lync Server Centralised Logging Service Agent in Windows Services so that the configuration is updated. 

The moral of this story is to only use Global policies for Centralised Logging, unless you have a very good reason not to!

Wrap up

This Deep Dive into the Centralised Logging Service has clarified how the service works, and how you can use it to get the information you need out of the system. Hopefully, this will leave you in a position to start debugging using it instead of leaning on the old Lync Logging Tool. As far as Custom Scenarios go, I would suggest you think about the type of information you may be needing out the system to fix common problems and tailoring some custom scenarios ahead of time, because you don’t really want to be messing around when the faults start rolling in. Keep logging, and bye for now.

PS: But what if you don’t like Powershell? Well, have no fear, as my next post will include a new Centralised Logging Tool GUI for your logging pleasure. 

Read more →