Syslog Monitoring Walkthrough with Systems Center Operations Manager 2007

Building on excellent posts here, here and here and feedback on the forums here, I am going to describe our Operations manager 2007 syslog setup.


First step was to build a group of servers that will be our Syslog collectors in Opsmgr.  Populate your Syslog Collectors group w/ the servers you will be pointing your syslog messages towards.  I created a new management pack called _Default_Syslog that will hold all the work in this section. 



Once I have my group, I want to create a new rule.  Under the rule wizard, I select Alert Generating Rules –> Event Based-> Syslog (Alert). 

I am going to have 3 Syslog rule.  One for each state, warning, error and informational.  The first rule I am making is the error rule.  So we are going to name it, Syslog Error.  (I know, imaginative)

Since I am collecting Alerts, it seems “Alert” would be a good rule target but I will admit to being a bit hazy on this area of SCOM and what it implies.  If you know better, please let me know.

For rule target, I am just going to pick Windows Computers.  Note that at some point I tried to target this at the Agent and this caused problems when I wanted to use a Management server as a collector.  To quote Graham Davies from the System Center Operations Manager General forums:

If you want the rule to run on the RMS \ Gateway then you'll need to target the appropriate class - management server. Although the Management Servers include "agent capability", they are not agents and hence won't get the rule if you target agent.

MAKE SURE you unselect Rule is Enabled or you will turn on syslog collection for all of your servers.  We will override to turn on these rules for our specific group later.




For Event Expressions, I want to catch all events w/ a Severity of less than or equal to 3.  So I build my Filter like so:



For my alerts, I like to get more data so I am going to put the following in the description:

Facility:
$Data/EventData/DataItem/Facility$
Severity:
$Data/EventData/DataItem/Severity$
Priority:
$Data/EventData/DataItem/Priority$
Priority Name
$Data/EventData/DataItem/PriorityName$
Time Generated:
$Data/EventData/DataItem/TimeStamp$
Hostname:
$Data/EventData/DataItem/HostName$
Message:
$Data/EventData/DataItem/Message$

Click create to finish the wizard.

Repeat the above for the Warning Alerts.  You will only need to change the Name, the severity on the alert and make your Filter equal to 4 like so:



And again for your Informational alerts.  Your filter will be greater than or equal to 5 and don’t forget to change your severity.

I chose to capture these as infomational alerts for now, I am going to see how spammy this gets.  another option is to create these as Event collection rules without alerts.  That will allow you to hold on to the data but not alert on it.  It will depend on your alerting requirements.  You can also manage what you send from your sending device’s syslog configuration.

To enable these rules on specific servers, right click on your rule and choose ‘Overrides’ –> Override the Rule –> For a group…



 Select your Syslog Servers group and override the rule to enable it on these servers.



You can verify that the syslog listener is enabled by getting on the machine you expect to be listening and checking for UDP 514 in netstat.  Open a command windows and type ‘netstat –an | findstr 514’  (findstr = grep).  You want to see a line like the below.  If you don't get anything, you are not listening for syslog packets.



In order to make this easy to see, I also created a view.  Under the SCOM Console Monitoring view, find your _default_syslog folder –> right click –> New –> Alert View. Include alerts ‘created by specific sources’.  Pick your rules that you just created.





You should end up w/ a view like (your source will probably be a computer name):




More info I found two ways to display more info in the line.  The first way is the most straightforward and I typed it up first so I am going to show it anyway.  You probably want the second way.  B)
The first way:

You can add more information to your alert and then set your view to show it.  Under your alert configuration, you can add custom alert fields.  I added fields for both my hostname and my priority name.



Send a few more test syslogs.  BTW, the kiwi syslog generator works perfectly for this.  Update your view so that you show Custom Field 1 and Custom Field 2 from the Display tab under your view properties.



Note that for some reason my extra columns were not displaying initially.  I had to delete and recreate the view.  Since my source was always the same, I also stopped showing the source.  You can see the syslog entry created after the change has populated the custom fields:



The second way:

This initially involved some (simple) XML munging so I was going to have an advanced and a simple method, the simple being the one above.  Once I figured out how in the XML, it turns out you can do it in the GUI as well. 

I first exported the MP and saw the below XML

          <Name>Syslog Error</Name>
          <Description>Facility:
{0}
Severity:
{1}
Priority:
{2}
Priority Name
{3}
Time Generated:
{4}
Hostname:
{5}
Message:
{6}</Description>

Note that the curly bracked enclosed items are expanded to our variables.  So {5} is analogous to $Data/EventData/DataItem/HostName$ in our GUI based alert description.  Putting $Data/EventData/DataItem/HostName$  in the alert name doesn’t work but putting {5} does.  So I changed the XML to

          <Name>Syslog Error: {5} - {3}</Name>
          <Description>Facility:
{0}
Severity:
{1}
Priority:
{2}
Priority Name
{3}
Time Generated:
{4}
Hostname:
{5}
Message:
{6}</Description>

When I checked my alert view, voila!, My alert name has changed as in the screen shot below.  My next thought was why can’t I do that from the gui?  So, i edited my alert name to be: Syslog Warning: {5} - {3}



Back to my alert view and, check it out, useful alert names!!  Maybe I am too excited about this but it makes it much more supportable in my opinion.  B)



One note, I am putting the syslog collector on other running servers.  We have only a few devices that are sending data here and only critical events are expected.  YMMV, you may want to dedicate a server to this.