OpsMgr by Example: The Exchange Management Pack

This blog entry is the next in a series of Operations Manager-related items that review the steps performed to install, configure and tune management packs in real-world environments.

Installation:

  • Download the Exchange 2003 Management Pack (http://www.microsoft.com/downloads/details.aspx?FamilyId=9FF454F4-6D34-4FB9-9E0B-F5B68C6EDC4F&displaylang=en), and the Exchange Management Pack Guide (http://download.microsoft.com/download/7/4/d/74deff5e-449f-4a6b-91dd-ffbc117869a2/om2007_mp_exsrvr2003.doc).
  • Read the Management Pack guide – cover to cover. There are important pieces you need to know that this document spells out in detail.
  • Import the Exchange Management Pack (either using the Operations console or PowerShell).
  • Deploy the OpsMgr agent to all Exchange Servers. The agent must be deployed to all Exchange Servers. Agentless configurations will NOT work for the Exchange Management Pack.
  • Get a list of all Exchange Servers from the Operations console. In the Authoring node, navigate to Authoring -> Groups -> Microsoft Exchange 2003 Server Group. Right-click on the group(s) and select View Group Members.
  • Enable Agent Proxy configuration on all Exchange Servers identified from the groups. This is in the Administration node under Administration -> Device Management -> Agent Managed. Right-click on each Exchange server, select Properties, then the Security tab, and check the box to “Allow this agent to act as a proxy and discover managed objects on other computers.” This has to be done for EVERY EXCHANGE SERVER, even if the server is added after your initial configuration of OpsMgr.
  • Download and run the Exchange 2003 MP Wizard (http://go.microsoft.com/fwlink/?LinkId=82103) on one of the Exchange servers in the environment. Run the wizard using an Exchange Full Administrator and take the default configurations.
  • Enable the Exchange Topology View in the Operations console -> Authoring -> Management Pack Objects -> Object Discoveries. Find the Exchange 2003 Topology Discovery and override it for a specific object choosing the Exchange server that you want to perform this role (set it to True).
  • Enable the mailbox and mailflow rules. To enable these rules, go to Authoring / Rules and search on “message tracking”. Sort the results by the “Enabled by Default” field, and find the following two rules: (There are 8 reports based on these two rules. Because the rules are not enabled by default, the reports are not visible until you set up an override. Thanks to Bernie Chouinard for pointing this out! There is also an error in the collect message tracking statistic vbscript which generates an error in the OpsMgr event log.)
    • Performance Collection Rule to Collect Message Tracking Log Statistics – Top Destinations by Message Count
    • Performance Collection Rule to Collect Message Tracking Log Statistics – Top Destinations by Size
  • Configure overrides to Enable these rules for all objects of Type: Exchange Database Storage.
  • Check to make sure that Exchange shows up under Monitoring -> Distributed Applications as a distributed application which is in the Healthy, Warning or Critical state. If it is in the “Not Monitored” state, check for Exchange servers which are not installed or are in a “gray” state. This may take some time to populate after all of the above tasks have been completed.
  • Several of the “Top” 100″ reports return blank data. This is because the Rule IDs associated with the reports are misconfigured and must be manually edited. Perform the following steps:
    • On the Report Server, open a browser and navigate to http://localhost/reports. Select Microsoft.Exchange.Server.2003.Monitoring
    • Find and select the “Report.Exchange.Top100MailboxesbySize” report (it does not have a rpdl extension)
    • Select the Properties tab, then select the Parameters link on left-hand margin
    • Scroll down and find the RuleID String parameter, and replace the value with 2EE6F2C1-4C8B-AFA9-D615-238F6AA73E8C
    • Click Apply, then run the Top100 mailboxes report to verify that data is now being returned.
    • Repeat these actions for the following Rule IDs:
      • Performance Collection Rule to Collect Mailbox Statistics –  Top 100 Mailboxes by Message Count
        New RuleID = 55BBEDA5-C09C-7C06-602F-20C85723EACE
      • Performance Collection Rule to Collect Mailbox Statistics – Top 100 Mailboxes by Size
        New RuleID = 2EE6F2C1-4C8B-AFA9-D615-238F6AA73E8C
      • Performance Collection Rule to Collect Public Folder Statistics – Top 100 Public Folders by Size
        New RuleID = 5D3DAEDA-56E6-909A-FAB8-AF021AA1A61E
      • Performance Collection Rule to Collect Public Folder Statistics – Top 100 Public Folders by Message Count
        New RuleID = B2032940-E1E0-975F-42F0-302C7B5F21DB

Tuning/Alerts to Look for: The following are alerts we encountered and resolved while tuning the Exchange Management pack.

Alert: Multiple/any alert with “Baseline” in the title

Issue: Default sensitivity levels within the Exchange management pack.

Resolution: See blog articles: http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!183.entry and http://ops-mgr.spaces.live.com/blog/cns!3D3B8489FCAA9B51!189.entry for details on tuning baseline counters.

Alert: The Internet Information Service NNTP Virtual Server named NNTPSVC/1 is unavailable as the virtual server has been stopped.

Issue: On Exchange servers this service is required to install but it is not required after it is installed.

Resolution: If this service is disabled/not in use you can remove it. To remove the service, log into the server and use “sc delete NNTPSvc”. Or you can create an override to ignore this on Microsoft Exchange 2003 Server Group, as NNTP was required for the installation but can be disabled after the installation has been completed.

Alert: Verify Test Mailboxes: This Exchange Server does not have any MOM test mailboxes.

Issue: Test mailboxes are created by the Exchange Configuration Wizard.

Resolution: Run the Configuration Wizard to create the mailboxes.

Alert: No MOM test mailbox account for some mailbox databases

Issue: Test mailboxes are created by the Exchange Configuration Wizard.

Resolution: Run the configuration Wizard creating test mailboxes on each database or disable the rule.

Alert: Replication is not occurring – All replication partners have failed to synchronize

Issue: The Alert Description is the key on this alert.

Resolution: Alert description of “AD Replication Monitoring : All replication partners are now replicating successfully” is a success condition and does not require any intervention other than closing the alert.

Alert: Some replication partners have failed to synchronize

Issue: A domain controller was offline and unable to be synchronized with.

Resolution: Bring the domain controller back online.

Alert: Outlook Web Access logon failure: Unexpected error during synthetic Outlook Web Access logon

Issue: OWA Logon failure: OWA can only be configured to be monitored if the site runs on HTTPS.

Resolution: Disabling the rule (For all objects of type: Exchange OWA), as this environment only runs with HTTP on the OWA configuration.

Alert: Exchange ActiveSync logon failure: Unexpected Error

Issue: Exchange EAS not required in the environment.

Resolution: Disabled the rule for all types of type Exchange EAS, as this functionality is not used in the environment.

Alert: The 3GB virtual address space option is not enabled

Issue: The 3GB configuration should be used for Exchange servers except for those which are functioning as bridgeheads or front-end servers (per the Exchange Best Practices Analyzer [BPA]).

Resolution: Disabling this rule for the front-end servers or bridgehead servers in the environment.

Alert: Failed to probe the state of monitored services

Issue: This was occurring on the SMTP services on an Exchange server which the administrators has manually restarted.

Resolution: The alert was notifying on a true business-impacted situation. Requested the administrators to put the server into maintenance mode prior to making changes like this, unless it is an emergency situation.

Alert: Data Publisher object is not installed

Issue: This was a system which was misidentified as an Exchange sever that was using a third party product to provide Exchange restoration functionality.

Resolution: Disabled the rule for this system through an override.

Alert: Microsoft Windows Internet Information Server 2003 NNTP Virtual Server is Unavailable.

Issue: NNTP Service Down on non-active cluster node

Resolution: The NNTP service is supposed to be down since it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group. NNTP was not used on Exchange and could also have been removed as a service from the systems. 

Alert: Microsoft Windows Internet Information Server 2003 SMTP Virtual Server is Unavailable.

Issue: SMTP Service Down on non-active cluster node

Resolution: The SMTP service is supposed to be down as it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group. 

Alert: Microsoft Windows Internet Information Server 2003 Web Site is Unavailable

Issue: Web Service Down on non-active cluster node

Resolution: The Web service is supposed to be down as it is running on a cluster and the system showing this error is not the active node in the cluster. Created a group for these servers which are running Exchange and are part of the cluster and disabled the rules for the group. 

Alert: Check Services FE Monitor reported a problem 

Issue: Product knowledge on this: “Services State monitoring with this registry key is a legacy from the MOM 2005 Exchange 2003 MP. This monitor is included since configuration is possible from within the Exchange Configuration Wizard. OpsMgr 2007 provides a dedicated health model for monitoring Windows Service Health.”.

Resolution: Right-click and choose Overrides, Disable the Monitor for all objects of type: Exchange 2003 Role.

Alert: Exchange EAS monitor reported a problem

Issue: Synthetic Exchange ActiveSync requires SSL

Resolution: Closed the alert as it had not repeated for 2 days and had a 15 minute schedule to run. Issue repeated. EAS logon verification: Cannot measure EAS availability for the following URL: 0x80131537(-214233033) Invalid URI: The format of the URI could not be determined. Found the following information at MyItForum:

This script problem is caused by OMA and EAS virtual directories not being SSL-enabled. So in order to correct it, simply enable SSL:

  • Open Internet Information Services (IIS Manager).
  • Connect to the server name of your front-end Exchange server.
  • Drill down to Web Sites, then to the web site.
  • Locate the two virtual directories named OMA and Microsoft-Server-ActiveSync.
  • Open the properties of the virtual directories, choose the Directory Security tab.
  • Under Secure communications, click Edit.
  • Check the box labeled “Require security channel (SSL)”.

Alert: No MOM test mailbox account for some mailbox databases

Issue: No MOM mailboxes were created on a per-storage group when running the configuration Wizard. The alert is being created expecting that per-store monitoring will be configured which is not the case in this environment.

Resolution: Disable this rule for all objects (of type Exchange 2003 role) because this rule is monitoring on a per-store basis but we are monitoring on a per-server basis. Closed the alerts.

 

Alert: SSL is not configured on this Exchange server 

Issue: This occurs on servers which have SSL enabled if they do not require usage of SSL within IIS. Back-end servers communicate with front-end servers via HTTP not HTTPS so SSL should not be required on the back-end Exchange servers. We found the following information at Notes from the Underground…

“SSL in a Front-End/Back-End Scenario

Although it’s possible to implement SSL on a front-end (FE) server, resulting in all transmitted data between the FE and your client browsers being encrypted, you should be aware that you can’t use SSL between any FE and back-end (BE) servers—it simply doesn’t work. This means that if your FE server is placed in a perimeter network (also known as a demilitarized zone, or DMZ), all traffic between the FE and BE would be unencrypted. So if you’re planning such a scenario, consider using IPSec between the FEs and BEs. More and more organizations place their FEs directly on their private networks (and instead place an ISA server or similar in the DMZ), which eliminates this security risk.”

Resolution: Disabled the alert on Exchange back-end servers.

Alert: Calendaring agent failed with error while saving appointment

Issue: Calendaring agent failed with error code 0x8004010f while saving appointment.

Resolution: Good links on this: http://www.eventid.net/display.asp?eventid=8206&eventno=1103&source=EXCDO&phase=1. Lots of product knowledge on this related to virus scanners, registry settings, etc. This is a result of an event ID of 8206 on the Exchange server.

 

Alert: Disabled user does not have a master account SID.

Issue: The user does not have “Associated external account” permission and the Exchange server does not have the hotfix available to resolve this issue.

Resolution: To resolve this, open the user account in Active Directory Users and Computers, go to Properties, Exchange Advanced, Mailbox Rights. For the Self account we added the “Associated external account” permission which resolves the error. The error itself does re-appear, but it appears with the next user identified in the environment which had the issue. If there are a large number of these in your environment you can also locate them by going to each Exchange back-end server, and doing a Filter on event number 9548 within the application event log. A hotfix is available for this, available at: http://support.microsoft.com/kb/916783. (This information is a subset of what was originally posted at http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!835.entry.)

Alert: Low Free Disk Space

Issue: Part of the Exchange Management Pack checks for free space on all drives including those which do not have Exchange directories or files on them. This activates a warning at less than 5% free disk space and less than 1000 MB of free disk space on Exchange server drives that do NOT have the transaction logs or queue files on them.

Resolution: Free disk space on the drive. See the “Logical Disk Free Space is Low” entry for potential approaches to free disk space on the drive.

Alert: Very low free disk space

Issue: Part of the Exchange Management Pack checks for free space on all drives including those which do not have Exchange directories or files on them. This activates an error at less than 2% free disk space and less than 400 MB of free disk space on Exchange server drives which do NOT have the transaction logs or queue files on them.

Resolution: Free disk space on the drive. See the “Logical Disk Free Space is Low” entry for potential approaches to free disk space on the drive.

Alert: Logical Disk Free Space is Low

Issue: Low disk space on a drive within a server being monitored by OpsMgr.

Resolution: Can either free up disk space on the drive or configure an override for the drive to change the monitoring configurations for the drive (see http://cameronfuller.spaces.live.com/blog/cns!A231E4EB0417CB76!1001.entry for details on how to do this override). Other items to consider:

  • If the page file is currently on the drive which is critical on drive space, it can be moved to another drive.
  • The “disk cleanup” wizard can also be used to provide methods to free up disk space (right-click on the drive, go to properties, click the disk cleanup button).
  • If the drive is critical on available free disk space, automatic updates can be turned off in the control panel and the c:\windows\softwaredistribution\download folder can be removed (of course, automatic updates will not occur after this change is made).
  • The default IIS configuration puts the IIS log files under C:\WINDOWS\system32\LogFiles\W3SVC1. These can be moved within the Internet Information Services (IIS) Manager by clicking on the properties of the web sites, under the properties of the log files. The log files can either be moved or disabled if required.
  • Exchange log files can take up a large amount of disk space on a drive if the Exchange server is not being backed up regularly. When the Exchange server has a full backup completed the log files are removed. If an Exchange server is critical on space on the log drive, determine if backups are occurring and if they are not, perform an ntbackup of the Exchange files to truncate the logs. Circular logging (which removes this type of a situation) can also be enabled in some configurations but is not recommended if there is any mailbox data on the system.

Alert: MAPI Logon Failure.

Issue: This occurred almost immediately after running the Exchange 2003 Management Pack configuration wizard.

Resolution: The issue was resolved when the Wizard completed its configurations and had only repeated once. Ran the “MAPI Logon” task to validate that the issue had been resolved and confirmed no errors. Closed out the alert.

Alert: MAPI session closed due to excessive number of store objects in use.

Issue: Exceeded the maximum of 250 objects of type “objtMessage” (1 repeat). Or exceeded the maximum of 32 objects of type “session” (0 repeats). Or Exceeded the maximum of 500 objects of type “objtFolder”.

Resolution: Microsoft resolutions in the Product Knowledge. Eventid.net has http://www.eventid.net/display.asp?eventid=9646&eventno=3449&source=MSExchangeIS&phase=1 on this. Microsoft KB article on this: http://support.microsoft.com/default.aspx/kb/830836.

 

Alert: Outlook Web Access logon failure: Unexpected error during synthetic Outlook Web Access logon

Issue: OWA Logon failed. Cannot measure OWA availability. Unexpected error. No Exchange virtual servers and virtual directory (SSL enabled) can be found on this server to form a valid URL. Try providing the url in the custom urls registry key.

If the name in URL matches the name in the certificate, we learned that when SSL is enabled, the MP reports an error like this when ‘Require SSL’ checkbox is not checked on the Directory Security tab of the website. See Andy Dominey’s blog writeup on this: http://myitforum.com/cs2/blogs/adominey/archive/2007/04/10/mom-2005-and-om-2007-exchange-2003-management-pack-issue.aspx

This rule requires OWA to be installed with SSL and to have require SSL checked on the system. It will NOT work without both of these configured. This also requires that the name matches the name on the certificate.

Resolution: Enable SSL and require SSL on the OWA server. If the name of the URL doesn’t match the certificate this rule will not work.
Update: Microsoft has a resolution to this which is available at http://support.microsoft.com/default.aspx/kb/919356 for the error 0x80131502(-2146233086) Index was out of range.

Alert: The MAD Monitoring thread was unable to read the CPU usage information.

Issue: This had repeated 8 times in 5 days/16 hours. The MAD Monitoring thread was unable to read the CPU usage information, error ‘0x800706be’. From the summary, if this happens occasionally it can be safely ignored. If it happens every five minutes then there is an issue.

Resolution: Closed the alert as it was not occurring “frequently”.

 

Alert: The Offline Address List (OAL) Generator could not generate full details for some entries in the OAL. To see which entries are affected, event logging for the OAL must be set to at least medium.

Issue: MSExchangeSA event id 9320.

Resolution: Eventid link on this: http://www.eventid.net/display.asp?eventid=9320&eventno=3692&source=MSExchangeSA&phase=1. The Microsoft article on this is available at http://support.microsoft.com/default.aspx/kb/908496.

 

Alert: The Offline Address List Generator could not generate full details because the total size of the details information is greater than 64 kilobytes.

Issue: See the Microsoft support article.

Resolution: The Microsoft article on this is available at http://support.microsoft.com/default.aspx/kb/908496.

Advertisements
This entry was posted in Tuning and Configuration. Bookmark the permalink.

10 Responses to OpsMgr by Example: The Exchange Management Pack

  1. Operations says:

    We received the following email / question from Jim Gray, but could not reply to him because of his email preferences.
     

    From: Jim GraySent: 8/29/2007 12:56 PMTo: Operations ManagerSubject: RE: Your blog entry "OpsMgr by Example: The Exchange Management Pack"
    Great article. When monitoring Exch in a cluster, we have the agent installed in the physical nodes and for some reason, the cluster appears in Agentless. Is this normal? Stepping through your example I see the cluster and not the physical nodes while inspecting the members of the Exchange 2003 group.  
     
    The answer is: Cluster nodes appear as agentless managed.
     

  2. Stefo says:

    OK, My Cluster nodes appear as agentless but i can\’t see this server as BackEnd with mailbox.
     
    it\’s right?
     

  3. Operations says:

     
    Stefano, we’re not running any Exchange clusters at the moment so we can’t validate – sorry :(. You may want to check with the newsgroups.

  4. Operations says:

    Update for Stefano\’s question of 9/7/2007:
     
    Exchange servers begin to appear in the agentless managed section when you have Exchange clusters. They appear AFTER you set the proxy setting on the physical exchange servers in the environment.

  5. Ron says:

    For some reason, Exchange Service is appearing as Not Monitored under distributed applicaitons, although all the servers in the organization appear in a Healthy managed state.  What would i do to troubleshoot this?

  6. Operations says:

    Cameron is working with Ron on this one – they work at the same company 🙂

  7. Diane says:

    Does any one have any information regarding the Exchange 2007 management pack?  There are only 6 available reports (the rest are set to Visible=False) and no message tracking rules available to enable such as in the Exchange 2003 management pack.  Are the remaining reports just not converted to OpsMgr yet?  Will they be in the next version of the Exchange 2007 management pack?

  8. Operations says:

    Your best luck will be when there is a native management pack available for Exchange 2007. Not ETA on that at this point, though.

  9. Tony says:

    http://support.microsoft.com/kb/948096.

    This link can no longer be found, can someone point to a reason or an updated version?
    I am having the exact same issue with the Top 100 coming up empty, and just want to know the most recent fix for this issue.

    Thank you in advance.

    • The article appears to have been pulled and there is no replacement / updated version. However, we are specifying the information that was in the article here – the link to the article was just as backup.
      The non-working link has been removed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s