A couple of nights ago I was contacted by one of our analysts in the ClearPointe Network Operations Center (NOC) about the following alert that had been triggered by a customer’s Exchange 2010 server.

First, it should be noted that I do not profess to be an Exchange 2010 guru, but I began investigating and troubleshooting. The customer was not experiencing an outage or any issues but rather one of the many PowerShell cmdlets that run in Systems Center Operations Manager was indicating a monitoring issue that needed to be addressed. After about 90 minutes of troubleshooting, I suspected that the problem might somehow be due to shrapnel left over from an Exchange upgrade and decided to continue the following morning.

I resumed troubleshooting and research the next morning. Manually running the PowerShell cmdlet in the Exchange Management Shell produced the same results that were observed in the alert the previous night:

During my Internet research that morning, I stumbled across a blog posting at SysAdmin Flakshack which dealt with the exact alert I was troubleshooting. His post confirmed what I suspected – the problem results from a remnant of Exchange 2003 not being removed from Active Directory properly during an upgrade to Exchange 2010. To resolve the problem, open the ADSI Edit tool and navigate to the following:

CN=Configuration,DC=yourdomain,DC=local
   CN=Services
      CN=Microsoft Exchange
         CN=YourOrganizationName
            CN=Administrative Groups
               CN=Your Old Exchange 2003 AdministrativeGroupName
                  CN=Servers

If the upgrade to Exchange 2010 is complete and there are no more Exchange 2003 servers in the environment, the Servers container should be empty, which it was in my case. Delete the Servers container and the problem is resolved, as show by rerunning the Test-MAPIConnectivity PowerShell cmdlet:

Do not delete the Exchange 2003 Administrative Groups container as that could create additional problems. Delete only the Servers container.

I spent approximately 3 hours researching and troubleshooting this Operations Manager alert; however, once I found the solution described above, the problem was resolved within seconds! I hope this post might help save another administrator some of his or her valuable time.

As I stated in my previous post, I have been troubleshooting a System Center Operations Manager (SCOM) alert that has been triggered on our two Lync 2010 edge servers here at ClearPointe. The alert, Event ID 26004 from source Health Service Modules and shown below, is triggered every 13 minutes and states that the Windows Event Log Provider is unable to open the Office Communications Server (OCS) event log.

The two servers in question were at one time our Office Communications Server edge servers; however, as a part of our recent telephone system upgrade, we upgraded from OCS to Lync Server 2010. With the upgrade to Lync 2010, the Office Communications Server event log was replaced with a Lync Server log – the Office Communications Server log no longer exists!

I spent a considerable amount of time, too much in fact, trying to figure out how and why the Windows Event Log Provider seemed to think that the log still existed. Initially, I stopped the System Center Management service (HealthService.exe), deleted the Health Service State directory located at C:\Program Files\System Center Operations Manager 2007, and restarted the service. This caused the server to discard all previously downloaded management packs and download a complete new set from it’s designated management server. I did this in hope that somehow an OCS management pack had remained in its cache after the Lync upgrade. The alert, however, fired again like clockwork on its 13 minute schedule.

I then searched the system registry and file system of each server trying to find anything related to OCS. The searches of each server turned up nothing. During my Internet research of blog posts, Microsoft TechNet articles, and forums, I happened to come across a somewhat unrelated article about manually creating an event log in the system registry. Could I really manually create an Office Communications Server event log via the registy that the Windows Event Log Provider could access, and thereby achieve a solution to my dilema? It seemed too easy!

On each of the Lync 2010 edge servers, I accessed the registry and navigated to HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\eventlog. I right-clicked on the eventlog key and selected New > Key from the pop-up menu.

I named the new key Office Communications Server and closed the Registry Editor. I reopened the Event Viewer applet and did, indeed, see an Office Communications Server event log (although nothing will ever be written to it). Now, all I needed was for the Windows Event Log Provider to see the new log and successfully access it. I watched the Operations Manager event log. Thirteen minutes passed and no Error event was triggered. Twenty-six minutes passed and, again, no Error event.

I wish I had stumbled across that post about manual event log creation sooner! It would have saved me a lot of time and frustration!

Yesterday I was troubleshooting a System Center Operations Manager (SCOM) alert triggered from each of our Lync 2010 edge servers stating that the Windows Event Log Provider could not access the Office Communications Server event log. (More on the resolution of that issue in my next post.) My troubleshooting and research eventually led me to the Lync Server 2010 Management Pack Guide. On page 39, in the “Management Pack problems and resolutions“ section, I noticed that edge servers are not discovered by default. This led me to check our implementation. Sure enough, our Lync edge servers had not been discovered as holding the EdgeServer Role.

To enable the discovery of the EdgeServer Role, from the Authoring space of the System Center Operations Manager Console, expand Management Pack Objects, and click on Object Discoveries.

On the menu bar, click the Find button to add the Look for: bar to the Object Discoveries pane.

In the Look for: text box, enter “ls central topology discovery” and click Find Now.

In the filtered results in the Object Discoveries pane, right-click on LS Central Topology Discovery.

From the pop-up menu select Overrides > Override the Object Discovery > For a specific object of class: LS Discovery Script. This opens the Override Properties dialog box as shown below.

Tick the checkbox on the Discover EdgeServer Role row and change the Override Value to True.

Select the destination management pack from the drop down list at the bottom of the dialog box (it is recommended to not save the override to the Default Management Pack).

Click the OK button.

Within four hours, which is the default setting for this object discovery, your Lync edge servers should appear in the Monitoring space of the System Center Operation Manager Console under Monitoring > Microsoft Lync Server 2010 Health > Servers > 4. Edge Servers.

Anyone utilizing SCOM to monitor their Lync Server implementation, should check to ensure that the EdgeServer Role discovery was enabled when they imported the Lync Server 2010 Management Pack.

I cannot take credit for finding this little tidbit, but I am posting it here for my possible future reference and, hopefully, to help someone else who is experiencing a similar problem.

Recently, one of our clients had begun to experience a degradation in the speed of document uploads into a SharePoint document library, at times as much as ten times slower than normal. During research and troubleshooting efforts, a blog post by Phil Harding was discovered that offered a not-so-obvious solution to the problem.

In Internet Explorer, access Internet Options and click on the Connections tab in the Internet Options dialog box. Then click the LAN settings button near the bottom of the Internet Options dialog box. On the Local Area Network (LAN) Settings dialog box that opens, uncheck the Automatically detect settings option. Also, ensure that the Bypass proxy server for local addresses option is checked. Click OK twice to close the Internet Options dialog box.

This quick, little tweak of the browser settings resolved our client’s slow upload performance. The fix probably took less time to enact than it took for a document to upload into the document library prior to the fix!

My daughter called earlier this evening and said that the keyboard on her Dell Studio laptop had suddenly stopped functioning. I remoted into the laptop using LogMeIn and did a little troubleshooting. I was unable to open the System event log so I began to suspect malicious software. I started a full system scan with Microsoft Forefront Client Security and disconnected from my remote session since the scan would take some time to complete.

In the meantime, I did some research on the Internet and came across a string of posts in a Dell support forum. It seems that quite a few people have experienced the non-responsive keyboard problem and it was resolved by draining the “flea power” on the laptop. What in the world is “flea power”, you ask? I asked the same question.

In layman terms, “flea power” represents a very small amount of power. When you switch the computer into standby, the power supply swithes over to the +5VFP circuitry to conserve power consumption, and put inactive components in a quiescent state. Even if the computer is off, but still plugged in, this flea power is supplied to the CPU in most instances.

The process to drain the flea power is quite simple. Power down the laptop, remove the AC adapter, and remove the battery. Press and hold the power button for 5 seconds. Reinstall the battery, plug in the AC adapter, and power up the laptop.

After following these instructions, miraculously the keyboard on my daughter’s laptop was functioning once again. I’m not yet sure why the existence of this small amount of reserve power suddenly has a negative impact on the keyboard, but we are glad we found this simple solution to the problem!

Oh, and, as for the virus scan, it was not run in vain. The scan did find several instances of trojans and malware on the computer, which we promptly removed.

VMware is replacing their original ESX computer virtualization product with ESXi. VMware ESXi is a smaller footprint version of ESX and does not include the ESX Service Console, which led me to investigate the proper method of configuring SNMP (Simple Network Management Protocol) on an ESXi server. This information is readily available from various locations throughout the Internet, but I am posting it here primarily for my reference. Should this post provide assistance to others, all the better!

As it turns out, it is quite simple to enable and configure SNMP on a VMware ESXi server. VMware provides a free download of its vSphere CLI command set which allows you to run common system administration commands against an ESXi system from any machine with network access to the targeted system.

Open the vSphere CLI and navigate to the bin directory. Run the vicfg-snmp command as shown below, replacing the placeholders with your data, to configure the community string, target machine, and port. A note here for Windows users, you must add the .pl file extension to the vicfg-snmp command in order for the command to complete successfully.

Configuring SNMP community string, target, and port

Now that SNMP is configured, you must enable it by running the following command, again from within the bin directory and including the .pl file extension if your are running on Windows.

Command to enable SNMP

That’s all there is to it! SNMP is now configured and enabled on your VMware ESXi server, although it is a good idea to run one final command to verify your settings, as shown below.

Command to verify the SNMP settings

The vSphere CLI package can be installed on Linux and Microsoft Windows systems. The vSphere CLI commands run on top of the vSphere SDK for Perl. The vSphere CLI installer installs both the vSphere CLI and vSphere SDK for Perl.

I use virtual machines (VMs) quite often for testing, evaluating, and troubleshooting. Invariably, during the process I discover that I did not make my virtual hard drive (VHD) large enough. Our developer introduced me to VirtualBox, an open source virtualization product for creating and managing VMs, several months ago. When I experienced my first VHD that had insufficient space, he also introduced me to Damn Small Linux (DSL) and a method for expanding my VHD. Well, it actually does not expand the existing disk but rather copies the bits to a new, larger VHD. I am documenting the process here for my future reference, although I hope this post will also help others.

The original VHD must be attached to the virtual IDE controller rather than the virtual SATA controller, so it might be necessary in your case to detach the small VHD from the SATA controller and attach it to the IDE controller before proceeding. Access the Storage settings for your VM and select the IDE Controller in the Storage Tree window. Click the Add Hard Disk button on the right side of the IDE Controller section. Create a new VHD of the desired size (named NewHardDisk1.vdi by default as shown below). Then select the CD-ROM drive in the IDE Controller section. In the Attributes section to the right, click the button that looks like a CD and mount the Damn Small Linux ISO that you have previously downloaded from their website.

Start the VM and allow it to boot into DSL. Open a Terminal session and type the following to elevate to root privileges:

sudo su

Now you need to identify the disks with the following command (the last character is a lowercase “L”):

fdisk -l

The final command initiates the bit-by-bit transfer from the smaller disk to the larger disk. “If=” designates the source, or smaller, disk and “of=” designates the destination, or larger, disk (note that your disks may be named differently that what is shown below):

dd if=/dev/hda of=/dev/hdb

Now sit back, be patient, and grab a cup of coffee. Depending on the size of your source disk, the transfer to the destination disk can take a while. There is no progress bar or anything to indicate the progress of the transfer, although there are additional commands that you can run in the Terminal window to provide this information. My experience has been that it can take up to two hours to transfer a 20 GB disk to a larger disk. The terminal session will return to a prompt when complete.

Shut down the VM and detach the smaller disk which is no longer needed. Also, “eject” the DSL ISO from the CD-ROM drive. Start the VM. In Windows, access Disk Management and you will see that your C: drive is still shown as the smaller size; however, to the right, you will see additional unused space. Right-click on the C: drive and select Extend Volume… from the pop-up menu.

Accept the default settings as you step through the wizard to extend the volume. Once complete, you will now have a larger C: drive in your virtual machine.

Recently, while troubleshooting a Windows Server Update Services (WSUS) issue on a managed server, I discovered that the settings on the Automatic Updates dialog box were greyed out with the Turn off Automatic Updates option selected as shown below. I had already verified that the server was not receiving the appropriate WSUS settings from Group Policy and could find no other GPO or local policy taking precedence over the desired policy settings.

In order to make the Automatic Updates options configurable once again, I had to edit the system registry as follows:

1. Click Start > Run, type regedit.exe in the textbox, and click the OK button.
2. Navigate to HKEY_LOCAL_MACHINE \ SOFTWARE \ Policies \ Microsoft \ Windows \ WindowsUpdate \ AU
3. In the right pane, delete the AUOptions and NoAutoUpdate values.
4. Navigate to HKEY_CURRENT_USER \ SOFTWARE \ Microsoft \ Windows \CurrentVersion \ Policies \ WindowsUpdate
5. In the right pane, delete the DisableWindowsUpdateAccess value.

Once the registry values had been deleted, the Automatic Updates settings were configurable. I forced the server to update it’s Group Policy settings by running gpupdate /force from a command prompt. Another check of the registry indicated that the server was now receiving the appropriate WSUS settings via Group Policy. After the forced Group Policy update, the settings on the Automatic Updates dialog box were once again greyed out; however, the Automatic (recommended) option was selected as expected.

At ClearPointe, we have two conference rooms and reservations for each room is created via the Outlook client and Exchange 2007. Recently, I was asked to troubleshoot an issue where reservation requests to one of the rooms was not returning an email acknowledgement as expected. While troubleshooting and resolving the problem, I noticed another behavior that I did not particularly like — when meeting reservations are displayed on the Outlook resource calendar, the subject line of the meeting request (i.e., the subject of the meeting) is replaced with the organizer of the meeting (i.e., the person who created the meeting request).

While researching this issue, I quickly discovered that this behavior is by default although, in my humble opinion, not the optimal behavior as most would rather see the purpose of the meeting as the subject on the calendar rather than the organizer. Besides, the organizer is already displayed in the calendar reservation so the default behavior displays the organizer twice!

The solution to this problem is simple and involves the use of the Exchange Management Shell (I am beginning to develop a real appreciation for PowerShell!). At the prompt of the Exchange Management Shell, enter the following cmdlet:

Set-MailboxCalendarSettings -Identity (ResourceMailboxAlias) -AddOrganizerToSubject:$false -DeleteSubject:$false

That is all there is to it! Now your resource room reservations will appear on the Outlook calendar with the subject of the meeting, the resource room where the meeting is to take place, and the organizer of the meeting.

Recently, one of our NOC Supervisors was troubleshooting an Operations Manager Agent heartbeat failure on one of the remote servers that we manage. When he attempted to restart the System Center Management service, it failed and returned the following error:

Windows could not start the OpsMgr Health Service on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code -2147467259.

The following event was logged in the event log of the remote computer:

After trying the basic steps we follow to get an Agent to resume communications without success, he turned to our good, old friend Google for additional research on the problem. He came across a post on J.C. Hornbeck’s SMS&MOM blog site that resolved the problem.

According to the instructions in the blog post, our engineer started ETL Tracing in Verbose Mode by running StartTracing.cmd Verbose from within the C:\Program Files\System Center Operations Manager\Tools directory. He then attempted to restart the System Center Management service, which failed as expected. The tracing was then stopped and he evaluted the output, looking for something similar to the following:

src EventLogUtil_cpp195 Common::EventLogUtil::LogEvent F00 D88 1 36432528
01\11\2008-14:59:14 Information Logging informational event with args
“OpsMgr_dor”, “NULL”,”NULL”, “NULL”, “NULL”, “NULL”, “NULL”, “NULL”, “NULL”
SecureStorageManager SecureStorageMGState_cpp75 SecureStorageMGState::Create F00
36C 1 172271493 01\11\2008-14:59:14 Error Unexpected return value :
2(ERROR_FILE_NOT_FOUND)
SecureStorageManager SecureStorageManager_cpp976
CSecureStorageManager::NotifyManagementGroupsList F00 36C 1 43840688
01\11\2008-14:59:14 Error Unable to create per-MG state object
SecureStorageManager SecureStorageManager_cpp1584
CSecureStorageManager::NotifyManagementGroupsList F00 36C 1 4294967295
01\11\2008-14:59:14 Error Unable to initialize per-management group state
ConnectorManager ConnectorManager_cpp5165
CConnectorManager::informManagementGroupState F00 36C 1 160 01\11\2008-14:59:14
Error Secure Storage Manager failed during MG notification :
-2147467259(E_FAIL)
ConnectorManager ConnectorManager_cpp1480 CConnectorManager::Start F00 36C 1 3393
01\11\2008-14:59:14 Error Unable to inform components about management
groups : -2147467259(E_FAIL)
HealthServiceExecutive HealthServiceExecutive_cpp1537
CHealthServiceExecutive::ManagerStartup F00 36C 1 235378759 01\11\2008-14:59:14
Error Start of 1 manager failed with code -2147467259(E_FAIL).
HealthServiceExecutive HealthServiceExecutive_cpp1777
CHealthServiceExecutive::ServiceInitialization F00 36C 1 4294967295
01\11\2008-14:59:14 Error ManagerStartup failed with code
-2147467259(E_FAIL).
HealthServiceExecutive HealthServiceExecutive_cpp1908
CHealthServiceExecutive::OnStartService F00 36C 1 160 01\11\2008-14:59:14
Error ServiceInitialization failed with code -2147467259(E_FAIL).
HealthServiceExecutive HealthServiceExecutive_cpp1401
CHealthServiceExecutive::ErrorShutdownServiceNoLock F00 36C 1 1455
01\11\2008-14:59:14 Error Shutting down service due to error. The supplied
error code is -2147467259(E_FAIL) and the current service state is 0×1.
HealthServiceExecutive HealthServiceExecutive_cpp2710
CHealthServiceExecutive::ShutdownServiceNoLock F00 36C 1 55405504
01\11\2008-14:59:14 Information Health service is stopping.

The error can be caused when the WindowsAccountLockDownSD registry key located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HealthService\Parameters\Management Group\ManagementGroupName is invalid or missing.

He then accessed another computer that was functioning properly, exported the WindowsAccountLockDownSD key from the registry, copied it to the problematic server and imported the key into the registry of that machine. He was then able to successfully start the System Center Management service.

Thanks to J.C. Hornbeck for supplying the solution to the problem, and thank you to ClearPointe engineer Neil Dicus for locating and implementing the solution.