Tag Archives: BSOD

Wisdom-Fu: E-mail alert when you find a memory dump

Res software is probably best know for their PowerFuse product which provides powerful and granular control of a system and a user’s environment.  They also have a terrific product called Wisdom which they describe as “Run Book  Automation for Windows.”  We utilize Wisdom every day to manage our XenApp farm and related servers.  I could spend pages and pages gushing about Wisdom, but I’m going to use this post to show how I use Wisdom to accomplish certain tasks. Naturally, the solution I present can be accomplished in a variety of ways, but I find Wisdom to be elegant and have a very short learning curve.  On top of that, it provides extensive,detailed change management and reporting which many products to not.

Scenario

Typically, a XenApp environment has many, many variables at work that can compromise the stability of a system.  At times, this results in a crash dump and reviewing these dumps can give insight to what caused the problem.

NOTE: If you have a affinity for punishing yourself and want to actually dive into dumps, I highly recommend Crash Dump Analysis by Dmitry Vostokov.  Check out his minidump analysis series to get started.

In our environment, we have many servers and occasionally one will crash, reboot, and come back into production before we get an e-mail alert and we would not know if a dump was generated unless we connected to the server to find it.  I will describe how I created a Wisdom module to detect dump files on a server, copy them to a central location, and send an e-mail alert to the team.

Step 1 – Determine if a dump file exists

You can set up your server to create full and mini dumps by going into Computer properties, clicking the Advanced tab, and selecting Startup and Recovery.

image

This window will show you where the dump files are being created. Full memory dumps are written to: %SystemRoot%\MEMORY.DMP and minidumps are written to: %SystemRoot%\Minidump\

Here is the Wisdom Module that we’ll dive into:

image

The Execute Command task:

image

The command line is: if not exist %WINDIR%\Minidump\*.dmp EXIT /B 1

This is a conditional statement that looks for any file with a *.DMP extension in the Minidump directory.  If the file exists, then the command will successfully end with an error code of 0 (as set by Wisdom) otherwise it will fail and exit with an error code of 1.

Step 2 – E-mail someone that a memory dump file was found

The send e-mail task

image

I’m highlighting the Condition portion of this task because this is where the conditional logic of the previous task (and its exit codes comes into play).  The condition is whether the previous task was successfully completed.  If so, the we set a DUMPEXISTS parameter to 1 if true or 0 if false.

Step 3 – Copy the memory dump to a network share and off the server

The perform file operations task

image

The task creates a directory on a share (based on the server name – more on this later), copies the memory dump file to that location and then deletes it from the server.  The condition on this task is the value of DUMPEXISTS, which we set in the previous task.  If the e-mail task ran, then DUMPEXISTS is set to one, so this task will run and move the dump to a network share.

The remaining 3 tasks for this module repeat the previous 3 for the other memory dump type.

Wrapping Up

The SERVERNAME parameter (which is used when we copy the memory dump to the network) is simply formed by the %COMPUTERNAME% variable.  Wisdom, luckily, has access to the environment variables that are set on the machine the task is run on.

image

Finally, you should set this module to run on every reboot, then you’ll get e-mail alerts that memory dumps were generated if a server crashes.

Building Block

Wisdom allows you to export any resource, module, project or run book as an XML file that can be imorted to another Wisdom database.  Another wonderful feature.  I’ve sanitized a building block of this module for the community.  Due to WordPress file extension restrictions, I’ve renamed the building block with a .DOC extension.  Change it to .XML and you should be able to import it.

module_get dmp files if they exist

Thanks,
Alain

Advertisements