Tag Archives: memory dump

PowerShell: Check for Dump Files and Alert

Intro

Citrix Provisioning Services has become a cornerstone technology in many XenApp and XenDesktop deployments.  Provisioned devices rely on a write-cache to store new data while they are running and Citrix best practice recommends pointing pagefiles, event logs, and other write-intensive files to this write-cache.  This also includes EdgeSight data.  The one draw back is that if the write-cache fills up, the provisioned device stops working.  I’ve written a simple PowerShell script that will scan the write-cache for DMP files and the dedicateddumpfile.sys and send an e-mail alert if something is found.

The script

###############################################################################
## Title       : check-dmpfiles.ps1
## Description : Checks writecache drive for dmp files and e-mails if found
## Author      : Alain Assaf
## Date        : 08/30/2011
## Notes       :
#### Changelog ################################################################
# -When - What - Who
# -08/30 -Initial script - Alain Assaf
###############################################################################

$emailBody=$null
$dmplocation=$null
$dmplocation="d:\"
$searchfor = @("*.dmp","dedicateddumpfile.sys")

### Script Body ###############################################################
$results = Get-ChildItem $dmplocation -Include $searchfor -Recurse -Force -ErrorAction SilentlyContinue
if (!$results) {
exit
} else {
foreach ($file in $results) {
$dmpfile = (Get-ChildItem $file -force).Name
$dmpfilesize = '{0:N2}' -f (((Get-Item $file -force).length) / 1048576.0)
$dmpfiledir = (Get-ChildItem $file -force).DirectoryName
$emailBody = $emailBody + "`nFound: $dmpfile"
$emailBody = $emailBody + "`nSize: $dmpfilesize MB"
$emailBody = $emailBody + "`nIn directory: $dmpfiledir"
$emailBody = $emailBody + "`n------------------------------------------------------------------------------"
}
$drvroot = (Get-Item $file -force).psdrive.Root
$drvfreespace = '{0:N2}' -f ((Get-Item $file -force).psdrive.Free / 1073741824.0)
$emailBody = $emailBody + "`n$drvroot Drive free space: $drvfreespace GB"
}

### E-mail Creation: Variables, Sending #######################################
#$emailFrom = "dmpfilecheck@company.com"
$emailTo ="sbcteam@company.com"
$subject = "ALERT: Dump file found on $hostname"
$smtpServer = "192.168.1.1"
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$msg = new-object Net.Mail.MailMessage($emailFrom, $emailTo, $subject, $emailBody)
if ($emailBody) {$smtp.send($msg)}

Script Logic Flow

Here’s an example e-mail:

Assaf, Alain
From: dmpfilecheck
Sent: Thursday, September 01, 2011 9:19 PM
To: Assaf, Alain
Subject: ALERT: Dump file found on XENAPPPVS

Found: rscorsvc.exe.dmp
Size: 86.65 MB
In directory: D:\Citrix\System Monitoring\Data
------------------------------------------------------------------------------
Found: rscorsvc.exe.hang.dmp
Size: 95.80 MB
In directory: D:\Citrix\System Monitoring\Data
------------------------------------------------------------------------------
Found: rscorsvc.exe.shutdown.dmp
Size: 66.71 MB
In directory: D:\Citrix\System Monitoring\Data
------------------------------------------------------------------------------
D:\ Drive free space: 9.24 GB

Thanks,
Alain

Wisdom-Fu: E-mail alert when you find a memory dump

Res software is probably best know for their PowerFuse product which provides powerful and granular control of a system and a user’s environment.  They also have a terrific product called Wisdom which they describe as “Run Book  Automation for Windows.”  We utilize Wisdom every day to manage our XenApp farm and related servers.  I could spend pages and pages gushing about Wisdom, but I’m going to use this post to show how I use Wisdom to accomplish certain tasks. Naturally, the solution I present can be accomplished in a variety of ways, but I find Wisdom to be elegant and have a very short learning curve.  On top of that, it provides extensive,detailed change management and reporting which many products to not.

Scenario

Typically, a XenApp environment has many, many variables at work that can compromise the stability of a system.  At times, this results in a crash dump and reviewing these dumps can give insight to what caused the problem.

NOTE: If you have a affinity for punishing yourself and want to actually dive into dumps, I highly recommend Crash Dump Analysis by Dmitry Vostokov.  Check out his minidump analysis series to get started.

In our environment, we have many servers and occasionally one will crash, reboot, and come back into production before we get an e-mail alert and we would not know if a dump was generated unless we connected to the server to find it.  I will describe how I created a Wisdom module to detect dump files on a server, copy them to a central location, and send an e-mail alert to the team.

Step 1 – Determine if a dump file exists

You can set up your server to create full and mini dumps by going into Computer properties, clicking the Advanced tab, and selecting Startup and Recovery.

image

This window will show you where the dump files are being created. Full memory dumps are written to: %SystemRoot%\MEMORY.DMP and minidumps are written to: %SystemRoot%\Minidump\

Here is the Wisdom Module that we’ll dive into:

image

The Execute Command task:

image

The command line is: if not exist %WINDIR%\Minidump\*.dmp EXIT /B 1

This is a conditional statement that looks for any file with a *.DMP extension in the Minidump directory.  If the file exists, then the command will successfully end with an error code of 0 (as set by Wisdom) otherwise it will fail and exit with an error code of 1.

Step 2 – E-mail someone that a memory dump file was found

The send e-mail task

image

I’m highlighting the Condition portion of this task because this is where the conditional logic of the previous task (and its exit codes comes into play).  The condition is whether the previous task was successfully completed.  If so, the we set a DUMPEXISTS parameter to 1 if true or 0 if false.

Step 3 – Copy the memory dump to a network share and off the server

The perform file operations task

image

The task creates a directory on a share (based on the server name – more on this later), copies the memory dump file to that location and then deletes it from the server.  The condition on this task is the value of DUMPEXISTS, which we set in the previous task.  If the e-mail task ran, then DUMPEXISTS is set to one, so this task will run and move the dump to a network share.

The remaining 3 tasks for this module repeat the previous 3 for the other memory dump type.

Wrapping Up

The SERVERNAME parameter (which is used when we copy the memory dump to the network) is simply formed by the %COMPUTERNAME% variable.  Wisdom, luckily, has access to the environment variables that are set on the machine the task is run on.

image

Finally, you should set this module to run on every reboot, then you’ll get e-mail alerts that memory dumps were generated if a server crashes.

Building Block

Wisdom allows you to export any resource, module, project or run book as an XML file that can be imorted to another Wisdom database.  Another wonderful feature.  I’ve sanitized a building block of this module for the community.  Due to WordPress file extension restrictions, I’ve renamed the building block with a .DOC extension.  Change it to .XML and you should be able to import it.

module_get dmp files if they exist

Thanks,
Alain