Runbooks or How to Make Documentation As Painful As Possible



I’ve begun the process of creating a Runbook from scratch to document a XenDesktop/XenApp environment that is completely new to me. Runbooks are useful…right? I certainly hope so, as I’m committing a lot of time to creating an exhaustive document. Having all your infrastructure and operations detailed in one place is probably best for knowledge transfer or training.  Unfortunately, it’s also great for outsourcing.

/rant on
In my prior job, the company decided that it did not want to be in the “IT business” anymore. So, as many companies are doing these days they fired “let go” the IT staff and brought in a third-party to manage operations, infrastructure, development, what have you. When this morale-destroying decision was made, my staff were directed (by me) to work with the outsourcing company to regurgitate all their accumulated knowledge. If you’ve been spared this exercise, you are fortunate. What follows is depressing and frustrating. Imparting detailed technical knowledge is difficult, especially when you’re having to document operations and infrastructure that were never documented before. Combine that with the fact that you are obsolescing yourself during this process, and you’ve got to wonder why you should come to work at all.

After many, many meetings, all this information was put into runbooks. When I reviewed them, everything was very high-level and only did a good job of describing our environment. Very little was detailed on the operations side. If you have knowledgeable staff and workers who are familiar with your technology, then I suppose this is sufficient.
/rant off

In my case, I don’t have an operations group to rely on. So, I want to document not only infrastructure in detail, but also operations. This will assist in training, but I also want to use this Runbook to improve this environment with automation. None exists at this stage and it is a barrier to scaling out the environment to support more users. So where do we start?

Define Infrastructure

First, take a 30,000 foot view of your environment. What are the major components? You’ve got a network, server infrastructure and possibly storage. Maybe start with an outline view

  1. Network
  2. Hypervisor
  3. Storage

This can be further refined (for example):

  1. Access Layer
  2. Servers
    1. Infrastructure
    2. XenApp
    3. XenDesktop
  3. Applications
  4. Users
  5. …and so on

Yeah, outlines are nice, but how do you get started?

I’m not going to sugar-coat it, this is the toughest thing to put together especially if you don’t have anything yet. The best way to get started is to keep expanding your outline with more and more detail. You can note procedures for changes, user management, application testing and so on. I’ve previously done a blog post on documenting Citrix Policies using PowerShell and MS Word. I would also highly recommend using Carl Webster’s documentation scripts. This can fill in a lot of detail for you. Carl has the following scripts available:

  • Active Directory
  • DHCP (DHCP must from from Windows Server 2012)
  • NetScaler
  • PVS (5.6 – 7.1)
  • XenApp 5 for Windows Server 2003
  • XenApp 5 for Windows Server 2008
  • XenApp 6
  • XenApp 6.5
  • XenDesktop 4

The scripts are available here: Where to Get Copies of the Various Documentation Scripts

Carl is developing other documentation scripts (XenDesktop 5 for one), so keep checking his web site for updates.

The Long Haul

I’m still actively working on my runbook (123 pages, with attachments almost 200) and have benefited from information I had from the previous environment administrators and earlier jobs, but I have my document always open and I’m making additions and edits to it daily.

So What About You?

I hope this post has inspired you to work on a run book or at least encourage documenting more of your environment and procedures. Please leave any tips, ideas, or questions in the comments. I’d like to know what you do to document and I’d like to know if you don’t document.


Powershell: Installed Software Versions


When you take over an existing environment, there are always surprises that keep one up at night. One of the most important applications that I deliver with my cloud environment was developed in-house. It was never designed to run in a multi-user environment and each installation is specific to a single user. This is a mess to say the least, but one of the largest issues is how to maintain, update, and easily deploy to new users. This post will begin with determining how many versions of this software are installed.

First the Good News…

I love starting with the good news. All the installs are on one XenApp server. This will allow us to iterate though some number of directories and query all the EXE’s for their version without making RPC’s or some other method of file query.

Now the Bad News…

Each install generates 3 (sort-of) random sub-directories, so there is no fixed pattern to where the main EXE resides. I say sort-of because at least they follow a pattern that we can rely on. RegEx to the rescue! The rest of the bad news is that all versions of the software under the latest will not automatically update. Thus I need this script to know how big a problem I have and which users I need to contact to update their software.


I can add more fields to the CSV by doing an AD user lookup and getting an email address and/or phone number as I will have to contact these users to upgrade their software. This will require either Microsoft or Quest Active Directory PowerShell modules.

The Script

Creates a list of users and their installed Software Version.
Creates a list of users and their installed Software Version.

It is recommended that this script be run as an admin.
.PARAMETER UserDirectory
Defaults to %HOMEDRIVE%\Users
Used to iterate through the Users directory and look for Software installs
.PARAMETER Outputfolder
Defaults to current folder location.
Enter a path to output the file.
PS C:\PSScript > .\check-Softwareversion.ps1
Will use all default values.
$env:homedrive = C:

Output file will be in the current directory.
None.  You cannot pipe objects to this script.
No objects are output from this script.  This script creates a CVS document.
NAME: check-Softwareversion.ps1
CHANGE LOG - Version - When - What - Who
1.00 - 06/04/2014 - Inititail script - Alain Assaf
AUTHOR: Alain Assaf
LASTEDIT: June 4, 2014
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>

[parameter(Position = 0, Mandatory=$False )]

[parameter(Position = 1, Mandatory=$False )]

$Appdata = "AppData\Local\apps\2.0"
$verInfo = New-Object System.Collections.ArrayList
$datetime = get-date -format "MM-dd-yyyy_HH-mm"
$Software = "somesoftware.exe"

#Get user list from $UserDirectory
$UserFolders = Get-ChildItem $UserDirectory

foreach ($UserDir in $UserFolders) {
$User = $UserDir.Name
$UserAppdata = "$UserDirectory\$User\$AppData"
if (test-path $UserAppdata) {
$subdir1 = Get-ChildItem $UserAppdata | where {$ -match '\S{8}.\S{3}'}
$TempDir = "$UserAppdata\$subdir1"
$subdir2 = Get-ChildItem $TempDir | where {$ -match '\S{8}.\S{3}'}
$TempDir2 = "$TempDir\$subdir2"
$subdir3 = Get-ChildItem $TempDir2 | where {$ -like 'other..dir*'}
$SoftwareDir = "$TempDir2\$subdir3"
$SoftwareApp = Get-ChildItem $Softwaredir | where {$_.Name -eq $Software}
$SoftwareVersion = ($SoftwareApp.VersionInfo).FileVersion
$SoftwareFileName = ($SoftwareApp.VersionInfo).FileName
$verInfo += New-Object psObject -Property @{'User'=$user;'SoftwareDir'=$SoftwareDir;'SoftwareFilename'=$SoftwareFileName;'SoftwareVersion'=$SoftwareVersion}

#Write report to CSV file
$LogFileName = $Software + "_VersionInfo" + $datetime + ".csv"
$LogFile = $Outputfolder + $LogFilename
$verInfo | Export-Csv -Path $LogFile


Powershell: Automating User Creation


I recently began a new job and it is unique for me. The organization I work for actually helps to generate revenue, and we are a customer of  corporate IT. This is a first for me where I’m not working directly for IT. I had just left a job where corporate viewed IT as too complex, too costly, and not responsive to the business. As a result, 80%-90% of the staff was fired (or re-badged) and all IT service and infrastructure was outsourced. As a manager, I had to preside over the knowledge transfer of my teams to the outsource company and then let  them go. At the same time, I had to maintain some productivity in a morale sapping environment. Needless it say is was rough, but many of my former team members got great jobs afterward as did I.

This outsourcing process had already occurred with my current employer. So, the business engages IT to implement projects, but in my case, they were not able to provide this particular service. So I find that I’m running an independent cloud implementation of XenDesktop. This includes AD (users, groups, OU’s, GPO’s – no trust with corporate AD), DHCP, DNS, File servers, XenApp, vDisk creation, Remote Access, Microsoft updates, and software testing/installs/upgrades. Basically everything from soup to nuts.

This project needs to grow to include many more users, but there is no automation present. So I’ve begun examining the biggest “pain” points in this environment to scale it out. First, we will look at user creation.


It has been years since I’ve been saddled with user account management (except for service accounts and the like). Our cloud users have to remember a separate set of credentials to use our cloud environment; this is already a barrier to adoption. To relieve this, we have made our usernames match the corporate domain usernames, and left it up to the user to make the passwords the same. User account creation begins with the user filling out a SharePoint form. Then we start a manual process to create the user’s account. To relieve this, I exported a request to a CSV file and wrote a PowerShell script that runs in the cloud to create the user, set the password, set the user profile locations, and assign groups. I reviewed several websites for ideas and I noted them in my script’s help area.


For the purposes of this script, the CSV file has the following header fields:

Status Records status of user request in SharePoint site.
Name In the format of Lastname, Firstname
Account Corporate AD username
Password Corporate AD Password
Cloud Version Used to assign groups
Email Address Corporate email
Position/Title Title – used to assign groups
Office Number Work Phone
Mobile Number Mobile Phone
Department Cost Center Added to notes in user account
Manager Added to notes in user account (we do not assume that the manager has a user account in the Cloud AD domain)
Manager Office Number Added to notes in user account
Location or Region Used for contact info
Modality Used for Organization Info
Group Used for Organization Info

It is assumed that you are running this script as a domain admin and the server you are running it from has the Microsoft AD PowerShell cmdlets installed.

If you wish to get feedback from the script while it is running use the -verbose parameter – i.e. >create-clouduser.ps1 -verbose


I don’t have an SMTP server setup in my cloud environment. When I do, I will add routines to the script to check a “New User” mailbox for new emails and then send confirmation to these users that their account was created. This will allow me to bridge the 2 domains (until corporate IT agrees to set up a domain trust).

The Script

    Reads CSV file and creates a user in the YOURCOMPANY domain.
    Reads CSV file and creates a user in the YOURCOMPANY domain.  

    It is recommended that this script be run as an admin. In addition, the Microsoft Active Directory Powershell Cmdlets must be available for user creation.
    Defaults to OU=Users,DC=YOURCOMPANY,DC=local
    OU location where user account will be created. Input must match above format.
    Defaults to any CSV file in C:\scripts\NewUserRequests Folder
    Enter a path to folder where CSV files exist. CSV files must have the following fields:
    Cloud Version
    Email Address
    Office Number
    Mobile Number
    Department Cost Center
    Manager Office Number
    Location or Region
    PS C:\PSScript > .\create-cloudixuser.ps1
    Will use all default values.
    User OU=Users,DC=YOURCOMPANY,DC=local
    PS C:\PSScript > .\create-cloudixuser.ps1 -verbose
    Will use all default values.
    User OU=Users,DC=YOURCOMPANY,DC=local
    Will write feedback/progress messages.
    None.  You cannot pipe objects to this script.
    To see feedback messages use the -verbose common parameter. No objects are output from this script.  This script creates a user creation log.
    NAME: create-clouduser.ps1
    VERSION: 1.00
    CHANGE LOG - Version - When - What - Who
                 1.00 - 06/06/2014 - Initial script - Alain Assaf
    AUTHOR: Alain Assaf
    LASTEDIT: June 6, 2014
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>
<a href=""></a>

    [parameter(Position = 0, Mandatory=$False )]    

    [parameter(Position = 1, Mandatory=$False )]

### FUNCTION: get-mymodule #####################################################
Function Get-MyModule {
    if(-not(Get-Module -name $name)) {
        if(Get-Module -ListAvailable | Where-Object { $ -eq $name }) {
            Import-Module -Name $name
        } #end if module available then import
        else { $false } #module not available
        } # end if not module
    else { $true } #module already loaded
### FUNCTION: get-mymodule #####################################################  

### FUNCTION: get-csvdata #####################################################
function get-csvdata($CSVFolder) {
    $DataFiles = (Get-ChildItem $CSVFolder -recurse -force | Where { $_.Name -like "*.csv" } | Foreach-Object -process { $_.FullName })
    $DataFilesCount = ($DataFiles | measure).count
    Write-Verbose "Discovered $DataFilesCount CSV Data files in $CSVFolder"  

    ForEach ($DataFilesItem in $DataFiles) {
        $FileInfo = Get-Item $DataFilesItem
        $LogDate = $FileInfo.LastWriteTime
        Write-Verbose "Reading data from $DataFilesItem ($LogDate ) "
        [array]$CSVData += Import-CSV $DataFilesItem -header status,name,Account,Password,CloudVer,Email,Title,WorkNumber,MobileNumber,CostCenter,Manager,ManagerWorkPhone,Location,Modality,Group
        ## Only use the header if you want to rename the attributes for the imported objects. If the column headers in the CSV are fine, don't use -header
    [int] $CSVDataCount = $CSVData.Count
    Write-Verbose "Imported $CSVDataCount records"
    Return ($CSVData)
### FUNCTION: get-csvdata #####################################################  

#Import Module(s)
if (!(get-mymodule activedirectory)) {
    write-verbose "Microsoft Active Directory PowerShell Cmdlet not available."
    write-verbose "Please run this script from a system with the Microsoft Active Directory PowerShell Cmdlets installed."

$datetime = get-date -format "MM-dd-yyyy_HH-mm"
$CloudProfilePath = "\\PROFILESERVER.YOURCOMPANY.local\upm\"
$CloudRDSProfilePath = "\\PROFILESERVER.YOURCOMPANY.local\profiles\"
$ScriptRunner = (get-aduser $env:username | select name).name  

#Confirm OU is valid
If (!([adsi]::Exists("LDAP://$UserOU"))) {
    write-verbose "$UserOU IS NOT VALID"
    write-verbose "Please use the following format: OU=Users,DC=YOURCOMPANY,DC=local"

#Confirm CSV Folder exists and has CSV files. If True get the files, otherweise exit
if (test-path $UserFolder) {
    if ((Get-ChildItem $UserFolder | where {$ -like "*.csv"}) -ne $null) {
        $UserList = get-csvdata($UserFolder)
    } else {
        write-verbose "No CSV files in $UserFolder."
} else {
    write-verbose "$UserFolder is not valid"

#Create Users
foreach ($user in $UserList) {
    if (!([bool]([adsisearcher]"samaccountname=$user.Account").FindOne()) -and ($user.status -ne 'Status')) {
        #Get first and last name
        $pos = ($",")
        $SurName = ($, $pos)
        $GivenName = ($$pos+2)
        #Create other account name data
        $DisplayName = $GivenName+ " " + $SurName
        $Initials = ($GivenName).Substring(0,1) + ($SurName).Substring(0,1)
        $Username = $user.Account
        #Get Cloud password to the Philips CODE password
        $Password = $user.Password
        #Create Description
        $Description = $User.Modality + " " + $user.Title
        #Create Contact Info
        $MobilePhone = $User.MobileNumber
        $OfficePhone = $User.WorkNumber
        $Email = $User.Email
        #Create Organization Info
        $Company = "YOUR COMPANY"
        $Department = $User.Modality + " " + $user.Group
        $Divsion = $User.Modality
        $EmployeeNumber = $User.Account
        $Manager = $user.Manager
        $Office = $User.Location
        $Title = $user.Title
        #Create User Profile Location
        $ProfilePath = $CloudProfilePath + $Username
        #Create User in AD
        new-aduser -name $Displayname -DisplayName $DisplayName -GivenName $GivenName -Surname $SurName -SamAccountName $UserName -UserPrincipalName $UPN -Description $Description -MobilePhone $MobilePhone -OfficePhone $OfficePhone -EmailAddress $Email -Company $Company -Department $Department -Division $Division -EmployeeNumber $EmployeeNumber -Office $Office -Title  $Title -ProfilePath $ProfilePath -Path $UserOU
        write-verbose "New AD User Account created for $DisplayName"
        #Set Password
        set-adaccountpassword -Identity $Username -NewPassword (ConvertTo-SecureString -AsPlainText $Password -Force)
        #Set Remote Desktop Services User Profile
        $RDSProfilePath = $CloudRDSProfilePath + $username
        Get-ADUser $username | ForEach-object {
            $ADSI = [ADSI]('LDAP://{0}' -f $_.DistinguishedName)
            try {
            } catch { Write-Verbose $Error[0] }
        #Add User to groups
        #Based on Cloud Version - modify as needed to match AD groups
        $GrpArray = @()
        switch ($user.cloudver) {
            "Field Service" {}
            "Work from Home" {$GrpArray += "WFH"}
            "Pilot" {}
            "QA" {}
            "Sales" {}
            "Inventory" {}
        #Based on Title - modify as needed to match AD groups
        switch ($user.title) {
            "Admin Assistant" {}
            "Director" {$GrpArray += "Leadership"}
            "FEP" {$GrpArray += ""}
            "Helpdesk" {$GrpArray += "Helpdesk"}
            "Helpdesk LEAD" {$GrpArray += "Domain Admins"; $GrpArray += "Helpdesk"}
            "FSE" {$GrpArray += "FSE"}
            "Pre-Sales" {$GrpArray += "Sales"}
            "RF Engineer" {$GrpArray += "Some-Application-Users" ; $GrpArray += "RF"}
            "RLM" {}
            "Sales" {$GrpArray += "Sales"}
            "SDC" {$GrpArray += "SDC"}
            "TC" {$GrpArray += "TC"}
            "Other" {}
        Add-ADPrincipalGroupMembership $username -MemberOf $GrpArray
        write-verbose "The following groups were added to $DisplayName:"
        write-verbose "$GrpArray"
        #Add Notes to user
        get-aduser $username -properties info | foreach { Set-ADUser -Identity $_.samaccountname -Replace @{info="$($ - $Manager"} }
        get-aduser $username -properties info | foreach { Set-ADUser -Identity $_.samaccountname -Replace @{info="$($`r`nManager Phone - $($user.ManagerWorkPhone)"} }
        get-aduser $username -properties info | foreach { Set-ADUser -Identity $_.samaccountname -Replace @{info="$($`r`nEmployee Cost Center - $($user.costcenter)"} }
        get-aduser $username -properties info | foreach { Set-ADUser -Identity $_.samaccountname -Replace @{info="$($`r`nAccount created on: $datetime by $ScriptRunner"} }
        #Enable Account
        Enable-ADAccount -Identity $Username