The Cloud is Blind: Tricerat Simplify Monitoring

NOTE: I was not asked by Tricerat to review their product, but I am taking a look at it to see if it would provide better visibility to my current XenDesktop/XenApp environment.

Intro

It's always Citrix

Citrix is a blessing and a curse for any virtualization admin. A blessing, because it allows you to deliver your company’s resources in a very dynamic and flexible way. A curse, because Citrix touches every aspect of your company’s infrastructure, and as any Citrix admin knows, any fault or degradation in that infrastructure can be magnified in your Citrix environment. If you’re lucky, you have the tools to identify these faults and if you are really lucky, you can inform other support teams to fix their problem. More often than not, this is a luxury and you have to rely on EdgeSight and whatever troubleshooting skills you have developed to fix problems like this. I’m not knocking EdgeSight, but it still requires a dedicated resource to configure, manage, and monitor and has little to no visibility into the other layers that make up your environment.

With more and more environments moving to the cloud, reporting and monitoring can be hampered due to the following reasons:

  1. Your cloud provider only offers/supports one monitoring tool.
  2. Your cloud provider only allows access to the application layer so you cannot install an appliance on a hypervisor, so you must use a product like EdgeSight.
  3. Your cloud provider’s architecture is so abstracted that monitoring is very limited (i.e. Amazon’s Workspaces).
  4. Your cloud provider does not want you digging into their architecture and pointing fingers at them.

When you’re working in a cloud environment, there are more variables that are out of your control. I’m reviewing various monitoring solutions that I hope will help me deal with performance issues in my cloud deployment and give me insight to the Cloud Provider’s resources as well as issues with my corporate resources.

Tricerat

Tricerat, known for addressing the terrible state of printing in  virtual environments, has developed a monitoring solution that “…provides out-of-the-box functionality and setup wizards that enable administrators to monitor their XenApp and XenDesktop deployments across the physical and virtual infrastructure, to determine root causes and automate fix actions.”

Install

Downloading the 30-day trial is as simple as providing some contact info and downloading the setup file (both 32-bit and 64-bit available). It is recommended that you perform the install with an account that is a domain admin and also a farm admin of your XenApp, XenDesktop, and PVS farms.

Setup

(I’m using Slideshare to present the install screenshots)

 

Note:If you are installing Simplify Monitoring on a system that already hosts a web site, you should run MonitorITCFU.exe which is located in the \Program FIles (x86)\Simplify Monitoring\Bin folder. This utility will allow you to change the HTTP port to something other than 80 which is the default. When you open it, you must go to File –> Open and select the Default.btc file to make change. Once you’ve saved your changes, then stop and restart the “MonitorIT Server Service”.

First Launch (after a reboot)

Hey, look! A desktop icon. Let’s click on it:

firstlaunch1

If you’ve looked at several monitoring solutions, the below interface will look familiar…

firstlaunch2

This is the same monitoring product that Goliath Technologies has OEM’d to other companies. Let’s give Tricerat the benefit of the doubt and see what their years of experience gives us.

First, there is a Help Guide:

firstlaunch3

My environment is based on VMWare, so let’s see how that is configured.

firstlaunch4

I added my vCenter server…

firstlaunch5

…and my hosts were added…

firstlaunch6

Clicking Next goes to the Host Licensing screen. You are limited to 10 hosts in the 30-day trial:

firstlaunch7

After completion, you are presented with an option to install Simplify Monitoring’s Console as a Plug-In into vCenter.

firstlaunch8

After you exit this wizard, you presented with a next step configuration guide.

firstlaunch9

So, I want to “See alerts and the health of your system” to see if there are any red flags I did not know about.

firstlaunch10

Hey, one of my servers is showing yellow. As you can see from the icons, we’ve got the typical RYG alert, the type of system (ESX, Linux, Windows), and the system status. After a few seconds, I got lots of red…

firstlaunch11

Time to panic? Let’s dive in and see what the alert is about (you can sort the system status to only show faults and to move all the faults to the top – which is why the above picture only shows red). The first system is a virtual NetScaler, so I’ll click on one of my XenDesktops to see what is triggering the alert.

firstlaunch12

The alert is due to available logical drive space. Clicking on the Watch/Alert Detail (the icon under Inf in the above picture) gives us…

firstlaunch13

Again, this environment is very new to me. The above location is created when MCS is used. I know that PVS was used in this environment, but it is not active now, so this alert is correct, but not relevant. Let’s see if we can silence this alert and see what else is going on. Going back to the Configuration Guide, I’m going to click on “View or Modify default alerting conditions and thresholds”

firstlaunch14

To configure Monitoring rules, click on the Configure tab and then Monitoring Rules.

firstlaunch15

The rule we want, “VMWare Virtual Machnine Alert” covers all major metrics.

firstlaunch16

I added the path (not including the drive letter) that was causing an issue to the Exclude field and then acknowledged those existing issues. Now back to the dashboard to see what we can see.

firstlaunch17

Yeah, but what about my Citrix environment?

Simplify Monitoring provides many different built-in rule sets that cover all major components of a Citrix environment:

image

Note the Windows and SQL dependency checks that are related to delivering Citrix XenApp. Each rule can be assigned to a server or set of servers. When you select a rule you can click on Edit to get more detail.

image

This screen allows you to assign the rule to a group of servers. If you click on the Service radio button, the screen changes to show the service that is monitored and all systems that are running an agent. This may be easier to assign the rule in your environment:

image

You can also configure the schedule of the alert…

image

…the method of notification…

image
image  image image
image

…and any remediation methods…

image

You can also recreate custom rules to monitor any number of items/systems:

image

How do I see it all?

To see your alerts you go to the Monitor Tab. The first screen will be the groups (if any) that were configured.

image

You can click on Categories to see your infrastructure grouped:

image

Status will show all detected devices in your environment:

image

Alerts will list all currently detected alerts.

image

If you click CPU, Memory, Storage, and Availability you will get a high-level look at these metrics. You can sort on any column by clicking on it.

image
image
image

You can also see your datastores from the perspective of your hypervisor if you click on Virtual Storage.

image

Tricerat has also included a nice graphing module and covers the main pain points of monitoring a system holistically. I unfortunately do not have any graphs from my environment available, but I grabbed a screen shot from a presentation here to give you an idea:

image

You setup a dashboard and add graphs to it.  It’s a nice view of your system and you can cycle through different dashboards to show you your VMWare, XenApp, and  XenDesktop environments.

Reports

You can also schedule and run reports on most metrics in your environment. I’m not able to show any screen shots of this, but I would recommend viewing a nice video of the major features of Tricerat Simplify Monitoring here:

Thanks,
Alain

Advertisements

PowerShell: E-mail Server Load and Assigned LE

NOTE: I recently updated this script for XenApp 7.x environments.
Note: I wrote this script for a XenApp 5.0 environment with PowerShell 2.0 and CTP3 XenApp Cmdlets.

Update: I found that the simple ping used with Test-Connection PowerShell cmdlet was not identifying servers that were not available for login. In our environment we occasionally have a server come back from a scheduled reboot and ping, but not allow RDP/ICA connections. I’ve incorporated a Test-Port function from Aaron Wurthmann’s PowerShell Script Connect-RDP.ps1

Intro

Citrix has utilized load balancing in XenApp for years. This feature allows you to set parameters in a load evaluator, apply it to a server, and have Citrix manage how users are assigned to servers. Despite the numerous tools available on the market that report on your XenApp farm utilization, Citrix user load balancing remains the easiest to implement (and cheapest). That being said, beyond running the “qfarm /load” command over and over, it’s difficult to report on your farm’s load. Thankfully, with the Citrix PowerShell commands we can regularly report on a server’s load (including the load evaluator rules) and e-mail that information.

The Script

###############################################################################
## Title : get-ctxLoadAndLE.ps1
## Description : Gathers server load, assigned LE, and active and disconnected sessions
## Author : Alain Assaf
## Date : 01/11/2012
## Sources : Logging and e-mail sections from Raido
## Sources : http://powershell.com/cs/blogs/ebook/archive/2008/10/23/chapter-4-arrays-and-hashtables.aspx
## Sources : http://technet.microsoft.com/en-us/library/ff730946.aspx
## Sources : http://technet.microsoft.com/en-us/library/ff730936.aspx
## Sources : [test-port function] http://irl33t.com/blog/2011/03/powershell-script-connect-rdp-ps1
## Notes : Assumes Citrix PowerShell cmdlets are installed
#### Changelog ################################################################
# -When - What - Who
# -01/11/2012 -Initial script -Alain Assaf/Chemtura
# -01/18/2012 - Changed way I get user sessions because it was timing out
# -02/20/2012 - Added sendTo variable to add mulitple receipients
# -03/05/2012 - Added lines to include LE Rules
# -04/26/2012 - Added Test-Port function from Aaron Wurthmann (aaron (AT) wurthmann (DOT) com)
###############################################################################

# Test-Port function to test RDP availability
# Written by Aaron Wurthmann (aaron (AT) wurthmann (DOT) com)
function Test-Port{
 Param([string]$srv=$strhost,$port=3389,$timeout=300)
 $ErrorActionPreference = "SilentlyContinue"
 $tcpclient = new-Object system.Net.Sockets.TcpClient
 $iar = $tcpclient.BeginConnect($srv,$port,$null,$null)
 $wait = $iar.AsyncWaitHandle.WaitOne($timeout,$false)
 if(!$wait)
 {
 $tcpclient.Close()
 Return $false
 }
 else
 {
 $error.Clear()
 $tcpclient.EndConnect($iar) | out-Null
 Return $true
 $tcpclient.Close()
 }
}
#Assign e-mail(s) to $sendto variable and SMTP server to $SMTPsrv
$sendto = SOMEONE@YOURDOMAIN.com
$SMTPsrv = 127.0.0.1

#Initialize array
$finalout = @()

#add XenApp Cmdlets
Add-PSSnapin -name *citrix* -erroraction silentlycontinue

#Get server list
$ctxservers = get-xaserver | select -Property servername | sort -Property servername

#Get user sessions
$ctxsrvSessions = Get-XASession | select -property SessionID,ServerName,AccountName,Protocol,State| where {($_.State -eq 'Active' -or $_.State -eq 'Disconnected') -and $_.Protocol -eq 'Ica'}

#Create a new object array with all the data we need
foreach ($srv in $ctxservers) {
 $ctxsrv = $srv.servername
 if (test-port $ctxsrv) {
  $srvload = get-xaserverload -servername $ctxsrv | select -Property ServerName,LoadEvaluatorName,Load,Rules
  $srvactive = @($ctxsrvSessions | where {$_.Servername -eq $ctxsrv -and $_.State -eq 'Active'}).count
  $srvdisconn = @($ctxsrvSessions | where {$_.Servername -eq $ctxsrv -and $_.State -eq 'Disconnected'}).count
  $objctxsrv = new-object System.Object
  $objctxsrv | Add-Member -type NoteProperty -name ServerName -value $srvload.servername
  $objctxsrv | Add-Member -type NoteProperty -name LEName -value $srvload.LoadEvaluatorName
  $objctxsrv | Add-Member -type NoteProperty -name Load -value $srvload.Load
  $objctxsrv | Add-Member -type NoteProperty -name Active -value $srvactive
  $objctxsrv | Add-Member -type NoteProperty -name Disconnected -value $srvdisconn
  $objctxsrv | Add-Member -type NoteProperty -name 'Server User Load' -value $srvload.Rules[0].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'Load Throttling' -value $srvload.Rules[1].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'Memory Usage' -value $srvload.Rules[2].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'CPU Usage' -value $srvload.Rules[3].Load.ToString()
  $finalout += $objctxsrv
 } else {
  $objctxsrv = new-object System.Object
  $objctxsrv | Add-Member -type NoteProperty -name ServerName -value $ctxsrv
  $objctxsrv | Add-Member -type NoteProperty -name LEName -value "OFFLINE"
  #$objctxsrv | Add-Member -type NoteProperty -name Load -value "0"
  #$objctxsrv | Add-Member -type NoteProperty -name Active -value "0"
  #$objctxsrv | Add-Member -type NoteProperty -name Disconnected -value "0"
  $finalout += $objctxsrv
 }
}

#Create HTML Header
$head = '<style>
META{http-equiv:refresh content:30;}
BODY{font-family:Verdana;}
TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse;}
TH{font-size:12px; border-width: 1px;padding: 2px;border-style: solid;border-color: black;background-color:Aquamarine}
TD{font-size:12px; border-width: 1px;padding: 2px;border-style: solid;border-color: black;background-color:GhostWhite}
</style>
<script type="text/javascript">
$(function(){
var linhas = $("table tr");
$(linhas).each(function(){
var Valor = $(this).find("td:first").html();
if(Valor == "OFFLINE"){
  $(this).find("td").css("background-color","Red");
}else if(Valor == "Running"){
  $(this).find("td").css("background-color","Green");
}
});
});
</script>
'
#$header = "<H5>XenApp DASHBOARD</H5>"
$title = "XenApp DASHBOARD"
#$body = $finalout | ConvertTo-Html -head $head -body $header -title $title
#$body = $finalout | ConvertTo-Html -head $head -title $title

#Create mail parameters
$messageParameters = @{
 Subject = "Report: Server Load and Assigned LE for XenApp Farm"
 Body = $finalout | ConvertTo-Html -head $head -title $title | out-string
 From = "Admin-Citrixtask@chemtura.com"
 To = $sendto
 SmtpServer = "$SMTPsrv"
}

#Send e-mail with Server load data
Send-MailMessage @messageParameters -BodyAsHtml

#Uncomment to output the results in an HTA file to view in a browser
#$finalout | ConvertTo-Html -head $head -title $title | out-file $env:temp\report.hta
#ii $env:temp\report.hta

Example e-mail

Currently the script records the following:

  • Server Name
  • Assigned Load Evaluator
  • Server Load
  • Active Sessions
  • Disconnected Sessions
  • Load Evaluator Rules (you will have to modify the above script depending on which LE rules are using)
    • User Load
    • Load Throttling
    • Memory Usage
    • CPU Usage
image
click on image to make it larger

I created a repeating scheduled task on a XenApp server in the farm I wanted to report on. Now my team gets an hourly report of the farm load.

Explore more

Load Manager Values Explained

Load Manager Rules Explained

Performance Monitor Counters Used by Load Manager

Troubleshooting Load Balancing Issues

How to Enable the Load Manager Log for Load Balancing Issues

Conclusion

Please try it out and let me know how it works for you. As you can see above, I borrowed a javascript that was supposed to color the cells for an OFFLINE server red, but have not been able to get it to work. If anyone has a solution, I will be happy to update this post and spotlight you, your blog, or twitter account.

As always, I welcome all comments and questions.

Thanks,
Alain

EdgeSight: What’s the Point?

Intro

In my experience, EdgeSight is not typically deployed in XenApp environments. The reasons for this are many including, it seems redundant and overlaps with an already installed solution, it’s hard to use (sorry Citrix, this is a great tool, but it almost requires a dedicated resource to manage it), most XenApp administrators/managers do not know why or how it should be used, and finally you need a platinum license in order to run the agent in advanced mode which is a deal-breaker for some companies.

In this post, I will cover a use case for EdgeSight that measures memory utilization of IE after a registry change is made.

Why Internet Explorer will kill your memory

Trond Eirik Haavarstein at XenAppBlog wrote a 2-part series on running Internet Explorer 7 or 8 in a terminal server environment and its impact on memory utilization. He shows how memory is used by multiple tabs in IE and references an MSDN Blog about a registry change that will modify the Tab Process Growth.

Any XenApp/Terminal Services environment is an exercise in resource management so we wanted to apply this change in our staging environment (prior to applying it in production) to determine if it will have a positive impact.

EdgeSight – Oh I get it!

After applying the registry change via a GPO, we waited a few days in order to gather enough data in EdgeSight to make a before/after comparison. Now we will walk through selecting the report we need and running it with the correct criteria.

Log into EdgeSight and go to the Browse Tab to select the report we need:

image

Here’s what you see (click on the picture to make it bigger)…

image

What! 143 reports? How do I get what I need? Well, we need to report on the performance of Internet Explorer. EdgeSight refers to applications as processes.  These can be executables launched by users (published applications) or processes run by system accounts. Click on Process under the Object Type table. Now we see (click on the picture to make it bigger)…

image

We’ve filtered our report list to just 33 items. If we wish to further filter our results, we can select Historical for the time frame and Performance for Data Type which gives us 15 reports to look through.  Since we know we’re looking for a report related to memory, we could have just as easily typed “memory” in the search field which gives us the following…

image

Then click on Processes for the Object Type would give us just 3 results…

image

You can see that you can narrow your choices in a couple of different ways.  The report we need is the Process Memory Usage. Let’s click on this report and see what the default result is.

image

As you can see above, we are looking at the entire environment monitored by EdgeSight and looking at the top 20 processes for the past week. The resulting report is grouped by Process, then Device, and then user…

image

We need to narrow the parameters of this report to give us the info we need for Internet Explorer…

We can first choose to limit the Department to just our XenApp servers where we made the registry change. Next we can hit the Category drop down and select Web Browsers.

image

If you only have IE installed on your XenApp servers, this makes it easy. Otherwise, you will have to check the Optional Parameters and find the process in the Process Picker Window…

image

Type iexplore.exe into the Filter window, make sure the By File Name radio button is checked and click filter

image

Select it and click Ok. Now Internet Explorer is the selected process in the Optional Parameters section.

image

Clicking on Go will show us the top 20 instances of Internet Explorer’s usage of Virtual, Private, and Working Set memory over the period of 3/22 to 3/29. For a detailed explanation of memory, I would refer to Mark Russinovich’s blog post here.

image

We want to track the change in memory usage over time, so we have to modify how the data is grouped. Changing the grouping to Date, Process, and All gives us the following:

image

Now it’s a matter of selecting the correct dates and comparing the results to show how the change we made has positively impacted IE memory usage.

Here are the results I recorded with EdgeSight in our staging environment:

Before:

After:

image

I found that the IE memory footprint was reduced by 10,000 – 30,000 kb after applying the registry change. That works out to around 10-30 MB per user which isn’t too bad in shared environment.

Have you tried this registry change and if so what was your result? How did you measure it? Comment Below

Thanks,
Alain