Are You There EdgeSight? It’s me Worker

Intro

If you rely on EdgeSight to provide accurate and timely information about your farm you have to assume that all your EdgeSight Worker Agents are functioning as expected.  Or do you?  In this post, we will review the information that the EdgeSight console provides you as well as creating a dashboard that can give you a detailed information on your EdgeSight Worker agents.

EdgeSight Console: Configuration Tab

The first place you can review the health of your EdgeSight server and its agents in under the Configuration Tab

image

Along the left-hand side of this screen you will see way to configure your workers, alerts, and other server settings. We’re going to spotlight some items under Server Configuration and Server Status. image

Server Configuration: Status
Your first overview of server health comes when you click on Status under Server Configuration.

image

The first line lists the workers that were and were not updated in the current 24 hour period as well as newly added workers.  Right away you see (in this case) that 48 workers updated and 32 did not.  That’s a large portion of EdgeSight agents that have not uploaded their data into your database and therefore any reports you are running will not include these systems.  The question becomes which systems did not update and why?

ES_ZQUEUE…Gesundheit!

The service on the EdgeSight server that processes payloads from the worker agents is the es_zqueue (seen under Server Script Host Status).  This process is not reporting any issues and there are no pending payloads to process (we’ll look at this more later).

Server Status: Messages
Message Status lists all the system messages generated by EdgeSight.  This includes Agent errors, payload errors, and new agents alerts to name a few.

image

Here you will see which servers had a payload issue (Data Upload), but not a reason why systems have not updated the database.

Server Status: Server Script Host
Clicking on this in your EdgeSight Console will show you the following screen:

image

Here you will see the various modules that keep the EdgeSight database updated, cleaned, and running smoothly. The core_zpd_loader 1 and 2 manage the data payloads from devices with the EdgeSight agent including errors. Clicking on the triangle will reveal the following menu.

image

Clicking on View Log will allow us to investigate why a payload might have failed or created an error.

4/13/2012 5:18:03 AM: PayloadLoader: Starting payload load for C:\Program Files (x86)\Citrix\System Monitoring\Server\EdgeSight\Data\WebLoad\Inst_33.zpd
4/13/2012 5:18:11 AM: PayloadLoader: Payload load completed with errors for C:\Program Files (x86)\Citrix\System Monitoring\Server\EdgeSight\Data\WebLoad\Inst_33.zpd. Error: -2146233088: Citrix.EdgeSight.Loader. System.Exception

As we can see if this example, the payload completed with an error and we can try searching Citrix to see if there is a resolution related to this error, but we do not see which server failed to upload any data.

I’ve walked through the diagnostic information that is available in the EdgeSight console to show that we still do not have a clear sign of which servers have updated the EdgeSight database recently. To address this issue, I did some digging around in the EdgeSight database and created a query that links the instance, machine, and OS_version tables.

The Query

DECLARE @tzbias INT
SELECT @tzbias = case when use_daylight = '0' then standard_bias else daylight_bias end from timezone where tzid = 13
SELECT    i.instid, m.name as 'System', ip_address AS 'IP', product_version AS 'ES Version',
CASE dept_set_type    WHEN 1 THEN 'XenApp' WHEN 2 THEN 'Endpoint' END AS 'ES Agent',
CONVERT(VARCHAR,DATEADD(mi,@tzbias,last_sync),100) AS 'Last Sync',
CONVERT(VARCHAR,DATEADD(mi,@tzbias,last_config_start),100) AS 'Last Config Check',
CONVERT(DECIMAL(19, 2),(last_db_size/1048576.0)) AS 'Last FBDB Size (MB)',
CASE o.short_name    WHEN 'Windows Server 2008' THEN 'W2K8'
WHEN 'Windows Server 2008 R2' THEN 'W2K8R2'
WHEN 'Windows Server 2003' THEN 'W2K3'
WHEN 'Windows XP'           THEN 'XP'
ELSE 'Other' END AS 'OS',
CASE o.ptype        WHEN 'Standard x64 Edition' THEN 'Std x64'
WHEN 'Professional'         THEN 'Pro'
WHEN 'Enterprise Edition'   THEN 'Ent'
WHEN 'Standard Edition'     THEN 'Std'
WHEN 'Enterprise x64 Edition' THEN 'Ent x64'
ELSE 'Other' END AS 'Edition',
sp_level,
CONVERT(VARCHAR,DATEADD(mi,@tzbias,tstamp),111) AS 'Date Added',
cps_farm_name,i.cps_product_name, i.cps_product_version, i.cps_product_service_pack
FROM instance i,machine m, os_version o
WHERE m.machid=i.machid and i.osid = o.osid
ORDER BY dateadd(mi,@tzbias,last_sync) DESC

The Report

image

Click on the image to see a larger version..

With this dashboard (I created it based on the query above in SQL Reporting Services) you can quickly see which servers have updated  the database (Last Sync) and which have updated their local EdgeSight worker agents (Last Config Ck). Armed with this information you can review your EdgeSight Agent worker schedules or check the agent on the system in question to make sure it is communicating with the EdgeSight server.

As always I welcome all questions and comments.

Thanks,
Alain

Advertisements

PowerShell: E-mail Server Load and Assigned LE

NOTE: I recently updated this script for XenApp 7.x environments.
Note: I wrote this script for a XenApp 5.0 environment with PowerShell 2.0 and CTP3 XenApp Cmdlets.

Update: I found that the simple ping used with Test-Connection PowerShell cmdlet was not identifying servers that were not available for login. In our environment we occasionally have a server come back from a scheduled reboot and ping, but not allow RDP/ICA connections. I’ve incorporated a Test-Port function from Aaron Wurthmann’s PowerShell Script Connect-RDP.ps1

Intro

Citrix has utilized load balancing in XenApp for years. This feature allows you to set parameters in a load evaluator, apply it to a server, and have Citrix manage how users are assigned to servers. Despite the numerous tools available on the market that report on your XenApp farm utilization, Citrix user load balancing remains the easiest to implement (and cheapest). That being said, beyond running the “qfarm /load” command over and over, it’s difficult to report on your farm’s load. Thankfully, with the Citrix PowerShell commands we can regularly report on a server’s load (including the load evaluator rules) and e-mail that information.

The Script

###############################################################################
## Title : get-ctxLoadAndLE.ps1
## Description : Gathers server load, assigned LE, and active and disconnected sessions
## Author : Alain Assaf
## Date : 01/11/2012
## Sources : Logging and e-mail sections from Raido
## Sources : http://powershell.com/cs/blogs/ebook/archive/2008/10/23/chapter-4-arrays-and-hashtables.aspx
## Sources : http://technet.microsoft.com/en-us/library/ff730946.aspx
## Sources : http://technet.microsoft.com/en-us/library/ff730936.aspx
## Sources : [test-port function] http://irl33t.com/blog/2011/03/powershell-script-connect-rdp-ps1
## Notes : Assumes Citrix PowerShell cmdlets are installed
#### Changelog ################################################################
# -When - What - Who
# -01/11/2012 -Initial script -Alain Assaf/Chemtura
# -01/18/2012 - Changed way I get user sessions because it was timing out
# -02/20/2012 - Added sendTo variable to add mulitple receipients
# -03/05/2012 - Added lines to include LE Rules
# -04/26/2012 - Added Test-Port function from Aaron Wurthmann (aaron (AT) wurthmann (DOT) com)
###############################################################################

# Test-Port function to test RDP availability
# Written by Aaron Wurthmann (aaron (AT) wurthmann (DOT) com)
function Test-Port{
 Param([string]$srv=$strhost,$port=3389,$timeout=300)
 $ErrorActionPreference = "SilentlyContinue"
 $tcpclient = new-Object system.Net.Sockets.TcpClient
 $iar = $tcpclient.BeginConnect($srv,$port,$null,$null)
 $wait = $iar.AsyncWaitHandle.WaitOne($timeout,$false)
 if(!$wait)
 {
 $tcpclient.Close()
 Return $false
 }
 else
 {
 $error.Clear()
 $tcpclient.EndConnect($iar) | out-Null
 Return $true
 $tcpclient.Close()
 }
}
#Assign e-mail(s) to $sendto variable and SMTP server to $SMTPsrv
$sendto = SOMEONE@YOURDOMAIN.com
$SMTPsrv = 127.0.0.1

#Initialize array
$finalout = @()

#add XenApp Cmdlets
Add-PSSnapin -name *citrix* -erroraction silentlycontinue

#Get server list
$ctxservers = get-xaserver | select -Property servername | sort -Property servername

#Get user sessions
$ctxsrvSessions = Get-XASession | select -property SessionID,ServerName,AccountName,Protocol,State| where {($_.State -eq 'Active' -or $_.State -eq 'Disconnected') -and $_.Protocol -eq 'Ica'}

#Create a new object array with all the data we need
foreach ($srv in $ctxservers) {
 $ctxsrv = $srv.servername
 if (test-port $ctxsrv) {
  $srvload = get-xaserverload -servername $ctxsrv | select -Property ServerName,LoadEvaluatorName,Load,Rules
  $srvactive = @($ctxsrvSessions | where {$_.Servername -eq $ctxsrv -and $_.State -eq 'Active'}).count
  $srvdisconn = @($ctxsrvSessions | where {$_.Servername -eq $ctxsrv -and $_.State -eq 'Disconnected'}).count
  $objctxsrv = new-object System.Object
  $objctxsrv | Add-Member -type NoteProperty -name ServerName -value $srvload.servername
  $objctxsrv | Add-Member -type NoteProperty -name LEName -value $srvload.LoadEvaluatorName
  $objctxsrv | Add-Member -type NoteProperty -name Load -value $srvload.Load
  $objctxsrv | Add-Member -type NoteProperty -name Active -value $srvactive
  $objctxsrv | Add-Member -type NoteProperty -name Disconnected -value $srvdisconn
  $objctxsrv | Add-Member -type NoteProperty -name 'Server User Load' -value $srvload.Rules[0].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'Load Throttling' -value $srvload.Rules[1].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'Memory Usage' -value $srvload.Rules[2].Load.ToString()
  $objctxsrv | Add-Member -type NoteProperty -name 'CPU Usage' -value $srvload.Rules[3].Load.ToString()
  $finalout += $objctxsrv
 } else {
  $objctxsrv = new-object System.Object
  $objctxsrv | Add-Member -type NoteProperty -name ServerName -value $ctxsrv
  $objctxsrv | Add-Member -type NoteProperty -name LEName -value "OFFLINE"
  #$objctxsrv | Add-Member -type NoteProperty -name Load -value "0"
  #$objctxsrv | Add-Member -type NoteProperty -name Active -value "0"
  #$objctxsrv | Add-Member -type NoteProperty -name Disconnected -value "0"
  $finalout += $objctxsrv
 }
}

#Create HTML Header
$head = '<style>
META{http-equiv:refresh content:30;}
BODY{font-family:Verdana;}
TABLE{border-width: 1px;border-style: solid;border-color: black;border-collapse: collapse;}
TH{font-size:12px; border-width: 1px;padding: 2px;border-style: solid;border-color: black;background-color:Aquamarine}
TD{font-size:12px; border-width: 1px;padding: 2px;border-style: solid;border-color: black;background-color:GhostWhite}
</style>
<script type="text/javascript">
$(function(){
var linhas = $("table tr");
$(linhas).each(function(){
var Valor = $(this).find("td:first").html();
if(Valor == "OFFLINE"){
  $(this).find("td").css("background-color","Red");
}else if(Valor == "Running"){
  $(this).find("td").css("background-color","Green");
}
});
});
</script>
'
#$header = "<H5>XenApp DASHBOARD</H5>"
$title = "XenApp DASHBOARD"
#$body = $finalout | ConvertTo-Html -head $head -body $header -title $title
#$body = $finalout | ConvertTo-Html -head $head -title $title

#Create mail parameters
$messageParameters = @{
 Subject = "Report: Server Load and Assigned LE for XenApp Farm"
 Body = $finalout | ConvertTo-Html -head $head -title $title | out-string
 From = "Admin-Citrixtask@chemtura.com"
 To = $sendto
 SmtpServer = "$SMTPsrv"
}

#Send e-mail with Server load data
Send-MailMessage @messageParameters -BodyAsHtml

#Uncomment to output the results in an HTA file to view in a browser
#$finalout | ConvertTo-Html -head $head -title $title | out-file $env:temp\report.hta
#ii $env:temp\report.hta

Example e-mail

Currently the script records the following:

  • Server Name
  • Assigned Load Evaluator
  • Server Load
  • Active Sessions
  • Disconnected Sessions
  • Load Evaluator Rules (you will have to modify the above script depending on which LE rules are using)
    • User Load
    • Load Throttling
    • Memory Usage
    • CPU Usage
image
click on image to make it larger

I created a repeating scheduled task on a XenApp server in the farm I wanted to report on. Now my team gets an hourly report of the farm load.

Explore more

Load Manager Values Explained

Load Manager Rules Explained

Performance Monitor Counters Used by Load Manager

Troubleshooting Load Balancing Issues

How to Enable the Load Manager Log for Load Balancing Issues

Conclusion

Please try it out and let me know how it works for you. As you can see above, I borrowed a javascript that was supposed to color the cells for an OFFLINE server red, but have not been able to get it to work. If anyone has a solution, I will be happy to update this post and spotlight you, your blog, or twitter account.

As always, I welcome all comments and questions.

Thanks,
Alain

EdgeSight: What’s the Point?

Intro

In my experience, EdgeSight is not typically deployed in XenApp environments. The reasons for this are many including, it seems redundant and overlaps with an already installed solution, it’s hard to use (sorry Citrix, this is a great tool, but it almost requires a dedicated resource to manage it), most XenApp administrators/managers do not know why or how it should be used, and finally you need a platinum license in order to run the agent in advanced mode which is a deal-breaker for some companies.

In this post, I will cover a use case for EdgeSight that measures memory utilization of IE after a registry change is made.

Why Internet Explorer will kill your memory

Trond Eirik Haavarstein at XenAppBlog wrote a 2-part series on running Internet Explorer 7 or 8 in a terminal server environment and its impact on memory utilization. He shows how memory is used by multiple tabs in IE and references an MSDN Blog about a registry change that will modify the Tab Process Growth.

Any XenApp/Terminal Services environment is an exercise in resource management so we wanted to apply this change in our staging environment (prior to applying it in production) to determine if it will have a positive impact.

EdgeSight – Oh I get it!

After applying the registry change via a GPO, we waited a few days in order to gather enough data in EdgeSight to make a before/after comparison. Now we will walk through selecting the report we need and running it with the correct criteria.

Log into EdgeSight and go to the Browse Tab to select the report we need:

image

Here’s what you see (click on the picture to make it bigger)…

image

What! 143 reports? How do I get what I need? Well, we need to report on the performance of Internet Explorer. EdgeSight refers to applications as processes.  These can be executables launched by users (published applications) or processes run by system accounts. Click on Process under the Object Type table. Now we see (click on the picture to make it bigger)…

image

We’ve filtered our report list to just 33 items. If we wish to further filter our results, we can select Historical for the time frame and Performance for Data Type which gives us 15 reports to look through.  Since we know we’re looking for a report related to memory, we could have just as easily typed “memory” in the search field which gives us the following…

image

Then click on Processes for the Object Type would give us just 3 results…

image

You can see that you can narrow your choices in a couple of different ways.  The report we need is the Process Memory Usage. Let’s click on this report and see what the default result is.

image

As you can see above, we are looking at the entire environment monitored by EdgeSight and looking at the top 20 processes for the past week. The resulting report is grouped by Process, then Device, and then user…

image

We need to narrow the parameters of this report to give us the info we need for Internet Explorer…

We can first choose to limit the Department to just our XenApp servers where we made the registry change. Next we can hit the Category drop down and select Web Browsers.

image

If you only have IE installed on your XenApp servers, this makes it easy. Otherwise, you will have to check the Optional Parameters and find the process in the Process Picker Window…

image

Type iexplore.exe into the Filter window, make sure the By File Name radio button is checked and click filter

image

Select it and click Ok. Now Internet Explorer is the selected process in the Optional Parameters section.

image

Clicking on Go will show us the top 20 instances of Internet Explorer’s usage of Virtual, Private, and Working Set memory over the period of 3/22 to 3/29. For a detailed explanation of memory, I would refer to Mark Russinovich’s blog post here.

image

We want to track the change in memory usage over time, so we have to modify how the data is grouped. Changing the grouping to Date, Process, and All gives us the following:

image

Now it’s a matter of selecting the correct dates and comparing the results to show how the change we made has positively impacted IE memory usage.

Here are the results I recorded with EdgeSight in our staging environment:

Before:

After:

image

I found that the IE memory footprint was reduced by 10,000 – 30,000 kb after applying the registry change. That works out to around 10-30 MB per user which isn’t too bad in shared environment.

Have you tried this registry change and if so what was your result? How did you measure it? Comment Below

Thanks,
Alain