Session recording deep dive and troubleshooting final version
Transcript of Session recording deep dive and troubleshooting final version
© 2016 Citrix | Confidential
Learning Together to Deliver the Future18-21 July, 2016
Session Recording Deep Dive and Troubleshooting
[email protected] | 涂振凯Senior Technical Support Engineer2016/07/19
© 2016 Citrix | Confidential
Agenda
• What is Session Recording
• Session Recording Architecture & Work Flow
• Components in Details
• Deployment Tips & Troubleshooting
© 2015 Citrix | Confidential
What’s Session Recording
© 2016 Citrix | Confidential
Product History
Editon Main Production Version RemarkSmartAuditor 1.1 Citrix Presentation Server 4.5/ 5 for
Windows 2003End of life since 31-Mar-2013
SmartAuditor 1.2 Citrix XenApp 5.0 for Windows 2008 End of life since 13-Jan-2015SmartAuditor 1.3 Citrix XenApp 6.0/6.5Session Recording 7.6 XenApp and XenDesktop 7.6 Released Q1 2015Session Recording 7.8 XenApp and XenDesktop 7.8 Provide the ability to record
the Desktop OS VDA
The SmartAuditor feature is abandoned from the XenApp and XenDesktop 7.0 to 7.5, since it’s very welcomed by Customers from the APAC region, it
returns since as a part of XenApp and XenDesktop 7.6
© 2016 Citrix | Confidential
What is Session Recording
• Session Recording is like a Digital Video Recorder…– Session Recording allows you to record the on-screen activity of any user session hosted
from a VDA machine
• Session Recording leverages Citrix VDA to provide…– Policy-based recording of an ICA session & associated session information– Manages recording & logs ICA sessions to persistent storage– Ability to search a catalog of recorded ICA sessions– Playback of recorded ICA sessions
© 2016 Citrix | Confidential
Session Recording Key Capabilities
• Records the entire ICA session
• Provides powerful policy-based recording capabilities
• Provides catalog search of recorded sessions
• Records user sessions very efficiently– A typical 8 hour outlook recording is about 20MB
• Digitally signs recorded session files to ensure data integrity
• Records session data centrally and stores it securely
• Automates session recording without requiring client-side software
© 2016 Citrix | Confidential
Use Case – User behavior monitoring / audit
• Management team wish to Record / Monitor / Audit User Activity– High-value / sensitive transactions– Users with high-privilege access– Security, corporate and regulatory policy compliance– Guest/ third party employee access to corporate systems
© 2016 Citrix | Confidential
Use Case – Collect feedback about the Product Design
• Collect feedback about product design from customer’s behavior – After deploying a new prototype
website or launching A/B test on User Experience or Navigation, Product Manager or Product Designer could check the recorded sessions to see if end user likes the new changes or feels hard to navigate
© 2016 Citrix | Confidential
Use Case - Technical Support and Troubleshooting
• Accelerate problem troubleshooting– Record the one-time issue
– Record the hard to reproduce issue
– Problem fast review
© 2015 Citrix | Confidential
Session Recording Architecture & Workflow
© 2016 Citrix | Confidential
Session Recording Architecture and Workflow
Unsecure Network
Client Users
Secure DatacenterSession Recording Agent
Server VDA or Desktop VDA
Session Recording Policy Console
SR ServerSession Recording Player
3rd Party Archive Solution
Storage
1
2
2
3
4
5
Establish ICA Connection
Verify recording Policy
Send Session Data
Retrieve Session Data
Archive files
Log Session Data and Write to Storage
2
3
4
5
0
1
Delivery ControllerApplication Enum
0
0 Predefined configurationSet the recording policy
© 2016 Citrix | Confidential
Components in Details
© 2016 Citrix | Confidential
Session Recording Server Components• Session Recording Broker– Installed as an IIS /ASP.NET hosted web application – Responsible for communicating with the Session Recording Database to enforce
policy query decisions and communicating with the Session Recording Player to manage access to session recordings
• Session Recording Storage Manager– Installed as a Windows service– Main Function:
– Writing Session Recording data to disk– Writes session metadata to database– Generates digital signatures– Records performance data (use perfmon)
– Characteristics– Does NOT interact with Session Recording Broker
– Can detect missing and duplicate data
– Can be restarted without data loss
© 2016 Citrix | Confidential
Session Recording Database• The Storage Manager writes session recording file metadata and
policies in the Session Recording Database– Can co-exist on SQL Server with other databases– Database schema installer will create appropriate login and user security settings– SQL Server can be clustered, mirrored and also support AlwaysOn for Sql Server
2012 – Availability features such as replication and mirroring are supported– Session metadata is approximately 1KB per recording
• Installation pre-requisites for Database:– SQL Server 2008 R2 SP2 Enterprise and Express editions– SQL Server 2012 SP1, Express, Standard, and Enterprise Editions– SQL Server 2014, Express, Standard, and Enterprise Editions.
© 2016 Citrix | Confidential
Session Recording Agent• The Session Recording Agent is the component which will be installed on the VDA
• Responsible for recording session data
• The Session Recording Driver is installed as part of the agent and is responsible for gathering session recording data
• XenApp Platinum License is required
• Supported Windows operating systems:– Microsoft Windows Server 2012 R2– Microsoft Windows Server 2012– Microsoft Windows Server 2008 R2 with Service Pack 1
• Requirements:– Microsoft Message Queuing (MSMQ), with Active Directory integration disabled, and MSMQ HTTP support
enabled– .NET Framework Version 3.5 Service Pack 1
• Data collected– Screen updates– Mouse activity– Session information
• No keyboard logging currently
© 2016 Citrix | Confidential
Session Recording Agent – Rollover
• Rollover prevents files from becoming– too large in size
– too long in duration
• Large files or files of too long duration– difficult to download
– poorer searching performance in Player
– can cause problems with “dormant” file detection and archiving
• Session Recording Agent also prevents rollover of– Too short sessions
– Too small files
Important Notes: The rollover setting does not apply to VDI desktop sessions for XenDesktop 7.8/7.9. In those cases, each recording file has a maximum size limit of 1GB and activities are not recorded after that limit is reached.
© 2016 Citrix | Confidential
Recording Files – Rollover Data Packets
Header
Completed recording file with linked rollover file
Start MD Login MD End MD Signature
Header Start MD Login MD
Next File GUID
Previous File GUID
Start reason = Rollover
End reason = Rollover
© 2016 Citrix | Confidential
Session Recording Policy Console
• The Session Recording Policy Console provides the ability to manage policies related to recording of ICA sessions
• The Session Recording Policy Console is implemented as an MMC Console Snapin
• Installation pre-requisites (verified before installation):– Supported Windows operating systems:
– Microsoft Windows Server 2012 R2– Microsoft Windows Server 2012– Microsoft Windows Server 2008 R2 with Service Pack 1
– Requirements:– .NET Framework Version 3.5 Service Pack 1 (Windows Server 2008 R2 only) or .NET Framework
Version 4.5.1 or 4.6.
© 2016 Citrix | Confidential
Session Recording Player
• The application used to replay captured session recordings files
• Only interacts with Session Recording Broker component
• Search option for metadata ,E.g. date/time, user, application, server, etc
• Installation pre-requisites (verified before installation):– Supported Windows operating systems:
– Microsoft Windows 8 / 8.1; Microsoft Windows 7 with Service Pack 1(Stated in Citrix Edocs)– For optimal results, install Session Recording Player on a workstation with:
– Screen resolution of 1024 x 768– Color depth of at least 32-bit– Memory: 1GB RAM (minimum). Additional RAM and CPU/GPU resources can improve performance
when playing graphics intensive recordings; especially when there are a lot of animations in the recordings.
© 2016 Citrix | Confidential
Data storage
• Session recording data is stored on a central file store in flat files– Multiple directories can be defined– Storage Manager will distribute files over directories
• Directories are created by year, month, date– E.g. E:\Recordings\2016\05\20
© 2016 Citrix | Confidential
Other Components and Utilities• Session Recording Authorization Console– A utility that enables Session Recording Server administrators to add users to pre-determined user roles
• Session Recording Custom Event API– API for the Session Recording software which enables ISVs to inject custom data through a third-party
application into a session recording
• Session Recording Player SDK– An SDK for use by ISVs to write third-party Session Recording Player extensions which can display
custom event data injected into the recorded session using the Session Recording Custom Event API
• icldb– A command line utility that enables you to run queries and perform maintenance of the Session Recording
Database
• iclstat– A command line utility for the Session Recording Server that enables you to view metadata information
about a session recording file
© 2016 Citrix | Confidential
Session Recording Database – icldb Utility
• Citrix Session Recording Database Utility– Perform maintenance operations and queries
• Located on Session Recording Server in
C:\Program Files\Citrix\Session Recording\Server\Bin
© 2016 Citrix | Confidential
Session Recording Database – icldb Utility
• Archive older recordings, option to move physical files
• Remove older recordings, option to remove physical files
• Restore recorded files into database
• Import or rebuild data from set of physical recordings– this will overwrite existing record if present
• Locate recording on disk by file GUID
• Query database schema version
• Remove all records
© 2015 Citrix | Confidential
Deployment Tips & Troubleshooting
© 2016 Citrix | Confidential
Deployment Tips
© 2016 Citrix | Confidential
Deployment Tips 1
• When Machine Creation Services (MCS) or Provisioning Services creates a VDA with configured master image and Microsoft Message Queuing (MSMQ) installed, the VDA has the same QMId as the MSMQ. This might cause various issues, such as:– Sessions mighty not be recorded even if the recording agreement is accepted.– The session logoff signal might not be received by the Session Recording server, which
leads to the session always in Live status.
• Tips– Use a powershell script to create a unique and persistent QMId for each VDA
• Reference:– http://
docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-whats-new/xad-xaxd76-knownissues.html
© 2016 Citrix | Confidential
Deployment Tip 2
• When recording a session with a resolution higher than or equal to 4096 x 4096, there might be fragments in the recording appearance.
• When you change your XenApp or XenDesktop license type, the change does not take effect immediately for Session Recording. Tips: Restart the VDA machine.
• You might receive an Installation failed error in the following two cases. You can ignore the message, but to avoid receiving the message, restart the machine before reinstalling the Session Recording components. – Uninstalled the Session Recording components, and then reinstalled them without restarting
the machine.– Installation failed and rollback happened, and then you tried to reinstall the Session
Recording components without restarting the machine.
© 2016 Citrix | Confidential
Deployment Tips 3
• Limitation for Session Recording to support the Pre-Launched application sessions [BUG0561109]Problem:– If the active policy tries to match the application name, the application launched in the pre-
launched session will not be matched, which results in the session not being recorded.– If the active policy records every application, when the user logs into the Windows Receiver
(at the same time the pre-launched session is established) a notification for recording will appear and the empty session and any applications that will be launched in this session later will be recorded.
– Workaround:– Publish the applications in separate Delivery Groups according to their recording policy. Do not use the
application name as the recording condition. This will ensure pre-launch sessions will be recorded. However, notifications will still appear.
• Session Recording does not support Framehawk display mode.
© 2016 Citrix | Confidential
Deployment Tips 4 – for Desktop Agent
• You cannot record the Windows 7 desktop sessions correctly when Legacy Graphics Mode is enabled by XenDesktop site policy and Disk-based Caching is enabled by Citrix Receiver for Windows policy. Those recordings show a black screen.
• Tips:– Disable Disk-based Caching by deploying with GPO to the machines on which you
installed Citrix Receiver for Windows. For more information about disabling Disk-based Caching, see http://support.citrix.com/article/CTX123169
© 2016 Citrix | Confidential
Troubleshooting
© 2016 Citrix | Confidential
Troubleshooting Methodology
Define the issue clearly
Gather information
Analyze and list possible cause
Possible Cause Verifying
Root Cause Documentation
• Issue Behavior• Expected
Behavior
• Review logs• Reproduce the
problem• Enable logging
• Rank the possible causes
• Create Action Plan
• Verify the action plan for the possible cause
• Document resolution and root cause for future reference
Redefine problem
Get further information
© 2016 Citrix | Confidential
General Troubleshooting Session Recording
• Session not recorded– Check server names and protocols in configurations– Check if the recording notification can be seen
– Yes: MSMQ or DB issue– Check Agent CDF Trace to see policy query results
– Should not record (actually should): Broker or Agent issue– Timeout: network or configuration issue
– Check Agent CDF Trace to see if metadata is obtained correctly– Correct: Broker issue– Incorrect: Agent issue
• Playback error or corruption– Check client type and version which launched the session
– Play recording file locally– Check player CDF trace
© 2016 Citrix | Confidential
General Troubleshooting Session Recording
• Check system event log first– Most of the service errors/warnings are logged
• Troubleshooting MSMQ– Open “Server Management” or “Computer Management” “Message Queuing”– Check status of private queue named citrixsmauddata– Check incoming (Server) or outgoing (Agent) packets in
http[s]://<servername>/msmq/private$/CitrixSmAudData
• Troubleshooting IIS– Ensure protocol used for connect is correct (HTTP/HTTPS)– Ensure correct certificate is used for HTTPS– Access http[s]://<servername>/SessionRecordingBroker/<name>.rem?wsdl for testing– <name> can be RecordPolicy, Player or PolicyAdministration
© 2016 Citrix | Confidential
Case Study 1Failed to record session
© 2016 Citrix | Confidential
Case Study1 – Failed to Record SessionProblem Symptom:Cannot record after migrating of the recording data and restart the Session Recording Server
Information Gathering
• Session Recording once worked fine• The problem only happened after migrate the
recording data and a server restart• Event Log – Nothing found• Reproduce the issue • End-user can get the Recording notification when
launch the application It means agent can get the record policy correctly
Possible CauseMSMQ or DB
Get further information for MSMQ
© 2016 Citrix | Confidential
Case Study1 – Failed to Record Session• Checked the MSMQ outbound queue in the
Agent Massive outbound message queue found
• Checked the MSMQ inbound queue in the Server No message queue found
• Checked the database connectivity OK
• Agent Side Message Queue status is waiting to connect
• MSDN tells me should check if the agent can telnet server side port 1801
• Verified that server side port 1801 cannot be Telnet
• The ntstat –ano shows that 1801 is listening on the loopback ip 127.0.0.1 which is different from my working environment
© 2016 Citrix | Confidential
Case Study1 – Failed to Record Session• Problem Analysis• It obviously is a problem of the MSMQ
• https://support.microsoft.com/en-us/kb/2554746
• Fixed the issue• Document the root
cause
© 2016 Citrix | Confidential
Case Study 2Records files gets deleted
© 2016 Citrix | Confidential
Case Study2 – Recording file gets deletedProblem Symptom:1. This is a newly built
environment2. The user does not get the
record notification window when launch the application
3. When the application is launched , *.icl file is generated
4. Once we close the session, the .icl file gets deleted
• Checked the agent event log first Find the error log from the source “Citrix Session Recording Agent” with the event id 0 , the error is “Exception caught while obtaining session metadata or running record policy query.”
Information Gathering
© 2016 Citrix | Confidential
Case Study2 – Failed to record the session
Another session recording log tells you , it’s using the fallback policy : No not record
Agent configuration check
Verified with customer if the SR Server is correctly configured and if they are using the SSL communication
The answer is no
So here is the misconfiguration issue
Note:If the answer is yes, we need to check if the certificate is installed correctly, and also verify if the certificate is trusted by the agent
© 2016 Citrix | Confidential
Most frequently asked question
© 2016 Citrix | Confidential
FAQ – Disk planning for Session Recording for Server VDA
• Q: How to plan the Session Recording disk?
• A: The example stated in KB CTX200869– Recording 5,000 Outlook sessions over an eight-hour work day consumes about 100GB of
storage space – One Outlook session will consume about 20 MB for an eight-hour work day
• Additionally information is:– As Session Recording is capturing screen update for the ICA Session, also depends on the
graphic content of the application, session resolution and color depth; we suggest to select a group of end-user to perform a pilot run for a week, then we can calculate the requirement of the disk based on the pilot run result.
– As a best practice for session recording, Windows Media Redirection and Flash Redirection are suggested to be enabled in the ICA Session.
© 2015 Citrix | Confidential
Resources
© 2016 Citrix | Confidential
Configuring Security Features of Session Recording
http://support.citrix.com/article/CTX200868
Building a Highly Scalable Session Recording System
http://support.citrix.com/article/CTX200869
Troubleshooting Recording Issues in Citrix SmartAuditor
http://archive.citrite.net/article/CTX114819
Troubleshooting Session Recording http://docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-monitor-article/xad-session-recording/xad-sr-trouble.html
Session Recording FP2 installation video
Server: https://youtu.be/Q47GwgsdW-IAgent: https://youtu.be/pLynm6S9gZk Player: https://youtu.be/ZTfKCuaYWVc
© 2016 Citrix | Confidential
Q & A ?
© 2016 Citrix | Confidential
Work better. Live better.Work better. Live better.