ICC12 NEW

Lecture Topic 12Lecture Topic 12

Usability Testing - part 1 Usability Testing - part 1

Dr. SENG LOKEDr. SENG LOKE

References: E-Commerce Fundamentals and Applications, by Henry Chan et al., John Wiley and Sons Pty Ltd, 2001

References: E-Commerce Fundamentals and Applications, by Henry Chan et al., John Wiley and Sons Pty Ltd, 2001

Lecture 12Lecture 12Lecture 12Lecture 12 USABILITY

• Usability can be defined as:“the extend to which end-users can successfully accomplish required goals in a specified environment subject to various comfort, effectiveness and satisfaction criteria” (Thimbleby, SIGCHI94)

• Usability is becoming an integral part of the software development lifecycle.

• Software developers aim at improving their product from the end-users viewpoint which involves identifying problems with the interface, and after some careful analysis of the problems, the interface is redesigned with the hindering aspects removed.

• The interface design is the main factor affecting usability, since it is the only part of the system with which the user interacts with.

Lecture 12Lecture 12Lecture 12Lecture 12 USABILITY TESTING

• Usability testing usually entails building prototypes of the system and testing it on users.

• Usability testing is a broad context where the interface is put under a test, with users critically analysing the system, evaluating the design layout and flow of interactions by interacting with the implemented version of the system.

• The main reasons for adopting this approach are that it can be used :– to measure user performance and satisfaction, – to identify and assess usability defects,– to provide ideas as to how a system might be improved,– to educate design teams.

Lecture 12Lecture 12Lecture 12Lecture 12USABILITY TESTING

• Many usability testing strategies have been developed. Most of these testing can be categorised into two main categories:– Empirical Testing– Inspection Testing

Lecture 12Lecture 12Lecture 12Lecture 12USABILITY TESTING

Phases of product usability testing

Phase Purposes Objects

Investigation Set usability objectives Discover opportunities

Existing solutions

Design and build Resolve conflicts Answer questions Debug Measure against

objectives

Early concepts Simulations Prototypes Partial products Training Documentation

Post release Get data forsales/marketing

Input to next release Competitive analysis

Finished product Competitive product

Lecture 12Lecture 12Lecture 12Lecture 12 Empirical Testing

• Empirical testing involves the user testing the product under review in a testing laboratory.

• Users are asked to use the system to complete a work related task that is likely to target problem areas.

• The users’ task results are recorded, often through a video camera. • The users’ concerns, which are expressed through the use of a

questionnaire, are also documented.• The results are analysed by the development team in an attempt to refine

their product.• The primary focus of empirical testing if the product is still in a prototype

stage is to use the results of the testing for refining the final version of the product.

Lecture 12Lecture 12Lecture 12Lecture 12 Empirical Testing

• If the product to be tested is in the final stage of development, this approach is useful for determining whether one selects a given product.

• There are three types of Empirical testing:–Fixed Laboratory Usability Testing–Portable Usability Testing– Remote Usability Testing

Lecture 12Lecture 12Lecture 12Lecture 12Empirical Testing : Fixed Laboratory Testing

• Fixed laboratory testing involves the usability testing of a product in a fixed laboratory. This method is often referred to as in-house testing, since the users have to come to the developer to complete the usability test.

• The user interact with the product by completing a work-related task which is created by the development team.

• To assist the development team in analysing the results of the usability test, the tests are recorded using a video camera installed in the laboratory.

Lecture 12Lecture 12Lecture 12Lecture 12

• ADVANTAGE:– All necessary equipment is already in place. This will make it easier for the development team to conduct the usability test and record all essential data.

• LIMITATION:– Can be very expensive, a fully equipped fixed laboratory costs between $50,000 to $100,000 and even more to operate and maintain. A big proportion of the amount goes towards recruiting and training of staff.– The user has to come to the developer to perform the test. (Explain what consequences this may cause ?)

Empirical Testing : Fixed Laboratory Testing


• IMPROVED Fixed Laboratory– The recent version of fixed laboratory is designed to address the problems of the earlier version of the laboratory.– The main emphasis is to design a comfortable, natural office scenario for the users and to eliminate the artificial element that is often associated with usability testing in a fixed laboratory.– This improved version fixed laboratory has a separate observation room, which is linked to the main laboratory through a one-way glass. This enables the observer to view the tester without the tester conscientious of an observer peering over their shoulder.



• IMPROVED Fixed Laboratory (ctd.)– This new layout enables testers to proceed without interruption as they would while performing the task in their own work environment.– Very often, the layout of the observation room is as similar as possible to the users real working environment.– However, this improved laboratory still does not address the problem of high costs.– Note that this fixed laboratory testing is normally used for building a new sophisticated software system or a new operating system whose usability will not be affected by internet connection, download time etc. (eg. Microsoft windows operating system releases).


Lecture 12Lecture 12Lecture 12Lecture 12Empirical Testing: Portable Usability Testing

• Portable usability testing was introduced to address the problem of high cost found in a fixed laboratory setting.

• Portable usability testing takes the testing laboratory to the users.

• The product is tested while the users interact with it in their own work environment.


• Portable Testing is best suited when the customer base is widely distributed.

• The developers will test a wide range of users from each of the customer sites.

• The recording of the task results of the user is best accomplished with the use of a video camera. However, an over the shoulder video camera can place undue stress on the users, which could jeopardise the field test results.

• The recording equipment must be setup in such a manner that it does not interfere with the usability testing process.

Empirical Testing: Portable Usability Testing


• The SETUP:• The computer output is directed to two

monitors.• The video camera should focus on the second

monitor.• The test results can then be recorded without

the video equipment getting in the user’s way.

• Another important element is to encourage the users to think out loud while they undertake the tasks. These verbal comments can prove to be valuable for analysis.

Empirical Testing: Portable Usability Testing


• Remote usability testing is a usability testing where neither the developers or the users have to travel.

• Users are observed and tested through the aid of computer networks and modem connections in a time efficient and cost effective manner.

• Can you think of any limitations of this type of usability testing?

Empirical Testing: Remote Usability Testing


• In general, the main advantage of empirical testing is that it is able to directly monitor the performance reactions of the user to the particular interface and one does not have to rely on a usability expert trying to second guess what a user’s reaction might be.

• The disadvantage is that it is only useful for testing an implemented product, and therefore the ability to correct the problem immediately is greatly reduced.

Empirical Testing: Summary


• The test layout in an empirical testing consists of:– the test subject (the user)– the computer with the software being tested– the test monitor (the tester)

• The user must perform a set of tasks and answer a set of questionnaire prepared earlier by the tester. The tasks must be designed such that:

– they be work related to stimulate the manner in which the software would actually be used in the work environment– they cover most of the significant functions of the package so that all significant and frequently used features of the interface are tested– the time of the test session clearly reflects work periods and is not too short (too artificial) or too long (may cause tiredness).

Empirical Testing: Test Procedure


• Usability Inspection testing is often referred to as “discount usability engineering methods”.

• Inspection testing is performed by usability inspectors who will inspect the design layout of an interface and comment based on the result of their test and experiences.

• Results from inspection methods influence the design layout of an interface since it captures potential problems at an early stage in the development life cycle.

• Most commonly used Inspection testing procedures are:– Cognitive Walkthrough– Heuristic Evaluation

Inspection Testing


• People who are evaluating the interface (called the inspectors) are asked to answer a set of derived questions for each step of the user task scenario. These questions are based on the relationships between user’s goals and the actions available for achieving these.

• This inspection testing can be done during the design stage especially when the dynamic aspects of the web site/application is determined.

• The user task scenario can be drawn from the Web Navigation Analysis Model (lecture 10) by defining a scenario for each branch of the hierarchical structure in the Site Navigation Net.

Inspection Testing: Cognitive Walkthrough

Lecture 12Lecture 12Lecture 12Lecture 12 Inspection Testing: Cognitive Walkthrough

Enter Booking Details

Prc: sesion activePoc:

Di:Do:

Booking Details Confirm


Di:Do:

WB

2

WB1

Booking Receipt


Di:Do:

WB

3

MT3

Search Lot


Di:Do:

Transfer Details Confirm


Di:Do:

Enter Transfer Details


Di: Lot no.Do:

Lot Result List


Di: Warehouse IDDo: Lot.no.

Transfer Receipt


Di:Do:

GT

1G

T5

GT

4G

T3

GT

2

Search Lot


Di:Do:

Delivery Details Confirm


Di:Do:

Enter Delivery Details


Di: Lot no.Do:

Lot Result List


Di: Warehouse IDDo: Lot no.

Delivery Receipt


Di:Do:

GD

5G

D4

GD

3G

D2

MT3

SNN2

GT

1.2

GD1

GD

1.2

MC3

Booking details

Booking details

• For example, we can perform a cognitive walkthrough test on the ‘Warehouse Booking’ activity as shown in the left most branch of the SNN.


Cognitive Walkthrough Sheet exampleTask description: Warehouse Booking WB1(-MT3) -

WB2-WB3

Tester name: ………..

Problems identified:

1. Failure to identify a goal

in the booking detail form, there is a ‘+’ sign which indicates a facility to add a new transporter if it does not exist in the list. User might fail to identify this goal.

2. Failure to identify the correct flow

after clicking ‘+’ sign and transferred to the ‘Add New Transporter’ form (MT3), the user might fail to identify the correct action to come back to the enter booking form.


Cognitive Walkthrough Sheet example (ctd.)

3. Correct action does not match goal

in the booking confirmation form, the user might want to edit the booking details (goal=edit). The correct action here is to click on the ‘Cancel’ button, which does not match to the user’s goal.


• Cognitive walkthrough requires an in depth, step-by-step evaluation of the interface.

• However, it often requires a substantial investment in training.

• See also: http://www.infodesign.com.au/usabilityresources/evaluation/conductingwalkthroughs.asp

Inspection Testing: Cognitive Walkthrough

http://www.infodesign.com.au/usabilityresources/evaluation/conductingwalkthroughs.asp


• In heuristic evaluation, the interface is analysed based on a set of known heuristic rules.

• In general, those rules can be summarised into the following nine usability heuristics:

1. Visibility of system status2. Match between system and real world3. User control and freedom4. Consistency and standards5. Error prevention6. Recognition rather than recall7. Flexibility and efficiency of use8. Aesthetic and minimal design9. Helping users recognise, diagnose and recover from

errors

Inspection Testing: Heuristic Evaluation

look & feel orfunctionality?

subjective & objective?


• Heuristic evaluation can be improved by a group of inspectors working independently on the same system interface. The inspectors do their initial evaluations individually and then collaborate afterwards.

• BENEFITS of Heuristic evaluation:– intuitive and easy to motivate people to do it– does not require advanced planning–can be used early in the development process



Example of Heuristic issues:• Visibility of system status

– Does the system let one know where one is (site title, page title, window title)?– Does the system acknowledge that an action requested has been carried out?

• Consistency and standard with respect to the following:– Message display methods (prompts, warning, helps)– Colour use (entry form, menu/submenu, foreground/background)– Keys definition– Data entry method– Navigation, dialogue, window display method– Search facility



Example of Heuristic issues (ctd.):

• Match between system and real world– Menu hierarchy that is consistent with the real world– Terminology used is the same as in real life of that domain

• Flexibility and efficiency of use:– Browser compatibility– Time taken for task completion– Unproductive period– Download time



Example of Heuristic issues (ctd.):• Helping users recognise, diagnose and recover

from errors:– Does the system explain why an action cannot be performed?– Does the system give prompts on how to proceed?– If an action is not allowed, does thet system give the reason?



• Heuristic evaluation is one of the most widely used method at assessing the usability of an interface at the design stage.

• Heuristic evaluation has been described as the usability assessment method which identifies the most serious usability problems with the least amount of effort.

• However, it can only be used to find usability problems. It cannot assess the performance of the user while they interact with the system like empirical user testing can.

• Another problem is that heuristic evaluation is biased towards more experienced inspectors, since they seem to predict more problems than people without usability expertise.


Further reading on Heuristic Evaluation

• Case studies:– http://www.bls.gov/ore/htm_papers/st960160.htm– http://www.infodesign.com.au/usabilityresources/evaluation/webevaluation.asp

• Background: – http://www.useit.com/papers/heuristic/

http://www.bls.gov/ore/htm_papers/st960160.htm

http://www.infodesign.com.au/usabilityresources/evaluation/webevaluation.asp

Next Lecture ...Next Lecture ...Next Lecture ...Next Lecture ...

Next Lecture: Usability Testing - part 2

ICC12 NEW

Documents

Transcript of ICC12 NEW