'Operational Testing – Walking A Mile In The Users Boots' by Gitte Ottosen
-
Upload
eurostar-conference -
Category
Technology
-
view
172 -
download
0
description
Transcript of 'Operational Testing – Walking A Mile In The Users Boots' by Gitte Ottosen
Operational Testing:
Walking a Mile in the User's Boots
© Sogeti
Systematic
• Established in 1985• Approx. 500 employees;
58% hold a MSc og PhD in software engineering
• Microsoft, ESRI and SAP partner
• 97% of our customers would recommend Systematic to other customers
• Customers in 35 countries
• CMMI Level 5 certified
• For more info visit:
www.systematic.com
Aarhus – HeadquartersSoftware and systems development
TampereProducts and services
LondonProducts and services
WashingtonProducts and services
Page 2
© Sogeti
Systematic’s Business Areas
A leading international company in delivering reliable and simplesolutions to people who make critical decisions every day
Vision:
Mission: Simplifying Critical Decision Making
Inte
ropera
bili
tyC
om
mand a
nd C
ontr
ol
Sit
uati
on A
ware
ness
Ele
ctro
nic
Warf
are
Support
Clin
ical In
form
ati
on S
yst
em
s
Managem
ent
Info
rmati
on S
yst
em
sB
oard
er
Contr
ol
Vari
ous
Finance
Transp
ort
Agri
cult
ure
Govern
ment
Defe
nce
Healt
hC
are
Inte
lligence
and
Nati
onal Secu
rity
Inte
gra
tion S
erv
ices
© Sogeti
The Sitaware Product Suite
© Sogeti
OVERVIEW
SitaWare Headquarters
Usable at multiple organisational levels
Improving situational awareness
Speeding up and supporting the Military Decision Making Process
Reducing planning preparation time
Swift andimproved ability to adjust on-going operations
© Sogeti
Product vs Project
SitaWare COTS Product
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
SitaWare Product Add-in
Customer specific
extension
Customer specific
extension
Customer specific
extension
Customer specific
extension
© Sogeti
The Background – How we Work
GUI
Communication Layer
Database
Business logic
Featu
re 1
Featu
re 2
The System
© Sogeti
How we Test
© Sogeti
Test in the Whole Lifecycle
Feature Kick-off
Feature A
Feature B
Code cut-off
Feature C
System test
Testing Stories
Test ”Qualify Feature”
Bug hunts
UnitIntegrationSystem
Sprint 3Sprint 1 Sprint 2
Increment
“Sprint test”
System Integration test
© Sogeti
The Challenges
• The domain• The complexity • The workflows• The environment - Interoperability
Sensors
Communications
WiFi
HF
UHFVHF
Works with many different comunication products
GyroGPS
Laser RangeFinder
Integrates to a variety of sensor products
SitaWare Headquarters
© Sogeti
Operational Test
© Sogeti
Moving the Focus
Test that the concept holds
Test the SYSTEM as a whole
Attack the system as a user would
Transfer fresh domain knowledge to the project
© Sogeti
An Iterative Process
Execute opr. test
Defining the
concept
Scenario and environment
ready
Execute test
Learning
Refining the
concept
Defining TSP and
environment
Project Product
Time
© Sogeti
Defining the Concept – Take One
• A project context• The operational test specification• The test environment• Internal test execution
© Sogeti
The Project Environment
© Sogeti
The Result and the Challenges
New bugs foundSystem used in a whole new way/contextDomain knowledge was transferred to the
developer/tester÷ Used as regression test – or misused÷ Stopped reading – executing by memory÷ Went faster and faster÷ Running the same scenario every time
© Sogeti
Defining the Concept – Take Two
• The real concept - Product• Defining the operational basis• Defining the organization
OPORDExercise Lightning
Fist1. Situation2. Mission3. Execution4. …5. …
© Sogeti
The First Execution – Getting Ready
Establishing the
environment
Defining the scenario
Getting the right people
Getting the training
© Sogeti
The Execution
© Sogeti
The Result of First Execution
• The concept was validated• New defects were identified• Knowledge was transferred into the organization• Testers were trained (Internal as well as external)
© Sogeti
What did we learn?
• Need for structure• Needs a lot of organisation• Clear division of responsibilities• Communication - communication – communication
© Sogeti
Following Executions
• Getting more people with the right profile• A better environment – a ”real organization”• Inviting UX to participate as observers - get to know the
users
• Inviting test to participate as assistants - get to know the domain
• What else:
– Usability test with the end users– Conceptual discussions
© Sogeti
Experiences
– Best for systems with larger work flows– Needs a lot of preparation– Need a lot of resources
• People• Environment• Test data setup• Scenarios• ….. And time
© Sogeti
The Way Ahead
• Extending the scenarios – Special operations– Including more applications from the product suite
• Finding new profiles – Fit the changing operational profiles– Other nationalities?
• More active involvement of operational testers in the planning phase
• The focus – from concept validation to bug hunting
• The schedule– Split up into shorter scenarios – max 2-4 hours– Feedback more often
© Sogeti
So What’s in it for us?
• Value on many levels– Verification of the concept
– Finding new defects– More operational knowledge gathered
– Testers get a better domain knowledge– Operational testers used for sparring as well, input for
the roadmap
© Sogeti