Usability of Keyword-driven Schema-agnostic Search - A Comparative Study of Keyword Search, Faceted...
-
Upload
thanh-tran -
Category
Technology
-
view
335 -
download
0
Transcript of Usability of Keyword-driven Schema-agnostic Search - A Comparative Study of Keyword Search, Faceted...
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
1
Usability of Schema-agnostic SearchESWC 2010 paper presentation
Thanh Tran1, Tobias Mathäß2, Peter Haase2
1 AIFB (KIT)2 fluid Operations GmbH
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Outline
Motivation - Complex Information Needs Overview - Schema-agnostic Search Frontend Conceptual Study of Schema-agnostic Search
Approaches from Process-oriented Point of View What are the steps? What are the actions required by the user? What is the state of the art?
Experimental Study of Schema-agnostic Search Systems using Task-based Experiments
2
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Motivating Scenario: Complex information needs on the Web of Data
Growing number of triples and links Web of linked data Complex information needs (entity search, fact
search, and particularly relation search)
Tran Thanh: Schema-agnostic Search3
Mary is a novice computer science student at KIT. She is eager to learn more about this vast research field and decided to find information about research work of researchers at AIFB.
How to express complex information needs? Formal structured query languages require knowledge
about the underlying data schemas Web data is dynamically evolving
z) rk(y,researchWo AIFB) name(x, x), worksAt(y )researcher type(y,.,).( yxz
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Contributions
We identified a particular class of search paradigms that we refer to as schema-agnostic search frontends
From a process-oriented viewpoint, we conducted a theoretical study of four widely used approaches representing the class of keyword-driven schema-agnostic search
We implemented these approaches under one common framework and carried out an experimental evaluation to assess efficiency, effectiveness and usability
Tran Thanh: Schema-agnostic Search4
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Schema-agnostic Search Frontend
Schema-agnostic frontend the user does not know the schema
Schema-agnostic search approaches Application-specific form-based interfaces Generic form-based interfaces: faceted search & browsing Graphical query interfaces Natural language interfaces IR-style keyword search DB-style keyword search
– Computing queries vs. computing complex results – Auto-completion: query vs. result completion
5
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Keyword-driven Schema-agnostic Search Process
Search paradigm from the process point of view A search process that starts with the use of
keywords followed by result inspection, refinement and further browsing, and ends with the retrieval of final results.1. IR-style keyword search + resource-based browsing2. IR-style keyword search + faceted-based browsing3. DB-style keyword search + result completion4. DB-style keyword search + query completion
Tran Thanh: Schema-agnostic Search6
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Keyword Search
QueryCompletion
ResultCompletion
IR-style Keyword Search + Resource-based Browsing
Tran Thanh: Schema-agnostic Search7
Keyword Search Result Set Inspection
ResourceInspection
Resource-basedNavigation
Facet-basedResult Set Inspection
Facet-basedExpansion & Refinement
Query SetInspection
Further Inspection, Browsing and Retrieval
Result Tuple Set Inspection
Search Initial Inspection FacetedSearch
“Information about research work of researchers at AIFB”
Querytype(y,researcher), worksAt(y,x), name(x,AIFB), publication(y,z)
Keywordsk1=publication, k2=researcher, K3=aifb
Result SetElena SimperlAndreas HarthRudi StuderThanh Tran
Result SetAIFB’s Publication 1AIFB’s Publication 2
FacetsPublicationNameAddressProject
Result SetTran’s Publication 3Tran’s Publication 4
Result Tuple SetPublication 1 AIFB HarthPublication 2 AIFB StuderPublication 3 AIFB TranPublication 4 AIFB Tran
ResourceTran
k2,k3
k1,k2,k3
Further Inspection, Browsing and RetrievalSearch Initial Inspection
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Keyword Search
QueryCompletion
ResultCompletion
IR-style Keyword Search + Faceted Browsing
Tran Thanh: Schema-agnostic Search8
Keyword Search Result Set Inspection
ResourceInspection
Resource-basedNavigation
Facet-basedResult Set Inspection
Facet-basedExpansion & Refinement
Query SetInspection
Further Inspection, Browsing and Retrieval
Result Tuple Set Inspection
Search Initial Inspection FacetedSearch
“Information about research work of researchers at AIFB”
Querytype(y,researcher), worksAt(y,x), name(x,AIFB), publication(y,z)
Keywordsk1=publication, k2=researcher, K3=aifb
Result SetElena SimperlAndreas HarthRudi StuderThanh Tran
Result SetAIFB’s Publication 1AIFB’s Publication 2
FacetsPublicationNameAddressProject
Result SetTran’s Publication 3Tran’s Publication 4
Result Tuple SetPublication 1 AIFB HarthPublication 2 AIFB StuderPublication 3 AIFB TranPublication 4 AIFB Tran
ResourceTran
k2,k3
k1,k2,k3
Further Inspection, Browsing and RetrievalSearch Initial Inspection
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Keyword Search
QueryCompletion
ResultCompletion
DB-style Keyword Search + Result Completion
Tran Thanh: Schema-agnostic Search9
Keyword Search Result Set Inspection
ResourceInspection
Resource-basedNavigation
Facet-basedResult Set Inspection
Facet-basedExpansion & Refinement
Query SetInspection
Further Inspection, Browsing and Retrieval
Result Tuple Set Inspection
Search Initial Inspection FacetedSearch
“Information about research work of researchers at AIFB”
Querytype(y,researcher), worksAt(y,x), name(x,AIFB), publication(y,z)
Keywordsk1=publication, k2=researcher, K3=aifb
Result SetElena SimperlAndreas HarthRudi StuderThanh Tran
Result SetAIFB’s Publication 1AIFB’s Publication 2
FacetsPublicationNameAddressProject
Result SetTran’s Publication 3Tran’s Publication 4
Result Tuple SetPublication 1 AIFB HarthPublication 2 AIFB StuderPublication 3 AIFB TranPublication 4 AIFB Tran
ResourceTran
k2,k3
k1,k2,k3
Further Inspection, Browsing and RetrievalSearch Initial Inspection
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Keyword Search
QueryCompletion
ResultCompletion
IR-style Keyword Search + Faceted Browsing
Tran Thanh: Schema-agnostic Search10
Keyword Search Result Set Inspection
ResourceInspection
Resource-basedNavigation
Facet-basedResult Set Inspection
Facet-basedExpansion & Refinement
Query SetInspection
Further Inspection, Browsing and Retrieval
Result Tuple Set Inspection
Search Initial Inspection FacetedSearch
“Information about research work of researchers at AIFB”
Querytype(y,researcher), worksAt(y,x), name(x,AIFB), publication(y,z)
Keywordsk1=publication, k2=researcher, K3=aifb
Result SetElena SimperlAndreas HarthRudi StuderThanh Tran
Result SetAIFB’s Publication 1AIFB’s Publication 2
FacetsPublicationNameAddressProject
Result SetTran’s Publication 3Tran’s Publication 4
Result Tuple SetPublication 1 AIFB HarthPublication 2 AIFB StuderPublication 3 AIFB TranPublication 4 AIFB Tran
ResourceTran
k2,k3
k1,k2,k3
Further Inspection, Browsing and RetrievalSearch Initial Inspection
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Objectives
Given complex needs, we aimed to assess whether users are able to address them
(effectiveness) how much effort they have to invest to address
them (efficiency) and how is their experience as a whole (to
obtain initial usability results)
Tran Thanh: Schema-agnostic Search11
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Information Workbench
Vorlesung Knowledge Discovery - Institut AIFBFacets
Term Completions
Keywords
Query & Resultcompeltions
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Information Workbench
13
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Tasks
Nine tasks of varying complexity All tasks were solvable Tasks aborted when not solvable with reasonable
effort / when more than 3 minutes are needed
Tran Thanh: Schema-agnostic Search14
Task Description Search Type
Task 1 Find The Beatles (the band) Entity Search
Task 4 Find the occupation of Barack Obama Fact Search
Task 9 Find all people whose birthplace is Albany, together with their deathplace
Relation Search
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Overall Setting
19 participants DBpedia together with YAGO as a domain-
independent dataset Top-10 queries for query completion and top-25
results for result completion Standard commodity PCs, Intel Pentium Core2Duo
with 2 x 2Ghz CPU and 3 GB RAM All actions taken by the participants and the
system responses were recorded as screencasts Study results based on these screencasts and
questionnaires Tran Thanh: Schema-agnostic Search15
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Effectiveness
Tran Thanh: Schema-agnostic Search16
Entity-Queries Fact-Queries Tuple-Queries -
10.00
20.00
30.00
40.00
50.00
Percentage of Unsolved Tasks
keyword searchfaceted searchresult completionquery translation
No difference for entity search Slight difference for fact search Largest difference for relation search
Keyword search requires large browsing effort to go through all entities DB-style query completion best (5 % failure) DB-style result completion requires inspecting large number of results
& asking queries with very specific keywords
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Efficiency
Tran Thanh: Schema-agnostic Search17
Keywordsearch Faceted Search Result-Completion Query-Completion -
20.00
40.00
60.00
80.00
100.00
120.00
Average Time Needed (in seconds)Entity-QueriesFact-QueriesTuple-QueriesOverall
Time needed for keyword search and faceted search increases rapidly with the complexity of the information needs.
No significant increase for completion-based approaches Slight differences: result completions faster than query
completions in some cases
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Experimental Study: Usablity
Tran Thanh: Schema-agnostic Search18
0
4
8
12
Most popular
very easy
pretty easy
fair pretty hard
very hard
0123456789
10
Ease of Query Completion
very easy
pretty easy
fair pretty hard
very hard
0123456789
10
Ease of Result Completion
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Conclusions
Schema-agnostic search is essential for searching the large and evolving Web of data!
Keyword-driven schema-agnostic search is natural evolution of Web search, from simple document search to entity search to complex relation search that address complex information needs Simple IR-style keyword search is sufficient for simple
information needs (entity and fact search) Faceted search not superior when need is specific Query- and result completion have advantages for
complex needs (relation search) Query completion more effective and usable while
result completion slightly more efficient
Tran Thanh: Schema-agnostic Search19
KIT – die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)
Thanks!
Tran Duc [email protected]
http://sites.google.com/site/kimducthanh/