Personal Experience Computing
description
Transcript of Personal Experience Computing
![Page 1: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/1.jpg)
1
Personal Experience Computing
朱浩華Intelligent Space Group
(許永真 , 黃寶儀 , 洪一平 , 傅立成 )CSIE, EE, INM (Graduate Institute of Networking & Multimedi
a)National Taiwan University
![Page 2: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/2.jpg)
2
Research Interests
Pervasive (Ubiquitous) & Mobile Computing– User Interfaces– Applications– Middleware– Systems– Networking– Security
![Page 3: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/3.jpg)
3
Personal Experience Computing• Imagine a wearable camera can record your entir
e life ….– Autobiography
– Memory augmentation (google search your past)
– Relive past memory (memory triggers)
– Sharing personal experience
• Personal experience computing is about computing support for– Recording, archiving, retrieving, searching, analyzi
ng (annotating), editing, sharing, etc., of personal experiences.
• Largest database ever• 4G+ Killer Applications
![Page 4: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/4.jpg)
4
![Page 5: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/5.jpg)
5
Personal Experience computing is a hot topic!
• Pervasive 2004– Workshop on Memory and Sharing of Experiences
• ACM Multimedia 2004– Keynote speech (Gordon Bell)– Workshop on Continuous Archival & Retrieval of Personal Experi
ences
• Microsoft Research: MyLifeBits• DARPA LifeLog Initiative • UK Grand Challenges in Computer Science: #3 Memorie
s for Life • Memex Device (V. Bush 60’s)
![Page 6: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/6.jpg)
6
Design & Evaluation of mProducer: a Mobile Authoring Tool for Personal E
xperience Computing
Chao-Ming Teng, Chon-In Wu, Yi-Chao Chen, Hao-hua Chu & Jane Yung-jen
Hsu
![Page 7: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/7.jpg)
7
Gadget & Lifestyle Trend
• Proliferation of camera-equipped phones, digital cameras, and camcorders
• Implication: Average consumers want to be content producers of their own personal experiences– Everyone can be a news reporter.– Record where they go, what they do, what they see,
and hear– Share personal experience (edit before share)– Anywhere, anytime
![Page 8: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/8.jpg)
8
Mobile Contents = Personal Experiences
Type Producers ToolsPC contents Mass media
contentsProfessional
content providers
PCs & Professional
content creation tools
Mobile Contents
Personal experience
contents
Average consumers
Cell phones & mProducer
Fundamental change in the type of contents!
![Page 9: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/9.jpg)
9
Why edit from mobile devices?
• PC Alternative:– Capture contents on mobile devices– Transfer them to PC– Use PC-based tools to edit & share them
• Reasons:– Reduce the amount of time between capturing &
sharing of time-sensitive content• Share anytime, anywhere
– Use simple, intuitive user interfaces– Record important events as keepsakes
![Page 10: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/10.jpg)
10
Mobile Editing vs. Desktop Editing
• Desktop Authoring Tools:– Professional, skillful content creators– Focused user attention– Resource-abundant desktop environment
• Mobile Authoring Tools:– Casual, unskillful consumers– Limited user attention– Resource-poor mobile environment
![Page 11: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/11.jpg)
11
Mobile Challenges
• Limited mobile storage– Toshiba T08 Mobile Phone with 8 MB Storage– Only 3 minutes of video: 5 FPS & 240x320
• Limited mobile computing resources– Image/video processing techniques are computational
intensive
• Specialized user interfaces– Small screen, inconvenient input methods, limited use
r attention– Simplicity, ease-of-use, good learnability, reasonable
quality of editing contents
![Page 12: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/12.jpg)
12
mProducer’s Solution
• Specialized UI– Location-based content management
• Location-based mental model
– Keyframe-based editing• Efficient & good enough quality
• Limited mobile storage– Storage constrained uploading
• Do we need to download uploaded frames back during editing? • No need to edit frame-by-frame (keyframe editing good enough)
• Limited mobile computing power– Sensor-assisted automated editing
• Automated editing is computationally intensive
![Page 13: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/13.jpg)
13
The rest of the talk is details
• Design & Architecture
• Storage constrained uploading
• Sensor-assisted automated editing (using tilt sensor)
• Specialized User Interface
• Related Work
• Conclusion & Future Work
![Page 14: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/14.jpg)
14
Overall Design
• Camera Phone (mobile storage) ↔ Wireless Network ↔ Storage Server – Mobile storage uploading into server storage
• Capturing phase & editing phase– Typical usage: repeated patterns of capturing
& editing
• Sharing phase (future work)
![Page 15: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/15.jpg)
15
(1) Captured Raw Data
Input
(2) Shot Boundary
Detection (SBD)
(3) Shaking Artifact Detection & Removal
Using Tilt Sensor
(4) Motion-JPEG
Encoding
(5) Keyframe Selection Algorithm (KSA)
(6) Storage Constrained
Uploading (SCU)
Buffer Space
Local Storage
Remote Storage
(1) Map-Based Content
Management
(3) Keyframe-based Storyboard Editing
Capturing Phase
Editing Phase
(2) Browse & select a clip
![Page 16: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/16.jpg)
16
Limited Mobile Storage(Storage Constrained Uploading)
• Naïve approach:– When the mobile storage is full, offload all
new contents to server
• Problems with naïve approach:– Download previously uploaded contents
during editing phase– Transfer contents later be cut by a user– Slow content transfer over wireless network
![Page 17: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/17.jpg)
17
Observation (Insight)
• User studies showed that editing at keyframe granularity is preferred over frame-by-frame granularity on mobile devices.– Consider user effort, attention, computing power, etc.
• Keep keyframes in mobile storage• Offload non-keyframes• Avoid downloading frames at editing phase
![Page 18: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/18.jpg)
18
Storage Constrained Uploading
• Prioritize frames– 2 levels: Keyframe, non-keyframes– Multiple levels: keyframes, I, P, B (MPEG)
• Adjust editing granularity– Types of frames needed during editing phase
• One challenge: multiple clips– Keep all clips at the same editing granularity– Round-robin offloading across clips
![Page 19: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/19.jpg)
19
Limited mobile computing power: (Sensor-Assisted Automated Editing)
• Existing automated tools use image processing to extract meta-data information– Shaking detection (forget to hit stop button)– Lighting level detection– Group shot (scene) similarity– Computational expensive
![Page 20: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/20.jpg)
20
Observation (Insight)
• Sensors can achieve the same result as image processing– But use much less computation!
• Tilt sensor -> camera shaking
• Ideal for mobile devices
![Page 21: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/21.jpg)
21
Tilt Sensor
![Page 22: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/22.jpg)
22
Experiment to Identify Camera Shaking Pattern
![Page 23: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/23.jpg)
23
User Interface Design
• Tradeoff between– Simplicity (ease-of-use, short learning curve, r
educed user effort)– Quality of edited production
• Two parts in UI:– Location-based content management– Keyframe-based storyboard editing
![Page 24: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/24.jpg)
24
StoryboardMaterial PoolLocation-based Content Management
![Page 25: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/25.jpg)
25
Location-based Content Management
• Two ways people mentally group clips:– Recording time– Recording location
• Observation (Insight): – User studies showed that grouping by location is
preferred.– Location information is more visual– Time information is more abstract
• Use GPS to annotate clips with location meta-data.
![Page 26: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/26.jpg)
26
Keyframe-based Editing
• Previous work uses keyframes to expedite video browsing (video summary).
• Apply keyframe understanding to keyframe editing:– Concern #1: editing is more demanding than underst
anding– Concern #2: reduction of editing quality
• Understand the tradeoff between user efforts & editing quality.
![Page 27: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/27.jpg)
27
Tradeoff between User Efforts & Editing Quality
• User studies to understand this tradeoff– Reduction in user-perceived
quality & produced contents acceptable to users
– Reduction in user efforts or improvement in task completion time
– Effectiveness when combining with slideshow player or storyboard player
Keyframe editing
![Page 28: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/28.jpg)
28
User Studies #1 (three editing interfaces)
• (UI-A): Frame-by-frame editing with a video player (the scaled-down version of conventional desktop editing interface)
• (UI-B): Keyframe-only editing with slideshow player• (UI-C): Keyframe-only editing with storyboard player
![Page 29: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/29.jpg)
29
User Studies #1 Setup
• Participants: – 8 males & 3 females– 20 ~ 41 years (mean 24)– 3 males have experience
with PDA– 5 have experience using
PC-based video editing tool
• Procedures:– Brief them on software with
demo
– Each participant captures 6 minutes of video with 3 2-minutes clips on campus
– Edit 3 clips using each of 3 interfaces
– Task completion time is measured
– At the end, each participant fills questionnaire scoring 3 interfaces.
– Interview participants
![Page 30: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/30.jpg)
30
User Studies #1 Result:Task Completion Time
Frame-by-frame Keyframe +slideshow
Keyframe + storyboard
![Page 31: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/31.jpg)
31
User Studies #1 ResultInterview on task completion time
• Storyboard UI helps them to see several keyframes at the same time, so they can quickly identify which frames or shots they did not like and remove them.
• Problem with frame-by-frame editing was that it required uninterrupted, focused attention on the screen. Mobile environment can be distracting and make it difficult to maintain continuous attention for a long period of time.– Friends calling, people walking by, surrounding noises, etc.
![Page 32: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/32.jpg)
32
User Studies #1 Result:Subjective Satisfaction on 3 Editing Interfaces
Questions (Rank three editing UIs)
1 Perceived quality of editing
2 Ease-of-use
3 Ease of learning
4 Overall editing experience
![Page 33: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/33.jpg)
33
User Studies #1 ResultInterview on subjective satisfaction
• Keyframe-only storyboard UI produced the best perceived editing qualilty– Even better than frame-by-frame!– Why? Casual users not willing to spend time finding mark-in &
mark-out boundary points for unwanted contents. Our shot boundary detection algorithm finds better boundary points.
• Advantages of keyframe-only storyboard UI– Users can quickly move among shots, very useful during editing.– Users can quickly delete unwanted shots with a single click.
• All participants found keyframe-only + storyboard best• 7/11 participants found keyframe-only + slideshow better
than frame-by-frame
![Page 34: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/34.jpg)
34
User Studies #2 Setup(overall user experience)
• Location-based content management & keyframe-based storyboard editing UI
• Participants: – 5 males & 2 females– 21 ~ 33 years (mean 24)– 3 have experience with PD
A– 3 have experience PC vide
o editing tool
• Procedures:– Brief them on software with
demo– Each participant was asked
to shoot any type of footage on campus about 10 minutes, 2+ clips
– Each participant was asked to edit two clips chosen randomly, and to think aloud.
![Page 35: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/35.jpg)
35
User Studies #2 Result (overall user experience)
• Participants feedbacks were very possible.– “A pretty cool tool to use”– “The keyframe-only storyboard is very helpful for me t
o delete contents that I do not like. Editing tools on desktop PCs should incorporate this feature too!”
– “Map based content management is very informative for choosing which clip to edit”
– Slideshow interface is better for understanding, whereas storyboard is better for editing
– Provide location tracking indoor
![Page 36: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/36.jpg)
36
Related Work
• Lots of related work on automated indexing of media:– Annotating 5W: Who, Where, When, What, How– ContextCam (Gatech), Garage Cinema (Berkeley), Context-Aware Phot
ography (Viktoria Institute), Audio-Based Memory Aid (MIT)
• Keyframe-based editing on PC:– speed up editing of home videos by grouping similar keyframes in piles
(based on color similarity)– Require a large screen for displaying piles
• Collaborative editing tools by Lara to download and edit at different fidelity– Focus on replica inconsistency, did not address limited storage & UI iss
ues
• To our knowledge, this was the first ever mobile video editing tool.
![Page 37: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/37.jpg)
37
Conclusion
• Future mobile contents = personal experiences• Future content production:
– Average consumers use mobile device to capture and edit personal experience contents
– Everyone becomes a news reporter!
• Casual interface for editing of personal experiences on the fly
![Page 38: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/38.jpg)
38
Future Work
• Sharing of personal experience– Dissemination with privacy & security protection
• Explore other types of sensors (orientation, emotion, voice, accelerometer, etc.) to automate – Editing personal experiences– Sharing personal experiences– Recalling personal experiences
• Do you have any cool ideas or applications on personal experience computing? – Let me know!
![Page 39: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/39.jpg)
39
References• Homepage: http://www.csie.ntu.edu.tw/~hchu• Chao-Ming Teng, Hao-hua Chu, Chon-In Wu, “mProducer: Authoring Multim
edia Personal Experiences on Mobile Phones”, IEEE International Conference on Multimedia and Expo (ICME’2004), Taipei, Taiwan, June 2004.
• Chao-ming Teng, Chon-in Wu, Yi-chao Chen, Hao-hua Chu, Yung-jen Hsu, “Design and Evaluation of mProducer: a Mobile Authoring Tool for Personal Experience Computing”, ACM Mobile and Ubiquitous Multimedia (MUM) 2004, College Park, Maryland, October, 2004.
• Chon-in Wu, Chao-ming (James) Teng, Yi-chao Chen, Tung-yun Lin, Hao-hua Chu, Jane Yun-jen Hsu, “Point-of-Capture Archiving and Editing of Personal Experiences from a Mobile Device”, Submitted to ACM Journal of Personal and Ubiquitous Computing (PUC): Special Issue on Memory and Sharing of Experiences, October, 2004.
![Page 40: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/40.jpg)
40
How to get meta-data (context) out of media (content)?
(TRADITIONAL) Content Analysisvs.
(NEW) Context-aware Media Capturing
Discussion Panel,
ACM Multimedia 2004
![Page 41: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/41.jpg)
41
Semantic Gap in Multimedia
• Gap between low-level signal analysis & high-level semantic description– High level language query on media content:
• “Find the last picture of my car before my wife drove it over the ditch.”
– Low level content descriptions
• Consider the number of digital cameras, camera phones, and camcorders– Little non-text media contents searchable on the
Internet
![Page 42: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/42.jpg)
42
Content Analysis Approach
• Analysis of low-level features (color, shape, etc.)• Inference of high-level multimedia meta-data
(cat, tree, etc.)• Different algorithms, different media types,
different application domains• Have the expected results and contribution to
multimedia information retrieval and media access been achieved?
![Page 43: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/43.jpg)
43
Context-aware Media Capturing
• Get meta-data (context) at point-of-capture– Location, Time, Calendar, (use sensors)
• Face recognition– If the system knows that you are at home, …
• Building recognition– If the system knows that you are on NCKU Campus
…
• Activity recognition– If the system knows that you are in a bathroom …
![Page 44: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/44.jpg)
44
The Promised Land? A hybrid approach of
Context-aware Media Capturing (Point-of-Capture) +
Content Analysis (Post)? +
Before Content Creation (Pre)?
![Page 45: Personal Experience Computing](https://reader035.fdocument.pub/reader035/viewer/2022062809/5681571b550346895dc4baa1/html5/thumbnails/45.jpg)
45
Interesting Topics
• Media-on-demand services– TiVo (digital video recorders)– Watch what I want to watch whenever I want to watch them (d
eath of mass market media)
• Life Log (capturing “everything personal”)– Wearable computers– Personal experiences (capturing, editing, sharing, searching, e
tc., for everyday people)
• Content analysis (CIA)• Digital Storytelling • Massive multiplayer games• Systems (service composition)