eric c. larson | eclarson.com
flipping the clinic
Assistant Professor Computer Science and Engineering
in home health monitoring using mobile phones
MiPhone 9 Now with iColon.
> slide to unlock
mobileComp Sci. & Engr.
databases & data science
algorithms arch
AI & robotics
OS
networking
languages
symbolic computing
software
mhealth
what is mhealth?
mHealth (em-‘helth )an abbreviation for mobile health, a term used for the practice of medicine and public health supported by mobile devices
the promise of mHealth:eliminate doctor visits remote / automatic diagnosis equalize developing countries
stress check
telemedicinefitness trainer
heart rate
current mhealthremote training43,000 apps for health on the app store
96% are for calorie counting & exercise 4% are remote monitoring !
yet to be a disruptive mHealth technology
consider physician’s needsconnecting with patient tracking baselines personalized trending data managing chronic disease
75% of all US healthcare spending is on chronic disease
source: center for disease control, 2014
mHealth sensing in the home for managing disease
compliance?cost?
privacy?data reliability?
compliance
phone as a sensorbaseline quantitysensor
embedded sensors processing
estimated
accelerometer gyroscope magnetometer /compass dual camera / flash 1+ microphones proximity sensor capacitive sensor gps motorized actuator wireless antenna (s)
compliance++; cost--;
data reliability?
what can the mobile phone sense with clinical accuracy?
ongoing research
lung function jaundice
ongoing research
lung function jaundice
spirometer lung function??
lung function
asthma COPD cystic fibrosis
evaluates pulmonary impairments
spirometer
device that measures amount of air inhaled and
exhaled.
using a spirometer
flow
volume
volum
e
time
using a spirometer
flow
volume
volum
e
time
volume-time graphvo
lume
time
volume-time graphvo
lume
time1 sec.
FEV1
FVCFEV1% = FEV1/FVC
FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
FEV1% = FEV1/FVC
> 80% healthy60 - 79% mild40 - 59% moderate
< 40% severe
flow-volume graphflo
w
volume
flow
volumeFEV1 FVC
1 sec.
PEF
PEF: Peak Expiratory Flow FEV1: Forced Expiratory Volume in 1 second FVC: Forced Vital Capacity
flow-volume graph
flow
volume
normalobstructive
flow-volume graph
obstructive diseases
!
resistance in air path leads to reduced air flow
restrictive diseases
!
lungs are unable to pump enough air and pressure
flow-volume graphFlo
w
Volume
normal
restrictiveobstructive
clinical spirometry
home spirometry
!
faster detection rapid recovery
trending
home spirometry
high cost barrier patient compliance
less coaching limited integration
challenges with
flow rate volume
lung functionairflow sensor
sound pressure microphone processing
estimated
SpiroSmart
availability cost portability more effective coaching interface integrated uploading
Using SpiroSmart
Using SpiroSmart
]
Using SpiroSmart
0 1 2 3 4 5 6 7−1
−0.5
0
0.5
1
time(s)
amplitude
flow features estimation
0 1 2 3 4 5 6 7−1
−0.5
0
0.5
1
time(s)
amplitude
envelope detection
0 1 2 3 4 5 6 7−1
−0.5
0
0.5
1
time(s)
amplitude
resonance tracking0 1 2 3 4 5 6 7
−1
−0.5
0
0.5
1lpc8raw
time(s)
amplitude
flow estimation features
vocal tractsource output 0 1 2 3 4 5 6 7−1
−0.5
0
0.5
1lpc8raw
time(s)
amplitude
auto-regressive estimate
time(s)
frequency(Hz)
1 2 3 4 5 60
500
1000
1500
2000
2500
0 1 2 3 40
5
10
15
Flow
(L/s
)
Volume(L)
0 2 4 6 8 100
1
2
3
4
time(s)
Volu
me(
L)
0 1 2 3 40
5
10
15
Flow
(L/s
)Volume(L)
0 2 4 6 8 100
1
2
3
4
time(s)
Volu
me(
L)
audio
flow features
measures regression
curve regression
lung functionFEV1FVCPEF
study design
x 3
x 3
study enrollment
participants 5218-75 years old, mostly healthy
study a
participants 1012-17 years old, mixed healthy/abnormal
study b
participants 5610-69 years old, mostly abnormal
study c
enrolled by hospitals
resultsmeasures regressionPE
F, Sp
iroSm
art 16
14 12 10 8 6 4 22 4 6 8 10 12 14 16
FEV1
, Spi
roSm
art
!!
6 5 4 3 2 1
1 2 3 4 5 6
FVC,
Spi
roSm
art
!7 6 5 4 3 2
2 3 4 5 6 7 FEV1
%, S
piro
Smar
t100
90 80 70 60 50 40 40 50 60 70 80 90 100
FEV1%, actual
FEV1, actualPEF, actual
FVC, actual
resultsmeasures regressionPE
F, Sp
iroSm
art 16
14 12 10 8 6 4 22 4 6 8 10 12 14 16
FEV1
, Spi
roSm
art
!!
6 5 4 3 2 1
1 2 3 4 5 6
FVC,
Spi
roSm
art
!7 6 5 4 3 2
2 3 4 5 6 7 FEV1
%, S
piro
Smar
t100
90 80 70 60 50 40 40 50 60 70 80 90 100
FEV1%, actual
FEV1, actualPEF, actual
FVC, actual
−2 0 2 4 60
5
10
15
volume(L)
flow(L/s)
−2 0 2 4 60
5
10
15
volume(L)
flow(L/s)
−2 0 2 4 60
5
10
15
volume(L)
flow(L/s)
−2 0 2 4 60
5
10
15
volume(L)
flow(L/s)
−1 0 1 2 3 40
2
4
6
8
volume(L)
flow(L/s)
−1 0 1 2 3 40
2
4
6
8
volume(L)
flow(L/s)
resultscurves regression
can SpiroSmart curves be used for diagnosis?
• 5 pulmonologists
• normal/abnormal subjects curves
• unaware if from SpiroSmart / spirometer
survey
results
normal minimal obstructive mild obstructive moderate obstructive severe obstructive
restrictive
inadequate
error 8%false negative
4%
false positive 14%
one off 10%
identical 64%
!
!
abnormal vs normal
96%
appropriate for trending and screening
global health non-profit
patient and doctors pharmaceutical drug trials
in 10 years, COPD will surpass AIDS/HIV as the leading cause of death in developing nations
ongoing research
lung function jaundice
ongoing research
lung function jaundice
Kernicterus StatisticsKernicterus:+21+($8+mill)+
Hazardous+jaundice:++1158+
($50,000)+
Extreme+jaundice:++2,317+($20,000)+
Severe+jaundice:+35,000+($8,500)+
Phototherapy:+290,000++($1,000)+
Visible+jaundice:+3.5+million+
Births/year:+4.1+million+
In the US Middle- & low-income countries:
• 75,000 cases kernicterus/year
• 114,000 newborn deaths/year
• 65% newborn deaths from kernicterus
kernicterus 21
($8 million)
Bhutani et al., Pediatric Research 2010
Total Serum Bilirubin
Medical GoldStandard
Transcutaneous
TcB
Bilirubinometer
TcBNon-invasive
Correlates 0.75-0.93
$7000
Quick results
Screening tool for TSB
20 !!
15 !!
10 !!
5 !!
0
0.5 1 2 3 4 5 6 Age (days)In Hospital
Biliru
bin
(mg/
dL)
Newborn Bilirubin Levels
75th percentile
25th percentile
Visual Assessment • Parents • Many physicians • Traveling practitioners
In Hospital At Home
Screening Challenges
Tend to underestimate
bilirubin level in blood
jaundice levelblood draw
yellowness camera processing
estimated
bilicam
Study Evaluation100 newborn participants
• <1 day old when enrolled
• 59% white !
2 medical centers !
Data collected by medical professionals using iPhone 4S
Photos: with & without flash
TSB (ground truth)
TcB (control)
3 - 5 days old
Data Collection App
!
Standardize • white balance • card position • phone position
!
Noisy Data
Automatic Quality Control
✔ ✖ ✖
✖ ✖ ✖
Ideal Glare Overexposed
Occlusion Shadow Underexposed
Algorithm Overview
Bilirubin Estimate
Color Balance
Extract Features
Apply Regressions
400 500 600
Wavelength (nm)
Rela
tive
Bili
rubi
n
Abs
orpt
ion
Prob
abili
ty
Bilirubin Absorption Properties
Feature Extraction
Gradient (of RGB channels)
with & without flash
skin
RGB
Cr
CbY
YCbCr
a*
b*
L*
L*a*b*
Extract Features
Apply Regressions
Color Balance
Algorithm Overview
Bilirubin Estimate
Regression EnsembleRegression
Regression
Regression
Regression
RegressionBilirubin Estimate
Combine
Regression Ensemble
No
90th percentileYes
regressions agree
mean
Least Angle Regressions
LARS-Lasso Elastic Net
Support Vector Regressions
k-Nearest Neighbor
Random Forest Regression
Bilirubin Estimate
Results
0
5
10
15
20
25
0 5 10 15 20 25
TSB Ground Truth (mg/dl)
Estim
ated
Bili
rubi
n (m
g/dl
)
Results
0
5
10
15
20
25
0 5 10 15 20 25
TSB Ground Truth (mg/dl)
Estim
ated
Bili
rubi
n (m
g/dl
)
Results
0
5
10
15
20
25
0 5 10 15 20 25
TSB Ground Truth (mg/dl)
Estim
ated
Bili
rubi
n (m
g/dl
) BiliCamrank order 0.85 correlation
Results
0
5
10
15
20
25
0 5 10 15 20 25
TSB Ground Truth (mg/dl)
Estim
ated
Bili
rubi
n (m
g/dl
)
TcBs correlate 0.75 - 0.93
BiliCamrank order 0.85 correlation
TcBrank order 0.92 correlation
Interpretation20 !!
15 !!
10 !!
5 !!
0
high risk
high intermediate risk
low intermediate risk
low risk
Bili
rubi
n (m
g/dL
)
0.5 1 2 3 4 5 6
Age (days)
20 !!
15 !!
10 !!
5 !!
0
high risk
high intermediate risk
low intermediate risk
low risk
Bili
rubi
n (m
g/dL
)
0.5 1 2 3 4 5 6
Age (days)
Bhutani Nomogram
20 !!
15 !!
10 !!
5 !!
0
high riskhigh intermediate risk
low intermediate risk
low risk
Bili
rubi
n (m
g/dL
)
0.5 1 2 3 4 5 6
Age (days)
Interpretation
9 high risk cases based on TSB
20 !!
15 !!
10 !!
5 !!
0
high riskhigh intermediate risk
low intermediate risk
low risk
Bili
rubi
n (m
g/dL
)
0.5 1 2 3 4 5 6
Age (days)
Interpretation
BiliCam 2/9 missed high risk (22%)85% blood draws avoided
20 !!
15 !!
10 !!
5 !!
0
high riskhigh intermediate risk
low intermediate risk
low risk
Bili
rubi
n (m
g/dL
)
0.5 1 2 3 4 5 6
Age (days)
Interpretation
BiliCam 2/9 missed high risk (22%) 85% blood draws avoided
TcB 2/8 missed high risk (25%) 88% blood draws avoided
BiliCam is sufficient for newborn Jaundice screening, but it is unknown how user error affects reliability
ongoing research
lung function jaundice
ongoing research
lung function jaundice
cervical cancer screening
affect sensingintra ocular pressure
oxygen volume VO2
cardiac output & blood pressure
> slide to unlock
Thank You!
eric c. larson | eclarson.com
flipping the clinic
Assistant Professor Computer Science and Engineering
in home health monitoring using mobile phonescollaborators: !Suku Nair Eric Bing Joseph Camp Dinesh Rajan Sohail Rafiqi Mark Wang Shwetak Patel Jim Stout, MD Jim Taylor, MD Margaret Rosenfeld, MD Gaetano Boriello Mayank Goel Lilian DeGreef
eclarson.com [email protected] @ec_larson