Anomaly Detectionby

ADGM/LVAE

NaotoMizunoMentor:Tanaka-san,Okanohara-san

Introduction

• Anomalydetection• Data• NABDataset(Artificial)• (Otherdataarenotopentothispresentation)

• Model• AuxiliaryVAE(ADGM)• LadderVAE• VAE(previouswork)

Variational Auto-Encoder(VAE)

• Weassumethatthedata𝑥 aregeneratedfromthelatentvariables𝑧.

• Weuseneuralnetworkasencoder anddecoder.

x x

zz

𝑞$(𝑧|𝑥) 𝑝)(𝑥|𝑧)Encoder Decoder

Data

LatentVariable

VAE

• Weuselowerboundoflog 𝑝) 𝑥 aslossfunction.

log 𝑝) 𝑥 ≥ 𝐸9: 𝑧 𝑥 log𝑝) 𝑥, 𝑧𝑞$ 𝑧 𝑥

𝑝) 𝑥, 𝑧 = 𝑝) 𝑥|𝑧 𝑝) 𝑧

• 𝑝) 𝑧 :Standardnormaldistribution• In training, 𝑧 is chosen from 𝑞$ 𝑧 𝑥 .

ADGM• Semi-supervisedLearning• Detectlabel𝑦 andreconstructdata𝑥.• Auxiliaryvariableincreasetheflexibilityofthemodel.

x

az y

Data

LatentVariable

AuxiliaryVariable

Label

x

az y

x

az y

SDGM

ObjectivefunctionofADGM• Forlabeleddata

• Lowerbound+classificationloss

𝐿 𝑥, 𝑦 = −𝐸9: 𝑎, 𝑧 𝑥, 𝑦 log𝑝) 𝑥, 𝑦, 𝑎, 𝑧𝑞$ 𝑎, 𝑧 𝑥, 𝑦 − 𝛼𝐸9: 𝑎 𝑥 log 𝑞$ (𝑦|𝑎, 𝑥)

• Forunlabeleddata𝑈 𝑥 = −𝐸9: 𝑎, 𝑦, 𝑧 𝑥 log

𝑝) 𝑥, 𝑦, 𝑎, 𝑧𝑞$ 𝑎, 𝑦, 𝑧 𝑥

• Total𝐽 = J 𝐿(𝑥K, 𝑦K)

�

MN,ON

+J𝑈(𝑥Q)�

MR

ADGMforMNIST

• Semi-supervisedlearning• 100labeled,60000unlabeled• Testerror

ADGM:0.96%SDGM:1.32%

• Generateimage• Choosing𝑧 fromGaussian• Generatewitheach𝑦

SDGM

Withoutauxiliaryvariable

AuxiliaryVAE• UnsupervisedLearning• Severalsamplinglayers(1or2)

x

z

a

a

z

x

z

a

a

z

LadderVAE

• Severalsamplinglayers(~5)• VAEwithseveralsamplinglayersisdifficulttotrain.

• Sharingtheinformationbetweendecoder andencoder.

x x

zd

d z

d z

LadderVAE

• Encoderusedecoderoutputasprior.

𝜎9T =1

𝜎V9WT + 𝜎XWT

𝜇9 =𝜇V9𝜎V9WT + 𝜇X𝜎XWT

𝜎V9WT + 𝜎XWT z

d

𝜇X, 𝜎XPrior

Likelihood Posterior

Sampling

𝜇V9, 𝜎V9 𝜇9, 𝜎9

Anomalydetection

• Modelistrainedwithoutanomalydata.• Modelcannotreconstructanomalydata.𝐴𝑛𝑜𝑚𝑎𝑙𝑦𝑆𝑐𝑜𝑟𝑒 = log 𝐸 log 𝑝)(𝑥|𝑧)

• MNISTwithnoise

NABDataset(Artificial)

• Weconvertrawdatatospectrogram.• Spectrogram:theamplitudesataparticularfrequencyandtime.

• Input:theamplitudesata time.

rawdata

anomalyspectrogram

singleinput

time

frequ

ency

NAB

• Scoresincreaseatanomaly.

train test

ADGM

LVAE

NAB

• Inthiscasemodelscannotdetectanomaly.• Smallinputvaluetendstoresultinsmallscore.

train test

ADGM

LVAE

Conclusion

• AnomalydetectionusingADGM/LVAE.• Anomalyisdetectedaslowprobabilitydata.

• PerformancesarealmostsameasVAE.• Manysamplinglayersarebetter(?)