Data Virtualization for Business Intelligence SystemsRevolutionizing Data Integration for Data Warehouses

Rick F. van der Lans

([FHUSWFUHDWHGIRU,QIRUPDWLFD&RUSRUDWLRQ&KDSWHU7KH)XWXUHRI'DWD9LUWXDOL]DWLRQ_

'DWD9LUWXDOL]DWLRQIRU%XVLQHVV,QWHOOLJHQFH6\VWHPV5HYROXWLRQL]LQJ'DWD,QWHJUDWLRQIRU'DWD:DUHKRXVHV&RS\ULJKW(OVHYLHU,QF$OOULJKWVUHVHUYHG

7KH)XWXUHRI'DWD9LUWXDOL]DWLRQ$FFRUGLQJWR-DPHV0DUNDULDQ&72RI,QIRUPDWLFD&RUSRUDWLRQ([SRQHQWLDO JURZWK LQ WUDGLWLRQDO WUDQVDFWLRQDO GDWD SOXV QHZ LQIRUPDWLRQ IURP VRFLDOPHGLD FDOOGHWDLO UHFRUGV VHQVRUV DQG GHYLFHV DQG JHRORFDWLRQ V\VWHPV KDV PDGH LW LPSHUDWLYH WKDWEXVLQHVVHV WDNH WKH OHDG LQKDUQHVVLQJGDWD WRGHULYHQHZ LQVLJKWV IRUFRPSHWLWLYHDGYDQWDJH7RWXUQ WKHVH QHZ LQVLJKWV LQWR EXVLQHVV RSSRUWXQLWLHV ERWK EXVLQHVV DQG ,7 H[HFXWLYHV DUH EHLQJFKDOOHQJHGWRUHWKLQNWKHLULQIRUPDWLRQPDQDJHPHQWSUDFWLFHVEUHDNGRZQRUJDQL]DWLRQDODQGGDWDVLORVDQGLPSURYHEXVLQHVV,7FROODERUDWLRQ:KLOHPDQ\FRPSDQLHVNQRZWKDWGDWDLVNH\WRGULYLQJFRPSHWLWLYHDGYDQWDJHWKH\DOVRNQRZWKDWWKH\PD\QRWEHDVHIIHFWLYHDVWKH\FRXOGEHDWWUDQVIRUPLQJLWLQWRDXVHIXODVVHW7KHGDWDLVWKHUHEXWLWVIUDJPHQWHGDFURVVV\VWHPVLWVLQWKHFORXGLWVRQWKHGHVNWRSLWVRQPRELOHGHYLFHVLWVLQDYDULHW\RIGDWDVRXUFHV,QIDFWLWVHYHU\ZKHUHDQGLWVGXSOLFDWHGDJDLQDQGDJDLQ,WVRIWHQQRWDYDLODEOHZKHQLWVQHHGHGDQGZKHQLWLVDYDLODEOHLWPLJKWEHIXOORIHUURUVRUGRHVQRWGHOLYHUWKHYDOXHGHVLUHGWRLPSURYHRSHUDWLRQVDQGLQFUHDVHUHYHQXH$QGDVGDWDJURZVWKHFRVWWRPDQDJHLWRQO\LQFUHDVHV7KLVLVSDUWLFXODUO\WUXHLQWKHFDVHRIEXVLQHVVLQWHOOLJHQFHWKHWRSSULRULW\IRUWKH&,2$FFRUGLQJWR)RUUHVWHU 5HVHDUFK ,QF $V GDWD YROXPHV DQG LQIRUPDWLRQ FRPSOH[LW\ FRQWLQXH WR VN\URFNHWWUDGLWLRQDO EXVLQHVV LQWHOOLJHQFH WRROV WU\ KDUG WR NHHS XS ZLWK HYHU LQFUHDVLQJ DQG FKDQJLQJGHPDQGV%XW LWVDQXSKLOOEDWWOH DQGEXVLQHVV LQWHOOLJHQFH WRROVDQGDSSOLFDWLRQVGRQRWDOZD\VNHHS XS WKH ULJKW OHYHO RI SDFH DQG DGYDQFHPHQW $V D UHVXOW WKH ULIW EHWZHHQ EXVLQHVVUHTXLUHPHQWV IRU RQGHPDQG LQIRUPDWLRQ DQG UHDOWLPH GHFLVLRQV DQG EXVLQHVV LQWHOOLJHQFHDSSOLFDWLRQVDQG,7VXSSRUWVWDIIVDELOLW\WRVXSSRUWWKHPFRQWLQXHVWRJURZ$FFRUGLQJWRWKHUHSRUW

RI%,UHTXLUHPHQWVFKDQJHRQEHWZHHQDGDLO\DQGPRQWKO\EDVLV RI WKH UHVSRQGHQWV VDLG WKH\ KDYH WR DVN GDWD DQDO\VWV WR FUHDWH FXVWRP UHSRUWV IRUWKHP

RIFXVWRPUHSRUWUHTXHVWVUHTXLUHDFXVWRPFXEHRUGDWDPDUWWRDQVZHUWKHUHTXHVW RI UHVSRQGHQWV FLWHG WKDW LW WDNHV EHWZHHQ GD\V DQGPRQWKV WR JHW WKHLU %, UHTXHVWVIXOILOOHG

,W LVFOHDU LQRUGHU WR WXUQGDWD LQWRQHZ LQVLJKWVDQG WKXVRSSRUWXQLWLHVEXVLQHVVHVQHHG WRSD\JUHDWHUDWWHQWLRQWRGDWDLQWHJUDWLRQWRSURYLGHPRUHDFWLRQDEOHWUXVWZRUWK\DQGWLPHO\LQIRUPDWLRQLQ WKHLU%,SURMHFWVDQGLQLWLDWLYHV7KH\QHHGWRSXWGDWD LQWHJUDWLRQDQGEXVLQHVVXVHUGULYHQVHOIVHUYLFH WRZRUN IRUXQFRYHULQJQHZ LQVLJKWV WKDW WKHEXVLQHVVQHHGVDQGFDQ WUXVW'RLQJ WKLVZLOOGULYH JURZWK UHGXFH FRVWV DQG GHOLYHU LQQRYDWLRQV WKURXJK VSRWWLQJ SDWWHUQV DQG WUHQGV ZKLOHPHHWLQJFRPSOLDQFHDQGULVNPDQGDWHVPRUHHIIHFWLYHO\%XWKRZGR\RXDFFRPSOLVKWKLV"+RZWR0D[LPL]H5HWXUQRQ'DWDZLWK'DWD9LUWXDOL]DWLRQ+HUHDUHVRPHFULWLFDOIDFWRUVWKDWRUJDQL]DWLRQVPXVWFRQVLGHULQRUGHUWRPD[LPL]HWKHLUUHWXUQRQGDWD

7KH\ PXVW EH PRUH EXVLQHVVIRFXVHG DQG WKLQN IURP D SHUVSHFWLYH RI DQ HQGWRHQGEXVLQHVVXVHRILQIRUPDWLRQDQGDVVRFLDWHGSURFHVVHV

7KH\PXVW HPSOR\ VHOIVHUYLFH DQG EXVLQHVV,7 FROODERUDWLRQ EHVW SUDFWLFHV WR HQDEOH WKHEXVLQHVVWRRZQWKHLUGDWDZKLOH,7UHWDLQVFRQWURODQGJRYHUQDQFH

7KH\ PXVW HPSOR\ DJLOH GDWD LQWHJUDWLRQ WHFKQLTXHV WKDW WKH FXW ZDLW DQG ZDVWH LQ WKHSURFHVVZKLOHFRPSOHPHQWLQJWUDGLWLRQDODSSURDFKHVWR%,

7KH\ QHHG WR GHILQH XVHUGULYHQ VHUYLFHOHYHO DJUHHPHQWV 6/$V DURXQG GDWD ODWHQF\DFFXUDF\FRQVLVWHQF\DYDLODELOLW\DQGXSWLPHDQGXQGHUVWDQGV\VWHPLPSOLFDWLRQV

([FHUSWFUHDWHGIRU,QIRUPDWLFD&RUSRUDWLRQ&KDSWHU7KH)XWXUHRI'DWD9LUWXDOL]DWLRQ_

'DWD9LUWXDOL]DWLRQIRU%XVLQHVV,QWHOOLJHQFH6\VWHPV5HYROXWLRQL]LQJ'DWD,QWHJUDWLRQIRU'DWD:DUHKRXVHV&RS\ULJKW(OVHYLHU,QF$OOULJKWVUHVHUYHG

7KH\ PXVW XVH GDWD WR FRPSOHPHQW WKHLU EXVLQHVV VWUDWHJ\ DQG WHFKQLFDO LQIUDVWUXFWXUHLQVWHDGRIUHLQYHQWLQJWKHHQWLUHZKHHO

2IODWHGDWDYLUWXDOL]DWLRQKDVHYROYHGDVDQDJLOHGDWDLQWHJUDWLRQFRQFHSWWRHQDEOHPRUHDJLOH%,+RZHYHU WUDGLWLRQDO %, DSSURDFKHV LQFOXGLQJ GDWD LQWHJUDWLRQ GDWD ZDUHKRXVLQJ DQG RWKHUFRPSOH[ GDWD SURFHVVLQJ LQLWLDWLYHV DUH QRW JRLQJ DZD\ DQ\WLPH VRRQ 7KLV LV EHFDXVH GDWD ZLOOFRQWLQXH WR EH KHWHURJHQHRXV GLUW\ DQG VHPDQWLFDOO\ GLIIHUHQW DFURVV V\VWHPV 7R VXFFHHGGDWDYLUWXDOL]DWLRQPXVWFRH[LVWUHXVHDQGFRPSOHPHQWH[LVWLQJLQIUDVWUXFWXUHDQGLQYHVWPHQWVPDGHWRVROYHWKHVHSUREOHPVUDWKHUWKDQEHD%DQG$LGIRUDVPDOOVXEVHWRIVSHFLDOXVHFDVHV,WPXVWDOVRLQYROYHWKHEXVLQHVVXVHUHDUO\DQGRIWHQWRHQVXUHWKDWWKHGDWDLVWUXVWZRUWK\%H\RQG/RRNLQJ8QGHUWKH+RRG7KHWULFNLVWRJDLQFRPSHWLWLYHDGYDQWDJHE\DFFHOHUDWLQJWKHGHOLYHU\RIFULWLFDOGDWDDQGUHSRUWVDQG EH DEOH WR WUXVW DQG FRQVXPH WKHP LQVWDQWO\ %XW GDWD YLUWXDOL]DWLRQ PXVW EH GRQH ULJKW WRVXSSRUW WKH FULWLFDO VXFFHVV IDFWRUV 9HU\ RIWHQ GDWD YLUWXDOL]DWLRQ ERUURZV KHDYLO\ IURP LWV GDWDIHGHUDWLRQOHJDF\7KHSULPDU\XVHFDVHGDWDIHGHUDWLRQGRHVZHOOLVWRDFFHVVDQGPHUJHDOUHDG\FOHDQHGDQGFRQIRUPHGGDWDLQUHDOWLPHOHDYLQJWKHKHDY\OLIWLQJIRURWKHUSURFHVVLQJORJLFWRPDNHWKLVSRVVLEOH6R WKH WLPHDGYDQWDJHJDLQHG LV ORVWDVRQH UHDOL]HV WKH IHGHUDWHGGDWDKDG WREHSUHSSHGIRUIHGHUDWLRQ$VDUHVXOWWKH52,VLPSO\GLVDSSHDUV6RGRJREH\RQG ORRNLQJXQGHU WKHKRRGDQGDVND IHZKDUGTXHVWLRQV7RZKDWH[WHQWGRHV WKHVROXWLRQ VXSSRUW GDWD WUDQVIRUPDWLRQ" ,V LW QRPLQDO OLPLWHG WRZKDW \RX FDQ GR SURJUDPPDWLFDOO\WKURXJK 64/ RU ;4XHU\" ,V WKHUH DQ\ GDWD SURILOLQJ RU ZLOO \RX UHTXLUH VWDJLQJ DQG IXUWKHUSURFHVVLQJ",VLWSURILOLQJRIERWKORJLFDQGVRXUFHVMXVWVRXUFHVRUQHLWKHU",VGDWDFOHDQVLQJDQGFRQIRUPLQJVLPSOLVWLFKDQGFRGHGRUQRQH[LVWHQW"+RZDERXWUHXVH"&DQ\RXTXLFNO\DQGHDVLO\UHXVHYLUWXDOYLHZV IRUDQ\XVHFDVH LQFOXGLQJEDWFK"7RGRGDWDYLUWXDOL]DWLRQ ULJKW LW UHTXLUHVDGHHS DQG WKRURXJK XQGHUVWDQGLQJ RI YHU\ FRPSOH[ SUREOHPV WKDW H[LVW LQ WKH GDWD LQWHJUDWLRQGRPDLQ6RZKDWVRXUSHUVSHFWLYH"6LPSO\SXWGDWDYLUWXDOL]DWLRQPXVWWDNHDSDJHIURPWKHZRUOGRIYLUWXDOPDFKLQHV 'DWD YLUWXDOL]DWLRQ PXVW GR WKH KHDY\ OLIWLQJ RI DFFHVVLQJ SURILOLQJ FOHDQVLQJWUDQVIRUPLQJDQGGHOLYHULQJIHGHUDWHGGDWDWRDQGIURPDQ\DSSOLFDWLRQRQGHPDQG,WPXVWKDQGOHDOOWKHXQGHUO\LQJGDWDFRPSOH[LW\LQRUGHUWRSURYLGHFRQIRUPHGDQGWUXVWHGGDWDUHXVLQJWKHORJLFIRUHLWKHUEDWFKRUUHDOWLPHRSHUDWLRQZKHWKHUWKURXJK64/62$5(67-621RUQHZDFURQ\PV\HW WREHVSHFLILHG'DWDYLUWXDOL]DWLRQPXVWEHEXLOWJURXQGXSRQ WKHFXW WKHZDLWDQGZDVWHEHVWSUDFWLFHVGLVFXVVHGLQWKHERRNRQ/HDQ,QWHJUDWLRQE\6FKPLGWDQG/\OH%\VWDUWLQJZLWKDORJLFDOGDWDPRGHOJLYLQJEXVLQHVVDQG,7UROHEDVHGYLVLELOLW\LQWRWKHGDWDHDUO\LQ WKH SURFHVV HQDEOLQJ GDWD SURILOLQJ RQ IHGHUDWHG GDWD WR VKRZ DQG UHVROYH WKH GDWD LQWHJULW\LVVXHVDSSO\LQJDGYDQFHGWUDQVIRUPDWLRQVLQFOXGLQJGDWDTXDOLW\LQUHDOWLPHWRIHGHUDWHGGDWDDQGFRPSOHWHO\DQGLQVWDQWO\UHXVLQJWKHLQWHJUDWLRQORJLFRUYLUWXDOYLHZVIRUEDWFKRUUHDOWLPH\RXFDQFXWWKHZDLWDQGZDVWHWKURXJKRXWWKHSURFHVV%\OHYHUDJLQJRSWLPL]DWLRQVSDUDOOHOLVPSLSHOLQLQJLGHQWLW\ UHVROXWLRQ DQG RWKHU FRPSOH[ WUDQVIRUPDWLRQDO FDSDELOLWLHV \RX FDQ RQO\ ILQG LQ DPDWXUHGDWDLQWHJUDWLRQSODWIRUPGDWDYLUWXDOL]DWLRQFDQHQDEOHPRUHDJLOHEXVLQHVVLQWHOOLJHQFH)LQDOO\ZLWKHQWHUSULVHVJHQHUDWLQJKXJHYROXPHVRIGDWD WKHW\SHVRIGDWDFKDQJLQJHQRUPRXVO\DQGQHHGIRUVKRUWHUGDWDSURFHVVLQJVSHHGVGDWDYLUWXDOL]DWLRQFDQPD[LPL]HWKHUHWXUQRQGDWD