ABSTRACT
YH OXVLRQV
INTRODUCTION
[LFRORJLFDO WHVW PHWKRGV HYROYHG HDUOLHU LQ WKLV FHQWXU\ LQ ]HGQHHGE\ERWKLQGXVWU\DQGJRYHUQPHQWUHJXODWRU\DJHQFLHV RGXFWV WRHQVXUHFRQVXPHUVDIHW\$VDUHVXOWRIDQXPEHURI IURP FRQVXPHU SURGXFWV LQ WKH V &RQJUHVV SDVVHG WKH DQG&RVPHWLF$FWRI 86&RGHZKLFKPDQGDWHG LQJIRUFHUWDLQSURGXFWV LQWKHULVNDVVHVVPHQWRIPDQ\SURGXFWVLVWKHHYDOXDWLRQRI HH\HLQMXU\RUH\HLUULWDWLRQ7KH'UDL]HUDEELWH\HLUULWDWLRQ
HWKRG IRU HYDOXDWLQJ WKH RFXODU LUULWDWLRQ SRWHQWLDO RI D ORSHG LQ UHVSRQVH WR WKHQHHG IRU WKLV W\SH RI LQIRUPDWLRQ )ULHGHQZDOGHWDO,QWKLVWHVWDPDWHULDOLVLQVWLOOHG RUDEELWVGLIIHUHQWPRGLÀFDWLRQVRIWKHWHVWUHTXLUHGLIIHUHQW DQGWKHRFXODUUHVSRQVHRIWKHDQLPDOVLVPRQLWRUHGXVLQJ
DVWDQGDUGL]HGVFRU XSWRG
$QLPDOZHOIDUH UHVHDUFKSXUSRVHVDU WKH 8QLYHUVLWLHV )HG KXPDQHWHFKQLTXHLQ WRWKHSXEOLFDWLRQRI PHQWDO7HFKQLTXHLQ WKHWKUHH5VUHJDUG
7KHPDQGDWH IRU WR HQVXUH WKH VDIHW\ JURXSVKDYHFRQÀGHQ IRU HVWLPDWLQJ WKH K SURYLGH DGHTXDWH VD 'UDL]H WHVWZLOO KDY XVHIXOLQWKHULVNDVV
)RUYDOLGDWLRQQ DEOHZKLFKPHDQV U PHDQV WKHPHWKRG L 1,(+6 7R DQG UHJXODWRU\ DFFH ZHUH GHYHORSHG E\ 2UJDQL]DWLRQIRU(FR WKH(XURSHDQ&HQWHU DODQGWKH,Q QDWLYH0HWKRGV ,&& >1,(+6@
$Q DOWHUQDWLYHP EHHQYDOLGDWHGE\D LQGXVWU\ DQG86 UH DJHQF\DFFHSWDQFHR
7KHGHYHORSPHQ E\FXUUHQWVFLHQWLÀF DJHQF\ DQG LQGXVWU\ QRQDQLPDOWR[LFROR VFLHQWLÀFDQGPHFKD
$Q LQGXVWU\ZLG YDOLGDWHGDQGUHJXOD XDWLRQV7KHFRPSHWL DQGPDUNHWQHZSUR UHJDUGLQJ WKH VDOH R SURGXFWVDIHW\WHVWLQ
LQJV\VWHPIRU WKHFRUQHD WKHFRQMXQFWLYD DQG WKH LULV IRU
FRQFHUQV RYHU WKH XVH RI DQLPDOV IRU DOO W\SHV RI WHVWLQJ DQG RVHLQWKHHDUO\V$WWKDWWLPH&KDUOHV+XPHIRXQGHURI HUDWLRQ IRU $QLPDO:HOIDUH SURSRVHG ´D VFLHQWLÀF VWXG\ RI ODERUDWRU\DQLPDOH[SHULPHQWVµ%DOOV7KLVVWXG\OHG 5XVVHOODQG%XUFK7KH3ULQFLSOHVRI+XPDQH([SHUL ZKLFKWKHFRQFHSWVRIUHGXFWLRQUHÀQHPHQWDQGUHSODFHPHQW LQJWKHXVHRIDQLPDOVLQH[SHULPHQWDWLRQZHUHGHÀQHG ERWK LQGXVWU\DQGJRYHUQPHQW UHJXODWRU\DJHQF\ VFLHQWLVWV LV RI SURGXFWV WKDW DUHPDGH DYDLODEOH IRU FRQVXPHU XVH %RWK FHLQXVLQJWKHFXUUHQWO\DFFHSWHG'UDL]HUDEELWH\HWHVWPHWKRG XPDQ H\H VDIHW\ RI D SURGXFW7RPDLQWDLQ WKLV LQWHJULW\ DQG IHW\ LQIRUPDWLRQ QHZ WHVW PHWKRGV GHYHORSHG WR UHSODFH WKH H WREHYDOLGDWHG DQGZLOO KDYH WRSURYLGH LQIRUPDWLRQ WKDW LV HVVPHQWSURFHVV HZWR[LFRORJLFDO WHVWPHWKRGVPXVWGHPRQVWUDWH WKH\DUHUHOL HSHDWDEOH DQG UHSURGXFLEOH DQG WKDW WKH\ DUH UHOHYDQWZKLFK V SUHGLFWLYH DQG KDV D ELRORJLFDO EDVLV IRU WKH VWDWHG SXUSRVH HQVXUH WKH VWDQGDUGL]DWLRQ RI WKHVH GHWHUPLQDWLRQV YDOLGDWLRQ SWDQFH FULWHULD IRU HYDOXDWLQJ QHZ WR[LFRORJLFDO WHVW PHWKRGV WKUHH GLIIHUHQW LQWHUQDWLRQDO RUJDQL]DWLRQV LQ WKH V WKH QRPLF&RRSHUDWLRQDQG'HYHORSPHQW2(&'2(&' IRUWKH9DOLGDWLRQRI$OWHUQDWLYH0HWKRGV(&9$0%DOOVHW WHUDJHQF\&RRUGLQDWLQJ&RPPLWWHHRQWKH9DOLGDWLRQRI$OWHU 9$0 1DWLRQDO ,QVWLWXWH IRU(QYLURQPHQWDO+HDOWK6FLHQFHV
HWKRG WKDW FDQ UHSODFH WKH'UDL]H UDEELW H\H WHVW KDV QRW \HW Q\RI WKHVHRUJDQL]DWLRQV7KLVSDSHUSURYLGHVDQRYHUYLHZRI JXODWRU\ SHUVSHFWLYHV UHODWHG WR WKH YDOLGDWLRQ DQG UHJXODWRU\ I'UDL]HDOWHUQDWLYHWHVWPHWKRGV
INDUSTRY PERSPECTIVE
WDQGXVHRIDQLPDODOWHUQDWLYHPHWKRGVE\LQGXVWU\DUHGULYHQ VRFLDOHFRQRPLFDQGSROLWLFDOFRQVLGHUDWLRQV%RWKJRYHUQPHQW VFLHQWLVWV DUH HQFRXUDJLQJ GHYHORSPHQW DQG XVH RI YDOLGDWHG JLFDOWHVWPHWKRGVVLQFHWKH\KDYHWKHSRWHQWLDOWRSURYLGHEHWWHU QLVWLFGDWDDQGWKHUHIRUHEHWWHUKXPDQULVNDVVHVVPHQW H SUREOHP KDV EHHQ WKH ODFN RI DYDLODEOH QRQZKROH DQLPDO
WRU\DFFHSWHGWR[LFRORJLFDOWHVWPHWKRGVIRUSURGXFWVDIHW\HYDO WLYHQHVVRIPDQ\PDQXIDFWXUHUVKLQJHVRQWKHLUDELOLW\WRGHYHORS GXFWV$OOJOREDOFRPSDQLHVDUHIDFHGZLWKSRWHQWLDOOHJLVODWLRQ I DQLPDOWHVWHGSURGXFWV2UJDQL]DWLRQV WKDW KDYH WR HQJDJH LQ JDUHIDFHGZLWKDQXQXVXDOW\SHRIWHFKQRORJLFDOGLVFRQWLQXLW\
8WWHUEDFN³W ZLWKRXWKDYLQJDGHT WRZDUGGHYHORSLQJDS WRWKHVWUDWHJLFQHHG DWWKLVWLPHLVWKDWYD DUHQRWDYDLODEOH
$ PDMRU HIIRUW GHYHORSPHQWDQGYDO WKH'UDL]HUDEELWH\H GHYHORSHG6HYHUDOR LW\%&23DVVD\* FXODUL]DWLRQDVVD\& FRQWUDFW WR[LFRORJ\ SURGXFWVDIHW\HYDOX 'UDL]H DOWHUQDWLYH WH SURGXFWODEHOLQJFODL
'XULQJ WKH ODWH YLWUR WR[LFRORJ\ ODER DQLPDO WR[LFLW\ WHVWLQ /DERUDWRU\,975/ WKHQHHGIRUQRQFOLQLF VDIHW\HYDOXDWLRQV2 LQYLWUR WHVWPHWKRGV DFFHSWDQFH7KHÀUVW DVVHVVPHQWRIWKHKX WLRQV&RPSDQ\VFLH +&(7FHOOOLQHDQG 7 PRGHO 7KH +&( PHWKRGVIRUWKHDVVH .DKQDQG:DONHU HWDO2QHRI GDWDRQWKHH\HLUULWDW IRUPDOLQWHUODERUDWRU WHVWPHWKRGDQGWKH WKHWRSLFRI&KDSWHU
7KHYDOLGDWLRQD SURYLGHLQGXVWU\VFLH ZLOO IDFLOLWDWH WKHZL +&(77(3DVVD\Z JRRGZLOOHIIRUWRI WK DUHWUXHIRUFRPSDQL 'UDL]HDOWHUQDWLYHPH .ODXVQHUHWDO
5HSODFLQJ WKH ' SURPRWHWKHFRPSHWL WKHIROORZLQJDGYDQW
KHGLOHPPDRIQHHGLQJWRDGRSWDQLPDOUHSODFHPHQWWHVWPHWKRGV XDWHPHWKRGVDYDLODEOH IRU WKLVSXUSRVH8QIRUWXQDWHO\ HIIRUWV SURSULDWHDOWHUQDWLYHWHVWPHWKRGVDUHSURJUHVVLQJVORZO\UHODWLYH VRILQGXVWU\IRUWKHQHZWHFKQRORJ\7KHGLOHPPDRILQGXVWU\ OLGDWHGDQGUHJXODWRU\DFFHSWHGDQLPDODOWHUQDWLYHWHVWPHWKRGV
RI LQGXVWU\ VFLHQWLVWV IRU PRUH WKDQ D GHFDGH KDV EHHQ WKH LGDWLRQRIQHZDOWHUQDWLYHWHVWVPHWKRGVIRUWKHUHSODFHPHQWRI WHVW$QXPEHURI'UDL]HDOWHUQDWLYH WHVWPHWKRGVKDYHEHHQ IWKHVHPHWKRGVOLNHWKHERYLQHFRUQHDORSDFLW\DQGSHUPHDELO DXWKHURQHWDODQGWKHFKRULRDOODQWRLFPHPEUDQHYDV $09$%DJOH\HWDODUHXVHGE\VRPHLQGXVWULDODQG ODERUDWRULHV DV FRPSRQHQWV RI D WHVWLQJ VFKHPH IRU LQKRXVH DWLRQVRUIRUSURGXFWGHYHORSPHQW,IYDOLGDWHG WKHVHRURWKHU VW PHWKRGV FRXOG DOVR EH XVHG WR JHQHUDWH GDWD WR VXSSRUW PVDQGWRVXSSRUWULVNDVVHVVPHQWVIRUUHJXODWRU\VXEPLVVLRQV V DQG HDUO\ V D QXPEHU RI FRPSDQLHV HVWDEOLVKHG LQ UDWRULHV GHGLFDWHG WRPHHWLQJ WKH FRPSDQ\·V QHHGV IRU QRQ J$VDQH[DPSOH WKH*LOOHWWH ,Q9LWUR7HVWLQJDQG5HVHDUFK ZDVHVWDEOLVKHGLQDVSDUWRIWKHFRPSDQ\·VUHVSRQVHWR DOLQYLWURRUDOWHUQDWLYHWR[LFRORJLFDOWHVWPHWKRGVIRUSURGXFW QHRIWKHPDMRUREMHFWLYHVRIWKH,975/ZDVWRGHYHORSQHZ ZLWK WKHJRDOEHLQJ WKHLUHYHQWXDOYDOLGDWLRQDQG UHJXODWRU\ JRDOZDVWKHGHYHORSPHQWRIQRQFOLQLFDOWHVWPHWKRGVIRUWKH PDQH\HLUULWDWLRQSRWHQWLDORIFKHPLFDOVDQGSURGXFWIRUPXOD QWLVWVGHYHORSHGDQHZKXPDQFRUQHDOHSLWKHOLDOFHOO OLQH WKH XVHGWKHVHFHOOVWRGHYHORSDQLQYLWURFRUQHDOPRGHOWKH+&( 7 PRGHO ZDV XVHG WR GHYHORS VHYHUDO GLIIHUHQW LQ YLWUR WHVW VVPHQWRIWKHH\HLUULWDWLRQSRWHQWLDORIFKHPLFDOVDQGSURGXFWV :DUGHWDO.UXV]HZVNLHWDO&ORWKLHU WKHPHWKRGVWKH+&(77(3DVVD\KDVEHHQXVHGWRSURYLGH LRQSRWHQWLDORISURGXFWVWRVWDIIWR[LFRORJLVWVDQGKDVXQGHUJRQH \SUHYDOLGDWLRQDQGYDOLGDWLRQVWXGLHV:DUGHWDO7KLV UHVXOWVRIWKHSUHYDOLGDWLRQVWXG\IRUWKH+&(77(3DVVD\DUH LQWKLVYROXPH QG UHJXODWRU\DFFHSWDQFHRI WKH+&(77(3 WHVWPHWKRGZLOO QWLVWVZLWKDQDOWHUQDWLYHWRWKH'UDL]HUDEELWH\HWHVW9DOLGDWLRQ GHU DFFHSWDQFH DQG XVH RI WKH WHVWPHWKRG9DOLGDWLRQ RI WKH LOOFRQIHU LQGXVWU\ZLGHUHFRJQLWLRQDQGZLOOGHPRQVWUDWH WKH HFRPSDQ\ LQSURPRWLQJ WKH WKUHH5V7KHVHVDPHSULQFLSOHV HVWKDWDUHVXSSRUWLQJWKHGHYHORSPHQWDQGYDOLGDWLRQRIRWKHU
WKRGVVXFKDV0DW7HN·V(SL2FXODUŒWHVWPHWKRG6WHUQHWDO UDL]H UDEELW WHVW ZLWK D QRQ²DQLPDOEDVHG WHVW VFKHPH ZLOO WLYHDGYDQWDJHRIDQRUJDQL]DWLRQE\SURYLGLQJVRPHRUDOORI DJHVVFLHQWLÀFDOO\VXSHULRUSURGXFWVDIHW\HYDOXDWLRQV³WKH
SRWHQWLDOWRSURYLGHE ULVNDVVHVVPHQWV RSPHQWDQGUHIRUPX ELRORJLFDOUHVSRQVHV WLRQLQDGYHUVHUHVSR OR\DOW\ PDLQWHQ PDUNHWVIRUH[DPSO LPDJHDQGSXEOLFUH
$QXPEHURI LQW WKHLVVXHVRIDOWHUQDWL E\LQGXVWU\WUDGHDVV 7HFKQLFDO&RPPLWWH DWLRQVKDYHVXSSRUWH ODERUDWRU\YDOLGDWLRQ VXSSRUWHGE\ LQGXVW PDUL]HGE\6SLHOPDQ KDYHEHHQWKDWGLIIHU UHVXOWVDQGWKDWQRQH DVUHSODFHPHQWVIRU
7KH ,/6,$$7E DGGUHVVLVVXHVWKDWZR WHVWPHWKRGV7KHLUX WKHRSKWKDOPRORJ\FR PHHWLQJVZLWKSDQHOV LUULWDWLRQWKDWPLJKW LQGXFHG KXPDQ H\H DYDLODELOLW\DQGXWLOLW $$7%OD]NDH FRQGXFWLQJKXPDQWH H[SHUWVWRGHYHORSDK GDWDSURVSHFWLYHO\RU FDOV DQG SURGXFWV % UHFHQWO\FXUWDLOHGPX ,/6,$$7KDVGLVEDQ
7KHDFFHSWHGPH LVWKH'UDL]HUDEELW DJHQFLHV LQGLIIHUHQ JXLGHOLQHVIRUULVND WHVW QLQH UDEELWVZ DQG YDULRXV PRGLÀ 6XEVWDQFHV$FW)+ )HGHUDO ,QVHFWLFLGH
HWWHUVFLHQWLÀFDQGPHFKDQLVWLFGDWDDQGWKHUHIRUHEHWWHUKXPDQ SURGXFWGLIIHUHQWLDWLRQ³VFLHQWLÀFDOO\VXSHULRUSURGXFWGHYHO ODWLRQGXHWRPRUHVHQVLWLYHDQGVSHFLÀFPHDVXUHVRIUHOHYDQW FRVWVDYLQJV³WKHVHPD\EHLQGLUHFWIRUH[DPSOHDUHGXF QVHVGXHWREHWWHUSURGXFWUHIRUPXODWLRQRULQFUHDVHGFXVWRPHU DQFH RI FRPSHWLWLYH SRVLWLRQ LQ SRWHQWLDOO\ UHVWULFWHG IXWXUH HWKH(XURSHDQ8QLRQFRXQWULHVDQGHQKDQFHGFRUSRUDWH ODWLRQV35FDSLWDO HULQGXVWU\FRRSHUDWLYHHIIRUWVKDYHEHHQFRQGXFWHG WRDGGUHVV YHPHWKRGVGHYHORSPHQWDQGYDOLGDWLRQ7KHVHLQFOXGHDFWLYLWLHV RFLDWLRQVDQGE\WKH,QWHUQDWLRQDO/LIH6FLHQFHV,QVWLWXWH,/6, HIRU$OWHUQDWLYHVWR$QLPDO5HVHDUFK$$77KHWUDGHDVVRFL GQXPHURXVZRUNVKRSVDQGYDOLGDWLRQVWXGLHV7KHPDQ\LQWHU VWXGLHV IRUDOWHUQDWLYHV WR WKH'UDL]HH\H WHVW WKDWKDYHEHHQ U\DUH WRRQXPHURXV WRGLVFXVVKHUHEXW WKH\KDYHEHHQVXP Q,QJHQHUDOWKHFRQFOXVLRQVRIWKHVHYDOLGDWLRQVWXGLHV HQWPHWKRGVGLIIHULQWKHLUDELOLW\WRSUHGLFWWKHLQYLYR'UDL]H RIWKHDOWHUQDWLYHPHWKRGVWKDWZHUHHYDOXDWHGFRXOGEHDFFHSWHG WKH'UDL]HWHVWEDVHGRQWKHH[LVWLQJGDWD URXJKW WRJHWKHU LQGXVWU\ DFDGHPLF DQG UHJXODWRU\ VFLHQWLVWV WR XOGSURPRWHWKHGHYHORSPHQWDQGYDOLGDWLRQRI'UDL]HDOWHUQDWLYH QLTXHDFWLYLWLHVKDYHLQFOXGHGWKHIROORZLQJDFWLYLWLHVWRLQIRUP PPXQLW\RILQGXVWU\·VQHHGVIRUQRQFOLQLFDORFXODUWHVWPHWKRGV RIRSKWKDOPRORJ\H[SHUWVWRLGHQWLI\PHFKDQLVPVRIKXPDQH\H EHPRGHOHGE\LQYLWURV\VWHPVWKHGHYHORSPHQWRIDFKHPLFDO LQMXU\ FODVVLÀFDWLRQ VFKHPH DQG D SLORW VWXG\ WR HYDOXDWH WKH \RIUHWURVSHFWLYHKXPDQFKHPLFDOLQGXFHGH\HLQMXU\GDWD,/6, WDO1XVVHQEODWWHWDO7RDOOHYLDWHWKHFRQFHUQVLQ VWLQJWKH,/6,$$7KDVZRUNHGZLWKDSDQHORIRSKWKDOPRORJ\ XPDQH\HLQMXU\FODVVLÀFDWLRQVFKHPHWKDWFDQEHXVHGWRFROOHFW UHWURVSHFWLYHO\IURPDFFLGHQWDOKXPDQH\HH[SRVXUHVWRFKHPL OD]ND HW DO 6LQFH WKH FRQVXPHUSURGXFW LQGXVWU\ KDV FKRILWVIRUPHUVXSSRUWIRUDOWHUQDWLYHPHWKRGVGHYHORSPHQWWKH GHG)XWXUHZRUNLQWKLVDUHDLVXQFHUWDLQ
REGULATORY PERSPECTIVE
WKRGIRUGHWHUPLQLQJWKHH\HLUULWDWLRQSRWHQWLDORIDVXEVWDQFH H\HLUULWDWLRQWHVW'UDL]HHWDO$QXPEHURIUHJXODWRU\ WFRXQWULHV UHTXLUH'UDL]H WHVWGDWDDVDSDUWRI WKHLU WHVWLQJ
VVHVVPHQW:LOOLDPV,/6,$$7,QWKHRULJLQDO HUH XVHG7KH WHVW KDV EHHQPRGLÀHG WR XVH IHZHU DQLPDOV HG YHUVLRQV KDYH EHHQ VSHFLÀHG E\ WKH )HGHUDO +D]DUGRXV 6$ WKH(QYLURQPHQWDO3URWHFWLRQ$JHQF\(3$XQGHUWKH )XQJLFLGH DQG 5RGHQWLFLGH $FW ),)5$ DQG WKH 7R[LF
6XEVWDQFHV&RQWURO DQG'HYHORSPHQW2 LUULWDWLRQWHVWSURWRF KDVSURYLGHG GHYHORSPHQWRIWKH
5HJXODWRU\DJHQF DFXWHH\HLUULWDWLRQD KXPDQKHDOWK7KH) WHFWLRQ $JHQF\ (3$ 'HSDUWPHQWRI7UDQV LVWUDWLRQ 26+$ DQ 'UDL]HH\HWHVWGDWD
$PRQJ UHJXODWR VXEPLVVLRQRI'UDL]H 1XWULWLRQ&)6$1 )HGHUDO)RRG'UXJ DSSURYDOIRUFRVPHWL ´IRUHYDOXDWLQJWKHV ,Q WKHLU HIIRUWVWRGHYHORSDQ )'$·V&HQWHUIRU'UX GUXJV DQG WKH\ GR )'$·V&HQWHUIRU%LR LQWKLVDUHD
7KH(3$2IÀFHR SURFHGXUHVWRPHHWW 86&RGH DQ 86&RGHHWV KD]DUGRXVWRKXPDQ &RVPHWLF$FW WKXV WKH'UDL]HUDEELWH\H H\H LUULWDWLRQ UHYHUV YDOLGDWHG DQG DFFHS LUULWDQWVVXFKWKDWWKH LQYLWUR WHVWV\VWHPV WKLVFULWHULRQ
3ULRUWRUHJXODWRU WR EHFRPH D YDOLGDW 8QLWHG6WDWHVDUHHYD E\ WKH ,&&9$0 7 YDOLGDWLRQDQGUHJXOD 7KHVHJXLGHOLQHVDUH WDQFHRI7R[LFRORJLFD ´WKHSURFHVVZKHUHE\ DSDUWLFXODUSXUSRVHµ ´D VFLHQWLÀF SURFHV
$FW76&$DQGWKH2UJDQL]DWLRQIRU(FRQRPLF&RRSHUDWLRQ (&'$OLVWRIWKHUHJXODWRU\JXLGHOLQHVIRUWKHYDULRXVH\H ROVLVSURYLGHGLQ'L3DVTXDOHDQG+D\HV:LOKHOPXV DQH[FHOOHQWKLVWRULFDO UHYLHZRIRFXODU WR[LFRORJ\DQG WKH 'UDL]HWHVW LHVQHHGGDWD WRHYDOXDWH WKHSRWHQWLDO IRUDSURGXFW WRFDXVH VRQHRIWKHFRPSRQHQWVLQWKHDVVHVVPHQWRILWVRYHUDOOULVNWR RRGDQG'UXJ$GPLQLVWUDWLRQ)'$ WKH(QYLURQPHQWDO3UR WKH &RQVXPHU 3URGXFW 6DIHW\ &RPPLVVLRQ &36& WKH SRUWDWLRQ'27WKH2FFXSDWLRQDO6DIHW\DQG+HDOWK$GPLQ G RWKHU UHJXODWRU\ DJHQFLHVZRUOGZLGH DFFHSW DQGRU UHTXLUH IRUULVNDVVHVVPHQWDQGIRUODEHOLQJSXUSRVHV U\ DJHQFLHV WKH )'$ KDV WDNHQ WKH OHDG LQ HQFRXUDJLQJ WKH DOWHUQDWLYHWHVWGDWD7KH&HQWHUIRU)RRG6DIHW\DQG$SSOLHG LVWKH)'$GLYLVLRQLQYROYHGZLWKWKHVDIHW\RIFRVPHWLFVWKH DQG&RVPHWLF$FW$OWKRXJK WKH\GRQRW UHTXLUHSUHPDUNHW FVWKH)'$&)6$1KDVUHOLHGRQWKH'UDL]HH\HLUULWDWLRQWHVW DIHW\RIDVXEVWDQFHLQWURGXFHGLQWRRUDURXQGWKHH\Hµ)'$ SRVLWLRQSDSHURQDQLPDOXVH WKH)'$VDLG WKH\HQFRXUDJH GLPSOHPHQWQRQZKROHDQLPDOWHFKQLTXHV)'$7KH J(YDOXDWLRQDQG5HVHDUFK&'(5UHJXODWHVRYHUWKHFRXQWHU QRW XVH WKH 'UDL]H WHVW DQG DFWLYHO\ GLVFRXUDJH LWV XVH 7KH ORJLFV(YDOXDWLRQDQG5HVHDUFK&%(5IROORZV&'(5·VOHDG
I3UHYHQWLRQ3HVWLFLGHVDQG7R[LF6XEVWDQFHVUHYLHZVWHVWLQJ KHGDWDUHTXLUHPHQWVRIWKH7R[LF6XEVWDQFHV&RQWURO$FW G WKH)HGHUDO ,QVHFWLFLGH)XQJLFLGH DQG5RGHQWLFLGH$FW HT7KHVXEVWDQFHVUHJXODWHGE\WKH(3$DUHW\SLFDOO\PRUH KHDOWKWKDQWKRVHUHJXODWHGXQGHUWKH)HGHUDO)RRG'UXJDQG SUHPDUNHW UHYLHZ LV UHTXLUHG7KH JXLGHOLQHV LQGLFDWH XVH RI WHVWWRHYDOXDWHH\HFRUURVLRQLUUHYHUVLEOHWLVVXHGDPDJHDQG LEOH H\H LQMXU\7KHJXLGHOLQHV VWDWH WKDW ´UHVXOWV IURPZHOO WHG LQ YLWUR WHVW V\VWHPVPD\ VHUYH WR LGHQWLI\ FRUURVLYHV RU WHVWPDWHULDOQHHGQRWEHWHVWHGLQYLYRµ(3$+RZHYHU WKDWDUHZHOOYDOLGDWHGDQGDFFHSWHGGRQRW\HWH[LVWWRPHHW
\DJHQF\DFFHSWDQFHDQHZ'UDL]HDOWHUQDWLYHPHWKRGZLOOKDYH HGPHWKRG 1HZ WR[LFRORJLFDO WHVWPHWKRGV GHYHORSHG LQ WKH OXDWHGDQGUHFRPPHQGHGWRDSHHUUHYLHZSDQHOIRUYDOLGDWLRQ KH ,&&9$0 KDV GHÀQHG WKH FULWHULD DQG SURFHVVHV IRU WKH WRU\DFFHSWDQFHRIQHZDQGUHYLVHGWR[LFRORJLFDOWHVWPHWKRGV
GHVFULEHGLQWKHUHSRUWWLWOHG9DOLGDWLRQDQG5HJXODWRU\$FFHS O7HVW0HWKRGV1,(+69DOLGDWLRQKDVEHHQGHÀQHGDV WKHUHOLDELOLW\DQGUHOHYDQFHRIDSURFHGXUHDUHHVWDEOLVKHGIRU %DOOVHWDO9DOLGDWLRQLVGHÀQHGE\WKH,&&9$0DV V GHVLJQHG WR FKDUDFWHUL]H WKH RSHUDWLRQDO FKDUDFWHULVWLFV
DGYDQWDJHVDQGOLP UHOHYDQFHµ1,(+6 ZKLFKDWHVWPHWKRG DQG GHVFULEHV ZKHWK 1,(+6 5H SHUIRUPHGUHSURGXFL
5HJXODWRU\DJHQ UHJXODWRU\XVH5HJX DQG ,&&9$0ZLOO S WKH DSSURSULDWH DJHQ PHWKRGLVOHIWXSWRW JXLGHOLQHV UHTXLUH WK VXEVWLWXWH UHSODFHP VXEPLVVLRQGRFXPHQ WLYHWHVWPHWKRGZLOO ZLOOKDYHWRGHWHUPL RIGDWDLWUHTXLUHVIUR WRDJHQFLHVLQVHYHUD WKHQHZWHVWPHWKRG QHZ WHVW PHWKRG GD VFKHPH DQGRU KLVWR ´SUHSRQGHUDQFHRIH DVDVXSSOHPHQWWR'
$QLVVXHRIVSHF 'UDL]H UDEELW H\H G GHYHORSHGWRSUHGLFW &KDPEHUV7 VWDWHVWKDW´GDWDJHQ HQGSRLQWRILQWHUHVW H[LVWLQJWHVWRUWKH UHIHUHQFHPHWKRGGD PHWKRGVKDVEHHQWK DVWKHWHVWVSHFLHVLV LVDYDLODEOHIRUWKHUD +RZHYHUWKH'UDL]H WKH VSHFLHVELRORJLF 'UDL]H UDEELW GDWD *HUVKEHLQ DQG0F' PDQ\W\SHVRILUULWDW :DONHU2QH WKHPRQNH\H\HZK FRQFOXGHGWKDWWKHUD KXPDQRFXODUUHVSR
0DQ\VFLHQWLVWVV RSPHQWDQGYDOLGDWLR WHVWDVWKH´JROGVWDQ
LWDWLRQVRIDWHVWPHWKRGDQGWRGHPRQVWUDWHLWVUHOLDELOLW\DQG 7KHUHOHYDQFHRID WHVWPHWKRGGHVFULEHV WKHH[WHQW WR ZLOOFRUUHFWO\SUHGLFWRUPHDVXUHWKHELRORJLFDOHIIHFWRILQWHUHVW HU WKH WHVW LVPHDQLQJIXO DQG XVHIXO IRU D SDUWLFXODU SXUSRVH OLDELOLW\ LV D PHDVXUH RI WKH GHJUHH WR ZKLFK D WHVW FDQ EH EO\ERWKZLWKLQDQGEHWZHHQODERUDWRULHV1,(+6 F\DFFHSWDQFH LV WKHDFFHSWDQFHRIDYDOLGDWHG WHVWPHWKRGIRU ODWRU\DFFHSWDQFHFULWHULDKDYHEHHQGHÀQHGE\WKH,&&9$0 URYLGH UHFRPPHQGDWLRQV UHJDUGLQJ D UHYLHZHG WHVWPHWKRG WR FLHV KRZHYHU WKH SURFHVV IRU WKH DFFHSWDQFH RI D QHZ WHVW KHLQGLYLGXDODJHQF\1,(+67KH,&&9$0VXEPLVVLRQ DW WKH ´LQWHQGHG UHJXODWRU\ XVHV DQG UDWLRQDOH HJ VFUHHQ HQW RU DGMXQFWµ IRU D QHZ WHVW PHWKRG EH GHVFULEHG LQ WKH W,&&9$07KHW\SHVRIGDWDSURYLGHGE\DQDOWHUQD EHFRPHHYLGHQWGXULQJWKHYDOLGDWLRQSURFHVVDQGHDFKDJHQF\ QHZKHWKHU LWZLOODFFHSWDQDOWHUQDWLYHPHWKRGDQGZKDW W\SH PWKHPHWKRG$OWHUQDWLYHWHVWPHWKRGGDWDFRXOGEHVXEPLWWHG OFRQWH[WVLQFOXGLQJVWDQGDORQHQHZWHVWPHWKRGGDWDLH KDVEHHQYDOLGDWHGDQGDFFHSWHGIRUWKDWSDUWLFXODUSXUSRVH WD WKDW LV VXSSOHPHQWHG E\ DGGLWLRQDO LQIRUPDWLRQ LH D WLHU ULFDO 'UDL]H GDWD RQ VLPLODU SURGXFWV WKDW LV VXEPLWWHG DV D YLGHQFHµDSSURDFKRUQHZWHVWPHWKRGGDWDWKDWLVVXEPLWWHG UDL]HWHVWGDWDWKDWZLOOVHUYHWRUHGXFHWKHQXPEHURIDQLPDOV LDOFRQFHUQ WRPDQ\UHJXODWRU\DJHQF\VFLHQWLVWV LV WKHXVHRI DWD WR HYDOXDWH WKH SHUIRUPDQFH RI QHZ WHVW PHWKRGV WKDW DUH WKHKXPDQUHVSRQVHWRH\HLUULWDQWV1XVVHQEODWWHWDO KH,&&9$0UHJXODWRU\DFFHSWDQFHFULWHULDUHODWHGWRWKLVLVVXH HUDWHGE\WKHPHWKRGVKRXOGDGHTXDWHO\PHDVXUHRUSUHGLFWWKH DQGGHPRQVWUDWHDOLQNDJHEHWZHHQHLWKHUWKHQHZWHVWDQGDQ QHZWHVWDQGHIIHFWVLQWKHWDUJHWVSHFLHVµ1,(+67KH WDXVHGWRHYDOXDWHWKHSHUIRUPDQFHRIQHZ'UDL]HDOWHUQDWLYH H'UDL]HUDEELWH\HLUULWDWLRQWHVW7KHFRQWLQXHGXVHRIWKHUDEELW GXHLQSDUW WRWKHODUJHGDWDEDVHRIRFXODUUHVSRQVHGDWDWKDW EELWWKXVWR[LFRORJLVWVIHHOFRPIRUWDEOHLQWKHXVHRIWKHPHWKRG WHVWKDVEHHQFLWHGDVDSRRUWHVWIRUPDQ\UHDVRQVLQFOXGLQJ DO GLIIHUHQFHV DQG WKH ODUJH FRHIÀFLHQW RI YDULDWLRQV VHHQ LQ %HFNOH\ %HFNOH\ HW DO :HLO DQG 6FDOD RQDOG ,/6,$$77KH UDEELW RFXODU UHVSRQVH WR LQJPDWHULDOVIUHTXHQWO\RYHUSUHGLFWVWKHKXPDQRFXODUUHVSRQVH ODUJHVWXG\ WKDWFRPSDUHGRFXODU UHVSRQVHV LQ WKH UDEELWDQG LFKLVFRQVLGHUHGWREHDQH[FHOOHQWPRGHOIRUWKHKXPDQH\H
EELWH\HZDVDSRRUSUHGLFWRURIWKHPRQNH\DQGWKHUHIRUHWKH QVH*UHHQHWDO XSSRUWWKHDOWHUQDWLYHXVHRIKXPDQGDWDIRUWHVWPHWKRGGHYHO QDQGFODLPWKDWLWVKRXOGUHSODFHWKH'UDL]HUDEELWH\HLUULWDWLRQ GDUGµ1XVVHQEODWWHWDO&KDPEHUV%OD]NDHWDO
+RZHYHUVXIÀ IRU QHZ WHVW PHWKRG UHWURVSHFWLYH RU SURV FRQFHUQDERXWLWVSUR KXPDQH\HWHVWLQJLV WLRQDO WHVWLQJFDQEH GHVFULEHG D SODQ IR H[SHFWHG WR FDXVH L SURGXFWV WKDWFRXOG RSLQLRQWKDWVXIÀFLHQ RIGDWDQHHGHGWRYDO DQGWKHHWKLFDOLVVXHV WREHDGGUHVVHGEHIR
7KHPDLQJRDOR %DVHGRQ0HFKDQLVP XODWRU\ 1HHGVµ ZDV GHPRQVWUDWHGUHOHYDQ YDOLGDWLRQ VWXGLHV7 KDYHVRPHFRUUHODWLR DPHFKDQLVWLFEDVLV SUHVHQWWLPHRQO\WZ ODUŒ DVVD\ DUH EHLQ %RWKRIWKHVHPHWKR
7KH ,&&9$0Z DQGWKHQDVD $FW 3XEOLF /DZ ,&&9$0 KDG DOUHD &RUURVLWH[Š DVVD\ IR WKHPXULQHORFDO O\P PDWLWLVSRWHQWLDORIF VWDJHVRIHYDOXDWLRQ PDNLQJ H[FHOOHQW SU VFLHQWLVWVIRUQHZYD
7KH,&&9$0YD QDWLRQDO HIIRUWV RI WK PHWKRGYDOLGDWLRQFU YDOLGDWHGDQDOWHUQDW KDYHEHHQSURSRVHGW PHWKRGWRWKH'UDL]H FRQGXFWHG EHIRUH WK PDQ\FDVHVWKHPHFK NQRZQ RUZHUH QRW
FLHQWKXPDQH\HLUULWDWLRQGDWDLQWKHTXDQWLWDWLYHIRUPQHHGHG HYDOXDWLRQ GRHV QRW H[LVW 7KHUH LV WKH SRWHQWLDO IRU XVLQJ SHFWLYH DFFLGHQWDO KXPDQ H\H LQMXU\ GDWD EXW WKHUH LV DOVR D EDEOHODFNRIVXIÀFLHQWLQIRUPDWLRQ6RPHVFLHQWLVWVWKLQNWKDW DGHTXDWHO\UHJXODWHGXQGHUFXUUHQWJXLGHOLQHVDQGWKDWDGGL FRQGXFWHG WRGHYHORS WKHGDWDQHHGHG&KDPEHUV KDV U SURVSHFWLYHO\ FROOHFWLQJ KXPDQ H\H GDWD IRU SURGXFWV QRW QMXU\ DQG IRU FROOHFWLQJ DFFLGHQWDO H[SRVXUH LQIRUPDWLRQ IRU EHH[SHFWHG WR UHVXOW LQH\H LQMXU\2WKHU VFLHQWLVWVDUHRI WKH WKXPDQH\HWHVWLQJFRXOGQRWEHSHUIRUPHGWRREWDLQWKHUDQJH LGDWHDQHZPHWKRG7KHUHDUHGLFKRWRPLHVLQERWKWKHVFLHQWLÀF DVVRFLDWHGZLWKWKHFROOHFWLRQDQGXVHRIKXPDQGDWDWKDWQHHG UHWKLVW\SHRIGDWDZLOOEHDYDLODEOH
DISCUSSION AND CONCLUSIONS
IWKHV\PSRVLXPHQWLWOHG´'HYHORSPHQWRI3UHGLFWLYH0HWKRGV VRI(\H,UULWDWLRQDWWKH2FXODU6XUIDFH0HHWLQJ,QGXVWU\5HJ WR SUHVHQW DOWHUQDWLYH H\H LUULWDWLRQ WHVW PHWKRGV WKDW KDYH D FHDQGWKHUHIRUHFRXOGEHFRQVLGHUHGDVFDQGLGDWHVIRUIXWXUH KHPHWKRGV VHOHFWHG IRUSUHVHQWDWLRQKDYH SURGXFHGGDWD WKDW QWRWKHUHIHUHQFHPHWKRGWKH'UDL]HUDEELWH\HWHVWDQGKDYH IRU WKHLUXVHDVD WHVWPHWKRGIRUKXPDQH\H LUULWDWLRQ$W WKH RRIWKHWHVWPHWKRGVWKH+&(77(3DVVD\DQGWKH(SL2FX J FRQVLGHUHG IRU VXEPLVVLRQ WR WKH ,&&9$0 IRU YDOLGDWLRQ GVDUHGLVFXVVHGLQGLIIHUHQWFKDSWHUVRIWKHVHSURFHHGLQJV DV RQO\ UHFHQWO\ HVWDEOLVKHG ÀUVW DV DQ DG KRF FRPPLWWHH LQ SHUPDQHQWFRPPLWWHH LQE\WKH,&&9$0$XWKRUL]DWLRQ $W WKH WLPH RI WKLV V\PSRVLXP 1RYHPEHU G\ IDFLOLWDWHG WKH YDOLGDWLRQ RI WZR DOWHUQDWLYH PHWKRGV WKH U DVVHVVLQJ WKH GHUPDO FRUURVLYLW\ SRWHQWLDO RI FKHPLFDOV DQG SKQRGHDVVD\//1$IRUDVVHVVLQJWKHDOOHUJLFFRQWDFWGHU KHPLFDOVDQGLWKDGDERXWVL[DGGLWLRQDOPHWKRGVLQGLIIHUHQW ,&&9$01,&($70 ,&&9$01,&($70KDVEHHQ RJUHVV WRZDUG PHHWLQJ WKH QHHGV RI LQGXVWU\ DQG UHJXODWRU\ OLGDWHGWR[LFRORJLFDOWHVWPHWKRGV OLGDWLRQFULWHULDDQGSURFHVVHVHYROYHGIURPWKHRQJRLQJLQWHU H (&9$0 DQG WKH2(&' DQG RWKHU JURXSV LQ GHÀQLQJ WHVW LWHULD+RZHYHUDW WKLV WLPHQRQHRI WKHVHRUJDQL]DWLRQVKDYH LYHPHWKRGWRWKH'UDL]HUDEELWH\HWHVW$QXPEHURIUHDVRQV
RH[SODLQWKHIDLOXUHRISUHYLRXVVWXGLHVWRYDOLGDWHDQDOWHUQDWLYH WHVW3XUFKDVHHWDO7KHHDUO\YDOLGDWLRQVWXGLHVZHUH H FXUUHQW FRQFHSWV RI YDOLGDWLRQ FULWHULD ZHUH HVWDEOLVKHG ,Q DQLVWLFEDVLVRIWKHWHVWPHWKRGVWKDWZHUHHYDOXDWHGZHUHQRW GHÀQHG 3UHYLRXV YDOLGDWLRQ VWXGLHVZHUH VHW XS WR YDOLGDWH D
UHSODFHPHQWWHVWIRU FULWHULDDQGSURFHVVH
'UDL]HDOWHUQDWLY WHVWPHWKRG YDOLGDWL VFLHQWLVWV WKLQN WKDW UHSODFHWKH'UDL]HWH 'UDL]HWHVWDUHQRWZ WHVWV DV VFUHHQV XQW $QRWKHUDSSURDFKLV DEDWWHU\ZLWKWKHOLP VSHFLÀFSXUSRVHIRUZ JXLGHOLQHV7KLVPHD UDQJHRILQYLYRLUULW LPSRUWDQFHRIWKLVF WKXVIDUDV'UDL]HDO
$QRWKHUXQLTXHF WKHLQYLYRWHVWHYDOXD WLVVXHV WKH FRUQHD VSHFLÀFSDUWRIWKH' WKHLU ULVN DVVHVVPHQ FRQFOXVLRQ WKDW WKH VXIÀFLHQWWRPRGHOD WLVVXHVFRUHVXVLQJS IURPWKHGLIIHUHQWRF LQIRUPDWLRQLVPLVVH RIWKHGDWDREWDLQHG DOVRVXSSRUWWKLVFRQ WLVVXHVFRUHVIRUWKHF ZLWK WKH HQGSRLQW R 'UDL]HPD[LPXPDY
7KH,&&9$0YD XVHIXOIRUULVNDVVHVV 8QGHUWKH,& PHWKRGEXWWKHFULWH DWLHUWHVWLQJVFKHPH ,&&9$0KDVHVWDEO OLVKHVWHVWPHWKRGSH
6FLHQWLVWV LQYROYH ORRNDW WKHXQLTXHF KDYHEHHQGLVFXVVHG VFLHQWLVWVDQGRWKHU,& IRUHDFKPHWKRGWKDW WKLVSURFHVVDQGPXV DQGUHJXODWRU\DJHQF WHVWLQJ7KLVZLOOHQVX VXVWDLQWKHLUFRPSHWL
WKH'UDL]HH\HWHVWDQGDUHFRQVLGHUHGWREHUDWKHUULJLGLQWKH VWKDWZHUHXVHG HPHWKRGVKDYHWKHLURZQXQLTXHFKDOOHQJHVZKHQLWFRPHVWR RQ2QO\ D IHZ RI WKHVH LVVXHVZLOO EH GLVFXVVHG KHUH0DQ\ WHVW EDWWHULHV UDWKHU WKDQ RQH WHVW PHWKRG ZLOO EH QHHGHG WR VW7KHHVVHQWLDOFRPSRQHQWVIRUDUHSODFHPHQWVFKHPHIRUWKH HOOGHÀQHGDWWKLVWLPH2QHDSSURDFKLVWRYDOLGDWHLQGLYLGXDO LO WKH\ FDQ EH DVVHPEOHG LQWR PRUH SUHGLFWLYH WHVW EDWWHULHV WRYDOLGDWHLQGLYLGXDOPHWKRGVWKDWDUHH[SHFWHGWREHSDUWRI LWDWLRQVFOHDUO\GHÀQHGDQGWKHPHWKRGYDOLGDWHGIRURQO\WKH KLFKLWLVGHÀQHG7KLVSURFHVVLVFRPSDWLEOHZLWKWKH,&&9$0 QVWKDWD'UDL]HDOWHUQDWLYHWHVWFRXOGEHYDOLGDWHGIRUDOLPLWHG DWLRQFKHPLFDOFODVVRUSK\VLFDOIRUPRIWKHWHVWPDWHULDO7KH RQFHSWFDQQRWEHRYHUVWDWHG$OOQRQDQLPDOPHWKRGVHYDOXDWHG WHUQDWLYHVKDYHKDGDWOHDVWRQHRIWKHVH OLPLWDWLRQV KDOOHQJHIRUWKHYDOLGDWLRQRI'UDL]HDOWHUQDWLYHPHWKRGVLVWKDW WHVWKHELRORJLFDOUHVSRQVHWRDPDWHULDOLQWKUHHGLIIHUHQWRFXODU WKH LULV DQG WKH FRQMXQFWLYD0DQ\ UHJXODWRU\ DJHQFLHV XVH D UDL]HWHVWGDWDIUHTXHQWO\WKHFRUQHDORSDFLW\VFRUHLQPDNLQJ WV %RWK VWDWLVWLFDO DQG HPSLULFDOO\ GHULYHG GDWD VXSSRUW WKH UHVXOWV IURP D UHOHYDQW FRUQHDO WHVW PHWKRG DORQH VKRXOG EH OORIWKHRFXODUVXUIDFHUHVSRQVHV/RYHOODQDO\]HG'UDL]H ULQFLSDOFRPSRQHQWDQDO\VLVDQGVKRZHGWKDWWKH'UDL]HVFRUHV XODUWLVVXHVDUHVWDWLVWLFDOO\UHODWHGWRRQHDQRWKHUDQGWKDWOLWWOH GLQXVLQJRQO\WKHWRWDO'UDL]HVFRUH0$6RU00$6$QDO\VLV E\WHVWLQJVXUIDFWDQWIRUPXODWLRQVLQWKH+&(77(3DVVD\ FOXVLRQ.UXV]HZVNLHWDO6:DUG7KH'UDL]H RQMXQFWLYDODQGWKHLULVUHVSRQVHVIRUWKHPDWHULDOVFRUUHODWHG I WKH+&(77(3 DVVD\ DVZHOO DV WKH FRUQHDO VFRUH DQG WKH HUDJHVFRUH0$6 OLGDWLRQFULWHULDUHTXLUHWKDWDWHVWPHWKRGSURYLGHGDWDWKDWDUH PHQWDQGQRWQHFHVVDULO\UHSODFHDQH[LVWLQJWHVWPHWKRG1,(+6 &9$0FULWHULDDWHVWPHWKRGFDQEHYDOLGDWHGDVDUHSODFHPHQW ULDDOVRSURYLGHIRUYDOLGDWLRQRIPHWKRGVWKDWZRXOGEHSDUWRI SDUWRIDWHVWEDWWHU\RUXVHGDVDQDGMXQFWRUVFUHHQLQJDVVD\ LVKHGFULWHULD WKDWDUHÁH[LEOHDQG WKHYDOLGDWLRQSURFHVVHVWDE UIRUPDQFHDQGHIIHFWLYHRXWFRPH1,(+6 G LQ WKH YDOLGDWLRQ SURFHVV RI'UDL]H DOWHUQDWLYHPHWKRGVPXVW KDOOHQJHV WRUHSODFLQJ WKH'UDL]HUDEELWH\H WHVWVRPHRIZKLFK LQWKLVFKDSWHU7KHWHVWPHWKRGGHYHORSHUVWKHUHJXODWRU\DJHQF\ &9$0VWDNHKROGHUVQHHGWRZRUNWRJHWKHUWRDGGUHVVWKHVHLVVXHV
LVHYDOXDWHG7KH,&&9$0YDOLGDWLRQFULWHULDDUHFRPSDWLEOHZLWK WEHLQWHUSUHWHGZLWKWKHVHSULQFLSOHVLQPLQGVRWKDWERWKLQGXVWU\ LHVFDQREWDLQWKHQHZWHFKQRORJLHVWKH\QHHGWRUHSODFHDQLPDO UHWKDWEHWWHUVFLHQFHLVXVHGLQULVNDVVHVVPHQWVDQGWKDWLQGXVWULHV WLYHHGJHIRUWKHGHYHORSPHQWRIVDIHQHZSURGXFWV
REFERENCES
