D G M : ISTINGUISHING OALS AND EANS T C P -R HE ASE OF RE EGISTRATION Alison  Ledgerwood   UC  Davis Goals to Means: Specific improve our research science practices Distinguish high vs. low confidence findings WHAT  MAKES  US  CONFIDENT  IN   OUR  OWN  RESULTS?   •  Given  what  we  now  know  about  power,   p-­‐hacking,  sequenKal  analyses,  etc.   •  When  do  we  trust  our  own  findings  a  lot  vs.   more  tentaKvely?   •  High  confidence  vs.  low  confidence  findings   •  High  confidence:  I’d  bet  money  I  could  replicate  it   •  Lower  confidence:  Let’s  replicate  before  beSng DEVELOPING  LAB  BEST  PRACTICES:   THE  PROCESS   1.  Learn  about  and  criKcally  evaluate  the   suggesKons  (lab  meeKngs/journal  clubs)   2.  Develop  a  set  of  shared  (and  sKll  evolving)   best  pracKces  for  our  lab   •  What  maximizes  the  info  we  get  from  our   work?   •  What  are  our  resource  limitaKons?   •  What’s  comfortable  and  feasible  right  now? LEDGERWOOD  LAB  BEST  PRACTICES:   SAMPLE  SIZES   •  Set  target  n’s  ahead  of  Kme   •  Power  analysis  when  possible   •  Based  on  previous  findings,  meta-­‐analysis,  etc.   •  Consider  power  analyses  that  take  into   account  effect  size  heterogeneity  and   publicaKon  bias   (see  McShane  &  Bockenholt,  2014  PPS;  Perugini  et  al.,  2014  PPS)   •  Otherwise,  minimum  target  cell  size  n  =  50 WHY  N  =  50?   •  Completely  arbitrary   •  But,  beher  than  n  =  20   •  Power  to  detect  a  small  effect:  Less  than  10%   •  When  power  dips  below  10%,  you’re  not  only  likely   to  overesKmate  the  effect  size…you  may  get  the   sign  wrong  (Gelman  &  Carlin,  2014)   •  Power  to  detect  a  medium  effect:  34%   •  When  power  gets  below  50%,  start  having  effect   size  exaggeraKon  problems  (i.e.,  results  are  less   informaKve).   See Gelman & Carlin, 2014 PPS for more on Type S and Type M errors WHY  N  =  50?   •  n  =  50   •  Power  to  detect  a  small  effect:  17%   (But  at  least  avoid  sign  problems)   •  Power  to  detect  a  medium  effect:  70%   (Avoid  exaggeraKon  problems)   See Gelman & Carlin, 2014 PPS for more on Type S and Type M errors DEVELOPING  LAB  BEST  PRACTICES:   THE  PROCESS   1.  Lab  meeKngs/discussions  about  arKcles  and   conference  presentaKons   2.  Develop  a  set  of  shared  (and  sKll  evolving)   best  pracKces  for  our  lab   •  What  maximizes  the  info  we  get  from  our   work?   •  What  are  our  resource  limitaKons?   •  What’s  comfortable  and  feasible  right  now?   3.  What  infrastructure  will  make  this  easy? EXPERIMENT  ARCHIVE  FORM   Available at http://ledgerwood.faculty.ucdavis.edu/resources