SMALL SAMPLE SIZE SOLUTIONS Researchers often have difficulties collecting enough data to test their hypotheses, either because target groups are small or hard to access, or because data collection entails prohibitive costs. Such obstacles may result in data sets that are too small for the complexity of the statistical model needed to answer the research question. This unique book provides guidelines and tools for implementing solutions to issues that arise in small sample research. Each chapter illustrates statistical methods that allow researchers to apply the optimal statistical model for their research question when the sample is too small. This essential book will enable social and behavioral science researchers to test their hypotheses even when the statistical model required for answering their research question is too complex for the sample sizes they can collect. The statistical models in the book range from the estimation of a population mean to models with latent variables and nested observations, and solutions include both classical and Bayesian methods. All proposed solutions are described in steps researchers can implement with their own data and are accompanied with annotated syntax in R. The methods described in this book will be useful for researchers across the social andbehavioralsciences,rangingfrommedicalsciencesandepidemiologytopsychology, marketing,andeconomics. Prof. Dr. Rens van de Schoot works as a Full Professor teaching Statistics for Small Data Sets at Utrecht University in the Netherlands and as an Extra-ordinary Professor at North-West University in South Africa. He obtained his PhD cum laude onthetopicofapplyingBayesianstatisticstoempiricaldata. Dr. Milica Miočević is an Assistant Professor in the Department of Psychology at McGill University. She received her PhD in Quantitative Psychology from Arizona StateUniversityin2017.Dr.Miočević’sresearchevaluatesoptimalwaystouseBayes- ianmethodsinthesocialsciences,particularlyformediationanalysis. EUROPEAN ASSOCIATION OF METHODOLOGY TheEuropeanAssociationofMethodology (EAM)servestopromoteresearchanddevelop- mentofempiricalresearchmethodsinthe fieldsoftheBehavioural,Social,Educational, HealthandEconomicSciencesaswellasinthe fieldofEvaluationResearch. Homepage:www.eam-online.org. The purpose of the EAM book series is to advance the development and appli- cation of methodological and statistical research techniques in social and behav- ioral research. Each volume in the series presents cutting-edge methodological developments in a way that is accessible to a broad audience. Such books can be authored, monographs, or edited volumes. SponsoredbytheEuropeanAssociationofMethodology,theEAMbookseriesis opentocontributionsfromtheBehavioral,Social,Educational,HealthandEconomic Sciences. Proposals for volumes in the EAM series should include the following: (1) title; (2) authors/editors; (3) a brief description of the volume’s focus and intended audience; (4) a table of contents; (5) a timeline including planned completion date. Proposalsareinvitedfromallinterestedauthors.Feelfreetosubmitaproposaltoone ofthemembersoftheEAMbookserieseditorialboardbyvisitingtheEAMwebsite http://eam-online.org. Members of the EAM editorial board are Manuel Ato (Uni- versity of Murcia), Pamela Campanelli (Survey Consultant, UK), Edith de Leeuw (UtrechtUniversity)andVasjaVehovar(UniversityofLjubljana). Volumes in the series include Van de Schoot/Miočević: Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners, 2020 Davidov/Schmidt/Billiet/Meuleman: Cross-Cultural Analysis: Methods and Applications, 2nd edition, 2018 Engel/Jann/Lynn/Scherpenzeel/Sturgis: Improving Survey Methods: Lessons from Recent Research, 2015 Das/Ester/Kaczmirek: Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies, 2011 Hox/Roberts: Handbook of Advanced Multilevel Analysis, 2011 De Leeuw/Hox/Dillman: International Handbook of Survey Methodology, 2008 Van Montfort/Oud/Satorra: Longitudinal Models in the Behavioral and Related Sciences, 2007 SMALL SAMPLE SIZE SOLUTIONS A Guide for Applied Researchers and Practitioners Edited by Rens van de Schoot and ̌ Milica Miocevic´ Firstpublished2020 byRoutledge 2ParkSquare,MiltonPark,Abingdon,Oxon,OX144RN 52VanderbiltAvenue,NewYork,NY10017 andbyRoutledge RoutledgeisanimprintoftheTaylor&FrancisGroup,aninformabusiness ©2020selectionandeditorialmatter,RensvandeSchootandMilicaMiočević; individualchapters,thecontributors TherightofRensvandeSchootandMilicaMiočevićtobeidentified astheauthorsoftheeditorialmaterial,andoftheauthorsfortheir individualchapters,hasbeenassertedinaccordancewithsections77and 78oftheCopyright,DesignsandPatentsAct1988. TheOpenAccessversionofthisbook,availableatwww.taylorfrancis.com, hasbeenmadeavailableunderaCreativeCommonsAttribution-Non Commercial-NoDerivatives4.0license. Trademarknotice:Productorcorporatenamesmaybetrademarksor registeredtrademarks,andareusedonlyforidentificationand explanationwithoutintenttoinfringe. LibraryofCongressCataloging-in-PublicationData Acatalogrecordforthistitlehasbeenrequested ISBN:978-0-367-22189-8(hbk) ISBN:978-0-367-22222-2(pbk) ISBN:978-0-429-27387-2(ebk) TypesetinBembo bySwales&Willis,Exeter,Devon,UK CONTENTS Introduction viii RensvandeSchootandMilicaMiočević Listofsymbols xi PARTI Bayesiansolutions 1 1 IntroductiontoBayesianstatistics 3 MilicaMiočević,RoyLevy,andRensvandeSchoot 2 Theroleofexchangeabilityinsequentialupdatingoffindings fromsmallstudiesandthechallengesofidentifyingexchangeable datasets 13 MilicaMiočević,RoyLevy,andAndreaSavord 3 AtutorialonusingtheWAMBSchecklisttoavoidthe misuseofBayesianstatistics 30 RensvandeSchoot,DucoVeen,LaurentSmeets,SonjaD.Winter, andSarahDepaoli 4 TheimportanceofcollaborationinBayesiananalyses withsmallsamples 50 DucoVeenandMartheEgberts vi Contents 5 AtutorialonBayesianpenalizedregressionwithshrinkagepriors forsmallsamplesizes 71 SaravanErp PARTII n=1 85 6 Onebyone:thedesignandanalysisofreplicatedrandomized single-caseexperiments 87 PatrickOnghena 7 Single-caseexperimentaldesignsinclinicalintervention research 102 MarijaMaricandVeravanderWerff 8 Howtoimprovetheestimationofaspecificexaminee’s (n ¼ 1)mathabilitywhentestdataarelimited 112 KimberleyLekandIngridArts 9 Combiningevidenceovermultipleindividualanalyses 126 FayetteKlaassen 10 Goingmultivariateinclinicaltrialstudies:aBayesian frameworkformultiplebinaryoutcomes 139 XynthiaKavelaars PARTIII Complexhypothesesandmodels 155 11 Anintroductiontorestriktor:evaluatinginformativehypotheses forlinearmodels 157 LeonardVanbrabantandYvesRosseel 12 Testingreplicationwithsmallsamples:applicationstoANOVA 173 MariëlleZondervan-ZwijnenburgandDominiqueRijshouwer 13 Smallsamplemeta-analyses:exploringheterogeneityusing MetaForest 186 CasparJ.vanLissa Contents vii 14 Itemparcelsasindicators:why,when,andhowtousethemin smallsampleresearch 203 CharlieRioux,ZacharyL.Stickley,OmololaA.Odejimi, andToddD.Little 15 Smallsamplesinmultilevelmodeling 215 JoopHoxandDanielMcNeish 16 Smallsamplesolutionsforstructuralequationmodeling 226 YvesRosseel 17 SEMwithsmallsamples:two-stepmodelingandfactorscore regressionversusBayesianestimationwithinformativepriors 239 SanneC.SmidandYvesRosseel 18 Importantyetunheeded:somesmallsampleissuesthatare oftenoverlooked 255 JoopHox Index 266 INTRODUCTION ̌ Rens van de Schoot and Milica Miocevic´ Researchers often have difficulties collecting enough data to test their hypotheses, either because target groups are small (e.g., patients with severe burninjuries); data are sparse (e.g., rare diseases), hard to access (e.g., infants of drug-dependent mothers),ordatacollectionentailsprohibitivecosts(e.g.,fMRI,measuringphono- logical difficulties of babies); or the study participants come from a population that ispronetodrop-out(e.g.,becausetheyarehomelessorinstitutionalized).Suchobs- tacles may result in data sets that are too small for the complexity of the statistical model needed to answer the research question. Researchers could reduce the requiredsamplesizefortheanalysisbysimplifyingtheirstatisticalmodels.However, thismayleavethe“true”researchquestionsunanswered.Assuch,limitationsassoci- ated with small data sets can restrict the usefulness of the scientific conclusions and mightevenhamperscientificbreakthroughs. Thefieldofmethodologicalsolutionsforissuesduetosmallsamplesizesisdevelop- ingrapidly,andfastsoftwareimplementationsofthesemethodsarebecomingincreas- inglyavailable.However,theselectionoftextsonstatisticalmethodsforsmallsample researchwithcomplexmodelsissparse.InMarch2018,weorganizedthefirstedition of the Small Sample Size Solutions conference (S4; www.uu.nl/s4) with the goal of bringingtogetherappliedresearcherswhoencounterissuesduetosmallsamples,and statisticiansworkingonsolutionstosuchissues.TheaimoftheS4Conferencewasto share information, learn about new developments, and discuss solutions for typical smallsamplesizeproblems.Thechaptersinthecurrentvolumedescribesomeofthe solutions to small sample size issues presented at the first S4 Conference. The list of contributors includes both established authors who provide an overview of available methods in a particular field, and early-career researchers working on promising innovative solutions. The authors ofthe chapters reviewed at least one other chapter in this volume, and each chapter was written with the goal of being accessible for applied researchers and students with a basic knowledge of statistics. Note that Introduction ix collecting more data, if at all possible, is always preferred, and that the methods describedinthecurrentbookarealastresort. The current book provides guidelines and tools for implementing a variety of solutions to issues that arise in small sample research, along with references for fur- ther (technical) details. The book includes solutions for estimation of population means, regression analyses, meta-analyses, factor analyses, advanced structural equa- tion models with latent variables, and models for nested observations. The types of solutionsconsistofBayesianestimationwithinformativepriors,variousclassicaland Bayesian methods for synthesizing data with small samples, constrained statistical inference, two-step modeling, and data analysis methods for one participant at a time. All methods require a strong justification of the choice of analytic strategy andcomplete transparencyaboutallstepsinthe analysis. The bookisaccompanied by state-of-the-art software solutions, some of which will only be released next year. All proposed solutions are described in steps researchers can implement with their own data and are accompanied with annotated syntax in R available on the Open Science Framework (osf.io/am7pr/). The content of the substantive applications spans a variety of disciplines, and we expect the book to be of interest to researchers within and outside academia who are working with small samples sizes. The book is split into three parts: Part I contains several chapters that describe and make use of Bayesian statis- tics. Chapter 1 offers a gentle introduction to the main ingredients in Bayesian analyses and provides necessary information for understanding Bayesian param- eter estimation and Bayes Factors. Chapter 2 offers a discussion of exchangeabil- ity and its role in the choice of sources of prior information in Bayesian analyses, which is relevant when combining datasets. Chapter 3 provides an extension of the When-to-Worry-and-How-to-Avoid-the-Misuse-of-Bayesian- Statistics (WAMBS) checklist, which is a 10-point checklist used to ensure opti- mal practices when applying Bayesian methods, extended to include prior and posterior predictive checking. Chapter 4 illustrates difficulties that can arise when implementing Bayesian solutions to a complex model and offers sugges- tions for avoiding these issues by making use of the effective sample size and divergent transitions. Chapter 5 provides a tutorial on Bayesian penalized regres- sion for scenarios with a small sample size relative to the complexity of the stat- istical model by applying so-called “shrinkage priors” that shrink small effects towards zero while leaving substantial effects large. Part II is composed of chapters on methods for analyzing data from a single participant. Chapter 6 introduces single-case experimental designs (n¼ 1) and provides background information for analyzing a single-case experimental design (SCED) using unilevel design-based analysis. Chapter 7 discusses SCEDs in detail and provides an example of tests of effectiveness and change processes. Chapter 8 introduced a shiny app that allows researchers to supplement test scores of a single participant with teacher input or scores from other students in order to obtain a more accurate estimate of a given student’s ability. Chapter 9