Methods of Multivariate Analysis Second Edition Methods of Multivariate Analysis Second Edition ALVINC.RENCHER BrighamYoungUniversity AJOHNWILEY&SONS,INC.PUBLICATION ∞ Thisbookisprintedonacid-freepaper. Copyright(cid:1)c 2002byJohnWiley&Sons,Inc.Allrightsreserved. PublishedsimultaneouslyinCanada. Nopartofthispublicationmaybereproduced,storedinaretrievalsystemortransmittedinanyform orbyanymeans,electronic,mechanical,photocopying,recording,scanningorotherwise,exceptas permittedunderSections107or108ofthe1976UnitedStatesCopyrightAct,withouteithertheprior writtenpermissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copyfeeto theCopyrightClearanceCenter,222RosewoodDrive,Danvers,MA01923,(978)750-8400,fax(978) 750-4744.RequeststothePublisherforpermissionshouldbeaddressedtothePermissionsDepartment, JohnWiley&Sons,Inc.,605ThirdAvenue,NewYork,NY10158-0012,(212)850-6011,fax(212) 850-6008.E-Mail:[email protected]. Fororderingandcustomerservice,call1-800-CALL-WILEY. LibraryofCongressCataloging-in-PublicationData Rencher,AlvinC.,1934– Methodsofmultivariateanalysis/AlvinC.Rencher.—2nded. p. cm.—(Wileyseriesinprobabilityandmathematicalstatistics) “AWiley-Intersciencepublication.” Includesbibliographicalreferencesandindex. ISBN0-471-41889-7(cloth) 1. Multivariateanalysis. I. Title. II. Series. QA278.R452001 519.5(cid:2)35—dc21 2001046735 PrintedintheUnitedStatesofAmerica 10 9 8 7 6 5 4 3 2 1 Contents 1. Introduction 1 1.1 WhyMultivariateAnalysis?, 1 1.2 Prerequisites, 3 1.3 Objectives, 3 1.4 BasicTypesofDataandAnalysis, 3 2. MatrixAlgebra 5 2.1 Introduction, 5 2.2 NotationandBasicDefinitions, 5 2.2.1 Matrices,Vectors,andScalars, 5 2.2.2 EqualityofVectorsandMatrices, 7 2.2.3 TransposeandSymmetricMatrices, 7 2.2.4 SpecialMatrices, 8 2.3 Operations, 9 2.3.1 SummationandProductNotation, 9 2.3.2 AdditionofMatricesandVectors, 10 2.3.3 MultiplicationofMatricesandVectors, 11 2.4 PartitionedMatrices, 20 2.5 Rank, 22 2.6 Inverse, 23 2.7 PositiveDefiniteMatrices, 25 2.8 Determinants, 26 2.9 Trace, 30 2.10 OrthogonalVectorsandMatrices, 31 2.11 EigenvaluesandEigenvectors, 32 2.11.1 Definition, 32 2.11.2 I+AandI−A, 33 2.11.3 tr(A)and|A|, 34 2.11.4 PositiveDefiniteandSemidefiniteMatrices, 34 2.11.5 TheProductAB, 35 2.11.6 SymmetricMatrix, 35 v vi CONTENTS 2.11.7 SpectralDecomposition, 35 2.11.8 SquareRootMatrix, 36 2.11.9 SquareMatricesandInverseMatrices, 36 2.11.10 SingularValueDecomposition, 36 3. CharacterizingandDisplayingMultivariateData 43 3.1 MeanandVarianceofaUnivariateRandomVariable, 43 3.2 CovarianceandCorrelationofBivariateRandomVariables, 45 3.2.1 Covariance, 45 3.2.2 Correlation, 49 3.3 ScatterPlotsofBivariateSamples, 50 3.4 GraphicalDisplaysforMultivariateSamples, 52 3.5 MeanVectors, 53 3.6 CovarianceMatrices, 57 3.7 CorrelationMatrices, 60 3.8 MeanVectorsandCovarianceMatricesforSubsetsof Variables, 62 3.8.1 TwoSubsets, 62 3.8.2 ThreeorMoreSubsets, 64 3.9 LinearCombinationsofVariables, 66 3.9.1 SampleProperties, 66 3.9.2 PopulationProperties, 72 3.10 MeasuresofOverallVariability, 73 3.11 EstimationofMissingValues, 74 3.12 DistancebetweenVectors, 76 4. TheMultivariateNormalDistribution 82 4.1 MultivariateNormalDensityFunction, 82 4.1.1 UnivariateNormalDensity, 82 4.1.2 MultivariateNormalDensity, 83 4.1.3 GeneralizedPopulationVariance, 83 4.1.4 DiversityofApplicationsoftheMultivariateNormal, 85 4.2 PropertiesofMultivariateNormalRandomVariables, 85 4.3 EstimationintheMultivariateNormal, 90 4.3.1 MaximumLikelihoodEstimation, 90 4.3.2 DistributionofyandS, 91 4.4 AssessingMultivariateNormality, 92 4.4.1 InvestigatingUnivariateNormality, 92 4.4.2 InvestigatingMultivariateNormality, 96 CONTENTS vii 4.5 Outliers, 99 4.5.1 OutliersinUnivariateSamples, 100 4.5.2 OutliersinMultivariateSamples, 101 5. TestsonOneorTwoMeanVectors 112 5.1 MultivariateversusUnivariateTests, 112 5.2 Testson(cid:1)with(cid:2)Known, 113 5.2.1 ReviewofUnivariateTestfor H :µ=µ 0 0 withσ Known, 113 5.2.2 MultivariateTestfor H :(cid:1)=(cid:1) with(cid:2)Known, 114 0 0 5.3 Testson(cid:1)When(cid:2)IsUnknown, 117 5.3.1 ReviewofUnivariatet-Testfor H :µ=µ withσ 0 0 Unknown, 117 5.3.2 Hotelling’sT2-Testfor H :(cid:1)=(cid:1) with(cid:2)Unknown, 117 0 0 5.4 ComparingTwoMeanVectors, 121 5.4.1 ReviewofUnivariateTwo-Samplet-Test, 121 5.4.2 MultivariateTwo-SampleT2-Test, 122 5.4.3 LikelihoodRatioTests, 126 5.5 TestsonIndividualVariablesConditionalonRejectionof H by 0 theT2-Test, 126 5.6 ComputationofT2, 130 5.6.1 ObtainingT2fromaMANOVAProgram, 130 5.6.2 ObtainingT2fromMultipleRegression, 130 5.7 PairedObservationsTest, 132 5.7.1 UnivariateCase, 132 5.7.2 MultivariateCase, 134 5.8 TestforAdditionalInformation, 136 5.9 ProfileAnalysis, 139 5.9.1 One-SampleProfileAnalysis, 139 5.9.2 Two-SampleProfileAnalysis, 141 6. MultivariateAnalysisofVariance 156 6.1 One-WayModels, 156 6.1.1 UnivariateOne-WayAnalysisofVariance(ANOVA), 156 6.1.2 MultivariateOne-WayAnalysisofVarianceModel (MANOVA), 158 6.1.3 Wilks’TestStatistic, 161 6.1.4 Roy’sTest, 164 6.1.5 PillaiandLawley–HotellingTests, 166 viii CONTENTS 6.1.6 UnbalancedOne-WayMANOVA, 168 6.1.7 SummaryoftheFourTestsandRelationshiptoT2, 168 6.1.8 MeasuresofMultivariateAssociation, 173 6.2 ComparisonoftheFourManovaTestStatistics, 176 6.3 Contrasts, 178 6.3.1 UnivariateContrasts, 178 6.3.2 MultivariateContrasts, 180 6.4 TestsonIndividualVariablesFollowingRejectionof H bythe 0 OverallMANOVATest, 183 6.5 Two-WayClassification, 186 6.5.1 ReviewofUnivariateTwo-WayANOVA, 186 6.5.2 MultivariateTwo-WayMANOVA, 188 6.6 OtherModels, 195 6.6.1 HigherOrderFixedEffects, 195 6.6.2 MixedModels, 196 6.7 CheckingontheAssumptions, 198 6.8 ProfileAnalysis, 199 6.9 RepeatedMeasuresDesigns, 204 6.9.1 Multivariatevs.UnivariateApproach, 204 6.9.2 One-SampleRepeatedMeasuresModel, 208 6.9.3 k-SampleRepeatedMeasuresModel, 211 6.9.4 ComputationofRepeatedMeasuresTests, 212 6.9.5 RepeatedMeasureswithTwoWithin-Subjects FactorsandOneBetween-SubjectsFactor, 213 6.9.6 RepeatedMeasureswithTwoWithin-Subjects FactorsandTwoBetween-SubjectsFactors, 219 6.9.7 AdditionalTopics, 221 6.10 GrowthCurves, 221 6.10.1 GrowthCurveforOneSample, 221 6.10.2 GrowthCurvesforSeveralSamples, 229 6.10.3 AdditionalTopics, 230 6.11 TestsonaSubvector, 231 6.11.1 TestforAdditionalInformation, 231 6.11.2 StepwiseSelectionofVariables, 233 7. TestsonCovarianceMatrices 248 7.1 Introduction, 248 7.2 TestingaSpecifiedPatternfor(cid:2), 248 7.2.1 Testing H : (cid:2)=(cid:2) , 248 0 0 CONTENTS ix 7.2.2 TestingSphericity, 250 7.2.3 Testing H : (cid:2)=σ2[(1−ρ)I+ρJ], 252 0 7.3 TestsComparingCovarianceMatrices, 254 7.3.1 UnivariateTestsofEqualityofVariances, 254 7.3.2 MultivariateTestsofEqualityofCovarianceMatrices, 255 7.4 TestsofIndependence, 259 7.4.1 IndependenceofTwoSubvectors, 259 7.4.2 IndependenceofSeveralSubvectors, 261 7.4.3 TestforIndependenceofAllVariables, 265 8. DiscriminantAnalysis:DescriptionofGroupSeparation 270 8.1 Introduction, 270 8.2 TheDiscriminantFunctionforTwoGroups, 271 8.3 RelationshipbetweenTwo-GroupDiscriminantAnalysisand MultipleRegression, 275 8.4 DiscriminantAnalysisforSeveralGroups, 277 8.4.1 DiscriminantFunctions, 277 8.4.2 AMeasureofAssociationforDiscriminantFunctions, 282 8.5 StandardizedDiscriminantFunctions, 282 8.6 TestsofSignificance, 284 8.6.1 TestsfortheTwo-GroupCase, 284 8.6.2 TestsfortheSeveral-GroupCase, 285 8.7 InterpretationofDiscriminantFunctions, 288 8.7.1 StandardizedCoefficients, 289 8.7.2 Partial F-Values, 290 8.7.3 CorrelationsbetweenVariablesandDiscriminant Functions, 291 8.7.4 Rotation, 291 8.8 ScatterPlots, 291 8.9 StepwiseSelectionofVariables, 293 9. ClassificationAnalysis:AllocationofObservationstoGroups 299 9.1 Introduction, 299 9.2 ClassificationintoTwoGroups, 300 9.3 ClassificationintoSeveralGroups, 304 9.3.1 EqualPopulationCovarianceMatrices:Linear ClassificationFunctions, 304 9.3.2 UnequalPopulationCovarianceMatrices:Quadratic ClassificationFunctions, 306