Emotion Recognition A Pattern Analysis Approach Amit Konar & Aruna Chakraborty EMOTION RECOGNITION EMOTION RECOGNITION A Pattern Analysis Approach Editedby AMITKONAR ArtificialIntelligenceLaboratory DepartmentofElectronicsandTelecommunicationEngineering JadavpurUniversity Kolkata,India ARUNACHAKRABORTY DepartmentofComputerScience&Engineering St.Thomas’CollegeofEngineering&Technology Kolkata,India Copyright©2015byJohnWiley&Sons,Inc.Allrightsreserved. PublishedbyJohnWiley&Sons,Inc.,Hoboken,NewJersey. PublishedsimultaneouslyinCanada. Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedinanyformor byanymeans,electronic,mechanical,photocopying,recording,scanning,orotherwise,exceptas permittedunderSection107or108ofthe1976UnitedStatesCopyrightAct,withouteithertheprior writtenpermissionofthePublisher,orauthorizationthroughpaymentoftheappropriateper-copyfeeto theCopyrightClearanceCenter,Inc.,222RosewoodDrive,Danvers,MA01923,(978)750-8400, fax(978)646-8600,oronthewebatwww.copyright.com.RequeststothePublisherforpermission shouldbeaddressedtothePermissionsDepartment,JohnWiley&Sons,Inc.,111RiverStreet, Hoboken,NJ07030,(201)748-6011,fax(201)748-6008. LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsin preparingthisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyor completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesof merchantabilityorfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysales representativesorwrittensalesmaterials.Theadviceandstrategiescontainedhereinmaynotbesuitable foryoursituation.Youshouldconsultwithaprofessionalwhereappropriate.Neitherthepublishernor authorshallbeliableforanylossofprofitoranyothercommercialdamages,includingbutnotlimitedto special,incidental,consequential,orotherdamages. ForgeneralinformationonourotherproductsandservicespleasecontactourCustomerCare DepartmentwiththeU.S.at877-762-2974,outsidetheU.S.at317-572-3993orfax317-572-4002. Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprint, however,maynotbeavailableinelectronicformat. LibraryofCongressCataloging-in-PublicationData: Konar,Amit. Emotionrecognition:apatternanalysisapproach/AmitKonar,ArunaChakraborty. pagescm Includesindex. ISBN978-1-118-13066-7(hardback) 1.Human-computerinteraction. 2.Artificialintelligence. 3.Emotions–Computersimulation. 4.Patternrecognitionsystems. 5.Context-awarecomputing. I.Chakraborty,Aruna,1977- II.Title. QA76.9.H85K6552014 004.01’9–dc23 2014024314 PrintedintheUnitedStatesofAmerica. 10 9 8 7 6 5 4 3 2 1 Toourparents CONTENTS Preface xix Acknowledgments xxvii Contributors xxix 1 IntroductiontoEmotionRecognition 1 AmitKonar,AnishaHalder,andArunaChakraborty 1.1 BasicsofPatternRecognition, 1 1.2 EmotionDetectionasaPatternRecognitionProblem, 2 1.3 FeatureExtraction, 3 1.3.1 FacialExpression–BasedFeatures, 3 1.3.2 VoiceFeatures, 7 1.3.3 EEGFeaturesUsedforEmotionRecognition, 9 1.3.4 Gesture-andPosture-BasedEmotionalFeatures, 11 1.3.5 MultimodalFeatures, 12 1.4 FeatureReductionTechniques, 15 1.4.1 PrincipalComponentAnalysis, 15 1.4.2 IndependentComponentAnalysis, 16 1.4.3 EvolutionaryApproachtoNonlinearFeatureReduction, 16 1.5 EmotionClassification, 17 1.5.1 NeuralClassifier, 17 1.5.2 FuzzyClassifiers, 21 1.5.3 HiddenMarkovModelBasedClassifiers, 22 1.5.4 k-NearestNeighborAlgorithm, 22 1.5.5 Na¨ıveBayesClassifier, 23 vii viii CONTENTS 1.6 MultimodalEmotionRecognition, 24 1.7 StimulusGenerationforEmotionArousal, 24 1.8 ValidationTechniques, 26 1.8.1 PerformanceMetricsforEmotionClassification, 27 1.9 Summary, 27 References, 28 AuthorBiographies, 44 2 ExploitingDynamicDependenciesAmongActionUnitsfor SpontaneousFacialActionRecognition 47 YanTongandQiangJi 2.1 Introduction, 48 2.2 RelatedWork, 49 2.3 ModelingtheSemanticandDynamicRelationshipsAmongAUs WithaDBN, 50 2.3.1 ADBNforModelingDynamicDependencies amongAUs, 51 2.3.2 ConstructingtheInitialDBN, 54 2.3.3 LearningDBNModel, 55 2.3.4 AURecognitionThroughDBNInference, 59 2.4 ExperimentalResults, 60 2.4.1 FacialActionUnitDatabases, 60 2.4.2 EvaluationonCohnandKanadeDatabase, 61 2.4.3 EvaluationonSpontaneousFacialExpression Database, 62 2.5 Conclusion, 64 References, 64 AuthorBiographies, 66 3 FacialExpressions:ACross-CulturalStudy 69 ChandraniSaha,WashefAhmed,SomaMitra,DebasisMazumdar, andSushmitaMitra 3.1 Introduction, 69 3.2 ExtractionofFacialRegionsandEkman’sActionUnits, 71 3.2.1 ComputationofOpticalFlowVectorRepresentingMuscle Movement, 72 3.2.2 ComputationofRegionofInterest, 73 3.2.3 ComputationofFeatureVectorsWithinROI, 74 3.2.4 FacialDeformationandEkman’sActionUnits, 75 3.3 CulturalVariationinOccurrenceofDifferentAUs, 76 3.4 ClassificationPerformanceConsideringCulturalVariability, 79 3.5 Conclusion, 84 References, 84 AuthorBiographies, 86