Ergebnisse der Mathematik und Volume 64 ihrer Grenzgebiete 3.Folge A Series of Modern Surveys in Mathematics EditorialBoard L.Ambrosio,Pisa V.Baladi,Paris G.-M.Greuel,Kaiserslautern M.Gromov,Bures-sur-Yvette G.Huisken,Tübingen J.Jost,Leipzig J.Kollár,Princeton G.Laumon,Orsay U.Tillmann,Oxford J.Tits,Paris D.B.Zagier,Bonn Forfurthervolumes: www.springer.com/series/728 Nihat Ay (cid:2) Jürgen Jost (cid:2) Hông Vân Lê (cid:2) Lorenz Schwachhöfer Information Geometry NihatAy HôngVânLê InformationTheoryofCognitiveSystems MathematicalInstituteofASCR MPIforMathematicsintheSciences CzechAcademyofSciences Leipzig,Germany Praha1,CzechRepublic and SantaFeInstitute,SantaFe,NM,USA LorenzSchwachhöfer DepartmentofMathematics JürgenJost TUDortmundUniversity GeometricMethodsandComplexSystems Dortmund,Germany MPIforMathematicsintheSciences Leipzig,Germany and SantaFeInstitute,SantaFe,NM,USA ISSN0071-1136 ISSN2197-5655(electronic) ErgebnissederMathematikundihrerGrenzgebiete.3.Folge/ASeriesofModernSurveys inMathematics ISBN978-3-319-56477-7 ISBN978-3-319-56478-4(eBook) DOI10.1007/978-3-319-56478-4 LibraryofCongressControlNumber:2017951855 Mathematics Subject Classification: 60A10, 62B05, 62B10, 62G05, 53B21, 53B05, 46B20, 94A15, 94A17,94B27 ©SpringerInternationalPublishingAG2017 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. Printedonacid-freepaper ThisSpringerimprintispublishedbySpringerNature TheregisteredcompanyisSpringerInternationalPublishingAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Preface Information geometry is the differential geometric treatment of statistical models. Ittherebyprovidesthemathematicalfoundationofstatistics.Informationgeometry therefore is of interest both for its beautiful mathematicalstructure and for the in- sightitprovidesintostatisticsanditsapplications.Informationgeometrycurrently is a very active field. For instance, Springer will soon launch a new topical jour- nal “Information Geometry”. We therefore think that the time is appropriate for a monograph on information geometry that develops the underlying mathematical theoryinfullgeneralityandrigor,thatexplorestheconnectionstoothermathemat- icaldisciplines,andthatprovesabstractandgeneralversionsoftheclassicalresults of statistics, like the Cramér–Rao inequality or Chentsov’s theorem. These, then, arethepurposesofthepresentbook,andwehopethatitwillbecomethestandard referenceforthefield. ParametricstatisticsasintroducedbyR.Fisherconsidersparametrizedfamilies of probability measures on some finite or infinite sample space Ω. Typically, one wishes to identify a parameter so that the resulting probability measure best fits the observation among the measures in the family. This naturally leads to quanti- tative questions, in particular, how sensitively the measures in the family depend ontheparameter.Forthis,ageometricperspectiveisexpedient.Thereisanatural metric, the Fisher metric introduced by Rao, on the space of probability measures on Ω. This metric is simply the projective or spherical metric obtained when one considersaprobabilitymeasureasanon-negativemeasurewithascalingfactorto renderitstotalmassequaltounity.TheFishermetricthusisaRiemannianmetric thatinducesacorrespondingstructureonparametrizedfamiliesofprobabilitymea- suresasabove.Furthermore,movingfromonereferencemeasuretoanotheryields an affine structure as discovered by S.I. Amari and N.N. Chentsov. The investiga- tion of these metric and affine structures is therefore called information geometry. Information-theoretical quantities like relative entropies (Kullback–Leibler diver- gences)thenfindanaturalgeometricinterpretation. Information geometry thus provides a way of understanding information- theoreticquantities,statisticalmodels,andcorrespondingstatisticalinferencemeth- ods in geometric terms. In particular, the Fisher metric and the Amari–Chentsov v vi Preface structurearecharacterizedbytheirinvarianceundersufficientstatistics.Severalge- ometricformalismshavebeenidentifiedaspowerfultoolstothisendandemphasize respective geometric aspects of probability theory. In this book, we move beyond theapplicationsinstatisticsanddevelopbothafunctionalanalyticandageometric theorythatareofmathematicalinterestintheirownright.Inparticular,thetheory ofduallyaffinestructuresturnsouttobeananalogueofKählergeometryinareal asopposedtoacomplexsetting. Also,astheconceptofShannoninformationcanberelatedtotheentropycon- ceptsofBoltzmannandGibbs,thereisalsoanaturalconnectionbetweeninforma- tiongeometryandstatisticalmechanics.Finally,informationgeometrycanalsobe usedasafoundationofimportantpartsofmathematicalbiology,likethetheoryof replicatorequationsandmathematicalpopulationgenetics. Samplespacescouldbefinite,butmoreoftenthannot,theyareinfinite,forin- stance,subsetsofsome(finite-oreveninfinite-dimensional)Euclideanspace.The spacesofmeasuresonsuchspacesthereforeareinfinite-dimensionalBanachspaces. Consequently, the differential geometric approach needs to be supplemented by functional analytic considerations. One of the purposes of this book therefore is to provide a general framework that integrates the differential geometry into the functionalanalysis. Acknowledgements We would like to thank Shun-ichi Amari for many fruitful discussions. This work wasmainlycarriedoutattheMaxPlanckInstituteforMathematicsintheSciences inLeipzig.IthasalsobeensupportedbytheBSIatRIKENinTokyo,theASSMS, GCUinLahore-Pakistan,theVNUforSciencesinHanoi,theMathematicalInsti- tuteoftheAcademyofSciencesoftheCzechRepublicinPrague,andtheSantaFe Institute.Wearegratefulfortheexcellentworkingconditionsandfinancialsupport of these institutions during extended visits of some of us. In particular, we should liketothankAntjeVandenbergforheroutstandinglogisticsupport. TheresearchofJ.J. leadingtohisbookcontributionhasreceivedfundingfrom the European Research Council under the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no. 267087. The research of H.V.L.issupportedbyRVO:67985840. vii Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 ABriefSynopsis . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 AnInformalDescription . . . . . . . . . . . . . . . . . . . . . . 6 1.2.1 TheFisherMetricandtheAmari–ChentsovStructure forFiniteSampleSpaces . . . . . . . . . . . . . . . . . 7 1.2.2 InfiniteSampleSpacesandFunctionalAnalysis . . . . . 8 1.2.3 ParametricStatistics . . . . . . . . . . . . . . . . . . . . 10 1.2.4 ExponentialandMixtureFamiliesfromthePerspective ofDifferentialGeometry . . . . . . . . . . . . . . . . . 14 1.2.5 InformationGeometryandInformationTheory . . . . . . 15 1.3 HistoricalRemarks. . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4 OrganizationofthisBook . . . . . . . . . . . . . . . . . . . . . 20 2 FiniteInformationGeometry . . . . . . . . . . . . . . . . . . . . . 25 2.1 ManifoldsofFiniteMeasures . . . . . . . . . . . . . . . . . . . 25 2.2 TheFisherMetric . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3 GradientFields . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 2.4 Them-ande-Connections. . . . . . . . . . . . . . . . . . . . . 42 2.5 TheAmari–ChentsovTensorandtheα-Connections . . . . . . . 47 2.5.1 TheAmari–ChentsovTensor . . . . . . . . . . . . . . . 47 2.5.2 Theα-Connections . . . . . . . . . . . . . . . . . . . . 50 2.6 CongruentFamiliesofTensors . . . . . . . . . . . . . . . . . . 52 2.7 Divergences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 2.7.1 Gradient-BasedApproach . . . . . . . . . . . . . . . . . 68 2.7.2 TheRelativeEntropy . . . . . . . . . . . . . . . . . . . 70 2.7.3 Theα-Divergence . . . . . . . . . . . . . . . . . . . . . 73 2.7.4 Thef-Divergence . . . . . . . . . . . . . . . . . . . . . 76 2.7.5 Theq-GeneralizationoftheRelativeEntropy. . . . . . . 78 ix x Contents 2.8 ExponentialFamilies . . . . . . . . . . . . . . . . . . . . . . . 79 2.8.1 ExponentialFamiliesasAffineSpaces . . . . . . . . . . 79 2.8.2 ImplicitDescriptionofExponentialFamilies . . . . . . . 84 2.8.3 InformationProjections . . . . . . . . . . . . . . . . . . 91 2.9 HierarchicalandGraphicalModels . . . . . . . . . . . . . . . . 100 2.9.1 InteractionSpaces . . . . . . . . . . . . . . . . . . . . . 101 2.9.2 HierarchicalModels . . . . . . . . . . . . . . . . . . . . 108 2.9.3 GraphicalModels . . . . . . . . . . . . . . . . . . . . . 112 3 ParametrizedMeasureModels . . . . . . . . . . . . . . . . . . . . 121 3.1 TheSpaceofProbabilityMeasuresandtheFisherMetric . . . . 121 3.2 ParametrizedMeasureModels. . . . . . . . . . . . . . . . . . . 135 3.2.1 TheStructureoftheSpaceofMeasures . . . . . . . . . . 139 3.2.2 TangentFibrationofSubsetsofBanachManifolds . . . . 140 3.2.3 PowersofMeasures . . . . . . . . . . . . . . . . . . . . 143 3.2.4 ParametrizedMeasureModelsandk-Integrability . . . . 150 3.2.5 Canonicaln-Tensorsofann-IntegrableModel . . . . . . 164 3.2.6 SignedParametrizedMeasureModels . . . . . . . . . . 168 3.3 ThePistone–SempiStructure . . . . . . . . . . . . . . . . . . . 170 3.3.1 e-Convergence . . . . . . . . . . . . . . . . . . . . . . . 170 3.3.2 OrliczSpaces . . . . . . . . . . . . . . . . . . . . . . . 172 3.3.3 ExponentialTangentSpaces . . . . . . . . . . . . . . . . 176 4 TheIntrinsicGeometryofStatisticalModels . . . . . . . . . . . . 185 4.1 ExtrinsicVersusIntrinsicGeometricStructures . . . . . . . . . . 185 4.2 ConnectionsandtheAmari–ChentsovStructure . . . . . . . . . 189 4.3 TheDualityBetweenExponentialandMixtureFamilies . . . . . 201 4.4 CanonicalDivergences. . . . . . . . . . . . . . . . . . . . . . . 210 4.4.1 DualStructuresviaDivergences. . . . . . . . . . . . . . 210 4.4.2 AGeneralCanonicalDivergence . . . . . . . . . . . . . 213 4.4.3 RecoveringtheCanonicalDivergenceofaDuallyFlat Structure . . . . . . . . . . . . . . . . . . . . . . . . . . 215 4.4.4 ConsistencywiththeUnderlyingDualisticStructure . . . 217 4.5 StatisticalManifoldsandStatisticalModels . . . . . . . . . . . . 219 4.5.1 StatisticalManifoldsandIsostatisticalImmersions . . . . 220 4.5.2 MonotoneInvariantsofStatisticalManifolds . . . . . . . 223 4.5.3 ImmersionofCompactStatisticalManifoldsintoLinear StatisticalManifolds . . . . . . . . . . . . . . . . . . . . 226 4.5.4 ProofoftheExistenceofIsostatisticalImmersions . . . . 228 4.5.5 ExistenceofStatisticalEmbeddings. . . . . . . . . . . . 238 Contents xi 5 InformationGeometryandStatistics . . . . . . . . . . . . . . . . . 241 5.1 CongruentEmbeddingsandSufficientStatistics . . . . . . . . . 241 5.1.1 StatisticsandCongruentEmbeddings . . . . . . . . . . . 244 5.1.2 MarkovKernelsandCongruentMarkovEmbeddings . . 253 5.1.3 Fisher–NeymanSufficientStatistics . . . . . . . . . . . . 261 5.1.4 InformationLossandMonotonicity . . . . . . . . . . . . 263 5.1.5 Chentsov’sTheoremandItsGeneralization . . . . . . . . 268 5.2 EstimatorsandtheCramér–RaoInequality . . . . . . . . . . . . 277 5.2.1 EstimatorsandTheirBias,MeanSquareError,Variance . 277 5.2.2 AGeneralCramér–RaoInequality . . . . . . . . . . . . 281 5.2.3 ClassicalCramér–RaoInequalities . . . . . . . . . . . . 286 5.2.4 EfficientEstimatorsandConsistentEstimators . . . . . . 287 6 FieldsofApplicationofInformationGeometry . . . . . . . . . . . 295 6.1 ComplexityofCompositeSystems . . . . . . . . . . . . . . . . 295 6.1.1 AGeometricApproachtoComplexity . . . . . . . . . . 296 6.1.2 TheInformationDistancefromHierarchicalModels . . . 298 6.1.3 TheWeightedInformationDistance . . . . . . . . . . . . 307 6.1.4 ComplexityofStochasticProcesses . . . . . . . . . . . . 317 6.2 EvolutionaryDynamics . . . . . . . . . . . . . . . . . . . . . . 327 6.2.1 NaturalSelectionandReplicatorEquations . . . . . . . . 328 6.2.2 ContinuousTimeLimits . . . . . . . . . . . . . . . . . . 333 6.2.3 PopulationGenetics . . . . . . . . . . . . . . . . . . . . 336 6.3 MonteCarloMethods . . . . . . . . . . . . . . . . . . . . . . . 348 6.3.1 LangevinMonteCarlo . . . . . . . . . . . . . . . . . . . 350 6.3.2 HamiltonianMonteCarlo . . . . . . . . . . . . . . . . . 351 6.4 Infinite-DimensionalGibbsFamilies . . . . . . . . . . . . . . . 354 AppendixA MeasureTheory . . . . . . . . . . . . . . . . . . . . . . . 361 AppendixB RiemannianGeometry . . . . . . . . . . . . . . . . . . . 367 AppendixC BanachManifolds . . . . . . . . . . . . . . . . . . . . . . 381 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403