DigitalSignalProcessingwithKernelMethods Digital Signal Processing with Kernel Methods JoséLuisRojo-Álvarez DepartmentofSignalTheoryandCommunications UniversityReyJuanCarlos Fuenlabrada(Madrid) and CenterforComputationalSimulation UniversidadPolitécnicadeMadrid,Spain ManelMartínez-Ramón DepartmentofElectricalandComputerEngineering TheUniversityofNewMexico Albuquerque,NewMexico USA JordiMuñoz-Marí DepartmentofElectronicsEngineering UniversitatdeValència Paterna(València),Spain GustauCamps-Valls DepartmentofElectronicsEngineering UniversitatdeValència Paterna(València),Spain Thiseditionfirstpublished ©JohnWiley&SonsLtd Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise, exceptaspermittedbylaw.Adviceonhowtoobtainpermissiontoreusematerialfromthistitleisavailable athttp://www.wiley.com/go/permissions. TherightofJoséLuisRojo-Álvarez,ManelMartínez-Ramón,JordiMuñoz-Marí,GustauCamps-Vallstobe identifiedastheauthorsoftheeditorialmaterialinthisworkhasbeenassertedinaccordancewithlaw. RegisteredOffices JohnWiley&Sons,Inc.,RiverStreet,Hoboken,NJ,USA JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,POSQ,UK EditorialOffice TheAtrium,SouthernGate,Chichester,WestSussex,POSQ,UK Fordetailsofourglobaleditorialoffices,customerservices,andmoreinformationaboutWileyproducts visitusatwww.wiley.com. Wileyalsopublishesitsbooksinavarietyofelectronicformatsandbyprint-on-demand.Somecontentthat appearsinstandardprintversionsofthisbookmaynotbeavailableinotherformats. LimitofLiability/DisclaimerofWarranty MATLABⓇandSimulinkisatrademarkofTheMathWorks,Inc.andisusedwithpermission.The MathWorksdoesnotwarranttheaccuracyofthetextorexercisesinthisbook.Thiswork’suseordiscussion ofMATLABⓇsoftwareorrelatedproductsdoesnotconstituteendorsementorsponsorshipbyThe MathWorksofaparticularpedagogicalapproachorparticularuseoftheMATLABⓇsoftware. Whilethepublisherandauthorshaveusedtheirbesteffortsinpreparingthiswork,theymakeno representationsorwarrantieswithrespecttotheaccuracyorcompletenessofthecontentsofthisworkand specificallydisclaimallwarranties,includingwithoutlimitationanyimpliedwarrantiesofmerchantability orfitnessforaparticularpurpose.Nowarrantymaybecreatedorextendedbysalesrepresentatives,written salesmaterialsorpromotionalstatementsforthiswork.Thefactthatanorganization,website,orproductis referredtointhisworkasacitationand/orpotentialsourceoffurtherinformationdoesnotmeanthatthe publisherandauthorsendorsetheinformationorservicestheorganization,website,orproductmay provideorrecommendationsitmaymake.Thisworkissoldwiththeunderstandingthatthepublisherisnot engagedinrenderingprofessionalservices.Theadviceandstrategiescontainedhereinmaynotbesuitable foryoursituation.Youshouldconsultwithaspecialistwhereappropriate.Further,readersshouldbeaware thatwebsiteslistedinthisworkmayhavechangedordisappearedbetweenwhenthisworkwaswrittenand whenitisread.Neitherthepublishernorauthorsshallbeliableforanylossofprofitoranyother commercialdamages,includingbutnotlimitedtospecial,incidental,consequential,orotherdamages. LibraryofCongressCataloging-in-Publicationdata Names:Rojo-Álvarez,JoséLuis,–author.|Martínez-Ramón,Manel,–author.| Muñoz-Marí,Jordi,author.|Camps-Valls,Gustau,–author. Title:Digitalsignalprocessingwithkernelmethods/byDr.JoséLuisRojo-Álvarez, Dr.ManelMartínez-Ramón,Dr.JordiMuñoz-Marí,Dr.GustauCamps-Valls. Description:Firstedition.|Hoboken,NJ:JohnWiley&Sons,.|Includesbibliographical referencesandindex.| Identifiers:LCCN(print)|LCCN(ebook)|ISBN(pdf)| ISBN(epub)|ISBN(cloth) Subjects:LCSH:Signalprocessing–Digitaltechniques. Classification:LCCTK.(ebook)|LCCTK..R(print)|DDC./–dc LCrecordavailableathttps://lccn.loc.gov/ CoverdesignbyWiley Coverimage:©AF-studio/Gettyimages Setin/ptWarnockbySPiGlobal,Pondicherry,India           v Contents AbouttheAuthors xiii Preface xvii Acknowledgements xxi ListofAbbreviations xxiii PartI FundamentalsandBasicElements  1 FromSignalProcessingtoMachineLearning  . ANewScienceisBorn:SignalProcessing  .. SignalProcessingBeforeBeingCoined  .. :BirthoftheInformationAge  .. s:AudioEngineeringCatalyzesSignalProcessing  . FromAnalogtoDigitalSignalProcessing  .. s:DigitalSignalProcessingBegins  .. s:DigitalSignalProcessingBecomesPopular  .. s:SiliconMeetsDigitalSignalProcessing  . DigitalSignalProcessingMeetsMachineLearning  .. s:NewApplicationAreas  .. s:NeuralNetworks,FuzzyLogic,andGeneticOptimization  . RecentMachineLearninginDigitalSignalProcessing  .. TraditionalSignalAssumptionsAreNoLongerValid  .. EncodingPriorKnowledge  .. LearningandKnowledgefromData  .. FromMachineLearningtoDigitalSignalProcessing  .. FromDigitalSignalProcessingtoMachineLearning  2 IntroductiontoDigitalSignalProcessing  . OutlineoftheSignalProcessingField  .. FundamentalsonSignalsandSystems  .. DigitalFiltering  .. SpectralAnalysis  .. Deconvolution  .. Interpolation  vi Contents .. SystemIdentification  .. BlindSourceSeparation  . FromTime–FrequencytoCompressedSensing  .. Time–FrequencyDistributions  .. WaveletTransforms  .. Sparsity,CompressedSensing,andDictionaryLearning  . MultidimensionalSignalsandSystems  .. MultidimensionalSignals  .. MultidimensionalSystems  . SpectralAnalysisonManifolds  .. TheoreticalFundamentals  .. LaplacianMatrices  . TutorialsandApplicationExamples  .. RealandComplexSignalProcessingandRepresentations  .. Convolution,FourierTransform,andSpectrum  .. Continuous-TimeSignalsandSystems  .. FilteringCardiacSignals  .. NonparametricSpectrumEstimation  .. ParametricSpectrumEstimation  .. SourceSeparation  .. Time–FrequencyRepresentationsandWavelets  .. ExamplesforSpectralAnalysisonManifolds  . QuestionsandProblems  3 SignalProcessingModels  . Introduction  . VectorSpaces,Basis,andSignalModels  .. BasicOperationsforVectors  .. VectorSpaces  .. HilbertSpaces  .. SignalModels  .. ComplexSignalModels  .. StandardNoiseModelsinDSP  .. TheRoleoftheCostFunction  .. TheRoleoftheRegularizer  . DigitalSignalProcessingModels  .. SinusoidalSignalModels  .. SystemIdentificationSignalModels  .. SincInterpolationModels  .. SparseDeconvolution  .. ArrayProcessing  . TutorialsandApplicationExamples  .. ExamplesofNoiseModels  .. AutoregressiveExogenousSystemIdentificationModels  .. NonlinearSystemIdentificationUsingVolterraModels  .. SinusoidalSignalModels  Contents vii .. Sinc-basedInterpolation  .. SparseDeconvolution  .. ArrayProcessing  . QuestionsandProblems  .A MATLABsimpleInterpToolboxStructure  4 KernelFunctionsandReproducingKernelHilbertSpaces  . Introduction  . KernelFunctionsandMappings  .. MeasuringSimilaritywithKernels  .. Positive-DefiniteKernels  .. ReproducingKernelinHilbertSpaceandReproducingProperty  .. Mercer’sTheorem  . KernelProperties  .. Tikhonov’sRegularization  .. RepresenterTheoremandRegularizationProperties  .. BasicOperationswithKernels  . ConstructingKernelFunctions  .. StandardKernels  .. PropertiesofKernels  .. EngineeringSignalProcessingKernels  . ComplexReproducingKernelinHilbertSpaces  . SupportVectorMachineElementsforRegressionandEstimation  .. SupportVectorRegressionSignalModelandCostFunction  .. MinimizingFunctional  . TutorialsandApplicationExamples  .. KernelCalculationsandKernelMatrices  .. BasicOperationswithKernels  .. ConstructingKernels  .. ComplexKernels  .. ApplicationExampleforSupportVectorRegressionElements  . ConcludingRemarks  . QuestionsandProblems  PartII FunctionApproximationandAdaptiveFiltering  5 ASupportVectorMachineSignalEstimationFramework  . Introduction  . AFrameworkforSupportVectorMachineSignalEstimation  . PrimalSignalModelsforSupportVectorMachineSignalProcessing  .. NonparametricSpectrumandSystemIdentification  .. OrthogonalFrequencyDivisionMultiplexingDigitalCommunications  .. ConvolutionalSignalModels  .. ArrayProcessing  . TutorialsandApplicationExamples  viii Contents .. NonparametricSpectralAnalysiswithPrimalSignalModels  .. SystemIdentificationwithPrimalSignalModelγ-filter  .. ParametricSpectralDensityEstimationwithPrimalSignalModels  .. TemporalReferenceArrayProcessingwithPrimalSignalModels  .. SincInterpolationwithPrimalSignalModels  .. OrthogonalFrequencyDivisionMultiplexingwithPrimalSignalModels  . QuestionsandProblems  6 ReproducingKernelHilbertSpaceModelsforSignalProcessing  . Introduction  . ReproducingKernelHilbertSpaceSignalModels  .. KernelAutoregressiveExogenousIdentification  .. KernelFiniteImpulseResponseandtheγ-filter  .. KernelArrayProcessingwithSpatialReference  .. KernelSemiparametricRegression  . TutorialsandApplicationExamples  .. NonlinearSystemIdentificationwithSupportVector Machine–AutoregressiveandMovingAverage  .. NonlinearSystemIdentificationwiththeγ-filter  .. ElectricNetworkModelingwithSemiparametricRegression  .. PromotionalData  .. SpatialandTemporalAntennaArrayKernelProcessing  . QuestionsandProblems  7 DualSignalModelsforSignalProcessing  . Introduction  . DualSignalModelElements  . DualSignalModelInstantiations  .. DualSignalModelforNonuniformSignalInterpolation  .. DualSignalModelforSparseSignalDeconvolution  .. SpectrallyAdaptedMercerKernels  . TutorialsandApplicationExamples  .. NonuniformInterpolationwiththeDualSignalModel  .. SparseDeconvolutionwiththeDualSignalModel  .. DopplerUltrasoundProcessingforFaultDetection  .. SpectrallyAdaptedMercerKernels  .. InterpolationofHeartRateVariabilitySignals  .. DenoisinginCardiacMotion-ModeDopplerUltrasoundImages  .. IndoorLocationfromMobileDevicesMeasurements  .. ElectroanatomicalMapsinCardiacNavigationSystems  . QuestionsandProblems  8 AdvancesinKernelRegressionandFunctionApproximation  . Introduction  . Kernel-BasedRegressionMethods  .. AdvancesinSupportVectorRegression  .. Multi-outputSupportVectorRegression  Contents ix .. KernelRidgeRegression  .. KernelSignal-to-NoiseRegression  .. Semi-supervisedSupportVectorRegression  .. ModelSelectioninKernelRegressionMethods  . BayesianNonparametricKernelRegressionModels  .. GaussianProcessRegression  .. RelevanceVectorMachines  . TutorialsandApplicationExamples  .. ComparingSupportVectorRegression,RelevanceVectorMachines,and GaussianProcessRegression  .. Profile-DependentSupportVectorRegression  .. Multi-outputSupportVectorRegression  .. KernelSignal-to-NoiseRatioRegression  .. Semi-supervisedSupportVectorRegression  .. BayesianNonparametricModel  .. GaussianProcessRegression  .. RelevanceVectorMachines  . ConcludingRemarks  . QuestionsandProblems  9 AdaptiveKernelLearningforSignalProcessing  . Introduction  . LinearAdaptiveFiltering  .. LeastMeanSquaresAlgorithm  .. RecursiveLeast-SquaresAlgorithm  . KernelAdaptiveFiltering  . KernelLeastMeanSquares  .. DerivationofKernelLeastMeanSquares  .. ImplementationChallengesandDualFormulation  .. ExampleonPredictionoftheMackey–GlassTimeSeries  .. PracticalKernelLeastMeanSquaresAlgorithms  . KernelRecursiveLeastSquares  .. KernelRidgeRegression  .. DerivationofKernelRecursiveLeastSquares  .. PredictionoftheMackey–GlassTimeSerieswithKernelRecursiveLeast Squares  .. BeyondtheStationaryModel  .. ExampleonNonlinearChannelIdentificationandReconvergence  . ExplicitRecursivityforAdaptiveKernelModels  .. RecursivityinHilbertSpaces  .. RecursiveFiltersinReproducingKernelHilbertSpaces  . OnlineSparsificationwithKernels  .. SparsitybyConstruction  .. SparsitybyPruning  . ProbabilisticApproachestoKernelAdaptiveFiltering  .. GaussianProcessesandKernelRidgeRegression  .. OnlineRecursiveSolutionforGaussianProcessesRegression  x Contents .. KernelRecursiveLeastSquaresTracker  .. ProbabilisticKernelLeastMeanSquares  . FurtherReading  .. SelectionofKernelParameters  .. Multi-KernelAdaptiveFiltering  .. RecursiveFilteringinKernelHilbertSpaces  . TutorialsandApplicationExamples  .. KernelAdaptiveFilteringToolbox  .. PredictionofaRespiratoryMotionTimeSeries  .. OnlineRegressionontheKINKDataset  .. TheMackey–GlassTimeSeries  .. ExplicitRecursivityonReproducingKernelinHilbertSpace andElectroencephalogramPrediction  .. AdaptiveAntennaArrayProcessing  . QuestionsandProblems  PartIII Classification,Detection,andFeatureExtraction  10 SupportVectorMachineandKernelClassificationAlgorithms  . Introduction  . SupportVectorMachineandKernelClassifiers  .. SupportVectorMachines  .. MulticlassandMultilabelSupportVectorMachines  .. Least-SquaresSupportVectorMachine  .. KernelFisher’sDiscriminantAnalysis  . AdvancesinKernel-BasedClassification  .. LargeMarginFiltering  .. Semi-supervisedLearning  .. MultipleKernelLearning  .. Structured-OutputLearning  .. ActiveLearning  . Large-ScaleSupportVectorMachines  .. Large-ScaleSupportVectorMachineImplementations  .. RandomFourierFeatures  .. ParallelSupportVectorMachine  .. Outlook  . TutorialsandApplicationExamples  .. ExamplesofSupportVectorMachineClassification  .. ExampleofLeast-SquaresSupportVectorMachine  .. Kernel-FilteringSupportVectorMachineforBrain–ComputerInterface SignalClassification  .. ExampleofLaplacianSupportVectorMachine  .. ExampleofGraph-BasedLabelPropagation  .. ExamplesofMultipleKernelLearning  . ConcludingRemarks  . QuestionsandProblems 