《深度自然语言处理》课程教学课件（Natural language processing with deep learning）15 Machine translation

西安交通大学Natural LanguageProcessingwithDeepLearningXIANHAOTONGUNIVERSITYMachine Translation龚通大学Chen Li2023

Chen Li 2023 Machine Translation Natural Language Processing with Deep Learning

Outline1.Pre-Neural Machine Translation2.NeuralMachineTranslation交通大学

1. Pre-Neural Machine Translation 2. Neural Machine Translation Outline

Outline1.Pre-Neural Machine Translation2.Neural Machine Translation交通大学

1. Pre-Neural Machine Translation 2. Neural Machine Translation Outline

Pre-Neural Machine TranslationMachineTranslationMachineTranslation (MT)is thetask oftranslatinga sentencexfromone language (thesourcelanguage)toasentenceyinanotherlanguage(thetargetlanguage)L'hommeestnelibre,etpartoutilestdanslesfersX:通大学y:Manisbornfree,buteverywhereheisinchains-Rousseau

l Machine Translation Machine Translation (MT) is the task of translating a sentence x from one language (the source language) to a sentence y in another language (the target language). x: L'homme est né libre, et partout il est dans lesfers y: Man is born free, but everywhere he is in chains – Rousseau Pre-Neural Machine Translation

Pre-Neural Machine TranslationThe early history of MT:1950sMachinetranslationresearchbeganintheearly1950sonmachineslesspowerfulthanhighschool calculatorsFoundationalworkon automata,formal languages,probabilities,and informationtheory MT heavily funded by military, but basically just simple rule-basedsystemsdoingwordsubstitutionHuman language is more complicated than that, and varies moreacross languages!Little understanding of natural language syntax,semantics,pragmaticsProblem soon appeared intractable

l The early history of MT: 1950s • Machine translation research began in the early 1950s on machines less powerful than high school calculators • Foundational work on automata, formal languages, probabilities, and information theory • MT heavily funded by military, but basically just simple rule-based systems doing word substitution • Human language is more complicated than that, and varies more across languages! • Little understanding of natural language syntax, semantics, pragmatics • Problem soon appeared intractable Pre-Neural Machine Translation

Pre-Neural Machine Translation( SMT )1990s-2010s:Statistical MachineTranslationCoreidea:LearnaprobabilisticmodelfromdataSupposewe'retranslatingFrench→EnglishWewanttofindbestEnglishsentencey,givenFrench sentencexargmax, P(yl)UseBayesRuletobreak thisdownintotwo componentsto belearned separately:= argmax, P(α|y)P(y)Translation ModelLanguage ModelModelshowtowriteModelshowwordsandphrasesgood English (fluency)should be translated (fidelity))LearntfromparalleldataLearntfrommonolingual data

l 1990s-2010s: Statistical Machine Translation （ SMT ） • Core idea: Learn a probabilistic model from data • Suppose we’re translating French → English. • We want to find best English sentence y, given French sentence x • Use Bayes Rule to break this down into two components to be learned separately: Translation Model Models how words and phrases should be translated (fidelity). Learnt from parallel data. Language Model Models how to write good English (fluency). Learnt from monolingual data. Pre-Neural Machine Translation

Pre-Neural Machine Translation(SMT)1990s-2010s:Statistical MachineTranslationQuestion: How to learn translation model P(αy)?First,needlargeamountofparalleldata(e.g,pairsofhuman-translatedFrench/Englishsentences)TheRosettaStoneAncient EgyptianDemotic大味AncientGreek

l 1990s-2010s: Statistical Machine Translation （SMT） • Question: How to learn translation model ? • First, need large amount of parallel data (e.g., pairs of human-translated French/English sentences) Ancient Egyptian Demotic Ancient Greek The Rosetta Stone Pre-Neural Machine Translation

Pre-Neural Machine TranslationLearningalignmentforSMTQuestion:HowtolearntranslationmodelP(αy)fromtheparallelcorpus?Break it down further: Introduce latent a variable into the model: P(c, ay)whereaisthe alignment,i.e.word-levelcorrespondencebetweensource sentencexandtargetsentenceyfliegeMorgerichnachKanadazur KonferenzwillflyTomorotheconferenceCanada

l Learning alignment for SMT • Question: How to learn translation model from the parallel corpus? • Break it down further: Introduce latent a variable into the model: where a is the alignment, i.e. word-level correspondence between source sentence x and target sentence y Pre-Neural Machine Translation

Pre-Neural Machine TranslationWhatisalignment?Alignmentisthecorrespondencebetweenparticularwordsinthetranslatedsentence pair..Typologicaldifferencesbetweenlanguages leadtocomplicatedalignments!.Note:Somewordshavenocounterpartxnano"spurious"oasuoexnapwordsedeLe.JaponJapan.Japanshakensecoueshakenbyparbytwodeux-nouveauxtwonewquakesseismesnewquakesExamples from:"The Mathematics of Statistical MachineTranslation:ParameterEstimation",www.aclweb.org/anthology/J93-2003htnBPOE

l What is alignment? Alignment is the correspondence between particular words in the translated sentence pair. • Typological differences between languages lead to complicated alignments! • Note: Some words have no counterpart Examples from: “The Mathematics of Statistical Machine Translation: Parameter Estimation", Brown et al, 1993. http://www.aclweb.org/anthology/J93-2003 Pre-Neural Machine Translation

Pre-NeuralMachineTranslationAlignmentiscomplexAlignmentcanbemany-to-oneeThe:.Lexne0balance-restewas-Thethe:.appartenaitterritorybalanceof.auxwasthetheaboriginal.autochtonesterritorypeopleofthemany-to-oneaboriginalalignmentspeople:http://www.aclweb.org/anthologv/193-2003Examples from:"TheMathematicsof StatisticalMachineTranslation:Parameter Estimation,Brownetal,1993

l Alignment is complex Alignment can be many-to-one Examples from: “The Mathematics of Statistical Machine Translation: Parameter Estimation", Brown et al, 1993. http://www.aclweb.org/anthology/J93-2003 Pre-Neural Machine Translation