《深度自然语言处理》课程教学课件（Natural language processing with deep learning）04 Language Model & Distributed Representation（1/6）

西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (1)交通大学ChenLicli@xjtu.edu.cn2023

Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (1) Natural language processing with deep learning

Outlines1. Traditional Language Model (LM)2.Distributed Representation交道大学

Outlines 1. Traditional Language Model (LM) 2. Distributed Representation

Outlines1.Traditional Language Model (LM)2.Distributed Representation交道大学

Outlines 1. Traditional Language Model (LM) 2. Distributed Representation

OverviewAnewNLPtaskLanguage modelingRelated conceptsMarkov chainn-gram modelMaximumLikelihoodEstimation(MLELaplace smoothing, additive smoothingGood Turing

• Language modeling A new NLP task Overview • Markov chain • n-gram model • Maximum Likelihood Estimation (MLE) • Laplace smoothing, additive smoothing • Good Turing Related concepts

LanguageModelDefinition: compute probability distribution over the words in a sentence交道大学

• Definition: compute probability distribution over the words in a sentence Language Model

LanguageModelDefinition: compute probability distribution over the words in a sentenceApplication: predicting the next word in a sequence given the sequence ofwordsalreadypresentbooklaptopthe girl opened herdoormind交通大学

• Definition: compute probability distribution over the words in a sentence • Application: predicting the next word in a sequence given the sequence of words already present Language Model the girl opened her book laptop door mind

LanguageModelDefinition: compute probability distribution overthe words in a sentenceApplication: predicting the next word in a sequence given the sequence ofwords already presentbooklaptopthe girl opened herdoormindFormal Definition:Given a sequence of words x(1), x(2),., x(t), estimate the probabilitydistribution of the next word x(t+1)P(x(t+1)|x(t), .. (1)x(t+1) could be any word in the vocabulary V = (W1, .., Wivi} Language Model is a system like this

Language Model the girl opened her book laptop door • Formal Definition: Given a sequence of words � (1) , � (2) ,., � (�) , estimate the probability distribution of the next word � (t+1) 。 � � (�+1)|� (�) , ., � (1) � (t+1) could be any word in the vocabulary V = {�1, ., �|�|} 。 • Language Model is a system like this mind • Definition: compute probability distribution over the words in a sentence • Application: predicting the next word in a sequence given the sequence of words already present

LanguageModelLanguage Model could also be regarded as a system assigning a probabilitytoatext.交道大学

• Language Model could also be regarded as a system assigning a probability to a text. Language Model

LanguageModelLanguage Model could also be regarded as a system assigning a probabilityto a text.For example, the probability of a text x(1), x(2) ,., x(T) (Language Model)couldbedenotedasP(x(1), . (T) = P(x(1) × P(x(2)|x(1) × . × P(x(T)|x(T-1),.. (1)= I/= P(x() |x(t-1), ., (1)ThisiswhatLanguageModelcomputes

• Language Model could also be regarded as a system assigning a probability to a text. • For example, the probability of a text � (1) , � (2) ,., � (�) (Language Model) could be denoted as P � (1) , ., � (�) = � � (1) × � � (2) � (1) × ∙∙∙ × � � (�) � (�−1) , ., � (1) = �=1 � � � (�) � (�−1) , ., � (1) Language Model This is what Language Model computes

LanguageModelHave youeverused Language Model?交通大学

• Have you ever used Language Model？ Language Model