《深度自然语言处理》课程教学课件（Natural language processing with deep learning）05 Language Model & Distributed Representation（2/6）

西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (2)交通大学ChenLicli@xjtu.edu.cn2023

Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (2) Natural language processing with deep learning

Outlines1. Word2Vector (W2V)2. Training LM3. Evaluating LM

Outlines 1. Word2Vector (W2V) 2. Training LM 3. Evaluating LM

Outlines1.Word2Vector(W2V2. Training LM3. Evaluating LM

Outlines 1. Word2Vector (W2V) 2. Training LM 3. Evaluating LM

WordEmbeddingConstruct a dense vector to represent each word, making this vector similartothevectorofwordsinsimilarcontexts0.2860.792-0.177banking:-0.1070.109-0.5420.3490.271

Word Embedding • Construct a dense vector to represent each word, making this vector similar to the vector of words in similar contexts. 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 banking =

WordEmbeddingConstruct a dense vectorto representeach word,makingthis vector similartothevectorofwordsinsimilarcontexts0.2860.792-0.177banking:-0.1070.109-0.5420.3490.271Word vectorsaresometimes calledword embeddings orword representationsThese are distributed representations

Word Embedding • Construct a dense vector to represent each word, making this vector similar to the vector of words in similar contexts. 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 banking = Word vectors are sometimes called word embeddings or word representations. These are distributed representations

Word vectors representword meaningvisualizationneedhelpcomeqo0.286take0.792keepqive-0.177getmakemeetcontinue-0.107seeexpect=0.109wantbecomeexpect-0.542thinkremainsay0.349areis0.271beweraas0.487beingbeenhadhashave

Word vectors represent word meaning- visualization 0.286 0.792 −0.177 −0.107 0.109 −0.542 0.349 0.271 0.487 expect =

Word vectorsrepresentword meaning-visualizationfoodeatlaptopDistributional vectorsrepresentedby a D-dimensionalvector where D<<V,where V is size of VocabularyThegreatest contribution of distributed representation is tomakerelatedorsimilarwordscloserindistanceand solve the problem of curseof dimensiontoa certain extent

Word vectors represent word meaning- visualization The greatest contribution of distributed representation is to make related or similar words closer in distance and solve the problem of curse of dimension to a certain extent

Word vectorsrepresentword meaning-visualization著名的类比King-Man+Woman=QueenMale-FemaleThegreatest contributionof distributed representationisto makerelatedorsimilarwordscloserindistanceandsolvetheproblemofcurseofdimensiontoacertainextent

Word vectors represent word meaning- visualization The greatest contribution of distributed representation is to make related or similar words closer in distance and solve the problem of curse of dimension to a certain extent

Distributed and Distributional RepresentationNotes:Distributed representation refers to the form of textrepresentation,whichislowdimensionaland densecontinuousvectorDistributional Representationis akind of methodto obtaintextrepresentation,whichuses co-occurrence matrixto obtainthesemanticrepresentationofwords.Eachlineof co-occurrencematrixcanbe regarded as the vector representation ofcorrespondingwords逸大

Distributed and Distributional Representation Distributed representation refers to the form of text representation, which is low dimensional and dense continuous vector Distributional Representation is a kind of method to obtain text representation, which uses co-occurrence matrix to obtain the semantic representation of words. Each line of co-occurrence matrix can be regarded as the vector representation of corresponding words. Notes:

Word2vecWord2vecTomasMikolov.etal.2013交通大学1.MikolovT,ChenK,Corrado G,etal.Efficient estimation ofword representations in vectorspace[J].arXivpreprintarXiv:1301.3781,2013

Word2vec • Tomáš Mikolov, et al. 2013 Word2vec 1. Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space[J]. arXiv preprint arXiv:1301.3781, 2013