《深度自然语言处理》课程教学课件（Natural language processing with deep learning）06 Language Model & Distributed Representation（3/6）

西安交通大学Natural languageprocessingwith deeplearningXIANHAOTONGUNIVERSITYLanguage Model&Distributed Representation (3)交通大学ChenLicli@xjtu.edu.cn2023

Chen Li cli@xjtu.edu.cn 2023 Language Model & Distributed Representation (3) Natural language processing with deep learning

Outlines1. NNLM2. CBOW3. Skip-gram4. Hierarchical softmax& Negative sampling5. Glove

Outlines 1. NNLM 2. CBOW 3. Skip-gram 4. Hierarchical softmax & Negative sampling 5. Glove

Outlines1.NNLM2. CBOW3. Skip-gram4. Hierarchical softmax& Negative sampling5. Glove

Outlines 1. NNLM 2. CBOW 3. Skip-gram 4. Hierarchical softmax & Negative sampling 5. Glove

Neural NetworkLanguage ModelsReviewthetaskof language modelsx(t)Input: word sequence x(1), x(2)...Output: the probability distribution of the next word P(x(t+1) |x(t),..r(1)NNLM road map (1):HLBL(Mnih,2009)NNLMGloveWord2vec(Turian,(Huang)(Bengio,(Pennington,2010)2012)(Mikolov,2013)C&W2003)2014)(Collobert,2008)1)Task-specificembedding交通大2)X-word2vec3)Understandingandinterpretation

Neural Network Language Models Review the task of language models • Input: word sequence � (1) , � (2) ,., � (�) • Output: the probability distribution of the next word �(� (�+1)|� (�) ,., � (1)) l NNLM road map (1)： NNLM (Bengio, 2003) HLBL (Mnih, 2009) C&W (Collobert, 2008) (Turian, 2010) (Huang, 2012) Word2vec (Mikolov, 2013) Glove (Pennington, 2014) 1) Task-specific embedding 2) X-word2vec 3) Understanding and interpretation

Neural Network Language ModelsReviewthetaskof language modelsInput: word sequence x(1), x(2)... x(t)Output: the probability distribution of the next word P(x(t+1)|x(t),...x(1Neural network model based on window?girlheropened

Neural Network Language Models Review the task of language models • Input: word sequence � (1) , � (2) ,., � (�) • Output: the probability distribution of the next word �(� (�+1)|� (�) ,., � (1)) Neural network model based on window？ girl opened her

Neural Network Language Models交通大学thegirlheropenedabandonFixedWindow

Before her mum arrives the girl opened her _ abandon Fixed window Neural Network Language Models

Neural Network Language Models交通大学Word vector (one-hot,distributedthegirlheropenedrepresentation......)x(2)x(3)x(4)x(1)x(1), x(2),x(3), x(4)

the girl opened her _ � (1) , � (2) ,� (3) , � (4) � (1) � (2) � (3) � (4) Word vector（one-hot, distributed representation.） Neural Network Language Models

Neural Network Language ModelsConcatenate word vectorse = [e(1); e(2); e(3); e(4)]零电电餐2囍6606Word vector (one-hot,distributedthegirlheropenedrepresentation......)x(2)x(3)x(4)x(1)x(1), x(2),x(3), x(4)

� = [� (1) ; � (2) ; � (3) ; � (4)] Concatenate word vectors Neural Network Language Models the girl opened her _ � (1) , � (2) ,� (3) , � (4) � (1) � (2) � (3) � (4) Word vector（one-hot, distributed representation.）

Neural Network Language ModelsHiddenLayerh = f(We+b)WConcatenate word vectorse = [e(1); e(2); e(3); e(4)]Q相电#06Word vector (one-hot,distributedthegirlheropenedrepresentation......x(2)x(3)x(4)x(1)x(1), x(2),x(3), x(4)

� � = [� (1) ; � (2) ; � (3) ; � (4)] Concatenate word vectors ℎ = �(�� + �1) Hidden Layer Neural Network Language Models the girl opened her _ � (1) , � (2) ,� (3) , � (4) � (1) � (2) � (3) � (4) Word vector（one-hot, distributed representation.）

Neural Network Language ModelsbookslaptopsOutput Layer = softmax(Uh + b2) E RIVIZOOUHiddenLayerh = f(We+b)WConcatenate word vectorse = [e(1); e(2); e(3); e(4)]Q0QQ：#606Word vector (one-hot, distributedthegirlheropenedrepresentation......x(1)x(2)x(3)x(4)x(1), x(2),x(3), x(4)

� � books laptops a zoo � = [� (1) ; � (2) ; � (3) ; � (4)] Concatenate word vectors ℎ = �(�� + �1) Hidden Layer � = 𝑠��𝑓�(�ℎ + �2) ∈ ℝ|�| Output Layer Neural Network Language Models the girl opened her _ � (1) , � (2) ,� (3) , � (4) � (1) � (2) � (3) � (4) Word vector（one-hot, distributed representation.）