广东工业大学：《机器学习》课程教学资源（课件讲义）第18讲 变换器模型 Transformer

BERT Transformer 李宏毅 Hung-yi Lee

Transformer 李宏毅 Hung-yi Lee 1

Sequence-to-sequence (Seq2seq) Input a sequence,output a sequence The output length is determined by model. Speech 你好嗎 Recognition N T 機 器學 習 machine learning Machine Translation N N' machine learning 機器學習 Speech Translation Language without text 2

Speech Recognition N T Input a sequence, output a sequence The output length is determined by model. Machine Translation N N’ 機 器 學 習 machine learning Sequence-to-sequence (Seq2seq) Speech Translation 機 器 學 習 machine learning 你 好 嗎 Language without text 2

Hokkien(閩南語、台語) Speech →“母湯” Recognition ???? (Speech of Hokkien) Speech “不行” (Speech of Hokkien) Translation Local soap operas(鄉土割)on YouTube (Speech of Hokkien,Chinese subtitle) Using 1500 hours of data for training 什度，沒有榜棒糖了

Hokkien (閩南語、台語) (Speech of Hokkien) Speech Recognition “母湯” Speech Translation “不行” ??????? … Local soap operas (鄉土劇) on YouTube (Speech of Hokkien, Chinese subtitle) Using 1500 hours of data for training (Speech of Hokkien) 3

Hokkien(閩南語、台語) Background music noises? Don't care Noisy transcriptions? Don't care… ·Phonemes of Hokkien? No "硬train一發” (Ying Train Yi Fa)

Hokkien (閩南語、台語) • Background music & noises? • Noisy transcriptions? • Phonemes of Hokkien? “硬train一發” (Ying Train Yi Fa) Don’t care … Don’t care … No …… 4

咸謝李仲翊同學提供實驗結果 Hokkien(閩南語、台語) 你的身體撐不住 沒事你為什麼要請假 要生了嗎 Answer:不會膩嗎 我有幫廠長拜託 Answer::我拜託廠長了 To learn more:https://sites.google.com/speech.ntut.edu.tw/fsw/home/challenge-2020 5

Hokkien (閩南語、台語) 你 的 身 體 撐 不 住 沒 事 你 為 什 麼 要 請 假 要 生 了 嗎 Answer:不 會 膩 嗎 我 有 幫 廠 長 拜 託 Answer: 我 拜 託 廠 長 了 To learn more: https://sites.google.com/speech.ntut.edu.tw/fsw/home/challenge-2020 感謝 李仲翊 同學提供實驗結果 5

Text-to-Speech 感謝張凱為同學提供實驗結果 Vocoder- (TTS)Synthesis Encoder Post-processing CBHG CBHG RNN RNN RNN Attention RNN RNN RNN Attention is applied to all decoder output Taiwanese Speech Pre-net Pre-net Pre-net Pre-net Synthesis Input embeddings hel l o! <BOS: Source of data:台灣媠聲2.0 Decoder 歡迎來到台大語音處理實驗室 最近肺炎真嚴重·要記得戴口罩、 勤洗手，有病就要看醫生 6

Text-to-Speech (TTS) Synthesis 感謝張凱為同學提供實驗結果 歡迎來到台大語音處理實驗室 Source of data: 台灣媠聲2.0 最近肺炎真嚴重，要記得戴口罩、 勤洗手，有病就要看醫生 Taiwanese Speech Synthesis 6

Seq2seq for Chatbot "Hello!How are you today?" input seq2seq response “HiW [PERSON 1:Hi PERSON 2:Hello How are you today PERSON T:T am good thank you how are you. [PERSON 2:Great,thanks My children and I were just about to watch Game of Thrones. Training [PERSON 1:Nice How old are your children? [PERSON 2:I have four that range in age from 10 to 21.You? data: PERSON 1:I do not have children at the moment. [PERSON 2:That just means you get to keep all the popcorn for yourself. [PERSON 1:And Cheetos at the moment! [PERSON 2:Good choice.Do you watch Game of Thrones? [PERSON 1:No,I do not have much time for TV. [PERSON 2:I usually spend my time painting:but,I love the show

Seq2seq for Chatbot input seq2seq Training data: response “Hi” “Hello! How are you today?” 7

Most Natural Language Processing applications... Question Context Answer What is a major importance ...Southern California is a major major economic of Southern California in relation economic center for the state center to California and the US? of California and the US.... What is the translation Most of the planet is Der GroSteil der from English to German? ocean water. Erde ist Meerwasser Question What is the Harry Potter star Daniel Harry Potter star Answering summary? Radcliffe gains access to a Daniel Radcliffe gets reported 320 million fortune... E320M fortune. (QA) Hypothesis:Product and geography Premise:Conceptually cream are what make cream skimming skimming has two basic Entailment work.Entailment,neutral, dimensions-product and geography. or contradiction? A stirring,funny and finally Is this sentence transporting re-imagining of positive positive or negative? Beauty and the Beast and (sentiment analysis) 1930s horror film. decaNLP QA can be done by seq2seq question,context Seq2seq answer https://arxiv.org/abs/1806.08730 https://arxiv.org/abs/1909.03329

Most Natural Language Processing applications … QA can be done by seq2seq question, context Seq2seq answer https://arxiv.org/abs/1806.08730 https://arxiv.org/abs/1909.03329 Question Answering (QA) (sentiment analysis) 8

Deep Learning for Human Language Processing 深度學習與人類語言處理 One slide for this course Model Model Model Model Model class Model class Source webpage:https://speech.ee.ntu.edu.tw/~hylee/dlhlp/2020-spring.html 9

Deep Learning for Human Language Processing 深度學習與人類語言處理 Source webpage: https://speech.ee.ntu.edu.tw/~hylee/dlhlp/2020-spring.html 9

Seq2seq for Syntactic Parsing Is it a sequence? S Model VP Output NP ADJV Model deep learning is very powerful Input 10

deep learning is very powerful VP S NP ADJV Seq2seq for Syntactic Parsing Model Input Model Output Is it a sequence? 10