广东工业大学：《机器学习》课程教学资源（课件讲义）第21讲 生成式网络模型（自编码器 Deep Auto-encoder）

Unsupervised Learning: Deep Auto-encoder

Unsupervised Learning: Deep Auto-encoder

Unsupervised Learning "We expect unsupervised learning to become far more important in the longer term.Human and animal learning is largely unsupervised:we discover the structure of the world by observing it,not by being told the name of every object." LeCun,Bengio,Hinton,Nature 2015 As I've said in previous statements:most of human and animal learning is unsupervised learning.If intelligence was a cake, unsupervised learning would be the cake,supervised learning would be the icing on the cake,and reinforcement learning would be the cherry on the cake.We know how to make the icing and the cherry,but we don't know how to make the cake. Yann LeCun,March 14,2016(Facebook)

Unsupervised Learning “We expect unsupervised learning to become far more important in the longer term. Human and animal learning is largely unsupervised: we discover the structure of the world by observing it, not by being told the name of every object.” – LeCun, Bengio, Hinton, Nature 2015 As I've said in previous statements: most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we don't know how to make the cake. - Yann LeCun, March 14, 2016 (Facebook)

Auto-encoder Usually <784 Compact NN code representation of Encoder the input object 28X28=784 Learn together Can reconstruct code NN Decoder the original object

Auto-encoder NN Encoder NN Decoder code Compact representation of the input object code Can reconstruct the original object Learn together 28 X 28 = 784 Usually <784

Recap:PCA Minimize(x-龙)2 As close as possible encode decode W hidden layer Input layer (linear） output layer Bottleneck later Output of the hidden layer is the code

Recap: PCA 𝑥 Input layer 𝑊 𝑥 ො 𝑊𝑇 output layer hidden layer (linear) 𝑐 As close as possible Minimize 𝑥 − 𝑥 ො 2 Bottleneck later Output of the hidden layer is the code encode decode

Deep Auto-encoder Symmetric is not necessary. Of course,the auto-encoder can be deep As close as possible W2 Initialize by RBM Code layer-by-layer Reference:Hinton,Geoffrey E.,and Ruslan R.Salakhutdinov."Reducing the dimensionality of data with neural networks."Science 313.5786(2006):504-507

Initialize by RBM layer-by-layer Reference: Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 313.5786 (2006): 504-507 • Of course, the auto-encoder can be deep Deep Auto-encoder Input Layer Layer Layer bottle Output Layer Layer Layer Layer Layer … … Code As close as possible 𝑥 𝑥 ො 𝑊1 𝑊1 𝑇 𝑊2 𝑊2 𝑇 Symmetric is not necessary

Deep Auto-encoder Original /234 Image PCA /334 Deep Auto-encoder /234 1000 73A

Deep Auto-encoder Original Image PCA Deep Auto-encoder 784 784 784 1000 500 250 30 30 250 500 1000 784

空 函 1000

784 784 784 1000 500 250 2 2 250 500 1000 784

More:Contractive auto-encoder Ref:Rifai,Salah,et al."Contractive Auto-encoder auto-encoders:Explicit invariance during feature extraction."Proceedings of the 28th International Conference on De-noising auto-encoder Machine Learning(ICML-11).2011. As close as possible ◆ encode decode Add noise Vincent,Pascal,et al."Extracting and composing robust features with denoising autoencoders."/CML,2008

Auto-encoder • De-noising auto-encoder 𝑥 𝑥 ො 𝑐 encode decode Add noise 𝑥′ As close as possible More: Contractive auto-encoder Ref: Rifai, Salah, et al. "Contractive auto-encoders: Explicit invariance during feature extraction.“ Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011. Vincent, Pascal, et al. "Extracting and composing robust features with denoising autoencoders." ICML, 2008

Deep Auto-encoder-Example NN C 5 Encoder 3 8 6 0,° 878 8e4 23 。 PCA降到 32-dim 5 .3 Pixel -tSNE 0 89

Deep Auto-encoder - Example Pixel -> tSNE 𝑐 NN Encoder PCA 降到 32-dim

Auto-encoder Text Retrieval Vector Space Model Bag-of-word this is 1 word string: query “This is an apple'”e 0 an 1 apple 1 pen 0 document Semantics are not considered

Auto-encoder – Text Retrieval word string: “This is an apple” … this is a an apple pen 1 1 0 1 1 0 Bag-of-word Semantics are not considered. Vector Space Model document query