李宏毅ML lecture-16 unsupervised Learning Auto-encoder

Auto-encoder

两个神经网络，一个降维，压缩（compact）即Encoder, 把图片转换为code。一个根据code重建（reconstruct）原始的图片。

这两个神经网络单独无法训练，必须同时训练。

Recap: PCA

训练一个三层神经网络实现PCA，使输入和输出接近，同时存在一个Bottleneck（瓶颈） later layer。这个layer的参数小于input layer的参数，这样就实现了encode和decode。

Deep Auto-encoder

增加一些隐藏层，就实现了Deep Auto-encoder.

Reference: Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. “Reducing the dimensionality of data with neural networks.” Science 313.5786 (2006): 504-507

Deep Auto-encoder 结果非常好

提高auto-encoder的能力，可以选择增加噪声

Ref: Rifai, Salah, et al. "Contractive auto-encoders: Explicit invariance during feature extraction.“ Proceedings of the 28th International Conference on Machine Learning (ICML-11). 2011.
Vincent, Pascal, et al. “Extracting and composing robust features with denoising autoencoders.” ICML, 2008.

Vector Space Model

搜索图片

直接搜索效果并不好

Reference: Krizhevsky, Alex, and Geoffrey E. Hinton. “Using very deep autoencoders for content-based image retrieval.” ESANN. 2011.

建立一个auto-encoder

在code上搜寻

pre-training

监督学习使用的不多了。
可以用于半监督学习，非标注数据大量的时候。

Learning More - Restricted Boltzmann Machine

• Neural networks [5.1] : Restricted Boltzmann machine – definition https://www.youtube.com/watch?v=p4Vh_zMw-HQ&index=36&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH
• Neural networks [5.2] : Restricted Boltzmann machine – inference https://www.youtube.com/watch?v=lekCh_i32iE&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&index=37
• Neural networks [5.3] : Restricted Boltzmann machine - free energy https://www.youtube.com/watch?v=e0Ts_7Y6hZU&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&index=38

Learning More - Deep Belief Network

• Neural networks [7.7] : Deep learning - deep belief network https://www.youtube.com/watch?v=vkb6AWYXZ5I&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&index=57
• Neural networks [7.8] : Deep learning - variational bound https://www.youtube.com/watch?v=pStDscJh2Wo&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&index=58
• Neural networks [7.9] : Deep learning - DBN pre-training https://www.youtube.com/watch?v=35MUlYCColk&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH&index=59

auto-encoding 在CNN上

自动编码机

反池化

反卷积