Deep Learning for Images, Soundwaves, and Character Strings

Опубликовано 17 августа 2016, 22:36

Deep neural networks that contain many layers of non-linear feature detectors fell out of favor because it was hard to get enough labeled data to train many millions of parameters and difficult to optimize the connection weights really well. Both of these problems can be overcome by first training a multi-layer belief net to form a top-down generative model of unlabeled input data and then using the features discovered by the belief net to initialize a bottom-up neural net. The neural net can then be discriminatively fine-tuned on a smaller set of labelled data. Marc'Aurelio Ranzato has recently used this deep learning method to create a very good generative model of natural images and Navdeep Jaitly has used it to learn features that are derived directly from the raw sound wave and outperform the features that are usually used for phoneme recognition. An alternative way to deal with the difficult optimization problem is to develop a more sophisticated optimizer that works really well for artificial neural networks -- something that the optimization community has often suggested but never done. Ilya Sutskever has recently used an excellent 'Hessian free' optimizer developed by James Martens to learn a recurrent neural network that predicts the next character in a string. 'He was elected President during the Revolutionary War and forgave Opus Paul at Rome' is an example of what this neural net generates after being trained on character strings from Wikipedia.